| *** camunoz has joined #openstack-infra | 00:00 | |
| *** john_studarus has quit IRC | 00:01 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix zuul-web sql connections https://review.openstack.org/572210 | 00:04 |
|---|---|---|
| *** heyongli has quit IRC | 00:05 | |
| *** heyongli has joined #openstack-infra | 00:05 | |
| *** eernst_ has quit IRC | 00:11 | |
| *** eernst has joined #openstack-infra | 00:12 | |
| *** ihar has quit IRC | 00:14 | |
| *** heyongli has quit IRC | 00:15 | |
| *** SumitNaiksatam has quit IRC | 00:15 | |
| *** heyongli has joined #openstack-infra | 00:15 | |
| *** rwsu has quit IRC | 00:15 | |
| *** kien-ha has joined #openstack-infra | 00:17 | |
| *** eernst_ has joined #openstack-infra | 00:19 | |
| *** eernst has quit IRC | 00:19 | |
| *** eernst_ has quit IRC | 00:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Handle websocket client hangups better https://review.openstack.org/573034 | 00:22 |
| corvus | they collided; that's a trivial resolution fix. i'll self-approve. | 00:22 |
| *** threestrands has quit IRC | 00:25 | |
| *** heyongli has quit IRC | 00:25 | |
| *** heyongli has joined #openstack-infra | 00:25 | |
| *** threestrands has joined #openstack-infra | 00:28 | |
| *** rwsu has joined #openstack-infra | 00:28 | |
| *** heyongli has quit IRC | 00:35 | |
| *** heyongli has joined #openstack-infra | 00:36 | |
| *** ihar has joined #openstack-infra | 00:40 | |
| *** heyongli has quit IRC | 00:46 | |
| *** heyongli has joined #openstack-infra | 00:46 | |
| *** heyongli has quit IRC | 00:56 | |
| *** heyongli has joined #openstack-infra | 00:56 | |
| *** rpioso is now known as rpioso|afk | 01:00 | |
| *** camunoz has quit IRC | 01:04 | |
| openstackgerrit | Merged openstack-infra/zuul master: Handle websocket client hangups better https://review.openstack.org/573034 | 01:04 |
| *** heyongli has quit IRC | 01:06 | |
| *** heyongli has joined #openstack-infra | 01:06 | |
| *** felipemonteiro has joined #openstack-infra | 01:12 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:14 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:14 | |
| *** hongbin has joined #openstack-infra | 01:16 | |
| *** kjackal has quit IRC | 01:16 | |
| *** heyongli has quit IRC | 01:16 | |
| *** heyongli has joined #openstack-infra | 01:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: webapp: fix browser return https://review.openstack.org/573053 | 01:17 |
| *** felipemonteiro has quit IRC | 01:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: webapp: fix browser return https://review.openstack.org/573053 | 01:20 |
| *** kjackal has joined #openstack-infra | 01:22 | |
| *** felipemonteiro_ has quit IRC | 01:24 | |
| *** heyongli has quit IRC | 01:27 | |
| *** heyongli has joined #openstack-infra | 01:27 | |
| *** threestrands has quit IRC | 01:35 | |
| *** threestrands has joined #openstack-infra | 01:35 | |
| *** threestrands has quit IRC | 01:36 | |
| *** heyongli has quit IRC | 01:37 | |
| *** threestrands has joined #openstack-infra | 01:37 | |
| *** heyongli has joined #openstack-infra | 01:37 | |
| *** kiennt26 has joined #openstack-infra | 01:37 | |
| *** threestrands has quit IRC | 01:38 | |
| *** threestrands has joined #openstack-infra | 01:38 | |
| *** threestrands has quit IRC | 01:38 | |
| *** threestrands has joined #openstack-infra | 01:38 | |
| *** threestrands has quit IRC | 01:39 | |
| *** threestrands has joined #openstack-infra | 01:40 | |
| *** threestrands has quit IRC | 01:40 | |
| *** threestrands has joined #openstack-infra | 01:40 | |
| corvus | 2018-06-06 22:15:02,694 p=28922 u=root | ERROR! Attempted to execute "/etc/ansible/hosts/openstack" as inventory script: Inventory script (/etc/ansible/hosts/openstack) had an execution error: Traceback (most | 01:40 |
| corvus | that's on puppetmaster | 01:40 |
| *** threestrands has quit IRC | 01:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: zuul grafana: fix load average graph https://review.openstack.org/573038 | 01:42 |
| clarkb | corvus: does foreground ansible work? it was working fir me earlier today to update git | 01:42 |
| *** edmondsw has joined #openstack-infra | 01:42 | |
| corvus | clarkb: nope | 01:43 |
| corvus | sudo ansible 'zuul01.openstack.org' -m shell -a 'hostname' | 01:43 |
| corvus | same error | 01:43 |
| corvus | the line i pasted earlier was the first occurence, so that should time-bound us | 01:44 |
| corvus | here's the full tb: http://paste.openstack.org/show/722845/ | 01:44 |
| clarkb | I'm guessing a lib just updated shade or oscc | 01:45 |
| corvus | i don't see any puppet-user actions logged related to that | 01:45 |
| clarkb | actually no, fungi was doing clouds.yaml work for new cloud that could be related | 01:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Ensure cherrypy listen port is int type https://review.openstack.org/573060 | 01:47 |
| *** edmondsw has quit IRC | 01:47 | |
| *** heyongli has quit IRC | 01:47 | |
| *** heyongli has joined #openstack-infra | 01:47 | |
| *** kjackal has quit IRC | 01:48 | |
| *** dpawlik has joined #openstack-infra | 01:49 | |
| corvus | i'm confused | 01:50 |
| corvus | i don't see a change for that. clouds.yaml doesn't match what's in system-config, but puppetmaster isn't in emergency | 01:50 |
| corvus | oh, maybe the file was manually changed and that broke ansible enough that it couldn't run puppet and re-write the file | 01:51 |
| clarkb | ya | 01:51 |
| corvus | clarkb, fungi: this timestamp is suspiciously close to when things broke: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-06-06.log.html#t2018-06-06T21:56:08 | 01:52 |
| *** dpawlik has quit IRC | 01:53 | |
| corvus | clarkb, fungi: i'm going to try to manually remove the packethost entry and see if that gets things working again | 01:54 |
| corvus | ansible -m shell works now | 01:55 |
| corvus | i guess i'll check back in 35 minutes and see if zuul has been updated and i can restart zuul-web | 01:56 |
| *** r-daneel has joined #openstack-infra | 01:56 | |
| *** heyongli has quit IRC | 01:57 | |
| *** heyongli has joined #openstack-infra | 01:58 | |
| *** heyongli has quit IRC | 02:08 | |
| *** aeng has quit IRC | 02:08 | |
| *** heyongli has joined #openstack-infra | 02:08 | |
| *** threestrands has joined #openstack-infra | 02:11 | |
| *** threestrands has quit IRC | 02:11 | |
| *** threestrands has joined #openstack-infra | 02:11 | |
| *** heyongli has quit IRC | 02:18 | |
| *** heyongli has joined #openstack-infra | 02:18 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for tagged attach bug 1775491 https://review.openstack.org/573069 | 02:19 |
| openstack | bug 1775491 in tempest "test_tagged_attachment randomly fails with: sequence or collection is not empty: [{u'type': u'disk', u'bus': u'pci', u'address': u'0000:00:07.0', u'serial': u'56e9a4a0-51fc-4319-845b-02c6a21d5b16', u'tags': [u'volume-tag']}]" [Undecided,New] https://launchpad.net/bugs/1775491 | 02:19 |
| *** slaweq has joined #openstack-infra | 02:21 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 02:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 02:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 02:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/labels route https://review.openstack.org/553979 | 02:25 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/nodes route https://review.openstack.org/553998 | 02:25 |
| *** slaweq has quit IRC | 02:26 | |
| *** heyongli has quit IRC | 02:28 | |
| *** heyongli has joined #openstack-infra | 02:28 | |
| ianw | do we ahve any examples of puppet matching on hostnames? https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/grafana.pp#n56 clearly doens't work | 02:29 |
| clarkb | ianw I think we largely do that through top level site.pp matchers | 02:30 |
| ianw | oooohh, it probably helps if you actually use the right hostname :/ | 02:31 |
| ianw | i'd actually be pretty confident about just upgrading grafana01 after playing | 02:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: grafana: fix hostname typo https://review.openstack.org/573072 | 02:34 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for tagged attach bug 1775491 https://review.openstack.org/573069 | 02:34 |
| openstack | bug 1775491 in tempest "test_tagged_attachment randomly fails with: sequence or collection is not empty: [{u'type': u'disk', u'bus': u'pci', u'address': u'0000:00:07.0', u'serial': u'56e9a4a0-51fc-4319-845b-02c6a21d5b16', u'tags': [u'volume-tag']}]" [High,Confirmed] https://launchpad.net/bugs/1775491 | 02:34 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:35 | |
| *** Kevin_Zheng has joined #openstack-infra | 02:38 | |
| *** heyongli has quit IRC | 02:38 | |
| *** heyongli has joined #openstack-infra | 02:39 | |
| *** rlandy|rover has quit IRC | 02:39 | |
| *** heyongli has quit IRC | 02:49 | |
| *** roman_g has quit IRC | 02:49 | |
| *** heyongli has joined #openstack-infra | 02:49 | |
| *** d0ugal_ has joined #openstack-infra | 02:49 | |
| *** eernst has joined #openstack-infra | 02:50 | |
| *** d0ugal has quit IRC | 02:51 | |
| *** mriedem has quit IRC | 02:52 | |
| *** psachin has joined #openstack-infra | 02:52 | |
| *** kiennt26 has quit IRC | 02:53 | |
| *** heyongli has quit IRC | 02:59 | |
| *** heyongli has joined #openstack-infra | 02:59 | |
| *** yamahata has quit IRC | 02:59 | |
| *** lifeless_ has joined #openstack-infra | 03:03 | |
| *** lifeless has quit IRC | 03:03 | |
| *** boris_42_ has quit IRC | 03:05 | |
| *** kien-ha has quit IRC | 03:05 | |
| *** iyamahat_ has quit IRC | 03:08 | |
| *** heyongli has quit IRC | 03:09 | |
| openstackgerrit | Merged openstack-infra/system-config master: grafana: fix hostname typo https://review.openstack.org/573072 | 03:09 |
| *** heyongli has joined #openstack-infra | 03:09 | |
| *** rosmaita has quit IRC | 03:10 | |
| *** threestrands has quit IRC | 03:13 | |
| *** heyongli has quit IRC | 03:19 | |
| *** heyongli has joined #openstack-infra | 03:20 | |
| *** rh-jelabarre has quit IRC | 03:22 | |
| *** Bhujay has joined #openstack-infra | 03:28 | |
| *** heyongli has quit IRC | 03:30 | |
| *** heyongli has joined #openstack-infra | 03:30 | |
| *** edmondsw has joined #openstack-infra | 03:30 | |
| *** rh-jelabarre has joined #openstack-infra | 03:31 | |
| *** kiennt26 has joined #openstack-infra | 03:32 | |
| *** ramishra has joined #openstack-infra | 03:32 | |
| *** edmondsw has quit IRC | 03:35 | |
| corvus | restarted zuul-web from master | 03:37 |
| *** germs has quit IRC | 03:37 | |
| *** zhangfei has joined #openstack-infra | 03:37 | |
| *** germs has joined #openstack-infra | 03:38 | |
| *** heyongli has quit IRC | 03:40 | |
| *** heyongli has joined #openstack-infra | 03:40 | |
| *** lpetrut has joined #openstack-infra | 03:42 | |
| *** dpawlik has joined #openstack-infra | 03:49 | |
| *** heyongli has quit IRC | 03:50 | |
| *** heyongli has joined #openstack-infra | 03:50 | |
| *** dpawlik has quit IRC | 03:54 | |
| *** eernst has quit IRC | 04:00 | |
| *** yamamoto has joined #openstack-infra | 04:00 | |
| *** heyongli has quit IRC | 04:00 | |
| *** heyongli has joined #openstack-infra | 04:01 | |
| openstackgerrit | kaka proposed openstack-infra/zuul-website master: Trivial: update url to new url https://review.openstack.org/573079 | 04:01 |
| *** eernst has joined #openstack-infra | 04:02 | |
| *** udesale has joined #openstack-infra | 04:02 | |
| *** ccamacho has quit IRC | 04:05 | |
| *** jcoufal has joined #openstack-infra | 04:07 | |
| *** jcoufal has quit IRC | 04:08 | |
| *** heyongli has quit IRC | 04:11 | |
| *** heyongli has joined #openstack-infra | 04:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ensure cherrypy listen port is int type https://review.openstack.org/573060 | 04:13 |
| *** rh-jelabarre has quit IRC | 04:13 | |
| *** lpetrut has quit IRC | 04:14 | |
| *** germs has quit IRC | 04:16 | |
| *** rh-jelabarre has joined #openstack-infra | 04:20 | |
| *** osinfra-grafana has joined #openstack-infra | 04:21 | |
| *** heyongli has quit IRC | 04:21 | |
| *** heyongli has joined #openstack-infra | 04:21 | |
| *** osinfra-grafana has quit IRC | 04:21 | |
| *** kjackal has joined #openstack-infra | 04:22 | |
| *** hongbin has quit IRC | 04:26 | |
| *** links has joined #openstack-infra | 04:28 | |
| *** annp has joined #openstack-infra | 04:29 | |
| *** openstackgrafana has joined #openstack-infra | 04:31 | |
| *** heyongli has quit IRC | 04:31 | |
| *** heyongli has joined #openstack-infra | 04:31 | |
| *** dhajare has joined #openstack-infra | 04:32 | |
| *** openstackgrafana has quit IRC | 04:35 | |
| *** masayukig1 has left #openstack-infra | 04:36 | |
| *** openstackgrafana has joined #openstack-infra | 04:36 | |
| *** openstackgrafana has quit IRC | 04:36 | |
| *** openstackgrafana has joined #openstack-infra | 04:38 | |
| *** openstackgrafana has joined #openstack-infra | 04:39 | |
| *** openstackgrafana has quit IRC | 04:39 | |
| *** openstackgrafana has joined #openstack-infra | 04:39 | |
| *** kjackal has quit IRC | 04:40 | |
| *** heyongli has quit IRC | 04:41 | |
| *** heyongli has joined #openstack-infra | 04:42 | |
| *** masayukig_ has quit IRC | 04:43 | |
| *** openstackgrafana has quit IRC | 04:44 | |
| *** openstackgrafana has joined #openstack-infra | 04:45 | |
| openstackgrafana | Testing grafana alerts! | 04:45 |
| openstackgrafana | Testing grafana alerts! | 04:45 |
| ianw | openstackgrafana: thank you for the alert | 04:45 |
| *** heyongli has quit IRC | 04:52 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 04:52 | |
| *** heyongli has joined #openstack-infra | 04:52 | |
| *** SumitNaiksatam has quit IRC | 04:53 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 04:53 | |
| *** dhajare has quit IRC | 04:56 | |
| *** dhajare_ has joined #openstack-infra | 04:56 | |
| *** heyongli has quit IRC | 05:02 | |
| *** heyongli has joined #openstack-infra | 05:02 | |
| *** auristor has quit IRC | 05:03 | |
| *** pgadiya has joined #openstack-infra | 05:08 | |
| *** pgadiya has quit IRC | 05:08 | |
| *** auristor has joined #openstack-infra | 05:10 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** heyongli has quit IRC | 05:12 | |
| *** heyongli has joined #openstack-infra | 05:12 | |
| *** slaweq has quit IRC | 05:15 | |
| *** ykarel|training has joined #openstack-infra | 05:17 | |
| *** edmondsw has joined #openstack-infra | 05:19 | |
| *** vivsoni has quit IRC | 05:19 | |
| *** vivsoni has joined #openstack-infra | 05:19 | |
| *** eernst has quit IRC | 05:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove direct tagging/branching for QA repos https://review.openstack.org/572750 | 05:21 |
| *** heyongli has quit IRC | 05:22 | |
| *** e0ne has joined #openstack-infra | 05:23 | |
| *** heyongli has joined #openstack-infra | 05:23 | |
| *** edmondsw has quit IRC | 05:23 | |
| *** e0ne has quit IRC | 05:25 | |
| *** slaweq has joined #openstack-infra | 05:28 | |
| *** Bhujay has quit IRC | 05:28 | |
| *** slaweq has quit IRC | 05:32 | |
| *** heyongli has quit IRC | 05:33 | |
| *** heyongli has joined #openstack-infra | 05:33 | |
| *** iranzo has joined #openstack-infra | 05:41 | |
| *** heyongli has quit IRC | 05:43 | |
| *** heyongli has joined #openstack-infra | 05:43 | |
| openstackgerrit | kaka proposed openstack-infra/zuul-website master: Trivial: update url to new url https://review.openstack.org/573079 | 05:44 |
| *** cshastri has joined #openstack-infra | 05:47 | |
| *** dteselkin has left #openstack-infra | 05:48 | |
| *** dpawlik has joined #openstack-infra | 05:50 | |
| *** gyee has quit IRC | 05:53 | |
| *** heyongli has quit IRC | 05:53 | |
| *** heyongli has joined #openstack-infra | 05:53 | |
| *** pcichy has quit IRC | 05:54 | |
| *** dpawlik has quit IRC | 05:55 | |
| *** pcichy has joined #openstack-infra | 06:00 | |
| *** ykarel|training has quit IRC | 06:02 | |
| *** heyongli has quit IRC | 06:03 | |
| *** heyongli has joined #openstack-infra | 06:04 | |
| *** ykarel|training has joined #openstack-infra | 06:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/573088 | 06:07 |
| *** heyongli has quit IRC | 06:14 | |
| *** heyongli has joined #openstack-infra | 06:14 | |
| *** ykarel|training has quit IRC | 06:14 | |
| *** pblaho has joined #openstack-infra | 06:16 | |
| *** lpetrut has joined #openstack-infra | 06:23 | |
| *** heyongli has quit IRC | 06:24 | |
| *** heyongli has joined #openstack-infra | 06:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/573088 | 06:28 |
| *** dklyle has quit IRC | 06:29 | |
| *** namnh has joined #openstack-infra | 06:31 | |
| *** e0ne has joined #openstack-infra | 06:31 | |
| *** heyongli has quit IRC | 06:34 | |
| *** heyongli has joined #openstack-infra | 06:35 | |
| *** pcaruana has joined #openstack-infra | 06:35 | |
| *** zoli is now known as zoli|wfh | 06:38 | |
| *** zoli|wfh is now known as zoli | 06:38 | |
| *** e0ne has quit IRC | 06:41 | |
| *** hashar has joined #openstack-infra | 06:41 | |
| *** jesslampe has quit IRC | 06:43 | |
| *** heyongli has quit IRC | 06:44 | |
| *** jesusaur has quit IRC | 06:44 | |
| *** heyongli has joined #openstack-infra | 06:45 | |
| *** ccamacho has joined #openstack-infra | 06:52 | |
| *** aojea has joined #openstack-infra | 06:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/labels.html web page https://review.openstack.org/553980 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/nodes.html web page https://review.openstack.org/553999 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add jobs graph rendering https://review.openstack.org/537869 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add project pipeline rendering https://review.openstack.org/537871 | 06:54 |
| *** heyongli has quit IRC | 06:55 | |
| *** heyongli has joined #openstack-infra | 06:55 | |
| *** jesusaur has joined #openstack-infra | 06:56 | |
| *** slaweq has joined #openstack-infra | 07:00 | |
| *** slaweq has quit IRC | 07:00 | |
| *** slaweq has joined #openstack-infra | 07:00 | |
| *** heyongli has quit IRC | 07:05 | |
| *** heyongli has joined #openstack-infra | 07:05 | |
| *** edmondsw has joined #openstack-infra | 07:07 | |
| *** e0ne has joined #openstack-infra | 07:10 | |
| *** armaan has joined #openstack-infra | 07:10 | |
| *** e0ne has quit IRC | 07:11 | |
| *** edmondsw has quit IRC | 07:11 | |
| *** amoralej|off is now known as amoralej | 07:14 | |
| *** heyongli has quit IRC | 07:15 | |
| *** heyongli has joined #openstack-infra | 07:15 | |
| *** jesslampe has joined #openstack-infra | 07:21 | |
| *** jistr|mtgs is now known as jistr | 07:22 | |
| *** yamahata has joined #openstack-infra | 07:23 | |
| *** jiapei has joined #openstack-infra | 07:24 | |
| *** heyongli has quit IRC | 07:25 | |
| *** nmathew has joined #openstack-infra | 07:26 | |
| *** heyongli has joined #openstack-infra | 07:26 | |
| *** tosky has joined #openstack-infra | 07:26 | |
| *** openstackgrafana has quit IRC | 07:29 | |
| *** dklyle has joined #openstack-infra | 07:32 | |
| *** openstackgrafana has joined #openstack-infra | 07:33 | |
| *** openstackgrafana has joined #openstack-infra | 07:33 | |
| *** openstackgrafana has quit IRC | 07:34 | |
| *** heyongli has quit IRC | 07:36 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** heyongli has joined #openstack-infra | 07:36 | |
| *** openstackgrafana has joined #openstack-infra | 07:36 | |
| *** openstackgrafana has quit IRC | 07:38 | |
| *** openstackgrafana has joined #openstack-infra | 07:38 | |
| openstackgrafana | Testing grafana alerts! | 07:38 |
| *** openstackgrafana has quit IRC | 07:38 | |
| *** openstackgrafana has joined #openstack-infra | 07:39 | |
| *** openstackgrafana has quit IRC | 07:39 | |
| *** openstackgrafana has joined #openstack-infra | 07:42 | |
| ltomasbo | hi folks, any of you know if there is any problem with the gates? I have a patch that got +2 from Zuul yesterday and it is not being merged (https://review.openstack.org/#/c/562249/) | 07:45 |
| *** heyongli has quit IRC | 07:46 | |
| *** heyongli has joined #openstack-infra | 07:46 | |
| AJaeger | ltomasbo: you might want to rebase it first | 07:47 |
| AJaeger | ltomasbo: seems it's fine now | 07:48 |
| ltomasbo | AJaeger, ok, we flipped +W to see if that helps | 07:48 |
| *** openstackgrafana has joined #openstack-infra | 07:48 | |
| ltomasbo | AJaeger, going again through the gates, lets see if it merges after passing them | 07:49 |
| *** dpawlik has joined #openstack-infra | 07:49 | |
| *** zhangfei has quit IRC | 07:51 | |
| *** jaosorior has joined #openstack-infra | 07:52 | |
| *** dhajare_ has quit IRC | 07:53 | |
| *** dpawlik has quit IRC | 07:54 | |
| *** heyongli has quit IRC | 07:56 | |
| *** heyongli has joined #openstack-infra | 07:56 | |
| *** jpich has joined #openstack-infra | 07:57 | |
| AJaeger | ltomasbo: you see it's based on a change that has been updated since, sometimes gerrit does not like that | 07:58 |
| ltomasbo | AJaeger, was based on a change that was merged already, right? | 08:01 |
| ltomasbo | AJaeger, anyway, I'll rebase if not working this time | 08:02 |
| *** zhangfei has joined #openstack-infra | 08:02 | |
| *** alexchadin has joined #openstack-infra | 08:04 | |
| *** gfidente has joined #openstack-infra | 08:05 | |
| *** heyongli has quit IRC | 08:06 | |
| *** heyongli has joined #openstack-infra | 08:07 | |
| hwoarang | clarkb: thank you for the new tag. i see that opensuse-150 nodes are still in 'queued' state and no jobs are starting | 08:07 |
| *** dhajare has joined #openstack-infra | 08:08 | |
| hwoarang | so maybe it takes a bit longer to become alive or maybe it's something else. i don't know if anyone can quickly look on the status of these nodes | 08:08 |
| *** jaosorior has quit IRC | 08:10 | |
| hwoarang | or maybe the images haven't been built with the new glean yet | 08:10 |
| hwoarang | which could be the most likely cause | 08:11 |
| *** roman_g has joined #openstack-infra | 08:11 | |
| *** shardy has joined #openstack-infra | 08:14 | |
| *** openstackgrafana has quit IRC | 08:16 | |
| *** heyongli has quit IRC | 08:17 | |
| *** heyongli has joined #openstack-infra | 08:17 | |
| *** openstackgrafana has joined #openstack-infra | 08:18 | |
| *** openstackgrafana has quit IRC | 08:18 | |
| *** openstackgrafana has joined #openstack-infra | 08:19 | |
| *** openstackgrafana has quit IRC | 08:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 08:20 |
| *** openstackgrafana has joined #openstack-infra | 08:21 | |
| *** openstackgrafana has joined #openstack-infra | 08:21 | |
| *** openstackgrafana has quit IRC | 08:22 | |
| *** dpawlik has joined #openstack-infra | 08:25 | |
| *** ykarel|training has joined #openstack-infra | 08:26 | |
| *** heyongli has quit IRC | 08:27 | |
| *** heyongli has joined #openstack-infra | 08:27 | |
| *** openstackgrafana has joined #openstack-infra | 08:28 | |
| *** openstackgrafana has quit IRC | 08:28 | |
| *** openstackgrafana has joined #openstack-infra | 08:29 | |
| *** openstackgrafana has quit IRC | 08:29 | |
| *** d0ugal_ has quit IRC | 08:29 | |
| *** d0ugal has joined #openstack-infra | 08:29 | |
| *** d0ugal has quit IRC | 08:29 | |
| *** d0ugal has joined #openstack-infra | 08:29 | |
| *** dpawlik has quit IRC | 08:30 | |
| *** openstackgrafana has joined #openstack-infra | 08:34 | |
| *** openstackgrafana has joined #openstack-infra | 08:35 | |
| *** electrofelix has joined #openstack-infra | 08:35 | |
| *** openstackgrafana has quit IRC | 08:36 | |
| *** openstackgrafana has joined #openstack-infra | 08:36 | |
| *** openstackgrafana has quit IRC | 08:37 | |
| *** heyongli has quit IRC | 08:37 | |
| *** heyongli has joined #openstack-infra | 08:37 | |
| *** openstackgrafana has joined #openstack-infra | 08:39 | |
| *** Bhujay has joined #openstack-infra | 08:39 | |
| *** openstackgrafana has quit IRC | 08:39 | |
| *** openstackgrafana has joined #openstack-infra | 08:40 | |
| *** Miouge has left #openstack-infra | 08:40 | |
| *** openstackgrafana has quit IRC | 08:40 | |
| *** Miouge has joined #openstack-infra | 08:40 | |
| *** openstackgrafana has joined #openstack-infra | 08:40 | |
| openstackgrafana | [Alerting] Test notification : Someone is testing the alert notification within grafana. | 08:40 |
| ianw | hwoarang: hmm, last opensuse15 seems to be opensuse-150-0000000477 | opensuse-150 | nb02 | qcow2,raw,vhd | ready | 00:20:39:55 | 08:41 |
| ianw | hwoarang: started a new build | 08:42 |
| ianw | https://nb02.openstack.org/opensuse-150-0000000478.log | 08:42 |
| hwoarang | ianw: yeah i checked https://nb01.openstack.org and that seems older. so maybe some nodes have the old image? | 08:43 |
| ianw | watch nl01.openstack.org/dib-image-list & image-list to see it upload | 08:43 |
| hwoarang | forgive my ingorance I am not 100% sure how all these things are connected together :) | 08:43 |
| ianw | hwoarang: yep, the last image is 477 which is 20 hours old | 08:43 |
| hwoarang | interesting.477 is not listed on nb01.openstack.org | 08:44 |
| hwoarang | 476 is the last one | 08:44 |
| ianw | no, that's just the builder, it depends on which builder picks it up | 08:44 |
| hwoarang | ah ok i see | 08:44 |
| hwoarang | ok will keep an eye on it | 08:44 |
| ianw | the thing to watch is dib-image-list and image-list | 08:44 |
| ianw | i've got a fix out so those pages don't return json to browsers -> https://review.openstack.org/#/c/573053/ | 08:45 |
| ianw | in the mean time, you can just curl it with 'text/plain' to make it easier to see | 08:45 |
| hwoarang | hehe true | 08:45 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 08:45 |
| hwoarang | thank you | 08:45 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: webapp: fix browser return https://review.openstack.org/573053 | 08:47 |
| *** heyongli has quit IRC | 08:47 | |
| *** heyongli has joined #openstack-infra | 08:48 | |
| *** openstackgrafana has quit IRC | 08:49 | |
| *** jaosorior has joined #openstack-infra | 08:54 | |
| *** kamren has quit IRC | 08:54 | |
| *** edmondsw has joined #openstack-infra | 08:55 | |
| *** kamren has joined #openstack-infra | 08:55 | |
| *** janki has joined #openstack-infra | 08:57 | |
| *** heyongli has quit IRC | 08:58 | |
| *** heyongli has joined #openstack-infra | 08:58 | |
| *** edmondsw has quit IRC | 09:00 | |
| *** dingyichen has quit IRC | 09:03 | |
| *** Bhujay has quit IRC | 09:06 | |
| *** kiennt26 has quit IRC | 09:07 | |
| *** heyongli has quit IRC | 09:08 | |
| *** heyongli has joined #openstack-infra | 09:08 | |
| *** dtantsur|afk is now known as dtantsur | 09:09 | |
| *** derekh has joined #openstack-infra | 09:09 | |
| *** lifeless_ has quit IRC | 09:17 | |
| *** lifeless has joined #openstack-infra | 09:18 | |
| *** heyongli has quit IRC | 09:18 | |
| *** heyongli has joined #openstack-infra | 09:18 | |
| *** nmathew has quit IRC | 09:22 | |
| *** armaan has quit IRC | 09:27 | |
| *** armaan has joined #openstack-infra | 09:28 | |
| *** heyongli has quit IRC | 09:28 | |
| *** heyongli has joined #openstack-infra | 09:29 | |
| *** sambetts|afk is now known as sambetts | 09:33 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 09:38 |
| AJaeger | Jeffrey4l: https://review.openstack.org/#/c/568174/ needs a rebase | 09:39 |
| *** heyongli has quit IRC | 09:39 | |
| *** heyongli has joined #openstack-infra | 09:39 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config master: Retire kolla-kubernetes project - step 4 remove from infra system https://review.openstack.org/568174 | 09:40 |
| Jeffrey4l | AJaeger, done ^^ | 09:40 |
| *** pbourke has quit IRC | 09:42 | |
| *** pbourke has joined #openstack-infra | 09:42 | |
| AJaeger | Jeffrey4l: LGTM, +2 | 09:43 |
| Jeffrey4l | thanks | 09:43 |
| *** links has quit IRC | 09:48 | |
| *** heyongli has quit IRC | 09:49 | |
| *** heyongli has joined #openstack-infra | 09:49 | |
| *** dklyle has quit IRC | 09:50 | |
| *** david-lyle has joined #openstack-infra | 09:50 | |
| *** david-lyle has quit IRC | 09:52 | |
| *** dklyle has joined #openstack-infra | 09:52 | |
| *** dklyle has quit IRC | 09:53 | |
| *** david-lyle has joined #openstack-infra | 09:53 | |
| *** david-lyle has quit IRC | 09:53 | |
| *** e0ne has joined #openstack-infra | 09:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/labels route https://review.openstack.org/553979 | 09:57 |
| *** heyongli has quit IRC | 09:59 | |
| *** heyongli has joined #openstack-infra | 09:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/nodes route https://review.openstack.org/553998 | 10:02 |
| *** dklyle has joined #openstack-infra | 10:04 | |
| *** links has joined #openstack-infra | 10:05 | |
| *** pcichy has quit IRC | 10:06 | |
| *** heyongli has quit IRC | 10:09 | |
| *** heyongli has joined #openstack-infra | 10:10 | |
| *** openstackgrafana has joined #openstack-infra | 10:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 10:13 |
| *** namnh has quit IRC | 10:15 | |
| *** heyongli has quit IRC | 10:20 | |
| *** heyongli has joined #openstack-infra | 10:20 | |
| *** openstackgrafana has quit IRC | 10:27 | |
| *** openstackgrafana has joined #openstack-infra | 10:27 | |
| *** dizquierdo has joined #openstack-infra | 10:28 | |
| *** heyongli has quit IRC | 10:30 | |
| *** ykarel|training has quit IRC | 10:30 | |
| *** heyongli has joined #openstack-infra | 10:30 | |
| *** openstackgrafana has quit IRC | 10:37 | |
| *** links has quit IRC | 10:39 | |
| *** apetrich has quit IRC | 10:40 | |
| *** heyongli has quit IRC | 10:40 | |
| *** heyongli has joined #openstack-infra | 10:40 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add opensuse-150 to nodepool dsvm https://review.openstack.org/572416 | 10:41 |
| *** edmondsw has joined #openstack-infra | 10:43 | |
| *** kbaikov[m] has quit IRC | 10:44 | |
| *** openstackgrafana has joined #openstack-infra | 10:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 10:47 |
| *** edmondsw has quit IRC | 10:47 | |
| openstackgrafana | [Alerting] Test notification : Someone is testing the alert notification within grafana. | 10:48 |
| ianw | test | 10:49 |
| *** heyongli has quit IRC | 10:50 | |
| *** heyongli has joined #openstack-infra | 10:51 | |
| *** eliqiao has joined #openstack-infra | 10:53 | |
| Tengu | test successful. | 10:55 |
| Tengu | :3 | 10:55 |
| *** links has joined #openstack-infra | 10:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add helga-grafana to grafana for IRC alerts https://review.openstack.org/573183 | 11:00 |
| *** dpawlik has joined #openstack-infra | 11:00 | |
| *** heyongli has quit IRC | 11:01 | |
| *** caphrim007 has quit IRC | 11:01 | |
| *** heyongli has joined #openstack-infra | 11:01 | |
| *** dizquierdo has quit IRC | 11:01 | |
| *** caphrim007 has joined #openstack-infra | 11:02 | |
| ianw | infra-root: ^ this is what is producing the test alert from openstackgrafana. grafana is pretty sensible about it's alerts and i think with a bit of fiddling with stats levels we can probably get pretty good automated notification of out-of-bounds conditions as we desire | 11:02 |
| ianw | this is currently hand-deployed on grafana02, which is in emergency while i keep fiddling | 11:03 |
| *** dpawlik has quit IRC | 11:05 | |
| *** udesale has quit IRC | 11:07 | |
| *** heyongli has quit IRC | 11:11 | |
| *** heyongli has joined #openstack-infra | 11:11 | |
| openstackgrafana | [Alerting] Zuul Job Queue alert : The Zuul build queue is averaging above 1 for 5 minutes | 11:12 |
| *** yamamoto has quit IRC | 11:14 | |
| *** pkovar has joined #openstack-infra | 11:17 | |
| *** yamahata has quit IRC | 11:19 | |
| *** jpena is now known as jpena|off | 11:20 | |
| *** heyongli has quit IRC | 11:21 | |
| *** heyongli has joined #openstack-infra | 11:21 | |
| *** edmondsw has joined #openstack-infra | 11:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/labels.html web page https://review.openstack.org/553980 | 11:22 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/nodes.html web page https://review.openstack.org/553999 | 11:24 |
| *** heyongli has quit IRC | 11:31 | |
| *** heyongli has joined #openstack-infra | 11:32 | |
| *** dizquierdo has joined #openstack-infra | 11:33 | |
| *** ldnunes has joined #openstack-infra | 11:33 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add jobs graph rendering https://review.openstack.org/537869 | 11:37 |
| *** ykarel|training has joined #openstack-infra | 11:38 | |
| *** pkovar has left #openstack-infra | 11:39 | |
| *** heyongli has quit IRC | 11:42 | |
| *** heyongli has joined #openstack-infra | 11:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add project pipeline rendering https://review.openstack.org/537871 | 11:43 |
| *** zhangfei has quit IRC | 11:43 | |
| *** jcoufal has joined #openstack-infra | 11:46 | |
| *** heyongli has quit IRC | 11:52 | |
| *** heyongli has joined #openstack-infra | 11:52 | |
| *** dpawlik has joined #openstack-infra | 11:52 | |
| *** alexchadin has quit IRC | 11:56 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/irc-meetings master: Update Designate IRC meeting to bi weekly alternating https://review.openstack.org/573204 | 11:56 |
| *** alexchadin has joined #openstack-infra | 11:57 | |
| *** dpawlik has quit IRC | 11:57 | |
| *** alexchadin has quit IRC | 11:57 | |
| *** alexchadin has joined #openstack-infra | 11:57 | |
| *** alexchadin has quit IRC | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 11:59 | |
| *** jpena|off is now known as jpena | 12:01 | |
| *** heyongli has quit IRC | 12:02 | |
| *** heyongli has joined #openstack-infra | 12:02 | |
| *** katkapilatova has joined #openstack-infra | 12:03 | |
| *** dizquierdo has quit IRC | 12:04 | |
| *** eharney has quit IRC | 12:08 | |
| *** kgiusti has joined #openstack-infra | 12:10 | |
| *** yamamoto has joined #openstack-infra | 12:10 | |
| *** nicolasbock has joined #openstack-infra | 12:11 | |
| *** jbadiapa has joined #openstack-infra | 12:12 | |
| *** armaan has quit IRC | 12:12 | |
| *** heyongli has quit IRC | 12:12 | |
| *** armaan has joined #openstack-infra | 12:13 | |
| *** heyongli has joined #openstack-infra | 12:13 | |
| *** dave-mccowan has joined #openstack-infra | 12:14 | |
| *** dave-mcc_ has joined #openstack-infra | 12:16 | |
| *** armaan has quit IRC | 12:17 | |
| *** dbecker has joined #openstack-infra | 12:17 | |
| *** rosmaita has joined #openstack-infra | 12:18 | |
| *** dave-mccowan has quit IRC | 12:18 | |
| *** yamamoto has quit IRC | 12:21 | |
| *** heyongli has quit IRC | 12:23 | |
| *** heyongli has joined #openstack-infra | 12:23 | |
| *** apetrich has joined #openstack-infra | 12:30 | |
| *** hemna_ has joined #openstack-infra | 12:31 | |
| *** heyongli has quit IRC | 12:33 | |
| *** heyongli has joined #openstack-infra | 12:33 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Add cookbook-openstackclient repo jobs https://review.openstack.org/571506 | 12:33 |
| *** alexchadin has joined #openstack-infra | 12:34 | |
| *** VW has joined #openstack-infra | 12:35 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Add cookbook-openstackclient repo jobs https://review.openstack.org/571506 | 12:38 |
| *** mriedem has joined #openstack-infra | 12:38 | |
| *** alexchadin has quit IRC | 12:39 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Add cookbook-openstackclient repo jobs https://review.openstack.org/571506 | 12:39 |
| *** trown|outtypewww is now known as trown | 12:41 | |
| *** heyongli has quit IRC | 12:43 | |
| *** heyongli has joined #openstack-infra | 12:43 | |
| *** jbadiapa has quit IRC | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| *** heyongli has quit IRC | 12:53 | |
| *** heyongli has joined #openstack-infra | 12:54 | |
| *** rcernin has quit IRC | 12:56 | |
| *** esberglu has joined #openstack-infra | 12:57 | |
| openstackgerrit | Oliver Walsh proposed openstack/diskimage-builder master: WIP: Add DIB element to install NVIDIA GPU drivers https://review.openstack.org/573223 | 12:58 |
| *** tpsilva has joined #openstack-infra | 13:02 | |
| *** heyongli has quit IRC | 13:04 | |
| *** heyongli has joined #openstack-infra | 13:04 | |
| *** myoung|off is now known as myoung | 13:05 | |
| *** eharney has joined #openstack-infra | 13:07 | |
| *** udesale has joined #openstack-infra | 13:09 | |
| *** VW has quit IRC | 13:09 | |
| *** VW has joined #openstack-infra | 13:10 | |
| *** alexchadin has joined #openstack-infra | 13:13 | |
| *** apetrich has quit IRC | 13:13 | |
| *** janki has quit IRC | 13:13 | |
| *** heyongli has quit IRC | 13:14 | |
| *** heyongli has joined #openstack-infra | 13:14 | |
| *** VW has quit IRC | 13:15 | |
| AJaeger | config-core, a few repos where created and need jobs - and some cleanups. Please review: https://review.openstack.org/#/c/571506/ https://review.openstack.org/#/c/568174/ https://review.openstack.org/#/c/568916/ https://review.openstack.org/#/c/572558/ https://review.openstack.org/#/c/571925/ https://review.openstack.org/#/c/572065/ https://review.openstack.org/#/c/572276/ | 13:20 |
| AJaeger | https://review.openstack.org/#/c/572017/ | 13:20 |
| * mnaser pulls up sleves | 13:20 | |
| * AJaeger thanks mnaser | 13:22 | |
| *** alexchad_ has joined #openstack-infra | 13:23 | |
| *** alexchad_ is now known as alexchadin_ | 13:23 | |
| *** alexchadin has quit IRC | 13:24 | |
| *** heyongli has quit IRC | 13:24 | |
| *** heyongli has joined #openstack-infra | 13:25 | |
| *** apetrich has joined #openstack-infra | 13:26 | |
| *** ykarel|training has quit IRC | 13:27 | |
| mnaser | AJaeger: taken care of | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire kolla-kubernetes project - step 4 remove from infra system https://review.openstack.org/568174 | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Add jobs for ansible-role-openstacksdk https://review.openstack.org/568916 | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new OSA project base jobs https://review.openstack.org/572558 | 13:30 |
| *** sthussey has joined #openstack-infra | 13:30 | |
| *** openstackgerrit has quit IRC | 13:34 | |
| *** heyongli has quit IRC | 13:34 | |
| *** heyongli has joined #openstack-infra | 13:35 | |
| *** openstackgerrit has joined #openstack-infra | 13:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove direct tagging/branching for yaql https://review.openstack.org/572065 | 13:38 |
| *** dave-mcc_ has quit IRC | 13:41 | |
| *** dave-mccowan has joined #openstack-infra | 13:42 | |
| fungi | corvus: clarkb: oops on the clouds.yaml edit. i had tested `openstack server list` with that and it worked, but i figured puppet was going to revert it to the original template on its next run so i didn't bother to remove it. curious why that entry was freaking out the inventory even though openstackclient was able to use it | 13:44 |
| *** heyongli has quit IRC | 13:45 | |
| *** heyongli has joined #openstack-infra | 13:45 | |
| *** dave-mccowan has quit IRC | 13:46 | |
| *** rlandy has joined #openstack-infra | 13:46 | |
| *** rlandy is now known as rlandy|rover | 13:47 | |
| *** dave-mccowan has joined #openstack-infra | 13:48 | |
| *** dave-mcc_ has joined #openstack-infra | 13:50 | |
| *** armaan has joined #openstack-infra | 13:52 | |
| *** dave-mccowan has quit IRC | 13:53 | |
| *** dpawlik has joined #openstack-infra | 13:53 | |
| *** katkapilatova has left #openstack-infra | 13:53 | |
| *** zhangfei has joined #openstack-infra | 13:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused puppet beaker job templates https://review.openstack.org/571925 | 13:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused legacy jobs https://review.openstack.org/572276 | 13:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove openstack-planet-unittest legacy job https://review.openstack.org/572017 | 13:55 |
| *** heyongli has quit IRC | 13:55 | |
| *** heyongli has joined #openstack-infra | 13:55 | |
| AJaeger | cmurphy: again merge conflict for you ;( could you check https://review.openstack.org/#/c/571929/ - and ask for reviews afterwards, please? | 13:57 |
| *** Bhujay has joined #openstack-infra | 13:58 | |
| *** dpawlik has quit IRC | 13:58 | |
| cmurphy | AJaeger: thanks, I wonder why gerrit doesn't email me when that happens any more :( | 13:59 |
| AJaeger | cmurphy: maybe somebody else here knows? infra-root ^ | 13:59 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert infra's beaker jobs for zuulv3 https://review.openstack.org/571929 | 14:01 |
| cmurphy | AJaeger: ^ | 14:01 |
| AJaeger | any config-core for another +2 to move forward with cmurphy 's stack, please? ^ | 14:03 |
| frickler | cmurphy: IIUC earlier there was a job that checked for merge conflicts, that also triggered emails. now zuul/gerrit somehow checks this by itself => no more emails. I'm sad about that, too | 14:03 |
| *** udesale has quit IRC | 14:03 | |
| *** udesale has joined #openstack-infra | 14:04 | |
| *** annp__ has joined #openstack-infra | 14:04 | |
| *** heyongli has quit IRC | 14:05 | |
| *** heyongli has joined #openstack-infra | 14:05 | |
| cmurphy | frickler: :( that's unfortunate for my email-based workflow | 14:06 |
| mordred | cmurphy, AJaeger: +A | 14:07 |
| cmurphy | \o/ at least my change queue is getting shorter | 14:07 |
| *** amoralej|lunch is now known as amoralej | 14:07 | |
| fungi | cmurphy: AJaeger: as frickler noted we used to test for merge conflicts using zuul and leave a comment on changes. we stopped doing that once we upgraded to a new enough gerrit that it has a feature to indicate whether a change needs rebasing. but yes, the unfortunate down-side is that gerrit doesn't have a feature to e-mail change subscribers once a change has a probable merge conflict | 14:09 |
| fungi | the up side is that you should be able to add a section of changes needing rebase to a gerrit dashboard/query | 14:10 |
| fungi | though i don't know which query parameter that is right off the top of my head | 14:11 |
| *** jiapei has quit IRC | 14:11 | |
| AJaeger | cmurphy: please, double check your change queue, there are more merge conflicts | 14:11 |
| frickler | infra-root: I'm seeing some strangeness on https://review.openstack.org/573202 , the osc-functional-devstack job is marked as POST_FAILURE, though according to the logs and also for ara it failed in the run phase | 14:11 |
| cmurphy | AJaeger: lol | 14:11 |
| cmurphy | you all expect me to be so on top of things | 14:12 |
| AJaeger | cmurphy: ;) | 14:12 |
| mordred | frickler: it's possible that the run phase didn't produce an artifact that the post-run was looking for? | 14:12 |
| fungi | frickler: something about the way the run failed caused the post-logs playbook to also fail | 14:13 |
| frickler | but shouldn't that show up in the job-output.txt? | 14:14 |
| *** felipemonteiro has joined #openstack-infra | 14:15 | |
| *** kjackal has joined #openstack-infra | 14:15 | |
| *** heyongli has quit IRC | 14:15 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:15 | |
| *** heyongli has joined #openstack-infra | 14:16 | |
| *** alexchadin has joined #openstack-infra | 14:17 | |
| fungi | not if it got truncated, which looks like may have been the case according to the tooltip in ara | 14:17 |
| fungi | "playbook execution is either in progress or was interrupted; data will be inconsistent" | 14:17 |
| frickler | cmurphy: fyi that search should give you all your patches in merge conflict: "-is:mergeable owner:self status:open". note the "-" that negates the first condition | 14:18 |
| *** alexchadin_ has quit IRC | 14:18 | |
| frickler | fungi: that is shown also for passing patches. seems to be an artefact because ara report gets copied before ansible run finishes | 14:20 |
| *** felipemonteiro has quit IRC | 14:20 | |
| frickler | s/patches/jobs/ | 14:20 |
| corvus | mordred: can hosts in the emergency file be regexes? | 14:22 |
| corvus | or globs? | 14:22 |
| fungi | frickler: oh, yep, i hadn't noticed that before but i see it for example at http://logs.openstack.org/02/573202/1/check/openstack-tox-pep8/5281f30/ara-report/ | 14:23 |
| fungi | frickler: maybe that started happening with the change to sqlite-based ara reporting? | 14:23 |
| *** links has quit IRC | 14:23 | |
| frickler | fungi: I think so, yes. I always assumed that it was likely intentional/unavoidable | 14:24 |
| *** jamesmcarthur has joined #openstack-infra | 14:25 | |
| *** heyongli has quit IRC | 14:26 | |
| *** heyongli has joined #openstack-infra | 14:26 | |
| fungi | dmsimard like will know when he's around | 14:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert puppet-openstackci jobs to zuulv3 https://review.openstack.org/571937 | 14:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker xenial jobs voting https://review.openstack.org/571965 | 14:27 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Add experimental puppet 4 beaker-rspec jobs https://review.openstack.org/571966 | 14:27 |
| fungi | s/like/likely/ | 14:27 |
| corvus | i added ze* to emergency file to prepare for ansible upgrade | 14:30 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert puppet-openstackci jobs to zuulv3 https://review.openstack.org/571937 | 14:32 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker xenial jobs voting https://review.openstack.org/571965 | 14:32 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Add experimental puppet 4 beaker-rspec jobs https://review.openstack.org/571966 | 14:32 |
| cmurphy | AJaeger: git is hard apparently | 14:32 |
| *** VW has joined #openstack-infra | 14:34 | |
| AJaeger | cmurphy: sometimes ;) | 14:35 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Convert puppet-openstackci jobs to zuulv3 https://review.openstack.org/571937 | 14:35 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker xenial jobs voting https://review.openstack.org/571965 | 14:35 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Add experimental puppet 4 beaker-rspec jobs https://review.openstack.org/571966 | 14:35 |
| *** heyongli has quit IRC | 14:36 | |
| *** heyongli has joined #openstack-infra | 14:36 | |
| dtroyer | https://review.openstack.org/#/c/569605/ needs another +2 for StarlingX Zuul jobs | 14:42 |
| *** ccamacho has quit IRC | 14:42 | |
| *** rajinir has joined #openstack-infra | 14:43 | |
| *** rpioso|afk is now known as rpioso | 14:43 | |
| *** iranzo has quit IRC | 14:45 | |
| *** heyongli has quit IRC | 14:46 | |
| *** neiloy has joined #openstack-infra | 14:46 | |
| *** heyongli has joined #openstack-infra | 14:46 | |
| mnaser | fungi beat me to it :) | 14:48 |
| *** apetrich has quit IRC | 14:48 | |
| AJaeger | mnaser: want to look at cmurphy's stack at https://review.openstack.org/#/c/571937 instead? | 14:49 |
| mnaser | alrighty | 14:50 |
| *** roman_g has quit IRC | 14:52 | |
| *** lifeless_ has joined #openstack-infra | 14:53 | |
| *** lifeless has quit IRC | 14:53 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert infra's beaker jobs for zuulv3 https://review.openstack.org/571929 | 14:54 |
| cmurphy | thanks y'all | 14:55 |
| *** heyongli has quit IRC | 14:56 | |
| *** alexchadin has quit IRC | 14:56 | |
| *** heyongli has joined #openstack-infra | 14:57 | |
| *** caphrim007_ has joined #openstack-infra | 15:00 | |
| *** dpawlik has joined #openstack-infra | 15:01 | |
| *** dizquierdo has joined #openstack-infra | 15:02 | |
| openstackgerrit | Merged openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 15:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 15:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Update to Ansible 2.5 https://review.openstack.org/562668 | 15:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 15:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Add zuul jobs for StarlingX https://review.openstack.org/569605 | 15:02 |
| *** caphrim007 has quit IRC | 15:03 | |
| *** apetrich has joined #openstack-infra | 15:04 | |
| *** dpawlik has quit IRC | 15:05 | |
| *** erlon has joined #openstack-infra | 15:06 | |
| *** heyongli has quit IRC | 15:07 | |
| *** d0ugal has quit IRC | 15:07 | |
| Shrews | and now we wait... | 15:07 |
| *** heyongli has joined #openstack-infra | 15:07 | |
| *** zoli is now known as zoli|gone | 15:07 | |
| *** zoli|gone is now known as zoli | 15:07 | |
| fungi | as mentioned yesterday, i can be around in ~50 minutes to pitch in, once security sig meeting and tc office hour conclude | 15:08 |
| *** annp__ has quit IRC | 15:09 | |
| corvus | Shrews, clarkb, fungi: i've disabled puppet on ze*. perhaps the next step should be to stop all executors, then manually run puppet on them, then restart them (probably one at a time). whatchya think? | 15:09 |
| Shrews | corvus: i had assumed that was the plan, so... ++ | 15:09 |
| clarkb | that seems reasonable we can also stop, puppet, start one at a time | 15:09 |
| clarkb | rather than stopping all of them (I think) | 15:10 |
| Shrews | actually, clarkb's plan sounds a bit safer | 15:10 |
| corvus | clarkb: that's possible too, though it means that jobs become increasingly likely to be restarted multiple times. | 15:11 |
| corvus | how about we start with one? :) | 15:11 |
| clarkb | start with one works then we can do the rest | 15:11 |
| * clarkb is still catching up it does appear that the zuul changes have merged, neat | 15:12 | |
| *** zhangfei has quit IRC | 15:12 | |
| corvus | i'm looking into something. i haven't started yet. | 15:13 |
| corvus | http://grafana.openstack.org/dashboard/db/zuul-status?panelId=18&fullscreen | 15:13 |
| corvus | that graph looks weird | 15:13 |
| corvus | because there is no scheduler running | 15:14 |
| corvus | 2018-06-07 15:05:59,685 is the last log entry | 15:16 |
| dtroyer | fungi, mnaser: thanks! | 15:16 |
| clarkb | corvus: was it intentionally stopped? | 15:16 |
| clarkb | (trying to understand if the scheduler not running is the weird thing or the contents of the graph given that state) | 15:16 |
| corvus | it coincides with puppet performing the install/upgrade | 15:16 |
| *** heyongli has quit IRC | 15:17 | |
| *** heyongli has joined #openstack-infra | 15:17 | |
| corvus | clarkb: my understanding is currently that the scheduler stopped for an unknown reason around the time that puppet last ran, that released all the nodes which is why the graph looked like that. the graph is what caught my attention. | 15:18 |
| clarkb | got it | 15:18 |
| corvus | Jun 7 15:05:59 zuul01 kernel: [12242746.987209] traps: zuul-scheduler[17679] general protection ip:5c175e sp:7f75adff7d10 error:0 in python3.5[400000+3a9000] | 15:18 |
| clarkb | related to updating zuul under it possibly. | 15:19 |
| openstackgerrit | Zane Bitter proposed openstack-dev/cookiecutter master: Convert to new docs PTI https://review.openstack.org/573294 | 15:19 |
| corvus | that was during the install_zuul task | 15:19 |
| corvus | that's interesting. i wonder if ansible 2.5 brought in a new binary dependency or something | 15:20 |
| corvus | anyway, i guess we're doing this as a full shutdown :) | 15:20 |
| clarkb | it links against openssl via paramiko | 15:20 |
| corvus | i'll stop all the executors and mergers | 15:20 |
| clarkb | ya I don't think doing a rolling update matters much given ^ | 15:20 |
| corvus | we can use dmsimard's emergency queue backup to restore | 15:21 |
| *** apetrich has quit IRC | 15:22 | |
| *** ramishra has quit IRC | 15:22 | |
| corvus | i'm updating zuul on all the executors | 15:23 |
| *** cshastri has quit IRC | 15:23 | |
| corvus | on | 15:24 |
| corvus | (ignore that ^) | 15:24 |
| *** jesslampe has quit IRC | 15:24 | |
| *** jesslampe has joined #openstack-infra | 15:24 | |
| corvus | should we send a status notice? | 15:27 |
| *** heyongli has quit IRC | 15:27 | |
| *** heyongli has joined #openstack-infra | 15:27 | |
| corvus | 2/10 executors stopped | 15:28 |
| clarkb | sure. How about #status notice Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly. | 15:28 |
| corvus | lgtm | 15:28 |
| fungi | wfm | 15:29 |
| clarkb | #status notice Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly. | 15:29 |
| openstackstatus | clarkb: sending notice | 15:29 |
| corvus | 5/10 | 15:29 |
| -openstackstatus- NOTICE: Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly. | 15:30 | |
| corvus | i've restarted the mergers | 15:30 |
| corvus | i'll start the scheduler now and re-enqueue | 15:31 |
| *** slaweq_ has joined #openstack-infra | 15:32 | |
| openstackstatus | clarkb: finished sending notice | 15:32 |
| *** slaweq has quit IRC | 15:33 | |
| corvus | paramiko.ssh_exception.SSHException: Signature verification (ssh-ed25519) failed. | 15:33 |
| corvus | i don't know which gerrit host that is | 15:33 |
| corvus | presumably odl? | 15:33 |
| clarkb | ya ours shouldn't have eliptic keys | 15:33 |
| clarkb | I will double check | 15:33 |
| clarkb | rsa and dsa only in review_site/etc | 15:34 |
| corvus | yeah, when i run ssh as the zuul user with that key i get a prompt | 15:34 |
| corvus | where's the authorized_keys file? | 15:34 |
| clarkb | corvus: you mean in puppet? I'll dig it up | 15:35 |
| clarkb | modules/openstack_project/manifests/zuul_merger.pp and modules/openstack_project/manifests/zuul_prod.pp have known_hosts_content parameters | 15:36 |
| corvus | i mean on disk | 15:36 |
| corvus | and yeah, derp, i meant known_hosts | 15:36 |
| clarkb | /home/zuul/.ssh/known_hosts according to puppet-zuul | 15:37 |
| corvus | there's no file there | 15:37 |
| *** heyongli has quit IRC | 15:37 | |
| corvus | hrm, but zuul sets this: client.set_missing_host_key_policy(paramiko.WarningPolicy()) | 15:37 |
| *** heyongli has joined #openstack-infra | 15:38 | |
| mordred | did paramiko release again and break us again? | 15:38 |
| mordred | or, rather, did installing ansible bump paramiko past what we want in zuul? | 15:39 |
| corvus | mordred: maybe? it may have been upgraded when we upgraded zuul? | 15:39 |
| Shrews | ansible has a paramiko requirement, but doesn't limit version | 15:39 |
| corvus | we don't limit the version either | 15:39 |
| Shrews | in their requirements.txt | 15:39 |
| Shrews | hmm | 15:39 |
| clarkb | in site.pp we only set known_hosts_content on ze* not zuul* or zm* | 15:39 |
| clarkb | corvus: ^ I think that is why you don't see that file on those hosts. Is that preventing things from working right now? | 15:40 |
| corvus | clarkb: only odl; otherwise, it's a spurious traceback every 5 seconds | 15:40 |
| corvus | paramiko last released mar 13 | 15:40 |
| clarkb | got it, I'll work on a patch | 15:40 |
| corvus | clarkb: to do what? | 15:40 |
| clarkb | corvus: to add the known hosts file then we can add in odl | 15:41 |
| corvus | clarkb: but we don't seem to need a known hosts file for openstack's gerrit | 15:41 |
| openstackgrafana | [OK] Zuul Job Queue alert : The Zuul build queue is averaging above 1 for 5 minutes | 15:42 |
| clarkb | corvus: that could be a paramiko bug with ed25519 verification not supporting the warning? | 15:42 |
| *** cshastri has joined #openstack-infra | 15:42 | |
| *** ChanServ sets mode: +o corvus | 15:42 | |
| *** openstackgrafana was kicked by corvus (openstackgrafana) | 15:42 | |
| *** corvus sets mode: -o corvus | 15:42 | |
| clarkb | our known_hosts config is out of date for the new ip addrs for new gerrit too. I wonder if pabelanger has a change up for that | 15:43 |
| corvus | clarkb: maybe... | 15:43 |
| corvus | clarkb: maybe we should drop odl for the moment? | 15:43 |
| clarkb | the config we do have I mean | 15:43 |
| clarkb | corvus: we can do that, we aren't doing anything with it yet aiui so shouldn't affect anything | 15:44 |
| corvus | i will manually do that; clarkb can you push up a change? | 15:44 |
| clarkb | corvus: yes | 15:45 |
| *** gyee has joined #openstack-infra | 15:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Revert "Add opendaylight connection to zuul" https://review.openstack.org/573300 | 15:47 |
| *** heyongli has quit IRC | 15:48 | |
| corvus | all the executors are stopped. scheduler and mergers are up and running | 15:48 |
| corvus | i'll start one of the executors now | 15:48 |
| *** heyongli has joined #openstack-infra | 15:48 | |
| corvus | we're running cat jobs still; things are a little slower with only 7 mergers. we're at "S" | 15:49 |
| corvus | okay, ze01 is running a job | 15:50 |
| * tobiash is crossing fingers | 15:50 | |
| clarkb | https://zuul.openstack.org/stream.html?uuid=bd6d7e85b7584f818d34073429da19aa&logfile=console.log I am watching that one | 15:51 |
| corvus | it's at least gotten through the base pre-playbook | 15:52 |
| Shrews | i guess we don't really have anything to *clean* the ze builds dir. lots of builds from Feb | 15:53 |
| clarkb | Shrews: if things crash ungracefully I think we do leave those behind | 15:54 |
| clarkb | (like if the ze damon goes away unexpectedl) | 15:55 |
| corvus | yeah. i could dust this off: https://review.openstack.org/531510 | 15:55 |
| Shrews | ++, i was about to suggest a startup cleanup thingy | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Delete stale jobdirs at startup https://review.openstack.org/573302 | 15:56 |
| corvus | there, i cherry-picked it onto master so we don't forget about it | 15:56 |
| *** eernst has joined #openstack-infra | 15:56 | |
| Shrews | http://logs.openstack.org/05/572805/5/check/openstack-tox-pep8/bd6d7e8/job-output.txt.gz#_2018-06-07_15_50_26_472858 | 15:57 |
| Shrews | Ansible Version: 2.5.4 | 15:57 |
| Shrews | \o/ | 15:57 |
| *** esberglu has quit IRC | 15:57 | |
| corvus | that pep8 job passed | 15:57 |
| corvus | the accompanying sphinx job failed | 15:57 |
| clarkb | and it succeeded. The docs job for that change failed but it is a docs change | 15:57 |
| *** janki has joined #openstack-infra | 15:57 | |
| clarkb | entirely possible that is a valid failure, looking at that now | 15:58 |
| *** heyongli has quit IRC | 15:58 | |
| corvus | 2018-06-07 15:55:12.104348 | ubuntu-xenial | /home/zuul/src/git.openstack.org/openstack/nova-specs/doc/source/specs/rocky/approved/nova-image-download-via-rbd.rst:7:Title overline too short. | 15:58 |
| *** dpawlik has joined #openstack-infra | 15:58 | |
| *** heyongli has joined #openstack-infra | 15:58 | |
| corvus | so i think we've established that "not all jobs will fail" :) | 15:58 |
| tobiash | \o/ | 15:59 |
| corvus | which is probably good enough to start up the rest of the executors and re-enqueue from the queue backup | 15:59 |
| clarkb | ara is workign too | 15:59 |
| clarkb | corvus: wfm | 15:59 |
| Shrews | that docs job failed the previous run, fwiw | 15:59 |
| clarkb | Shrews: ya and corvus found the error so that all looks good | 16:00 |
| Shrews | ah | 16:00 |
| corvus | all executors started, re-enqueuing | 16:00 |
| clarkb | specifically a warning that was treated as an error | 16:00 |
| *** dpawlik has quit IRC | 16:02 | |
| corvus | status notice The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra | 16:06 |
| corvus | ^? | 16:06 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert puppet-openstackci jobs to zuulv3 https://review.openstack.org/571937 | 16:07 |
| corvus | clarkb, Shrews, fungi: ^ status msg look ok? | 16:08 |
| clarkb | yes | 16:08 |
| *** heyongli has quit IRC | 16:08 | |
| fungi | lgtm | 16:08 |
| corvus | #status notice The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra | 16:08 |
| openstackstatus | corvus: sending notice | 16:08 |
| Shrews | yep | 16:08 |
| fungi | and i'm freed up now to help with any lingering regressions we might notice | 16:08 |
| *** heyongli has joined #openstack-infra | 16:08 | |
| Shrews | can't wait for 2.6 to be released a few minutes from now! | 16:09 |
| -openstackstatus- NOTICE: The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra | 16:10 | |
| *** diablo_rojo has quit IRC | 16:10 | |
| fungi | clearly the solution is to wait until ansible is bored with making any further changes to the current version and about to release a new one | 16:11 |
| *** janki has quit IRC | 16:11 | |
| openstackstatus | corvus: finished sending notice | 16:12 |
| corvus | Shrews: i don't care! we did it! we upgraded to the latest version of ansible! | 16:12 |
| clarkb | ha | 16:12 |
| clarkb | seriously though did their releaes cadence pick up steam or am I just not paying attention | 16:13 |
| Shrews | great. now we have to _really_ think about this multi-version thing... | 16:13 |
| corvus | Shrews, tobiash: thank you for doing the actual work on that :) | 16:13 |
| corvus | Shrews: ++ | 16:13 |
| clarkb | seems like when I needed a bug fix it took 9 months but now that we ahve to keep up with them its every few months :) | 16:13 |
| *** yamamoto has quit IRC | 16:14 | |
| corvus | clarkb: i can neither confirm nor deny that the release cadence is designed to annoy you personally ;) | 16:14 |
| Shrews | i thought our release cadence with zuul+nodepool was designed to do that? | 16:14 |
| corvus | that i can confirm | 16:14 |
| clarkb | ha | 16:15 |
| clarkb | this reminds me I have a fix for logging that I need to rebase then we should totally do a release :P | 16:15 |
| *** lifeless has joined #openstack-infra | 16:16 | |
| *** lifeless_ has quit IRC | 16:16 | |
| tobiash | I finally can remove one of the biggest parts from our staging tree :) | 16:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker xenial jobs voting https://review.openstack.org/571965 | 16:18 |
| *** heyongli has quit IRC | 16:18 | |
| *** heyongli has joined #openstack-infra | 16:19 | |
| *** dizquierdo has quit IRC | 16:20 | |
| *** dbecker has quit IRC | 16:21 | |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| corvus | i'm going to work on the ssh connection issue now | 16:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add timestamps to multiline debug message logs https://review.openstack.org/572845 | 16:23 |
| *** myoung is now known as myoung|biaf | 16:23 | |
| *** SumitNaiksatam has quit IRC | 16:25 | |
| *** yamamoto has quit IRC | 16:26 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add experimental puppet 4 beaker-rspec jobs https://review.openstack.org/571966 | 16:26 |
| *** heyongli has quit IRC | 16:29 | |
| *** heyongli has joined #openstack-infra | 16:29 | |
| clarkb | I'm going to keep an eye on failures for a bit so that I don't get sucked into something that prevents me from helping debug issues from $projects | 16:32 |
| clarkb | but also breakfast | 16:32 |
| *** myoung|biaf is now known as myoung|lunch | 16:32 | |
| openstackgerrit | Scott Hussey proposed openstack-infra/project-config master: (airship) Disable verify permission for CI https://review.openstack.org/573315 | 16:33 |
| *** dizquierdo has joined #openstack-infra | 16:35 | |
| *** lifeless_ has joined #openstack-infra | 16:35 | |
| *** lifeless has quit IRC | 16:36 | |
| *** dklyle has quit IRC | 16:36 | |
| corvus | hrm. the error is not the same error you get from something not being in known_hosts | 16:38 |
| *** heyongli has quit IRC | 16:39 | |
| *** jamesmcarthur has quit IRC | 16:39 | |
| *** heyongli has joined #openstack-infra | 16:39 | |
| *** jamesmcarthur has joined #openstack-infra | 16:40 | |
| *** jpich has quit IRC | 16:41 | |
| *** felipemonteiro_ has quit IRC | 16:42 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:42 | |
| *** gtmanfred has quit IRC | 16:44 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** tbarron has quit IRC | 16:46 | |
| corvus | aha | 16:47 |
| corvus | paramiko used ssh-ed25519 as the host key algo, but openssh settled on ecdsa-sha2-nistp256 | 16:48 |
| corvus | if i tell openssh to use ssh-ed25519 it also fails with a signature error | 16:48 |
| *** lpetrut has quit IRC | 16:49 | |
| *** heyongli has quit IRC | 16:49 | |
| *** heyongli has joined #openstack-infra | 16:49 | |
| *** derekh has quit IRC | 16:50 | |
| *** gtmanfred has joined #openstack-infra | 16:50 | |
| *** dbecker has joined #openstack-infra | 16:50 | |
| *** Swimingly has joined #openstack-infra | 16:50 | |
| *** Swimingly has quit IRC | 16:50 | |
| *** Swimingly has joined #openstack-infra | 16:50 | |
| *** tbarron has joined #openstack-infra | 16:50 | |
| mordred | corvus: wow | 16:51 |
| *** SumitNaiksatam has joined #openstack-infra | 16:52 | |
| AJaeger | clarkb: could you as PTL put https://review.openstack.org/#/c/572371/ on your review list, please? | 16:54 |
| clarkb | AJaeger: yes | 16:54 |
| *** e0ne has quit IRC | 16:55 | |
| clarkb | AJaeger: done, reviewed the governance change too | 16:57 |
| *** ccamacho has joined #openstack-infra | 16:57 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch to specific flavor and slowly bump capacity https://review.openstack.org/573369 | 16:58 |
| *** jamesmcarthur has joined #openstack-infra | 16:58 | |
| mnaser | clarkb: ^ | 16:58 |
| clarkb | mnaser: that removes boot from volume was that intended as well? | 16:58 |
| mnaser | clarkb: yes, as the commit message mentions, that flavor already has ephermal disk so it'll make lives easier | 16:59 |
| clarkb | cool | 16:59 |
| clarkb | oh right because we were leaking volumes | 16:59 |
| mnaser | yup | 16:59 |
| clarkb | +2 thanks | 16:59 |
| *** heyongli has quit IRC | 16:59 | |
| *** heyongli has joined #openstack-infra | 17:00 | |
| *** jamesmca_ has joined #openstack-infra | 17:00 | |
| AJaeger | thanks, clarkb | 17:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove beaker-*-rake jobs https://review.openstack.org/571927 | 17:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy puppet-openstackci jobs https://review.openstack.org/571939 | 17:01 |
| AJaeger | rebased these two cleanups ^ | 17:01 |
| *** jamesmcarthur has quit IRC | 17:03 | |
| *** dtantsur is now known as dtantsur|afk | 17:04 | |
| *** diablo_rojo has joined #openstack-infra | 17:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add cookbook-openstackclient repo jobs https://review.openstack.org/571506 | 17:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 17:06 |
| clarkb | AJaeger: mnaser ^ updated to address the reviews | 17:06 |
| *** dizquierdo has quit IRC | 17:06 | |
| AJaeger | could I get reviews for https://review.openstack.org/#/c/571938/ and https://review.openstack.org/571927 (two cleanups9 if somebody has spare cycles, please? Also, for https://review.openstack.org/572785 and https://review.openstack.org/572564 (new jobs for some repos) | 17:07 |
| AJaeger | clarkb: LGTM | 17:08 |
| *** lpetrut has joined #openstack-infra | 17:08 | |
| *** ccamacho has quit IRC | 17:08 | |
| *** ccamacho has joined #openstack-infra | 17:08 | |
| *** ccamacho has quit IRC | 17:09 | |
| *** ccamacho has joined #openstack-infra | 17:09 | |
| *** heyongli has quit IRC | 17:10 | |
| *** heyongli has joined #openstack-infra | 17:10 | |
| *** r-daneel has quit IRC | 17:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add afsmon project https://review.openstack.org/572371 | 17:16 |
| *** jamesmca_ has quit IRC | 17:17 | |
| *** jpena is now known as jpena|off | 17:17 | |
| *** heyongli has quit IRC | 17:20 | |
| *** heyongli has joined #openstack-infra | 17:20 | |
| *** ccamacho has quit IRC | 17:21 | |
| *** ccamacho has joined #openstack-infra | 17:21 | |
| *** lpetrut has quit IRC | 17:22 | |
| *** yamahata has joined #openstack-infra | 17:23 | |
| *** yamamoto has joined #openstack-infra | 17:23 | |
| *** sambetts is now known as sambetts|afk | 17:24 | |
| *** trown is now known as trown|lunch | 17:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use updated openstackci beaker jobs https://review.openstack.org/571938 | 17:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tox coverage job https://review.openstack.org/572785 | 17:27 |
| *** Bhujay has quit IRC | 17:27 | |
| *** udesale has quit IRC | 17:28 | |
| corvus | fascinating -- if you have a host key in known_hosts, paramiko will re-order its preferred key types to favor that | 17:30 |
| *** heyongli has quit IRC | 17:30 | |
| clarkb | corvus: so adding an rsa key to known hosts should fix it? | 17:30 |
| *** heyongli has joined #openstack-infra | 17:30 | |
| corvus | so one way of asking paramiko to use a certain key type is to have a host key of that type already | 17:30 |
| fungi | for the same host, or a different host? | 17:31 |
| *** yamamoto has quit IRC | 17:31 | |
| *** boden has joined #openstack-infra | 17:31 | |
| fungi | zuul/nodepool activity has really picked up over the last couple of days, and we've been maxed out on nodes pretty much solid for the past 12 hours | 17:31 |
| *** boris_42_ has joined #openstack-infra | 17:32 | |
| *** myoung|lunch is now known as myoung | 17:32 | |
| fungi | must be the mad rush for the r2 milestone | 17:32 |
| corvus | for the same host | 17:32 |
| *** psachin has quit IRC | 17:32 | |
| *** amoralej is now known as amoralej|off | 17:33 | |
| fungi | i suppose that makes sense. otherwise it would warn you about a different host key than the one you'd accepted which was still valid? | 17:33 |
| *** openstackgerrit has quit IRC | 17:34 | |
| corvus | yeah, makes sense. but it'd only be an issue if the remote started offering a new algo | 17:34 |
| boden | hi, I’m having an issue with the vmware-tox-lower-constraints job. If I don’t manually install the projects in tox ini (pip install -q -e giturl) it doesn’t appear to be installing master copies of dependencies in the job (http://logs.openstack.org/05/573005/2/check/vmware-tox-lower-constraints/946d5e3/job-output.txt.gz). If I do install them in tox, then it doesn’t appear to include the depends on patch presumably because | 17:35 |
| boden | pip install in my toxo overwrites them…. Can someone help me understand the proper way to do this? | 17:35 |
| clarkb | boden: can you link to your tox.ini file | 17:36 |
| *** lpetrut has joined #openstack-infra | 17:36 | |
| *** openstackgerrit has joined #openstack-infra | 17:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Enable doc and release notes for python-tempestconf https://review.openstack.org/572564 | 17:36 |
| clarkb | boden: in general though the way the "siblings" insatllation happens which ensure you get the depends on patch is that the tox jobs will do a tox run only install deps. Then update the tox env with the content from the zuul required-projects then run tox only running the test comands | 17:37 |
| boden | clarkb current copy of tox ini https://github.com/openstack/vmware-nsx/blob/master/tox.ini but also see changes I’ve been testing in the patch https://review.openstack.org/#/c/573005/2 | 17:37 |
| clarkb | boden: for all of that to work the job must have the require-projects specifed in the case of no explicit depends-on, explicit depends on shouldn't need that | 17:38 |
| boden | clarkb ack, note that this seems to only occur in our lower constraints job.. the py jobs are geen | 17:38 |
| clarkb | boden: it is possible the lower constraints job isn't using the siblings stuff | 17:38 |
| clarkb | hrm no it does | 17:38 |
| clarkb | http://logs.openstack.org/05/573005/2/check/vmware-tox-lower-constraints/946d5e3/job-output.txt.gz#_2018-06-07_13_34_48_897627 that is where it sees neutron | 17:39 |
| boden | clarkb do I need to define the constraints job in the project.yaml and add the required-projects to it? | 17:40 |
| *** heyongli has quit IRC | 17:40 | |
| corvus | boden, clarkb: neutron has to be a required projects for siblings to install it | 17:40 |
| *** heyongli has joined #openstack-infra | 17:41 | |
| clarkb | corvus: ah so depends-on isn't sufficient there? | 17:41 |
| corvus | nope, we filter like this: projects: "{{ zuul.projects.values() | selectattr('required') | list }}" | 17:41 |
| corvus | https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox/tasks/siblings.yaml#n13 | 17:42 |
| boden | clarkb corvus I’m sorry, but far from skilled here… are we saying that I should add neutron as a required projects for all my jobs (project.yaml) and also add a job for lower constraints with the required projects? | 17:42 |
| corvus | i think the idea was to avoid depends-on actually changing what the job tests :) | 17:42 |
| clarkb | corvus: got it | 17:43 |
| clarkb | boden: generally yes, it will have to be a required-project for the depends on in the context of tox siblings to work | 17:44 |
| *** cshastri has quit IRC | 17:46 | |
| corvus | boden: something like this: http://paste.openstack.org/show/722915/ | 17:46 |
| fungi | note that not necessarily all your jobs would need need neutron as a required-project, just the jobs which expect neutron to be installed | 17:46 |
| boden | corvus clarkb many thanks.. I’ll work on an update to projects.yaml and then hopefully bug you again quick for your input :) | 17:47 |
| corvus | boden: some of the other ones may already have this, for example, if you're using this template: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n361 | 17:47 |
| fungi | (i was about to say just the jobs which need neutron installed from source, but i don't think neutron is releasing to pypi yet, so this would be the only way to get neutron anyway i suppose) | 17:47 |
| corvus | clarkb, fungi: https://github.com/paramiko/paramiko/blob/master/paramiko/transport.py#L161 | 17:47 |
| boden | corvus yes, but I was told previously I didn’t need to add neturon to some of them (it failed when I did)… and the constraints job was never added (obviously) | 17:47 |
| boden | thanks again | 17:48 |
| boden | let me go break soem stuff now | 17:48 |
| *** felipemonteiro__ has joined #openstack-infra | 17:48 | |
| clarkb | ecdsa over rsa? interesting | 17:48 |
| fungi | corvus: so that's its default preferred order i guess? | 17:48 |
| clarkb | I was under the impression that anything with "nist" in it should be avoided | 17:49 |
| clarkb | or maybe that was ecdsa specific | 17:49 |
| corvus | clarkb, fungi: yeah, i think so. paramiko seems to most favor ed25519; openssh seems to prefer ecdsa-sha2-nistp256 | 17:49 |
| fungi | clarkb: yeah, ssh-ed25519 and ecdsa-sha2-nistp* are all elliptic curve so i guess the paramiko devs feel ecc is superior to rsa is superior to dss | 17:49 |
| *** iranzo has joined #openstack-infra | 17:51 | |
| *** heyongli has quit IRC | 17:51 | |
| fungi | presumably they expect that if the server administrator has added an ecc key of some kind, that should be used when possible and rsa is only intended as a backwards-compatible fallback in those situations | 17:51 |
| *** heyongli has joined #openstack-infra | 17:51 | |
| clarkb | corvus: is the problem that difference between them because the key isn't being accepted properly? | 17:51 |
| *** felipemonteiro_ has quit IRC | 17:52 | |
| corvus | clarkb: i'm having trouble parsing that sentence :( | 17:52 |
| clarkb | corvus: do you think the problem we had is related to paramiko and openssh having different preferences? | 17:52 |
| clarkb | hopefully that is better | 17:52 |
| fungi | or is the problem that paramiko is accepting an ecc host key for the channel even though it doesn't have an ecc client key to use? | 17:53 |
| corvus | clarkb: ah, i'm starting to suspect there's just something broken about the ed25519 algo on the remote end, since both paramiko and openssh fail when using that. but openssh *happens* to have a key order that puts it lower, so we don't usually see the error. | 17:53 |
| clarkb | gotcha | 17:53 |
| clarkb | also if openssh also fails then ya I would agree likely the remote | 17:54 |
| fungi | yeah, broken host key sounds plausible | 17:54 |
| clarkb | iirc gerrit only recently added support for that key type too so could be implementation problem as well | 17:54 |
| fungi | especially since i think they farm all that out to bc | 17:54 |
| fungi | so could even be a bc bug | 17:54 |
| corvus | can someone more up to date on this clarify the difference between ssh-ed25519 and ecdsa-sha2-nistp256 for me? or point me in the right direction? | 17:55 |
| fungi | rather, that mina-sshd farms that out to bc | 17:55 |
| fungi | corvus: they're different curves (different initial parameter sets if you will) | 17:55 |
| clarkb | corvus: two different algorithms aiui. The nist one uses unverifiable magic numbers that make some groups concerned iirc | 17:55 |
| fungi | ed25519 is basically djb's somewhat-arbitrarily chosen initialization for a curve, which has gained a lot of momentum as a de facto standard | 17:56 |
| corvus | so on the remote side, will gerrit have a ssh-ed25519 key, and a ecdsa-sha2-nistp256 key, and a rsa key, and ...etc? | 17:56 |
| fungi | yes | 17:56 |
| clarkb | corvus: yes | 17:56 |
| corvus | okay, so reporting something like "the ssh-ed25519 key on git.opendaylight.org doesn't seem to work" is a sensible thing to take over to the lf-relmgt channel | 17:57 |
| openstackgerrit | boden proposed openstack-infra/project-config master: fix vmware-nsx jobs https://review.openstack.org/573386 | 17:57 |
| *** e0ne has joined #openstack-infra | 17:57 | |
| fungi | corvus: yes, i believe so | 17:57 |
| clarkb | may want to clarify it is the server's host key but yes | 17:57 |
| corvus | sorry #lf-releng | 17:58 |
| corvus | i'm off there to do that now :) | 17:58 |
| *** armaan_ has joined #openstack-infra | 17:58 | |
| *** armaan has quit IRC | 17:58 | |
| fungi | a paste of the openssh error signature error from explicitly attempting to use ssh-ed25519 to connect to their gerrit would likely be helpful | 17:59 |
| fungi | what's the hostname of their gerrit? | 18:00 |
| clarkb | fungi: git.opendaylight.org iirc | 18:00 |
| clarkb | they run it under /review | 18:00 |
| fungi | thanks | 18:00 |
| *** heyongli has quit IRC | 18:01 | |
| *** heyongli has joined #openstack-infra | 18:01 | |
| corvus | zxiiro: i just reported an issue in #lf-releng, but if you want to see the history that lead to that conclusion it's all above ^ | 18:02 |
| clarkb | corvus: I'm going to push an update to our puppet to have known_hosts files in more places | 18:08 |
| *** r-daneel has joined #openstack-infra | 18:09 | |
| corvus | clarkb: i'm not certain that paramiko will use that | 18:10 |
| corvus | clarkb: yeah, zuul is only going to use the system host keys | 18:11 |
| *** apetrich has joined #openstack-infra | 18:11 | |
| corvus | which, to be fair, might be a better approach for us to take across the board :) | 18:11 |
| *** heyongli has quit IRC | 18:11 | |
| clarkb | hrm | 18:11 |
| *** heyongli has joined #openstack-infra | 18:11 | |
| clarkb | ok I'll update what we currently have to be at least accurate | 18:12 |
| corvus | if we wanted to add load_host_keys() to zuul, i think that would be okay | 18:14 |
| corvus | http://docs.paramiko.org/en/2.4/api/client.html#paramiko.client.SSHClient.load_host_keys | 18:14 |
| clarkb | least surprise for me would be to respect the current users known hosts as well as system hosts file | 18:15 |
| corvus | so i think the ways to solve this problem are: (1) fix git.opendaylight.org; (2) add an ssh-rsa host key to /etc/ssh/known_hosts; (3) add an ssh-rsa host key to ~zuul/.ssh/known_hosts [requires adding load_host_keys support to gerrit driver in zuul] | 18:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Update review.o.o host key data https://review.openstack.org/573397 | 18:16 |
| corvus | i should probably confirm that #3 works before we do it; i'll do that after lunch; i have to afk right now though | 18:16 |
| fungi | clarkb: the docs for load_system_host_keys() indicate it already has that behavior: " filename is left as None, an attempt will be made to read keys from the user’s local known hosts file, as used by OpenSSH..." | 18:19 |
| fungi | er, "If filename is left as None, ..." | 18:19 |
| *** mriedem has quit IRC | 18:20 | |
| clarkb | and zuul doesn't set it so ya it must be looking at those files already? | 18:21 |
| *** heyongli has quit IRC | 18:21 | |
| *** heyongli has joined #openstack-infra | 18:22 | |
| clarkb | I think 3) for zuul01 should get this sorted then? | 18:22 |
| fungi | that's how i interpret the paramiko docs, at least. i haven't looked under th hood to see if it's working as advertised | 18:22 |
| *** SumitNaiksatam has quit IRC | 18:23 | |
| zxiiro | corvus: cool looking now | 18:25 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Add opendaylight connection to zuul" https://review.openstack.org/573300 | 18:25 |
| clarkb | fungi: now that ansible update is largely behind us did you see that clouds.yaml on puppetmaster broke ansible yesterday evening | 18:26 |
| clarkb | fungi: corvus had to manually update it | 18:26 |
| clarkb | not sure if this will affect your changes to get those updated properly | 18:26 |
| mordred | clarkb: oh - what was the break? | 18:28 |
| *** SumitNaiksatam has joined #openstack-infra | 18:29 | |
| clarkb | mordred: I dunno corvus debugged it but oscc in openstack inventory script stopped being able to load the file. I expect that fungi updated it by hand based on irc timestamps | 18:29 |
| clarkb | and puppet couldn't run to fix it because anibsle could not run | 18:29 |
| clarkb | mostly want ot make sure fungi is aware so that the addition of new clouds doesn't rely on broken/out of date info | 18:30 |
| *** lpetrut has quit IRC | 18:31 | |
| *** aojea has quit IRC | 18:31 | |
| *** claudiub has joined #openstack-infra | 18:31 | |
| mordred | ugh. well - that seems like a terrible bug | 18:31 |
| *** heyongli has quit IRC | 18:32 | |
| *** heyongli has joined #openstack-infra | 18:32 | |
| fungi | clarkb: yeah, i replied to you and corvus in scrollback before you were likely awake | 18:33 |
| fungi | i did manually edit that file and then tested it with openstackclient and it worked fine | 18:33 |
| fungi | i didn't revert my test changes because i assumed (wrongly!) that puppet was just going to revert it to the templated version in a few minutes anyway | 18:33 |
| *** diablo_rojo has quit IRC | 18:33 | |
| fungi | sorry about that :/ | 18:33 |
| clarkb | aha that explains it then, thanks | 18:34 |
| fungi | very much confused though as to how it broke the ansible inventory | 18:34 |
| *** diablo_rojo has joined #openstack-infra | 18:34 | |
| clarkb | should be able to recreate it in a file to the side then manually select that file to debug it further | 18:34 |
| mordred | ++ I'd love a reproduction case | 18:34 |
| fungi | and, yeah... worried the packethost addition change might reintroduce that when it merges, so will keep an eye out | 18:35 |
| fungi | i basically just copied the limestone section and then edited the handful of parameters we needed to do packethost | 18:35 |
| clarkb | maybe a formatting bug? | 18:36 |
| fungi | and took out some of the overrides we don't yet know will be needed | 18:36 |
| *** lpetrut has joined #openstack-infra | 18:36 | |
| fungi | mmm... maybe? don't openstackclient and ansible both use os-client-config to parse it though? | 18:36 |
| fungi | so if it were a parsing error i'd expect them to both fail | 18:37 |
| clarkb | yes | 18:37 |
| *** gfidente has quit IRC | 18:37 | |
| *** trown|lunch is now known as trown | 18:38 | |
| *** gfidente has joined #openstack-infra | 18:38 | |
| *** gfidente has joined #openstack-infra | 18:38 | |
| *** lpetrut has quit IRC | 18:40 | |
| fungi | i'll see if we logged any sort of traceback | 18:40 |
| *** lpetrut has joined #openstack-infra | 18:40 | |
| clarkb | fungi: I +2'd the change to add in the new clouds.yaml config | 18:40 |
| clarkb | left a couple of notes that we might cleanup now or later | 18:41 |
| fungi | thanks! | 18:41 |
| *** heyongli has quit IRC | 18:42 | |
| *** heyongli has joined #openstack-infra | 18:42 | |
| *** e0ne has quit IRC | 18:43 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to create selection plans by summit https://review.openstack.org/573407 | 18:43 |
| *** spotz has quit IRC | 18:45 | |
| *** e0ne has joined #openstack-infra | 18:45 | |
| *** diablo_rojo has quit IRC | 18:46 | |
| *** spotz has joined #openstack-infra | 18:46 | |
| fungi | clarkb: mordred: yeah, this was the traceback http://paste.openstack.org/show/722919/ so i have a feeling i made bad assumptions in https://review.openstack.org/573021 about not needing to include a values dict under the region? | 18:47 |
| *** kjackal has quit IRC | 18:47 | |
| clarkb | fungi: look at city cloud if not doing extra values you just list the names looks like | 18:47 |
| *** armaan_ has quit IRC | 18:48 | |
| fungi | right, that's what i'm thinking in retrospect i should have done | 18:48 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to create selection plans by summit https://review.openstack.org/573407 | 18:48 |
| *** jesslampe has quit IRC | 18:49 | |
| *** jesslampe has joined #openstack-infra | 18:50 | |
| *** SumitNaiksatam has quit IRC | 18:50 | |
| fungi | no idea why that didn't break openstackclient when i tested it, but i guess maybe it doesn't exercise quite the same paths | 18:50 |
| *** SumitNaiksatam has joined #openstack-infra | 18:50 | |
| fungi | clarkb: i'll incorporate your suggestions into my next patchset | 18:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to specific flavor and slowly bump capacity https://review.openstack.org/573369 | 18:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Add cookbook-openstackclient repo jobs https://review.openstack.org/571506 | 18:51 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove beaker-*-rake jobs https://review.openstack.org/571927 | 18:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 18:52 |
| clarkb | AJaeger: mnaser ^ I had to update our linter check for system-required jobs | 18:52 |
| clarkb | fungi: mordred ^ that may interest you as well | 18:52 |
| *** heyongli has quit IRC | 18:52 | |
| *** neiloy has quit IRC | 18:52 | |
| *** greghaynes has quit IRC | 18:52 | |
| *** heyongli has joined #openstack-infra | 18:52 | |
| corvus | clarkb, fungi: the lf folks say it's a known bug in gerrit 2.14 and will be fixed in 2.15. so probably our manual ssh-rsa host key workaround is the way to go for now. i verified that will work with a test script. | 18:53 |
| clarkb | corvus: cool, did you see fungi's note that paramiko docs say that the load system host keys function will look for the filein the homedir if no filepath is passed? | 18:54 |
| *** gfidente has quit IRC | 18:55 | |
| corvus | clarkb: oh i missed that. i think that means we're good to go then as soon as we have puppet write ~zuul/.ssh/known_hosts | 18:55 |
| corvus | clarkb, fungi: and i just verified that with my test script too. | 18:55 |
| clarkb | yup why don't I update my change to fix those then we can add on odl's in a followup | 18:55 |
| corvus | k. lunch for me now. biab | 18:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Apply up to date gerrit host key to zuul servers https://review.openstack.org/573397 | 18:57 |
| clarkb | there | 18:57 |
| clarkb | corvus: we might suggest to lf that they delete their ed25519 host key(s) and restart gerrit. I think that will fix it for clients too | 18:58 |
| clarkb | unless gerrit makes a new one I guess ugh | 18:58 |
| clarkb | with openssh at least it doesn't create the files if they are missing (as we found with glean) | 18:59 |
| clarkb | hwoarang: speaking of glean, is latest glean working for you? | 18:59 |
| *** kjackal has joined #openstack-infra | 18:59 | |
| mordred | clarkb, fungi: I left some comments on https://review.openstack.org/#/c/573021/2 | 18:59 |
| *** greghaynes has joined #openstack-infra | 19:00 | |
| *** neiloy has joined #openstack-infra | 19:00 | |
| clarkb | mordred: we do identity CRUD operations though so probably want that too? | 19:00 |
| clarkb | things like change passwd? | 19:00 |
| *** beisner-sick is now known as beisner | 19:00 | |
| mordred | clarkb: shouldn't need it - it won't hurt anything - I mostly just wanted to communicate that it doesn't do what one might think at first | 19:02 |
| *** lpetrut has quit IRC | 19:02 | |
| clarkb | well if v2 is preferred (it was once not sure if currently) then the lack of the domain and all that will make it sad won't it? | 19:02 |
| *** heyongli has quit IRC | 19:02 | |
| *** heyongli has joined #openstack-infra | 19:03 | |
| mordred | right - but there is a domain. but identity_api_version != auth_type: v3password - the two are completely unrelated | 19:03 |
| clarkb | right but they use the same auth details don't they? | 19:04 |
| clarkb | so if you were to use v2 but only have v3 auth details it would haev a sad | 19:04 |
| mordred | nope - it would be fine | 19:04 |
| clarkb | (and vice versa v3 has a sad if not including the domain) | 19:04 |
| clarkb | is it only this direction that has problems? | 19:04 |
| mordred | wait - I think we're still saying different things | 19:04 |
| clarkb | maybe | 19:04 |
| clarkb | CRUD doesn't rely on any auth details once you've auth'd I guess | 19:04 |
| clarkb | even if you are changing those details | 19:04 |
| mordred | the default for auth_type is 'password' which autodetects v2password or v3password from the contents of the auth dict | 19:05 |
| mordred | if you manually put auth_type: v2password and then added _domain_name entries - or v3password and omitted them, authentication would have a sad | 19:05 |
| mordred | once you have authenticated, identity_api_version controls which endpoint should be found from the catalog | 19:05 |
| mordred | but you make those calls with the token you got from the authentication step | 19:06 |
| *** eernst has quit IRC | 19:06 | |
| clarkb | right, so you'd only need to set that if you wanted to add a domain? or similar? otherwise it could v2 and be confused? | 19:06 |
| mordred | yes - that's right | 19:07 |
| *** eernst has joined #openstack-infra | 19:07 | |
| mordred | all that said - there is no downside to setting identity_api_version - I mostly just wanted to vomit pedantic info your way for completeness | 19:08 |
| *** Goneri has joined #openstack-infra | 19:11 | |
| clarkb | I guess thats mostly what I'm worried about we are relying on v3 for auth and if you need to chagne those dtails will need v3 crud | 19:13 |
| *** heyongli has quit IRC | 19:13 | |
| *** heyongli has joined #openstack-infra | 19:13 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Remove legacy puppet-openstackci jobs https://review.openstack.org/571939 | 19:14 |
| anteaya | my provnce is having an election today so I'm a bit distracted with that | 19:15 |
| anteaya | not that anyone was waiting me on anything | 19:15 |
| anteaya | but just in case you care | 19:15 |
| clarkb | anteaya: is it a public holiday? In oregon we have like 20 days to vote by mail so its barely special other than needing to remember to do it | 19:16 |
| clarkb | often wish it were a holiday though :) | 19:16 |
| anteaya | not a holiday, no | 19:16 |
| anteaya | but employers are obligated to ensure their employees have time enough to vote | 19:17 |
| anteaya | polls are open 9 to 9 so anyone who works that entire time has to be given time off to vote | 19:17 |
| anteaya | and we have advance polls and special polling which would include by mail voting | 19:18 |
| anteaya | but today is when most people go into a polling station and vote | 19:18 |
| anteaya | and tonight we learn the results | 19:18 |
| *** mugsie has quit IRC | 19:19 | |
| *** mugsie has joined #openstack-infra | 19:21 | |
| *** mugsie has quit IRC | 19:21 | |
| *** mugsie has joined #openstack-infra | 19:21 | |
| *** mugsie has quit IRC | 19:21 | |
| *** mugsie has joined #openstack-infra | 19:21 | |
| *** mugsie has quit IRC | 19:21 | |
| *** mugsie has joined #openstack-infra | 19:21 | |
| clarkb | here everyone votes my mail but they must receive your ballot by 8pm election day so you have to deliver to a collection box during the last few days | 19:21 |
| clarkb | then we have results later in the evening | 19:21 |
| anteaya | cool | 19:22 |
| *** eharney has quit IRC | 19:22 | |
| *** mugsie has quit IRC | 19:22 | |
| *** heyongli has quit IRC | 19:23 | |
| anteaya | I think there might be a few obstacles to implimenting that system in ontario | 19:23 |
| *** heyongli has joined #openstack-infra | 19:23 | |
| anteaya | in the urban places I think it would be okay | 19:23 |
| *** Guest68045 has joined #openstack-infra | 19:24 | |
| anteaya | but some of the province is very remote, so the getting to a given collection box might be a problem | 19:24 |
| anteaya | depends on where they were located and how many of them their were, I guess | 19:24 |
| *** Guest68045 has quit IRC | 19:24 | |
| *** mugsie_ has joined #openstack-infra | 19:24 | |
| clarkb | ya there are some logistical challenges like not everyone may have an address to receive a ballot so they do third party deliveries here aiui | 19:26 |
| clarkb | (I don't personally have that issue so not sure of all the details) | 19:26 |
| *** felipemonteiro__ has quit IRC | 19:26 | |
| anteaya | yes indeed, that would alsos be an issue | 19:27 |
| *** mugsie_ has quit IRC | 19:27 | |
| *** mugsie_ has joined #openstack-infra | 19:27 | |
| *** mugsie_ is now known as _mugsie | 19:27 | |
| anteaya | usually we get voter's cards in the mail, everyone registered on the voter's list is supposed to recieve one | 19:27 |
| *** lpetrut has joined #openstack-infra | 19:27 | |
| anteaya | except this time canada post had a technical oopsie and 500,000 people didn't get their voter's card | 19:27 |
| anteaya | I was one of them | 19:28 |
| anteaya | the helpful shrug has limited value in this circumstance | 19:28 |
| *** _mugsie is now known as mugsie_ | 19:28 | |
| *** d0ugal has joined #openstack-infra | 19:29 | |
| *** d0ugal has quit IRC | 19:29 | |
| *** d0ugal has joined #openstack-infra | 19:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 19:29 |
| *** diablo_rojo has joined #openstack-infra | 19:29 | |
| clarkb | mordred: ^ I'll get it right eventually | 19:29 |
| mordred | clarkb: so you say | 19:31 |
| clarkb | mostly I just want a thing that runs and fails then we can more easily point at kata for pushing it further along maybe | 19:32 |
| mordred | clarkb: hah. yeah. I should have caughtthat | 19:32 |
| hogepodge | so on zuul npm jobs, do we have a migration path to use 'yarn test' instead of 'npm test'? | 19:33 |
| *** heyongli has quit IRC | 19:33 | |
| *** heyongli has joined #openstack-infra | 19:33 | |
| clarkb | hogepodge: we probably want yarn specific jobs then projects can move when they are yarnable? | 19:33 |
| mordred | clarkb: the normal jobs use yarn if they find yarn.lock otherwise they use npm | 19:34 |
| clarkb | oh interesting | 19:34 |
| mordred | mordred@solace:~/src/git.openstack.org/openstack-infra/zuul$ yarn test | 19:34 |
| mordred | yarn run v1.7.0 | 19:34 |
| mordred | error Command "test" not found. | 19:34 |
| mordred | hogepodge: ^^ what does one need to do to have yarn test work? | 19:34 |
| mordred | hogepodge: ah - so yarn run works like npm run doesn't it? | 19:35 |
| hogepodge | ah, ok, so that means I should add the yarn.lock file to my project? | 19:35 |
| *** boris_42_ has quit IRC | 19:36 | |
| mordred | hogepodge: well ... the only thing it'll affect right now for thejobs is usingyarn install instead of npm install | 19:36 |
| mordred | BUT - we could extend that to use yarn run instead of npm run as well | 19:36 |
| *** kjackal has quit IRC | 19:36 | |
| hogepodge | mordred: this is the patch in question https://review.openstack.org/#/c/573404/ | 19:37 |
| mordred | hogepodge: and yes - although it sort of depends if your project is a library or an application | 19:37 |
| mordred | hogepodge: looking | 19:37 |
| mordred | hogepodge: yes - for that definitely git add yarn.lock | 19:37 |
| hogepodge | the most important changes being to package.json | 19:37 |
| hogepodge | we're migrating away from bower which is warning us the world (of bower) is about to end. | 19:38 |
| *** slaweq_ has quit IRC | 19:38 | |
| mordred | yes. bower should stop being used for sure | 19:38 |
| clarkb | I remember when we had to change everython TO bower it wasn't that long ago | 19:39 |
| mordred | hogepodge: fwiw, we're using yarn for zuul dashboard and storyboard-webclient | 19:39 |
| clarkb | I guess they decided their fully decentrailized system was bad afterall | 19:39 |
| mordred | clarkb: I think it was longer ago than it's pleasant to think about :) | 19:39 |
| hogepodge | I'm also thinking that in the interest of application maintenance (since we're dropping down to a really small team), requiring that the application be run inside of a docker container mainly to manage dependencies but also testing and development. | 19:39 |
| hogepodge | It's unpleasant to be thinking about all this right now. | 19:40 |
| * hogepodge wants refstack to go on autopilot so he can think about more interesting problems | 19:40 | |
| clarkb | putting things in a container doesn't change the fact you have to list and manage dependencies | 19:40 |
| mordred | hogepodge: https://review.openstack.org/#/c/528436/ <-- there's the storyboard-webclient patch moving from bower to yarn, fwiw | 19:40 |
| clarkb | they have to get into the container somehow | 19:40 |
| mordred | and yeah - I'd actually argue that yarn is pretty awesome - so docker will just add complexity ... but whatever floats your boat :) | 19:41 |
| hogepodge | clarkb: our dev workflow (docker or yolo) is different from our deploy workflow (puppet) and with these changes coming in our puppet manifest will break badly | 19:42 |
| mordred | hogepodge: lemme look at the refstack repo and see if there are any easy ways I can see to help you take advantage of anything else we've got | 19:42 |
| *** slaweq has joined #openstack-infra | 19:43 | |
| hogepodge | removing the puppet portion to install dependencies in production will just be the removal of another headache for me | 19:43 |
| *** heyongli has quit IRC | 19:43 | |
| *** heyongli has joined #openstack-infra | 19:44 | |
| clarkb | hogepodge: there is work to start migrating to less puppet in infra but is still in spec writing phase. Your input on that much appreciated if interested | 19:44 |
| mordred | hogepodge: k. well - lemme look at what's there and make some suggestions as to things that exist currently that can be leveraged to make your life easier | 19:44 |
| hogepodge | clarkb: sure, I'm happy to contribute. | 19:44 |
| hogepodge | mordred: my main issue right now is moving to yarn means going to at least Node 8. Xenial installs Node 6 by default, so I have to update the puppet deploy manifests to install the new repo and install the correct version of Node. | 19:46 |
| hogepodge | mordred: sure, that would be great. We have a bunch of technical debt we're trying to catch up on too (including migrating to Python 3 so that we're not badly burned by that either) | 19:46 |
| clarkb | hogepodge: thats fairly straightforward and we should have examples of that | 19:46 |
| clarkb | hogepodge: we've had to do similar with other things | 19:46 |
| hogepodge | clarkb: sure, It's just work that annoys me, especially when it's spread across multiple repositories (one of which we don't own) | 19:47 |
| * clarkb digs up an example | 19:47 | |
| mordred | hogepodge: yah. SO ... the pattern we use for these sorts of thigns elsewhere is to build the javascript stuff in zuul jobs and publish it to a tarball - that tarball then gets deployed on the target server - so no node is needed on the server at all | 19:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Add kata-run job to kata-containers/proxy https://review.openstack.org/572452 | 19:49 |
| clarkb | oh hey it did work this time | 19:49 |
| mordred | but - I totally hear you - I'm reading through things right now so I can make more specific suggestions | 19:49 |
| *** eernst has joined #openstack-infra | 19:49 | |
| * clarkb clones a kata-containers/proxy repo to make a pull request and see what happens | 19:51 | |
| clarkb | oh I have to click buttons in github first don't I | 19:52 |
| mordred | yup | 19:52 |
| mordred | hogepodge: ok. I can see several ways I can improve your life greatly | 19:53 |
| *** heyongli has quit IRC | 19:54 | |
| *** heyongli has joined #openstack-infra | 19:54 | |
| hogepodge | mordred: life being improved greatly would be welcome | 19:57 |
| mordred | hogepodge: I'll get you some patches in just a bit - the pattern you need/are using is one we've got in place in several other places, and you're not far from it | 19:58 |
| clarkb | I think I got ahead of myself with nodepool seeing the new label for kata jobs, but it totally tried to run a job then failed | 19:58 |
| hogepodge | mordred: cool | 19:58 |
| clarkb | the github bits would appear to be working | 19:58 |
| mordred | clarkb: woot | 19:59 |
| mordred | clarkb: you might also windup needing to add support to the base job for doing a submodule update | 19:59 |
| corvus | what's the plan to get to multi-tenancy? | 20:00 |
| mordred | corvus: you mean like a zuul tenant for kata? | 20:00 |
| corvus | yep | 20:00 |
| mordred | corvus: dunno. clarkb thoughts? | 20:01 |
| clarkb | I think we start by proving that zuul can work for them using third party check. Run a job like the kata-run job that just executes their base run.sh stuff. | 20:03 |
| corvus | i'm really worried about doing anything more than that | 20:03 |
| corvus | i think the experience is going to be terrible without multi-tenancy | 20:04 |
| *** eharney has joined #openstack-infra | 20:04 | |
| *** heyongli has quit IRC | 20:04 | |
| corvus | so i think setting this up, and then saying "go forth and try to use it" in a single-tenant configuration is the best way to encourage them to hate zuul | 20:04 |
| clarkb | I want to start ther ebecause they continue to push on jenkins (irc logs from today for example) so don't want to go full multitenant for them without an idea if they will use it long term | 20:04 |
| *** iranzo has quit IRC | 20:04 | |
| *** heyongli has joined #openstack-infra | 20:04 | |
| clarkb | and I agree I think to take advantage of zuul properly we will want a separate tenant for them so that they can have self testing changes | 20:05 |
| clarkb | I just don't want to make that a requirement, do all the work, then have them decide to keep using jenkins | 20:06 |
| corvus | clarkb: perhaps you should at least put the job in openstack-zuul-jobs? | 20:06 |
| clarkb | corvus: sure I can move it. I guess that way dependson will work | 20:06 |
| corvus | and changes to it are self-testing | 20:07 |
| clarkb | well only if we run the test in ozj's pipeline right? | 20:07 |
| clarkb | *check/gate | 20:07 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to update selection plan by summit https://review.openstack.org/573422 | 20:07 |
| clarkb | oh with dpeends on we can get that too yes | 20:07 |
| clarkb | as for adding a tenant I think we'll have to whitelabel a webserver for status unless the winterstack conversation ends up with a generic name then they can just be a generic tenant (and whitelable themselvesi f they like) | 20:08 |
| corvus | that wfm | 20:09 |
| mordred | ++ | 20:09 |
| *** felipemonteiro has joined #openstack-infra | 20:09 | |
| corvus | how do we propose a change to create zuul.katacontainers.io? | 20:09 |
| clarkb | corvus: that would be a vhost on zuul01 I expect, so through puppet for that? (assuming we host it in infra) | 20:10 |
| corvus | i mean the dns name | 20:10 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to update selection plan by summit https://review.openstack.org/573422 | 20:10 |
| clarkb | their dns is in our shared dns management iirc | 20:10 |
| clarkb | we should have access to it but I can double check | 20:11 |
| corvus | okay, so we can use the usual dns.py script? | 20:11 |
| clarkb | I think sop | 20:11 |
| clarkb | I'll check in a moment after I move the job definition | 20:11 |
| corvus | i imagine we should get someone to say that's okay first, but assuming it is, it sounds like we have a workable path forward | 20:11 |
| corvus | i'm a little less worried now :) | 20:11 |
| *** heyongli has quit IRC | 20:14 | |
| *** lpetrut has quit IRC | 20:15 | |
| *** heyongli has joined #openstack-infra | 20:15 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to get selection plan by id https://review.openstack.org/573424 | 20:15 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to get selection plan by id https://review.openstack.org/573424 | 20:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Move kata-run to this repo https://review.openstack.org/573426 | 20:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Use kata-runsh out of OZJ https://review.openstack.org/573427 | 20:18 |
| clarkb | corvus: ^ that should move the job | 20:18 |
| clarkb | corvus: and yes katacontainers.io is in our shared dns management system | 20:20 |
| clarkb | I too agree we should ask permission/let people know we may create/update records there if this progresses but don't expect that to be an issue | 20:21 |
| *** d0ugal has quit IRC | 20:21 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:22 | |
| corvus | both lgtm | 20:23 |
| *** heyongli has quit IRC | 20:24 | |
| *** heyongli has joined #openstack-infra | 20:25 | |
| *** felipemonteiro has quit IRC | 20:25 | |
| clarkb | To summarize, we'll start with third party check to prove that it can work. If kata want to take it further from there we will make a new zuul tenant for them and whitelabel it under katacontainers.io if winterstack naming is not done. If winterstack name is done they can be a tenant under winterstack | 20:30 |
| corvus | ++ | 20:30 |
| fungi | yeah, that all sounds great to me | 20:31 |
| *** e0ne has quit IRC | 20:32 | |
| *** e0ne has joined #openstack-infra | 20:33 | |
| clarkb | and for those that may not be super familiar the prove it can work bits are largely going to revolve around nested virt and possibly the azure nodepool driver. I think we've proven that zuul can work with github at this point | 20:34 |
| *** heyongli has quit IRC | 20:35 | |
| clarkb | and now lunch | 20:35 |
| *** heyongli has joined #openstack-infra | 20:35 | |
| *** pcaruana has quit IRC | 20:39 | |
| *** alkhodos_ has joined #openstack-infra | 20:41 | |
| *** kjackal has joined #openstack-infra | 20:42 | |
| *** e0ne has quit IRC | 20:44 | |
| *** heyongli has quit IRC | 20:45 | |
| *** heyongli has joined #openstack-infra | 20:45 | |
| *** VW_ has joined #openstack-infra | 20:45 | |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** mriedem1 has joined #openstack-infra | 20:47 | |
| *** armaan has joined #openstack-infra | 20:47 | |
| *** VW has quit IRC | 20:49 | |
| *** e0ne has quit IRC | 20:49 | |
| *** VW_ has quit IRC | 20:50 | |
| fungi | in later-than-lunch news, i'm obsessing over what internal temperature i need to get baked potatoes to in my smoker | 20:51 |
| fungi | also experimenting to see if the same holds true for rutabaga | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Accumulate errors in context managers https://review.openstack.org/573438 | 20:52 |
| *** trown is now known as trown|outtypewww | 20:52 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move kata-run to this repo https://review.openstack.org/573426 | 20:55 |
| *** heyongli has quit IRC | 20:55 | |
| *** heyongli has joined #openstack-infra | 20:55 | |
| *** mriedem1 is now known as mriedem | 20:58 | |
| *** kjackal has quit IRC | 20:58 | |
| *** kjackal has joined #openstack-infra | 20:58 | |
| *** germs has joined #openstack-infra | 20:59 | |
| *** dbecker has quit IRC | 21:00 | |
| *** germs has quit IRC | 21:01 | |
| *** Goneri has quit IRC | 21:01 | |
| *** germs has joined #openstack-infra | 21:03 | |
| *** germs has quit IRC | 21:04 | |
| *** erlon has quit IRC | 21:04 | |
| *** heyongli has quit IRC | 21:05 | |
| *** heyongli has joined #openstack-infra | 21:06 | |
| ianw | corvus: oh geez sorry, i put in a test alert for the grafana, saw it worked, deleted it, went to bed, but actually forgot to click OK :) | 21:07 |
| mnaser | http://grafana.openstack.org/dashboard/db/nodepool-vexxhost | 21:09 |
| mnaser | does anyone have any ideas why nothing appears thre | 21:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Use kata-runsh out of OZJ https://review.openstack.org/573427 | 21:10 |
| mnaser | https://review.openstack.org/#/c/573369/ merged 2 hours ago | 21:10 |
| mnaser | i guess that has to do with the issues clarkb saw with NODE_FAILURE | 21:10 |
| mordred | fungi: so - fwiw, I have recently been cooking baked potatoes for 3 hours at 350 | 21:10 |
| clarkb | mnaser: Its next on my list to look but finishing up lunch | 21:11 |
| mordred | fungi: which doesn't answer your question - but I've found the results of that to be extremely pleasing | 21:11 |
| fungi | mordred: yeah, challenge is slow-cooking in a smoker wher ethe temperature is somewhat variable... internets say 100c/212f | 21:11 |
| fungi | core temperature | 21:12 |
| mordred | yah | 21:12 |
| fungi | which is easy enough for me to measure with an instant-read digital thermometer | 21:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to delete selection plan by summit https://review.openstack.org/573442 | 21:12 |
| fungi | mordred: i'm less worried about potatoes, bigger question is whether the same holds true for rutabaga | 21:13 |
| fungi | but i'll be finding out soon enough | 21:13 |
| mordred | fungi: well - luckily they are vegetables -so the answer is "eat them and see if they are done" | 21:13 |
| fungi | bingo | 21:14 |
| *** myoung is now known as myoung|off | 21:14 | |
| fungi | my theory is that most tubers are done at a similar core temperature. curious to see if i'm right | 21:15 |
| clarkb | mordred: Exception: Unable to find flavor with min ram: 8000 | 21:15 |
| clarkb | er mnaser ^ | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** heyongli has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to delete selection plan by summit https://review.openstack.org/573442 | 21:16 |
| *** heyongli has joined #openstack-infra | 21:16 | |
| clarkb | mnaser: oh! openstackci-vexxhost has the new flavor but not openstackjenkins-vexxhost | 21:16 |
| clarkb | mnaser: so we need to add the flavor to the nodepool tenant | 21:16 |
| clarkb | sorry I should've checked both earlier | 21:16 |
| mnaser | clarkb: what’s the specs of the osf-highcpu-8 flavour thay you see? | 21:16 |
| mnaser | Ah yes that’ll explains it | 21:16 |
| clarkb | I'll double check the specs where I do see it too | 21:17 |
| ianw | fungi: cooks illustrated had a big thing on sweet potato, or yams, or whatever you call them. 9-10 minutes in microwave, then 1 hour at about 400F (200C). they go into the starch development etc etc | 21:17 |
| mnaser | So I added the flavour to the wrong project. Shucks | 21:17 |
| clarkb | ram: 8192 disk: 0 ephemeral: 0 vcpu: 0 | 21:17 |
| clarkb | mnaser: ^ | 21:17 |
| fungi | ianw: yeah, i'm not surprised there's some chemistry happening at certain temperature thresholds | 21:19 |
| *** nicolasbock has quit IRC | 21:22 | |
| *** armaan has quit IRC | 21:25 | |
| *** heyongli has quit IRC | 21:26 | |
| clarkb | mnaser: let me know if there is anything else I can do to help | 21:26 |
| *** heyongli has joined #openstack-infra | 21:26 | |
| *** felipemonteiro_ has quit IRC | 21:28 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:29 | |
| hogepodge | mordred: also, for some reason the npm tests are failing here with git access errors, if you have any ideas what's going on https://review.openstack.org/#/c/573404/ | 21:29 |
| clarkb | hogepodge: its trying to pull git@github.com:twbs/bootstrap.git over ssh which doesn't work because the zuul test user dones't have an ssh key for github | 21:32 |
| clarkb | I think | 21:32 |
| clarkb | before that it appears to use https | 21:33 |
| hogepodge | That's what it looks like to me too. Yeah, seems like https would be the answer since it's just an unprivileged pull. | 21:33 |
| *** rajinir has quit IRC | 21:34 | |
| *** heyongli has quit IRC | 21:36 | |
| *** heyongli has joined #openstack-infra | 21:36 | |
| mnaser | clarkb: done, they should now appear | 21:37 |
| clarkb | I see the flavor in a listing now, thanks | 21:38 |
| clarkb | mordred: ^ any idea how long shade/nodepool will cache the flavor list? | 21:38 |
| clarkb | I guess if it doesn't update reasonably soon I'll restart the daemon | 21:38 |
| *** edmondsw has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 21:38 |
| clarkb | I'll give it 5 minutes | 21:39 |
| clarkb | looks like we may set the flavors attribute if unset then never update it | 21:42 |
| *** boden has quit IRC | 21:43 | |
| clarkb | ya its still failing. infra-root I'm going to restart nodepool-launcher on nl03 fyi | 21:43 |
| clarkb | this is to pick up a flavor update in vexxhost | 21:44 |
| *** dhajare has quit IRC | 21:44 | |
| *** dhajare has joined #openstack-infra | 21:46 | |
| *** heyongli has quit IRC | 21:46 | |
| *** heyongli has joined #openstack-infra | 21:47 | |
| clarkb | mnaser: working now but /dev/vda1 14G 8.7G 4.1G 68% / that seems small? | 21:47 |
| mnaser | Ugh did I forget to setup local disk | 21:47 |
| clarkb | (I may have to disable it because I think jobs will fail on that | 21:47 |
| mnaser | Let me swap it quickly | 21:47 |
| clarkb | ok | 21:47 |
| mnaser | Of course my laptop crashes. Working on this. Just a quick reboot | 21:50 |
| *** boden has joined #openstack-infra | 21:52 | |
| mnaser | clarkb: recreated the flavor | 21:52 |
| mnaser | same name but different id | 21:53 |
| mnaser | nodepool will probably complain because the id is different | 21:53 |
| clarkb | ok I'll restart again so that it picks up the new data | 21:54 |
| *** janki has joined #openstack-infra | 21:55 | |
| *** heyongli has quit IRC | 21:57 | |
| mnaser | cool, should be good now | 21:57 |
| mnaser | i see ubuntu-xenial-vexxhost-ca-ymq-1-0004352087 | 21:57 |
| *** heyongli has joined #openstack-infra | 21:57 | |
| clarkb | ya its looking better now. I do see a problem with how nodepool allocates nodes though and having a label that can only be fulfilled by one cloud region | 21:59 |
| *** jcoufal has quit IRC | 21:59 | |
| *** dizquierdo has joined #openstack-infra | 22:00 | |
| clarkb | corvus: Shrews: the problem is if a provider is at capacity and it is the only provider that can provide a label it will decline the request anyway and let others try (and also decline) until the request fails | 22:00 |
| *** hashar has quit IRC | 22:00 | |
| *** boden has quit IRC | 22:01 | |
| *** boden has joined #openstack-infra | 22:01 | |
| clarkb | reading the code that may be a min ready specific problem | 22:02 |
| clarkb | and jobs would queue properly if the demand comes from a job instead | 22:02 |
| corvus | clarkb: are you sure? i was about to agree that is a design defect in the algorithm in general. | 22:03 |
| *** lifeless_ has quit IRC | 22:03 | |
| corvus | clarkb: oh, nope you're right :) | 22:04 |
| clarkb | corvus: ya we seem to only consider max_servers if the job is for min ready | 22:04 |
| clarkb | so I think its probably mostly ok | 22:04 |
| clarkb | but somethign we may want to improve later | 22:04 |
| *** lifeless has joined #openstack-infra | 22:04 | |
| clarkb | anyways time now to rebase my PR and see what happens | 22:04 |
| *** hamzy has quit IRC | 22:04 | |
| corvus | clarkb: yeah. tbh, if the only provider that can provide a label is at quota, it may not be terrible to have min-ready delayed :) | 22:05 |
| *** hamzy has joined #openstack-infra | 22:05 | |
| *** heyongli has quit IRC | 22:07 | |
| *** heyongli has joined #openstack-infra | 22:07 | |
| clarkb | https://zuul.openstack.org/stream.html?uuid=8fcee773f1444dc1809f6b3407f5479a&logfile=console.log its working :) | 22:07 |
| clarkb | I mean as far as running the job goes, thanks mnaser | 22:08 |
| corvus | it looks like "Make file matchers overridable" change has (more than) sufficient buy-in. how about we plan on putting that into production on monday? | 22:08 |
| clarkb | wfm | 22:09 |
| dmsimard | ansible basically wrote their own reno :/ https://github.com/ansible/ansible/commit/70c475da6c76bf9d34e56042b0f89c0e5ae16b79 | 22:09 |
| corvus | the cutover day is going to be sort of a wildcard day for zuul jobs. we won't really know what's going to break in advance. but hopefully not much will break, and it should be way easier to fix if it does. | 22:09 |
| mnaser | clarkb: I wonder if the mirrors are okay at our infra | 22:09 |
| mnaser | It’s been stuck at fetching dependencies for a while | 22:10 |
| clarkb | ya http and ssh to it is not happy | 22:10 |
| clarkb | looking into it now | 22:11 |
| clarkb | I'll try rebooting the instance | 22:12 |
| *** slaweq has quit IRC | 22:13 | |
| *** slaweq has joined #openstack-infra | 22:14 | |
| *** neiloy has quit IRC | 22:15 | |
| clarkb | mnaser: 71cdb525-cd30-4763-bb0e-90f4a11cc6db is the instance fwiw | 22:15 |
| clarkb | I'm waiting for it to come back after asking it "nicely" to reboot | 22:15 |
| clarkb | we may need to stop nodepool on vexxhost again and make a new server if this one is really out to lunch | 22:16 |
| clarkb | :/ | 22:16 |
| *** heyongli has quit IRC | 22:17 | |
| *** heyongli has joined #openstack-infra | 22:17 | |
| mnaser | clarkb: is it just this one or other jobs too? | 22:18 |
| *** kjackal has quit IRC | 22:18 | |
| clarkb | I think its all of them as its shared infra that mirror host | 22:18 |
| clarkb | it doesn't appear to be coming back so we should probably disable vexxhost until we can resurrect it | 22:19 |
| clarkb | or make a new one | 22:19 |
| *** slaweq has quit IRC | 22:19 | |
| *** eharney has quit IRC | 22:19 | |
| clarkb | I've set max-servers back to 0 and put nl03.o.o in emergency file | 22:20 |
| clarkb | corvus: noticed that the zuul executors are still in the emergency file should we remove them at this point | 22:20 |
| *** boden has quit IRC | 22:20 | |
| clarkb | #status log Added vexxhost back to nl03 but our mirror node is unhappy there so have temporarily disabled the cloud again until the mirror node is up and running | 22:21 |
| openstackstatus | clarkb: finished logging | 22:21 |
| *** SumitNaiksatam has quit IRC | 22:23 | |
| clarkb | mnaser: do you have a preference for us trying to understand what hapepned to that instance and fixing it or just deleting it and making a new one? | 22:23 |
| clarkb | mnaser: more than happy to help debug if we want ot go that route | 22:24 |
| *** rcernin has joined #openstack-infra | 22:24 | |
| corvus | clarkb: oh thanks! will do | 22:25 |
| corvus | clarkb: how's this look? https://etherpad.openstack.org/p/ixaVwbe3Yh | 22:25 |
| clarkb | corvus: lgtm | 22:26 |
| corvus | removed ze* from emergency | 22:27 |
| clarkb | mnaser: that said I'm running out of day where I can be useful. It is wedding anniversary today so i am on the hook for dinner tonight and I'll have to pop out soonish to make that happen | 22:27 |
| *** heyongli has quit IRC | 22:27 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 22:28 |
| *** heyongli has joined #openstack-infra | 22:28 | |
| corvus | i'm going to be away tomorrow | 22:28 |
| mnaser | clarkb: I have a few moments. What’s your cutoff? :p | 22:29 |
| mnaser | oh you know what | 22:30 |
| clarkb | mnaser: I have about an hour | 22:30 |
| mnaser | How hard is it to build a new mirror vm? | 22:30 |
| clarkb | mnaser: it shouldn't be hard am happy to do that instead | 22:30 |
| mnaser | So the idea behind that is | 22:30 |
| clarkb | wasn't sure how important it is for you on cloud side to dig into broken VMs | 22:30 |
| mnaser | I can set it up on that osf flavour | 22:30 |
| mnaser | So you can get non throttled network | 22:30 |
| mnaser | Fix two things at once. The v1 platform and instances are really really old. | 22:31 |
| clarkb | I think it is a v2 instance, | 22:31 |
| clarkb | v2-standard-2 | 22:31 |
| ianw | mnaser / clarkb : happy to help rebuild mirrors etc if it helps | 22:32 |
| mnaser | oh if it’s a v2 instance something is wrong | 22:32 |
| mnaser | Let me dig | 22:32 |
| *** edmondsw has joined #openstack-infra | 22:36 | |
| mnaser | oh | 22:36 |
| mnaser | hm | 22:37 |
| mnaser | it looks like the hypervisor that it sits on seems to have gone on vacation | 22:37 |
| clarkb | yay I'm being helpful :P | 22:37 |
| *** heyongli has quit IRC | 22:38 | |
| mnaser | and funnily enough | 22:38 |
| mnaser | you were the only vm on it | 22:38 |
| mnaser | lol | 22:38 |
| *** heyongli has joined #openstack-infra | 22:38 | |
| *** rwsu has quit IRC | 22:39 | |
| mnaser | clarkb: evacuated the instance onto another hypervisor | 22:40 |
| mnaser | it's pingable now | 22:40 |
| *** edmondsw has quit IRC | 22:41 | |
| clarkb | mnaser: thanks I'll check on it and if its running the apache service we can turn vexxhost back on | 22:41 |
| mnaser | clarkb: http://mirror.ca-ymq-1.vexxhost.openstack.org/ looks like thats on too | 22:41 |
| *** eernst has quit IRC | 22:42 | |
| clarkb | yup lgtm I'll undo my temporary disablement | 22:42 |
| clarkb | thank you for looking at that | 22:42 |
| *** eernst has joined #openstack-infra | 22:42 | |
| clarkb | max servers set back to 10 | 22:42 |
| mnaser | i'll stick around a bit more and see how things break in other ways :) | 22:42 |
| clarkb | and nl03 is removed from the emergency file | 22:43 |
| clarkb | ianw: ^ probably the biggest thing at this point is just keeping a rough eye on jobs running on vexxhost | 22:43 |
| *** eernst has quit IRC | 22:47 | |
| *** eernst has joined #openstack-infra | 22:48 | |
| *** heyongli has quit IRC | 22:48 | |
| *** heyongli has joined #openstack-infra | 22:48 | |
| *** felipemonteiro_ has quit IRC | 22:48 | |
| *** felipemonteiro has joined #openstack-infra | 22:49 | |
| clarkb | https://zuul.openstack.org/stream.html?uuid=4d84df0d28e642e28df28262b8799539&logfile=console.log should be happier now | 22:49 |
| clarkb | lets watch and find out | 22:49 |
| *** diablo_rojo has quit IRC | 22:50 | |
| *** eernst has quit IRC | 22:51 | |
| *** rwsu has joined #openstack-infra | 22:51 | |
| *** mriedem is now known as mriedem_afk | 22:52 | |
| clarkb | and now to debug why the script isn't working but thats awesome progress I Think | 22:52 |
| *** janki has quit IRC | 22:52 | |
| *** eernst has joined #openstack-infra | 22:53 | |
| *** eernst has quit IRC | 22:53 | |
| *** eernst has joined #openstack-infra | 22:54 | |
| mnaser | clarkb: don’t forget dinner :p | 22:54 |
| *** dizquierdo has quit IRC | 22:54 | |
| *** eernst has quit IRC | 22:54 | |
| *** eernst has joined #openstack-infra | 22:55 | |
| clarkb | I haven't yet, now that I have logs I'll share them with some people and then context switch to that | 22:55 |
| clarkb | I think we may need go installed so that will be a new pre task and also set GOPATH but that is work for another day | 22:56 |
| *** dklyle has joined #openstack-infra | 22:56 | |
| *** eernst has quit IRC | 22:58 | |
| *** eernst has joined #openstack-infra | 22:58 | |
| *** heyongli has quit IRC | 22:58 | |
| *** heyongli has joined #openstack-infra | 22:59 | |
| *** eernst_ has joined #openstack-infra | 23:00 | |
| *** eernst has quit IRC | 23:00 | |
| *** Goneri has joined #openstack-infra | 23:00 | |
| ianw | ok | 23:02 |
| *** eernst_ has quit IRC | 23:04 | |
| clarkb | ianw: based on the job above I think things should be happy at this point. I've also double checked the root device size and it is the more expected 75GB | 23:04 |
| *** adreznec has quit IRC | 23:04 | |
| *** adreznec has joined #openstack-infra | 23:06 | |
| ianw | clarkb: ok, i will keep an eye. it would be great if we could write some sort of basic rule to monitor things and alert us if things were out of bounds ;) | 23:08 |
| *** heyongli has quit IRC | 23:08 | |
| *** heyongli has joined #openstack-infra | 23:09 | |
| clarkb | ya though we removed nodepool's direct inspection of that stuff, maybe have something look at logstash data and trigger on that | 23:10 |
| clarkb | or a special job that just explicitly checks? | 23:10 |
| *** rpioso is now known as rpioso|afk | 23:15 | |
| *** heyongli has quit IRC | 23:19 | |
| *** heyongli has joined #openstack-infra | 23:19 | |
| *** felipemonteiro_ has joined #openstack-infra | 23:21 | |
| openstackgerrit | Zane Bitter proposed openstack-infra/project-config master: Add openstack-service-broker project https://review.openstack.org/573459 | 23:22 |
| *** felipemonteiro has quit IRC | 23:22 | |
| openstackgerrit | Zane Bitter proposed openstack-infra/project-config master: Add Zuul config for openstack-service-broker https://review.openstack.org/573460 | 23:23 |
| *** aeng has joined #openstack-infra | 23:25 | |
| *** Goneri has quit IRC | 23:27 | |
| *** heyongli has quit IRC | 23:29 | |
| *** heyongli has joined #openstack-infra | 23:29 | |
| ianw | clarkb: oh i was being cheeky talking about openstackgrafana alerting ... just getting last bits of grafyaml setup etc together | 23:34 |
| clarkb | ah. re that corvus didnt appear to be a big fan. Maybe we put it in an opt in channel instead of this one | 23:35 |
| *** hemna_ has quit IRC | 23:38 | |
| ianw | that's probably because i accidentally left a test alert running all night :) | 23:38 |
| *** heyongli has quit IRC | 23:39 | |
| *** heyongli has joined #openstack-infra | 23:39 | |
| corvus | well, it fairly accurately demonstrated why i don't think we should have an alerting system. we've talked about this a few times in the past. they require a lot of time tweaking settings to to avoid false alarms. especially for a team of volunteers with no service level agreements. i think monitoring is fine. even having a nagios page with some red/green boxes periodically could be useful. i | 23:43 |
| corvus | think active alerting is counter-productive. | 23:43 |
| *** diablo_rojo has joined #openstack-infra | 23:43 | |
| corvus | to be clear, i mean a nagios *web* page (noun, not verb) :) | 23:46 |
| ianw | corvus: i feel like if you've gone to the effort of writing grafana queries, setting some sane limits within the same dashboard file is going to be a win. i would, for example, like to know when afs partitions have filled up before all our reprepro runs start corrupting themselves | 23:46 |
| corvus | ianw: in my experience, disk space alerts are some of the highest maintenance alerts. | 23:48 |
| *** sthussey has quit IRC | 23:48 | |
| corvus | ianw: i think your desire is reasonable. but i don't feel like anyone on the infra team should feel compelled to see and respond to pages. so if you want to proceed, i'd appreciate it if you could make your system completely invisible to me, and i'd like us to agree as a group that participation is so voluntary as to not even be encouraged. | 23:49 |
| *** heyongli has quit IRC | 23:49 | |
| *** heyongli has joined #openstack-infra | 23:50 | |
| *** diablo_rojo has quit IRC | 23:51 | |
| corvus | (i really think that something more like a nagios dashboard where folks could see potential problems at a glance would fit the bill quite well and not be objectionable) | 23:51 |
| ianw | setting up the alerts parameters in grafana will actually flag the corresponding graph with red and a broken heart symbol | 23:54 |
| ianw | if the alert gets relayed here, well maybe we can discuss more widely when i have everything actually in place that it could | 23:55 |
| corvus | ianw: i don't understand your last sentence | 23:55 |
| auristor | ianw: with regards to afs partitions, in general, if its over 90% full, start thinking about growing the space or moving volumes | 23:56 |
| ianw | auristor: heh, yes, well i would like to know when that happens :) which is why i have spent a little time on monitoring our things a bit better | 23:56 |
| ianw | corvus: i mean with the new grafana, you can set up alert parameters for graphs. if an irc message gets triggered for it is another thing | 23:57 |
| auristor | sure but that doesn't require an alarm. its more like a weekly check list | 23:57 |
| auristor | I'm more concerned about the volumes that are approaching 2TB. | 23:57 |
| ianw | auristor: yeah, but the point for infra is that there is nobody who "owns" any particular thing who is sitting around with a checklist of "this is what i'll check today" | 23:58 |
| corvus | auristor: i think we've reduced the pypi mirror significantly. but if necessary, we can shard it (and likely most of the others) | 23:59 |
| corvus | ianw: an alert won't help that. it will just drive people away. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!