| hogepodge | clarkb: fungi: here's the proposed work list for the upcoming RefStack Server upgrade https://etherpad.openstack.org/p/refstack-update-feb-2018 | 00:00 |
|---|---|---|
| *** claudiub has quit IRC | 00:00 | |
| hogepodge | I'm not sure where the infra deployment scripts live. Once we update the puppet-refstack module we'll need to also update whatever repo manages the module deployment so we can add the new option | 00:01 |
| *** mguiney has joined #openstack-infra | 00:01 | |
| *** dingyichen has joined #openstack-infra | 00:02 | |
| *** caphrim007 has quit IRC | 00:06 | |
| ianw | corvus: is it possible only the administrator can create a ppa? i'm not seeing the option to on the openstack-ci-core page, but i do on others | 00:07 |
| ianw | only mordred seems to be an admin in openstack-ci-core, but i bet corvus gets permissions inherited from it's parent openstack-ci | 00:08 |
| *** felipemonteiro has joined #openstack-infra | 00:12 | |
| *** hemna_ has joined #openstack-infra | 00:13 | |
| *** felipemonteiro_ has joined #openstack-infra | 00:13 | |
| *** VW has joined #openstack-infra | 00:13 | |
| *** VW has quit IRC | 00:14 | |
| *** VW has joined #openstack-infra | 00:15 | |
| *** felipemonteiro has quit IRC | 00:17 | |
| *** edmondsw has quit IRC | 00:18 | |
| *** olaph1 has joined #openstack-infra | 00:19 | |
| *** olaph has quit IRC | 00:19 | |
| *** VW has quit IRC | 00:20 | |
| *** lbragstad has quit IRC | 00:22 | |
| *** olaph has joined #openstack-infra | 00:24 | |
| *** jamesmcarthur has joined #openstack-infra | 00:25 | |
| *** olaph1 has quit IRC | 00:25 | |
| *** olaph has quit IRC | 00:29 | |
| clarkb | hogepodge: openstack-infra/system-config is going to be the thing that consumes puppet-refstack | 00:29 |
| clarkb | hogepodge: you can push that change up now with a depends on | 00:29 |
| *** jamesmcarthur has quit IRC | 00:30 | |
| *** olaph has joined #openstack-infra | 00:30 | |
| mguiney | i'll get that up shortly, sorry | 00:31 |
| mguiney | things on fire, etc. | 00:31 |
| clarkb | no rush | 00:31 |
| clarkb | plan is for friday right? plenty of time :) | 00:31 |
| *** slaweq has joined #openstack-infra | 00:31 | |
| *** agopi_out has joined #openstack-infra | 00:32 | |
| *** wolverineav has quit IRC | 00:32 | |
| *** tosky has quit IRC | 00:33 | |
| *** wolverineav has joined #openstack-infra | 00:33 | |
| *** slaweq has quit IRC | 00:36 | |
| *** wolverineav has quit IRC | 00:37 | |
| *** links has joined #openstack-infra | 00:40 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add publish-openstack-sphinx-docs to blazar-dashboard jobs https://review.openstack.org/540771 | 00:44 |
| corvus | ianw: oh hrm, looking | 00:52 |
| corvus | ianw: try now | 00:53 |
| ianw | corvus: excellent, i have a create button now, thanks | 00:54 |
| ianw | presumably i just dput packages signed with my key from my profile, and it "just works" | 00:55 |
| corvus | ianw: cool, i'll promote all of us to administrators | 00:56 |
| *** esberglu has quit IRC | 01:00 | |
| mnaser | any infra-root around who would mind checking up on the reboot of `nb03.openstack.org` and `mirror01.ca-ymq-1.vexxhost.openstack.org` -- it should be a few seconds only | 01:02 |
| clarkb | mnaser: yes I can do it | 01:02 |
| mnaser | (and if you'd rather schedule it, i can do that too) | 01:02 |
| clarkb | now is fine | 01:02 |
| mnaser | awesome, thank you | 01:02 |
| mnaser | mirror starting now | 01:02 |
| clarkb | it might make some image builds unhappy but meh | 01:02 |
| clarkb | they can always be restarted | 01:03 |
| mnaser | nb03 started reboot | 01:03 |
| mnaser | mirror01.ca-ymq-1.vexxhost.openstack.org back up (but seems like port 80 is not responding, yet?) | 01:03 |
| clarkb | mnaser: ya it may be a little slow to spool up afs? | 01:04 |
| mnaser | clarkb: looks like its up now | 01:04 |
| clarkb | looks up now ya | 01:04 |
| mnaser | nb03.openstack.org done too but have no way of testing that | 01:04 |
| clarkb | ya it looks ok. | 01:05 |
| clarkb | pabelanger: ^ wasn't that a server you were going to clean up? | 01:05 |
| mnaser | awesome, thank you so much clarkb | 01:05 |
| clarkb | pabelanger: looks like it can't connect to the zk server due to firewall rules? if that is indeed something that needs cleaning up can we get that confirmed (I can help do the cleanign if it helps) | 01:06 |
| ianw | yeah, i strongly remember that one was turned off | 01:17 |
| openstackgerrit | megan guiney proposed openstack-infra/puppet-refstack master: Added option to set enable_anonymous_upload https://review.openstack.org/544145 | 01:18 |
| *** olaph1 has joined #openstack-infra | 01:19 | |
| *** caphrim007 has joined #openstack-infra | 01:20 | |
| *** VW has joined #openstack-infra | 01:22 | |
| *** olaph has quit IRC | 01:22 | |
| *** hongbin has joined #openstack-infra | 01:24 | |
| *** caphrim007 has quit IRC | 01:25 | |
| *** gyee has quit IRC | 01:25 | |
| *** jamesmcarthur has joined #openstack-infra | 01:26 | |
| *** felipemonteiro__ has joined #openstack-infra | 01:33 | |
| *** felipemonteiro_ has quit IRC | 01:33 | |
| *** hemna_ has quit IRC | 01:34 | |
| pabelanger | clarkb: mnaser: yah, it was powered off. Ready for delete | 01:38 |
| *** VW has quit IRC | 01:43 | |
| *** VW has joined #openstack-infra | 01:44 | |
| *** ijw has quit IRC | 01:45 | |
| *** edmondsw has joined #openstack-infra | 01:46 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:46 | |
| *** xarses has joined #openstack-infra | 01:48 | |
| *** VW has quit IRC | 01:49 | |
| *** felipemonteiro__ has quit IRC | 01:49 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add links to GrafYaml https://review.openstack.org/542395 | 01:50 |
| *** edmondsw has quit IRC | 01:51 | |
| *** caphrim007 has joined #openstack-infra | 01:53 | |
| *** esberglu has joined #openstack-infra | 01:57 | |
| *** esberglu has quit IRC | 01:57 | |
| *** felipemonteiro_ has quit IRC | 01:59 | |
| *** benj_ has quit IRC | 02:08 | |
| *** benj_ has joined #openstack-infra | 02:09 | |
| *** signed8bit_Zzz is now known as signed8b_ | 02:17 | |
| *** vivsoni has quit IRC | 02:23 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** ijw has joined #openstack-infra | 02:32 | |
| *** olaph has joined #openstack-infra | 02:33 | |
| *** slaweq has joined #openstack-infra | 02:33 | |
| *** olaph1 has quit IRC | 02:33 | |
| *** signed8b_ is now known as signed8bit_Zzz | 02:33 | |
| *** mriedem_afk has quit IRC | 02:34 | |
| *** lbragstad has joined #openstack-infra | 02:35 | |
| *** ijw has quit IRC | 02:37 | |
| *** slaweq has quit IRC | 02:37 | |
| *** notmyname has quit IRC | 02:38 | |
| *** iyamahat__ has quit IRC | 02:40 | |
| *** notmyname has joined #openstack-infra | 02:41 | |
| *** yamahata has quit IRC | 02:42 | |
| *** liusheng has quit IRC | 02:43 | |
| *** liusheng has joined #openstack-infra | 02:43 | |
| *** salv-orl_ has joined #openstack-infra | 02:45 | |
| *** ramishra has joined #openstack-infra | 02:45 | |
| *** myoung|bbl is now known as myoung | 02:46 | |
| *** salv-orlando has quit IRC | 02:48 | |
| *** d0ugal has quit IRC | 02:49 | |
| *** aviau has quit IRC | 02:55 | |
| *** aviau has joined #openstack-infra | 02:56 | |
| *** bhavik1 has joined #openstack-infra | 02:58 | |
| *** d0ugal has joined #openstack-infra | 02:59 | |
| *** dougwig has quit IRC | 03:01 | |
| *** bhavik1 has quit IRC | 03:01 | |
| *** rlandy has quit IRC | 03:15 | |
| *** ijw has joined #openstack-infra | 03:18 | |
| *** r-daneel has joined #openstack-infra | 03:23 | |
| *** harlowja has quit IRC | 03:30 | |
| *** VW has joined #openstack-infra | 03:32 | |
| *** ijw has quit IRC | 03:32 | |
| *** edmondsw has joined #openstack-infra | 03:34 | |
| *** edmondsw has quit IRC | 03:39 | |
| *** yamamoto has joined #openstack-infra | 03:43 | |
| *** links has quit IRC | 03:50 | |
| *** felipemonteiro has joined #openstack-infra | 03:54 | |
| *** d0ugal has quit IRC | 03:55 | |
| ianw | yay, ppa's built for arm64 (1.8 branch) and amd64 (latest 1.6 for hwe kernel) | 03:59 |
| *** sree has joined #openstack-infra | 03:59 | |
| ianw | i'm going to put the ppa on ze02, and do the hard reboot we discussed about the ivp6 stuff | 03:59 |
| *** d0ugal has joined #openstack-infra | 03:59 | |
| *** lbragstad has quit IRC | 04:02 | |
| *** links has joined #openstack-infra | 04:07 | |
| *** caphrim007_ has joined #openstack-infra | 04:08 | |
| *** dave-mccowan has quit IRC | 04:10 | |
| *** caphrim007 has quit IRC | 04:11 | |
| *** harlowja has joined #openstack-infra | 04:12 | |
| *** cshastri has joined #openstack-infra | 04:20 | |
| *** efried1 has joined #openstack-infra | 04:20 | |
| *** efried has quit IRC | 04:22 | |
| *** efried1 is now known as efried | 04:22 | |
| *** ykarel_ has joined #openstack-infra | 04:22 | |
| *** jamesmcarthur has joined #openstack-infra | 04:26 | |
| *** harlowja has quit IRC | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** slaweq has joined #openstack-infra | 04:34 | |
| *** lpetrut has joined #openstack-infra | 04:38 | |
| *** slaweq has quit IRC | 04:38 | |
| *** felipemonteiro has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Install HWE kernels for zuul-exectors https://review.openstack.org/544164 | 04:45 |
| *** hongbin has quit IRC | 04:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Install HWE kernels for zuul-exectors https://review.openstack.org/544164 | 04:57 |
| *** psachin` has joined #openstack-infra | 05:04 | |
| *** psachin has joined #openstack-infra | 05:04 | |
| *** VW has quit IRC | 05:06 | |
| *** VW has joined #openstack-infra | 05:07 | |
| ianw | #status log ze02.o.o was hard rebooted but didn't fix ipv6 issues. i detached and re-attached the network from the existing vm, and that seemed to help. DNS has been updated | 05:10 |
| openstackstatus | ianw: finished logging | 05:10 |
| *** lpetrut has quit IRC | 05:11 | |
| *** VW has quit IRC | 05:11 | |
| ianw | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=565&nodeid=node1_565&host_group_data= already looking better | 05:12 |
| ianw | not sure why it has query_ifName in there for nic though...? | 05:12 |
| *** harlowja has joined #openstack-infra | 05:14 | |
| *** lpetrut has joined #openstack-infra | 05:19 | |
| *** claudiub has joined #openstack-infra | 05:21 | |
| *** dhajare has joined #openstack-infra | 05:24 | |
| *** lpetrut has quit IRC | 05:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move to native nodepool boot jobs https://review.openstack.org/543329 | 05:35 |
| *** janki has joined #openstack-infra | 05:36 | |
| *** VW has joined #openstack-infra | 05:37 | |
| *** iyamahat has joined #openstack-infra | 05:37 | |
| *** lbragstad has joined #openstack-infra | 05:42 | |
| *** VW has quit IRC | 05:42 | |
| *** iyamahat has quit IRC | 05:45 | |
| *** psachin has quit IRC | 05:45 | |
| *** psachin` has quit IRC | 05:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Install HWE kernels for zuul-exectors https://review.openstack.org/544164 | 05:47 |
| *** pgadiya has joined #openstack-infra | 05:49 | |
| *** VW has joined #openstack-infra | 05:51 | |
| *** psachin has joined #openstack-infra | 05:51 | |
| *** olaph1 has joined #openstack-infra | 05:51 | |
| *** signed8bit_Zzz has quit IRC | 05:52 | |
| *** olaph has quit IRC | 05:52 | |
| *** kjackal has joined #openstack-infra | 05:55 | |
| *** VW has quit IRC | 05:56 | |
| *** rkukura has quit IRC | 05:56 | |
| *** slaweq has joined #openstack-infra | 05:59 | |
| *** slaweq has quit IRC | 06:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/544172 | 06:07 |
| *** coolsvap has joined #openstack-infra | 06:17 | |
| AJaeger | ianw, could you review these small changes, please? https://review.openstack.org/543192 and https://review.openstack.org/542102 and https://review.openstack.org/#/c/540771/ ? | 06:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/544172 | 06:18 |
| *** psachin has quit IRC | 06:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-tempest-dsvm-neutron-nova-next-full usage https://review.openstack.org/541477 | 06:20 |
| *** signed8bit has joined #openstack-infra | 06:22 | |
| *** masber has joined #openstack-infra | 06:22 | |
| *** rkukura has joined #openstack-infra | 06:23 | |
| *** VW has joined #openstack-infra | 06:23 | |
| *** harlowja has quit IRC | 06:24 | |
| *** VW has quit IRC | 06:28 | |
| *** psachin has joined #openstack-infra | 06:28 | |
| *** ykarel_ is now known as ykarel | 06:31 | |
| *** VW has joined #openstack-infra | 06:33 | |
| *** olaph has joined #openstack-infra | 06:36 | |
| *** VW has quit IRC | 06:37 | |
| *** olaph1 has quit IRC | 06:38 | |
| *** rosmaita has quit IRC | 06:51 | |
| *** pcrews has quit IRC | 06:51 | |
| *** pcrews has joined #openstack-infra | 06:52 | |
| *** VW has joined #openstack-infra | 06:56 | |
| *** dhajare has quit IRC | 06:58 | |
| *** pcichy has quit IRC | 06:59 | |
| *** VW has quit IRC | 07:01 | |
| openstackgerrit | Zhipeng Huang proposed openstack-infra/project-config master: Add release note jobs to Cyborg https://review.openstack.org/544189 | 07:02 |
| *** jchhatbar has joined #openstack-infra | 07:05 | |
| *** ianychoi_ is now known as ianychoi | 07:08 | |
| *** janki has quit IRC | 07:09 | |
| *** edmondsw has joined #openstack-infra | 07:11 | |
| *** jchhatbar is now known as janki | 07:14 | |
| *** edmondsw has quit IRC | 07:15 | |
| *** lbragstad has quit IRC | 07:17 | |
| *** alexchadin has joined #openstack-infra | 07:17 | |
| *** VW has joined #openstack-infra | 07:27 | |
| *** VW has quit IRC | 07:32 | |
| *** agopi_out has quit IRC | 07:36 | |
| *** armaan has joined #openstack-infra | 07:37 | |
| *** lpetrut has joined #openstack-infra | 07:42 | |
| *** kjackal has quit IRC | 07:44 | |
| *** ykarel_ has joined #openstack-infra | 07:47 | |
| *** slaweq has joined #openstack-infra | 07:48 | |
| *** andreas_s has joined #openstack-infra | 07:48 | |
| *** ykarel has quit IRC | 07:48 | |
| *** gema has joined #openstack-infra | 07:49 | |
| *** olaph has quit IRC | 07:50 | |
| *** olaph has joined #openstack-infra | 07:50 | |
| *** dhajare has joined #openstack-infra | 07:50 | |
| *** ykarel__ has joined #openstack-infra | 07:52 | |
| *** slaweq has quit IRC | 07:53 | |
| *** ykarel has joined #openstack-infra | 07:54 | |
| *** slaweq has joined #openstack-infra | 07:54 | |
| *** ykarel_ has quit IRC | 07:55 | |
| *** pcaruana has joined #openstack-infra | 07:56 | |
| *** ykarel__ has quit IRC | 07:57 | |
| *** eumel8 has joined #openstack-infra | 07:57 | |
| *** slaweq has quit IRC | 07:59 | |
| *** masber has quit IRC | 08:05 | |
| *** VW has joined #openstack-infra | 08:05 | |
| *** slaweq has joined #openstack-infra | 08:05 | |
| openstackgerrit | Zhipeng Huang proposed openstack-infra/project-config master: Add release note and doc jobs to Cyborg https://review.openstack.org/544189 | 08:08 |
| *** VW has quit IRC | 08:10 | |
| *** slaweq has quit IRC | 08:10 | |
| *** armaan has quit IRC | 08:13 | |
| *** gfidente has joined #openstack-infra | 08:14 | |
| *** gfidente has joined #openstack-infra | 08:14 | |
| *** zhipeng has joined #openstack-infra | 08:14 | |
| *** jpena|off is now known as jpena | 08:16 | |
| *** snapiri has joined #openstack-infra | 08:17 | |
| *** aviau has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:19 | |
| *** florianf has joined #openstack-infra | 08:20 | |
| *** amoralej|off is now known as amoralej | 08:20 | |
| *** yamamoto has quit IRC | 08:21 | |
| *** dingyichen has quit IRC | 08:21 | |
| *** yamamoto has joined #openstack-infra | 08:22 | |
| *** e0ne has joined #openstack-infra | 08:22 | |
| *** kjackal has joined #openstack-infra | 08:24 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:27 | |
| *** tesseract has joined #openstack-infra | 08:28 | |
| *** stakeda has quit IRC | 08:31 | |
| *** saop has joined #openstack-infra | 08:33 | |
| *** rcernin has quit IRC | 08:33 | |
| *** snapiri1 has joined #openstack-infra | 08:35 | |
| *** snapiri has quit IRC | 08:37 | |
| *** VW has joined #openstack-infra | 08:38 | |
| *** kjackal has quit IRC | 08:38 | |
| *** slaweq has joined #openstack-infra | 08:40 | |
| *** pcichy has joined #openstack-infra | 08:40 | |
| *** snapiri1 has quit IRC | 08:42 | |
| *** VW has quit IRC | 08:43 | |
| *** slaweq has quit IRC | 08:44 | |
| *** ttx has joined #openstack-infra | 08:50 | |
| *** oidgar has joined #openstack-infra | 08:51 | |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** links has quit IRC | 08:53 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** iyamahat has joined #openstack-infra | 09:05 | |
| *** ociuhandu has joined #openstack-infra | 09:05 | |
| *** ameeda has joined #openstack-infra | 09:06 | |
| *** links has joined #openstack-infra | 09:06 | |
| ameeda | hello, please check this bug https://bugs.launchpad.net/cinder/+bug/1740432 and let me what should I do | 09:06 |
| openstack | Launchpad bug 1740432 in Cinder " Reintroduce the StorPool driver" [Undecided,New] | 09:06 |
| *** Douhet has quit IRC | 09:06 | |
| *** yamahata has joined #openstack-infra | 09:07 | |
| *** hashar has joined #openstack-infra | 09:08 | |
| *** Douhet has joined #openstack-infra | 09:08 | |
| *** olaph has quit IRC | 09:08 | |
| *** ociuhandu has quit IRC | 09:10 | |
| *** olaph has joined #openstack-infra | 09:10 | |
| AJaeger | ameeda: you have to figure out whether there'S a documentation need - if there's none, mark it as wontfix | 09:10 |
| *** zhipeng has quit IRC | 09:11 | |
| AJaeger | ameeda: see also https://docs.openstack.org/doc-contrib-guide/doc-impact.html | 09:11 |
| *** VW has joined #openstack-infra | 09:17 | |
| *** slaweq has joined #openstack-infra | 09:18 | |
| *** ykarel is now known as ykarel|lunch | 09:19 | |
| *** kopecmartin has joined #openstack-infra | 09:19 | |
| gibi | gmann_: thanks for the ML post, I put my 2 cents in a reply. | 09:21 |
| gibi | wrong window | 09:21 |
| *** janki has quit IRC | 09:22 | |
| *** VW has quit IRC | 09:22 | |
| *** slaweq has quit IRC | 09:23 | |
| *** iyamahat has quit IRC | 09:23 | |
| *** slaweq has joined #openstack-infra | 09:24 | |
| *** yamahata has quit IRC | 09:24 | |
| *** olaph1 has joined #openstack-infra | 09:24 | |
| *** olaph has quit IRC | 09:25 | |
| *** jamesmcarthur has joined #openstack-infra | 09:25 | |
| *** kopecmartin has quit IRC | 09:26 | |
| *** slaweq_ has joined #openstack-infra | 09:27 | |
| *** pgadiya has quit IRC | 09:28 | |
| *** slaweq has quit IRC | 09:28 | |
| *** pgadiya has joined #openstack-infra | 09:29 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** kopecmartin has joined #openstack-infra | 09:31 | |
| *** yamahata has joined #openstack-infra | 09:31 | |
| *** pgadiya has quit IRC | 09:31 | |
| *** pgadiya has joined #openstack-infra | 09:33 | |
| *** pgadiya has quit IRC | 09:33 | |
| *** snapiri has joined #openstack-infra | 09:35 | |
| *** kjackal has joined #openstack-infra | 09:38 | |
| *** pgadiya has joined #openstack-infra | 09:38 | |
| *** pgadiya has quit IRC | 09:38 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** pblaho has joined #openstack-infra | 09:46 | |
| *** VW has joined #openstack-infra | 09:47 | |
| AJaeger | config-core, could you review https://review.openstack.org/544189 and https://review.openstack.org/540771 , please? Those add new docs jobs that are important for Queens... | 09:50 |
| *** VW has quit IRC | 09:52 | |
| *** alexchadin has quit IRC | 09:56 | |
| *** alexchadin has joined #openstack-infra | 09:57 | |
| ameeda | AJaeger: thank you | 09:58 |
| *** ykarel|lunch is now known as ykarel | 10:00 | |
| *** kopecmartin has quit IRC | 10:01 | |
| *** shardy has joined #openstack-infra | 10:02 | |
| *** VW has joined #openstack-infra | 10:02 | |
| *** kopecmartin has joined #openstack-infra | 10:03 | |
| *** pcaruana has quit IRC | 10:05 | |
| *** VW has quit IRC | 10:06 | |
| *** kopecmartin has quit IRC | 10:07 | |
| *** kopecmartin has joined #openstack-infra | 10:09 | |
| *** dsariel has joined #openstack-infra | 10:09 | |
| *** sree has quit IRC | 10:09 | |
| *** dsariel_ has joined #openstack-infra | 10:09 | |
| *** sree has joined #openstack-infra | 10:10 | |
| *** sree has quit IRC | 10:15 | |
| *** slaweq_ is now known as slaweq | 10:16 | |
| *** makowals has joined #openstack-infra | 10:19 | |
| *** pcaruana has joined #openstack-infra | 10:20 | |
| *** pcichy has quit IRC | 10:20 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix self fulfilling empty node requests https://review.openstack.org/544236 | 10:22 |
| *** VW has joined #openstack-infra | 10:25 | |
| *** ykarel_ has joined #openstack-infra | 10:26 | |
| *** zoli is now known as zoli|lunch | 10:27 | |
| *** zoli|lunch is now known as zoli | 10:28 | |
| *** owalsh is now known as owalsh_afk | 10:28 | |
| *** ykarel has quit IRC | 10:29 | |
| *** VW has quit IRC | 10:30 | |
| *** lucas-pto is now known as lucasagomes | 10:30 | |
| *** kjackal has quit IRC | 10:32 | |
| *** rfolco|off has quit IRC | 10:33 | |
| *** yamahata has quit IRC | 10:34 | |
| *** armaan has joined #openstack-infra | 10:34 | |
| *** ykarel_ is now known as ykarel | 10:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Setup translation jobs for Queens (latecomer) https://review.openstack.org/543192 | 10:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-openstack-sphinx-docs to blazar-dashboard jobs https://review.openstack.org/540771 | 10:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Add the vault charm as a new project https://review.openstack.org/542102 | 10:37 |
| *** ldnunes has joined #openstack-infra | 10:41 | |
| *** tosky has joined #openstack-infra | 10:44 | |
| *** edmondsw has joined #openstack-infra | 10:47 | |
| *** sambetts|afk is now known as sambetts | 10:48 | |
| *** slaweq_ has joined #openstack-infra | 10:50 | |
| *** ykarel_ has joined #openstack-infra | 10:52 | |
| *** priteau has joined #openstack-infra | 10:52 | |
| *** edmondsw has quit IRC | 10:52 | |
| *** ykarel has quit IRC | 10:54 | |
| *** slaweq_ has quit IRC | 10:55 | |
| saop | tristanC, hello | 11:00 |
| saop | tristanC, Whats the version of bubblewrap we should use in Xenial ubuntu, kind of messed our setup now? | 11:01 |
| *** lathiat has quit IRC | 11:01 | |
| *** ankkumar has joined #openstack-infra | 11:01 | |
| *** VW has joined #openstack-infra | 11:02 | |
| *** rfolco has joined #openstack-infra | 11:04 | |
| *** rfolco is now known as rfolco|rover | 11:05 | |
| *** VW has quit IRC | 11:07 | |
| *** kjackal has joined #openstack-infra | 11:07 | |
| *** olaph has joined #openstack-infra | 11:11 | |
| *** olaph1 has quit IRC | 11:13 | |
| *** jamesmcarthur has joined #openstack-infra | 11:19 | |
| openstackgerrit | Merged openstack-infra/zuul master: Extend stackdump to display the daemonize status https://review.openstack.org/542158 | 11:20 |
| *** jamesmcarthur has quit IRC | 11:23 | |
| *** olaph has quit IRC | 11:27 | |
| *** olaph1 has joined #openstack-infra | 11:27 | |
| *** RuiChen has quit IRC | 11:28 | |
| *** pbourke has quit IRC | 11:29 | |
| *** pcaruana has quit IRC | 11:30 | |
| *** pbourke has joined #openstack-infra | 11:30 | |
| *** zhipeng has joined #openstack-infra | 11:31 | |
| *** saop_ has joined #openstack-infra | 11:31 | |
| *** saop has quit IRC | 11:34 | |
| *** ankkumar has quit IRC | 11:35 | |
| *** sree has joined #openstack-infra | 11:36 | |
| *** alexchadin has quit IRC | 11:38 | |
| *** VW has joined #openstack-infra | 11:38 | |
| *** sree has quit IRC | 11:40 | |
| *** pcichy has joined #openstack-infra | 11:40 | |
| *** pcaruana has joined #openstack-infra | 11:42 | |
| *** saop_ has quit IRC | 11:43 | |
| *** VW has quit IRC | 11:44 | |
| *** sshnaidm|ruck has quit IRC | 11:46 | |
| *** alexchadin has joined #openstack-infra | 11:47 | |
| *** sree has joined #openstack-infra | 11:51 | |
| *** dsariel has quit IRC | 11:52 | |
| *** dsariel_ has quit IRC | 11:53 | |
| *** sree has quit IRC | 11:56 | |
| *** dave-mccowan has joined #openstack-infra | 11:56 | |
| *** sree has joined #openstack-infra | 11:58 | |
| *** sree_ has joined #openstack-infra | 12:00 | |
| *** sree_ is now known as Guest18968 | 12:01 | |
| *** lathiat has joined #openstack-infra | 12:02 | |
| *** sree has quit IRC | 12:03 | |
| *** armaan has quit IRC | 12:05 | |
| *** armaan has joined #openstack-infra | 12:05 | |
| *** HeOS has quit IRC | 12:07 | |
| *** dsariel has joined #openstack-infra | 12:07 | |
| *** cshastri_ has joined #openstack-infra | 12:08 | |
| *** dsariel_ has joined #openstack-infra | 12:08 | |
| *** zhipeng has quit IRC | 12:09 | |
| *** cshastri has quit IRC | 12:11 | |
| *** slaweq_ has joined #openstack-infra | 12:11 | |
| *** VW has joined #openstack-infra | 12:12 | |
| *** slaweq_ has quit IRC | 12:16 | |
| *** VW has quit IRC | 12:17 | |
| *** wolverineav has joined #openstack-infra | 12:17 | |
| *** zhipeng has joined #openstack-infra | 12:18 | |
| *** slaweq_ has joined #openstack-infra | 12:19 | |
| *** sshnaidm|ruck has joined #openstack-infra | 12:20 | |
| *** HeOS has joined #openstack-infra | 12:21 | |
| *** patriciadomin_ is now known as patriciadomin | 12:21 | |
| *** dtantsur|afk is now known as dtantsur | 12:21 | |
| *** slaweq_ has quit IRC | 12:23 | |
| *** yamamoto has quit IRC | 12:23 | |
| *** yamamoto has joined #openstack-infra | 12:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add release note and doc jobs to Cyborg https://review.openstack.org/544189 | 12:24 |
| *** armaan has quit IRC | 12:27 | |
| *** auristor has quit IRC | 12:31 | |
| *** katkapilatova has joined #openstack-infra | 12:31 | |
| *** zhipeng has quit IRC | 12:35 | |
| *** edmondsw has joined #openstack-infra | 12:36 | |
| *** auristor has joined #openstack-infra | 12:37 | |
| *** olaph has joined #openstack-infra | 12:37 | |
| *** olaph1 has quit IRC | 12:39 | |
| *** dsariel has quit IRC | 12:39 | |
| *** dsariel_ has quit IRC | 12:40 | |
| *** edmondsw has quit IRC | 12:40 | |
| *** makowals has quit IRC | 12:43 | |
| openstackgerrit | Merged openstack/ptgbot master: Separate data loading from init https://review.openstack.org/543978 | 12:46 |
| *** VW has joined #openstack-infra | 12:47 | |
| *** tosky has quit IRC | 12:47 | |
| openstackgerrit | Merged openstack/ptgbot master: Use deepcopy for dict copy operations https://review.openstack.org/543979 | 12:50 |
| *** edmondsw has joined #openstack-infra | 12:50 | |
| *** kjackal has quit IRC | 12:50 | |
| *** jpena is now known as jpena|lunch | 12:51 | |
| *** VW has quit IRC | 12:51 | |
| openstackgerrit | Merged openstack/ptgbot master: Replace ~wipe by more selective ~reload https://review.openstack.org/543980 | 12:52 |
| openstackgerrit | Merged openstack/ptgbot master: Add 'unbook' admin command https://review.openstack.org/543981 | 12:52 |
| openstackgerrit | Merged openstack/ptgbot master: Allow everyone to issue track moderator commands https://review.openstack.org/543982 | 12:52 |
| *** dsariel has joined #openstack-infra | 12:52 | |
| *** dsariel_ has joined #openstack-infra | 12:53 | |
| *** r-daneel has quit IRC | 12:54 | |
| *** zhipeng has joined #openstack-infra | 12:55 | |
| *** olaph1 has joined #openstack-infra | 12:56 | |
| *** olaph has quit IRC | 12:56 | |
| *** olaph1 is now known as olaph | 13:00 | |
| *** armaan has joined #openstack-infra | 13:00 | |
| *** ianychoi has quit IRC | 13:00 | |
| *** weshay_PTO is now known as weshay | 13:01 | |
| *** dhajare has quit IRC | 13:02 | |
| *** ykarel__ has joined #openstack-infra | 13:03 | |
| *** ianychoi has joined #openstack-infra | 13:04 | |
| *** r-daneel has joined #openstack-infra | 13:04 | |
| *** zhipeng has quit IRC | 13:04 | |
| *** jcoufal has joined #openstack-infra | 13:05 | |
| *** ykarel_ has quit IRC | 13:05 | |
| *** tosky has joined #openstack-infra | 13:06 | |
| *** ianychoi has quit IRC | 13:09 | |
| *** Guest18968 has quit IRC | 13:09 | |
| *** VW has joined #openstack-infra | 13:10 | |
| *** e0ne has quit IRC | 13:10 | |
| *** sree has joined #openstack-infra | 13:10 | |
| *** kjackal has joined #openstack-infra | 13:11 | |
| *** e0ne has joined #openstack-infra | 13:13 | |
| *** rosmaita has joined #openstack-infra | 13:14 | |
| *** sree has quit IRC | 13:15 | |
| *** yamamoto has quit IRC | 13:17 | |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| *** sshnaidm has joined #openstack-infra | 13:19 | |
| *** oidgar has quit IRC | 13:21 | |
| *** sshnaidm|ruck has quit IRC | 13:22 | |
| *** yamamoto has quit IRC | 13:22 | |
| *** eumel8 has quit IRC | 13:23 | |
| *** trown|outtypewww is now known as trown | 13:23 | |
| *** lbragstad has joined #openstack-infra | 13:23 | |
| *** tosky has quit IRC | 13:23 | |
| *** VW has quit IRC | 13:25 | |
| *** VW has joined #openstack-infra | 13:25 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 13:26 |
| *** priteau has quit IRC | 13:27 | |
| *** amoralej is now known as amoralej|lunch | 13:30 | |
| *** VW has quit IRC | 13:30 | |
| *** rlandy has joined #openstack-infra | 13:31 | |
| *** kgiusti has joined #openstack-infra | 13:32 | |
| AJaeger | config-core, could you review this stack to remove jenkins/scripts, please? https://review.openstack.org/#/c/541868/ | 13:39 |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Custom Template Type https://review.openstack.org/544305 | 13:39 |
| *** tosky has joined #openstack-infra | 13:40 | |
| *** jamesmcarthur has joined #openstack-infra | 13:41 | |
| *** jamesmcarthur has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** kiennt26 has joined #openstack-infra | 13:49 | |
| *** arxcruz|sick is now known as arxcruz | 13:50 | |
| *** jpena|lunch is now known as jpena | 13:50 | |
| *** owalsh_afk is now known as owalsh | 13:52 | |
| *** cshastri_ has quit IRC | 13:53 | |
| *** jonher- is now known as jonher | 13:55 | |
| *** armaan has quit IRC | 13:57 | |
| *** mriedem has joined #openstack-infra | 13:57 | |
| *** armaan has joined #openstack-infra | 13:58 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/puppet-ptgbot master: Deploy missing bootstrap-3.3.7.min.js https://review.openstack.org/544345 | 14:00 |
| *** jamesmcarthur has joined #openstack-infra | 14:00 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Remove leftover console.log https://review.openstack.org/544346 | 14:02 |
| fungi | any infra-puppet-core reviewers up for seconding a trivial but important (ptg-blocker) change to puppet-ptgbot? https://review.openstack.org/544345 | 14:04 |
| *** armaan has quit IRC | 14:05 | |
| *** andreas_s has quit IRC | 14:06 | |
| *** andreas_s has joined #openstack-infra | 14:06 | |
| *** andreas_s has quit IRC | 14:06 | |
| *** alexchadin has quit IRC | 14:08 | |
| *** alexchadin has joined #openstack-infra | 14:08 | |
| *** alexchadin has quit IRC | 14:08 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Remove tagging for Cyborg https://review.openstack.org/544349 | 14:09 |
| *** alexchadin has joined #openstack-infra | 14:09 | |
| *** alexchadin has quit IRC | 14:09 | |
| *** zhipeng has joined #openstack-infra | 14:09 | |
| *** alexchadin has joined #openstack-infra | 14:10 | |
| *** alexchadin has quit IRC | 14:10 | |
| *** alexchadin has joined #openstack-infra | 14:11 | |
| *** r-daneel has quit IRC | 14:11 | |
| *** alexchadin has quit IRC | 14:11 | |
| *** jamesmca_ has joined #openstack-infra | 14:11 | |
| AJaeger | smcginnis: do you have some automation for these? ^ | 14:11 |
| AJaeger | smcginnis: +2 | 14:11 |
| *** alexchadin has joined #openstack-infra | 14:11 | |
| *** alexchadin has quit IRC | 14:12 | |
| smcginnis | AJaeger: No, not at the moment. There is the start of an automated script to check for this case, but for now it's just remembering to do it for new projects. | 14:12 |
| *** alexchadin has joined #openstack-infra | 14:12 | |
| *** alexchadin has quit IRC | 14:12 | |
| *** zhipeng has quit IRC | 14:14 | |
| *** cshastri has joined #openstack-infra | 14:14 | |
| *** armaan has joined #openstack-infra | 14:15 | |
| *** oidgar has joined #openstack-infra | 14:16 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-neutron-nova-next-full job https://review.openstack.org/541479 | 14:17 |
| *** zoli is now known as zoli|afk | 14:19 | |
| *** zoli|afk is now known as zoli | 14:19 | |
| *** eharney has joined #openstack-infra | 14:19 | |
| *** Goneri has joined #openstack-infra | 14:20 | |
| AJaeger | infra-root, please see the zuul error in 541479 - how could this happen? | 14:20 |
| *** lucasagomes is now known as lucas-hungry | 14:21 | |
| mriedem | AJaeger: was just going to ask about that :) | 14:22 |
| *** alexchadin has joined #openstack-infra | 14:22 | |
| smcginnis | I still blame mriedem. | 14:23 |
| *** ykarel__ is now known as ykarel | 14:26 | |
| *** alexchadin has quit IRC | 14:26 | |
| odyssey4me | heh, I'm seeing it in a change I just pushed up too | 14:26 |
| fungi | AJaeger: first thing i'll check is whether we imported a project with an existing zuul config | 14:26 |
| AJaeger | fungi: no time to dig myself into this for the next couple of hours, thanks for looking | 14:27 |
| fungi | that | 14:27 |
| fungi | that's a dead-end anyway, we've had paunch for a very long time | 14:27 |
| pabelanger | morning | 14:28 |
| AJaeger | did we branch from an old version paunch that did not include the Zuul update? | 14:28 |
| *** r-daneel has joined #openstack-infra | 14:29 | |
| fungi | https://review.openstack.org/537875 removed that job from master on jan 27 | 14:29 |
| AJaeger | smcginnis: when did we create queens for paunch? | 14:29 |
| fungi | looks like it | 14:30 |
| fungi | branched from a merge commit dated december 4, a point in time prior to that removal | 14:30 |
| fungi | so that explains it | 14:30 |
| fungi | we created a branch from a now-unsupported point in history | 14:30 |
| fungi | at least as far as zuul's overall configuration consistency is concerned | 14:31 |
| pabelanger | yah, it looks like it | 14:31 |
| pabelanger | how was the branch created? | 14:31 |
| pabelanger | via gerrit ui? | 14:31 |
| fungi | looking | 14:31 |
| *** zhipeng has joined #openstack-infra | 14:31 | |
| *** amoralej|lunch is now known as amoralej | 14:31 | |
| pabelanger | we should be able to propose a patch to paunch to remove the OVB job | 14:32 |
| fungi | created by https://review.openstack.org/543601 which merged just under an hour ago | 14:32 |
| pabelanger | ah | 14:33 |
| AJaeger | can we merge *anything* right now or is Zuul refusing _everything_? | 14:33 |
| pabelanger | AJaeger: we need to patch paunch first | 14:33 |
| AJaeger | that's a loop hole ;( | 14:33 |
| smcginnis | Those branches were created by the requirements team since we did not get a final branching request from the teams. | 14:33 |
| *** links has quit IRC | 14:33 | |
| fultonj | hi corvus: about a week ago we talked about github triggering CI jobs on zuul. I saw "Report on GitHub from openstack-infra" merged. Is there any other update? | 14:33 |
| fungi | looks like we branched paunch from its most recent release tag, which was 2.2.0 from january 30, but tagged a commit from december prior to the config change | 14:34 |
| smcginnis | We should be able to backport that zuul change to the stable branch quickly, I would hope. | 14:34 |
| pabelanger | it is possible we need to update the branch logic, seems we can merge broken code behind zuul | 14:34 |
| fultonj | i'm basically looking for a live example to copy | 14:34 |
| smcginnis | fungi: We branched from the wrong point? I'm not sure I follow. | 14:34 |
| fungi | AJaeger: pabelanger: smcginnis: so yes, i think the multi-day nature of release requests caused us to lose a race on job renaming/removal | 14:35 |
| pabelanger | smcginnis: fungi: AJaeger: should be it: https://review.openstack.org/544358/ | 14:35 |
| pabelanger | doh | 14:35 |
| pabelanger | now a new | 14:35 |
| smcginnis | Hah, was going to say, doesn't look much better there. | 14:35 |
| pabelanger | os-apply-config | 14:35 |
| fungi | smcginnis: the release request was made on january 25 for the then-current master branch tip commit id, the config change merged to master on the 27th and the release was approved on the 30th | 14:36 |
| smcginnis | fungi: OK, I see what you're saying now. | 14:37 |
| fungi | https://review.openstack.org/537752 was the request for that 2.2.0 tag for paunch | 14:37 |
| smcginnis | Looks like two additional commits merged in that repo, plus a merge commit. | 14:37 |
| pabelanger | https://review.openstack.org/544359/ too | 14:38 |
| fungi | i wonder if we should consider raising a red flag if a branch moves beyond a release request before it's approved | 14:38 |
| pabelanger | fungi: yah, that might be a good idea now | 14:38 |
| smcginnis | fungi: I would guess it's not entirely uncommon. | 14:38 |
| fungi | right, that's why i didn't say reject | 14:39 |
| *** coolsvap has quit IRC | 14:39 | |
| fungi | though i'm not sure how to spot that | 14:39 |
| pabelanger | so, I think we'll need to force merge: https://review.openstack.org/544358/ then we should be able to recheck https://review.openstack.org/544359/ | 14:39 |
| fungi | at least not from the perspective of our ci workflow | 14:39 |
| pabelanger | if people would like to look^ | 14:39 |
| fungi | non-voting jobs wouldn't likely get run again because the results (in this case) were only a few days old | 14:40 |
| smcginnis | pabelanger: Failure on there. | 14:40 |
| AJaeger | do you want to #status notice once it's fixed? | 14:40 |
| pabelanger | smcginnis: on which | 14:40 |
| smcginnis | 544359 | 14:40 |
| smcginnis | This change depends on a change that failed to merge. | 14:40 |
| pabelanger | smcginnis: yah, expected. We need to force merge 554358 | 14:40 |
| pabelanger | but want to confirm | 14:41 |
| *** olaph1 has joined #openstack-infra | 14:43 | |
| *** jcoufal_ has joined #openstack-infra | 14:44 | |
| *** olaph has quit IRC | 14:46 | |
| *** jcoufal__ has joined #openstack-infra | 14:46 | |
| *** ianychoi has joined #openstack-infra | 14:47 | |
| *** jcoufal has quit IRC | 14:47 | |
| fungi | did we introduce two configuration syntax errors? | 14:48 |
| fungi | i can't seem to load https://review.openstack.org/554358 | 14:49 |
| fungi | The page you requested was not found, or you do not have permission to view this page. | 14:49 |
| *** jcoufal_ has quit IRC | 14:49 | |
| fungi | oh, you mistyped it | 14:49 |
| pabelanger | fungi: paunch and os-client-config so far have syntax errors | 14:49 |
| pabelanger | see links above | 14:49 |
| *** lbragsta_ has joined #openstack-infra | 14:50 | |
| pabelanger | sorry, os-apply-config* | 14:50 |
| fungi | got it, so probably the same issue introduced both places in parallel as they were part of the same release request | 14:50 |
| pabelanger | yah, believe so | 14:50 |
| fungi | and yeah, we'll need to bypass ci for at least one of them | 14:50 |
| fungi | i don't think it matters which | 14:50 |
| pabelanger | dealers choice | 14:50 |
| pabelanger | paunch might win, because other is using depends-on syntax | 14:51 |
| fungi | i'll do paunch | 14:51 |
| *** lbragstad has quit IRC | 14:51 | |
| pabelanger | ack | 14:52 |
| corvus | fultonj: i think we have most of the technicial issues worked out, but i haven't written up a policy for the tc to review yet | 14:52 |
| *** HeOS has quit IRC | 14:53 | |
| fultonj | corvus: if the technical issues are worked out, then is there an example i could follow to mimic the feature? | 14:53 |
| fultonj | or do you mean, it should work but there's no current example? | 14:53 |
| *** yamamoto has quit IRC | 14:53 | |
| fungi | #status log bypassed zuul and directly submitted https://review.openstack.org/#/c/544358 | 14:54 |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| openstackstatus | fungi: finished logging | 14:54 |
| *** yamahata has joined #openstack-infra | 14:54 | |
| fungi | rechecking the other now | 14:54 |
| pabelanger | k | 14:54 |
| pabelanger | looks to be enqueued now | 14:55 |
| *** jcoufal has joined #openstack-infra | 14:56 | |
| fungi | if it enqueues, i'm inclined to say we also bypass zuul to merge 544359 so we don't wait 3 hours for tripleo jobs to complete before getting zuul's configuration back to a coherent state | 14:56 |
| pabelanger | fungi: ++ | 14:56 |
| pabelanger | I see jobs running now | 14:56 |
| fungi | okay, i'll go ahead and merge it | 14:56 |
| *** yamamoto has quit IRC | 14:57 | |
| *** jcoufal__ has quit IRC | 14:57 | |
| *** yamamoto has joined #openstack-infra | 14:57 | |
| *** iyamahat has joined #openstack-infra | 14:58 | |
| fungi | #status log bypassed zuul and directly submitted https://review.openstack.org/#/c/544359 | 14:58 |
| openstackstatus | fungi: finished logging | 14:58 |
| fungi | hopefully that has us back on track again | 14:58 |
| pabelanger | I'll recheck 541479 from above | 14:58 |
| fungi | thanks | 14:59 |
| *** alexchadin has joined #openstack-infra | 15:00 | |
| pabelanger | and enqueued | 15:00 |
| corvus | fultonj: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n4 | 15:00 |
| pabelanger | AJaeger: mriedem: looks to be fixed now that fungi force merged our fixes | 15:00 |
| pabelanger | smcginnis: ^ | 15:01 |
| *** ansmith has joined #openstack-infra | 15:01 | |
| AJaeger | thanks, fungi | 15:01 |
| AJaeger | fungi, send a #status notice? Anybody proposeing a change in the last 60 mins hit this... | 15:02 |
| fungi | it was an interesting puzzle to solve | 15:02 |
| AJaeger | ;) | 15:02 |
| *** agopi_out has joined #openstack-infra | 15:02 | |
| *** coolsvap has joined #openstack-infra | 15:02 | |
| fungi | #status notice Due to a race in stable/queens branch creation and some job removals, Zuul has reported syntax errors for the past hour; if you saw a syntax error reported for "Job tripleo-ci-centos-7-ovb-containers-oooq not defined" you can safely recheck now | 15:03 |
| openstackstatus | fungi: sending notice | 15:03 |
| *** alexchadin has quit IRC | 15:03 | |
| *** sshnaidm is now known as sshnaidm|ruck | 15:04 | |
| -openstackstatus- NOTICE: Due to a race in stable/queens branch creation and some job removals, Zuul has reported syntax errors for the past hour; if you saw a syntax error reported for "Job tripleo-ci-centos-7-ovb-containers-oooq not defined" you can safely recheck now | 15:04 | |
| *** jamesmca_ has quit IRC | 15:05 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Custom Template Type https://review.openstack.org/544305 | 15:05 |
| *** bobh has joined #openstack-infra | 15:05 | |
| *** jamesmca_ has joined #openstack-infra | 15:06 | |
| openstackstatus | fungi: finished sending notice | 15:06 |
| fultonj | corvus: thanks, i'll lokk at that and docs and try to grok it. | 15:07 |
| *** iyamahat has quit IRC | 15:07 | |
| dmsimard | just skimmed backlog, I guess it's the first time we create a batch of branches under zuul v3 -- should be interesting to follow how Zuul behaves, we're basically adding N repo branches | 15:08 |
| dmsimard | gj on fixing things quickly | 15:08 |
| *** armaan has quit IRC | 15:09 | |
| *** jamesmca_ has quit IRC | 15:10 | |
| fungi | dmsimard: gj on the part of zuul devs for making the syntax error reporting so thorough. it didn't take much investigation to spot the cause and solution | 15:12 |
| fungi | because we had the right information up front | 15:12 |
| fungi | #thanks zuul v3 for having such detailed configuration syntax error reporting | 15:13 |
| openstackstatus | fungi: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 15:13 |
| dmsimard | +1 | 15:13 |
| *** olaph has joined #openstack-infra | 15:13 | |
| *** r-daneel has quit IRC | 15:14 | |
| *** olaph1 has quit IRC | 15:16 | |
| *** zhipeng has quit IRC | 15:16 | |
| AJaeger | could I get a review on https://review.openstack.org/#/c/541868/ (and rest of stack if you have time) to further remove project-config/jenkins/scripts , please? | 15:18 |
| dmsimard | I'm going to keep an eye on the Zuul memory to see if branching has an impact.. http://cacti.openstack.org/cacti/graph.php?local_graph_id=64792&rra_id=0&view_type=tree | 15:18 |
| *** lbragsta_ is now known as lbragstad | 15:19 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Custom Template Type https://review.openstack.org/544305 | 15:19 |
| *** cshastri has quit IRC | 15:20 | |
| *** r-daneel has joined #openstack-infra | 15:22 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** panda is now known as panda|mtg | 15:24 | |
| *** alexchadin has joined #openstack-infra | 15:30 | |
| *** lbragstad has quit IRC | 15:31 | |
| *** lucas-hungry is now known as lucasagomes | 15:31 | |
| *** hongbin has joined #openstack-infra | 15:31 | |
| *** VW has joined #openstack-infra | 15:32 | |
| *** felipemonteiro has joined #openstack-infra | 15:33 | |
| *** dklyle has joined #openstack-infra | 15:33 | |
| *** david-lyle has quit IRC | 15:33 | |
| *** slaweq has quit IRC | 15:33 | |
| *** lbragstad has joined #openstack-infra | 15:33 | |
| *** slaweq has joined #openstack-infra | 15:34 | |
| *** gibi has quit IRC | 15:34 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:36 | |
| *** slaweq has quit IRC | 15:38 | |
| *** jamesmca_ has joined #openstack-infra | 15:40 | |
| *** links has joined #openstack-infra | 15:40 | |
| *** ramishra has quit IRC | 15:40 | |
| *** felipemonteiro has quit IRC | 15:40 | |
| *** felipemonteiro_ has quit IRC | 15:43 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:43 | |
| fungi | disappearing to run some errands. back later | 15:49 |
| *** esberglu has joined #openstack-infra | 15:52 | |
| *** katkapilatova has quit IRC | 15:52 | |
| *** katkapilatova has joined #openstack-infra | 15:53 | |
| *** katkapilatova has quit IRC | 15:57 | |
| *** ykarel is now known as ykarel|away | 15:57 | |
| *** caphrim007_ has quit IRC | 15:58 | |
| *** olaph1 has joined #openstack-infra | 16:00 | |
| *** olaph has quit IRC | 16:01 | |
| *** alexchadin has quit IRC | 16:02 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Don't skip branching for projects with no releases https://review.openstack.org/544513 | 16:02 |
| *** ykarel|away has quit IRC | 16:05 | |
| *** ijw has joined #openstack-infra | 16:07 | |
| *** olaph1 is now known as olaph | 16:11 | |
| *** pcaruana has quit IRC | 16:13 | |
| *** yamamoto has quit IRC | 16:13 | |
| smcginnis | Could I get some eyes on https://review.openstack.org/544513 please. It seems we optimized our way out of being able to branch things like devstack. | 16:18 |
| *** kiennt26 has quit IRC | 16:20 | |
| *** electrofelix has quit IRC | 16:20 | |
| *** panda|mtg is now known as panda | 16:22 | |
| *** armaan has joined #openstack-infra | 16:23 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add stable/queens branch for Packstack gerritbot notifications https://review.openstack.org/544524 | 16:25 |
| *** yamamoto has joined #openstack-infra | 16:27 | |
| AJaeger | dmsimard, pabelanger, could I get a review on https://review.openstack.org/#/c/541868/ (and rest of stack if you have time) to further remove project-config/jenkins/scripts , please? | 16:27 |
| dmsimard | AJaeger: will try | 16:28 |
| *** psachin has quit IRC | 16:29 | |
| *** yamamoto has quit IRC | 16:31 | |
| *** rlandy is now known as rlandy|brb | 16:32 | |
| *** mriedem is now known as mriedem_afk | 16:34 | |
| *** slaweq has joined #openstack-infra | 16:34 | |
| *** iyamahat has joined #openstack-infra | 16:35 | |
| *** tosky has quit IRC | 16:36 | |
| *** slaweq_ has joined #openstack-infra | 16:36 | |
| *** imacdonn has quit IRC | 16:37 | |
| *** imacdonn has joined #openstack-infra | 16:37 | |
| *** slaweq has quit IRC | 16:39 | |
| *** alexchadin has joined #openstack-infra | 16:40 | |
| *** slaweq_ has quit IRC | 16:40 | |
| *** jcoufal_ has joined #openstack-infra | 16:41 | |
| *** aviau has quit IRC | 16:43 | |
| *** krtaylor has joined #openstack-infra | 16:43 | |
| *** aviau has joined #openstack-infra | 16:43 | |
| *** e0ne has quit IRC | 16:43 | |
| *** alexchadin has quit IRC | 16:44 | |
| *** jcoufal has quit IRC | 16:44 | |
| *** hemna_ has joined #openstack-infra | 16:47 | |
| *** rlandy|brb is now known as rlandy | 16:51 | |
| AJaeger | thanks | 16:52 |
| openstackgerrit | Merged openstack-infra/zuul master: Do not call merger:cat when all config items are excluded https://review.openstack.org/535509 | 16:54 |
| smcginnis | Any folks around that can take a look at https://review.openstack.org/544513 ? | 16:54 |
| smcginnis | Need to wait to branch devstack and grenade for stable/queens until that lands. | 16:55 |
| *** slaweq has joined #openstack-infra | 16:56 | |
| corvus | all of the zuulv3-projects changes have been uploaded | 16:56 |
| AJaeger | \o/ | 16:57 |
| AJaeger | how many merged - and how many not yet? How many failed due to bitrot? ;( | 16:57 |
| corvus | 93 are still open, out of i think 675 total? | 16:58 |
| corvus | that's been a pretty constant number, so i expect it to shrink over the next couple days | 16:58 |
| corvus | then we'll be left with maybe 50 or so that are stuck? | 16:58 |
| *** camunoz has joined #openstack-infra | 16:58 | |
| *** slaweq has quit IRC | 17:00 | |
| *** slaweq_ has joined #openstack-infra | 17:00 | |
| *** oidgar has quit IRC | 17:01 | |
| *** rosmaita has quit IRC | 17:01 | |
| AJaeger | not too bad - I fear I rechecked dozens ;( | 17:02 |
| *** suhdood has joined #openstack-infra | 17:02 | |
| *** efried is now known as efried_rollin | 17:02 | |
| *** lpetrut has joined #openstack-infra | 17:03 | |
| pabelanger | dmsimard: mind approving ARA change to remove non-voting jobs in gate now: https://review.openstack.org/543487/ | 17:05 |
| *** slaweq_ has quit IRC | 17:05 | |
| mnaser | corvus: maybe we might have to rerun it in case some projects branched before merging that change? so stable/queens might have project in it | 17:05 |
| openstackgerrit | Petr Kovar proposed openstack-infra/infra-manual master: Update content on project docs setup per project docs migration https://review.openstack.org/544545 | 17:06 |
| *** ameeda has quit IRC | 17:06 | |
| AJaeger | mnaser: yes, that is most likely the case | 17:07 |
| AJaeger | mnaser, corvus and projects might had changes in flight with project names in it. I tried to -1 everything one day with it, but still thinks could have slipped in. | 17:08 |
| *** beagles is now known as beagles_brb | 17:08 | |
| *** dtantsur is now known as dtantsur|afk | 17:09 | |
| *** VW has quit IRC | 17:09 | |
| *** VW has joined #openstack-infra | 17:10 | |
| *** ijw has quit IRC | 17:10 | |
| *** gyee has joined #openstack-infra | 17:11 | |
| corvus | yep, it'll be a slow iterative process. but we're not deprecating it, so we're not in a hurry. | 17:11 |
| *** yamamoto has joined #openstack-infra | 17:12 | |
| *** rosmaita has joined #openstack-infra | 17:13 | |
| *** VW has quit IRC | 17:15 | |
| *** signed8bit is now known as signed8bit_Zzz | 17:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move wheel-build scripts out of jenkins/scripts https://review.openstack.org/541868 | 17:15 |
| *** dsariel_ has quit IRC | 17:16 | |
| *** VW has joined #openstack-infra | 17:16 | |
| *** dsariel has quit IRC | 17:16 | |
| *** alexchadin has joined #openstack-infra | 17:16 | |
| *** pcaruana has joined #openstack-infra | 17:17 | |
| pabelanger | infra-root: 3 executors have lost console streaming now, ze01, ze04, ze09. Any objections to stopping / starting zuul-executor there? | 17:17 |
| *** yamamoto has quit IRC | 17:17 | |
| *** dougwig has joined #openstack-infra | 17:19 | |
| clarkb | pabelanger: might be opportunity to update the kernel and afs on them? | 17:20 |
| *** ramishra has joined #openstack-infra | 17:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544560 | 17:20 |
| pabelanger | clarkb: let me check if we got the new PPA setup | 17:21 |
| clarkb | pabelanger: ze02 has the afs ppa in apt sources list | 17:21 |
| dmsimard | oh, restarting the executors would also allow us to get new executor metrics https://review.openstack.org/#/c/541452/ | 17:21 |
| clarkb | not sure if that got as far as getting added to puppet yet though | 17:21 |
| *** links has quit IRC | 17:21 | |
| pabelanger | let me look | 17:21 |
| clarkb | just thinking that if that kernel update fixes the OOMing we are better off properly fixing it if we canthan just continuing to restart it | 17:21 |
| pabelanger | sure | 17:22 |
| *** slaweq has joined #openstack-infra | 17:22 | |
| *** tosky has joined #openstack-infra | 17:23 | |
| *** iyamahat has quit IRC | 17:23 | |
| *** jpich has quit IRC | 17:24 | |
| clarkb | smcginnis: approved | 17:26 |
| pabelanger | clarkb: https://review.openstack.org/544164/ found it | 17:26 |
| smcginnis | clarkb: Thanks! | 17:27 |
| pabelanger | I guess we'll need to keep an eye out for kernel security fixes, then publish them into our PPA | 17:27 |
| pabelanger | and hopefully 18.04 will be better | 17:28 |
| clarkb | pabelanger: not sure I understand, the ppa is only for the afs driver. The kernel is from ubuntu | 17:28 |
| clarkb | pabelanger: we will need to stay on top of updates to afs | 17:28 |
| pabelanger | Oh, right | 17:28 |
| pabelanger | yah, ignore me | 17:28 |
| pabelanger | I need more coffee | 17:28 |
| clarkb | I've +2'd the chagne but not approved it. I think we will need to do a manual dist-upgrade after that ppa gets in place and before rebooting with the new kernel | 17:29 |
| pabelanger | yah, I think so too | 17:29 |
| clarkb | infra-root want to look ever 544164 to see if you think it is ready for use on the executors? | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP add host/group vars https://review.openstack.org/544562 | 17:30 |
| *** caphrim007 has joined #openstack-infra | 17:30 | |
| *** yamahata has quit IRC | 17:31 | |
| pabelanger | clarkb: do we know the reason ubuntu doesn't ship HWE support for AFS package? | 17:31 |
| *** ramishra has quit IRC | 17:31 | |
| dmsimard | pabelanger: it's not in the upstream kernel so I guess it boils down to a lack of resource to rebase things in it | 17:32 |
| clarkb | pabelanger: since the kernel doesn't have a stable ABI and the afs driver is out of tree as the kernel moves ahead of afs compatibility breaks. Ubuntu isn't syncing their afs package with necessary updates to handle the newer kernels | 17:32 |
| corvus | clarkb, pabelanger: +3 | 17:32 |
| clarkb | basically any out of tree drivers will need to be updated. I think ubuntu does this for the nvidia drivers but unsure of any others | 17:32 |
| pabelanger | dmsimard: clarkb: ack, thanks | 17:32 |
| dmsimard | clarkb: they maintain zfs on linux as a dkms blob I think, perhaps they could do the same thing for afs. | 17:33 |
| *** beagles_brb is now known as beagles | 17:33 | |
| clarkb | dmsimard: oh ya I am zfs on linux with hwe and its working fine too | 17:33 |
| corvus | this is dkms | 17:33 |
| clarkb | so ya they definitely do it for some drivers | 17:33 |
| corvus | last time i looked it wasn't an api break | 17:33 |
| corvus | it was a package build break | 17:34 |
| clarkb | corvus: I think ianw mentioend there was an abi break this tiem around | 17:34 |
| clarkb | and he had to update the afs source too | 17:34 |
| corvus | i will be happy when we can turn hwe enablement back off | 17:34 |
| openstackgerrit | Merged openstack-infra/project-config master: Don't skip branching for projects with no releases https://review.openstack.org/544513 | 17:36 |
| clarkb | in my limited experience the kernel does break all sorts of things because they "can". I once had to rename an enum for pci states in an out of tree atheros ethernet driver in order to get working arch install. That was also the beginning of the end for using arch for me | 17:36 |
| clarkb | but not surprised that afs would need updating to handle newer kernels | 17:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544569 | 17:36 |
| clarkb | corvus: I have a hunch that the zuul executors might be the first of our servers to run 18.04 | 17:40 |
| *** jamesmcarthur has quit IRC | 17:40 | |
| clarkb | (and I think 18.04 doesn't have any major changes like systemd so hopefully that is a smooth transition) | 17:40 |
| *** signed8bit_Zzz is now known as signed8bit | 17:41 | |
| dmsimard | 18.04 drops unity and mir though (just kidding, doesn't impact us :p) | 17:41 |
| *** panda is now known as panda|off | 17:43 | |
| *** ijw has joined #openstack-infra | 17:44 | |
| dmsimard | SamYaple: hey, do you currently use the openstack-infra docker.io reverse proxy ? | 17:44 |
| *** jcoufal has joined #openstack-infra | 17:45 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:45 | |
| *** jamesmca_ has quit IRC | 17:46 | |
| clarkb | dmsimard: I think SamYaple was working on getting it updated so that it would work with newer docker /me checks if that change merged | 17:46 |
| *** iyamahat has joined #openstack-infra | 17:46 | |
| *** jamesmcarthur has joined #openstack-infra | 17:46 | |
| *** jamesmcarthur has quit IRC | 17:46 | |
| clarkb | https://review.openstack.org/#/c/542370/ did merge https://review.openstack.org/#/c/542474/ is the change to consume them looks like | 17:47 |
| *** jamesmcarthur has joined #openstack-infra | 17:47 | |
| *** jcoufal_ has quit IRC | 17:47 | |
| *** felipemonteiro_ has quit IRC | 17:48 | |
| *** lpetrut has quit IRC | 17:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 17:49 |
| SamYaple | dmsimard: yea but it was borked. | 17:49 |
| SamYaple | dmsimard: fixed now. >=17.10 docker-ce should use https://review.openstack.org/#/c/542474/2/playbooks/vars.yaml | 17:49 |
| *** alexchadin has quit IRC | 17:50 | |
| *** jcoufal_ has joined #openstack-infra | 17:50 | |
| *** jcoufal has quit IRC | 17:51 | |
| pabelanger | dmsimard: yah, really depends on which docker client you are using now. sadface | 17:51 |
| *** kjackal has quit IRC | 17:51 | |
| *** kjackal has joined #openstack-infra | 17:51 | |
| tosky | notmyname: hi! I tried to use your change on swift with https://review.openstack.org/#/c/544383/, and we moved a bit; not the failure is different | 17:51 |
| clarkb | pabelanger: well the new thing should work with old clients right? | 17:51 |
| tosky | notmyname: as you can see http://logs.openstack.org/19/304019/25/experimental/sahara-tests-scenario-py3/b186232/controller/logs/screen-s-proxy.txt.gz#_Feb_14_16_45_24_210470 | 17:51 |
| pabelanger | clarkb: no | 17:51 |
| clarkb | oh I thought it would, thats unfortunate but also very dockery of them | 17:51 |
| SamYaple | yea its pretty messedup | 17:51 |
| pabelanger | clarkb: yes, dockery | 17:51 |
| SamYaple | still better than docker registry though | 17:52 |
| tosky | notmyname: 'ImportError: No module named ceilometermiddleware.swift' - should it be added to the extra as well? Another extra? Extension of the same extra? Should we discuss on swift channel? | 17:52 |
| clarkb | how does docker registry work with both? | 17:52 |
| SamYaple | im still not surethey have a mechanism to delete oldimages inthe docker registry | 17:52 |
| *** jcoufal has joined #openstack-infra | 17:53 | |
| SamYaple | clarkb: the docker registry mirror actually caches and serves the data as if it wasnt a cache | 17:53 |
| SamYaple | and by cache imean "stores forever" | 17:53 |
| SamYaple | we didnt spend alot of time on the differences inbehaviour between apache and the dockerregistry though | 17:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Install HWE kernels for zuul-exectors https://review.openstack.org/544164 | 17:54 |
| clarkb | but from an http persective that doesn't matter much right? a typical user points at a registry without a trailing path and it works both with the old and new versions of the client? | 17:54 |
| *** jcoufal_ has quit IRC | 17:55 | |
| pabelanger | it is possible I over looked something, but when I use my local docker client on :8082, it will not work. Pulls from docker.io directly, after a redirect | 17:55 |
| SamYaple | well they point at the registry service which has logic for really old and new clients | 17:55 |
| *** hashar is now known as hasharAway | 17:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP add host/group vars https://review.openstack.org/544562 | 17:56 |
| *** lpetrut has joined #openstack-infra | 17:56 | |
| *** sambetts is now known as sambetts|afk | 17:56 | |
| clarkb | SamYaple: your change there to consume the mirror again appears to be working through? we've at least solved the issue going forward? | 17:58 |
| dmsimard | SamYaple: pretend we set up that docker daemon mirror configuration and we pull from somewhere else than docker.io, is it going to break things ? | 17:59 |
| clarkb | dmsimard: you would haev to point it at the other backend and possibly update the cdn/storage as appropriate | 18:00 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| clarkb | but in this case we are explicitly mirroring docker hub so not sure why we would do that | 18:01 |
| clarkb | s/mirroring/proxying/ | 18:01 |
| notmyname | tosky: yeah. in the swift channel is better, probably (alternative answer: I have no idea, but maybe someone else in -swift does) | 18:02 |
| SamYaple | clarkb: yea its goingthrough for sure. and pabelanger testedthat it was actually caching | 18:02 |
| dmsimard | clarkb: in RDO we pull some stuff from docker.io (ex: centos base image) and some other stuff from a private registry -- I was wondering if we had to bother with something to make both work | 18:02 |
| SamYaple | clarkb: i think dmsimard might be refering to quay.io or a private registry | 18:02 |
| dmsimard | I'm not familiar enough, I'll have to find out | 18:02 |
| dmsimard | SamYaple: yup | 18:02 |
| dmsimard | brb | 18:02 |
| SamYaple | dmsimard: the answer is no, but possible | 18:02 |
| * fungi is glad we no longer use jenkins... CVE-2018-6356 would have exposed the ssh private key to all our slaves | 18:03 | |
| SamYaple | fungi: oh boy... i use jenkins.... | 18:03 |
| clarkb | dmsimard: SamYaple you'd have to setup proxies for those on other ports in apache | 18:03 |
| clarkb | but should work in much the same way | 18:03 |
| fungi | SamYaple: hopefully you follow their security notes! | 18:03 |
| *** snapiri has quit IRC | 18:03 | |
| *** derekh has quit IRC | 18:03 | |
| SamYaple | fungi: haha its not a serious installation for me. im not too worried | 18:03 |
| SamYaple | the jenkins at my work though.... | 18:04 |
| SamYaple | glad i dont manage that | 18:04 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add tripleo-ci group to all other TripleO projects https://review.openstack.org/544580 | 18:04 |
| EmilienM | weshay, AJaeger ^ suite & fin | 18:04 |
| fungi | "any file within the Jenkins home directory accessible to the Jenkins master process could be downloaded." | 18:04 |
| SamYaple | nice | 18:04 |
| fungi | yeah, not great | 18:04 |
| fungi | https://jenkins.io/security/advisory/2018-02-14/ | 18:05 |
| openstackgerrit | Merged openstack-infra/nodepool master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544560 | 18:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 18:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:06 |
| *** yamahata has joined #openstack-infra | 18:07 | |
| *** signed8b_ has joined #openstack-infra | 18:08 | |
| *** kopecmartin has quit IRC | 18:08 | |
| *** gfidente has quit IRC | 18:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add stable/queens branch for Packstack gerritbot notifications https://review.openstack.org/544524 | 18:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tagging for Cyborg https://review.openstack.org/544349 | 18:10 |
| *** signed8bit has quit IRC | 18:11 | |
| clarkb | infra-root can we get a review on https://review.openstack.org/#/c/544345/1 to fix ptgbot website? Slightly time sensitive as we try to get things ready for ptg | 18:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 18:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:13 |
| fungi | infra-puppet-core reviewers: see clarkb's plea above | 18:14 |
| pabelanger | +3 | 18:14 |
| clarkb | pabelanger: left comment on fedora 27 switch change above. YOu missed an important piece :) | 18:15 |
| *** jpena is now known as jpena|off | 18:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 18:17 |
| pabelanger | clarkb: looking | 18:17 |
| clarkb | pabelanger: thank you for ptgbot review | 18:18 |
| pabelanger | clarkb: Oh, right. Thanks | 18:18 |
| *** kgiusti has left #openstack-infra | 18:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 18:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 18:19 |
| *** suhdood_ has joined #openstack-infra | 18:19 | |
| *** suhdood has quit IRC | 18:22 | |
| *** trown is now known as trown|lunch | 18:22 | |
| *** suhdood_ has quit IRC | 18:23 | |
| clarkb | another change that would be good to get eyes on is https://review.openstack.org/#/c/541950/ fixes infra jobs to handle tags. corvus you may be intrested in that as it may break some assumptions about the afs publishing marker files | 18:25 |
| *** psachin has joined #openstack-infra | 18:25 | |
| *** yamamoto has joined #openstack-infra | 18:27 | |
| corvus | clarkb: i can't think of anything it could break, can you? | 18:29 |
| clarkb | corvus: I am not aware of anything just wasn't sure if something was reading and parsing those files and expecting the branch to exist | 18:29 |
| corvus | clarkb: ah, afaik they're still just for human debugger consumption | 18:29 |
| clarkb | perfect | 18:29 |
| *** felipemonteiro__ has quit IRC | 18:29 | |
| corvus | er, i should say the content is such. the existence of the files of course is used by the module. | 18:30 |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Deploy missing bootstrap-3.3.7.min.js https://review.openstack.org/544345 | 18:30 |
| fungi | ttx: ^ merged, will take effect soonish | 18:31 |
| *** alexchadin has joined #openstack-infra | 18:32 | |
| *** yamamoto has quit IRC | 18:32 | |
| *** signed8b_ has quit IRC | 18:32 | |
| *** mriedem_afk is now known as mriedem | 18:32 | |
| *** signed8bit has joined #openstack-infra | 18:32 | |
| *** openstackgerrit has quit IRC | 18:33 | |
| *** kjackal has quit IRC | 18:33 | |
| *** alexchadin has quit IRC | 18:36 | |
| *** VW has quit IRC | 18:36 | |
| *** VW has joined #openstack-infra | 18:37 | |
| *** shardy has quit IRC | 18:38 | |
| *** zoli is now known as zoli|gone | 18:41 | |
| *** VW has quit IRC | 18:41 | |
| *** zoli|gone is now known as zoli | 18:42 | |
| *** VW has joined #openstack-infra | 18:42 | |
| *** jcoufal_ has joined #openstack-infra | 18:45 | |
| AJaeger | pabelanger: windmill jobs are broken again - aren't they? See https://review.openstack.org/#/c/540217/1 | 18:45 |
| AJaeger | fungi, a simple infra-manual review as time permits, please - https://review.openstack.org/#/c/544545 | 18:46 |
| fungi | i could use a break, thanks! | 18:46 |
| *** jcoufal has quit IRC | 18:48 | |
| pabelanger | AJaeger: yes, working on it | 18:48 |
| AJaeger | fungi: nothing critical ;) | 18:49 |
| *** florianf has quit IRC | 18:49 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/543127/ as next step for jenkins/script removal, please? | 18:50 |
| *** jpena|off is now known as jpena | 18:50 | |
| fungi | AJaeger: perhaps not critical, but was a nice diversion! | 18:51 |
| *** e0ne has joined #openstack-infra | 18:52 | |
| AJaeger | ;) | 18:52 |
| *** yamamoto has joined #openstack-infra | 18:52 | |
| *** yamamoto has quit IRC | 18:52 | |
| pabelanger | corvus: ianw: it doesn't look like nodepool dsvm jobs is working as epxect, I don't see any images building | 18:53 |
| pabelanger | corvus: sorry, that should have been clarkb | 18:53 |
| pabelanger | clarkb: ianw: http://logs.openstack.org/74/544574/3/check/nodepool-functional-py35/7e09d37/zuul-info/inventory.yaml | 18:54 |
| *** tesseract has quit IRC | 18:54 | |
| pabelanger | this is because it looks like devstack_localrc are not hashed together. | 18:54 |
| pabelanger | which sounds right | 18:54 |
| clarkb | pabelanger: there was a bug unsure if patch merged | 18:54 |
| clarkb | pabelanger: ianw disabled all image building by default then didnt renable it on the default job | 18:55 |
| clarkb | I'm making tea can dig up fix in a sec | 18:55 |
| *** jcoufal has joined #openstack-infra | 18:55 | |
| *** caphrim007_ has joined #openstack-infra | 18:56 | |
| AJaeger | pabelanger: https://review.openstack.org/#/c/543350 is end of ianw's stack | 18:56 |
| *** lucasagomes is now known as lucas-afk | 18:56 | |
| pabelanger | clarkb: okay, I might be looking in wrong place, but don't see setting for libs_from_git | 18:56 |
| AJaeger | the rest is merged | 18:56 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/nodepool/tree/.zuul.yaml#n44 | 18:56 |
| AJaeger | and https://review.openstack.org/#/c/543330/ removes legacy jobs in openstack-zuul-jobs, an easy review | 18:56 |
| AJaeger | config-core ^ | 18:56 |
| *** openstackgerrit has joined #openstack-infra | 18:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-ci group to all other TripleO projects https://review.openstack.org/544580 | 18:57 |
| pabelanger | AJaeger: I think this is different, I don't see devstack_localrc variables being setup | 18:57 |
| *** jcoufal_ has quit IRC | 18:57 | |
| *** armaan has quit IRC | 18:57 | |
| AJaeger | pabelanger: I'm sure ;) Just wanted to point out ianw's last work which should have fixed things | 18:57 |
| *** eharney has quit IRC | 18:57 | |
| clarkb | pabelanger: https://review.openstack.org/#/c/544125/ thats the fix I think | 18:58 |
| *** bobh has quit IRC | 18:58 | |
| *** armaan has joined #openstack-infra | 18:58 | |
| clarkb | I've approved it now that it has a second +2 | 18:58 |
| *** caphrim007 has quit IRC | 18:58 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Handle tags in infra docs publishing https://review.openstack.org/541950 | 18:59 |
| tosky | clarkb: so, about that extras thing, swift and devstack | 18:59 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update content on project docs setup per project docs migration https://review.openstack.org/544545 | 18:59 |
| *** jpena is now known as jpena|off | 18:59 | |
| tosky | clarkb: if something is defined in extras, and it's then installed with pip, wouldn't that prevent testing dependency between patches? Like for clients | 19:00 |
| pabelanger | clarkb: Ah, right. I didn't check src job | 19:01 |
| pabelanger | clarkb: thanks! | 19:01 |
| *** claudiub has quit IRC | 19:02 | |
| pabelanger | Okay, running out for a few errands | 19:03 |
| *** sshnaidm|ruck is now known as sshnaidm|off | 19:03 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Unpause Xenial build for non-src functional test https://review.openstack.org/544125 | 19:05 |
| *** jcoufal_ has joined #openstack-infra | 19:07 | |
| *** alexchadin has joined #openstack-infra | 19:07 | |
| *** jcoufal has quit IRC | 19:09 | |
| *** lpetrut has quit IRC | 19:10 | |
| *** jcoufal has joined #openstack-infra | 19:13 | |
| *** jcoufal_ has quit IRC | 19:15 | |
| AJaeger | tbarron: thanks for explanation on the manila change | 19:16 |
| tbarron | AJaeger: np, and sorry for the cryptic communication. | 19:16 |
| AJaeger | tbarron: no worries | 19:17 |
| *** jcoufal_ has joined #openstack-infra | 19:17 | |
| *** jcoufal has quit IRC | 19:19 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Improve file renaming in stage-output https://review.openstack.org/544606 | 19:20 |
| *** harlowja has joined #openstack-infra | 19:21 | |
| *** olaph1 has joined #openstack-infra | 19:25 | |
| *** olaph has quit IRC | 19:25 | |
| AJaeger | tosky: with https://review.openstack.org/#/c/510579 , do you have changes to remove the jobs from project-config and openstack-zuul-jobs? You don#t want to run both old and new ones in parallel. | 19:26 |
| *** rwsu has quit IRC | 19:28 | |
| *** rwsu has joined #openstack-infra | 19:29 | |
| tosky | AJaeger: yes, I do, but that review just merged and I was going to do the work tomorrow | 19:31 |
| *** dklyle has quit IRC | 19:33 | |
| AJaeger | tosky: fine with me... | 19:36 |
| *** jcoufal has joined #openstack-infra | 19:37 | |
| *** ralonsoh has quit IRC | 19:38 | |
| *** bobh has joined #openstack-infra | 19:38 | |
| *** jcoufal_ has quit IRC | 19:39 | |
| clarkb | infra-root I am taking the lock on the passwords file (will give you all a minute to warn me if you are editing it for some reason too) | 19:39 |
| fungi | thanks clarkb | 19:39 |
| *** alexchadin has quit IRC | 19:39 | |
| *** coolsvap has quit IRC | 19:41 | |
| *** trown|lunch is now known as trown|outtypewww | 19:42 | |
| *** yolanda has quit IRC | 19:44 | |
| *** jcoufal_ has joined #openstack-infra | 19:44 | |
| *** jcoufal has quit IRC | 19:44 | |
| *** yolanda has joined #openstack-infra | 19:45 | |
| fungi | heading back out for more errands, will return in a while | 19:46 |
| *** efried_rollin is now known as efried | 19:50 | |
| clarkb | ok lock is released | 19:51 |
| *** yamamoto has joined #openstack-infra | 19:53 | |
| *** caphrim007_ has quit IRC | 19:58 | |
| *** caphrim007 has joined #openstack-infra | 19:59 | |
| *** jcoufal has joined #openstack-infra | 19:59 | |
| *** jcoufal_ has quit IRC | 19:59 | |
| *** eharney has joined #openstack-infra | 19:59 | |
| *** jcoufal_ has joined #openstack-infra | 20:01 | |
| *** rfolco|rover is now known as rfolco|off | 20:01 | |
| *** felipemonteiro has joined #openstack-infra | 20:02 | |
| *** yamamoto has quit IRC | 20:02 | |
| *** jcoufal has quit IRC | 20:04 | |
| *** hasharAway is now known as hashar | 20:05 | |
| *** amoralej is now known as amoralej|off | 20:07 | |
| ianw | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64155&rra_id=0&view_type=tree&graph_start=1518552372&graph_end=1518638772 <- yay switching the interface on ze02 worked pretty clearly :) | 20:07 |
| clarkb | ianw: switching the interface? | 20:08 |
| ianw | i disconnected and reconnected the network via rax on it, trying to avoid a full rebuild | 20:09 |
| clarkb | ah | 20:10 |
| clarkb | ianw: did you see corvus approved the afs ppa chagne and kernel upgrade? | 20:10 |
| clarkb | as soon as I catch up on a few things I can help with reboots. I think pabelanger is interested as well | 20:10 |
| pabelanger | yah, in fact changes live on executors now | 20:11 |
| pabelanger | so, we need to dist-upgrade | 20:11 |
| pabelanger | stop zuul-executor | 20:11 |
| pabelanger | reboot | 20:11 |
| ianw | clarkb: ok cool. i should have got the versioning right in the ppa so that it overwrites | 20:12 |
| *** david-lyle has joined #openstack-infra | 20:12 | |
| clarkb | pabelanger: you might want to stop zuul-executor first | 20:13 |
| ianw | for arm64 it's 1.8.x branch, because that's where the changes are committed (https://launchpad.net/~openstack-ci-core/+archive/ubuntu/openafs-arm64) | 20:13 |
| pabelanger | clarkb: yah | 20:13 |
| ianw | that took a little more shoehorning :) | 20:13 |
| pabelanger | I'll start with ze01.o.o first | 20:13 |
| ianw | brb ... playing taxi for a bit | 20:14 |
| pabelanger | okay, ze01 stopping | 20:14 |
| pabelanger | will take a few moments | 20:14 |
| *** ldnunes has quit IRC | 20:16 | |
| *** alexchadin has joined #openstack-infra | 20:21 | |
| pabelanger | running dist-upgrade now | 20:21 |
| clarkb | I'm going to grab some lunch but then can help with ^ | 20:21 |
| *** alexchadin has quit IRC | 20:25 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:27 | |
| *** signed8bit is now known as signed8bit_Zzz | 20:28 | |
| *** felipemonteiro has quit IRC | 20:30 | |
| *** signed8bit_Zzz is now known as signed8bit | 20:33 | |
| pabelanger | okay, rebooting ze01 | 20:34 |
| tosky | AJaeger: I think that I can't still remove the legacy jobs until I finish backporting the zuulv3 jobs to the older branches | 20:34 |
| pabelanger | Linux ze01 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | 20:36 |
| pabelanger | looks like right kernel | 20:36 |
| pabelanger | and afs is working | 20:36 |
| pabelanger | clarkb: ianw: ^ | 20:36 |
| pabelanger | I'll hold off on other executors for the moment | 20:37 |
| AJaeger | tosky: do it quickly ;) | 20:37 |
| AJaeger | tosky: you don't want both sets of jobs running on master | 20:38 |
| tosky | AJaeger: trying, but of course... weird errors | 20:38 |
| AJaeger | the usual fun ;( | 20:39 |
| tosky | fungi: if I still see the error "Job tripleo-ci-centos-7-ovb-containers-oooq not defined", but the error in my review (on python-saharaclient) refers to a different repository (paunch), should I be concerned? | 20:39 |
| AJaeger | tosky: where? | 20:39 |
| tosky | fungi, AJaeger namely the error here: https://review.openstack.org/#/c/542847 | 20:39 |
| *** tpsilva has quit IRC | 20:39 | |
| AJaeger | infra-root, this just happened 8 minutes ago ^ | 20:40 |
| tosky | of course never easy :) | 20:40 |
| pabelanger | Hmm | 20:40 |
| tosky | AJaeger: about the backports, should I go back to ocata, or I can just ignore ocata at this point? | 20:40 |
| clarkb | I deleted the stable/queens branch of paunch and some other tripleo projects | 20:40 |
| clarkb | because they were branched too early and non functional | 20:40 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Update UI-Router https://review.openstack.org/384554 | 20:41 |
| clarkb | could be related? | 20:41 |
| pabelanger | did they recreate? | 20:41 |
| pabelanger | http://git.openstack.org/cgit/openstack/paunch/log/?h=stable/queens | 20:41 |
| clarkb | pabelanger: no | 20:41 |
| clarkb | because they need to branch next week or something | 20:41 |
| tosky | ok, but how is my job returning an error related to paunch? Is it due to the global namespace? | 20:41 |
| pabelanger | clarkb: I mean, I still see branch on git.o.o | 20:41 |
| clarkb | hrm | 20:41 |
| corvus | gerrit has a stable/queens for paunch | 20:42 |
| tosky | if it's the case, all review for queens should hit that error | 20:42 |
| corvus | https://review.openstack.org/#/admin/projects/openstack/paunch,branches | 20:42 |
| clarkb | uhg ya it recreated | 20:42 |
| clarkb | so something must be up with the release teams management | 20:42 |
| corvus | tosky: all changes that touch zuul config (ie, zuul.yaml) | 20:42 |
| clarkb | and I think tosky is broken due to how tripleo was broken when those branches were created | 20:42 |
| pabelanger | yah | 20:42 |
| pabelanger | our fixes from this morning are gone now | 20:43 |
| AJaeger | tosky: grab a coffee and wait ;) | 20:43 |
| clarkb | pabelanger: they were abandoned because tripleo specifically did not want to ahve a queens branch yet | 20:43 |
| clarkb | and then the branches were deleted | 20:43 |
| tosky | a bit late for coffee, I'm simply going to wash the dishes :) | 20:43 |
| pabelanger | clarkb: yah, so need to see why it was recreated | 20:43 |
| pabelanger | let me see if I can find some logs | 20:44 |
| AJaeger | tosky: grab a beer then or a wine :) | 20:44 |
| AJaeger | tosky: you need to backport it to all your active branches that you want tested | 20:45 |
| *** jcoufal has joined #openstack-infra | 20:46 | |
| AJaeger | clarkb: https://review.openstack.org/541645 just merged in releases - I guess it recreated the branch | 20:48 |
| *** jcoufal_ has quit IRC | 20:49 | |
| AJaeger | wait, https://review.openstack.org/544610 merged earlier | 20:49 |
| AJaeger | might have been https://review.openstack.org/544619 that merged and broke it | 20:50 |
| *** Goneri has quit IRC | 20:50 | |
| AJaeger | and afterwards https://review.openstack.org/544610 merges to remove the branches, so when 544619 merged, it recreated them. | 20:51 |
| AJaeger | Could that be the case? | 20:51 |
| clarkb | AJaeger: you'd probably need smcginnis to look over it | 20:52 |
| smcginnis | Just posted in -release, but I thought patches were automatically rebased on master before merging. | 20:54 |
| smcginnis | Shouldn't that prevent older patches that included the branch info from having that info included? | 20:54 |
| persia | Autorebase restores things that are present in the tree at the ref-to-be-merged and not present in the tree at HEAD of master, as I understand it. | 20:55 |
| smcginnis | So only additive? | 20:57 |
| AJaeger | smcginnis: I'm not following - but can't followup further here, hope somebody else can look further. | 21:01 |
| corvus | see discussion in #openstack-release for details, but we found what went wrong and why the branches were re-created | 21:03 |
| corvus | clarkb is going to delete them again shortly, and we expect them to stay deleted | 21:03 |
| AJaeger | config-core, could you put https://review.openstack.org/#/c/543127/ as next step for jenkins/script removal on your review queue, please? | 21:04 |
| AJaeger | thanks, corvus ! | 21:04 |
| * AJaeger signs off for today | 21:04 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| pabelanger | okay, moving on to ze03 for kernel updates, ze01 looks good | 21:06 |
| clarkb | tosky: things should be cleaned up now | 21:06 |
| clarkb | tosky: let us know if you still have trouble | 21:06 |
| *** dsariel_ has joined #openstack-infra | 21:08 | |
| tosky | clarkb: I'm retrying a recheck right now | 21:08 |
| *** esberglu has quit IRC | 21:08 | |
| *** dsariel has joined #openstack-infra | 21:08 | |
| *** esberglu has joined #openstack-infra | 21:09 | |
| AJaeger | still same error ;( https://review.openstack.org/#/c/542847/ | 21:09 |
| tosky | clarkb: same error | 21:09 |
| tosky | yep | 21:09 |
| * AJaeger signs really off | 21:10 | |
| pabelanger | dist-upgrade running now | 21:12 |
| clarkb | there is no queens branch now at least | 21:12 |
| fungi | okay, i'm back and catching up | 21:12 |
| *** psachin has quit IRC | 21:12 | |
| pabelanger | possible zuul still has it in memory? and hasn't reloaded configuration yet | 21:13 |
| fungi | looks like the tripleo branch situation was an unending source of excitement | 21:13 |
| *** esberglu has quit IRC | 21:13 | |
| smcginnis | fungi: You missed all the fun. | 21:13 |
| fungi | not all apparently | 21:13 |
| fungi | sounds like maybe branch deletion doesn't clear branch configuration from zuul? | 21:13 |
| ianw | yeah, i just got a random mail from a change complaining about tripleo jobs | 21:14 |
| pabelanger | fungi: yah, I wonder if we reload configuration on branch deletes | 21:14 |
| pabelanger | and since no branch, unsure how to force merge a fix | 21:15 |
| *** VW has quit IRC | 21:16 | |
| corvus | the running config reloaded fine, it may be only speculative configs that are getting the error | 21:16 |
| *** VW has joined #openstack-infra | 21:16 | |
| *** slaweq has quit IRC | 21:18 | |
| andreaf | corvus yeah I've got a couple of those two - same error on tripleo-ci-centos-7-ovb-containers-oooq | 21:19 |
| corvus | pabelanger: it's worth doing a full reconfig, that may clear it | 21:19 |
| *** kjackal has joined #openstack-infra | 21:19 | |
| corvus | feel free to do that while i continue to read code | 21:20 |
| *** felipemonteiro_ has quit IRC | 21:20 | |
| * tosky is ready to recheck when requested | 21:20 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:20 | |
| pabelanger | corvus: is that zuul-scheduler reload? | 21:21 |
| *** inc0 has quit IRC | 21:21 | |
| *** esberglu has joined #openstack-infra | 21:22 | |
| *** inc0 has joined #openstack-infra | 21:22 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Improve file renaming in stage-output https://review.openstack.org/544606 | 21:23 |
| corvus | pabelanger: /etc/init.d/zuul-scheduler reload | 21:24 |
| pabelanger | thanks | 21:24 |
| pabelanger | will do that now | 21:24 |
| *** jbadiapa has quit IRC | 21:24 | |
| *** olaph1 has quit IRC | 21:24 | |
| pabelanger | okay, should be running now | 21:25 |
| *** olaph has joined #openstack-infra | 21:26 | |
| *** jcoufal_ has joined #openstack-infra | 21:26 | |
| pabelanger | also, ze03.o.o has been rebooted and back online | 21:27 |
| *** pcaruana has quit IRC | 21:28 | |
| *** jcoufal has quit IRC | 21:29 | |
| pabelanger | and looks like zuul is finished reloading | 21:30 |
| pabelanger | tosky: do you mind rechecking again please | 21:30 |
| corvus | still no joy. i guess we'll have to restart | 21:30 |
| corvus | pabelanger: can you do that while i continue to debug? | 21:31 |
| tosky | pabelanger, corvus: should I recheck or not? | 21:31 |
| pabelanger | corvus: yup, will start saving queues | 21:31 |
| corvus | tosky: i already did | 21:31 |
| tosky | oh, oki | 21:31 |
| *** jcoufal has joined #openstack-infra | 21:32 | |
| clarkb | pabelanger: maybe double check there aren't any other release items queued first | 21:33 |
| clarkb | (since we seem to have had a few of those today and not paying attention to that was sadness) | 21:33 |
| pabelanger | clarkb: good idea | 21:34 |
| pabelanger | yah, I see 2 jobs running in release-post | 21:34 |
| pabelanger | I'll let them finish and update release team | 21:34 |
| clarkb | thanks | 21:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean master: DNM: Testing nodepool jobs https://review.openstack.org/536135 | 21:35 |
| *** jcoufal_ has quit IRC | 21:36 | |
| ianw | pabelanger: ^ just testing that f27 change... | 21:36 |
| pabelanger | ianw: thanks! I didn't get a chance to do that yet | 21:36 |
| *** mdbooth has quit IRC | 21:39 | |
| *** ari[m] has quit IRC | 21:39 | |
| *** jcoufal_ has joined #openstack-infra | 21:39 | |
| *** pcrews has quit IRC | 21:40 | |
| pabelanger | okay, release jobs look to be finished | 21:40 |
| pabelanger | will dump queues now | 21:40 |
| pabelanger | done, stopping scheduler | 21:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Clean up old images for testing https://review.openstack.org/544582 | 21:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool dsvm https://review.openstack.org/544583 | 21:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: WIP: Support ubuntu-beaver (18.04) with nodepool dsvm https://review.openstack.org/544574 | 21:41 |
| ianw | sorry, just had to rebase them for the test | 21:41 |
| *** akscram has left #openstack-infra | 21:41 | |
| *** jcoufal has quit IRC | 21:41 | |
| pabelanger | started, config reloading | 21:42 |
| *** esberglu has quit IRC | 21:43 | |
| *** jcoufal has joined #openstack-infra | 21:44 | |
| *** jcoufal_ has quit IRC | 21:45 | |
| pabelanger | corvus: does zuul-web also need to be restarted? zuul.o.o http looks down for me | 21:46 |
| corvus | pabelanger: yes, there's a bug that causes it to not survive scheduler restarts | 21:46 |
| pabelanger | k | 21:46 |
| corvus | i rechecked 542847, it looks good now | 21:47 |
| tosky | yep | 21:47 |
| pabelanger | starting to enqueue changes now | 21:47 |
| tosky | thanks | 21:47 |
| *** jcoufal_ has joined #openstack-infra | 21:48 | |
| *** Goneri has joined #openstack-infra | 21:48 | |
| *** jcoufal has quit IRC | 21:49 | |
| pabelanger | and done | 21:50 |
| pabelanger | I'm going to have to EOD here shortly but ze01 and ze03 are running HWE kernels now | 21:50 |
| *** pcrews has joined #openstack-infra | 21:50 | |
| clarkb | pabelanger: ok, process was stop executor, dist-upgrade, reboot? | 21:51 |
| clarkb | I can pick up 04 in a bit | 21:51 |
| *** jcoufal has joined #openstack-infra | 21:51 | |
| corvus | aha -- it's branch deletion *after* a failed tenant reconfiguration that's broken | 21:53 |
| fungi | neat | 21:53 |
| fungi | caching? | 21:53 |
| *** jcoufal_ has quit IRC | 21:54 | |
| *** jcoufal_ has joined #openstack-infra | 21:55 | |
| corvus | so much caching | 21:55 |
| corvus | i'm guessing that the failed reconfig caused us not to clear the cache | 21:56 |
| *** jcoufal has quit IRC | 21:57 | |
| corvus | though why the full reconfiguration didn't fix it is a second mystery... there will be 2 unit tests coming out of this. | 21:58 |
| *** rcernin has joined #openstack-infra | 22:00 | |
| *** jcoufal has joined #openstack-infra | 22:00 | |
| *** jcoufal_ has quit IRC | 22:01 | |
| *** slaweq has joined #openstack-infra | 22:04 | |
| *** jcoufal_ has joined #openstack-infra | 22:05 | |
| *** wolverineav has quit IRC | 22:05 | |
| *** wolverineav has joined #openstack-infra | 22:06 | |
| *** jcoufal has quit IRC | 22:07 | |
| clarkb | I'm asking ze04 to stop its executor now | 22:09 |
| *** wolverineav has quit IRC | 22:10 | |
| *** jcoufal has joined #openstack-infra | 22:10 | |
| *** slaweq has quit IRC | 22:11 | |
| *** jcoufal_ has quit IRC | 22:11 | |
| *** jcoufal_ has joined #openstack-infra | 22:14 | |
| *** jcoufal has quit IRC | 22:15 | |
| *** signed8bit is now known as signed8bit_Zzz | 22:16 | |
| *** signed8bit_Zzz is now known as signed8bit | 22:16 | |
| *** jcoufal has joined #openstack-infra | 22:16 | |
| *** bobh has quit IRC | 22:17 | |
| *** jcoufal_ has quit IRC | 22:20 | |
| *** dave-mccowan has quit IRC | 22:20 | |
| clarkb | ze04 is upgrading packages now | 22:21 |
| clarkb | hrm was it a mistake to do an interactive afs upgrade? | 22:23 |
| *** jcoufal_ has joined #openstack-infra | 22:23 | |
| clarkb | our cell is openstack.org and do we use a 500MB cache? | 22:23 |
| clarkb | ianw: pabelanger ^ | 22:23 |
| ianw | clarkb: it can take a minute to build the modules, is that what you mean? | 22:24 |
| *** jcoufal has quit IRC | 22:24 | |
| fungi | you can cancel out of the upgrade and rerun with DEBIAN_FRONTEND=noninteractive and --option 'Dpkg::Options::=--force-confold' | 22:24 |
| clarkb | ianw: no its asking me a bunch of configuration questions | 22:24 |
| fungi | send it a good old-fashioned sigint. apt is very forgiving | 22:24 |
| clarkb | ok | 22:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix branch deletion after failed reconfig https://review.openstack.org/544716 | 22:25 |
| fungi | debconf makes a nuisance of itself if you're relying on a configuration management system | 22:25 |
| corvus | fungi, pabelanger, clarkb: ^ that should take care of both bugs | 22:26 |
| * clarkb tries again | 22:26 | |
| fungi | corvus: thanks!!! | 22:26 |
| corvus | doh, forgot git add | 22:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix branch deletion after failed reconfig https://review.openstack.org/544716 | 22:26 |
| clarkb | fungi: thanks, I'm always wary of forcefully telling package managers to stop | 22:27 |
| fungi | it may complain that you need to run dpkg --configure -a | 22:27 |
| fungi | which, if you do that without DEBIAN_FRONTEND=noninteractive will take you right back to the same debconf prompts | 22:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add platform build jobs to nodepool experimental https://review.openstack.org/544717 | 22:29 |
| *** dsariel has quit IRC | 22:30 | |
| *** dsariel_ has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-infra | 22:31 | |
| *** threestrands has quit IRC | 22:31 | |
| *** threestrands has joined #openstack-infra | 22:31 | |
| *** VW_ has joined #openstack-infra | 22:34 | |
| *** VW has quit IRC | 22:38 | |
| *** VW_ has quit IRC | 22:39 | |
| clarkb | ok rebooting ze04 now | 22:40 |
| clarkb | looks like package installs and dkms stuff all went well | 22:40 |
| *** claudiub has joined #openstack-infra | 22:43 | |
| clarkb | ok it is running zuul-executor and I can ls things in the afs tree | 22:43 |
| *** claudiub has quit IRC | 22:43 | |
| clarkb | moving on to 05 now | 22:43 |
| ianw | clarkb: suse question just because you're here ... is zypper being replaced for <nextversion>? there was some chatter with dirk that things would be changing, and it seems that zypper is broken on debian testing, which doesn't bode well for it working on the next ubuntu | 22:44 |
| clarkb | not that I'm aware of tumbleweed is still using zypper | 22:45 |
| dirk | ianw: its still called zypper, just a newer version | 22:45 |
| dirk | ianw: the repository metadata changed a bit, and it can't be read by zypper of ubuntu (its the 13.2 version iirc, so a couple of years old) | 22:45 |
| ianw | oh, hey dirk :) | 22:45 |
| dirk | ianw: good morning, I guess you reminded me to go to bed now.. | 22:46 |
| ianw | dirk: i was just chasing up on adding the dep in the package. but it seems transition to testing is broken anyway https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812243 | 22:46 |
| openstack | Debian bug 812243 in zypper "zypper symbol not defined: symbol _ZN4zypp5CpeId11NoThrowType13lastMalformedE" [Grave,Open] | 22:46 |
| ianw | i'm just booting a fresh deb vm to see what's going on ... | 22:47 |
| dirk | that might be fixed already | 22:47 |
| dirk | around Feb 2016? or 2017? we did a NMU to fix this problem | 22:47 |
| dirk | and it was released as a update to ubuntu xenial | 22:47 |
| ianw | oh, ok, maybe it didn't make it into debian? https://tracker.debian.org/pkg/zypper seems it's blocked | 22:48 |
| dirk | thats quite possible. | 22:48 |
| dirk | basically the fix is to upload a new build of the same sources. its just some gcc whatever change of like two years ago that broke abi and noone rebuilt the package since then | 22:49 |
| *** pcichy has quit IRC | 22:49 | |
| dirk | so your fix to the dependencies will automatically solve this problem as well | 22:49 |
| *** pcichy has joined #openstack-infra | 22:50 | |
| *** aeng has quit IRC | 22:50 | |
| *** felipemonteiro__ has joined #openstack-infra | 22:51 | |
| *** kjackal has quit IRC | 22:52 | |
| ianw | ahh, this is lownmu too so it should be possible to fix this | 22:53 |
| clarkb | ze05 is updating packages now | 22:53 |
| *** jcoufal has joined #openstack-infra | 22:53 | |
| *** jcoufal_ has quit IRC | 22:53 | |
| *** mriedem has quit IRC | 22:55 | |
| *** felipemonteiro_ has quit IRC | 22:55 | |
| ianw | i really need to move my packages out of svn-buildpackage ... i can barely remember how to use svn these days | 22:56 |
| clarkb | I can barely remember how to package these days (/me hides in shame) | 22:57 |
| clarkb | I think I've built like 2 or three packages ever and that was all 10 years ago | 22:57 |
| fungi | update to git-buildpackage and start hosting the packaging sources on salsa.debian.org so you can swear at gitlab like everyone else | 22:59 |
| clarkb | we should give our gerrit a fun name like that too | 22:59 |
| dmsimard | btw it would be a good idea to check if ara on the executors got updated to 0.14.6 (up from 0.14.5) | 23:00 |
| dmsimard | (sorry I'm fighting $jenkins fires) | 23:00 |
| clarkb | ara==0.14.5 <- on ze01 | 23:00 |
| fungi | jenkins did seem to light a lot of things on fire this morning with that security announcement | 23:00 |
| clarkb | when was 0.14.6 published? | 23:00 |
| dmsimard | clarkb: yesterday | 23:01 |
| clarkb | ok | 23:01 |
| dmsimard | fungi: thanks for bringing that to our attention earlier, it didn't cross my feed | 23:01 |
| dmsimard | fungi: the worst part is that it's written even in a worse way than how intel describes meltdown in it's press releases | 23:02 |
| dmsimard | fungi: they word it in a somewhat ambiguous way to make you think that maybe only authenticated users are able to exploit it but really if you read "user accessing it, allowing anyone with Overall/Read access" https://jenkins.io/security/advisory/2018-02-14/ | 23:03 |
| dmsimard | You need "Overall/Read access" to even browse Jenkins unauthenticated | 23:03 |
| fungi | dmsimard: yep! we kept struggling with that back when we had an internet-facing jenkins webui | 23:04 |
| dmsimard | Thankfully our Jenkins' days are counted, we're skipping straight to Zuul v3 | 23:04 |
| fungi | and the jenkins upstream opinion seemed to be that you should keep it firewalled on an internal network and only ever expose it to "trusted" systems | 23:04 |
| clarkb | re ara I'm going to focus on executor kernel and afs upgrades and reboots | 23:05 |
| dmsimard | fungi: well yeah Jenkins is like backdoor as a service | 23:05 |
| clarkb | the ara update doesn't requier a zuul restart as it is forked and exec'd | 23:05 |
| clarkb | maybe someone else can investigate that on the lower numbers ze's | 23:05 |
| clarkb | I'm up to 05 now | 23:05 |
| dmsimard | clarkb: yeah no worries, the puppet update shenanigan might not work. I'll look at it when I have a chance, it's not a critical update | 23:06 |
| clarkb | ah ok if not critical then no rush | 23:06 |
| *** hongbin has quit IRC | 23:06 | |
| clarkb | ze05 is up now on new kernel and afs dirs can be listed | 23:07 |
| clarkb | moving on to 06 now | 23:07 |
| *** aeng has joined #openstack-infra | 23:07 | |
| dmsimard | the new executor metrics are working \o/ | 23:08 |
| dmsimard | I'll set something up in grafana when I have a chance | 23:08 |
| dmsimard | new metrics: https://i.imgur.com/VHqhFDF.png | 23:09 |
| *** edmondsw has quit IRC | 23:12 | |
| clarkb | 06 is getting packages updated now | 23:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add host/group vars https://review.openstack.org/544562 | 23:20 |
| *** jcoufal_ has joined #openstack-infra | 23:20 | |
| *** jcoufal has quit IRC | 23:21 | |
| clarkb | ianw: did you want to get any of these or were you happy enough just doing 02 :) | 23:23 |
| *** aeng has quit IRC | 23:25 | |
| ianw | clarkb: i can if you want :) | 23:26 |
| *** aeng has joined #openstack-infra | 23:26 | |
| clarkb | ianw: its going slowly enough that I might run out of daylight. But probably still amke it through all of them ebfore I call it a day | 23:27 |
| clarkb | 06 still dkms'ing | 23:27 |
| *** jcoufal has joined #openstack-infra | 23:27 | |
| ianw | what's left, 7-10 ? | 23:28 |
| clarkb | ya once 06 is done | 23:28 |
| clarkb | I think I'll still make it through them | 23:28 |
| ianw | i'll start at 10 | 23:28 |
| clarkb | wfm thanks | 23:28 |
| *** jcoufal_ has quit IRC | 23:28 | |
| *** jcoufal__ has joined #openstack-infra | 23:29 | |
| clarkb | ok 06 finally updated and has been rebooted. Zuul is running and afs is listable | 23:30 |
| clarkb | now 07 | 23:30 |
| *** jcoufal has quit IRC | 23:32 | |
| dmsimard | clarkb: you're lucky, we run out of daylight at like 4PM here during the winter | 23:32 |
| fungi | well, it's only 3:30pm where he is, i think | 23:34 |
| *** ajo is now known as ajo\pto | 23:35 | |
| clarkb | ya sunset is apparently 5:37 today | 23:35 |
| ianw | dirk: when around ... the problem here seems much deeper than just a new package dep. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797869 | 23:36 |
| openstack | Debian bug 797869 in src:libzypp "libzypp: shared library does not generate correct dependencies" [Serious,Open] | 23:36 |
| corvus | that's a whole 17 minutes later than montreal :) | 23:36 |
| *** e0ne has quit IRC | 23:36 | |
| clarkb | 07 is package updating now | 23:37 |
| fungi | it set here at home at 5:43pm local time. i win! | 23:37 |
| ianw | dirk: all that seems to need to be resolved before the new package (which has gpg2 deps) can move into testing. i'm afraid it's a bit much for me right now | 23:37 |
| * fungi knows the souther-hemispherers have us all beat | 23:38 | |
| *** mriedem has joined #openstack-infra | 23:38 | |
| ianw | but, this does seem like a warning shot that suse-minimal will be unbuildable on > xenial | 23:38 |
| clarkb | dnf hasn't trickled down to centos yet ahs it ? it shares a bunch of deps with zypper aiui so in theory that would work on centos if present | 23:39 |
| *** rlandy is now known as rlandy|bbl | 23:41 | |
| *** hashar has quit IRC | 23:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Invert executor ram statsd metric https://review.openstack.org/544742 | 23:41 |
| ianw | clarkb: no, centos is yum only. but for fedora-minimal we still just use yum on the base system to do the initial chroot install, then only once inside chroot do we use dnf | 23:44 |
| ianw | that may even work for opensuse-minimal, if zypper isn't on the host system? | 23:45 |
| *** dhill__ has quit IRC | 23:45 | |
| clarkb | maybe? I was just thinking we are likely to avoid zypper dep issues on a dnf system since there is a bit of overlap in deps (libsolv for example) | 23:46 |
| clarkb | though that could also backfire if they have conflicting needs in similar deps | 23:46 |
| clarkb | we've seen that before with python packaging :) | 23:46 |
| ianw | oh, so add a centos based builder? | 23:46 |
| clarkb | ya | 23:47 |
| ianw | i'm stopping & rebooting ze10 | 23:47 |
| clarkb | 07 rebooting now as well | 23:48 |
| *** jamesmcarthur has joined #openstack-infra | 23:48 | |
| *** signed8bit has quit IRC | 23:49 | |
| *** signed8bit has joined #openstack-infra | 23:49 | |
| clarkb | 07 look good. Doing 08 now | 23:50 |
| *** larsks has quit IRC | 23:51 | |
| ianw | well, libzypp spent 20 minutes grinding with g++ , put out "Test timeout computed to be: 9.99988e+06" which seems unlikely, failed all test cases, and died. so, yeah ... :) | 23:53 |
| *** jamesmcarthur has quit IRC | 23:53 | |
| *** kzaitsev_pi has quit IRC | 23:54 | |
| *** jcoufal has joined #openstack-infra | 23:55 | |
| ianw | pabelanger: f27 build looks good, thank http://logs.openstack.org/35/536135/3/check/nodepool-functional-py35-redhat-src/d1f49f0/controller/logs/builds/fedora-27-0000000001.log.gz | 23:55 |
| *** kzaitsev_pi has joined #openstack-infra | 23:55 | |
| dmsimard | corvus: haha yeah but we're past the winter solstice, thankfully getting some amount of extra daylight every day \o/ | 23:56 |
| fungi | ianw: are you saying suse doesn't test zupper builds on debian? for shame! ;) | 23:57 |
| fungi | er, zypper | 23:57 |
| *** jcoufal__ has quit IRC | 23:57 | |
| ianw | fungi: not really blaming anyone ... i was happy to get a nmu going for a missing package dep, but as soon as i saw "cmake" i was like ... uh oh ... :) | 23:59 |
| ianw | ze10 up with hwe, looks ok | 23:59 |
| ianw | clarkb: i'll do 09 now and then that's it? | 23:59 |
| *** felipemonteiro__ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
