| *** shardy_afk has quit IRC | 00:01 | |
| *** xarses_ has quit IRC | 00:03 | |
| *** slaweq has quit IRC | 00:03 | |
| *** emmaloui has quit IRC | 00:04 | |
| *** xarses_ has joined #openstack-infra | 00:06 | |
| pabelanger | clarkb: so, I can push those changed up in the next commit if you want, because right now prepare-docs-for-afs and prepare-infra-docs-for-afs are almost identical. But, I first want to get a publish job working with current logic, which we haven't tested with shade yet. Then work on optimizations | 00:07 |
|---|---|---|
| *** andreww has joined #openstack-infra | 00:08 | |
| *** shardy_afk has joined #openstack-infra | 00:09 | |
| *** rossella_s has quit IRC | 00:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Integration testing for mirror info script https://review.openstack.org/501904 | 00:09 |
| *** xarses_ has quit IRC | 00:10 | |
| *** rossella_s has joined #openstack-infra | 00:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add support for debian in configure-mirrors https://review.openstack.org/501537 | 00:11 |
| *** Apoorva has quit IRC | 00:12 | |
| *** Apoorva has joined #openstack-infra | 00:12 | |
| *** gildub has joined #openstack-infra | 00:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Integration testing for mirror info script https://review.openstack.org/501904 | 00:17 |
| clarkb | pabelanger: ya thats fine but the dirname thing should be fixed | 00:17 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add support for fedora in configure-mirrors https://review.openstack.org/501538 | 00:18 |
| *** vhosakot has quit IRC | 00:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Integration testing for mirror info script https://review.openstack.org/501904 | 00:19 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add support for opensuse in configure-mirrors https://review.openstack.org/501539 | 00:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove project pipeline definition from zuul-jobs https://review.openstack.org/501849 | 00:20 |
| *** srobert has quit IRC | 00:24 | |
| *** thorst has joined #openstack-infra | 00:27 | |
| *** Swami has quit IRC | 00:31 | |
| *** dmellado_ has joined #openstack-infra | 00:32 | |
| *** mwarad has joined #openstack-infra | 00:32 | |
| *** shardy_afk has quit IRC | 00:33 | |
| *** stevebaker has joined #openstack-infra | 00:33 | |
| *** derekh has quit IRC | 00:35 | |
| mordred | pabelanger, clarkb: I'm also fine with fixing prepare-docs-for-afs later - but when we do, I've already fixed it once here https://review.openstack.org/#/c/501381 | 00:35 |
| mordred | pabelanger, clarkb: so step one in fixing it would be, I believe, to sync in those versions | 00:35 |
| mordred | clarkb, pabelanger: actually - lemme just push up a fix | 00:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add prepare-docs-for-afs role https://review.openstack.org/501835 | 00:40 |
| mordred | that's just a rebase | 00:41 |
| *** shardy_afk has joined #openstack-infra | 00:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add prepare-docs-for-afs role https://review.openstack.org/501835 | 00:41 |
| mordred | clarkb, pabelanger: that syncs the playbook content that shoudl be the same to match what's in the other role | 00:41 |
| mordred | clarkb: which should fix your review comment too | 00:42 |
| mordred | pabelanger: I think you just got unlucky and copied the playbooks before the other patch landed | 00:42 |
| *** priteau has joined #openstack-infra | 00:45 | |
| *** liujiong has joined #openstack-infra | 00:45 | |
| *** efried has quit IRC | 00:47 | |
| *** squid has quit IRC | 00:49 | |
| *** priteau has quit IRC | 00:49 | |
| *** squid has joined #openstack-infra | 00:49 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| *** jamesmcarthur has joined #openstack-infra | 00:54 | |
| *** mwarad has quit IRC | 00:56 | |
| *** efried has joined #openstack-infra | 00:57 | |
| *** Apoorva_ has joined #openstack-infra | 00:58 | |
| *** jamesmcarthur has quit IRC | 00:59 | |
| *** slaweq has joined #openstack-infra | 00:59 | |
| *** Apoorva has quit IRC | 01:01 | |
| *** ramishra has joined #openstack-infra | 01:01 | |
| *** vhosakot has joined #openstack-infra | 01:01 | |
| *** mwarad has joined #openstack-infra | 01:02 | |
| *** Apoorva_ has quit IRC | 01:02 | |
| *** mwarad has quit IRC | 01:02 | |
| *** slaweq has quit IRC | 01:04 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move openstack-specific jobs to openstack-zuul-jobs https://review.openstack.org/501846 | 01:06 |
| *** rlandy|PTO is now known as rlandy | 01:07 | |
| *** LindaWang has joined #openstack-infra | 01:07 | |
| *** hamzy has joined #openstack-infra | 01:08 | |
| *** cuongnv has joined #openstack-infra | 01:09 | |
| *** harlowja has quit IRC | 01:11 | |
| *** mriedem has quit IRC | 01:14 | |
| *** thorst has quit IRC | 01:16 | |
| *** rlandy has quit IRC | 01:19 | |
| *** __Cyclone__ has quit IRC | 01:22 | |
| *** hongbin has joined #openstack-infra | 01:23 | |
| *** zhurong has joined #openstack-infra | 01:25 | |
| *** thorst has joined #openstack-infra | 01:28 | |
| *** vhosakot has quit IRC | 01:30 | |
| *** pcrews has quit IRC | 01:31 | |
| *** pcrews has joined #openstack-infra | 01:32 | |
| *** yamahata has quit IRC | 01:33 | |
| *** iyamahat has quit IRC | 01:37 | |
| *** skelso has joined #openstack-infra | 01:44 | |
| *** thorst has quit IRC | 01:46 | |
| *** skelso has quit IRC | 01:48 | |
| *** _Cyclone_ has joined #openstack-infra | 01:49 | |
| *** slaweq has joined #openstack-infra | 02:00 | |
| *** slaweq has quit IRC | 02:05 | |
| *** thorst has joined #openstack-infra | 02:07 | |
| *** thorst has quit IRC | 02:07 | |
| *** olaph1 has quit IRC | 02:08 | |
| *** zhurong has quit IRC | 02:11 | |
| *** xinliang has quit IRC | 02:12 | |
| *** zhurong has joined #openstack-infra | 02:14 | |
| *** xinliang has joined #openstack-infra | 02:24 | |
| *** andreww has quit IRC | 02:32 | |
| *** scottda_ has quit IRC | 02:37 | |
| *** priteau has joined #openstack-infra | 02:46 | |
| *** priteau has quit IRC | 02:50 | |
| *** jkilpatr has quit IRC | 02:59 | |
| *** annegentle has joined #openstack-infra | 03:02 | |
| *** nicolasbock has quit IRC | 03:11 | |
| *** markvoelker has quit IRC | 03:12 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 03:16 | |
| *** rkukura has quit IRC | 03:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Integration testing for mirror info script https://review.openstack.org/501904 | 03:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 03:22 |
| *** annegentle has quit IRC | 03:23 | |
| *** Sree_ has joined #openstack-infra | 03:26 | |
| *** Sree_ is now known as Guest27568 | 03:26 | |
| *** Guest24_ has joined #openstack-infra | 03:27 | |
| *** Guest24_ has quit IRC | 03:27 | |
| *** Guest24_ has joined #openstack-infra | 03:28 | |
| *** Guest24_ has quit IRC | 03:29 | |
| *** Guest24_ has joined #openstack-infra | 03:30 | |
| *** jascott1 has joined #openstack-infra | 03:37 | |
| *** udesale has joined #openstack-infra | 03:41 | |
| *** rkukura has joined #openstack-infra | 03:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Expand PATH for SUSE systems to include /usr/sbin https://review.openstack.org/501737 | 03:45 |
| *** hongbin has quit IRC | 03:56 | |
| *** xinliang has quit IRC | 03:59 | |
| *** liujiong_lj has joined #openstack-infra | 04:00 | |
| *** liujiong has quit IRC | 04:00 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Expand PATH for SUSE systems to include /usr/sbin https://review.openstack.org/501737 | 04:02 |
| *** slaweq has joined #openstack-infra | 04:03 | |
| *** cshastri has joined #openstack-infra | 04:04 | |
| *** xinliang has joined #openstack-infra | 04:06 | |
| *** slaweq has quit IRC | 04:07 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: Migrate machine inspection tests to requests_mock https://review.openstack.org/501942 | 04:07 |
| *** thorst has joined #openstack-infra | 04:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move pipeline definition from zuul-jobs to project-config https://review.openstack.org/501850 | 04:12 |
| *** thorst has quit IRC | 04:13 | |
| *** links has joined #openstack-infra | 04:19 | |
| *** gouthamr has quit IRC | 04:19 | |
| *** claudiub has joined #openstack-infra | 04:21 | |
| *** Guest27568 has quit IRC | 04:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move pipeline definition from zuul-jobs to project-config https://review.openstack.org/501850 | 04:29 |
| *** zhurong has quit IRC | 04:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add integration tests for validate-host https://review.openstack.org/501543 | 04:34 |
| *** yamahata has joined #openstack-infra | 04:35 | |
| *** coolsvap has joined #openstack-infra | 04:37 | |
| *** Guest24_ has quit IRC | 04:43 | |
| *** psachin has joined #openstack-infra | 04:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Integration testing for mirror info script https://review.openstack.org/501904 | 04:44 |
| *** marst has joined #openstack-infra | 04:45 | |
| *** marst has quit IRC | 04:50 | |
| *** rossella_s has quit IRC | 04:52 | |
| *** rossella_s has joined #openstack-infra | 04:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 04:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add integration test jobs https://review.openstack.org/501946 | 04:56 |
| *** sshnaidm|afk is now known as sshnaidm|off | 05:02 | |
| *** hichihara has quit IRC | 05:04 | |
| *** zhurong has joined #openstack-infra | 05:04 | |
| *** caphrim007 has joined #openstack-infra | 05:05 | |
| *** Guest24 has joined #openstack-infra | 05:06 | |
| *** Guest24 has quit IRC | 05:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:08 |
| *** caphrim0_ has joined #openstack-infra | 05:09 | |
| *** thorst has joined #openstack-infra | 05:09 | |
| *** caphrim007 has quit IRC | 05:13 | |
| *** markvoelker has joined #openstack-infra | 05:13 | |
| *** rossella_s has quit IRC | 05:13 | |
| *** thorst has quit IRC | 05:14 | |
| *** rossella_s has joined #openstack-infra | 05:15 | |
| *** gaurangt has quit IRC | 05:17 | |
| *** gaurangt has joined #openstack-infra | 05:19 | |
| *** e0ne has joined #openstack-infra | 05:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:23 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:28 |
| *** rossella_s has quit IRC | 05:30 | |
| *** rossella_s has joined #openstack-infra | 05:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a role to emit an informative header for logs https://review.openstack.org/501495 | 05:33 |
| *** Kevin_Zheng has quit IRC | 05:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:33 |
| *** iyamahat has joined #openstack-infra | 05:34 | |
| *** iyamahat_ has joined #openstack-infra | 05:34 | |
| *** iyamahat has quit IRC | 05:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add role to make /etc/ci/mirror_info.sh https://review.openstack.org/501578 | 05:41 |
| *** ykarel has joined #openstack-infra | 05:46 | |
| *** markvoelker has quit IRC | 05:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Build /etc/ci/mirror_info.sh script from ansible https://review.openstack.org/501954 | 05:52 |
| *** dims has quit IRC | 05:54 | |
| *** caphrim0_ has quit IRC | 05:59 | |
| *** caphrim007 has joined #openstack-infra | 05:59 | |
| *** dhajare has joined #openstack-infra | 06:00 | |
| AJaeger | ianw, could you put https://review.openstack.org/501382 , https://review.openstack.org/501392 and https://review.openstack.org/490430 on your review list, please? All just cosmetics... | 06:03 |
| *** dhajare has quit IRC | 06:04 | |
| *** slaweq has joined #openstack-infra | 06:04 | |
| AJaeger | thanks! | 06:05 |
| openstackgerrit | Merged openstack-infra/publications master: Fix glossary URL, use https https://review.openstack.org/490430 | 06:05 |
| *** aeng has quit IRC | 06:07 | |
| *** slaweq has quit IRC | 06:09 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Fix moved URLs https://review.openstack.org/501382 | 06:09 |
| *** thorst has joined #openstack-infra | 06:10 | |
| *** sflanigan has quit IRC | 06:11 | |
| *** thorst has quit IRC | 06:15 | |
| *** harlowja has joined #openstack-infra | 06:19 | |
| *** jascott1 has quit IRC | 06:21 | |
| *** dhajare has joined #openstack-infra | 06:24 | |
| *** rcernin has joined #openstack-infra | 06:32 | |
| *** harlowja has quit IRC | 06:40 | |
| *** e0ne has quit IRC | 06:41 | |
| *** pgadiya has joined #openstack-infra | 06:42 | |
| *** markvoelker has joined #openstack-infra | 06:44 | |
| *** priteau has joined #openstack-infra | 06:45 | |
| AJaeger | infra-root, I see a job waiting for 30mins now for a trusty job - and looking at grafana, see 100 error launch events per minute. Something looks wrong - see http://grafana.openstack.org/dashboard/db/nodepool | 06:46 |
| AJaeger | trusty node I mean | 06:46 |
| AJaeger | is ovh broken? http://grafana.openstack.org/dashboard/db/nodepool-ovh | 06:46 |
| *** priteau_ has joined #openstack-infra | 06:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Revert "Set OVH max-servers to 0"" https://review.openstack.org/501971 | 06:47 |
| AJaeger | if you agree with my diagnosis, please merge the above - might need hand merge since it will not get a trusty node ^ | 06:48 |
| * AJaeger needs to run some errands now | 06:48 | |
| *** priteau has quit IRC | 06:50 | |
| *** priteau_ has quit IRC | 06:51 | |
| *** dims has joined #openstack-infra | 06:51 | |
| *** slaweq has joined #openstack-infra | 06:55 | |
| *** rossella_s has quit IRC | 06:55 | |
| *** mudpuppy has quit IRC | 06:57 | |
| *** mudpuppy has joined #openstack-infra | 06:57 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use connection type supplied from nodepool https://review.openstack.org/501976 | 06:58 |
| *** rossella_s has joined #openstack-infra | 06:58 | |
| *** slaweq has quit IRC | 06:59 | |
| *** pgadiya has quit IRC | 07:00 | |
| *** rossella_s has quit IRC | 07:03 | |
| *** yolanda has joined #openstack-infra | 07:07 | |
| *** florianf has joined #openstack-infra | 07:07 | |
| *** rossella_s has joined #openstack-infra | 07:08 | |
| *** gcb has quit IRC | 07:08 | |
| *** thorst has joined #openstack-infra | 07:11 | |
| *** thorst has quit IRC | 07:16 | |
| *** markvoelker has quit IRC | 07:18 | |
| *** priteau has joined #openstack-infra | 07:19 | |
| *** jpich has joined #openstack-infra | 07:20 | |
| *** jascott1 has joined #openstack-infra | 07:21 | |
| *** gildub has quit IRC | 07:24 | |
| *** ralonsoh has joined #openstack-infra | 07:29 | |
| *** aarefiev_afk is now known as aarefiev | 07:31 | |
| *** tesseract has joined #openstack-infra | 07:31 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** chlong has joined #openstack-infra | 07:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Skip bifrost tinyipa jobs on xenial newton https://review.openstack.org/500773 | 07:44 |
| *** chlong has quit IRC | 07:45 | |
| *** egonzalez has joined #openstack-infra | 07:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in project description https://review.openstack.org/501392 | 07:56 |
| *** slaweq has joined #openstack-infra | 07:56 | |
| *** slaweq has quit IRC | 08:01 | |
| *** rossella_s has quit IRC | 08:02 | |
| *** rossella_s has joined #openstack-infra | 08:02 | |
| *** shardy_afk is now known as shardy | 08:07 | |
| *** rossella_s has quit IRC | 08:07 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Skip bifrost tinyipa jobs on xenial newton https://review.openstack.org/500773 | 08:10 |
| *** rossella_s has joined #openstack-infra | 08:10 | |
| *** thorst has joined #openstack-infra | 08:12 | |
| *** markvoelker has joined #openstack-infra | 08:15 | |
| *** thorst has quit IRC | 08:17 | |
| *** emmaloui has joined #openstack-infra | 08:23 | |
| *** bauzas is now known as bauwser | 08:25 | |
| *** chlong has joined #openstack-infra | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:34 | |
| *** chlong has quit IRC | 08:35 | |
| *** electrofelix has joined #openstack-infra | 08:38 | |
| *** priteau has quit IRC | 08:39 | |
| *** markvoelker has quit IRC | 08:49 | |
| *** chlong has joined #openstack-infra | 08:51 | |
| *** chlong_ has joined #openstack-infra | 08:54 | |
| *** chlong has quit IRC | 08:58 | |
| *** derekh has joined #openstack-infra | 08:58 | |
| *** liusheng has quit IRC | 09:00 | |
| *** liusheng has joined #openstack-infra | 09:00 | |
| *** hashar has joined #openstack-infra | 09:01 | |
| *** chlong_ has quit IRC | 09:04 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support username also for unmanaged cloud images https://review.openstack.org/500808 | 09:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add password to build and upload information https://review.openstack.org/502011 | 09:04 |
| *** thorst has joined #openstack-infra | 09:13 | |
| *** gmann_ has joined #openstack-infra | 09:14 | |
| *** gmann has quit IRC | 09:15 | |
| *** thorst has quit IRC | 09:17 | |
| *** dmellado_ has quit IRC | 09:18 | |
| *** dmellado_ has joined #openstack-infra | 09:20 | |
| *** chlong has joined #openstack-infra | 09:21 | |
| *** gmann_ has quit IRC | 09:23 | |
| *** ykarel|lunch is now known as ykarel | 09:23 | |
| *** chlong has quit IRC | 09:24 | |
| *** chlong has joined #openstack-infra | 09:25 | |
| *** dmellado_ has quit IRC | 09:29 | |
| *** shardy is now known as shardy_mtg | 09:29 | |
| *** dmellado_ has joined #openstack-infra | 09:30 | |
| *** gmann has joined #openstack-infra | 09:31 | |
| *** dmellado_ has quit IRC | 09:33 | |
| *** Sree_ has joined #openstack-infra | 09:36 | |
| *** Sree_ is now known as Guest11064 | 09:36 | |
| *** dmellado_ has joined #openstack-infra | 09:39 | |
| *** dmellado_ has quit IRC | 09:39 | |
| *** ramishra has quit IRC | 09:41 | |
| *** dmellado_ has joined #openstack-infra | 09:41 | |
| *** dmellado_ has quit IRC | 09:42 | |
| *** ramishra has joined #openstack-infra | 09:43 | |
| *** dmellado_ has joined #openstack-infra | 09:43 | |
| *** dmellado_ has quit IRC | 09:47 | |
| *** dmellado_ has joined #openstack-infra | 09:47 | |
| *** dhajare has quit IRC | 09:48 | |
| *** dmellado has quit IRC | 09:50 | |
| *** dmellado_ is now known as dmellado | 09:51 | |
| *** eroux has joined #openstack-infra | 09:52 | |
| *** chlong has quit IRC | 09:55 | |
| *** slaweq has joined #openstack-infra | 09:58 | |
| *** cuongnv has quit IRC | 09:59 | |
| *** slaweq has quit IRC | 10:03 | |
| *** zhurong has quit IRC | 10:04 | |
| *** e0ne has joined #openstack-infra | 10:07 | |
| *** dklyle has joined #openstack-infra | 10:09 | |
| *** liujiong_lj has quit IRC | 10:11 | |
| *** emmaloui has quit IRC | 10:12 | |
| *** emmaloui_ has joined #openstack-infra | 10:12 | |
| *** david-lyle has quit IRC | 10:12 | |
| *** szaher has quit IRC | 10:12 | |
| *** hrubi has quit IRC | 10:13 | |
| *** hrubi has joined #openstack-infra | 10:13 | |
| *** thorst has joined #openstack-infra | 10:14 | |
| *** jokke_ has quit IRC | 10:14 | |
| *** jokke_ has joined #openstack-infra | 10:14 | |
| *** szaher has joined #openstack-infra | 10:15 | |
| *** thorst has quit IRC | 10:18 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:20 | |
| *** wolverineav has joined #openstack-infra | 10:23 | |
| *** rkukura has quit IRC | 10:24 | |
| *** nicolasbock has joined #openstack-infra | 10:24 | |
| *** nicolasbock has quit IRC | 10:29 | |
| *** tosky has joined #openstack-infra | 10:29 | |
| *** udesale has quit IRC | 10:35 | |
| *** priteau has joined #openstack-infra | 10:36 | |
| *** nicolasbock has joined #openstack-infra | 10:42 | |
| *** dtantsur is now known as dtantsur|bbl | 10:43 | |
| *** sdague has joined #openstack-infra | 10:43 | |
| AJaeger | infra-root - if anybody is around - looks like we're not launching *anything* at all currently ;( The graphs at http://grafana.openstack.org/dashboard/db/nodepool looks really strange | 10:44 |
| *** aviau has quit IRC | 10:44 | |
| *** aviau has joined #openstack-infra | 10:45 | |
| *** pbourke has joined #openstack-infra | 10:45 | |
| AJaeger | #status notice Our CI systems experience a hickup, no new jobs are started. Please stay tuned and wait untils this resolved. | 10:45 |
| openstackstatus | AJaeger: sending notice | 10:45 |
| *** markvoelker has joined #openstack-infra | 10:46 | |
| *** yamamoto has quit IRC | 10:46 | |
| -openstackstatus- NOTICE: Our CI systems experience a hickup, no new jobs are started. Please stay tuned and wait untils this resolved. | 10:47 | |
| openstackstatus | AJaeger: finished sending notice | 10:48 |
| *** rossella_s has quit IRC | 10:51 | |
| *** gildub has joined #openstack-infra | 10:51 | |
| *** rossella_s has joined #openstack-infra | 10:53 | |
| *** iyamahat_ has quit IRC | 10:55 | |
| *** vipul has quit IRC | 10:55 | |
| *** vipul has joined #openstack-infra | 10:57 | |
| *** jkilpatr has joined #openstack-infra | 10:57 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** yamamoto has quit IRC | 11:02 | |
| *** LindaWang has quit IRC | 11:05 | |
| *** ykarel is now known as ykarel|afk | 11:06 | |
| *** jaypipes has joined #openstack-infra | 11:08 | |
| *** lihi- has joined #openstack-infra | 11:08 | |
| *** lihi has quit IRC | 11:09 | |
| *** dtantsur|bbl has quit IRC | 11:09 | |
| *** aviau has quit IRC | 11:09 | |
| *** dtantsur has joined #openstack-infra | 11:09 | |
| *** lihi- is now known as lihi | 11:09 | |
| *** dtantsur has quit IRC | 11:09 | |
| *** dtantsur has joined #openstack-infra | 11:09 | |
| *** jbadiapa has quit IRC | 11:09 | |
| *** rcernin has quit IRC | 11:09 | |
| *** jbadiapa has joined #openstack-infra | 11:09 | |
| *** aviau has joined #openstack-infra | 11:10 | |
| *** rakhmerov has quit IRC | 11:11 | |
| *** rakhmerov has joined #openstack-infra | 11:11 | |
| *** mat128 has joined #openstack-infra | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:14 | |
| *** thorst has joined #openstack-infra | 11:14 | |
| *** rhallisey has joined #openstack-infra | 11:17 | |
| *** jpena is now known as jpena|lunch | 11:19 | |
| *** thorst has quit IRC | 11:19 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** efried is now known as fried_rice | 11:21 | |
| *** onovy has quit IRC | 11:22 | |
| *** peterlisak has quit IRC | 11:22 | |
| *** tesseract has quit IRC | 11:23 | |
| *** peterlisak has joined #openstack-infra | 11:25 | |
| *** onovy has joined #openstack-infra | 11:25 | |
| *** tesseract has joined #openstack-infra | 11:27 | |
| *** alexchadin has joined #openstack-infra | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** rcernin has joined #openstack-infra | 11:36 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** alexchadin has quit IRC | 11:42 | |
| *** alexchadin has joined #openstack-infra | 11:43 | |
| *** dklyle has quit IRC | 11:44 | |
| *** dklyle has joined #openstack-infra | 11:44 | |
| *** thiagolib__ has joined #openstack-infra | 11:46 | |
| *** lihi has quit IRC | 11:48 | |
| *** lihi has joined #openstack-infra | 11:49 | |
| *** srobert has joined #openstack-infra | 11:51 | |
| *** pbourke has quit IRC | 11:53 | |
| *** slaweq has joined #openstack-infra | 12:00 | |
| *** thorst has joined #openstack-infra | 12:02 | |
| *** olaph has joined #openstack-infra | 12:03 | |
| *** trown|outtypewww is now known as trown | 12:04 | |
| *** slaweq has quit IRC | 12:05 | |
| *** bmjen has quit IRC | 12:06 | |
| *** dhajare has joined #openstack-infra | 12:06 | |
| *** bmjen has joined #openstack-infra | 12:07 | |
| *** Guest11064 has quit IRC | 12:09 | |
| *** Sree has joined #openstack-infra | 12:10 | |
| *** eroux has quit IRC | 12:13 | |
| *** tesseract has quit IRC | 12:14 | |
| *** tesseract has joined #openstack-infra | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:14 | |
| dirk | AJaeger: is that already preparation for the zuulv3 switchover? | 12:15 |
| *** dprince has joined #openstack-infra | 12:15 | |
| *** emmaloui_ has quit IRC | 12:15 | |
| *** shardy_mtg is now known as shardy | 12:16 | |
| *** andreas_s has joined #openstack-infra | 12:16 | |
| *** Sree has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| *** Hunner has quit IRC | 12:17 | |
| *** tonyb has quit IRC | 12:19 | |
| *** tonyb has joined #openstack-infra | 12:20 | |
| *** jpena|lunch is now known as jpena | 12:20 | |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** Hunner has joined #openstack-infra | 12:21 | |
| *** Hunner has quit IRC | 12:21 | |
| *** Hunner has joined #openstack-infra | 12:21 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** yamamoto has joined #openstack-infra | 12:22 | |
| *** emmaloui has joined #openstack-infra | 12:30 | |
| *** ramishra has quit IRC | 12:30 | |
| *** alee has quit IRC | 12:32 | |
| *** eumel8 has joined #openstack-infra | 12:32 | |
| *** LindaWang has joined #openstack-infra | 12:33 | |
| *** markvoelker has quit IRC | 12:34 | |
| *** markvoelker has joined #openstack-infra | 12:34 | |
| *** baoli has joined #openstack-infra | 12:36 | |
| dmsimard | dirk: no, unexpected | 12:36 |
| *** baoli has quit IRC | 12:36 | |
| dmsimard | AJaeger: ianw seemed to mention some odd issues last night | 12:36 |
| *** baoli has joined #openstack-infra | 12:37 | |
| andreaf | AJaeger: yeah a lot of nodes in building state :( it looks really broken | 12:37 |
| *** jaypipes has quit IRC | 12:38 | |
| *** tmorin1 has joined #openstack-infra | 12:39 | |
| *** pbourke has joined #openstack-infra | 12:39 | |
| *** rossella_s has quit IRC | 12:42 | |
| dmsimard | AJaeger / infra-root: ianw picked up on some oddities last night, not sure if related to the ongoing issues but re-posting in case | 12:43 |
| dmsimard | http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2017-09-08.log.html#t2017-09-08T00:28:54 | 12:43 |
| dmsimard | http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2017-09-08.log.html#t2017-09-08T03:14:39 | 12:43 |
| dmsimard | wish I could look bu no access :( | 12:45 |
| *** rossella_s has joined #openstack-infra | 12:45 | |
| *** andreas_s has quit IRC | 12:45 | |
| *** eharney has quit IRC | 12:46 | |
| *** verdurin has quit IRC | 12:47 | |
| *** akscram1 has quit IRC | 12:48 | |
| *** akscram1 has joined #openstack-infra | 12:49 | |
| *** srobert has quit IRC | 12:50 | |
| *** jascott1_ has joined #openstack-infra | 12:52 | |
| *** jascott1 has quit IRC | 12:54 | |
| *** rossella_s has quit IRC | 12:55 | |
| *** greghaynes has quit IRC | 12:56 | |
| *** rossella_s has joined #openstack-infra | 12:56 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** camunoz has joined #openstack-infra | 12:57 | |
| *** camunoz_ has joined #openstack-infra | 12:57 | |
| smcginnis | Some zuulv3 things were enabled yesterday I thought. Bad interaction maybe? | 12:58 |
| smcginnis | OVH was also brought back online IIRC. | 12:58 |
| *** psachin has quit IRC | 12:59 | |
| mordred | oh my | 13:00 |
| *** cshastri has quit IRC | 13:00 | |
| smcginnis | mordred: You broke the internet. | 13:01 |
| *** slaweq has joined #openstack-infra | 13:01 | |
| mordred | looks like something wrong with the images | 13:01 |
| *** greghaynes has joined #openstack-infra | 13:02 | |
| mordred | cat: /usr/local/jenkins/slave_scripts/afs-slug.sh: No such file or directory^M | 13:02 |
| mordred | that's happening in the ready script | 13:02 |
| *** kgiusti has joined #openstack-infra | 13:05 | |
| *** slaweq has quit IRC | 13:06 | |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** srobert has joined #openstack-infra | 13:11 | |
| *** srobert has quit IRC | 13:11 | |
| *** srobert has joined #openstack-infra | 13:12 | |
| dmsimard | mordred: oh yuck so nodepool is cycling through nodes non-stop because the ready script is failing ? | 13:12 |
| sdague | that would explain the yellow bar | 13:13 |
| dirk | https://review.openstack.org/#/c/499857/ looks related | 13:14 |
| dmsimard | mordred: https://github.com/openstack-infra/project-config/commit/8a7456ed9097a3b016cded77393c0408547e2918#diff-b8162a0f19d5488a41cd7159c59f831c | 13:14 |
| dmsimard | dirk: beat you to it :D | 13:14 |
| dirk | looks like configure_mirror.sh needs to be fixed | 13:15 |
| dmsimard | yeah, it would probably be faster to revert to the previous image | 13:15 |
| dmsimard | fixing configure_mirror.sh we would need to wait for a build+upload | 13:15 |
| dmsimard | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n80 | 13:15 |
| *** Dinesh_Bhor has quit IRC | 13:16 | |
| mordred | yup. that'sit | 13:16 |
| mordred | thanks for finding the commit | 13:16 |
| mordred | let's get afs-slug.sh re-added | 13:17 |
| *** Goneri has joined #openstack-infra | 13:17 | |
| dmsimard | mordred: that'd require an image rebuild | 13:17 |
| mordred | I will, in the mean time, delete the current images | 13:17 |
| dmsimard | mordred: nodepool keeps the previous image until the next rotation right ? | 13:17 |
| mordred | yup | 13:17 |
| dmsimard | ok | 13:17 |
| mordred | for exactly this type of situation :) | 13:17 |
| * dmsimard nods | 13:17 | |
| * dirk bets this never happened before | 13:17 | |
| mordred | dmsimard: can you make a patch real quick re-adding afs-slug while I got deleting some images? | 13:17 |
| *** korzen has joined #openstack-infra | 13:18 | |
| dmsimard | mordred: a revert or just add it back ? | 13:18 |
| odyssey4me | mordred out of curiousity I take it that means that images use the same name, and one is newer and therefore the newer one always gets used? or does nodepool figure out the newer image itself and designate its use by uuid when building? | 13:18 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Revert "Ansiblify the wheel build a little more" https://review.openstack.org/502055 | 13:19 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Revert "Ansiblify the wheel build a little more" https://review.openstack.org/502055 | 13:20 |
| mordred | odyssey4me: the second - we keep a local list of our images - so what you know as 'ubuntu-xenial' is actually currently ubuntu-xenial-1504835477 on rax-iad and ubuntu-xenial-1504835467 on rax-ord | 13:21 |
| mordred | oh goodie. now I can't get to review.o.o | 13:22 |
| mordred | there it is | 13:22 |
| odyssey4me | mordred ah, thanks - good to know | 13:22 |
| openstackgerrit | Lance Bragstad proposed openstack/ptgbot master: Fixup minor miss-spelling in README.md https://review.openstack.org/502057 | 13:22 |
| dmsimard | mordred: revert is conflicting due to commits you added on top | 13:22 |
| mordred | dmsimard: thanks - I'm gonna edit that just a sec | 13:22 |
| dmsimard | mordred: want me to do a squashed revert ? | 13:23 |
| odyssey4me | I have a task on the table to implement nodepool for our CI, but it looks like nodepool is a little between states... I might need a bit of a crash course at the PTG. :) | 13:23 |
| *** bobh has joined #openstack-infra | 13:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Re-add afs-slug.sh to the repo https://review.openstack.org/502055 | 13:24 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Revert "Ansiblify the wheel build a little more" https://review.openstack.org/502055 | 13:24 |
| dirk | dmsimard: mordred: cleaned that up | 13:24 |
| dmsimard | dirk: oops, you stepped on mordred | 13:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Ansiblify the wheel build a little more" https://review.openstack.org/502055 | 13:24 |
| mordred | oh. whoops | 13:25 |
| dmsimard | er | 13:25 |
| odyssey4me | lol | 13:25 |
| mordred | dmsimard: good - we did the same thing :) | 13:25 |
| mordred | dirk: I meant | 13:25 |
| dirk | mordred: sorry | 13:25 |
| dmsimard | ok, whew | 13:25 |
| dirk | yeah, commit message changed. | 13:25 |
| dirk | hmm, yours is better. | 13:25 |
| mordred | ok. I force-merged that so that we don't have to wait on nodepool - and so that it can get to th enodepool builders quicker | 13:25 |
| *** gildub has quit IRC | 13:26 | |
| sc` | have a 'business continuity' question. i've recently acquired the openstack namespace on chef's supermarket for publishing openstack deployment with the support of the foundation. i'd like to ensure that after my cycle as ptl, should i choose not to run again for a fourth, that the secrets are maintained beyond me and my password manager, so how shall i do that? | 13:27 |
| sc` | i'm asking here, as i've been told that infra would usually have been the gatekeeper | 13:27 |
| *** gouthamr has joined #openstack-infra | 13:27 | |
| *** hashar has quit IRC | 13:28 | |
| sc` | in the event the whole core team disperses again, i'd rather have some kind of plan in mind | 13:28 |
| *** hashar has joined #openstack-infra | 13:28 | |
| dirk | sc`: not a lot of people might be listening today due to travels to the PTG | 13:29 |
| dirk | sc`: I suggest dropping a n ote to the mailing list | 13:29 |
| sc` | ah. ptg | 13:29 |
| sc` | i should have planned for that, but i tend not to get in too many chef-related conversations at openstack events ;) | 13:30 |
| mordred | sc`: that is definitely the type of thing we can arrange to transfer into the care of the infra team -we have a nice pile of passwords for the various accounts to things | 13:30 |
| frickler | mordred: I mentioned review.o.o getting bumpy yesterday already, now it is near flatlining. maybe once you are waiting for all the nodepool rebuilds, you can give it a kick, too. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 13:30 |
| * mordred is getting all the fun this morning | 13:30 | |
| sc` | mordred: capital. something to work on once ptg is over and everyone is back home, i suspect | 13:31 |
| *** links has quit IRC | 13:31 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Disable ansible-hardening trusty gate in queens https://review.openstack.org/501880 | 13:32 |
| *** srobert has quit IRC | 13:32 | |
| *** rossella_s has quit IRC | 13:33 | |
| *** eharney has joined #openstack-infra | 13:34 | |
| *** rossella_s has joined #openstack-infra | 13:36 | |
| *** alee has joined #openstack-infra | 13:37 | |
| mordred | ok. we should now be killing our clouds booting new nodes | 13:37 |
| *** xyang1 has joined #openstack-infra | 13:40 | |
| * sc` pokes gerrit with a +1 stick of poking | 13:43 | |
| *** jbadiapa has quit IRC | 13:43 | |
| mordred | #status notice nodepool issue related to bad images has been resolved, builds should be coming back online soon. Restarted gerrit due to reasons. Happy Friday. | 13:45 |
| openstackstatus | mordred: sending notice | 13:45 |
| dmsimard | mordred: I know ready-scripts are no longer a thing in v3, but we're still going to be building images -- should we have an integration test that checks that the image builds and boots properly ? | 13:46 |
| -openstackstatus- NOTICE: nodepool issue related to bad images has been resolved, builds should be coming back online soon. Restarted gerrit due to reasons. Happy Friday. | 13:46 | |
| dmsimard | kind of heavy with all the repo caching but eh | 13:46 |
| mordred | dmsimard: we've talked about that- it's a thing that yolanda wanted/did with gozer a couple of years ago ... | 13:46 |
| mordred | dmsimard: biggest issue is that - the size of them is quite large | 13:47 |
| dmsimard | yeah | 13:47 |
| openstackstatus | mordred: finished sending notice | 13:48 |
| *** baoli has quit IRC | 13:48 | |
| mordred | there's also the idea of having nodepool build an image andupload it but not mark it ready until it has triggered a zuul job to run on it | 13:48 |
| *** scottda_ has joined #openstack-infra | 13:48 | |
| mordred | which might be a thing we can think about now with v3 | 13:48 |
| dmsimard | mordred: yeah, I was thinking about something along these lines -- have nodepool build the image, upload it, ensure that it can boot it and run some basic job and then mark it live | 13:49 |
| *** yamamoto has quit IRC | 13:49 | |
| *** Sree has joined #openstack-infra | 13:50 | |
| Shrews | that's unpossible. you can't both have zuul use only ready images AND not consider an image ready until zuul uses it | 13:50 |
| Shrews | so some thought is needed around that | 13:50 |
| mordred | Shrews: ++ | 13:50 |
| mordred | Shrews: definitely not a thing that can be done today for sure | 13:50 |
| mordred | Shrews: and many dragons | 13:51 |
| Shrews | so many dragons | 13:51 |
| *** hongbin has joined #openstack-infra | 13:51 | |
| mordred | Shrews: you're a dragon | 13:51 |
| Shrews | mordred: i'm the last dragon | 13:52 |
| Shrews | pcrews: OMG OMG OMG... http://www.slashfilm.com/samuel-l-jackson-is-shonuff-in-the-last-dragon-remake/ | 13:53 |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| Shrews | that's from 2008... where's my remake??? | 13:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Move working centos jobs in bifrost to check queue https://review.openstack.org/499525 | 13:56 |
| *** srobert has joined #openstack-infra | 13:57 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:57 | |
| AJaeger | mordred: thanks for getting us back on track! | 13:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to using stestr https://review.openstack.org/491035 | 13:58 |
| *** felipemonteiro__ has joined #openstack-infra | 13:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move gate-osc-placement-dsvm-functional-ubuntu-xenial to check https://review.openstack.org/501884 | 13:59 |
| mordred | AJaeger: \o/ | 13:59 |
| mordred | AJaeger: let's hope that's the last excitement for today | 13:59 |
| AJaeger | mordred: could you check http://grafana.openstack.org/dashboard/db/nodepool-ovh , please? | 13:59 |
| AJaeger | I'm not happy about the error node launch attempt numbers | 13:59 |
| AJaeger | Is that normal - or do we have a problem in ovh? | 14:00 |
| mordred | WELL ... | 14:00 |
| mordred | coupla things ... we may have quota settings set wrong for citycloud: | 14:00 |
| mordred | OpenStackCloudHTTPError: (403) Client Error for url: https://sto2.citycloud.com:8774/v2/bed89257500340af8d0fbe7141b1bfd6/servers Quota exceeded for cores, instances: Requested 8, 1, but already used 400, 50 of 400, 50 cores, instances | 14:00 |
| *** iyamahat has joined #openstack-infra | 14:00 | |
| mordred | OpenStackCloudHTTPError: (403) Client Error for url: https://kna1.citycloud.com:8774/v2/bed89257500340af8d0fbe7141b1bfd6/servers Quota exceeded for cores, instances: Requested 8, 1, but already used 400, 50 of 400, 50 cores, instances | 14:01 |
| AJaeger | that's citycloud, not ovh, isn't it? | 14:01 |
| sdague | mordred: I bet the tight loop just leaked a ton of instances | 14:02 |
| *** slaweq has joined #openstack-infra | 14:02 | |
| AJaeger | yeah - looking at http://grafana.openstack.org/dashboard/db/nodepool-ovh we delete more instances than we have quota | 14:02 |
| *** felipemonteiro_ has quit IRC | 14:02 | |
| *** esberglu has joined #openstack-infra | 14:02 | |
| *** coolsvap has quit IRC | 14:03 | |
| mordred | we're getting 500 errors from nova for ovh: 'Timed out waiting for a reply to message ID 51a1b2ff274c416fb1bdbfdc53a6acc6' | 14:03 |
| *** baoli has joined #openstack-infra | 14:03 | |
| *** bnemec is now known as beekneemech | 14:03 | |
| mordred | AJaeger: let's land https://review.openstack.org/#/c/501971/ | 14:05 |
| AJaeger | mordred: ok. | 14:05 |
| mordred | AJaeger: I don't think there's any reason for us to keep requesting servers from ovh if we can't boot them - and let the system settle down for a bit | 14:05 |
| mordred | AJaeger: I agre though about force-merging- it's gonna be a few hours at least before that job gets nodes :) | 14:06 |
| AJaeger | mordred: I had hoped we wouldn't need that one ;( Let's do it and give some time to settle... | 14:06 |
| *** slaweq has quit IRC | 14:07 | |
| *** alexchadin has quit IRC | 14:07 | |
| mordred | AJaeger: ++ | 14:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Revert "Set OVH max-servers to 0"" https://review.openstack.org/501971 | 14:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Set OVH max-servers to 0"" https://review.openstack.org/501971 | 14:10 |
| *** iyamahat_ has joined #openstack-infra | 14:13 | |
| AJaeger | EmilienM, mwhahaha, could you review https://review.openstack.org/#/c/499189/ , please? | 14:13 |
| mwhahaha | AJaeger: sure | 14:14 |
| *** iyamahat_ has quit IRC | 14:15 | |
| AJaeger | mwhahaha: thanks, then I'll +2 | 14:15 |
| *** iyamahat has quit IRC | 14:15 | |
| EmilienM | AJaeger: ack | 14:18 |
| *** marst has joined #openstack-infra | 14:22 | |
| *** dave-mccowan has joined #openstack-infra | 14:23 | |
| *** jamesmcarthur has quit IRC | 14:24 | |
| *** rbrndt has joined #openstack-infra | 14:24 | |
| *** annegentle has joined #openstack-infra | 14:32 | |
| *** ykarel|afk has quit IRC | 14:33 | |
| *** olaph has quit IRC | 14:34 | |
| *** emmaloui has quit IRC | 14:34 | |
| *** emmaloui has joined #openstack-infra | 14:34 | |
| *** jcoufal has joined #openstack-infra | 14:36 | |
| *** olaph has joined #openstack-infra | 14:36 | |
| *** korzen has quit IRC | 14:40 | |
| *** jamesmcarthur has joined #openstack-infra | 14:40 | |
| *** dhajare has quit IRC | 14:42 | |
| *** annegentle has quit IRC | 14:45 | |
| *** nicolasbock has quit IRC | 14:47 | |
| *** nicolasbock has joined #openstack-infra | 14:47 | |
| dmsimard | yay graphs are moving in the right direction now | 14:50 |
| *** xarses_ has joined #openstack-infra | 14:54 | |
| *** vhosakot has joined #openstack-infra | 14:56 | |
| *** baoli has quit IRC | 14:58 | |
| *** aarefiev is now known as aarefiev_afk | 14:58 | |
| *** alee has quit IRC | 14:59 | |
| *** shardy is now known as shardy_mtg | 15:00 | |
| *** camunoz_ has quit IRC | 15:01 | |
| *** egonzalez has quit IRC | 15:01 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Use wildecard in delorean-current whitelist https://review.openstack.org/502087 | 15:01 |
| *** dklyle has quit IRC | 15:02 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add whereto test as optional step in doc build https://review.openstack.org/502089 | 15:02 |
| *** slaweq has joined #openstack-infra | 15:03 | |
| *** david-lyle has joined #openstack-infra | 15:03 | |
| dmsimard | mordred: lol so RDO got hit by the same issue because we build the *same* image as upstream for our nodepool | 15:03 |
| dmsimard | mordred: so deleting the most recent image makes nodepool use the previous one ? | 15:03 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Use wildcard in delorean-current whitelist https://review.openstack.org/502087 | 15:04 |
| AJaeger | dhellmann: regarding 502089: With the move to zuulv3, the file you touched might not be used anymore. Please discuss with pabelanger and mordred how to do your change for zuulv3 as well. | 15:05 |
| dhellmann | AJaeger : thanks, I thought that might be the case. I figured at least this way we had something concrete to discuss. | 15:05 |
| *** caphrim007 has quit IRC | 15:07 | |
| *** baoli has joined #openstack-infra | 15:07 | |
| *** caphrim007 has joined #openstack-infra | 15:07 | |
| *** slaweq has quit IRC | 15:08 | |
| *** annegentle has joined #openstack-infra | 15:09 | |
| *** yamamoto has quit IRC | 15:11 | |
| *** caphrim007 has quit IRC | 15:12 | |
| *** alee has joined #openstack-infra | 15:14 | |
| *** felipemonteiro__ has quit IRC | 15:19 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:19 | |
| *** links has joined #openstack-infra | 15:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip devstack/non-doc jobs for keystone changes https://review.openstack.org/494018 | 15:22 |
| pabelanger | AJaeger: dhellmann: Ya, I'd love to sync up at PTG next week and give an overview of changes, and how we might be able to reduce some complexity for moving directories around | 15:22 |
| dhellmann | pabelanger : sounds like a good idea | 15:23 |
| *** dangers_` has joined #openstack-infra | 15:25 | |
| *** Sree has quit IRC | 15:26 | |
| *** links has quit IRC | 15:26 | |
| *** dangers_away has quit IRC | 15:26 | |
| *** Sree has joined #openstack-infra | 15:26 | |
| *** LindaWang has quit IRC | 15:27 | |
| *** Sree has quit IRC | 15:31 | |
| mordred | dmsimard: yes - deleting the most recent dib-image ... one sec, there's a doc | 15:31 |
| Shrews | dmsimard: yes, but you'd need to pause image building until it's corrected, or else it would happen again | 15:31 |
| mordred | dmsimard: https://docs.openstack.org/infra/system-config/nodepool.html#bad-images | 15:31 |
| mordred | Shrews: the underlying problem should now be fixed - we deleted a file from project-config that was needed | 15:32 |
| Shrews | mordred: yep, saw that. thx for fixing | 15:32 |
| Shrews | dmsimard: mordred: https://docs.openstack.org/infra/system-config/nodepool.html#bad-images | 15:33 |
| *** wolverineav has quit IRC | 15:33 | |
| Shrews | is what you're looking for maybe? | 15:33 |
| mordred | Shrews: if you have a spare second, I think the fedora-26 and one other started building before we landed the project-config change so may have gotten bogus scripts dir | 15:33 |
| *** felipemonteiro__ has quit IRC | 15:33 | |
| *** wolverineav has joined #openstack-infra | 15:33 | |
| *** felipemonteiro__ has joined #openstack-infra | 15:34 | |
| mordred | Shrews: but I couldn't delete them because they were in building state ... we should probalby proactively delete them just to be sure | 15:34 |
| mordred | dhellmann: so- quick sake of argument / question ... | 15:37 |
| *** shardy_mtg is now known as shardy | 15:37 | |
| *** rlandy is now known as rlandy|brb | 15:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-bup master: Don't backup /afs https://review.openstack.org/502103 | 15:37 |
| *** rlandy|brb is now known as rlandy | 15:37 | |
| *** wolverineav has quit IRC | 15:38 | |
| dhellmann | mordred : yes? | 15:38 |
| mordred | dhellmann: how hard would it be / bad of an idea to include that whereto check in openstacksphinx extension? | 15:38 |
| mordred | dhellmann: or is that a layer violation? | 15:38 |
| *** jamesmcarthur has quit IRC | 15:38 | |
| clarkb | infra-root in cleaning out my inbox I've noticed bup on at least one instance is attempting to backup afs. I've also noticed that the backup server has run out of disk so I guess I will be working on adding another volume to back the backups | 15:38 |
| dhellmann | yeah, I thought about that. I'm not sure. | 15:38 |
| mordred | kk | 15:38 |
| dhellmann | def. worth discussing as an option | 15:38 |
| dhellmann | if we built docs using "tox -e docs" I would just tell teams to add the command there, you know? | 15:39 |
| *** dprince has quit IRC | 15:39 | |
| dhellmann | so I wonder if we want to pull back all the way and think about that again before we shoehorn it in somewhere else | 15:39 |
| *** rguillebert has left #openstack-infra | 15:42 | |
| Shrews | mordred: hrm, not seeing any recent afs-slug errors in the logs | 15:44 |
| *** jamesmcarthur has joined #openstack-infra | 15:44 | |
| clarkb | infra-root I am going to begin the process of creating a third 1TB volume for the backup server in ord | 15:44 |
| *** hashar is now known as hasharAway | 15:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add whereto test as optional step in doc build https://review.openstack.org/502106 | 15:45 |
| mordred | dhellmann: ^^ | 15:45 |
| *** felipemonteiro_ has joined #openstack-infra | 15:46 | |
| mordred | dhellmann: speaking of pulling back all the way .. I actually have a VERY long email written about the PTI for docs | 15:46 |
| mordred | Shrews: cool -maybe we got it in in time | 15:46 |
| dhellmann | mordred : I look forward to reading it one day? :-) | 15:46 |
| pabelanger | clarkb: ack | 15:48 |
| *** felipemonteiro__ has quit IRC | 15:49 | |
| clarkb | pabelanger: mordred can you review https://review.openstack.org/#/c/502103/ as well (the node trying to backup all of afs isn't actually a valid backup node (review-dev) so its not super urgent but seems like a good belts and suspenders for if/when we have valid overlap of backups and afs mounted on the node | 15:50 |
| *** slaweq has joined #openstack-infra | 15:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add prepare-docs-for-afs role https://review.openstack.org/501835 | 15:54 |
| pabelanger | clarkb: +3 | 15:54 |
| pabelanger | mordred: clarkb: AJaeger: ^ new update to prepare-docs-for-afs to fix typo and comments | 15:54 |
| dirk | infra-root: http://logs.openstack.org/96/498596/4/gate/gate-nova-pep8-ubuntu-xenial/c644b67/console.html | 15:55 |
| pabelanger | dirk: looks like network issue cloning git | 15:55 |
| *** vhosakot has quit IRC | 15:56 | |
| *** dprince has joined #openstack-infra | 15:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add whereto test as optional step in doc build https://review.openstack.org/502106 | 15:56 |
| mordred | dhellmann, pabelanger: ^^ that one might not fail syntax checks :) | 15:57 |
| mordred | dhellmann: (pinging you on it as a way to start putting v3-style things in front of your eyes) | 15:57 |
| dhellmann | let the edumucatin' begin | 15:58 |
| AJaeger | pabelanger: LGTM | 15:58 |
| *** rkukura has joined #openstack-infra | 15:58 | |
| clarkb | ok resize2fs is running now | 15:58 |
| *** jascott1_ has quit IRC | 15:58 | |
| *** jascott1 has joined #openstack-infra | 15:59 | |
| clarkb | ok df reports 1TB free | 16:04 |
| *** rkukura has quit IRC | 16:04 | |
| clarkb | jeblair: looking at other cronspam, http://paste.openstack.org/show/620733/ appears to affect all our hosts. Is that familiar to you? I'm guessing that some config we use now results in a warning with newer exim? | 16:04 |
| *** mat128 has quit IRC | 16:05 | |
| *** slaweq has quit IRC | 16:06 | |
| jeblair | clarkb: doesn't ring a bell | 16:06 |
| *** slaweq has joined #openstack-infra | 16:06 | |
| clarkb | #status info | 16:07 |
| openstackstatus | clarkb: unknown command | 16:07 |
| clarkb | derp | 16:07 |
| clarkb | its log | 16:07 |
| *** tesseract has quit IRC | 16:08 | |
| *** wolverineav has joined #openstack-infra | 16:08 | |
| clarkb | #status log New 1TB cinder volume attached to Rax ORD backup server and backups filesystem extended to include that space. This was done in response to a full filesystem. Backups should begin functioning again on the next pulse. | 16:08 |
| openstackstatus | clarkb: finished logging | 16:08 |
| *** slaweq has quit IRC | 16:09 | |
| *** slaweq has joined #openstack-infra | 16:09 | |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| *** Apoorva has joined #openstack-infra | 16:11 | |
| *** yamamoto has joined #openstack-infra | 16:12 | |
| *** rkukura has joined #openstack-infra | 16:12 | |
| *** yamamoto has quit IRC | 16:16 | |
| *** rkukura has quit IRC | 16:17 | |
| pabelanger | mordred: minding reviewing https://review.openstack.org/501835/ again? | 16:17 |
| *** slaweq has quit IRC | 16:18 | |
| *** rkukura has joined #openstack-infra | 16:18 | |
| *** slaweq has joined #openstack-infra | 16:18 | |
| mordred | pabelanger: +3 | 16:18 |
| mordred | pabelanger: while you're in docs thinking space: https://review.openstack.org/#/c/502106/ | 16:19 |
| *** slaweq has quit IRC | 16:19 | |
| clarkb | ok with backups theoretically happy now. time to look at other things. mordred do we know why ovh suddenly was unhappy? | 16:19 |
| mordred | pabelanger: I actually think we could push that into the base tox role with an opt-in flag, but we should chat about the "batteries-included-vs-not" stuff next week before we do | 16:20 |
| clarkb | maybe related to the tight loop of boot delete due to afs problem? | 16:20 |
| mordred | clarkb: no - we were getting nova 500 errors on boot of nova not being able to get a message or something | 16:20 |
| mordred | clarkb: so we might have just slammed them too hard | 16:20 |
| clarkb | gotcha, so maybe we turn them back on and everything is ok? | 16:21 |
| mordred | clarkb: we should also check back in on citycloud - we were getting quota overrun errors on both regions | 16:21 |
| mordred | clarkb: maybe so | 16:21 |
| mordred | clarkb, Shrews: we really should add some consume-quota-values-from-cloud stuff to nodepool - at the very least as a double-check step | 16:21 |
| *** jamesmcarthur has quit IRC | 16:21 | |
| clarkb | mordred: do you know which region(s) in citycloud were unhappy? | 16:22 |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| mordred | like - grab instances and cpu core quotas if they're there, and if we're over what the cloud says we can do, even if we're under nodepool quota, don't try | 16:22 |
| mordred | clarkb: yes- one sec | 16:22 |
| mordred | sto2 and kna1 | 16:23 |
| mordred | clarkb: at least, those are the two I noticed and pasted in to irc | 16:23 |
| *** tmorin1 has quit IRC | 16:23 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/elastic-recheck master: Add query for OVB testenv failures https://review.openstack.org/502117 | 16:23 |
| clarkb | ok those are the two that are shared between v3 and v2. max-servers is 5 and 45 respecitvely though which should be right at quota | 16:24 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add prepare-docs-for-afs role https://review.openstack.org/501835 | 16:25 |
| clarkb | perhaps there is a race where we can go over that threshold? | 16:25 |
| *** jamesmcarthur has quit IRC | 16:26 | |
| clarkb | were we could attempt to boot a 46th node in v2 ? | 16:27 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config master: networking-odl: make rally/functional/fullstack with nitrogen voting https://review.openstack.org/501521 | 16:29 |
| *** szaher has quit IRC | 16:31 | |
| *** felipemonteiro_ has quit IRC | 16:32 | |
| *** felipemonteiro has joined #openstack-infra | 16:33 | |
| *** yamahata has quit IRC | 16:34 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:34 | |
| *** jpich has quit IRC | 16:35 | |
| *** szaher has joined #openstack-infra | 16:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Revert "Enable maintainConnectionCache" https://review.openstack.org/502121 | 16:38 |
| *** felipemonteiro has quit IRC | 16:38 | |
| clarkb | openstack usage show takes a long time to retunr | 16:38 |
| *** rkukura has quit IRC | 16:38 | |
| *** armax has joined #openstack-infra | 16:38 | |
| *** jamesmcarthur has joined #openstack-infra | 16:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rework publish-openstack-python-docs(-infra) job https://review.openstack.org/501363 | 16:40 |
| clarkb | oh and that only shows you cummulative usage not point in time? | 16:40 |
| *** rcernin has quit IRC | 16:40 | |
| *** rkukura has joined #openstack-infra | 16:40 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Load plugin configuration for the LIO job https://review.openstack.org/502122 | 16:40 |
| clarkb | mordred: is there no way to ask eg nova what it thinks my quota usage is? | 16:41 |
| *** derekh has quit IRC | 16:42 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Load tempest configuration for the LIO job https://review.openstack.org/502122 | 16:42 |
| *** claudiub has quit IRC | 16:42 | |
| *** yamamoto has joined #openstack-infra | 16:44 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** squid has quit IRC | 16:44 | |
| *** emmaloui has quit IRC | 16:45 | |
| *** rkukura has quit IRC | 16:45 | |
| *** squid has joined #openstack-infra | 16:45 | |
| *** rkukura has joined #openstack-infra | 16:46 | |
| *** shardy has quit IRC | 16:47 | |
| *** caphrim007 has joined #openstack-infra | 16:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 16:49 |
| *** rkukura has quit IRC | 16:49 | |
| *** rkukura has joined #openstack-infra | 16:51 | |
| *** jascott1 has quit IRC | 16:52 | |
| *** jcoufal has quit IRC | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| mordred | clarkb: yes, you can, and shade supports it | 16:54 |
| mordred | clarkb: BUT - last time I checked at least one of our clouds returned all zeros | 16:55 |
| clarkb | :/ | 16:55 |
| *** armax has quit IRC | 16:55 | |
| mordred | clarkb: that may no longer be true | 16:55 |
| *** Sree has joined #openstack-infra | 16:56 | |
| mordred | clarkb: shade aPI call is "get_compute_limits" if you want to play | 16:56 |
| *** yamamoto has quit IRC | 16:56 | |
| *** yamahata has joined #openstack-infra | 16:56 | |
| mordred | clarkb: http://paste.openstack.org/show/620739/ - just did that against my personal vexxhost account | 16:57 |
| mordred | clarkb: since the flavor object (that we get anyway in nodepool) contains a "vcpus" attribute | 16:58 |
| *** caphrim0_ has joined #openstack-infra | 16:59 | |
| mordred | clarkb: if we get limits, we should be able to do the math during quota calc to determine whether the request we want to make would be over quota or not | 16:59 |
| mordred | clarkb: get_compute_limits also returns 'total_cores_used': 2 and 'total_instances_used': 1 | 16:59 |
| clarkb | ya I think those values are most useful to the calculation | 17:00 |
| clarkb | since we can easily cross check that against what nodepool thinks it is using and log a warning/not boot if we are in weird situation | 17:00 |
| mordred | clarkb: ok - I just checked the infra rax account | 17:00 |
| *** Sree has quit IRC | 17:00 | |
| mordred | clarkb: and it's returning data | 17:00 |
| *** rkukura_ has joined #openstack-infra | 17:00 | |
| *** harlowja has joined #openstack-infra | 17:00 | |
| mordred | (they were the ones I remember returning zeros) | 17:00 |
| *** Sree has joined #openstack-infra | 17:01 | |
| *** rkukura has quit IRC | 17:01 | |
| *** rkukura_ is now known as rkukura | 17:01 | |
| mordred | clarkb: so yah- I think a simple/naive implementation that just grabs data and does a sanify check woudn't be terribly hard and could keep us from spinning when we're at or over quota | 17:01 |
| *** jpena is now known as jpena|off | 17:02 | |
| *** caphrim007 has quit IRC | 17:02 | |
| *** Swami has joined #openstack-infra | 17:04 | |
| *** Sree has quit IRC | 17:05 | |
| *** jascott1 has joined #openstack-infra | 17:05 | |
| *** ralonsoh has quit IRC | 17:06 | |
| *** rkukura has quit IRC | 17:10 | |
| *** esberglu has quit IRC | 17:11 | |
| *** rkukura has joined #openstack-infra | 17:12 | |
| *** jamesmca_ has joined #openstack-infra | 17:13 | |
| *** szaher has quit IRC | 17:15 | |
| *** vhosakot has joined #openstack-infra | 17:15 | |
| *** rkukura has quit IRC | 17:16 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config master: networking-odl: make rally/functional/fullstack with nitrogen voting https://review.openstack.org/501521 | 17:16 |
| *** tosky has quit IRC | 17:17 | |
| *** jamesmcarthur has quit IRC | 17:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Fix syntax error in prepare-docs-for-afs https://review.openstack.org/502128 | 17:19 |
| *** slaweq has joined #openstack-infra | 17:20 | |
| *** rkukura has joined #openstack-infra | 17:20 | |
| *** rkukura has quit IRC | 17:21 | |
| *** pbourke has quit IRC | 17:21 | |
| *** isq has joined #openstack-infra | 17:23 | |
| *** slaweq has quit IRC | 17:25 | |
| pabelanger | clarkb: dmsimard: I see a few held nodes in nodepool for you, can we delete them? | 17:25 |
| *** sambetts is now known as sambetts|afk | 17:25 | |
| *** yamamoto has joined #openstack-infra | 17:25 | |
| *** trown is now known as trown|lunch | 17:25 | |
| clarkb | pabelanger: this the centos server in rax? | 17:26 |
| clarkb | I don't need it if dmsimard is done with it | 17:26 |
| pabelanger | clarkb: ya, 2 on rax and another on citycloud | 17:27 |
| dmsimard | clarkb: you can delete it | 17:27 |
| dmsimard | Didn't learn anything useful from it | 17:27 |
| *** puiterwijk|cld is now known as patricku | 17:27 | |
| clarkb | pabelanger: there should only be one? | 17:28 |
| * clarkb looks | 17:28 | |
| clarkb | oh network overlay ya those can be deleted | 17:28 |
| *** patricku has quit IRC | 17:28 | |
| *** patricku has joined #openstack-infra | 17:28 | |
| *** yamamoto has quit IRC | 17:29 | |
| *** jamesmca_ has quit IRC | 17:29 | |
| *** rbrndt has quit IRC | 17:30 | |
| pabelanger | thanks, deleted | 17:30 |
| pabelanger | gate still recovering, looks like a bunch of centos nodes got launched which is satifying tripleo jobs | 17:31 |
| *** quite has quit IRC | 17:32 | |
| *** jcoufal has joined #openstack-infra | 17:35 | |
| *** e0ne has quit IRC | 17:35 | |
| *** jcoufal has quit IRC | 17:35 | |
| *** jcoufal has joined #openstack-infra | 17:36 | |
| ttx | clarkb: would be great to get +2A on https://review.openstack.org/#/c/501867/1 and get that link up before the event starts | 17:37 |
| *** ykarel has joined #openstack-infra | 17:37 | |
| *** quite has joined #openstack-infra | 17:38 | |
| clarkb | ttx: done | 17:38 |
| ttx | thanks! | 17:39 |
| *** esberglu has joined #openstack-infra | 17:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Revert "Enable maintainConnectionCache" https://review.openstack.org/502121 | 17:41 |
| *** tosky has joined #openstack-infra | 17:41 | |
| *** jcoufal has quit IRC | 17:42 | |
| *** jcoufal has joined #openstack-infra | 17:42 | |
| *** jcoufal has quit IRC | 17:45 | |
| *** esberglu has quit IRC | 17:45 | |
| *** jcoufal has joined #openstack-infra | 17:45 | |
| *** cody-somerville has joined #openstack-infra | 17:47 | |
| *** armax has joined #openstack-infra | 17:48 | |
| *** jcoufal has quit IRC | 17:48 | |
| *** jcoufal has joined #openstack-infra | 17:48 | |
| pabelanger | clarkb: I've see a lot dsvm jobs fail like http://logs.openstack.org/39/500139/6/check/gate-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend-ubuntu-xenial-nv/ab732e3//console.html today | 17:49 |
| *** electrofelix has quit IRC | 17:50 | |
| pabelanger | clarkb: not sure if limited to rax or something that recently changed | 17:50 |
| clarkb | I don't think anything recently changed in rax or in setup workspace other than moving swap setup to setup host | 17:50 |
| pabelanger | clarkb: http://status.openstack.org/elastic-recheck/#1686542 | 17:51 |
| pabelanger | large spike in timeout jobs | 17:51 |
| pabelanger | curious if related | 17:51 |
| *** Goneri has quit IRC | 17:52 | |
| pabelanger | maybe not, most seem to be related to tripleo right now | 17:54 |
| pabelanger | http://logs.openstack.org/29/499529/6/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container-upgrades-nv/537753d//console.html is inap | 17:55 |
| pabelanger | seems to have cleared up on newer jobs, not sure what happened | 17:58 |
| *** ykarel has quit IRC | 18:01 | |
| pabelanger | Ya, its like ansible failed to run for some reason | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file containing v2 to v3 mappings https://review.openstack.org/491804 | 18:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 18:09 |
| mordred | pabelanger: so that we don't forget, the v3 publish-to-pypi jobs still are using test_pypi | 18:11 |
| pabelanger | mordred: ya, I want to tag sandbox one more time, to confirm it is still working | 18:12 |
| pabelanger | then I think we can update to production creds | 18:12 |
| mordred | pabelanger: sweet | 18:12 |
| clarkb | pabelanger: that is a multinode jobs could be ssh problems like we have seen on puppetmaster? | 18:15 |
| clarkb | also one thing that changed is newer version of ansible | 18:15 |
| pabelanger | clarkb: ya, maybe? we don't seem to get any stdout from ansible-playbook | 18:16 |
| *** thiagolib__ has quit IRC | 18:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 18:20 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix syntax error in prepare-docs-for-afs https://review.openstack.org/502128 | 18:26 |
| *** caphrim0_ has quit IRC | 18:28 | |
| *** hasharAway has quit IRC | 18:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to openstack-tox-py35 job https://review.openstack.org/500330 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 18:28 |
| *** florianf has quit IRC | 18:29 | |
| *** hashar has joined #openstack-infra | 18:29 | |
| *** rbrndt has joined #openstack-infra | 18:29 | |
| openstackgerrit | German Eichberger proposed openstack-infra/project-config master: [WIP] Generate Octavia Amphora image artefact https://review.openstack.org/501890 | 18:30 |
| *** eharney has quit IRC | 18:31 | |
| clarkb | mordred: pabelanger do we want to try turning ovh back on? maybe just bump up max servers to ~50 in each region by hand to start? | 18:32 |
| AJaeger | clarkb: what about one region first? | 18:33 |
| *** armax has quit IRC | 18:33 | |
| clarkb | AJaeger: ya we can start with bhs1 first as it is the bigger of the two (so in theory can handle more load) | 18:34 |
| AJaeger | clarkb: it's worth trying - with the large backlog we have... | 18:34 |
| clarkb | AJaeger: ya thats why I think we should try to get ovh back if possible | 18:35 |
| clarkb | pabelanger: mordred let me go ahead and manually bump it up to 50 in bhs1 now | 18:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add chdir for prepare-docs-for-afs https://review.openstack.org/502136 | 18:35 |
| *** esberglu has joined #openstack-infra | 18:35 | |
| clarkb | actually lets go small, 5 | 18:36 |
| pabelanger | clarkb: wfm | 18:36 |
| AJaeger | clarkb: go for it ;) | 18:36 |
| clarkb | mordred: AJaeger pabelanger rax-dfw fedora26 just hit the afs slug problem | 18:37 |
| clarkb | did we not delete all the images with that problems/ | 18:37 |
| *** Sree has joined #openstack-infra | 18:37 | |
| pabelanger | clarkb: I haven't been following, possible it build again without fixes? | 18:38 |
| clarkb | we've got stale nodes in ovh-bhs1 that are building for hours too I'm trying to lcean those up so we'll see the new lower quota as useable | 18:39 |
| clarkb | pabelanger: ya maybe? | 18:39 |
| *** esberglu has quit IRC | 18:40 | |
| pabelanger | clarkb: sounds like we need to delete DIB again, so that should pickup fixes hopefully in rebuild | 18:40 |
| pabelanger | clarkb: I can delete the DIB now if you want | 18:42 |
| clarkb | pabelanger: that would be good so I can focus on getting ovh sorted | 18:43 |
| pabelanger | kk | 18:43 |
| clarkb | pabelanger: just delete the newest one I think? | 18:43 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add chdir for prepare-docs-for-afs https://review.openstack.org/502136 | 18:43 |
| clarkb | pabelanger: sincei n theory the oldest is old enough? | 18:43 |
| mordred | clarkb, AJaeger: there was a race in dib deleting | 18:43 |
| mordred | because I was still coffeeing | 18:43 |
| pabelanger | clarkb: ah | 18:44 |
| pabelanger | I deleted the latest DIB for fedora | 18:44 |
| pabelanger | nodepool should do the right thing now | 18:44 |
| mordred | I did the delete-dib-image then landed the project-config patch - and fedora26 started building immediately | 18:44 |
| *** eharney has joined #openstack-infra | 18:44 | |
| mordred | pabelanger: look at the dib ages and see if there are any about the same age as that fedora one was | 18:44 |
| mordred | pabelanger: I think one other image started building at the same time | 18:44 |
| pabelanger | clarkb: we also might want to move nb01 and nb02 both back to rackspace, because of network slowness in vexxhost | 18:45 |
| clarkb | ok nodes are botting in ovh bhs1. Also I think I may have found why citycloud is/was complaining. I set max-servers to 5 but we have two 3-node envs booting | 18:45 |
| clarkb | mordred: ^ | 18:45 |
| pabelanger | mordred: ya | 18:45 |
| clarkb | so does seem like there must be an accounting bug in nodepool somewhere | 18:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Also support IPv6 in the finger log streamer https://review.openstack.org/502137 | 18:46 |
| pabelanger | well, once a DIB has started building, you cannot delete it | 18:46 |
| pabelanger | and after it is built, we upload right away | 18:46 |
| *** esberglu has joined #openstack-infra | 18:46 | |
| pabelanger | so, there is a period of time, if you know a bad image is getting built, there is no way to stop it from uploading, going live | 18:46 |
| clarkb | hrm we should be abl to put a check in place on the upload side that doesn't proceed if state on dib image is deleting | 18:46 |
| clarkb | we haev ip addresses for ovh primary nodes. So it got that far along at least | 18:47 |
| clarkb | and they successfully built | 18:47 |
| clarkb | I think we can go ahead and turn them on proepr like | 18:47 |
| pabelanger | cool | 18:48 |
| *** sdague has quit IRC | 18:49 | |
| *** jcoufal has quit IRC | 18:54 | |
| AJaeger | looking at http://logs.openstack.org/20/500320/11/gate/tox-py35-on-zuul/8c20bdb/ , I see a 70 MB job-output.json file - and a 3.2 MB job-output.txt.gz . Should we compress the json file? | 18:54 |
| AJaeger | clarkb: shall I propose a revert of the revert of the revert? | 18:55 |
| AJaeger | (for ovh)? | 18:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "Revert "Revert "Set OVH max-servers to 0""" https://review.openstack.org/502140 | 18:55 |
| clarkb | AJaeger: ^ there it is | 18:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "Revert "Revert "Set OVH max-servers to 0""" https://review.openstack.org/502141 | 18:55 |
| clarkb | AJaeger: ++ to compressing both of them | 18:56 |
| AJaeger | clarkb: you beat me ;) | 18:56 |
| clarkb | by 10 seconds :) | 18:56 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: Migrate machine tests related to state transitions https://review.openstack.org/502142 | 18:56 |
| AJaeger | clarkb: I should have waited for your answer - abandoned mine | 18:57 |
| clarkb | ok | 18:57 |
| AJaeger | TheJulia: could you review https://review.openstack.org/500773 , please? | 18:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Also support IPv6 in the finger log streamer https://review.openstack.org/502137 | 18:57 |
| *** mat128 has joined #openstack-infra | 18:58 | |
| AJaeger | how can we teach codesearch about new repos? It does not find openstack-infra/zuul-jobs - I think we should have it going forward. | 19:00 |
| *** e0ne has joined #openstack-infra | 19:03 | |
| *** sdague has joined #openstack-infra | 19:03 | |
| clarkb | I think we have to restart the service | 19:04 |
| *** tinwood has quit IRC | 19:05 | |
| AJaeger | might be good to do this so that all zuul v3 repos are searchable for the v3 switch... | 19:07 |
| *** tinwood has joined #openstack-infra | 19:07 | |
| *** trown|lunch is now known as trown | 19:10 | |
| *** Apoorva_ has joined #openstack-infra | 19:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove root-marker from shell task https://review.openstack.org/502144 | 19:13 |
| pabelanger | AJaeger: ^does that make sense now? | 19:13 |
| pabelanger | AJaeger: we need to create root-marker twice on existing publishing jobs | 19:13 |
| pabelanger | because run-docs.sh creates it, then our JJB macro | 19:14 |
| *** Apoorva has quit IRC | 19:16 | |
| *** ijw_ has joined #openstack-infra | 19:19 | |
| *** ijw_ has quit IRC | 19:20 | |
| *** e0ne has quit IRC | 19:20 | |
| SamYaple | w/in 19 | 19:20 |
| *** ijw_ has joined #openstack-infra | 19:21 | |
| *** slaweq has joined #openstack-infra | 19:22 | |
| AJaeger | pabelanger: check closely for e.g. tagged builds: | 19:22 |
| AJaeger | marker is at doc/build/html/.root-marker | 19:22 |
| AJaeger | and then we move mv doc/build/html/$tag doc/publish - so there's no marker moved to doc/publish | 19:23 |
| AJaeger | pabelanger: lines 3 and 43 of main.yaml | 19:23 |
| clarkb | mordred: pabelanger can I get a second review on https://review.openstack.org/#/c/502140/ | 19:23 |
| AJaeger | pabelanger: so, double check that the document you publish really has a .root-marker file in all cases. | 19:24 |
| *** jascott1 has quit IRC | 19:24 | |
| AJaeger | pabelanger: I'm calling it a day now, have a great evening. | 19:25 |
| * AJaeger waves good bye | 19:25 | |
| * mordred hands AJaeger a pie | 19:25 | |
| AJaeger | yummie... thanks | 19:25 |
| mordred | AJaeger, clarkb: the codesearch config file should get written by jeepyb based on projects.yaml - so yah, we might be missing a service restart | 19:26 |
| pabelanger | AJaeger: thanks | 19:26 |
| *** slaweq has quit IRC | 19:27 | |
| mordred | clarkb: also, re: citycloud max-servers from scrollback - all the multinode quota accounting is new/different in v3, so we may just want to put a pin in that and look for if it's still there | 19:27 |
| clarkb | mordred: ya I think that is probably best | 19:29 |
| *** slaweq has joined #openstack-infra | 19:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Rework .root-marker for shell task https://review.openstack.org/502144 | 19:31 |
| *** cody-somerville has quit IRC | 19:32 | |
| *** Goneri has joined #openstack-infra | 19:32 | |
| *** mat128 has quit IRC | 19:33 | |
| *** cody-somerville has joined #openstack-infra | 19:33 | |
| *** jascott1 has joined #openstack-infra | 19:36 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: DNM: Try centos-7 and opensuse-423 devstack jobs https://review.openstack.org/502147 | 19:37 |
| *** Apoorva_ has quit IRC | 19:39 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Support IPv6 in the finger log streamer https://review.openstack.org/502137 | 19:42 |
| *** ijw_ has quit IRC | 19:42 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Make the Boot-From-Volume job voting https://review.openstack.org/502148 | 19:42 |
| *** alee has quit IRC | 19:42 | |
| *** Sree has quit IRC | 19:42 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Rework .root-marker for shell task https://review.openstack.org/502144 | 19:44 |
| *** tosky has quit IRC | 19:49 | |
| *** kgiusti has left #openstack-infra | 19:53 | |
| clarkb | pabelanger: looking at recent failures it looks like rax ord at least is just going slowly | 19:54 |
| clarkb | looks like could be network related | 19:54 |
| clarkb | repo setup and pip installs are not quick | 19:54 |
| *** rossella_s has quit IRC | 19:54 | |
| *** alee has joined #openstack-infra | 19:55 | |
| clarkb | access to tarballs is pretty quick from a random node i hopped on in ord | 19:56 |
| *** tosky has joined #openstack-infra | 19:56 | |
| clarkb | trying to clone nova now | 19:56 |
| clarkb | that seems pretty quick too | 19:56 |
| clarkb | 10MBps | 19:57 |
| clarkb | which is about 100mbps throughput | 19:57 |
| pabelanger | mordred: jeblair: clarkb: AJaeger: Yay: https://docs.openstack.org/sandbox/latest/ | 19:57 |
| pabelanger | I'm going to do another test before moving it to shade | 19:57 |
| clarkb | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=570 looking at that maybe we just overloaded the mirror | 19:58 |
| clarkb | there is a gap in the snmp and eth0 traffic spikes | 19:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: DNM: Try centos-7 and opensuse-423 devstack jobs https://review.openstack.org/502147 | 19:59 |
| AJaeger | pabelanger: real test is the next job publishing content - if there would be no .root-marker (there is one), it would delete the directory again. This looks good, so far - now try branch or tagging... | 20:00 |
| pabelanger | AJaeger: ya, that's what I'm doing now | 20:00 |
| pabelanger | AJaeger: sandbox now deleted in afs | 20:01 |
| jeblair | AJaeger: btw we merged a bunch of changes to the infra manual zuulv3 migration page while you were on vacation, if you want to look at it: file:///home/corvus/git/openstack-infra/infra-manual/doc/build/html/zuulv3.html | 20:01 |
| pabelanger | AJaeger: I'll then try a tab | 20:01 |
| pabelanger | tag* | 20:01 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Make the Boot-From-Volume job voting https://review.openstack.org/502148 | 20:01 |
| AJaeger | jeblair: that file URL does not work for me ;) | 20:03 |
| AJaeger | jeblair: will read later - but now disappear again. | 20:03 |
| jeblair | AJaeger: ha :) | 20:04 |
| jeblair | ftr the url is https://docs.openstack.org/infra/manual/zuulv3.html | 20:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add migration tool for v2 to v3 conversion https://review.openstack.org/491805 | 20:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file containing v2 to v3 mappings https://review.openstack.org/491804 | 20:07 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add UPPER_CONSTRAINTS_FILE file if it exists https://review.openstack.org/500320 | 20:08 |
| ttx | clarkb: for some reason the cityguide patch failed @ https://review.openstack.org/#/c/501867/1 | 20:12 |
| ttx | some probably should just recheck | 20:13 |
| clarkb | ttx: that is a known issue I have a change up to debug it. I will recheck | 20:13 |
| ttx | ok | 20:13 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: trigger tripleo scen1-4,7 on tht/environments https://review.openstack.org/501443 | 20:16 |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config master: Add propose-project-config-update job https://review.openstack.org/499845 | 20:25 |
| *** rossella_s has joined #openstack-infra | 20:28 | |
| *** mat128 has joined #openstack-infra | 20:29 | |
| *** hashar has quit IRC | 20:32 | |
| *** beekneemech has quit IRC | 20:35 | |
| *** trown is now known as trown|outtypewww | 20:35 | |
| *** mat128 has quit IRC | 20:38 | |
| *** Apoorva has joined #openstack-infra | 20:39 | |
| jlvillal | Anyone know who is in charge of the page: https://www.openstack.org/ptg/#tab_travel ? | 20:43 |
| jlvillal | I was thinking it would be useful if they gave some info on how to use mass-transit to get to the hotel | 20:43 |
| jlvillal | I added what I found to our Ironic PTG Etherpad: https://etherpad.openstack.org/p/ironic-queens-ptg | 20:44 |
| *** Apoorva has quit IRC | 20:44 | |
| *** Apoorva has joined #openstack-infra | 20:48 | |
| *** eharney has quit IRC | 20:49 | |
| *** Apoorva has quit IRC | 20:49 | |
| *** dprince has quit IRC | 20:49 | |
| *** Apoorva has joined #openstack-infra | 20:49 | |
| clarkb | jlvillal: https://launchpad.net/openstack-org is place to file bugs | 20:51 |
| clarkb | jlvillal: I can also ping them separately with that info | 20:51 |
| *** srobert has quit IRC | 20:52 | |
| *** funzo_ has quit IRC | 20:53 | |
| *** olaph has quit IRC | 20:53 | |
| clarkb | jlvillal: there is also the cityguide wiki page you could update | 20:53 |
| *** olaph has joined #openstack-infra | 20:53 | |
| clarkb | jlvillal: https://wiki.openstack.org/wiki/PTG/Queens/CityGuide | 20:53 |
| *** esberglu has quit IRC | 20:55 | |
| *** esberglu has joined #openstack-infra | 20:55 | |
| jlvillal | clarkb: Thanks! | 20:56 |
| *** gouthamr has quit IRC | 20:56 | |
| *** esberglu has quit IRC | 21:00 | |
| *** baoli has quit IRC | 21:00 | |
| *** rcernin has joined #openstack-infra | 21:06 | |
| *** annegentle has quit IRC | 21:10 | |
| *** camunoz has quit IRC | 21:11 | |
| *** cody-somerville has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:16 | |
| clarkb | before the weekend I'm running backups on ethercalc really quickly to just double check those are happy now | 21:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file containing v2 to v3 mappings https://review.openstack.org/491804 | 21:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: WIP Add npm-upload job https://review.openstack.org/502185 | 21:21 |
| *** felipemonteiro_ has quit IRC | 21:23 | |
| *** gouthamr has joined #openstack-infra | 21:23 | |
| *** Apoorva_ has joined #openstack-infra | 21:26 | |
| *** Apoorva has quit IRC | 21:29 | |
| *** cody-somerville has joined #openstack-infra | 21:30 | |
| *** ijw has joined #openstack-infra | 21:31 | |
| *** thorst has quit IRC | 21:32 | |
| *** marst has quit IRC | 21:32 | |
| *** cody-somerville has quit IRC | 21:35 | |
| *** caphrim007 has joined #openstack-infra | 21:40 | |
| *** caphrim0_ has joined #openstack-infra | 21:41 | |
| mordred | clarkb: http://logs.openstack.org/47/502147/2/check/devstack-legacy-ubuntu-xenial-tempest-dsvm-neutron-full/2bd2f6c/ legacy job with tempest with a green dot | 21:41 |
| *** priteau has quit IRC | 21:42 | |
| *** caphrim007 has quit IRC | 21:44 | |
| *** ijw has quit IRC | 21:45 | |
| *** alee has quit IRC | 21:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update roadmap in the README https://review.openstack.org/502188 | 21:52 |
| *** funzo has joined #openstack-infra | 21:53 | |
| *** xyang1 has quit IRC | 21:53 | |
| pabelanger | clarkb: where did we end up with OVH? Should it be back online? | 21:55 |
| clarkb | pabelanger: change to add it back is approved but I think caught in the backlog | 21:55 |
| pabelanger | clarkb: ah, cool. Did we want to enqueue in to gate? | 21:57 |
| pabelanger | I'm guessing https://review.openstack.org/502140/ | 21:57 |
| *** funzo has quit IRC | 21:58 | |
| clarkb | ok backup on ethercacle seemed to work | 21:58 |
| clarkb | pabelanger: that is fine by me, you may need to give it a positive verified vote valeu though | 21:59 |
| clarkb | enqueue woudln't enqueue it with a negative vote at least, not sure if 0 works | 21:59 |
| pabelanger | clarkb: ack | 22:00 |
| *** xarses_ has quit IRC | 22:09 | |
| *** iyamahat has joined #openstack-infra | 22:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file containing v2 to v3 mappings https://review.openstack.org/491804 | 22:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Revert "Set OVH max-servers to 0""" https://review.openstack.org/502140 | 22:13 |
| *** bobh has quit IRC | 22:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Consume validate_host parameters from site-variables https://review.openstack.org/501242 | 22:15 |
| *** iyamahat has quit IRC | 22:17 | |
| *** iyamahat has joined #openstack-infra | 22:18 | |
| *** rcernin has quit IRC | 22:19 | |
| *** olaph1 has joined #openstack-infra | 22:21 | |
| *** olaph has quit IRC | 22:23 | |
| jeblair | clarkb, mtreinish: in looking at the subunit-file section of the log processor config, i'm getting the sense that what we *really* want is "any subunit files from any gate, periodic, or post job". the filters in there roughly accomplish that, though i think they have some incidental holes (like, there's no reason we don't want grenade periodic results, we probably just forgot to add it). does that sound right? | 22:24 |
| mtreinish | jeblair: I don't think there are grenade periodic results, or at least there weren't when we added that originally | 22:25 |
| mtreinish | but yeah that is the basic idea. We originally had the scope more limited (just tempest jobs) but we stopped that filtering a long time ago | 22:26 |
| *** Apoorva_ has quit IRC | 22:27 | |
| jeblair | okay, i'll use that more general approach for the v3-ification of this for the first pass. we can always add more filtering later if needed, but if we simplify it, we may be able to make it more future-proof and have to tweak it less | 22:27 |
| *** wolverineav has quit IRC | 22:27 | |
| *** Apoorva has joined #openstack-infra | 22:27 | |
| *** wolverineav has joined #openstack-infra | 22:27 | |
| *** xarses_ has joined #openstack-infra | 22:27 | |
| mordred | jeblair: ++ | 22:28 |
| *** xarses_ has quit IRC | 22:31 | |
| *** thorst has joined #openstack-infra | 22:32 | |
| *** xarses_ has joined #openstack-infra | 22:33 | |
| jeblair | mordred: we don't have a great way to share a module between two (roles, playbooks, etc) do we? | 22:34 |
| *** armax has joined #openstack-infra | 22:34 | |
| jeblair | er lemme change channels on that | 22:34 |
| *** thorst has quit IRC | 22:37 | |
| *** dave-mccowan has quit IRC | 22:39 | |
| *** jamesmcarthur has joined #openstack-infra | 22:39 | |
| *** alee has joined #openstack-infra | 22:40 | |
| *** wolverineav has quit IRC | 22:41 | |
| *** jamesmcarthur has quit IRC | 22:44 | |
| *** xarses_ has quit IRC | 22:46 | |
| *** rbrndt has quit IRC | 22:50 | |
| *** rbrndt has joined #openstack-infra | 22:51 | |
| *** rbrndt has quit IRC | 22:51 | |
| *** slaweq has quit IRC | 22:51 | |
| *** rlandy has quit IRC | 22:52 | |
| clarkb | jeblair: the biggest thing is making sure we don't attempt to index things we don't understand in logstash because that reates error records and general slow down | 22:52 |
| clarkb | oh I'm thinking logstash but you are talking subunit | 22:52 |
| *** funzo has joined #openstack-infra | 22:54 | |
| mtreinish | clarkb: well if you pass a non-subunit file to the worker it will just raise an exception (which we made non-fatal) | 22:55 |
| mtreinish | so there shouldn't be any harm | 22:56 |
| mtreinish | the bigger risk is using subunit from something we don't want to be tracking (because of data pollution) | 22:56 |
| clarkb | ya I was confusing the two workes there | 22:57 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Make validate-host read from site-variables https://review.openstack.org/500592 | 22:57 |
| *** caphrim0_ has quit IRC | 22:58 | |
| *** funzo has quit IRC | 22:58 | |
| mtreinish | clarkb: yeah, it just made me think about another edge case | 22:59 |
| openstackgerrit | Merged openstack-infra/puppet-bup master: Don't backup /afs https://review.openstack.org/502103 | 23:00 |
| dmsimard | AJaeger, dirk, hwoarang: We're seeing the same suse devstack failures across different job results recently | 23:01 |
| dmsimard | http://logs.openstack.org/47/502147/2/check/gate-tempest-dsvm-neutron-full-opensuse-423-nv/a30d9c4/logs/devstacklog.txt.gz#_2017-09-08_21_00_40_012 | 23:01 |
| dmsimard | http://logs.openstack.org/47/502147/2/check/devstack-legacy-opensuse-423-tempest-dsvm-neutron-full/1d87625/logs/devstacklog.txt.gz#_2017-09-08_20_39_49_067 | 23:01 |
| dmsimard | tl;dr peakmem_tracker failing to start with subprocess.CalledProcessError: Command '['pmap', '-XX', '1']' returned non-zero exit status 1 | 23:02 |
| pabelanger | dmsimard: keep in mind, opensuse is -nv, so we likely need to do the same for zuulv3 conversion | 23:02 |
| dmsimard | Maybe something related to the recent devstack systemd/screen work ? | 23:02 |
| mtreinish | dmsimard: http://logs.openstack.org/47/502147/2/check/gate-tempest-dsvm-neutron-full-opensuse-423-nv/a30d9c4/logs/screen-peakmem_tracker.txt.gz#_Sep_08_20_49_13_233701 | 23:03 |
| dmsimard | pabelanger: correct, but we need to pay attention to non-votings jobs that start consistently failing when they were consistently passing before, it could be a result of our ongoing work | 23:03 |
| mtreinish | dmsimard: it looks like maybe a different version of that package on suse | 23:03 |
| mtreinish | that uses different args | 23:03 |
| pabelanger | dmsimard: yup | 23:03 |
| dmsimard | mtreinish: yes, I found the man page for that and it didn't have a -X | 23:04 |
| dmsimard | http://www.unix.com/man-page/suse/1/pmap/ | 23:04 |
| mtreinish | dmsimard: screen v systemd wouldn't come into play here, we've been only using systemd in the gate for > 6months | 23:04 |
| mtreinish | the recent change was the deprecated and not default screen mode was removed | 23:04 |
| *** vhosakot has quit IRC | 23:04 | |
| dmsimard | ¯\_(ツ)_/¯ | 23:05 |
| dmsimard | I don't know much about devstack, I just know that this job suddenly started failing :) | 23:05 |
| dmsimard | -XX Show everything the kernel provides | 23:05 |
| dmsimard | doesn't sound like there's an alternative in SUSE's version | 23:05 |
| *** Swami has quit IRC | 23:05 | |
| dmsimard | Looks like it comes from here I guess ? https://github.com/openstack-dev/devstack/blob/ee71ae03f6c7acc014eeb047090fa7d60c493882/tools/mlock_report.py#L29 | 23:06 |
| mtreinish | dmsimard: you can just disable the peakmem_tracker service on suse jobs then | 23:06 |
| dmsimard | do we have facilities in devstack to do one command for a distro and another for another ? | 23:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use 'git checkout' when checking out a tag https://review.openstack.org/502195 | 23:07 |
| mtreinish | there are | 23:07 |
| mtreinish | but that's a standalone python script that devstack runs so the devstack mechanism wouldn't work inside the script | 23:08 |
| dmsimard | Just found this and peakmem_tracker is in there https://github.com/SUSE-Cloud/automation/blob/master/scripts/jenkins/qa_devstack.sh#L114 | 23:08 |
| mtreinish | dmsimard: also that isn't a new thing that line has been there for > 7months | 23:08 |
| dmsimard | mtreinish: odd that it'd start failing now ? | 23:08 |
| dmsimard | mtreinish: the process was started in a screen before right ? would that end up being fatal on failure ? maybe with systemd it's different ? | 23:09 |
| mtreinish | dmsimard: no it's always been in systemd | 23:09 |
| dmsimard | damn, I suck | 23:09 |
| dmsimard | I have no clue then | 23:09 |
| dmsimard | maybe one of the suse experts I pinged know :D | 23:09 |
| mtreinish | running in screen was a manual thing users had to do by setting ENABLE_SCREEN in their local.conf | 23:09 |
| mtreinish | I'm pretty sure none of the gate jobs did that, because screen is super unreliable | 23:10 |
| mtreinish | (for what devstack used it for, not in general) | 23:10 |
| mtreinish | dmsimard: the likely thing is a packaging change on the distro | 23:11 |
| mtreinish | it's a bit late on a fri for me to track through changes on the suse package mirror though :) | 23:11 |
| dmsimard | mtreinish: do you know if it's possible to disable just that without re-declaring the whole enabled services line ? | 23:11 |
| *** hongbin has quit IRC | 23:12 | |
| mtreinish | dmsimard: add 'disable_service peakmem_tracker' to the localrc/local.conf | 23:13 |
| dmsimard | mtreinish: btw totally unrelated but if you're at the PTG we need to cross paths again, we briefly chatted about openstack-health and ara at the last openstack canada :) | 23:14 |
| dmsimard | now that ara is default in zuul v3, it changes a bit the dynamic | 23:14 |
| mtreinish | probably better to do 'disable_service peakmem_tracker memory_tracker' (since peakemem tracker is the deprecated name) | 23:14 |
| dmsimard | ara also gained subunit export in the meantime | 23:14 |
| dmsimard | mtreinish: thanks I'll try that | 23:14 |
| mtreinish | dmsimard: yes I'll be at the ptg | 23:14 |
| *** thorst has joined #openstack-infra | 23:14 | |
| mtreinish | zuul v3 is gonna require a bunch of changes to openstack-health (and subunit2sql) | 23:15 |
| mtreinish | it's something I've been meaning to address, but I haven't had the time | 23:15 |
| mordred | mtreinish: it's an area we should chat about | 23:15 |
| dmsimard | cool, we should definitely chat about it | 23:16 |
| mtreinish | mordred: well the biggest thing is all the metadata is going to change | 23:16 |
| mordred | mtreinish, dmsimard:there's also things we've pondered doing with subunit on the zuul-side - and after not too long there might be opportunities to integrate things lke openstack-health and subunit2sql in some new manner | 23:16 |
| mordred | *waves hands super wildly* | 23:16 |
| mordred | mtreinish: yah | 23:17 |
| dmsimard | mordred: tbh subunit is a pita to work with | 23:17 |
| dmsimard | I don't know how you guys do it | 23:17 |
| dmsimard | the little work I've had to do with it on ara has sucked | 23:17 |
| mtreinish | which is something jeblair gave me fair warning on, I just haven't done anything about it :) | 23:17 |
| mordred | dmsimard: it is - although it also supports multiple intelaced streams | 23:17 |
| mtreinish | dmsimard: yeah I drink heavily | 23:17 |
| *** sdague has quit IRC | 23:18 | |
| dmsimard | I really had to look at the source code to understand what was going on | 23:18 |
| dmsimard | and when I didn't understand anything from the source code, I looked at the unit/integration tests | 23:18 |
| dmsimard | and then I understood /a little bit/ | 23:18 |
| mordred | dmsimard: it's one of the reasons we've been so adamant about it - it opens the door to us actually observing the subunit stream in-flight and telling users about results before the jobs are finished | 23:18 |
| mtreinish | dmsimard: heh, until you look at another example and it comletely messes up your understanding | 23:18 |
| mtreinish | happens to me every other day | 23:18 |
| mtreinish | and i've been working with it for a long time | 23:19 |
| mordred | which has the potential to be very cool - but it also involves writing some code that already makes me head spin | 23:19 |
| *** thorst has quit IRC | 23:19 | |
| dmsimard | mordred: cue Raymond Hettinger "There has to be a better way" | 23:19 |
| * mordred tried to figure out how to interact with an incoming subunit stream embedded inside of a text stream the other day ... | 23:19 | |
| mordred | it did not go well | 23:19 |
| *** tosky has quit IRC | 23:19 | |
| dmsimard | so, anyway, let's catch up about openstack-health, v3 and ara sometime in denver. Good opportunities IMO, let me put an item on the infra pad about it. | 23:20 |
| mordred | dmsimard: ++ | 23:23 |
| pabelanger | infra-root: I'm going to turn down zm05-zm08 shortly, confirm zuulv2.5 works fine, then delete servers shortly after | 23:27 |
| mordred | pabelanger: ++ | 23:28 |
| *** baoli has joined #openstack-infra | 23:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use 'git checkout' when checking out a tag https://review.openstack.org/502195 | 23:29 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: DNM: Test centos-7, fedora-26 and opensuse-423 devstack jobs https://review.openstack.org/502147 | 23:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: DNM: Test centos-7, fedora-26 and opensuse-423 devstack jobs https://review.openstack.org/502147 | 23:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to using stestr https://review.openstack.org/491035 | 23:40 |
| *** cody-somerville has joined #openstack-infra | 23:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to normal tox-py35 job https://review.openstack.org/500330 | 23:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Switch to constraints version of tox job https://review.openstack.org/502196 | 23:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Consume validate_host parameters from site-variables https://review.openstack.org/501242 | 23:46 |
| *** dangers_` is now known as dangers_away | 23:49 | |
| mordred | pabelanger, jeblair, clarkb: can we go ahead and merge this: so that it's available for adding entries to as we add more project-templates and jobs and whatnot? | 23:50 |
| jeblair | mordred: what? | 23:50 |
| mordred | gah | 23:50 |
| mordred | I didn't paste did? | 23:50 |
| mordred | https://review.openstack.org/#/c/491804 - the mapping file change | 23:51 |
| jeblair | mordred: wfm | 23:51 |
| pabelanger | mordred: +2 | 23:53 |
| *** askb has quit IRC | 23:53 | |
| *** funzo has joined #openstack-infra | 23:55 | |
| *** funzo has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!