| *** bobh has joined #openstack-infra | 00:10 | |
| *** bobh has quit IRC | 00:14 | |
| *** bobh has joined #openstack-infra | 00:20 | |
| *** bobh has quit IRC | 00:24 | |
| *** bobh has joined #openstack-infra | 00:38 | |
| *** bobh has quit IRC | 00:39 | |
| *** dingyichen has joined #openstack-infra | 00:39 | |
| *** bobh has joined #openstack-infra | 00:49 | |
| *** bobh has quit IRC | 00:53 | |
| *** bobh has joined #openstack-infra | 01:03 | |
| *** bobh has quit IRC | 01:08 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** bobh has joined #openstack-infra | 01:13 | |
| *** slaweq has quit IRC | 01:16 | |
| *** bobh has quit IRC | 01:17 | |
| *** bobh has joined #openstack-infra | 01:23 | |
| *** bobh has quit IRC | 01:28 | |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:41 | |
| *** bobh has joined #openstack-infra | 01:45 | |
| *** bobh has quit IRC | 01:49 | |
| *** bobh has joined #openstack-infra | 01:52 | |
| *** armax has quit IRC | 01:54 | |
| *** bobh has quit IRC | 01:56 | |
| *** bobh has joined #openstack-infra | 02:03 | |
| *** bobh has quit IRC | 02:07 | |
| *** hongbin has joined #openstack-infra | 02:19 | |
| *** bobh has joined #openstack-infra | 02:24 | |
| *** efried1 has joined #openstack-infra | 02:26 | |
| *** efried has quit IRC | 02:28 | |
| *** efried1 is now known as efried | 02:28 | |
| *** bobh has quit IRC | 02:28 | |
| *** bobh has joined #openstack-infra | 02:31 | |
| *** bobh has quit IRC | 02:36 | |
| *** psachin has joined #openstack-infra | 02:44 | |
| *** rh-jelabarre has quit IRC | 02:45 | |
| *** bobh has joined #openstack-infra | 02:53 | |
| *** bobh has quit IRC | 02:57 | |
| *** bobh has joined #openstack-infra | 02:58 | |
| *** bobh has quit IRC | 03:02 | |
| *** chandankumar has quit IRC | 03:05 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** slaweq has quit IRC | 03:15 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update the Upstream Institute meeting channel https://review.openstack.org/591232 | 03:17 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove Cinder-Nova cross-project meeting https://review.openstack.org/591233 | 03:18 |
| *** bobh has joined #openstack-infra | 03:21 | |
| *** vivsoni_ has quit IRC | 03:22 | |
| *** bobh has quit IRC | 03:25 | |
| *** bobh has joined #openstack-infra | 03:30 | |
| *** bobh has quit IRC | 03:34 | |
| *** bobh has joined #openstack-infra | 03:44 | |
| *** bobh has quit IRC | 03:48 | |
| *** ramishra has joined #openstack-infra | 03:53 | |
| *** ramishra has quit IRC | 03:54 | |
| *** ramishra has joined #openstack-infra | 03:54 | |
| *** udesale has joined #openstack-infra | 03:57 | |
| *** vivsoni has joined #openstack-infra | 04:09 | |
| *** bobh has joined #openstack-infra | 04:14 | |
| *** bobh has quit IRC | 04:18 | |
| *** bobh has joined #openstack-infra | 04:23 | |
| *** dklyle has joined #openstack-infra | 04:23 | |
| *** bobh has quit IRC | 04:27 | |
| *** jaosorior has joined #openstack-infra | 04:28 | |
| *** dave-mccowan has quit IRC | 04:28 | |
| *** bobh has joined #openstack-infra | 04:30 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/591032 , https://review.openstack.org/#/c/591033, please? I like to get these quickly in to avoid conflicts and rebases. thanks! | 04:32 |
| ianw | AJaeger: LGTM, but i'd presume you'd want to watch these anyway because it touches a lot, so will leave for you to +w | 04:34 |
| AJaeger | ianw: ok, will +w. | 04:34 |
| *** bobh has quit IRC | 04:35 | |
| *** jamesmcarthur has joined #openstack-infra | 04:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Setup missing translation jobs for neutron-lbaas-dashboard https://review.openstack.org/590364 | 04:38 |
| *** dklyle has quit IRC | 04:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use translations-jobs-master-stable https://review.openstack.org/591032 | 04:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Use translation-jobs-master-only https://review.openstack.org/591033 | 04:40 |
| *** bobh has joined #openstack-infra | 04:40 | |
| *** jamesmcarthur has quit IRC | 04:42 | |
| AJaeger | ianw: could you also put https://review.openstack.org/591115 and https://review.openstack.org/590045 on your review queue, please? | 04:42 |
| *** bobh has quit IRC | 04:44 | |
| AJaeger | config-core, https://review.openstack.org/591026 and https://review.openstack.org/591027 remove duplicated docs jobs for openstack-zuul-jobs - please review | 04:47 |
| *** viks_ has joined #openstack-infra | 04:47 | |
| *** Bhujay has joined #openstack-infra | 04:50 | |
| *** bobh has joined #openstack-infra | 04:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Setup missing translation jobs for neutron-lbaas-dashboard https://review.openstack.org/590364 | 04:53 |
| openstackgerrit | Merged openstack-infra/project-config master: drop python 3.5 jobs from oslo.service https://review.openstack.org/591115 | 04:53 |
| *** rcernin has quit IRC | 04:54 | |
| *** rcernin has joined #openstack-infra | 04:54 | |
| *** bobh has quit IRC | 04:55 | |
| AJaeger | thanks, ianw ! | 04:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to publish-openstack-docs-pti for cyborg https://review.openstack.org/590650 | 04:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron-fullstack-python35 job to Neutron dashboard https://review.openstack.org/589584 | 04:59 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove double docs job https://review.openstack.org/591026 | 05:02 |
| *** bobh has joined #openstack-infra | 05:02 | |
| *** psachin has quit IRC | 05:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: [n-odl] Make the Y axis display correctly https://review.openstack.org/590275 | 05:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Add readthedocs web hooks for powervm https://review.openstack.org/590444 | 05:05 |
| *** bobh has quit IRC | 05:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Gerritbot: Enable notifications for openstack/openstack-chef https://review.openstack.org/589891 | 05:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Networking-ODL fullstack from grafana https://review.openstack.org/590305 | 05:12 |
| *** bobh has joined #openstack-infra | 05:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: enhance publish-openstack-sphinx-docs description https://review.openstack.org/590808 | 05:17 |
| AJaeger | ianw: updated ^ - thanks for review! | 05:17 |
| *** bobh has quit IRC | 05:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move openstack-zuul-jobs completely in-repo https://review.openstack.org/591027 | 05:18 |
| ianw | AJaeger: thanks, just suggestions :) it's pretty cool we can put full rst type things in there | 05:18 |
| AJaeger | ianw: yeah, it is... | 05:18 |
| *** openstackgerrit has quit IRC | 05:18 | |
| *** hongbin has quit IRC | 05:20 | |
| *** bobh has joined #openstack-infra | 05:21 | |
| *** bobh has quit IRC | 05:25 | |
| *** bobh has joined #openstack-infra | 05:32 | |
| *** bobh has quit IRC | 05:36 | |
| *** bobh has joined #openstack-infra | 05:40 | |
| *** chandankumar has joined #openstack-infra | 05:41 | |
| *** bobh has quit IRC | 05:44 | |
| *** bobh has joined #openstack-infra | 05:52 | |
| *** hwoarang has joined #openstack-infra | 05:53 | |
| *** ykarel has joined #openstack-infra | 05:54 | |
| *** bobh has quit IRC | 05:56 | |
| *** jaosorior has quit IRC | 05:59 | |
| *** janki has joined #openstack-infra | 05:59 | |
| *** pcaruana has joined #openstack-infra | 05:59 | |
| *** bobh has joined #openstack-infra | 06:00 | |
| *** dpawlik has joined #openstack-infra | 06:00 | |
| *** openstackgerrit has joined #openstack-infra | 06:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable add_host for trusted play and persist inventory https://review.openstack.org/590092 | 06:00 |
| *** mschuppert has joined #openstack-infra | 06:04 | |
| *** bobh has quit IRC | 06:04 | |
| *** dpawlik has quit IRC | 06:05 | |
| *** pcaruana has quit IRC | 06:05 | |
| *** dpawlik has joined #openstack-infra | 06:12 | |
| *** pcaruana has joined #openstack-infra | 06:19 | |
| *** bobh has joined #openstack-infra | 06:23 | |
| *** AJaeger has quit IRC | 06:23 | |
| *** ykarel_ has joined #openstack-infra | 06:25 | |
| *** udesale has quit IRC | 06:25 | |
| *** ykarel has quit IRC | 06:27 | |
| *** bobh has quit IRC | 06:28 | |
| ianw | pabelanger: can you double check https://review.openstack.org/#/c/586236 ... that & follow-on will clean up all the extra system-config puppet cruft for nodepool.o.o | 06:28 |
| *** ykarel__ has joined #openstack-infra | 06:33 | |
| *** alexchadin has joined #openstack-infra | 06:33 | |
| *** bobh has joined #openstack-infra | 06:34 | |
| *** AJaeger has joined #openstack-infra | 06:35 | |
| *** ykarel_ has quit IRC | 06:35 | |
| *** slaweq has joined #openstack-infra | 06:37 | |
| *** bobh has quit IRC | 06:38 | |
| *** bobh has joined #openstack-infra | 06:40 | |
| *** ykarel__ is now known as ykarel | 06:41 | |
| *** bobh has quit IRC | 06:44 | |
| openstackgerrit | Merged openstack-infra/system-config master: modules: mirror: Mirror the openSUSE Cloud Master repository https://review.openstack.org/589105 | 06:47 |
| AJaeger | infra-root, I see retry-limit on the periodic wheel builds: http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-centos-7/bc767ff/ and http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-bionic/b478c96/ | 06:48 |
| openstackgerrit | Merged openstack-dev/pbr master: import zuul job settings from project-config https://review.openstack.org/588700 | 06:48 |
| AJaeger | infra-root, I seee "legacy-install-afs-with-puppet" problem... - ianw has changes up for those AFAIK | 06:49 |
| *** bobh has joined #openstack-infra | 06:55 | |
| *** ssbarnea has joined #openstack-infra | 06:55 | |
| *** bobh has quit IRC | 07:00 | |
| ianw | AJaeger: yeah, i think that moving to those roles to install the AFS client is the best way forward | 07:00 |
| ianw | i'll try to sync with mordred to tomorrow on https://review.openstack.org/#/c/586236/2 and page in more of this new stuff i haven't quite kept up with | 07:01 |
| *** mpeterson has quit IRC | 07:01 | |
| *** udesale has joined #openstack-infra | 07:02 | |
| *** rcernin has quit IRC | 07:03 | |
| *** apetrich has joined #openstack-infra | 07:04 | |
| AJaeger | thanks, ianw | 07:04 |
| *** jpich has joined #openstack-infra | 07:07 | |
| *** evrardjp has joined #openstack-infra | 07:08 | |
| cmurphy | i think gerrit might want a restart http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 07:10 |
| *** psachin has joined #openstack-infra | 07:14 | |
| ianw | cmurphy: hrm, java seems to be okish "Java memory used: 18,906 Mb / 43,691 Mb" | 07:14 |
| ianw | are you seeing 500s? | 07:15 |
| *** bobh has joined #openstack-infra | 07:15 | |
| cmurphy | ianw: no it's just really slow for me | 07:15 |
| cmurphy | maybe it's just me | 07:15 |
| *** amoralej|off is now known as amoralej | 07:16 | |
| cmurphy | ianw: hmm i think i might just have been having ipv6 issues, so just me | 07:18 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Clean Neutron failure rates Grafana dashboard https://review.openstack.org/590239 | 07:18 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add graphs with overall number of jobs run https://review.openstack.org/591253 | 07:18 |
| *** mpeterson has joined #openstack-infra | 07:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Translation cleanup https://review.openstack.org/591280 | 07:19 |
| *** bobh has quit IRC | 07:19 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 07:20 |
| *** bobh has joined #openstack-infra | 07:21 | |
| *** bobh has quit IRC | 07:25 | |
| *** mpeterson has quit IRC | 07:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-website master: Link to Zuul Sphinx extension https://review.openstack.org/591282 | 07:29 |
| *** alexchadin has quit IRC | 07:31 | |
| *** mpeterson has joined #openstack-infra | 07:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Cleanup translation templates https://review.openstack.org/591283 | 07:34 |
| AJaeger | ianw: translation changes look fine, now we can cleanup - https://review.openstack.org/591280 and https://review.openstack.org/591283 (reviews welcome but no urgency) | 07:34 |
| *** alexchadin has joined #openstack-infra | 07:35 | |
| *** holser_ has joined #openstack-infra | 07:37 | |
| *** ccamacho has joined #openstack-infra | 07:40 | |
| *** rpittau has joined #openstack-infra | 07:43 | |
| *** pcaruana has quit IRC | 07:43 | |
| *** olivierbourdon38 has joined #openstack-infra | 07:44 | |
| *** jaosorior has joined #openstack-infra | 07:46 | |
| *** bobh has joined #openstack-infra | 07:48 | |
| *** holser_ has quit IRC | 07:52 | |
| *** bobh has quit IRC | 07:53 | |
| *** holser_ has joined #openstack-infra | 07:54 | |
| *** jpena|off is now known as jpena | 07:55 | |
| *** pcaruana has joined #openstack-infra | 07:57 | |
| *** slaweq_ has joined #openstack-infra | 07:57 | |
| *** slaweq has quit IRC | 07:59 | |
| *** ykarel is now known as ykarel|lunch | 07:59 | |
| *** abregman has joined #openstack-infra | 08:13 | |
| abregman | hello | 08:14 |
| abregman | can anyone add me to "tobiko-core" and "tobiko-release" groups? | 08:14 |
| *** psachin has quit IRC | 08:17 | |
| *** lucasagomes has joined #openstack-infra | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:20 | |
| *** psachin has joined #openstack-infra | 08:20 | |
| *** bobh has joined #openstack-infra | 08:29 | |
| *** jaosorior_ has joined #openstack-infra | 08:36 | |
| *** derekh has joined #openstack-infra | 08:36 | |
| *** jamesmcarthur has joined #openstack-infra | 08:38 | |
| ianw | abregman: done | 08:38 |
| abregman | ianw: thank you! :) | 08:38 |
| *** jaosorior has quit IRC | 08:40 | |
| *** alexchadin has quit IRC | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:42 | |
| *** alexchadin has joined #openstack-infra | 08:43 | |
| *** dtantsur|afk is now known as dtantsur | 08:49 | |
| *** ykarel|lunch is now known as ykarel | 08:56 | |
| *** dingyichen has quit IRC | 09:03 | |
| *** slaweq_ is now known as slaweq | 09:05 | |
| *** gary_perkins has joined #openstack-infra | 09:08 | |
| *** gfidente has joined #openstack-infra | 09:11 | |
| *** alexchadin has quit IRC | 09:12 | |
| *** ccamacho has quit IRC | 09:21 | |
| *** gfidente has quit IRC | 09:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Clean Neutron failure rates Grafana dashboard https://review.openstack.org/590239 | 09:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Add graphs with overall number of jobs run https://review.openstack.org/591253 | 09:32 |
| *** electrofelix has joined #openstack-infra | 09:33 | |
| *** alexchadin has joined #openstack-infra | 09:33 | |
| *** rpittau has quit IRC | 09:36 | |
| *** rpittau has joined #openstack-infra | 09:36 | |
| *** ccamacho has joined #openstack-infra | 09:39 | |
| *** jaosorior_ is now known as jaosorior | 09:39 | |
| *** apetrich has quit IRC | 09:47 | |
| *** tosky has joined #openstack-infra | 09:59 | |
| *** jaosorior has quit IRC | 10:09 | |
| *** roman_g_ has joined #openstack-infra | 10:10 | |
| *** jaosorior has joined #openstack-infra | 10:11 | |
| *** panda is now known as panda|ruck | 10:15 | |
| *** tosky has quit IRC | 10:19 | |
| *** eumel8 has joined #openstack-infra | 10:19 | |
| *** ccamacho has quit IRC | 10:31 | |
| *** ccamacho has joined #openstack-infra | 10:32 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Fix reject clauses in the absence of approvals https://review.openstack.org/589762 | 10:34 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Fix reject clauses in the absence of approvals https://review.openstack.org/589762 | 10:37 |
| *** gfidente has joined #openstack-infra | 10:37 | |
| *** ykarel is now known as ykarel|afk | 10:38 | |
| *** gouthamr has quit IRC | 10:44 | |
| *** dmellado has quit IRC | 10:46 | |
| *** stevebaker has quit IRC | 10:46 | |
| *** alexchadin has quit IRC | 10:53 | |
| *** holser_ has quit IRC | 11:03 | |
| *** alexchadin has joined #openstack-infra | 11:03 | |
| *** sambetts has joined #openstack-infra | 11:03 | |
| *** sambetts is now known as sambetts|afk | 11:04 | |
| *** sshnaidm is now known as sshnaidm|off | 11:04 | |
| *** slaweq_ has joined #openstack-infra | 11:06 | |
| *** slaweq has quit IRC | 11:06 | |
| *** jpena is now known as jpena|lunch | 11:14 | |
| *** alexchadin has quit IRC | 11:17 | |
| *** alexchadin has joined #openstack-infra | 11:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy scenario multinode job for nova https://review.openstack.org/583849 | 11:21 |
| *** slagle has joined #openstack-infra | 11:22 | |
| *** panda|ruck is now known as panda|ruck|lunch | 11:23 | |
| *** udesale has quit IRC | 11:27 | |
| *** amoralej is now known as amoralej|lunch | 11:32 | |
| *** rosmaita has joined #openstack-infra | 11:34 | |
| *** mriedem has joined #openstack-infra | 11:36 | |
| *** alexchadin has quit IRC | 11:36 | |
| *** alexchadin has joined #openstack-infra | 11:38 | |
| *** jamesmcarthur has joined #openstack-infra | 11:38 | |
| *** tosky has joined #openstack-infra | 11:41 | |
| *** jamesmcarthur has quit IRC | 11:42 | |
| *** rh-jelabarre has joined #openstack-infra | 11:43 | |
| *** holser_ has joined #openstack-infra | 11:44 | |
| *** sambetts|afk is now known as sambetts | 11:45 | |
| *** ykarel|afk is now known as ykarel | 11:45 | |
| sambetts | Hey Infra, have you ever seen nodepool get a node to Ready state then get stuck? In my third party CI I'm seeing nodepool get nodes all the way through to ready state then they remain locked and never get given out to jobs that need them | 11:51 |
| *** dave-mccowan has joined #openstack-infra | 11:51 | |
| *** alexchadin has quit IRC | 11:53 | |
| sambetts | restarting the nodepool-launcher service I see all the nodes become unlocked and suddenly start being handled again | 12:02 |
| *** panda|ruck|lunch is now known as panda|ruck | 12:02 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 12:09 |
| *** rpittau is now known as rpittau|afk | 12:10 | |
| *** rpittau|afk is now known as rpittau | 12:10 | |
| *** jpena|lunch is now known as jpena | 12:13 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 12:14 |
| *** jaosorior has quit IRC | 12:18 | |
| *** jaosorior has joined #openstack-infra | 12:19 | |
| *** bobh has quit IRC | 12:20 | |
| *** bobh has joined #openstack-infra | 12:21 | |
| *** amoralej|lunch is now known as amoralej | 12:25 | |
| *** boden has joined #openstack-infra | 12:25 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add tempest-scenario-job to Neutron failure rate dashboard https://review.openstack.org/583870 | 12:28 |
| *** jcoufal has joined #openstack-infra | 12:30 | |
| *** apetrich has joined #openstack-infra | 12:31 | |
| *** jistr is now known as jistr|call | 12:32 | |
| *** trown|outtypewww is now known as trown | 12:32 | |
| Shrews | sambetts: nodepool version? | 12:32 |
| sambetts | Shrews: 3.2.1 | 12:32 |
| *** tosky has quit IRC | 12:35 | |
| Shrews | 3.2.1 does not exist. are you running from master branch? | 12:35 |
| sambetts | Shrews: 3.2.1 is what nodepool --version just have me, I think ansible-role-nodepool installed from the git repo | 12:36 |
| sambetts | s/have/gave | 12:36 |
| Shrews | sambetts: you must be running an unreleased version from git. either way, we haven't seen problems like that for a while (some bugs in earlier versions that could have caused that) | 12:38 |
| Shrews | sambetts: there should be enough info in the launcher logs to figure out what is happening | 12:39 |
| sambetts | Shrews: thats whats kind of weird there were no errors in the laucher logs, I did see one in the launcher journal logs which seemed to be a kazoo communication error | 12:40 |
| sambetts | but zookeeper looks fine and is running on the same server :/ | 12:40 |
| *** jcoufal has quit IRC | 12:41 | |
| Shrews | sambetts: did it resume operations after the zk error? zk errors happen on occasion, especially if you only have a 1 node cluster | 12:41 |
| sambetts | Shrews: I can only assume not, and thats why the nodes where stuck in "locked" | 12:42 |
| *** jcoufal has joined #openstack-infra | 12:42 | |
| sambetts | Shrews: although it was able to continue making other VMs | 12:42 |
| sambetts | for other jobs | 12:42 |
| Shrews | sambetts: well that's normal communication (only a single zk connection per launcher) | 12:43 |
| Shrews | sambetts: you don't want to just look for errors. look to see if the launcher is "paused" at any point | 12:44 |
| Shrews | message is something like "Pausing request handling to handle..." | 12:44 |
| Shrews | if it never recovers from the pause, then that's the thing you'd need to track down | 12:45 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 12:45 |
| *** nicolasbock has joined #openstack-infra | 12:48 | |
| sambetts | Shrews: its kind of weird, basically the ones that were locked were stuck but new requests were able to continue it just never went back to cleanup/unlock the previously created ones | 12:49 |
| *** dmellado has joined #openstack-infra | 12:50 | |
| sambetts | Shrews: and because of the request to zuul job preallocated that seems to happen even though nodepool was able to create other ready nodes ok they didn't get given to the zuul jobs in the queue so were basically deadlocked against a locked nodepool node | 12:50 |
| Shrews | sambetts: are you sure zuul didn't have them locked? | 12:53 |
| sambetts | Shrews: the requests were never satified in the nodepool request-list and the nodes never went into in-use | 12:54 |
| sambetts | Shrews: as soon as I restarted the nodepool launcher all the locked nodes unlocked and nodepool was able to satify the requests and zuul picked up up from where it was | 12:55 |
| Shrews | sambetts: we have seen cases where zuul would be assigned nodes properly, but zuul never changed them to in-use or removed the request | 12:55 |
| Shrews | ah, well if you didn't restart zuul, problem is in the launcher | 12:56 |
| Shrews | not sure what that problem would be though | 12:56 |
| *** hwoarang has quit IRC | 12:56 | |
| *** psachin has quit IRC | 12:56 | |
| sambetts | yeah it was super strange, I'll try to keep and eye out for it to do it again | 12:57 |
| *** jamesmcarthur has joined #openstack-infra | 12:57 | |
| sambetts | this was the second time though, first time I thought it was a zuul issue because the queue was stuck and I could see ready nodes, so I ended up restarting zuul first and the nodepool so I wasn't sure which it was | 12:59 |
| sambetts | so this time I double checked nodepool first | 12:59 |
| *** hwoarang has joined #openstack-infra | 12:59 | |
| *** jroll has quit IRC | 12:59 | |
| *** jroll has joined #openstack-infra | 13:00 | |
| *** mriedem has quit IRC | 13:03 | |
| *** jistr|call is now known as jistr | 13:04 | |
| *** ramishra has quit IRC | 13:04 | |
| *** stevebaker has joined #openstack-infra | 13:05 | |
| *** eharney has joined #openstack-infra | 13:05 | |
| *** bobh has quit IRC | 13:06 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 13:08 |
| *** bobh has joined #openstack-infra | 13:09 | |
| *** nhicher has joined #openstack-infra | 13:11 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 13:11 |
| *** alexchadin has joined #openstack-infra | 13:12 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 13:14 |
| *** bobh has quit IRC | 13:14 | |
| *** bobh has joined #openstack-infra | 13:23 | |
| *** bobh has quit IRC | 13:25 | |
| *** SteelyDan is now known as dansmith | 13:25 | |
| *** bobh has joined #openstack-infra | 13:25 | |
| *** bobh has quit IRC | 13:28 | |
| *** bobh has joined #openstack-infra | 13:29 | |
| *** bobh has quit IRC | 13:34 | |
| *** bobh has joined #openstack-infra | 13:34 | |
| *** russellb has joined #openstack-infra | 13:36 | |
| *** bobh has quit IRC | 13:39 | |
| *** roman_g_ has quit IRC | 13:45 | |
| *** roman_g_ has joined #openstack-infra | 13:46 | |
| *** ramishra has joined #openstack-infra | 13:48 | |
| *** abregman has quit IRC | 13:50 | |
| *** eharney has quit IRC | 13:51 | |
| *** pbourke has quit IRC | 13:52 | |
| *** pbourke has joined #openstack-infra | 13:53 | |
| *** bobh has joined #openstack-infra | 13:59 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: [DNM] Testing CentOS images builds on Ubuntu Xenial https://review.openstack.org/591366 | 13:59 |
| *** bobh has quit IRC | 14:04 | |
| *** eharney has joined #openstack-infra | 14:06 | |
| *** bobh has joined #openstack-infra | 14:08 | |
| *** gema has quit IRC | 14:08 | |
| *** bnemec-pto is now known as bnemec | 14:10 | |
| *** bobh has quit IRC | 14:12 | |
| *** bobh has joined #openstack-infra | 14:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add exim role to base playbook https://review.openstack.org/589964 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove bridge from disabled and add puppet group https://review.openstack.org/591150 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Stop running puppet from puppetmaster https://review.openstack.org/591151 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 14:23 |
| *** janki has quit IRC | 14:24 | |
| *** janki has joined #openstack-infra | 14:24 | |
| *** bobh has quit IRC | 14:26 | |
| *** Bhujay has quit IRC | 14:27 | |
| *** alexchadin has quit IRC | 14:30 | |
| *** bobh has joined #openstack-infra | 14:30 | |
| *** ykarel is now known as ykarel|away | 14:30 | |
| *** alexchadin has joined #openstack-infra | 14:30 | |
| *** armax has joined #openstack-infra | 14:34 | |
| *** bobh has quit IRC | 14:35 | |
| *** ykarel|away has quit IRC | 14:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set fedora-27 min-ready to 0 https://review.openstack.org/591405 | 14:38 |
| pabelanger | config-core: preparing for fedora-27 removal^ | 14:38 |
| pabelanger | if all goes well, should be able to remove the images this afternoon | 14:39 |
| *** bobh has joined #openstack-infra | 14:40 | |
| *** bobh has quit IRC | 14:44 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/591280/ , https://review.openstack.org/591283 , and https://review.openstack.org/590808 , please? | 14:49 |
| AJaeger | pabelanger: https://review.openstack.org/#/c/588378/ needs to merge first, doesn't it? Are all dependencies in? | 14:49 |
| *** ccamacho has quit IRC | 14:50 | |
| pabelanger | AJaeger: not first, we'll still launch fedora-27, just not keep any online. Hoping that will land after lunch | 14:51 |
| AJaeger | pabelanger: ok | 14:52 |
| *** bobh has joined #openstack-infra | 14:54 | |
| *** ccamacho has joined #openstack-infra | 14:56 | |
| *** rpioso|afk is now known as rpioso | 14:57 | |
| sambetts | Shrews: it seems to have happened again during a CleanupWorker run there was a kazoo ConnectionLoss error which then seems to have cascaded resulting in a bunch of nodes stuck in locked, unsure if its related to https://review.openstack.org/#/c/589854/ | 14:58 |
| *** bobh has quit IRC | 14:59 | |
| *** bobh_ has joined #openstack-infra | 14:59 | |
| *** bobh_ has quit IRC | 15:03 | |
| *** alexchadin has quit IRC | 15:04 | |
| *** bobh has joined #openstack-infra | 15:10 | |
| *** bobh has quit IRC | 15:14 | |
| sambetts | Shrews: checking out the zookeeper logs it looks like it might be a disk I/O issue causing the ZK sessions to timeout, with zuul rsyncing the required repos to every job I guess the disk is getting hammered | 15:16 |
| *** Goneri has joined #openstack-infra | 15:17 | |
| *** janki has quit IRC | 15:19 | |
| Shrews | sambetts: zk can be very i/o intensive. SpamapS helped us find that it really needs to be on a node by itself | 15:19 |
| sambetts | Shrews: yeah I'm thinking I may need to do the same or at least give it its own disk separate from the one the zuul-executor is hammering | 15:20 |
| Shrews | that could explain some of the behavior you're seeing. long periods of inactivity in the launcher | 15:20 |
| *** bobh has joined #openstack-infra | 15:21 | |
| SpamapS | Yeah it has to be on a node by itself with plenty of IOPS | 15:21 |
| sambetts | Shrews: yeah, it doesn't look like nodepool recovers very well from zk outages | 15:22 |
| SpamapS | sambetts: SSD recommended. ;) | 15:22 |
| *** signed8bit has joined #openstack-infra | 15:22 | |
| SpamapS | sambetts: check your zookeeper logs. If it is warning about checkpoints taking a long time, that's your clue. | 15:22 |
| Shrews | sambetts: it *should* recover from zk outages just fine. but if it can't communicate with zk, it can't recover from that | 15:22 |
| Shrews | or if communication is severly delayed | 15:23 |
| sambetts | SpamapS: I'm seeing "fsync-ing the write ahead log in SyncThread took..." logs | 15:23 |
| *** holser_ has quit IRC | 15:24 | |
| sambetts | which seem to point to the same thing from looking at the troubleshooting guides | 15:24 |
| pabelanger | sambetts: this is all in one install? | 15:25 |
| *** bobh has quit IRC | 15:26 | |
| sambetts | pabelanger: yeah all running on one baremetal server | 15:26 |
| pabelanger | sambetts: once suggestion corvus said to me, but haven't switched back to test, is if you are doing all in one setup. Don't run the zuul-merger daemon, might see a little better performance if you stop it and have only zuul-executor doing the merges. IIRC | 15:27 |
| pabelanger | but also, +1 for moving zuul-executor to its own disk. very IO heavy | 15:27 |
| sambetts | cool I'll try out tuning zookeeper and see if that makes things a little more stable | 15:32 |
| mordred | corvus, clarkb, pabelanger: https://review.openstack.org/#/c/589964 is ready for review (and is green in the tests) | 15:33 |
| *** pcaruana has quit IRC | 15:37 | |
| *** bobh has joined #openstack-infra | 15:38 | |
| SpamapS | sambetts: yep that's the one. | 15:42 |
| *** bobh has quit IRC | 15:42 | |
| SpamapS | and yeah, I'd say you want some dedicated IOPS for zk | 15:42 |
| SpamapS | and if executor is fighting with zk, that will definitely hurt things | 15:43 |
| SpamapS | git pounds on the disk a bit | 15:43 |
| SpamapS | especially with big merges | 15:43 |
| clarkb | I'm guessing that the newer kernel performance boost we saw on executors was related to ^ and costs of page table isolation coming down | 15:44 |
| clarkb | I wouldn't publish a document claiming tat but its a good guess I think :P | 15:44 |
| *** olivierbourdon38 has quit IRC | 15:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Translation cleanup https://review.openstack.org/591280 | 15:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable add_host for trusted play and update inventory https://review.openstack.org/590092 | 15:47 |
| *** bobh has joined #openstack-infra | 15:48 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: enhance publish-openstack-sphinx-docs description https://review.openstack.org/590808 | 15:52 |
| *** bobh has quit IRC | 15:53 | |
| *** gouthamr has joined #openstack-infra | 15:54 | |
| *** signed8b_ has joined #openstack-infra | 15:54 | |
| pabelanger | clarkb: mordred: mind a review on https://review.openstack.org/591405/ to start process of removing fedora-27 from nodepool. We have no projects using it now | 15:56 |
| *** anticw has quit IRC | 15:56 | |
| *** signed8bit has quit IRC | 15:56 | |
| clarkb | hrm my vim decided to magically start using folds in rst docs for me | 16:00 |
| openstackgerrit | Merged openstack-infra/system-config master: Add exim role to base playbook https://review.openstack.org/589964 | 16:00 |
| clarkb | not sure I'm a fan | 16:00 |
| *** boden has quit IRC | 16:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove fedora-27 from nodepool https://review.openstack.org/591442 | 16:02 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove (legacy-)fedora-27 nodesets / jobs https://review.openstack.org/588378 | 16:02 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Flip fedora-latest integration tests voting https://review.openstack.org/590579 | 16:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 16:03 |
| *** e0ne has quit IRC | 16:03 | |
| *** bobh has joined #openstack-infra | 16:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Set fedora-27 min-ready to 0 https://review.openstack.org/591405 | 16:07 |
| AJaeger | config-core, could you review https://review.openstack.org/590642 , https://review.openstack.org/590641 and https://review.openstack.org/581278 , please? | 16:07 |
| *** bobh has quit IRC | 16:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Disable password auth on dib images https://review.openstack.org/591446 | 16:09 |
| *** boden has joined #openstack-infra | 16:09 | |
| *** dklyle has joined #openstack-infra | 16:10 | |
| *** eharney has quit IRC | 16:11 | |
| *** bobh has joined #openstack-infra | 16:12 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Cleanup translation templates https://review.openstack.org/591283 | 16:13 |
| *** slaweq_ has quit IRC | 16:15 | |
| *** bobh has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix web content copying in multi dashboard job https://review.openstack.org/591089 | 16:17 |
| openstackgerrit | Merged openstack-infra/project-config master: base-test: Validate test-emit-job-header work https://review.openstack.org/557948 | 16:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - base-test https://review.openstack.org/591453 | 16:22 |
| *** gyee has joined #openstack-infra | 16:23 | |
| clarkb | corvus: for https://review.openstack.org/#/c/581278/2/tox.ini do y ouwant to keep that there for future role tests like with the swift upload role? | 16:24 |
| *** eharney has joined #openstack-infra | 16:24 | |
| *** bobh has joined #openstack-infra | 16:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable password auth on dib images https://review.openstack.org/591446 | 16:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - base-test https://review.openstack.org/591453 | 16:24 |
| *** sambetts is now known as sambetts|afk | 16:26 | |
| *** olivierbourdon38 has joined #openstack-infra | 16:26 | |
| openstackgerrit | Merged openstack-dev/pbr master: switch documentation job to new PTI https://review.openstack.org/588701 | 16:27 |
| *** e0ne has joined #openstack-infra | 16:28 | |
| *** bobh has quit IRC | 16:29 | |
| *** bobh has joined #openstack-infra | 16:30 | |
| *** ccamacho has quit IRC | 16:31 | |
| *** jpich has quit IRC | 16:31 | |
| *** slaweq_ has joined #openstack-infra | 16:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: linters: Take a guess at the roles path https://review.openstack.org/590642 | 16:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Bump ansible for linters https://review.openstack.org/590641 | 16:33 |
| *** jpena is now known as jpena|off | 16:34 | |
| *** bobh has quit IRC | 16:34 | |
| corvus | clarkb: nah, we can add it back if we do | 16:36 |
| *** slaweq_ has quit IRC | 16:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Promote test-emit-job-header change https://review.openstack.org/591457 | 16:39 |
| *** pcaruana has joined #openstack-infra | 16:40 | |
| *** bobh has joined #openstack-infra | 16:41 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove testr https://review.openstack.org/581278 | 16:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "base-test: Validate test-emit-job-header work" https://review.openstack.org/557949 | 16:43 |
| *** masayukig has quit IRC | 16:43 | |
| *** bobh has quit IRC | 16:45 | |
| openstackgerrit | Han-Wen NIenhuys proposed openstack-infra/git-review master: test https://review.openstack.org/591458 | 16:48 |
| *** bobh has joined #openstack-infra | 16:50 | |
| *** slaweq_ has joined #openstack-infra | 16:52 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/zuul master: Remove non-working html code in div container https://review.openstack.org/591459 | 16:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove references to long-gone odsreg.o.o service https://review.openstack.org/589580 | 16:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 16:53 |
| openstackgerrit | Han-Wen NIenhuys proposed openstack-infra/git-review master: Remove support for outdated gerrit versions https://review.openstack.org/591460 | 16:54 |
| *** bobh has quit IRC | 16:54 | |
| *** slaweq_ has quit IRC | 16:57 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| AJaeger | cmurphy: is it safe to merge https://review.openstack.org/590142 now? The dependencies are in... | 17:00 |
| *** derekh has quit IRC | 17:01 | |
| *** bobh has joined #openstack-infra | 17:01 | |
| *** e0ne has quit IRC | 17:01 | |
| AJaeger | dmsimard: do you want to abandon https://review.openstack.org/571919 ? Please check again... | 17:01 |
| AJaeger | corvus: https://review.openstack.org/587178 is "DNM: test logs in swift" - shall I abandon or do you still need this? | 17:02 |
| cmurphy | AJaeger: yes it should be safe now | 17:02 |
| corvus | AJaeger: still using it | 17:03 |
| cmurphy | i guess i could propose a canary patch that depends-on it to make certain | 17:03 |
| corvus | i like the term 'canary patch' | 17:03 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: DNM Test https://review.openstack.org/590142 https://review.openstack.org/591462 | 17:04 |
| AJaeger | corvus: ok | 17:05 |
| *** bobh has quit IRC | 17:06 | |
| AJaeger | cmurphy: thanks. please tell us once it's fine to merge your change and I'll happily +A... | 17:06 |
| cmurphy | AJaeger: will do | 17:06 |
| *** trown is now known as trown|lunch | 17:07 | |
| AJaeger | clarkb: want to +A https://review.openstack.org/#/c/563790 now? and then https://review.openstack.org/#/c/563702/ ? | 17:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Add release jobs to ansible-role-tripleo-* https://review.openstack.org/590045 | 17:08 |
| clarkb | AJaeger: looking | 17:09 |
| clarkb | cmurphy: did we reach a conclusion on the symlink thing? | 17:09 |
| cmurphy | clarkb: i've been trying to test it on trusty and running into a different problem that i don't understand yet | 17:09 |
| clarkb | AJaeger: we should get corvus' input on the first one. I'm happy to merge it from a technical standpoint but not sure if corvus is ready from a user reconsumption of zuul-jobs standpoint | 17:10 |
| pabelanger | config-core: https://review.openstack.org/591442/ is ready to land now, we no longer have jobs using fedora-27, and 0 nodes online | 17:10 |
| AJaeger | clarkb: your call - please self-approve when ready... | 17:10 |
| *** slaweq_ has joined #openstack-infra | 17:11 | |
| *** yamahata has quit IRC | 17:11 | |
| clarkb | corvus: https://review.openstack.org/#/c/563790/1 can you indicate if you think we are ready for that one from a deprecation standpoint? | 17:12 |
| cmurphy | clarkb: seeing this bug https://github.com/nodemailer/nodemailer/issues/754 but not sure what's different about my env that's not in production | 17:12 |
| AJaeger | pabelanger: what about http://git.openstack.org/cgit/openstack-infra/nodepool/tree/devstack/plugin.sh#n248 ? | 17:12 |
| clarkb | cmurphy: we use nodejs 4.something | 17:12 |
| AJaeger | pabelanger: and similar places - I think that is not a problem, correct? | 17:13 |
| clarkb | cmurphy: possibly that or we are using an older version of nodemailer that was pulled in when we installed? | 17:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Remove legacy-centos-7-4-node nodeset https://review.openstack.org/591464 | 17:13 |
| clarkb | AJaeger: correct, that is for nodepool running against devstack in test nodes for testing | 17:14 |
| pabelanger | AJaeger: those are fine, we are actually building fedora-27 for nodepool dsvm jobs | 17:14 |
| *** bobh has joined #openstack-infra | 17:14 | |
| clarkb | we should clean those up though | 17:14 |
| pabelanger | I have a patch up to replace them with fedora-28 | 17:14 |
| clarkb | and maybe got to fedora latest there too | 17:14 |
| AJaeger | great | 17:14 |
| pabelanger | clarkb: we don't run nodepool on fedora-latest, just make sure we can build them | 17:14 |
| pabelanger | AJaeger: clarkb: https://review.openstack.org/559211/ | 17:15 |
| clarkb | pabelanger: yes I know | 17:15 |
| cmurphy | clarkb: the nodejs version that gets installed in the tests is 4.9.1 and the nodemailer version is nodemailer@4.6.7 | 17:15 |
| clarkb | pabelanger: I'm saying if we aren't using fedora-27 elsewhere maybe stop testing we can build and boot them | 17:15 |
| pabelanger | clarkb: agree, was to follow up for that after 559211 | 17:15 |
| *** slaweq_ has quit IRC | 17:15 | |
| clarkb | cmurphy: nodemailer@2.7.0 and node == v4.8.7 | 17:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-4-node nodeset https://review.openstack.org/591464 | 17:17 |
| clarkb | I'm trying to remember if we can delete the diskimages section when we remove the provider config. Removing from the provider list causes it to delete the image from the provider which is what we want | 17:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-3-node nodeset https://review.openstack.org/591465 | 17:18 |
| *** bobh has quit IRC | 17:18 | |
| clarkb | does removing it from diskimages list cause it to delete the diskimages as well? I guess we will find out? | 17:18 |
| * clarkb goes with lets find out :) | 17:18 | |
| *** openstackgerrit has quit IRC | 17:19 | |
| *** NobodyCam has quit IRC | 17:19 | |
| *** NobodyCam has joined #openstack-infra | 17:19 | |
| *** openstackgerrit has joined #openstack-infra | 17:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-2-node nodeset https://review.openstack.org/591466 | 17:20 |
| *** bobh has joined #openstack-infra | 17:20 | |
| corvus | clarkb, AJaeger: i think 563790 is okay to merge now, +3 | 17:22 |
| clarkb | corvus: thanks! | 17:22 |
| corvus | i calculated the date of merge for https://review.openstack.org/578238 and left a comment | 17:22 |
| *** bobh has quit IRC | 17:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove centos-7-3-node / centos-7-4-node https://review.openstack.org/591468 | 17:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove fedora-27 from nodepool https://review.openstack.org/591442 | 17:29 |
| clarkb | pabelanger: ^ | 17:29 |
| *** jamesmcarthur has quit IRC | 17:31 | |
| *** eharney has quit IRC | 17:31 | |
| *** eandersson has quit IRC | 17:32 | |
| *** slaweq_ has joined #openstack-infra | 17:32 | |
| *** jamesmcarthur has joined #openstack-infra | 17:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP Extract pep8 messages for inline comments https://review.openstack.org/589634 | 17:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: DNM: test pep8 failures https://review.openstack.org/591469 | 17:34 |
| *** bgmccollum has quit IRC | 17:37 | |
| cmurphy | AJaeger: https://review.openstack.org/590142 looks safe according to https://review.openstack.org/591462 | 17:37 |
| *** jamesmcarthur has quit IRC | 17:37 | |
| *** slaweq_ has quit IRC | 17:37 | |
| *** ianychoi_ has joined #openstack-infra | 17:38 | |
| pabelanger | cmurphy: ack | 17:39 |
| AJaeger | \o/ | 17:39 |
| AJaeger | cmurphy: +A | 17:39 |
| AJaeger | thanks! | 17:39 |
| *** jamesdenton has quit IRC | 17:39 | |
| *** jmorgan1 has quit IRC | 17:40 | |
| *** signed8b_ is now known as signed8bit_Zzz | 17:40 | |
| *** ianychoi has quit IRC | 17:41 | |
| *** jamesmcarthur has joined #openstack-infra | 17:41 | |
| AJaeger | clarkb: want to rebase https://review.openstack.org/571520 (infra-manual PTI change)? with the Stein python3 goal, this is ready to go ahead IMHO (might need some tweaking but general direction is fine to merge IMHO) | 17:42 |
| *** stephenfin_ has quit IRC | 17:42 | |
| clarkb | AJaeger: sure | 17:42 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Stop logging inventory in validate-host https://review.openstack.org/563790 | 17:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add workaround for ethercalc bug https://review.openstack.org/585292 | 17:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add idempotency test https://review.openstack.org/585293 | 17:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add test for ethercalc::apache https://review.openstack.org/585294 | 17:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Use versioncmp function https://review.openstack.org/590489 | 17:42 |
| AJaeger | clarkb: with 563790 now merged, shall I +A https://review.openstack.org/563702 as well? | 17:43 |
| cmurphy | clarkb: downgrading nodemailer to 2.7.0 seems to fix it on trusty, not sure whether it's worth trying to capture it in tests or just assume it will be fine | 17:43 |
| *** rosmaita has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-4-node nodeset https://review.openstack.org/591464 | 17:44 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-3-node nodeset https://review.openstack.org/591465 | 17:44 |
| *** stephenfin has joined #openstack-infra | 17:45 | |
| AJaeger | config-core, good cleanup of our jobs today, thanks! - currently ready to merge: https://review.openstack.org/#/c/590581/ https://review.openstack.org/#/c/584190/ | 17:46 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-centos-7-2-node nodeset https://review.openstack.org/591466 | 17:46 |
| *** jamesmcarthur has quit IRC | 17:46 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Revert "Revert "Remove unneeded required projects"" https://review.openstack.org/590142 | 17:46 |
| AJaeger | config-core, should we move forward with https://review.openstack.org/#/c/511828 "Use ensure-output-dirs in base-test" ? | 17:46 |
| *** yamahata has joined #openstack-infra | 17:47 | |
| clarkb | cmurphy: My hunch is that newer nodemailer needs newer node which is why it works fine on xenial with nodejs 6 | 17:47 |
| clarkb | cmurphy: we are probably ok to ignore that short term if we can use this work to update to xenial soon (that is my goal) | 17:47 |
| pabelanger | AJaeger: I think if we land http://lists.openstack.org/pipermail/openstack-infra/2018-August/006032.html first, it will be easier to test new log directory changes | 17:48 |
| *** bgmccollum has joined #openstack-infra | 17:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-manual master: Push PTI job config into projects https://review.openstack.org/571520 | 17:49 |
| clarkb | AJaeger: ^ rebased | 17:49 |
| AJaeger | thanks, clarkb | 17:50 |
| AJaeger | pabelanger: https://review.openstack.org/#/c/587165/ has some questions by ianw - could you address them first | 17:50 |
| pabelanger | sure, will look this afternoon | 17:51 |
| *** olivierbourdon38 has quit IRC | 17:52 | |
| *** bobh has joined #openstack-infra | 17:54 | |
| AJaeger | clarkb: commented - thanks | 17:55 |
| clarkb | AJaeger: didn't approve https://review.openstack.org/#/c/590581/1 sinc ethere is an open question on the import data ( I responded to that question and +2'd though in case they decide they don't need a seed repo) | 17:56 |
| *** jamesmcarthur has joined #openstack-infra | 17:56 | |
| clarkb | AJaeger: do you think I should just link to that docs section from our docs rather than duplicate it? | 17:57 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove centos-7-3-node / centos-7-4-node https://review.openstack.org/591468 | 17:57 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy scenario job https://review.openstack.org/584190 | 17:57 |
| *** signed8bit_Zzz is now known as signed8b_ | 17:57 | |
| *** bobh has quit IRC | 17:58 | |
| AJaeger | clarkb: see my previous discussions there, I think we're fine - but let's wait... | 17:59 |
| *** pcaruana has quit IRC | 17:59 | |
| AJaeger | clarkb: let's ask dhellmann for https://review.openstack.org/571520 - I think linking to a stein goal page will soon get outdated. Otherwise I consider linking a good idea | 18:00 |
| clarkb | AJaeger: ok I tried to spell out the two ideas I had i na response to your review. Hopefully that helps dhellmann | 18:00 |
| AJaeger | clarkb: alternative is to link to our document from that... | 18:00 |
| AJaeger | infra-root, do you want to move forward with this gerritbot stack: https://review.openstack.org/#/c/545469/ ? | 18:01 |
| dhellmann | clarkb, AJaeger : it will be easier for you to keep that list of exceptions up to date in the infra spec. We can update the goal to point there for reference. | 18:01 |
| clarkb | dhellmann: AJaeger ok let me update it with the list of exceptions on our end then | 18:01 |
| corvus | AJaeger: i'm only volunteering to review emergency patches or patches implementing the limnoria spec | 18:02 |
| *** bgmccollum has quit IRC | 18:02 | |
| AJaeger | corvus: ah... | 18:02 |
| *** jamesdenton has joined #openstack-infra | 18:04 | |
| *** bobh has joined #openstack-infra | 18:05 | |
| *** rlandy has joined #openstack-infra | 18:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-manual master: Push PTI job config into projects https://review.openstack.org/571520 | 18:08 |
| clarkb | AJaeger: dhellmann ^ something like that? | 18:09 |
| *** bobh has quit IRC | 18:09 | |
| AJaeger | clarkb: that part is fine - could you update the zuulv3 page as well that I linked as comment comment, please? | 18:10 |
| *** slaweq_ has joined #openstack-infra | 18:11 | |
| clarkb | AJaeger: the goals page you mean? | 18:11 |
| AJaeger | clarkb: update https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 18:11 |
| AJaeger | clarkb: that information is outdated now, so let's remove the section - and add a link to the section you just changed. That's basically my suggestion | 18:12 |
| clarkb | got it | 18:12 |
| *** amoralej is now known as amoralej|lunch | 18:14 | |
| *** amoralej|lunch is now known as amoralej|off | 18:14 | |
| *** slaweq_ has quit IRC | 18:15 | |
| *** greghaynes has quit IRC | 18:18 | |
| *** electrofelix has quit IRC | 18:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-manual master: Push PTI job config into projects https://review.openstack.org/571520 | 18:19 |
| clarkb | AJaeger: ^ I htink that is what you were looking for | 18:19 |
| *** bobh has joined #openstack-infra | 18:20 | |
| *** trown|lunch is now known as trown | 18:22 | |
| AJaeger | clarkb: indeed, thanks! | 18:23 |
| *** bobh has quit IRC | 18:24 | |
| *** e0ne has joined #openstack-infra | 18:30 | |
| *** bobh has joined #openstack-infra | 18:31 | |
| *** slaweq_ has joined #openstack-infra | 18:32 | |
| *** bobh has quit IRC | 18:36 | |
| *** slaweq_ has quit IRC | 18:37 | |
| *** bobh has joined #openstack-infra | 18:40 | |
| *** bobh has quit IRC | 18:45 | |
| clarkb | pabelanger: followup on my question early about removing dib images, it seems tohave just orked | 18:47 |
| *** jmorgan1 has joined #openstack-infra | 18:47 | |
| clarkb | fedora-27 is completely gone as a dib image and as a cloud image and is not on disk on the builder | 18:47 |
| pabelanger | clarkb: yah, we just first need to make sure we have 0 nodes online before landing that change. Because wa automatic when we set min-ready to 0, now we have to manually delete it, if no jobs are going to use it | 18:48 |
| *** slaweq_ has joined #openstack-infra | 18:53 | |
| pabelanger | I should say, before nodepoolv3, we would delete ready nodes with min-ready:0 now we don't. | 18:53 |
| *** bobh has joined #openstack-infra | 18:56 | |
| *** slaweq_ has quit IRC | 18:57 | |
| *** jmorgan1 has quit IRC | 18:57 | |
| *** hemna_ has joined #openstack-infra | 18:58 | |
| *** bobh has quit IRC | 19:00 | |
| *** jmorgan1 has joined #openstack-infra | 19:01 | |
| *** hwoarang has quit IRC | 19:02 | |
| *** jpena|off is now known as jpena | 19:04 | |
| *** jpena is now known as jpena|off | 19:06 | |
| *** bobh has joined #openstack-infra | 19:13 | |
| *** fdegir_ has joined #openstack-infra | 19:17 | |
| *** patricku_ has joined #openstack-infra | 19:17 | |
| *** srwilkers_ has joined #openstack-infra | 19:17 | |
| *** lxkong_ has joined #openstack-infra | 19:17 | |
| *** ameade_ has joined #openstack-infra | 19:18 | |
| *** bobh has quit IRC | 19:18 | |
| *** Tengu_ has joined #openstack-infra | 19:20 | |
| *** bstinson_ has joined #openstack-infra | 19:23 | |
| *** mnencia_ has joined #openstack-infra | 19:23 | |
| *** mgagne_ has joined #openstack-infra | 19:24 | |
| *** Tengu has quit IRC | 19:25 | |
| *** rajinir has quit IRC | 19:25 | |
| *** knikolla has quit IRC | 19:25 | |
| *** adriant has quit IRC | 19:25 | |
| *** austinsun[m] has quit IRC | 19:25 | |
| *** mnencia has quit IRC | 19:25 | |
| *** lxkong has quit IRC | 19:25 | |
| *** patricku has quit IRC | 19:25 | |
| *** davidlenwell has quit IRC | 19:25 | |
| *** fdegir has quit IRC | 19:25 | |
| *** mnasiadka has quit IRC | 19:25 | |
| *** ameade has quit IRC | 19:25 | |
| *** robcresswell has quit IRC | 19:25 | |
| *** serverascode_ has quit IRC | 19:25 | |
| *** kien-ha has quit IRC | 19:25 | |
| *** srwilkers has quit IRC | 19:25 | |
| *** tommylikehu has quit IRC | 19:25 | |
| *** kambiz has quit IRC | 19:25 | |
| *** HenryG has quit IRC | 19:25 | |
| *** jgrassler has quit IRC | 19:25 | |
| *** mgagne has quit IRC | 19:25 | |
| *** Miouge has quit IRC | 19:25 | |
| *** charz has quit IRC | 19:25 | |
| *** bstinson has quit IRC | 19:25 | |
| *** bobh has joined #openstack-infra | 19:25 | |
| *** mnencia_ is now known as mnencia | 19:25 | |
| *** patricku_ is now known as patricku | 19:25 | |
| *** srwilkers_ is now known as srwilkers | 19:25 | |
| *** lxkong_ is now known as lxkong | 19:25 | |
| *** fdegir_ is now known as fdegir | 19:25 | |
| *** ameade_ is now known as ameade | 19:25 | |
| *** Miouge- has joined #openstack-infra | 19:25 | |
| *** adriant has joined #openstack-infra | 19:26 | |
| *** amotoki has quit IRC | 19:27 | |
| *** knikolla has joined #openstack-infra | 19:27 | |
| *** bobh has quit IRC | 19:29 | |
| mordred | corvus: (or anybody) do you know why we have manage_exim: false on backup\d+\..*\.ci\.openstack\.org ? | 19:29 |
| * fungi does not | 19:30 | |
| *** amotoki has joined #openstack-infra | 19:30 | |
| *** holser_ has joined #openstack-infra | 19:31 | |
| fungi | backup01.ord.rax.ci.openstack.org is the only host in our inventory matching that pattern that i can find | 19:31 |
| clarkb | I think we wanted those nodes as static as possible | 19:32 |
| *** slaweq_ has joined #openstack-infra | 19:32 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Add beaker tests for firehose https://review.openstack.org/585691 | 19:32 |
| *** bobh has joined #openstack-infra | 19:34 | |
| *** holser_ has quit IRC | 19:35 | |
| *** slaweq_ has quit IRC | 19:36 | |
| *** bobh has quit IRC | 19:38 | |
| *** eharney has joined #openstack-infra | 19:40 | |
| *** apetrich has quit IRC | 19:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove base.yaml things from openstack_project::server https://review.openstack.org/585836 | 19:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Shift lists exim config to ansible https://review.openstack.org/591494 | 19:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Shift exim config for firehose and storyboard https://review.openstack.org/591495 | 19:43 |
| mordred | corvus, clarkb, fungi: ^^ those last two could use some eyeballs and/or feedback about approach | 19:44 |
| * mordred realized that having ansible run base.py and then puppet run will work fine ... except for the hosts with extra special exim config | 19:45 | |
| *** e0ne has quit IRC | 19:52 | |
| *** bobh has joined #openstack-infra | 19:53 | |
| *** xe1gyq has quit IRC | 19:58 | |
| *** bobh has quit IRC | 19:58 | |
| *** efried has quit IRC | 19:58 | |
| corvus | mordred: if we're going to treat the varibles as global, should we prefix all of them with the module name? | 20:00 |
| corvus | i was sort of expecting the role to have variables passed into it explicitly, which is why i didn't prefix them. to be honest, the whole global variable thing we have with hiera which gives us names like "zuul_gerrit_ssh_key_for_jenkins" or whatever is difficult to work with | 20:00 |
| corvus | it seems :( that the exim module has to have "exim_transports" as its parameter instead of "transports" | 20:00 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for firehose.o.o https://review.openstack.org/591500 | 20:02 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Turn off binary symlink in nodejs module https://review.openstack.org/590015 | 20:02 |
| *** bgmccollum has joined #openstack-infra | 20:04 | |
| *** rajinir has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Fix httpd mod defined types for puppet 4 https://review.openstack.org/590023 | 20:06 |
| corvus | mordred, fungi: afaik, lists.katacontainers.io is still on the old mailman/exim config. i don't know if anyone is planning on updating that or migrating it to lists.o.o. but its exim config is very different than lists.o.o, and is managed by the puppet mailman module. | 20:07 |
| fungi | yeah, and they're depending on dmarc mitigation options only available from the mailman version available on xenial | 20:10 |
| fungi | i assumed we wouldn't try to migrate them until after lists.o.o is upgraded | 20:10 |
| corvus | fungi: it doesn't sound like we manage that server. i suggest we cut it loose. | 20:10 |
| mordred | corvus: well - I think we can do the transports/exim_transports thing in a few different ways depending on how we want to organize it | 20:10 |
| *** slaweq_ has joined #openstack-infra | 20:11 | |
| fungi | it's in our tenant, so may be hard to omit from the intentory. maybe perpetually in the emergency disable list until we can migrate it (once we're running lists.o.o on xenial)? | 20:11 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: remove pypy jobs https://review.openstack.org/591504 | 20:12 |
| clarkb | Isnt it managed by puppet? | 20:12 |
| corvus | i had no idea it was running a different version | 20:12 |
| mordred | is it not using puppet-exim for exim? it looked like its config was using that | 20:13 |
| fungi | yeah, we launched it on xenial with the expectation the next lists.o.o upgrade was coming soonish (i forget who was talking about planning that upgrade though) | 20:13 |
| *** efried has joined #openstack-infra | 20:13 | |
| corvus | mordred: er, yes, sorry, it's puppet-exim that does that, but it has a bunch of hardcoded stuff in the config file to do it. maybe it's possible to reimplement with the exim role, i'm not sure. but if so, it'll be a unique set of routers/transports/etc that is different than lists.o.o. | 20:14 |
| mordred | corvus: well - looking at what was hardcoded - it looked the same as the routers/transports for lists.o.o (even though it was coming in through differenttemplating) | 20:15 |
| mordred | but it's entirely possible I was just totally reading it wrong | 20:15 |
| mordred | that said - we can totally put those routers/transports into host_vars intead of group_vars | 20:15 |
| corvus | mordred: lists.o.o looks in /srv/mailmain for the multi-domain support | 20:15 |
| mordred | ah. nod | 20:16 |
| *** slaweq_ has quit IRC | 20:16 | |
| clarkb | not sure I understand what the hardcoding is. looks pretty boring in kata_lists.pp | 20:16 |
| corvus | fungi: well, it seems we worked at cross-purposes; lists.k.o was supposed to be temporary until the multi-domain support was finished (see commit message in https://review.openstack.org/524322 ) but we've created a situation where we can't merge it. | 20:17 |
| clarkb | we could merge it if we forced microsoft users to usedifferent email providers or upgraded lists to xenial | 20:17 |
| *** e0ne has joined #openstack-infra | 20:18 | |
| clarkb | but yes I think that was an unintended side effect of people addressing a problem in front of them | 20:18 |
| mordred | corvus: the exim divergence is just the stuff in exim.conf right? the other stuff should be largely the same? | 20:19 |
| corvus | clarkb: we should talk about the "dmarc mitigation measures" before we upgrade lists to xenial. i have not yet seen one that i would be willing to implement on a mailing list server i have any responsibility for, so, at the very least, that warrants some discussion i think. | 20:19 |
| corvus | mordred: i think so? i can't think of anything else now. | 20:20 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to get my speaker presentations by role and selection plan https://review.openstack.org/591508 | 20:20 |
| corvus | fungi: what mitigation is being used? | 20:22 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to get my speaker presentations by role and selection plan https://review.openstack.org/591508 | 20:22 |
| fungi | corvus: the dmarc sender rewriting option | 20:23 |
| mordred | corvus: the current puppet-exim is able to splat out exim.conf for both lists.o.o and lists.kc.io (which is really all I'm trying to be able to do now) | 20:23 |
| mordred | I think the divergence is in mailman | 20:23 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to get my speaker presentations by role and selection plan https://review.openstack.org/591508 | 20:23 |
| mordred | which isn't being touched by any of the ansible work yet | 20:23 |
| *** bobh has joined #openstack-infra | 20:24 | |
| corvus | mordred: yes, it is. but the configuration it splats out is different | 20:24 |
| fungi | and yeah, looking back at channel history, i thought i had explicitly decided to use xenial because we were talking about upgrading lists.o.o to xenial, but the closer i look the more i suspect i just ended up with xenial because that was our default in launch-node.py | 20:24 |
| corvus | fungi: do you mean "dmarc_moderation_action: Munge From" ? | 20:25 |
| corvus | mordred: the "mailman_domains" variable in puppet-exim triggers the different behavior | 20:26 |
| fungi | corvus: yes, looks like that's the one. config_list just shows that it's option #1 | 20:27 |
| corvus | brb | 20:27 |
| fungi | so not smtp sender rewriting, but from header rewriting | 20:28 |
| *** bobh has quit IRC | 20:29 | |
| clarkb | and it only rewrites if the domain says they reject invalid email as specified by dmarc right? | 20:29 |
| mordred | corvus: ok. I grok the words you are saying - I have an idea of how to do that patch different/better that I think should work | 20:31 |
| corvus | mordred: ok cool, i was just trying to look up source links to point you at, but maybe you don't need that? | 20:31 |
| *** bobh has joined #openstack-infra | 20:32 | |
| *** slaweq_ has joined #openstack-infra | 20:32 | |
| mordred | yah - I think I should have taken that left at albequerque | 20:32 |
| corvus | mordred: my *guess* is that we probably can reimplement it for lists.k.o in the same pattern as lists.o.o, just the actual routers/transports will be different. | 20:33 |
| mordred | yup | 20:33 |
| mordred | the divergence in the routers/transports was the part I was missing when i looked before | 20:33 |
| corvus | so grepping for mailmain_domains in http://git.openstack.org/cgit/openstack-infra/puppet-exim/tree/templates/exim4.conf.erb and translating that into yaml | 20:33 |
| *** e0ne has quit IRC | 20:33 | |
| clarkb | yup if domain says reject or quarantine munge will munge otherwise should be unaffected | 20:34 |
| corvus | they should be really *similar*, so can probably copy-pasta 90% | 20:34 |
| *** apetrich has joined #openstack-infra | 20:34 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for review and review01 https://review.openstack.org/591516 | 20:34 |
| *** jcoufal has quit IRC | 20:34 | |
| fungi | clarkb: yes, only from addresses for domains which claim dmarc signatures should be enforced get rewritten from what i've seen. everyone else's from headers get left intact | 20:36 |
| *** bobh has quit IRC | 20:36 | |
| *** slaweq_ has quit IRC | 20:36 | |
| corvus | clarkb, fungi: it appears that option is implemented in in 2.1.16 | 20:37 |
| corvus | it's effectively the same as from_is_list in 2.1.16 | 20:37 |
| corvus | i'm still not sure it should be enabled, but if we wanted to talk about it, it's physically possible. | 20:38 |
| fungi | huh, interesting. for some reason i didn't think it showed up until 2.1.18 | 20:40 |
| fungi | i don't know that it makes sense for all lists, but apparently the kata-dev list had a lot of people getting kicked (most of the participants on the list are from intel and microsoft from what i gather) | 20:41 |
| corvus | i don't know that it would fruitful to rehash the dmarc discussion right now. i think we all are familiar with that. i'm just really not clear as to whether i have a supporting role in an RFC-violating list server or not. | 20:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Shift lists exim config to ansible https://review.openstack.org/591494 | 20:43 |
| corvus | cause "violate the RFCs" is a decision i would have expected us, as sysadmins, to make as a group | 20:43 |
| corvus | so the fact that decision was made for kata without that discussion makes it feel like we are not responsible for that server | 20:44 |
| corvus | but we keep talking about updating its config management like we are | 20:44 |
| fungi | yeah, i don't think any of the sysadmins set it. jbryce told me he turned it on in the list manager webui | 20:44 |
| corvus | during a normal upgrade, i would have expected us to learn about the option and set an appropriate site default | 20:45 |
| corvus | in 2.1.16, the option is disabled by default in the site config | 20:45 |
| fungi | ahh, as in 2.1.16 default server configuration disallows list owners turning it on? | 20:46 |
| corvus | fungi: yes, and in 2.1.18, the site admin can set the maximum permissiveness for the new option which applies to all lists | 20:46 |
| corvus | https://wiki.list.org/DEV/DMARC | 20:46 |
| mordred | corvus: I think https://review.openstack.org/591494 covers the exim difference just using the routers/transports options from your new exim role - but when you have headspace I'd also like to chat about the approach to variable naming approach - it seems we're finally to a complex enough example in an ansible patch to have somehting meaty to discuss at least | 20:47 |
| corvus | mordred: yeah, as much as it feels like bikeshedding, i think our experience_with_really_long_global_hiera_variable_names suggests it's worth talking about that now :) | 20:48 |
| mordred | oh -totally - I don't think it's bikeshedding at all, honestly | 20:48 |
| clarkb | its a per liet setting aiui | 20:48 |
| mordred | clarkb: it honestly took me a few to figure out what a liet was :) | 20:49 |
| clarkb | so we can operate a server that is happy and list admins can change this on their own | 20:49 |
| fungi | clarkb: per list but the server admin can choose whether to allow it to be set on any lists or forbid it server-wide | 20:49 |
| clarkb | sorry list | 20:49 |
| clarkb | fungi: ah | 20:49 |
| fungi | which is the point, i think | 20:49 |
| *** signed8b_ is now known as signed8bit_Zzz | 20:50 | |
| corvus | clarkb: yes, effectively we can, as sysadmins, set what level of breakage we're comfortable with, and list admins can break things up to and including that level :) | 20:50 |
| fungi | we didn't as a group decide we were okay with list owners being allowed to turn that option on | 20:50 |
| clarkb | from a usability perspective it is unfortunately painful to tell users to use a less bad email provider | 20:50 |
| * mordred is not happy about corporations with crappy IT departments who are not competent enough to run compliant email being allowed to break email for other people | 20:50 | |
| corvus | clarkb: from a usability perspetive it is unfortunately painful for users to recieve messages with incorrect from addresses | 20:51 |
| jbryce | I'm not sure there's been any pain for the users of that list. It actually fixed the pain of the list not working for many of them | 20:51 |
| clarkb | corvus: I agree but the alternative aiui is to drop the email? | 20:51 |
| mordred | otoh - those corporations have already effectively bullied a large portion of the internet, so we're in the situation where to the casual observer something seems 'wrong' with the service we're running | 20:51 |
| corvus | clarkb: there are several alternatives on that page | 20:52 |
| corvus | clarkb: i advocate reject (not drop) | 20:52 |
| jbryce | So I guess you have to weigh what you consider to be the actual purpose of having a mailing list | 20:52 |
| fungi | clarkb: one alternative is actually to turn on a feature which rejects attempts to subscribe from dmarc-enforced domains | 20:52 |
| corvus | mordred: i believe that reject would improve the user experience there; a clear message about the problem could be provided to the user | 20:52 |
| corvus | fungi: that's even better | 20:53 |
| *** bobh has joined #openstack-infra | 20:53 | |
| *** slaweq_ has joined #openstack-infra | 20:53 | |
| fungi | i can't recall if 2.1 added that but i saw it in mm3 | 20:53 |
| corvus | a point which i would like to continue to stress is that not allowing messages/subscription from dmarc domains *is* acting in accordance with the wishes of the owners of those domains. | 20:53 |
| corvus | dmarc is a way of saying this domain should not be used for this purpose | 20:53 |
| jbryce | circus: how does that improve the user experience? It just ends the user experience for people who are required to use an email on a domain that has dmarc in place | 20:54 |
| *** jamesmcarthur has quit IRC | 20:54 | |
| fungi | the users of those domains are unfortunately kept in the dark about that fact (and the operators of those domains either don't understand that's what it means when they set that up or are not interested in explaining to their users that's the case) | 20:54 |
| *** trown is now known as trown|outtypewww | 20:55 | |
| jbryce | (Phone irc client autocorrected corvus to circus) | 20:55 |
| corvus | jbryce: i, um, assume "circus" was addressed to me. it doesn't end the experience, we can direct folks to use a domain which permits people to send to mailing lists. we shouldn't help them work around their corporate policy. | 20:55 |
| mordred | :) | 20:55 |
| * mordred was :)'ing to circus, which he now wants to start calling corvus | 20:55 | |
| corvus | mordred: try it and see how it works out for you | 20:55 |
| mordred | corvus: I reserve the right to try it at some point in the future and to experience the effects | 20:56 |
| fungi | uhnfortunately the dmarc cancer has spread beyond corporate use and has infected a number of freemail sites too | 20:56 |
| corvus | actually documenting this and informing users of the issue and corrective actions will be much better than having things broken. | 20:56 |
| jbryce | Or adjust a simple setting that states clearly what the mail agent is doing (forwarding on behalf of) and make the list work for users who are trying to participate | 20:56 |
| *** bobh has quit IRC | 20:57 | |
| *** slaweq_ has quit IRC | 20:57 | |
| corvus | at the expense of users who are not at such domains | 20:57 |
| jbryce | How is at their expense? I think that's what I'm missing | 20:58 |
| jbryce | The kata list works fine for everyone now | 20:58 |
| jbryce | Have gotten zero additional complaints since applying that setting | 20:58 |
| fungi | other users are receiving messages with inaccurate from addresses (the header says it's from the mailing list's domain instead of the user's domain) | 20:59 |
| corvus | jbryce: i think in that case the sample size is small and selective. | 20:59 |
| corvus | one of the issues is what fungi just articulated | 20:59 |
| *** apetrich has quit IRC | 20:59 | |
| jbryce | What is the expense incurred by that? | 21:00 |
| corvus | as an example of how that can manifest -- users of MUAs with address books may now be accumulating incorrect addresses for their colleagues | 21:00 |
| *** Goneri has quit IRC | 21:00 | |
| corvus | replying to messages becomes even more confusing | 21:00 |
| jbryce | I receive the message. I clearly see who sent it. I can read, reply, communicate | 21:00 |
| corvus | jbryce: i'm not arguing that you are experiencing a problem | 21:01 |
| jbryce | The alternative is locking out a large and growing number of people which seems counterproductive considering the ultimate reason for a mailing list to exist | 21:02 |
| fungi | well, to say they're locked out is to assume they're unwilling to use other e-mail addresses to participate on mailing lists | 21:02 |
| mordred | we could switch to nntp, which yahoo has not (yet) broken | 21:02 |
| corvus | i'm arguing that the process violates the RFCs, which are the way the internet community agrees on how systems should interoperate, so that diverse systems can exist and work together. | 21:03 |
| *** bobh has joined #openstack-infra | 21:03 | |
| jbryce | I understand your argument | 21:03 |
| * mordred does not like being bullied by people who do not play by the rules | 21:03 | |
| jbryce | The reality is that a large part of the internet community does not operate that way | 21:04 |
| fungi | and another large (but vanishing if we don't fight for it) part of the internet does | 21:05 |
| corvus | that makes me sad, but does not convince me to change my opinion. such a thing could be (and usually has been) said about almost everything we do in our community. | 21:05 |
| clarkb | fungi: I think one of the struggles with that is at least some of our users have changed providers just to have the new provider evntually start enforcing dmarc too :/ We probably need to be able to point people to a location that does not intend on doing so | 21:05 |
| *** bobh has quit IRC | 21:08 | |
| fungi | it's been long predicted that global e-mail will end up entirely partitioned by these various uncoordinated adjustments to the protocol (and the ietf rfc process is no longer nearly so coordinated a means of extending internet protocols as it once was) | 21:10 |
| mordred | I do not like either option we have and I am mad at yahoo for putting us in this position. I believe it is important for our ability to collaborate openly for everyone to play by a defined and understood rulebook. I also understand that we have a growing number of people who are captive to employers who are responsible for violating such agreements in a way that is potentially out of their individual | 21:10 |
| mordred | control. there are literally no outcomes that are not problematic on at least one axis | 21:10 |
| *** slaweq_ has joined #openstack-infra | 21:11 | |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| *** bobh has joined #openstack-infra | 21:13 | |
| corvus | mordred: agreed. that seems like exactly the sort of thing that calls out for discussion and consensus. | 21:14 |
| *** slaweq_ has quit IRC | 21:16 | |
| *** bobh has quit IRC | 21:17 | |
| clarkb | corvus: with the reject option that means mailman tells the remote sender the email was rejected due to dmarc? | 21:17 |
| clarkb | (trying to understand how the various options end up expressing themselves to end users) | 21:17 |
| *** hwoarang has joined #openstack-infra | 21:18 | |
| corvus | clarkb: aiui, yes. | 21:19 |
| corvus | clarkb: since we use exim, and exim is turing complete, there's likely another option which isn't on there, which is to reject at SMTP time. | 21:19 |
| corvus | imo that's almost always better than a bounce | 21:19 |
| fungi | especially since reliable bounce conditions are often exploitable to turn into a reflection attack | 21:20 |
| *** haleyb is now known as haleyb|away | 21:20 | |
| *** signed8bit_Zzz is now known as signed8b_ | 21:21 | |
| clarkb | reading up on fastmail's policy plans here (since they are my current provider) they will set a p=reject policy in the future (date unspecified), this means for people using @fastmail.com and other fastmail domains they will also run into this | 21:21 |
| mordred | well that's disappointing - I thought they were supposed to be one of the good ones | 21:23 |
| clarkb | I use a custom domain though so can configure my own policy in DNS | 21:23 |
| clarkb | I think | 21:24 |
| *** slagle has quit IRC | 21:26 | |
| fungi | if they don't control your dns for that domain, then yes | 21:27 |
| *** slaweq_ has joined #openstack-infra | 21:33 | |
| mordred | infra-root: I've gotta go run some errands, so I think that's likely it for me today - I'll keep pushing on bridge tomorrow | 21:37 |
| clarkb | I'm totally nerd sniped by me email related dns records right now :/ | 21:37 |
| mordred | corvus: maybe ping me when you get up and we can chat through variable naming stuff before things get too busy? | 21:37 |
| jbryce | I feel the same way as mordred in that there are problems with all options | 21:38 |
| clarkb | I've had to update the gandi nameservers my domain users before they will let me manage my domain records :/ | 21:40 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoints to add/remove speakers from my presentations https://review.openstack.org/591526 | 21:41 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoints to add/remove speakers from my presentations https://review.openstack.org/591526 | 21:42 |
| corvus | clarkb: you can use our nameservers in exchange for a chocolate bar | 21:42 |
| clarkb | corvus: ha | 21:42 |
| jbryce | And like corvus said, as we look at opening up these collaboration services to more people, I think these usability things deserve a lot of consideration and discussion | 21:43 |
| clarkb | apparnetly my dkim setup is also broken because I have old deprecated dns records too. | 21:43 |
| *** slaweq_ has quit IRC | 21:43 | |
| jbryce | For something like this, it is sometimes not in our users' control. That will probably be a higher or lower proportion depending on different project communities | 21:44 |
| *** jamesmcarthur has quit IRC | 21:45 | |
| *** Goneri has joined #openstack-infra | 21:45 | |
| jbryce | Which may mean there's not one blanket config across the board that works for all of them. Or it may mean telling some of them they should set up mailing lists somewhere else. Or other alternatives | 21:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/devstack-gate master: Bump ansible to latest 2.5 version https://review.openstack.org/591527 | 21:49 |
| *** gfidente is now known as gfidente|fk | 21:49 | |
| *** e0ne has joined #openstack-infra | 21:50 | |
| pabelanger | clarkb: noticed by chance, give 2.3 is eol. figured we should try to update to 2.5 | 21:50 |
| pabelanger | and see what breaks | 21:50 |
| clarkb | pabelanger: thats fine, though considering 2.3 still works and d-g is on life support (in favor of devstack zuul native job) if things are really broken on 2.5 that might be a good reason to get off of d-g | 21:51 |
| *** bobh has joined #openstack-infra | 21:51 | |
| pabelanger | ++ | 21:51 |
| pabelanger | yah, maybe topic at PTG about pushing of legacy | 21:51 |
| *** bobh has quit IRC | 21:55 | |
| *** e0ne has quit IRC | 21:56 | |
| *** slaweq_ has joined #openstack-infra | 22:00 | |
| *** rcernin has joined #openstack-infra | 22:01 | |
| ianw | infra-root: could someone poke a second +2 on a pretty easy cleanup for nodepool logging files -> https://review.openstack.org/#/c/586236/ . will just let me clear out another review from my queue and cleanup some of hte puppet around nodepool.o.o | 22:03 |
| *** slaweq_ has quit IRC | 22:04 | |
| *** boden has quit IRC | 22:08 | |
| *** rlandy has quit IRC | 22:09 | |
| *** slaweq_ has joined #openstack-infra | 22:11 | |
| clarkb | ianw: done | 22:11 |
| *** rpioso is now known as rpioso|afk | 22:15 | |
| *** slaweq_ has quit IRC | 22:16 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-elastic_recheck master: Add beaker tests https://review.openstack.org/581389 | 22:18 |
| fungi | wow, i just got privmsg spam that seemed like it was attempting to exploit unsafe control sequence handling in some client/terminal | 22:24 |
| clarkb | fungi: huh | 22:26 |
| *** scarab_ has joined #openstack-infra | 22:27 | |
| clarkb | wow where did the day go its already 3:30pm and I'm yet to rereview ethercalc/etherpad and topic:puppet-4 related changes | 22:31 |
| *** scarab_ has quit IRC | 22:32 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use nodepool-builder.logging.conf from puppet-nodepool https://review.openstack.org/586236 | 22:32 |
| fungi | oh, analyzing the log from that privmsg looks like it was simply an annoying combination of unicode soft hyphens and raw tabs that confused my terminal's font placement | 22:32 |
| fungi | i was about to be surprised if someone managed to funnel terminal escapes through irc | 22:32 |
| clarkb | infra-root can I get reviews on https://review.openstack.org/#/c/591017/ which will allow us to upgrade etherpad servers to xenial (the -1 was due to some testing changes that were fixed I have rechecked) and https://review.openstack.org/#/c/571520/ changes our suggestions around where people put their job config | 22:33 |
| *** scarab_ has joined #openstack-infra | 22:33 | |
| clarkb | The second one is a policy update relfected through docs. Might be good to get a wider audience on that one | 22:34 |
| *** rtjure has quit IRC | 22:35 | |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| ianw | tonyb: might want to comment on https://review.openstack.org/#/c/571520 ? | 22:37 |
| *** scarab_ has quit IRC | 22:37 | |
| tonyb | ianw: Thanks, I'll go back and read it. I tought I'd left a comment but I can re-review it and the associated changes | 22:38 |
| ianw | yeah it has just had a few revs since | 22:40 |
| clarkb | The organization in that section changed a bit so I had to update the content a little | 22:40 |
| clarkb | the general idea behind it should be the same though | 22:41 |
| clarkb | ianw: I see you are reviewing ethercalc things, care to review https://review.openstack.org/#/c/590026/2 too? I can watch etherpad-dev and make sure it is happy with future parser | 22:48 |
| *** jamesmcarthur has quit IRC | 22:52 | |
| clarkb | ianw: also read the docs is taking a downtime 1700UTC august 18 for four hours | 22:53 |
| *** slaweq_ has joined #openstack-infra | 22:53 | |
| clarkb | I think you are past the make rtd jobs work again problems but if you see issues then likely just due to their migration | 22:53 |
| openstackgerrit | Merged openstack-infra/puppet-ethercalc master: Add workaround for ethercalc bug https://review.openstack.org/585292 | 22:56 |
| openstackgerrit | Merged openstack-infra/puppet-ethercalc master: Add idempotency test https://review.openstack.org/585293 | 22:56 |
| openstackgerrit | Merged openstack-infra/puppet-ethercalc master: Add test for ethercalc::apache https://review.openstack.org/585294 | 22:56 |
| openstackgerrit | Merged openstack-infra/puppet-ethercalc master: Use versioncmp function https://review.openstack.org/590489 | 22:57 |
| *** slaweq_ has quit IRC | 22:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Cleanup nodepool.openstack.org to be a zk server https://review.openstack.org/547248 | 23:00 |
| openstackgerrit | Merged openstack-infra/system-config master: Turn the future parser on for etherpad-dev https://review.openstack.org/590026 | 23:09 |
| *** slaweq_ has joined #openstack-infra | 23:11 | |
| tonyb | clarkb, ianw: Any idea when the code merged from https://review.openstack.org/#/c/550978 will be running and so I can query it? | 23:14 |
| *** slaweq_ has quit IRC | 23:15 | |
| clarkb | tonyb: will require we restart the scheduler and web daemons. I know there are other features people want to get in (like the file line mapping change) | 23:18 |
| clarkb | tonyb: looking at ps, that change should already be in place | 23:18 |
| clarkb | tonyb: yup I think http://zuul.openstack.org/api/job/devstack is what you are looking for | 23:19 |
| tonyb | clarkb: Oh cool. I'll give it a try. Are there docs for poke the API or shoudl I just follwo the bouncing ball? | 23:19 |
| tonyb | Oh that's easy ;P | 23:19 |
| tonyb | ... hmm perhaps I need he project bit that isn't merged. eitehrway I'll play and comment on thr PTI review | 23:20 |
| tonyb | Thanks! | 23:20 |
| openstackgerrit | Merged openstack-dev/pbr master: add python 3.6 unit test job https://review.openstack.org/589589 | 23:21 |
| *** tpsilva has quit IRC | 23:28 | |
| clarkb | etherpad-dev looks happy with future parser. http://paste.openstack.org/show/727959/ | 23:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add icons to index files https://review.openstack.org/591535 | 23:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/devstack-gate master: Bump ansible to latest 2.5 version https://review.openstack.org/591527 | 23:37 |
| clarkb | corvus: that is an interesting approach :) | 23:37 |
| corvus | clarkb: it is my second! | 23:38 |
| corvus | clarkb: i started with uploading the icons to the root of the container, and got as far as realizing that i would need to link to https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/corvustest/icons/unknown.png before i, um, decided to change tack. | 23:39 |
| clarkb | corvus: one comment (unused import) | 23:40 |
| corvus | (doing that isn't impossible, it just requires a whole bunch of new plumbing in the script to handle generating index files with knowledge of where they're eventually going to be uploaded; right now we have separated concerns -- index generation doesn't know anythig about uploading) | 23:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add icons to index files https://review.openstack.org/591535 | 23:41 |
| corvus | i bet i can guess which one | 23:41 |
| clarkb | corvus: we may want to consider only having a single copy of each image in the index too (right now I htink its an entire copy per usage) | 23:41 |
| corvus | clarkb: yes. this was very easy to write. :) | 23:41 |
| *** gfidente|fk is now known as gfidente | 23:42 | |
| clarkb | corvus: we could just embed the base64 content in generate_log_index as part of the head right? then your map returns could refer back to those items? | 23:43 |
| clarkb | would require embedded css I think | 23:43 |
| clarkb | basically same idea as existing change but reorganizes it to reduce file size | 23:44 |
| corvus | clarkb: yeah, i think that's what we do for the status page (though its days are numbered there i think) | 23:44 |
| clarkb | ianw: cmurphy fwiw ethercalc looks good after that stack merged too. I think that means I'm up for booting a new ethercalc tomorrow | 23:48 |
| ianw | oh good. it's nice when testing reflects reality :) | 23:50 |
| pabelanger | clarkb: looks like tripleo create some new nodesets, specific for their jobs: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/zuul.d/nodesets.yaml I'll dive into more why tomorrow and see if we can collapse them into existing nodesets in ozj or devstack | 23:51 |
| clarkb | if we can get digitized ethrepad change in too, I'll work on etherpad-dev and time permitting etherpad.o.o | 23:51 |
| pabelanger | if only to keep naming convention the same | 23:51 |
| clarkb | pabelanger: in general I think we are ok with that as it allows you to modify groups and group vars which may be useful to specific jobs? | 23:51 |
| clarkb | however those looks pretty boring and similar to what we already have :) | 23:52 |
| *** slaweq_ has joined #openstack-infra | 23:53 | |
| pabelanger | yah, very similar. Will look to see if we can colapse, if not that is also fine | 23:53 |
| *** slaweq_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
