| prometheanfire | hm yep | 00:00 |
|---|---|---|
| prometheanfire | invalid here too | 00:00 |
| * prometheanfire is just now reading backlog | 00:00 | |
| *** gagehugo has joined #openstack-infra | 00:01 | |
| *** rh-jelabarre has quit IRC | 00:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Remove AvoidDuplicates from mailman pipeline https://review.openstack.org/635078 | 00:04 |
| corvus | fungi: ^ | 00:04 |
| fungi | thanks! reviewing | 00:04 |
| *** yamamoto has quit IRC | 00:04 | |
| corvus | fungi: we certainly could see more issues, but this one really could break almost any message at random, so it's a plausible explanation for all of the issues. | 00:05 |
| corvus | fungi: here's the original: https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/Defaults.py.in#L675 | 00:07 |
| fungi | agreed. and thanks, i was just about to ask whether that was basically a copy from the default config with the AvoidDuplicates pipeline removed | 00:08 |
| *** whoami-rajat has quit IRC | 00:08 | |
| corvus | fungi: i do not know how to do that in mm3 | 00:10 |
| fungi | i've added it to https://etherpad.openstack.org/p/mm3poc | 00:13 |
| fungi | so we don't forget it's a possible stumbling block | 00:14 |
| *** ijw has joined #openstack-infra | 00:15 | |
| corvus | fungi: i'm pretty sure we *can* do the same thing in mm3 -- i'm seeing facilities for custom pipelines, etc. i still don't know how, but i'm fairly confident we won't be boxed in. | 00:17 |
| corvus | fungi: ah, here's the default in mm3: https://gitlab.com/mailman/mailman/blob/master/src/mailman/pipelines/builtin.py#L49 | 00:19 |
| fungi | thanks! i'll add a reference to that too | 00:19 |
| prometheanfire | make sure to refrence a commit sha as master can/will change over time | 00:20 |
| fungi | good call, doing now | 00:20 |
| *** ijw_ has joined #openstack-infra | 00:21 | |
| *** Emine has quit IRC | 00:23 | |
| *** ijw has quit IRC | 00:23 | |
| fungi | infra-root: https://review.openstack.org/635073 is another fairly trivial (puppet classvar plumbing exercise) xenial-upgrade change for the openstackid servers if you get a moment | 00:24 |
| *** rascasoft has joined #openstack-infra | 00:24 | |
| fungi | oh, and i just spotted an error in that one | 00:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Override FQDN defaults for OpenStackID servers https://review.openstack.org/635073 | 00:26 |
| *** gcutrini has quit IRC | 00:28 | |
| *** kjackal has quit IRC | 00:32 | |
| *** tosky has quit IRC | 00:34 | |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| *** rascasoft has quit IRC | 00:38 | |
| *** jamesmcarthur has quit IRC | 00:40 | |
| *** rfolco has quit IRC | 00:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add insecure-ci-registry01 to inventory https://review.openstack.org/635047 | 00:42 |
| *** mriedem has quit IRC | 00:43 | |
| *** wolverineav has quit IRC | 00:43 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove private_v4 parameters from static inventory https://review.openstack.org/635052 | 00:44 |
| *** wolverineav has joined #openstack-infra | 00:45 | |
| *** auristor has quit IRC | 00:45 | |
| *** wolverineav has quit IRC | 00:46 | |
| *** wolverineav has joined #openstack-infra | 00:46 | |
| *** auristor has joined #openstack-infra | 00:47 | |
| *** lbragsta_ has joined #openstack-infra | 00:50 | |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| *** lbragsta_ is now known as lbragstad__ | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:56 | |
| *** Swami has quit IRC | 00:57 | |
| *** rfolco has joined #openstack-infra | 00:59 | |
| *** hwoarang has quit IRC | 01:00 | |
| *** hwoarang has joined #openstack-infra | 01:01 | |
| *** wolverineav has quit IRC | 01:03 | |
| *** wolverineav has joined #openstack-infra | 01:04 | |
| *** mrodriguez has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:04 | |
| *** hamzy has joined #openstack-infra | 01:06 | |
| *** wolverineav has quit IRC | 01:08 | |
| *** bobh has joined #openstack-infra | 01:15 | |
| *** bobh has quit IRC | 01:16 | |
| *** wolverineav has joined #openstack-infra | 01:24 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add storyboard01 to Ansible inventory https://review.openstack.org/633420 | 01:29 |
| fungi | and updated that ^ to remove the private_ipv4 since i forgot it hadn't already merged | 01:29 |
| *** dims has quit IRC | 01:40 | |
| *** rascasoft has joined #openstack-infra | 01:45 | |
| *** bobh has joined #openstack-infra | 01:49 | |
| *** gcutrini has joined #openstack-infra | 01:50 | |
| *** rascasoft has quit IRC | 01:50 | |
| *** bobh has quit IRC | 01:52 | |
| *** armax has quit IRC | 01:53 | |
| *** agopi|off has quit IRC | 01:57 | |
| *** wolverineav has quit IRC | 02:06 | |
| *** wolverineav has joined #openstack-infra | 02:07 | |
| *** gcutrini has quit IRC | 02:09 | |
| *** dims has joined #openstack-infra | 02:11 | |
| *** wolverineav has quit IRC | 02:11 | |
| *** agopi|off has joined #openstack-infra | 02:16 | |
| *** dims has quit IRC | 02:25 | |
| *** yamamoto has joined #openstack-infra | 02:27 | |
| *** yamamoto has quit IRC | 02:32 | |
| *** dims has joined #openstack-infra | 02:33 | |
| *** sdake has quit IRC | 02:37 | |
| *** wolverineav has joined #openstack-infra | 02:38 | |
| *** adrianreza has joined #openstack-infra | 02:40 | |
| *** sdake has joined #openstack-infra | 02:42 | |
| *** wolverineav has quit IRC | 02:45 | |
| *** wolverineav has joined #openstack-infra | 02:46 | |
| *** wolverineav has quit IRC | 02:47 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:48 | |
| *** lbragstad__ has quit IRC | 02:52 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** gyee has quit IRC | 03:00 | |
| *** ekcs has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| *** psachin has joined #openstack-infra | 03:04 | |
| *** armax has joined #openstack-infra | 03:10 | |
| *** sdake has quit IRC | 03:23 | |
| *** apetrich has quit IRC | 03:24 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** Shrews has quit IRC | 03:27 | |
| *** Shrews has joined #openstack-infra | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:33 | |
| *** ykarel has joined #openstack-infra | 03:36 | |
| *** ccamacho has quit IRC | 03:39 | |
| *** rlandy has quit IRC | 03:50 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur has quit IRC | 04:03 | |
| *** roman_g has quit IRC | 04:04 | |
| *** ramishra has joined #openstack-infra | 04:06 | |
| *** sdake has joined #openstack-infra | 04:10 | |
| *** rascasoft has joined #openstack-infra | 04:16 | |
| *** ramishra has quit IRC | 04:16 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| *** rascasoft has quit IRC | 04:28 | |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** udesale has joined #openstack-infra | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:35 | |
| openstackgerrit | Merged openstack-infra/zuul master: Allow run to be list of playbooks https://review.openstack.org/592160 | 04:38 |
| *** chandankumar has joined #openstack-infra | 04:40 | |
| *** chandankumar has quit IRC | 04:41 | |
| *** chandankumar has joined #openstack-infra | 04:42 | |
| *** rascasoft has joined #openstack-infra | 04:45 | |
| *** ijw_ has quit IRC | 04:49 | |
| *** rascasoft has quit IRC | 04:52 | |
| *** janki has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has joined #openstack-infra | 05:05 | |
| *** ykarel has quit IRC | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:10 | |
| *** wolverineav has joined #openstack-infra | 05:11 | |
| *** zbr|ssbarnea has joined #openstack-infra | 05:13 | |
| *** chandankumar is now known as chkumar|ruck | 05:15 | |
| *** zbr has quit IRC | 05:16 | |
| *** wolverineav has quit IRC | 05:16 | |
| *** sdake has quit IRC | 05:25 | |
| *** ykarel has joined #openstack-infra | 05:26 | |
| *** sdake has joined #openstack-infra | 05:34 | |
| *** tkajinam_ has joined #openstack-infra | 05:41 | |
| *** tkajinam has quit IRC | 05:43 | |
| *** udesale has quit IRC | 05:45 | |
| *** udesale has joined #openstack-infra | 05:55 | |
| *** yamamoto has quit IRC | 06:01 | |
| *** pdeore has joined #openstack-infra | 06:02 | |
| *** verdurin has quit IRC | 06:03 | |
| *** jamesmcarthur has joined #openstack-infra | 06:07 | |
| *** jtomasek has joined #openstack-infra | 06:07 | |
| *** yamamoto has joined #openstack-infra | 06:10 | |
| *** jamesmcarthur has quit IRC | 06:11 | |
| *** sdake has quit IRC | 06:20 | |
| *** jtomasek has quit IRC | 06:27 | |
| *** verdurin has joined #openstack-infra | 06:30 | |
| *** tkajinam_ is now known as tkajinam | 06:34 | |
| *** quique|rover|off is now known as quiquell|rover | 06:42 | |
| *** jamesmcarthur has joined #openstack-infra | 06:44 | |
| *** kjackal has joined #openstack-infra | 06:47 | |
| *** jamesmcarthur has quit IRC | 06:48 | |
| *** ramishra has quit IRC | 06:51 | |
| *** ramishra has joined #openstack-infra | 06:51 | |
| *** dpawlik has joined #openstack-infra | 06:52 | |
| *** whoami-rajat has joined #openstack-infra | 06:53 | |
| *** priteau has joined #openstack-infra | 06:54 | |
| *** roman_g has joined #openstack-infra | 06:55 | |
| *** jtomasek has joined #openstack-infra | 06:57 | |
| *** ramishra has quit IRC | 07:00 | |
| *** ramishra has joined #openstack-infra | 07:01 | |
| *** armax has quit IRC | 07:02 | |
| *** tkajinam_ has joined #openstack-infra | 07:03 | |
| *** lpetrut has joined #openstack-infra | 07:03 | |
| *** tkajinam has quit IRC | 07:05 | |
| *** harlowja has quit IRC | 07:05 | |
| *** jmccrory has quit IRC | 07:06 | |
| *** harlowja has joined #openstack-infra | 07:06 | |
| *** jmccrory has joined #openstack-infra | 07:06 | |
| *** slaweq has joined #openstack-infra | 07:07 | |
| *** AJaeger has quit IRC | 07:09 | |
| *** udesale has quit IRC | 07:15 | |
| *** pgaxatte has joined #openstack-infra | 07:15 | |
| *** AJaeger has joined #openstack-infra | 07:15 | |
| *** kjackal has quit IRC | 07:16 | |
| *** jtomasek has quit IRC | 07:18 | |
| *** priteau has quit IRC | 07:18 | |
| *** jtomasek has joined #openstack-infra | 07:18 | |
| *** udesale has joined #openstack-infra | 07:20 | |
| *** gfidente has joined #openstack-infra | 07:22 | |
| *** snapiri has joined #openstack-infra | 07:26 | |
| *** rascasoft has joined #openstack-infra | 07:26 | |
| *** pcaruana has joined #openstack-infra | 07:29 | |
| *** markvoelker has joined #openstack-infra | 07:39 | |
| *** yboaron has joined #openstack-infra | 07:39 | |
| *** yamamoto has quit IRC | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:41 | |
| *** yamamoto has quit IRC | 07:41 | |
| *** yamamoto has joined #openstack-infra | 07:41 | |
| *** Emine has joined #openstack-infra | 07:49 | |
| *** apetrich has joined #openstack-infra | 07:50 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:51 | |
| *** ykarel is now known as ykarel|lunch | 07:59 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix error reporting for special task failures https://review.openstack.org/635110 | 08:01 |
| *** rpittau has joined #openstack-infra | 08:06 | |
| *** lpetrut has quit IRC | 08:07 | |
| *** dpawlik has quit IRC | 08:07 | |
| *** lpetrut has joined #openstack-infra | 08:08 | |
| *** rossella_s has joined #openstack-infra | 08:08 | |
| *** kopecmartin|off is now known as kopecmartin | 08:09 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:09 | |
| *** kjackal has joined #openstack-infra | 08:10 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** Emine has quit IRC | 08:12 | |
| *** markvoelker has quit IRC | 08:12 | |
| *** dpawlik has joined #openstack-infra | 08:14 | |
| *** tkajinam_ has quit IRC | 08:15 | |
| *** dpawlik has quit IRC | 08:20 | |
| *** panda|off is now known as panda | 08:32 | |
| *** dpawlik has joined #openstack-infra | 08:32 | |
| *** e0ne has quit IRC | 08:33 | |
| *** dpawlik has quit IRC | 08:36 | |
| *** jamesmcarthur has joined #openstack-infra | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:47 | |
| *** jamesmcarthur has quit IRC | 08:50 | |
| *** ramishra has quit IRC | 08:55 | |
| *** tosky has joined #openstack-infra | 08:56 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix error reporting for special task failures https://review.openstack.org/635110 | 08:56 |
| *** ramishra has joined #openstack-infra | 08:57 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| *** e0ne has joined #openstack-infra | 09:01 | |
| *** kjackal has quit IRC | 09:07 | |
| *** hwoarang has quit IRC | 09:07 | |
| *** jpena|off is now known as jpena | 09:09 | |
| *** xek has joined #openstack-infra | 09:09 | |
| *** ociuhandu has quit IRC | 09:09 | |
| *** markvoelker has joined #openstack-infra | 09:09 | |
| *** ociuhandu has joined #openstack-infra | 09:10 | |
| *** lpetrut has quit IRC | 09:14 | |
| *** ociuhandu has quit IRC | 09:15 | |
| *** shardy has joined #openstack-infra | 09:23 | |
| *** hwoarang has joined #openstack-infra | 09:25 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix error reporting for special task failures https://review.openstack.org/635110 | 09:26 |
| *** derekh has joined #openstack-infra | 09:34 | |
| *** markvoelker has quit IRC | 09:43 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:48 | |
| *** takamatsu_ has joined #openstack-infra | 09:57 | |
| *** takamatsu has quit IRC | 09:57 | |
| *** aojea has joined #openstack-infra | 09:59 | |
| *** takamatsu_ has quit IRC | 10:00 | |
| *** ociuhandu has joined #openstack-infra | 10:03 | |
| *** takamatsu_ has joined #openstack-infra | 10:03 | |
| *** ociuhandu has quit IRC | 10:07 | |
| *** Emine has joined #openstack-infra | 10:14 | |
| *** chkumar|ruck has quit IRC | 10:16 | |
| *** chandankumar has joined #openstack-infra | 10:17 | |
| *** Emine has quit IRC | 10:18 | |
| *** chandankumar is now known as chandan|ruck | 10:18 | |
| *** chandan|ruck has quit IRC | 10:20 | |
| *** Emine has joined #openstack-infra | 10:21 | |
| *** yamamoto has quit IRC | 10:23 | |
| *** psachin has quit IRC | 10:24 | |
| *** bhavikdbavishi has quit IRC | 10:30 | |
| *** chandankumar has joined #openstack-infra | 10:30 | |
| *** chandankumar has quit IRC | 10:38 | |
| *** slaweq has quit IRC | 10:40 | |
| *** markvoelker has joined #openstack-infra | 10:40 | |
| *** jamesmcarthur has joined #openstack-infra | 10:46 | |
| *** psachin has joined #openstack-infra | 10:50 | |
| *** jamesmcarthur has quit IRC | 10:51 | |
| *** pdeore has quit IRC | 10:55 | |
| *** slaweq has joined #openstack-infra | 10:57 | |
| *** ykarel is now known as ykarel|mtg | 10:57 | |
| *** Emine has quit IRC | 10:58 | |
| zbr|ssbarnea | can anyone recommend an rss2irc bot? | 11:00 |
| *** slaweq has quit IRC | 11:01 | |
| *** ccamacho has joined #openstack-infra | 11:13 | |
| *** markvoelker has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:17 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** ykarel|mtg is now known as ykarel | 11:34 | |
| *** wolverineav has joined #openstack-infra | 11:36 | |
| *** slaweq has joined #openstack-infra | 11:38 | |
| *** wolverineav has quit IRC | 11:41 | |
| *** ykarel has quit IRC | 11:43 | |
| *** ykarel has joined #openstack-infra | 11:43 | |
| *** takamatsu_ has quit IRC | 11:48 | |
| *** janki has quit IRC | 11:49 | |
| *** ginopc has joined #openstack-infra | 11:51 | |
| *** takamatsu_ has joined #openstack-infra | 11:52 | |
| *** janki has joined #openstack-infra | 11:54 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** Emine has joined #openstack-infra | 12:06 | |
| *** markvoelker has joined #openstack-infra | 12:09 | |
| *** janki has quit IRC | 12:23 | |
| *** takamatsu_ has quit IRC | 12:24 | |
| *** takamatsu has joined #openstack-infra | 12:24 | |
| *** jpich has quit IRC | 12:27 | |
| *** jpich has joined #openstack-infra | 12:27 | |
| *** jpena is now known as jpena|lunch | 12:28 | |
| *** ociuhandu has joined #openstack-infra | 12:35 | |
| *** ociuhandu has quit IRC | 12:35 | |
| *** ociuhandu has joined #openstack-infra | 12:35 | |
| *** rf0lc0 has joined #openstack-infra | 12:35 | |
| *** ociuhandu has quit IRC | 12:35 | |
| *** ociuhandu has joined #openstack-infra | 12:36 | |
| *** yamamoto has joined #openstack-infra | 12:37 | |
| *** rfolco has quit IRC | 12:37 | |
| *** sdake has joined #openstack-infra | 12:41 | |
| *** markvoelker has quit IRC | 12:42 | |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update SQLAlchemy url in docs https://review.openstack.org/635148 | 12:47 |
| *** jamesmcarthur has quit IRC | 12:50 | |
| *** bobh has joined #openstack-infra | 12:58 | |
| clarkb | johnsom: rm_work https://github.com/openstack/python-octaviaclient/pull/1 debian would like an update to make octavia client doc builds reproduceable | 12:59 |
| *** rlandy has joined #openstack-infra | 13:06 | |
| *** ociuhandu_ has joined #openstack-infra | 13:09 | |
| *** ociuhandu_ has quit IRC | 13:11 | |
| *** ociuhandu_ has joined #openstack-infra | 13:12 | |
| *** ociuhandu has quit IRC | 13:12 | |
| openstackgerrit | Merged openstack-infra/zuul master: Retry jobs on winrm send_input failed https://review.openstack.org/631964 | 13:13 |
| *** kjackal has joined #openstack-infra | 13:14 | |
| *** rh-jelabarre has joined #openstack-infra | 13:14 | |
| *** rf0lc0 is now known as rfolco | 13:14 | |
| *** bobh has quit IRC | 13:15 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Revert "Revert "Add a timeout for the image build"" https://review.openstack.org/633792 | 13:16 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove unused fields from AWS driver https://review.openstack.org/634768 | 13:19 |
| *** sdake has quit IRC | 13:23 | |
| *** sdake has joined #openstack-infra | 13:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add documentation for AWS/EC2 driver https://review.openstack.org/634769 | 13:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Rename aws flavor-name to instance-type https://review.openstack.org/635153 | 13:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove aws flavor-name backwards compat https://review.openstack.org/635154 | 13:27 |
| *** sdake has quit IRC | 13:27 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix error reporting for special task failures https://review.openstack.org/635110 | 13:28 |
| *** jpena|lunch is now known as jpena | 13:31 | |
| *** mriedem has joined #openstack-infra | 13:32 | |
| *** jamesdenton has quit IRC | 13:33 | |
| *** sdake has joined #openstack-infra | 13:34 | |
| *** janki has joined #openstack-infra | 13:35 | |
| *** trown|outtypewww is now known as trown | 13:35 | |
| *** slaweq has quit IRC | 13:35 | |
| mordred | clarkb: if you get bored, https://review.openstack.org/#/c/630889 is a devstack patch from ianw to support using bindep in devstack plugins (and its parent patch https://review.openstack.org/#/c/624852) | 13:36 |
| *** kjackal has quit IRC | 13:37 | |
| *** gcutrini has joined #openstack-infra | 13:39 | |
| *** sdake has quit IRC | 13:39 | |
| *** markvoelker has joined #openstack-infra | 13:39 | |
| mnaser | infra-root: i left this up for one of you to +W because it has a large impact and it would be good to have someone around when we +w it -- https://review.openstack.org/#/c/635014/1 | 13:40 |
| *** tosky has quit IRC | 13:44 | |
| *** sdake has joined #openstack-infra | 13:46 | |
| *** jamesdenton has joined #openstack-infra | 13:51 | |
| *** ykarel has quit IRC | 13:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-tempest-dsvm-full-drbd-devstack https://review.openstack.org/634972 | 13:52 |
| *** yamamoto has quit IRC | 14:00 | |
| *** sdake has quit IRC | 14:00 | |
| *** yamamoto has joined #openstack-infra | 14:00 | |
| *** yamamoto has quit IRC | 14:00 | |
| *** iurygregory has joined #openstack-infra | 14:00 | |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| *** sdake has joined #openstack-infra | 14:02 | |
| *** jtomasek has quit IRC | 14:04 | |
| *** jtomasek has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set explicit version for k8s-on-openstack https://review.openstack.org/634820 | 14:05 |
| *** yamamoto has quit IRC | 14:05 | |
| *** yamamoto has joined #openstack-infra | 14:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Increase number of k8s nodes to 4 https://review.openstack.org/628031 | 14:07 |
| iurygregory | Question about zuulv3 if someone knows: I'm going to inherit a job in python-ironicclienct using a job defined in ironic, but I would like the local.conf to not have a config that comes from the job I'm inheriting (this config it's not boolean so I can't set to False) | 14:07 |
| *** jtomasek has quit IRC | 14:08 | |
| *** jtomasek has joined #openstack-infra | 14:08 | |
| *** psachin has quit IRC | 14:10 | |
| *** sdake has quit IRC | 14:12 | |
| *** markvoelker has quit IRC | 14:13 | |
| mnaser | infra-root: just a heads up, i just saw this looking through logs -- NoConnectedServersError('No connected Gearman servers',) | 14:16 |
| mnaser | in the "Submit logstash processing jobs to log processors" task | 14:16 |
| *** sdake has joined #openstack-infra | 14:16 | |
| *** jcoufal has joined #openstack-infra | 14:17 | |
| *** sdake has quit IRC | 14:17 | |
| *** kgiusti has joined #openstack-infra | 14:18 | |
| *** ykarel has joined #openstack-infra | 14:19 | |
| *** cristoph has joined #openstack-infra | 14:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add documentation for AWS/EC2 driver https://review.openstack.org/634769 | 14:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Rename aws flavor-name to instance-type https://review.openstack.org/635153 | 14:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove aws flavor-name backwards compat https://review.openstack.org/635154 | 14:27 |
| *** eharney has joined #openstack-infra | 14:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Optionally silence git push in mirror-workspace-git-repos https://review.openstack.org/635166 | 14:33 |
| *** sdake has joined #openstack-infra | 14:34 | |
| *** gcutrini has quit IRC | 14:36 | |
| *** jtomasek has quit IRC | 14:44 | |
| *** betherly has joined #openstack-infra | 14:47 | |
| frickler | infra-root: mnaser: something still seems broken with http://status.openstack.org/elastic-recheck/ , maybe someone can debug further? note the year: "Last Elastic Search Index Update: Sun Feb 07 2106 06:29:57 GMT+0000 (UTC)" | 14:48 |
| *** wolverineav has joined #openstack-infra | 14:49 | |
| *** sdake has quit IRC | 14:50 | |
| *** betherly has quit IRC | 14:51 | |
| johnsom | clarkb: I | 14:52 |
| johnsom | am not 100% sure what they mean, but I will look into it. Thanks for the heads up. | 14:52 |
| *** wolverineav has quit IRC | 14:53 | |
| *** sdake has joined #openstack-infra | 14:58 | |
| AJaeger | johnsom: follow to the debian bug report, there's a patch | 14:58 |
| fungi | frickler: yeah, likely related to mnaser's observation about the log processor complaining its gearman connection went away | 14:59 |
| *** janki has quit IRC | 15:00 | |
| *** agopi|off is now known as agopi | 15:01 | |
| *** bobh has joined #openstack-infra | 15:03 | |
| *** quiquell|rover is now known as quique|rover|eat | 15:04 | |
| *** markvoelker has joined #openstack-infra | 15:10 | |
| dmsimard | zuul v3 live status in polygerrit (sharing from #zuul): https://gerrit.cesnet.cz/c/dummy/+/1389 | 15:11 |
| dmsimard | Ah, jobs have finished so the "live" status is no longer live but there was progress bars and everything | 15:12 |
| dmsimard | actual patch: https://gerrit-review.googlesource.com/c/plugins/zuul-status/+/213036 | 15:12 |
| clarkb | we had/have that but it killed zuul iirc | 15:12 |
| fungi | in fact the code still lives in the hideci.js overlay and is simply disabled by a variable | 15:13 |
| *** gcutrini has joined #openstack-infra | 15:13 | |
| fungi | in our case, we have (at least) hundreds of active contributors with likely dozens of gerrit changes open in different browser tabs. with those all continuously slamming the zuul api it got really overwhelmed | 15:14 |
| dmsimard | ah, that sort of makes sense | 15:14 |
| fungi | though back then, the status api was served by the main scheduler process | 15:15 |
| fungi | we might be able to withstand it with the status api served by the zuul-web daemon now | 15:15 |
| fungi | corvus: bad news, first new dkim validation failure i looked into this morning may be an entirely different cause. when exim received http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002410.html it had <> wrapping all the bare e-mail addresses in the from and to headers. when i received it, the <> had been stripped from all of them. maybe the deduplication pipeline does that too? | 15:16 |
| fungi | but seems like a stretch | 15:16 |
| fungi | perhaps it does in service of being able to more easily match and remove subscribers in the to header, but why would it alter the from header in that case? | 15:18 |
| *** diablo_rojo has joined #openstack-infra | 15:19 | |
| *** armax has joined #openstack-infra | 15:20 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for VolumeAttachmentNotFound c-vol bug 1814916 https://review.openstack.org/635190 | 15:25 |
| openstack | bug 1814916 in Cinder "MessagingTimeout in cinder-api during volume attachment update due to VolumeAttachmentNotFound in cinder-volume" [Medium,Confirmed] https://launchpad.net/bugs/1814916 | 15:25 |
| *** jamesmcarthur has joined #openstack-infra | 15:26 | |
| fungi | corvus: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002411.html (from the same user) has the same difference. he's one of the ones i previously observed consistently showing invalid dkim signatures, so i guess this is why | 15:27 |
| *** ociuhandu_ has quit IRC | 15:28 | |
| *** ociuhandu has joined #openstack-infra | 15:29 | |
| fungi | corvus: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002414.html also shows an invalid signature, and has a missing entry in the cc header, so we're definitely seeing multiple occurrences of the deduplication pipeline fouling things up | 15:30 |
| fungi | same for http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002429.html http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002430.html http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002431.html http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002432.html | 15:35 |
| fungi | http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002433.html http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002434.html | 15:35 |
| *** ekultails has joined #openstack-infra | 15:36 | |
| fungi | by far the most frequent cause from what i'm seeing | 15:36 |
| fungi | infra-root: it would be great to get https://review.openstack.org/635078 approved so we can more easily see what's still getting munged by the listserv | 15:40 |
| fungi | puppet-infra-core: ^ | 15:40 |
| *** quique|rover|eat is now known as quiquell|rover | 15:40 | |
| fungi | er, infra-puppet-core: ^ | 15:41 |
| *** pgaxatte has quit IRC | 15:41 | |
| fungi | infra-root: config-core: i'm approving the project-config base jobs cleanup now and keeping an eye on things https://review.openstack.org/635014 | 15:42 |
| mnaser | thanks fungi :) | 15:42 |
| frickler | fungi: corvus: +2 on https://review.openstack.org/635078 , feel free to merge when you can watch it | 15:42 |
| fungi | thanks frickler! | 15:43 |
| fungi | will hopefully get back around to that one shortly | 15:43 |
| dmsimard | Zuul looks a bit backlogged, is there something specific going on ? Lots of events in queue and despite all jobs for a change returning successfully, it still hasn't returned to gerrit | 15:43 |
| *** markvoelker has quit IRC | 15:43 | |
| dmsimard | graphs are a bit spiky but nothing in specific is jumping at me | 15:45 |
| *** zbr has joined #openstack-infra | 15:46 | |
| fungi | pretty good sized inbound traffic spike on the scheduler http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64796&rra_id=all | 15:46 |
| dmsimard | Seeing a lot of messages like this one: 2019-02-06 15:46:57,923 DEBUG zuul.Scheduler: Submitting tenant reconfiguration event for openstack due to event <GitTriggerEvent ref-updated gerrit oldrev:0000000000000000000000000000000000000000 newrev:2286c0211d673d2fc6f5b5cf03819a751ca0c971> in project gerrit | 15:47 |
| dmsimard | with different newrev's | 15:47 |
| *** zbr|ssbarnea has quit IRC | 15:47 | |
| *** udesale has joined #openstack-infra | 15:48 | |
| AJaeger | corvus: there's a config error in opendev/base-jobs, see http://zuul.openstack.org/config-errors | 15:49 |
| AJaeger | hope that 635014 fixes it... | 15:49 |
| fungi | assuming 635014 manages to merge | 15:49 |
| *** bhavikdbavishi has joined #openstack-infra | 15:50 | |
| corvus | AJaeger: if it didn't fix it, zuul would have told us | 15:50 |
| AJaeger | ;) | 15:50 |
| AJaeger | great! | 15:51 |
| AJaeger | wow, "Queue lengths: 951 events, 1 management events, 428 results." - haven't seen that in ages... | 15:52 |
| dmsimard | AJaeger: yes, I was trying to understand what is going on -- there seems to be some loop going on | 15:52 |
| AJaeger | at least it's decreasing - now only 910 events | 15:53 |
| clarkb | results can produce events like those that cause reconfigurations | 15:55 |
| clarkb | but in general it should run out of work that produces results eventually and causeit to catch up on events | 15:56 |
| corvus | no result in opendev's zuul should cause a ref-updated on gerrit's own gerrit. | 15:56 |
| dmsimard | clarkb: to give you an idea, it is about ~36 minutes behind right now | 15:57 |
| dmsimard | clarkb: all the jobs for a change completed 36 minutes ago and it still hasn't returned to gerrit | 15:57 |
| dmsimard | s/about/at least/ | 15:57 |
| fungi | dmsimard: which is roughly how long it's been since the anomalous traffic on eth0 started | 15:57 |
| fungi | so presumably zuul fetching *lots* of something | 15:57 |
| fungi | s/fetching/querying/ ? | 15:58 |
| *** jtomasek has joined #openstack-infra | 15:59 | |
| dmsimard | It's churning through the queue faster than it is growing right now so it'll eventually catch up but I'm still curious what happened :) | 16:00 |
| *** ykarel is now known as ykarel|away | 16:01 | |
| *** quiquell|rover is now known as quiquell|off | 16:01 | |
| corvus | it appears a very large number of refs changes on gerrit.googlesource.com | 16:02 |
| dmsimard | the timing appears to roughly correlate when a batch of nova changes with dependant changes landed in the check pipeline | 16:02 |
| corvus | unfortunately, we did not log which refs, so there's not much of a clue as to what. | 16:02 |
| fungi | ahh, right, we do have a connection there | 16:04 |
| clarkb | for dockerimage building iirc | 16:04 |
| fungi | i went digging in our gerrit thinking we might have a massive activity spike or something | 16:04 |
| dmsimard | corvus: does GitTriggerEvent know about which gerrit connection a ref update comes from ? could that be added to the logs ? | 16:04 |
| corvus | the log line from dmsimard bears close reading :) | 16:04 |
| fungi | "in project gerrit" should have been a tip-off, yes | 16:04 |
| corvus | and a GitTriggerEvent means it comes from a git-driver-based connection, not a gerrit-driver one | 16:05 |
| fungi | right-o | 16:05 |
| corvus | dmsimard: i believe the correspending "Scheduling event from" line does contain the connection name | 16:06 |
| corvus | which ... is apparently not logged by the git driver :( | 16:07 |
| dmsimard | corvus: indeed, I meant for another format of log line, hang on | 16:07 |
| *** gcutrini has quit IRC | 16:09 | |
| corvus | oh, zuul logs it, but we don't. | 16:10 |
| dmsimard | zuul is caught up \o/ | 16:10 |
| AJaeger | yeah \o/ | 16:10 |
| *** zbr|ssbarnea has joined #openstack-infra | 16:10 | |
| fungi | excellent | 16:10 |
| *** gcutrini has joined #openstack-infra | 16:11 | |
| *** tosky has joined #openstack-infra | 16:11 | |
| dmsimard | it looks like we'll have a bit of a thundering herd according to graphs because no builds started for a while | 16:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update git connection logging https://review.openstack.org/635204 | 16:13 |
| *** zbr has quit IRC | 16:13 | |
| *** rossella_s has quit IRC | 16:16 | |
| *** zbr has joined #openstack-infra | 16:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove some base jobs https://review.openstack.org/635014 | 16:23 |
| fungi | okay, now that's ^ in, checking on zuul for a bit | 16:24 |
| *** agopi is now known as agopi|brb | 16:25 | |
| fungi | looks like it hasn't reconfigured yet (at least the config error hasn't cleared yet) | 16:25 |
| *** zbr|ssbarnea has quit IRC | 16:26 | |
| *** xek has quit IRC | 16:26 | |
| *** xek has joined #openstack-infra | 16:26 | |
| *** ramishra has quit IRC | 16:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add storyboard01 to Ansible inventory https://review.openstack.org/633420 | 16:29 |
| *** gyee has joined #openstack-infra | 16:32 | |
| *** agopi|brb has quit IRC | 16:32 | |
| *** jamesmcarthur has quit IRC | 16:34 | |
| *** jtomasek has quit IRC | 16:35 | |
| *** jpich has quit IRC | 16:36 | |
| *** jpich has joined #openstack-infra | 16:36 | |
| *** jpich has quit IRC | 16:39 | |
| *** jpich has joined #openstack-infra | 16:39 | |
| *** markvoelker has joined #openstack-infra | 16:40 | |
| *** jpich has quit IRC | 16:44 | |
| *** jamesmcarthur has joined #openstack-infra | 16:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove openstack/instack https://review.openstack.org/635221 | 16:52 |
| *** agopi|brb has joined #openstack-infra | 16:52 | |
| corvus | fungi: config error has cleared now | 16:53 |
| fungi | yep, just saw it as well | 16:53 |
| corvus | mordred: how do you want to handle the gitea, pxc, and rook playbooks? | 16:54 |
| corvus | mordred: just want to stick them in run_all.sh as-is? | 16:54 |
| corvus | fungi: i just watched a job complete and upload logs | 16:57 |
| *** agopi|brb is now known as agopi | 16:58 | |
| corvus | clarkb, fungi: could we get an ssl cert for insecure-ci-registry.opendev.org? (yes, i see the irony) | 16:59 |
| clarkb | I guess if we snakeoil we have to make docker trust it somehow | 17:00 |
| *** sdake has quit IRC | 17:00 | |
| corvus | yes, we can do that (i do that for the buildset registries) but i'd rather not with this one, as eventually i'd like folks to be able to pull images from it fairly easily | 17:00 |
| corvus | (that won't work right off the bat, because the auth isn't sophisticated enough, but i'd like to plan for it) | 17:01 |
| *** udesale has quit IRC | 17:04 | |
| mordred | corvus: that's a great question | 17:05 |
| mordred | corvus: I'd sort of love to do zuul-cd for them ... but I think this tail is super long as it is - so maybe just run_all.sh for now, and we can handle breaking things out into individual things for zuul to trigger in the future? | 17:06 |
| clarkb | fwiw I think my changes ti make zuul cd work are still up for review | 17:07 |
| corvus | clarkb: oh i thought those landed | 17:07 |
| *** cristoph has quit IRC | 17:08 | |
| clarkb | https://review.openstack.org/#/c/604925/ | 17:08 |
| corvus | that's why i thought it had landed :) | 17:09 |
| clarkb | looks like I need to rebase and add another fix for thing pabelamger noticed | 17:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Rename aws flavor-name to instance-type https://review.openstack.org/635153 | 17:10 |
| fungi | corvus: ssl cert seems fine. if clarkb is busy, i think i still have one purchased at namecheap i haven't activated yet which we could use for that | 17:10 |
| clarkb | Im catching up on things at home but will try to swing around for updating things like that and doing reviews in a bit | 17:10 |
| clarkb | fungi you should go for it as I need to unpack and do laundry and a grocery run | 17:10 |
| fungi | corvus: however the domain validation via dns record requires a bit of a review dance for the opendev.org zone | 17:11 |
| corvus | fungi: understood | 17:11 |
| corvus | i'll update run_all.sh and after clarkb refreshes his patches, we can switch to zuulcd | 17:11 |
| mordred | corvus: ++ | 17:11 |
| *** markvoelker has quit IRC | 17:13 | |
| fungi | since zuul seems to be under control, i'm approving the mailman deduplication pipeline removal change (635078) and then getting started on the ssl cert for insecure-ci-registry.opendev.org while the former winds its way through the bowels of our ci/cd automation | 17:14 |
| *** sdake has joined #openstack-infra | 17:15 | |
| *** ociuhandu_ has joined #openstack-infra | 17:16 | |
| fungi | infra-root: i suspect we're failing to allocate any fedora-28 nodes at the moment... there's a trupleo-ci change sitting in check waiting for over two days now | 17:17 |
| fungi | er, tripleo-ci | 17:17 |
| fungi | also possible the node request for it got lost somehow, i guess | 17:17 |
| fungi | wild speculations. i haven't looked into it at all, just saw it | 17:17 |
| *** ociuhandu has quit IRC | 17:17 | |
| *** ginopc has quit IRC | 17:18 | |
| Shrews | fungi: hrm, i'll see if there's anything in nodepool logs | 17:18 |
| mordred | infra-root: I'm going to be AFK this afternoon starting in 1.5 hours | 17:19 |
| *** pcaruana has quit IRC | 17:19 | |
| mordred | corvus: there actually isn't a playbook for rook/xtradb ... you're working on run_all.sh - would you like me to work on a playbook? or is that already on your TDL | 17:20 |
| fungi | infra-root: see the build failure (you have to toggle ci and look at zuul's review comment) for openstack-tox-py36 on https://review.openstack.org/635194 | 17:20 |
| fungi | i have a feeling this is somehow related to the base jobs cleanup/move | 17:20 |
| fungi | maybe a race during reconfiguration? | 17:21 |
| corvus | mordred: there are playbooks in system-config/kubernetes/rook/rook-playbook.yaml | 17:21 |
| corvus | same for percona-xtradb-cluster | 17:21 |
| mordred | oh! so there are | 17:21 |
| mordred | corvus: https://review.openstack.org/#/c/626054/15/playbooks/k8s/storage-class.yaml is what confused me | 17:22 |
| mordred | corvus: that should just be deleted - there's already a storage-class.yaml in the xtradb dir | 17:22 |
| corvus | fungi: yes, i expect that's a race... inasmuch as probably every job which started before the reconfig is going to do that | 17:22 |
| corvus | knowing what we know now, i guess we should have removed the jobs first, while leaving the playbooks. | 17:23 |
| corvus | then removed the playbooks later. | 17:23 |
| corvus | not much we can do about it now, except maybe issue a status notice | 17:23 |
| fungi | on it | 17:23 |
| *** jamesmcarthur has quit IRC | 17:24 | |
| *** _walshh2 has joined #openstack-infra | 17:25 | |
| Shrews | fungi: i don't see any issue with nodepool, atm. a fedora-28 node was created and used a few minutes ago, afaict | 17:25 |
| *** jamesmcarthur_ has joined #openstack-infra | 17:25 | |
| fungi | #status notice Any changes failed around 16:30 UTC today with a review comment from Zuul like "ERROR Unable to find playbook" can be safely rechecked; this was an unanticipated side effect of our work to move base job definitions between configuration repositories. | 17:26 |
| openstackstatus | fungi: sending notice | 17:26 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove openstack/instack https://review.openstack.org/635221 | 17:26 |
| clarkb | fungi: re fedora there was a dbus issue on 29 ianw was fixing | 17:26 |
| clarkb | unsure of 28 | 17:26 |
| fungi | Shrews: yeah, so presumably something went sideways with the request for that particular job? | 17:26 |
| fungi | s/job/build/ | 17:26 |
| -openstackstatus- NOTICE: Any changes failed around 16:30 UTC today with a review comment from Zuul like "ERROR Unable to find playbook" can be safely rechecked; this was an unanticipated side effect of our work to move base job definitions between configuration repositories. | 17:27 | |
| *** jamesmcarthur_ has quit IRC | 17:27 | |
| *** ykarel|away has quit IRC | 17:28 | |
| Shrews | fungi: seems probable | 17:28 |
| openstackstatus | fungi: finished sending notice | 17:29 |
| corvus | incoming! | 17:29 |
| *** lucasagomes is now known as lucas-afk | 17:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Prevent gitea pods from being co-scheduled https://review.openstack.org/629655 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use project-config when syncing gitea repos https://review.openstack.org/629683 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update to gitea v1.6.3 https://review.openstack.org/629942 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Trigger deployment with gitea 1.6.3 https://review.openstack.org/635016 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update to gitea 1.7.1 https://review.openstack.org/634565 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Trigger deployment with gitea 1.7.1 https://review.openstack.org/635017 | 17:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run the gitea k8s playbooks https://review.openstack.org/635227 | 17:30 |
| corvus | mordred, fungi: ^ the last change is new, the rest are rebase/reordering | 17:30 |
| JpMaxMan | Greetings would someone mind adding my user to this group https://review.openstack.org/#/admin/groups/1985,members | 17:30 |
| corvus | i believe i have the right order so that we land the current configuration right before we land the change to actually run the playbooks, then we proceed with changes to perform upgrades which should exercise it. | 17:30 |
| corvus | JpMaxMan: done | 17:31 |
| *** sdake has quit IRC | 17:31 | |
| JpMaxMan | corvus: thanks! | 17:31 |
| _walshh2 | Hi all, hoping someone will be able to help me. We have an OpenStack project that we migrated from private github but we don't know how to create a stable branch from it through gerrit. Any ideas? | 17:32 |
| _walshh2 | https://github.com/openstack/devstack-plugin-vmax | 17:32 |
| *** e0ne has quit IRC | 17:33 | |
| *** shardy has quit IRC | 17:33 | |
| fungi | _walshh2: it looks like https://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/devstack-plugin-vmax.config isn't set up with create permissions for refs/heads/* so see https://docs.openstack.org/infra/manual/creators.html#creation-of-branches for instructions | 17:34 |
| *** rpittau has quit IRC | 17:34 | |
| _walshh2 | Thank you! | 17:35 |
| *** sdake has joined #openstack-infra | 17:35 | |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| fungi | my pleasure! | 17:36 |
| openstackgerrit | Merged openstack-infra/puppet-mailman master: Remove AvoidDuplicates from mailman pipeline https://review.openstack.org/635078 | 17:39 |
| *** bhavikdbavishi has quit IRC | 17:40 | |
| fungi | i'll start tailing syslog on lists.o.o now that's merged | 17:42 |
| *** kopecmartin is now known as kopecmartin|off | 17:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:42 | |
| fungi | last pulse hit the server around 17:30z so hopefully it'll make it into the next one around 18:15z | 17:43 |
| *** trown is now known as trown|lunch | 17:45 | |
| corvus | mordred: could you give https://review.openstack.org/635227 a once-over? | 17:47 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add documentation for AWS/EC2 driver https://review.openstack.org/634769 | 17:48 |
| corvus | i have 3 tasks i'm working on now: push gitea-k8s changes through, set up the intermediate registry, and work on redirect rules for git->gitea | 17:48 |
| *** hwoarang has quit IRC | 17:49 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zone-opendev.org master: Domain validation for insecure-ci-registry https://review.openstack.org/635232 | 17:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zone-opendev.org master: Revert "Domain validation for insecure-ci-registry" https://review.openstack.org/635233 | 17:52 |
| fungi | corvus: ^ | 17:52 |
| Shrews | infra-root: i'd like to try restarting the builder on nb03 again with the new build timeout code in place. any objections? | 17:52 |
| corvus | Shrews: ++ | 17:52 |
| *** hwoarang has joined #openstack-infra | 17:53 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Domain validation for insecure-ci-registry https://review.openstack.org/635232 | 17:53 |
| Shrews | corvus: woohoo! we have logs running as a daemon | 17:58 |
| Shrews | how excitingly unexciting | 17:58 |
| Shrews | going to let this run a while before doing the other builders | 17:59 |
| corvus | Shrews: the simple pleasures in life! | 17:59 |
| *** owalsh has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:01 | |
| fungi | my current foci are supporting the gitea soft-beta-prod rollout, continuing investigation of dmarc invalidation for the openstack-discuss ml, and xenial upgrades for the storyboard and openstackid servers | 18:01 |
| *** xek_ has joined #openstack-infra | 18:02 | |
| fungi | speaking of xenial upgrades, https://review.openstack.org/635073 is the next missing bit for openstackid (a fairly trivial puppet classvar plumbing exercise) | 18:03 |
| *** xek has quit IRC | 18:04 | |
| *** owalsh has joined #openstack-infra | 18:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add resources for deploying rook and xtradb to kuberenets https://review.openstack.org/626054 | 18:08 |
| *** markvoelker has joined #openstack-infra | 18:10 | |
| *** jpena is now known as jpena|off | 18:12 | |
| fungi | http://logs.openstack.org/83/629683/3/check/openstackci-beaker/a509678/ara-report/result/4ba369b0-c528-4f3e-a657-01f8f6c37ebf/ "rspec ./spec/acceptance/nodepool_builder_spec.rb:38 # nodepool builder should be idempotent" | 18:13 |
| fungi | i guess that's racy? | 18:13 |
| clarkb | if nodepool merges code in that window ya | 18:14 |
| openstackgerrit | Merged openstack-infra/system-config master: Add gitea k8s resource definitions and playbook https://review.openstack.org/626759 | 18:14 |
| clarkb | we have that problem.ina couple places because pip install will run again making it not idempote t | 18:14 |
| openstackgerrit | Merged openstack-infra/system-config master: Add modified repo header template https://review.openstack.org/629929 | 18:16 |
| openstackgerrit | Merged openstack-infra/system-config master: Set APP_NAME for gitea https://review.openstack.org/629898 | 18:16 |
| *** sdake has quit IRC | 18:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow git-test to use mod_rewrite https://review.openstack.org/635238 | 18:18 |
| corvus | clarkb, fungi: ^ can you merge that asap? i've hit a wall :( | 18:18 |
| *** sdake has joined #openstack-infra | 18:18 | |
| fungi | looking | 18:19 |
| fungi | quick-aproved | 18:20 |
| fungi | infra-root: the /srv/mailman/*/etc/mm_cfg_local.py files on lists.o.o were updated by 635078 at 18:15 utc to exclude the AvoidDuplicates pipeline from the GLOBAL_PIPELINE list. i'm continuing to monitor to make sure things are working for the lists | 18:23 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: [WIP] - URLTrigger based on the timer trigger https://review.openstack.org/635241 | 18:25 |
| fungi | corvus: key/cert/chain for insecure-ci-registry.opendev.org are now available in /root/certs/2019-02-06/ on bridge.o.o and i've un-wip'd 635233 to revert the temporary domain validation cname rr | 18:26 |
| corvus | fungi: thanks! | 18:27 |
| fungi | any time! | 18:27 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Revert "Domain validation for insecure-ci-registry" https://review.openstack.org/635233 | 18:27 |
| JpMaxMan | greetings is there something more I need to do other than +2 to get this change to merge? https://review.openstack.org/635240 | 18:34 |
| corvus | JpMaxMan: we probably need to add the project to zuul | 18:35 |
| corvus | JpMaxMan: i'll push up a change | 18:35 |
| JpMaxMan | corvus: oh ok thanks - it says needs verified label - that comes from zuul? | 18:36 |
| corvus | JpMaxMan: oh, actually, i think we're all set on the zuul side; i think we just need to add jobs to it | 18:36 |
| corvus | JpMaxMan: yep | 18:36 |
| corvus | JpMaxMan: one second and i'll show you how to update your change to do that | 18:37 |
| JpMaxMan | corvus; great thank you | 18:37 |
| corvus | JpMaxMan: ok, update your change to add a .zuul.yaml file with this content: https://docs.openstack.org/infra/manual/creators.html#add-jobs-for-your-project | 18:38 |
| corvus | JpMaxMan: (so the change should add both .gitreview and .zuul.yaml) | 18:38 |
| JpMaxMan | ok one sec | 18:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add zuul user to bridge.openstack.org https://review.openstack.org/604925 | 18:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 18:40 |
| clarkb | infra-root ^ that fixes the merge conflict issue for zuul cd, but I haven't sorted out known hosts yet. So maybe delay on reviewing that while I figure that out | 18:41 |
| mordred | corvus: run_all.sh patch looks great! | 18:41 |
| *** ekcs has joined #openstack-infra | 18:41 | |
| JpMaxMan | corvus: ok done and +2 so should merge now? | 18:42 |
| corvus | JpMaxMan: yep, and it has | 18:42 |
| JpMaxMan | corvus: like magic! thanks :) | 18:43 |
| corvus | JpMaxMan: the "noop" job is fast, so you shouldn't have to wait long between +2,approve and the merge. later on, we can add in jobs that perform test builds, etc. | 18:43 |
| *** markvoelker has quit IRC | 18:43 | |
| JpMaxMan | cool | 18:44 |
| *** ociuhandu_ has quit IRC | 18:47 | |
| *** wolverineav has joined #openstack-infra | 18:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add zuul user to bridge.openstack.org https://review.openstack.org/604925 | 18:50 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 18:50 |
| clarkb | infra-root ok I think I got that right. pabelanger thanks for the earlier review ^ now with known_hosts | 18:50 |
| *** ekultails has quit IRC | 18:53 | |
| fungi | just an update, messages are still coming through to the mailing lists. i haven't yet seen a dkim-signed message to openstack-discuss since the config update was deployed, so not sure yet if it's helping | 18:54 |
| fungi | but at least we don't seem to have broken mailman | 18:54 |
| *** trown|lunch is now known as trown | 18:55 | |
| fungi | going to try to keep reviewing gitea deployment changes but also sb meeting starts in 4 minutes | 18:56 |
| clarkb | fungi: cmurphy: I think that undefined group var thing on etherpad-dev is mostly a non issue for that server but does point out an issue with this upgrade we'll run into with other servers. I think that comes from our hiera.yaml doing a group_vars/{group}.yaml lookup and we set that value in manifests/site.pp for the etherpad-dev server to $group = etherpad-dev | 18:56 |
| clarkb | we don't rely on that for etherpad-dev I guess, but do rely on it elsewhere? | 18:56 |
| clarkb | oh hrm we do rely on that lookup working iwht etherpad-dev, but it works | 18:57 |
| clarkb | in any case we should probably figure that out before doing more hosts, but I think etherpad-dev is otherwise happy | 18:58 |
| fungi | yay! i'm caught up on the topic:opendev-gerrit stack just in time for the sb meeting | 19:00 |
| fungi | (it helped that those last few changes were really small!) | 19:00 |
| pabelanger | clarkb: +2 | 19:02 |
| *** wolverineav has quit IRC | 19:06 | |
| *** dpawlik has joined #openstack-infra | 19:06 | |
| *** wolverineav has joined #openstack-infra | 19:06 | |
| *** ijw has joined #openstack-infra | 19:07 | |
| *** wolverineav has quit IRC | 19:07 | |
| *** wolverineav has joined #openstack-infra | 19:07 | |
| *** ekultails has joined #openstack-infra | 19:09 | |
| *** gfidente is now known as gfidente|afk | 19:09 | |
| *** bhavikdbavishi has quit IRC | 19:10 | |
| *** bobh has quit IRC | 19:12 | |
| clarkb | fungi: cmurphy `hiera -d -c /etc/puppetlabs/puppet/hiera.yaml etherpad-dev_db_user group='etherpad-dev' ::environment='production'` does work on the server | 19:12 |
| clarkb | makes me think that maybe the comment about not using ::group because the variable is set at nodescope is related | 19:12 |
| clarkb | I'm guessing its a scoping problem since puppet4 is more picky about that | 19:12 |
| fungi | sounds likely, agreed | 19:13 |
| clarkb | https://puppet.com/docs/puppet/4.10/lang_scope.html#node-scope doesn't have any hints that I see around how you might refer to nodescope more explicitly | 19:16 |
| clarkb | note if I change it to ::group it does indeed fail to lookup on that hiera command | 19:16 |
| openstackgerrit | Merged openstack-infra/system-config master: Allow git-test to use mod_rewrite https://review.openstack.org/635238 | 19:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Override FQDN defaults for OpenStackID servers https://review.openstack.org/635073 | 19:18 |
| clarkb | best I can tell $group is correct for that var since it isn't at top scope | 19:19 |
| clarkb | maybe cmurphy will know what is up. But I think we should try to get that warning to go away before moving on | 19:19 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory https://review.openstack.org/633930 | 19:31 |
| JpMaxMan | Is there any special permissions group a user has to be in to access the change end point to create a change with the API? We keep getting a 403 forbidden response. | 19:31 |
| clarkb | JpMaxMan: any user should be able to createa a change, but they do need to be authenticated via the api key | 19:32 |
| JpMaxMan | -u 'user:password' --digest correct? | 19:33 |
| clarkb | JpMaxMan: https://review.openstack.org/Documentation/rest-api.html#authentication ya with the /a/ url prefix | 19:35 |
| JpMaxMan | Hmmm that's what we are doing | 19:36 |
| JpMaxMan | Ok will keep digging just wanted to make sure there wasn't a special permission needed to use the API for creating a change. | 19:37 |
| clarkb | JpMaxMan: do you hav ethe json content for the POST? | 19:37 |
| *** wolverineav has quit IRC | 19:38 | |
| JpMaxMan | http://paste.openstack.org/show/744643/ | 19:39 |
| clarkb | JpMaxMan: unless they are baked into gerrit in some unexpected way I don't think there are special things | 19:39 |
| *** wolverineav has joined #openstack-infra | 19:39 | |
| *** markvoelker has joined #openstack-infra | 19:40 | |
| *** wolverineav has quit IRC | 19:41 | |
| *** wolverineav has joined #openstack-infra | 19:41 | |
| fungi | JpMaxMan: and the password you're sending is one you generated at https://review.openstack.org/#/settings/http-password and matches the username you see there? | 19:41 |
| Shrews | #status log Nodepool builders all restarted on commit 6a4a8f with new build timeout feature | 19:41 |
| clarkb | okall 3 required fields are supplied, I would've expected that to work. You are using your username and password from https://review.openstack.org/#/settings/http-password ? | 19:41 |
| openstackstatus | Shrews: finished logging | 19:41 |
| clarkb | Shrews: yay | 19:42 |
| JpMaxMan | Correct | 19:42 |
| Shrews | \o/ | 19:42 |
| * fungi is baffled, but has also never attempted to create a change via the gerrit rest api... does it not need an initial patchset created at the same time? | 19:42 | |
| fungi | i would also expect an entirely different error as long as the account credentials and url are relevant | 19:43 |
| Shrews | infra-root: for those who may have missed it, nodepool builders are all restarted with new build timeout feature. I watched a full build complete on nb03 before restarting nb01 and nb02 | 19:44 |
| mordred | Shrews: woot! | 19:45 |
| clarkb | fungi: JpMaxMan ya maybe confirm some other request works with the /a/ path and the username:password | 19:45 |
| clarkb | like list changes | 19:45 |
| fungi | Shrews: let's pretend i've been gone for two weeks... do you have a one-sentence summary on what a nodepool build timeout does? ;) | 19:45 |
| Shrews | fungi: if a build does not complete within the default of 8 hours, the dib process is killed | 19:46 |
| fungi | an image build, specifically i guess | 19:46 |
| fungi | awesome! | 19:46 |
| Shrews | fungi: yes | 19:46 |
| clarkb | fungi: remember back in november after summit we had builders out to lunch? they did that again end of december/early january | 19:46 |
| clarkb | debugging it seemed to be because the image build was still running after many days | 19:46 |
| fungi | right-o | 19:46 |
| clarkb | this should mitigate that | 19:46 |
| fungi | that's an excellent improvement | 19:46 |
| *** bobh has joined #openstack-infra | 19:49 | |
| JpMaxMan | clarkb yeah other calls seem to work, I'm looping in gcutrini here who is actively working on this, he can provide some more info. I'll also try with my credentials to see if it makes a difference. | 19:51 |
| JpMaxMan | ok - actually - gcutrini that works for me | 19:55 |
| *** jamesmcarthur has quit IRC | 19:55 | |
| JpMaxMan | I'll PM you and we can work through why it doesn't for you | 19:55 |
| fungi | JpMaxMan: just a hunch, but https://review.openstack.org/#/admin/projects/openstack-infra/netlify-sandbox shows that "Require a valid contributor agreement to upload" is set to "TRUE" for that repo | 20:01 |
| fungi | possible you've agreed to the openstack icla and gcutrini has not? | 20:01 |
| JpMaxMan | fungi: good thought though he can use git review. | 20:02 |
| fungi | i don't know if that could cause a 403 forbidden when trying to call the change creation api method, but i suspect it could be | 20:02 |
| fungi | ahh, yeah if git-review is working to push a change to that repo with the same account then i guess that's not it | 20:03 |
| JpMaxMan | actually I just tried using his credentials and it worked, so must be something unique to his environment. | 20:03 |
| fungi | even better. gotta love that kind of failure | 20:03 |
| JpMaxMan | carry on nothing to se here :) | 20:03 |
| *** wolverineav has quit IRC | 20:04 | |
| *** ijw has quit IRC | 20:05 | |
| *** dpawlik has quit IRC | 20:05 | |
| *** wolverineav has joined #openstack-infra | 20:06 | |
| *** ijw has joined #openstack-infra | 20:06 | |
| *** ccamacho has quit IRC | 20:08 | |
| *** Emine has quit IRC | 20:10 | |
| *** wolverineav has quit IRC | 20:10 | |
| *** wolverineav has joined #openstack-infra | 20:11 | |
| *** markvoelker has quit IRC | 20:12 | |
| *** sdake has quit IRC | 20:13 | |
| *** wolverineav has quit IRC | 20:15 | |
| *** aojea has quit IRC | 20:18 | |
| *** ijw has quit IRC | 20:19 | |
| *** dpawlik has joined #openstack-infra | 20:22 | |
| *** sdake has joined #openstack-infra | 20:23 | |
| *** ijw has joined #openstack-infra | 20:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:25 | |
| *** dpawlik has quit IRC | 20:27 | |
| corvus | i've approved the opendev-gerrit stack up through the change which should start running the service playbooks | 20:29 |
| fungi | thanks! | 20:33 |
| fungi | i'm about to go wok up some fried rice, but will return to help with that as soon as i can | 20:33 |
| *** dtantsur is now known as dtantsur|afk | 20:39 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove openstack/instack https://review.openstack.org/635221 | 20:40 |
| *** panda is now known as panda|off | 20:41 | |
| *** jamesmcarthur has joined #openstack-infra | 20:41 | |
| fungi | corvus: mailman seems to still be deduplicating users from cc headers... do we need to restart the mailman-.* queue services for each of our listserv sites for the config changes to take effect? | 20:41 |
| *** jamesmcarthur has quit IRC | 20:41 | |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| corvus | fungi: oh, i bet so. | 20:42 |
| fungi | i can do that real quick then | 20:42 |
| fungi | okay, i've restarted the services for each of the 5 sites on lists.o.o | 20:44 |
| fungi | will continue to monitor new posts for signs of behavior change | 20:44 |
| corvus | i've added the docker registry secrets (cert/password) to private hostvars on bridge | 20:45 |
| *** wolverineav has joined #openstack-infra | 20:50 | |
| *** xek__ has joined #openstack-infra | 20:50 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for VolumeAttachmentNotFound c-vol bug 1814916 https://review.openstack.org/635190 | 20:51 |
| openstack | bug 1814916 in Cinder "MessagingTimeout in cinder-api during volume attachment update due to VolumeAttachmentNotFound in cinder-volume" [Medium,Confirmed] https://launchpad.net/bugs/1814916 | 20:51 |
| *** xek_ has quit IRC | 20:52 | |
| *** wolverineav has quit IRC | 20:54 | |
| *** sdake has quit IRC | 20:57 | |
| *** sdake has joined #openstack-infra | 20:59 | |
| *** e0ne has quit IRC | 21:00 | |
| *** gcutrini has quit IRC | 21:03 | |
| *** wolverineav has joined #openstack-infra | 21:06 | |
| *** markvoelker has joined #openstack-infra | 21:09 | |
| *** slaweq has joined #openstack-infra | 21:12 | |
| openstackgerrit | Merged openstack-infra/system-config master: Prevent gitea pods from being co-scheduled https://review.openstack.org/629655 | 21:13 |
| openstackgerrit | Merged openstack-infra/system-config master: Use project-config when syncing gitea repos https://review.openstack.org/629683 | 21:13 |
| openstackgerrit | Merged openstack-infra/system-config master: Run the gitea k8s playbooks https://review.openstack.org/635227 | 21:13 |
| *** ijw has quit IRC | 21:14 | |
| clarkb | I'm following up on mnasers logstash gearman thing and something seems off with the geard running on logstash01. It does accept connections but then status command doesn't return anything | 21:14 |
| clarkb | and grafana shows a backlog of about a million jobs? | 21:15 |
| *** ijw has joined #openstack-infra | 21:15 | |
| clarkb | comparing then to behavor of the workers it seems that its bogged down. We may just need to restart the geard | 21:16 |
| corvus | that means *something* is working. | 21:16 |
| corvus | yeah. it's entirely possible that it's spinning its wheels; ++restart. | 21:16 |
| *** ociuhandu has joined #openstack-infra | 21:16 | |
| clarkb | 2019-02-06 21:12:38,009 Connecting to logstash.openstack.org port 4730 then 2019-02-06 21:14:45,259 Connected to logstash.openstack.org port 4730 | 21:17 |
| clarkb | fwiw | 21:17 |
| *** xek__ has quit IRC | 21:17 | |
| clarkb | status works post restart, but not seeing workers reregister yet | 21:19 |
| *** kjackal has joined #openstack-infra | 21:20 | |
| clarkb | ok thing sseem to be moving now? I'll go through and make sure all the workers are happy (as they are bound to be in weird states too given the server was out to lunch) | 21:21 |
| *** sdake has quit IRC | 21:23 | |
| *** ijw has quit IRC | 21:27 | |
| *** wolverineav has quit IRC | 21:27 | |
| *** ijw has joined #openstack-infra | 21:27 | |
| *** wolverineav has joined #openstack-infra | 21:28 | |
| *** ijw has quit IRC | 21:28 | |
| *** ijw has joined #openstack-infra | 21:32 | |
| *** wolverineav has quit IRC | 21:32 | |
| *** ijw has quit IRC | 21:37 | |
| clarkb | ya a bunch of workers seemed to not reconnect so I went ahead and restarted them | 21:39 |
| clarkb | stuck in tcp wait land? I dunno | 21:39 |
| *** ociuhandu has quit IRC | 21:40 | |
| clarkb | #status log Restarted Geard on logstash01 and log processing workers on logstash-workerXX. Geard appeared to be out to lunch with a large queue and workers were not reconnecting on their own. | 21:41 |
| openstackstatus | clarkb: finished logging | 21:41 |
| fungi | oh, thanks for picking that back up! it fell off my radar with everything else that came up shortly after i got on the 'puter | 21:42 |
| *** markvoelker has quit IRC | 21:42 | |
| *** ijw has joined #openstack-infra | 21:48 | |
| *** wolverineav has joined #openstack-infra | 21:51 | |
| *** ijw has quit IRC | 21:53 | |
| *** slaweq has quit IRC | 21:55 | |
| *** zbr|ssbarnea has joined #openstack-infra | 22:00 | |
| *** gfidente|afk has quit IRC | 22:00 | |
| *** zbr|ssbarnea has quit IRC | 22:01 | |
| *** zbr has quit IRC | 22:02 | |
| fungi | okay, messages are still arriving to our mailing lists. i was a bit worried since there was a substantial lull following my queue manager restarts | 22:05 |
| clarkb | fungi: no worries, I think it should be happy now | 22:08 |
| *** ijw has joined #openstack-infra | 22:11 | |
| fungi | so far we've had only one post to openstack-discuss with a dkim signature since the process restarts. its signature validates correctly, but the message also doesn't have any of the hallmarks of those i've seen get mangled previously so not really much help yet | 22:15 |
| mgagne | FYI, we are having issues with inap-mtl01, it could take some time (maybe hours) to restore services. | 22:16 |
| *** ijw has quit IRC | 22:16 | |
| fungi | thanks for the heads-up mgagne! | 22:16 |
| smcginnis | Failure on some release job doc publishing. Known issue? " ERROR! the role 'write-root-marker' was not found in" | 22:18 |
| corvus | could be related to the base jobs move | 22:19 |
| corvus | smcginnis: link? | 22:19 |
| smcginnis | http://logs.openstack.org/17/178ce0386c8a9ae07635e20cc2546673edcbd2a6/release-post/publish-tox-docs-static/20eaeb3/job-output.txt.gz#_2019-02-06_22_15_13_414916 | 22:19 |
| *** jcoufal has quit IRC | 22:19 | |
| corvus | i see it, patch inbound | 22:23 |
| smcginnis | corvus: Thanks | 22:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add openstack-zuul-jobs to base-publish-tox-docs-static https://review.openstack.org/635347 | 22:27 |
| corvus | smcginnis, clarkb, fungi: ^ there may be a few more of those. we can continue to fix them as we find them, or seek them out. or we could make an openstack-base job which inherits from base, and adds openstack-zuul-jobs to the role path, then make that the tenant default base job. | 22:28 |
| fungi | that last thing sounds like more work | 22:29 |
| *** kgiusti has left #openstack-infra | 22:31 | |
| corvus | it's definitely good if you get paid by the keystroke | 22:31 |
| *** jcoufal has joined #openstack-infra | 22:31 | |
| corvus | but it's a bit more solid, in that we should end up with the same roles as before we started the move | 22:32 |
| *** agopi is now known as agopi|brb | 22:36 | |
| *** kjackal has quit IRC | 22:36 | |
| *** kjackal has joined #openstack-infra | 22:36 | |
| fungi | i guess i like the thoroughness of it, but if we don't expect many more it could be overkill | 22:38 |
| *** bobh has quit IRC | 22:38 | |
| corvus | there aren't that many roles in ozj, i'd guess < 5 more | 22:39 |
| *** sdake has joined #openstack-infra | 22:39 | |
| *** ekultails has quit IRC | 22:40 | |
| *** markvoelker has joined #openstack-infra | 22:40 | |
| *** agopi|brb has quit IRC | 22:40 | |
| *** roman_g has quit IRC | 22:43 | |
| *** roman_g has joined #openstack-infra | 22:44 | |
| fungi | so if we just git grep in there for the names of the roles we moved, we'll find them? | 22:45 |
| clarkb | in there being all the zuul job playbooks I think | 22:45 |
| fungi | okay, so codesearch.o.o (and hope there aren't any exclusively in non-master branches)? | 22:46 |
| fungi | it's times like this when i look forward to a zuul runtime configuration explorer | 22:47 |
| *** jcoufal has quit IRC | 22:48 | |
| *** jcoufal has joined #openstack-infra | 22:52 | |
| *** tkajinam has joined #openstack-infra | 22:53 | |
| fungi | another dkim-signed message to the ml, but it didn't have any additional recipients in cc | 22:54 |
| *** jcoufal has quit IRC | 22:54 | |
| fungi | odds are i'll have to wait until tomorrow to confirm whether mailman has stopped trying to deduplicate recipients | 22:54 |
| fungi | and also whether mta's like dell's which wrap bare addresses in <> are no longer mangled | 22:55 |
| fungi | (i suppose i could test both of those directly, but i'm not interested in polluting the ml nor fiddling with test lists... we'll know soon enough anyway) | 22:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-zuul-jobs to base-publish-tox-docs-static https://review.openstack.org/635347 | 22:56 |
| *** dmsimard has quit IRC | 22:57 | |
| *** eharney has quit IRC | 22:59 | |
| *** agopi|brb has joined #openstack-infra | 23:01 | |
| *** ijw has joined #openstack-infra | 23:03 | |
| *** agopi|brb is now known as agopi | 23:04 | |
| clarkb | fungi how are you checking dkim signature validity? is there a tool to do that directly? | 23:06 |
| fungi | clarkb: there are quite a few, but my mta does it for me and inserts a header with details | 23:07 |
| clarkb | ah | 23:08 |
| corvus | clarkb: one of which is the python dkim library: http://baruch.siach.name/blog/posts/verify_dkim_signature_from_command_line/ | 23:08 |
| fungi | and i have mutt configured to display that custom header by default, so i'm just looking at the message in mutt and not doing anything special | 23:08 |
| fungi | i either see DKIM_VALID or DKIM_INVALID along with DKIM_SIGNED appearing in the list of flags in my X-Spam-Status headers | 23:09 |
| corvus | incidentally, having made a start on the cgit redirects, i believe i can assure the rest of the team that they will be interesting. :) | 23:10 |
| fungi | i'm quite fond of using spamassassin as an inline exim filter for scoring and optionally rejecting messages with my personal mta | 23:10 |
| corvus | i just wrote my first recursive mod_rewrite rule. | 23:10 |
| mordred | corvus: so what you're telling me is that mod_rewrite is fully turing complete and by the time you're done you will have rewritten the zuul scheulder purely in rewrite rules? | 23:13 |
| *** markvoelker has quit IRC | 23:13 | |
| corvus | mordred: we might need to increase the stack limit for that, apparently it defaults to 10 | 23:14 |
| corvus | but other than that, i can see *no problems* with that plan! | 23:14 |
| corvus | (and yes, shortly after writing my first recursive mod_rewrite rule, i learned the stack limit) | 23:14 |
| mordred | corvus: maybe what you really need to do is bounce back and forth between mod_rewrite and TeX | 23:15 |
| corvus | mordred: i was just circling back around to the gitea thing... apparently it would help if we had ~root/.kube/config on bridge | 23:15 |
| mordred | corvus: yeah. except there's a thing we need to do there ... | 23:16 |
| mordred | corvus: since there is also the nodepool k8s | 23:16 |
| mordred | so we should probably make use of the ability to put more than one k8s cluster into a .kube/config and have various contexts defined | 23:16 |
| corvus | k. i'll see about writing that patch | 23:17 |
| corvus | (we don't actually have the nodepool k8s in root's config on bridge, so it's technically not a problem now, but i agree, especially on bridge, we should start out with support for multi-k8s) | 23:18 |
| mordred | ++ | 23:18 |
| mordred | corvus: fwiw, k8s-on-openstack pulls the k8s config back from the master to $(pwd)/admin.conf - our running of it does that in a /tmp, so we don't have it around from that task directly ... | 23:22 |
| mordred | corvus: but perhaps we could write a something that pulls the file and then does a yaml merge or something ... or maybe that's too much magic - just thinking out loud | 23:23 |
| *** dmsimard has joined #openstack-infra | 23:23 | |
| corvus | mordred: yeah, i'd like to do that, but maybe later? | 23:24 |
| mordred | yeah | 23:24 |
| *** rlandy is now known as rlandy|bbl | 23:28 | |
| *** wolverineav has quit IRC | 23:29 | |
| *** ijw has quit IRC | 23:29 | |
| *** ijw has joined #openstack-infra | 23:30 | |
| *** wolverineav has joined #openstack-infra | 23:30 | |
| fungi | looks like mailman is no longer deduping. i got http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002461.html just now with a valid dkim signature and a bunch of known subscribers still present in the cc | 23:32 |
| fungi | same for a couple more which arrived after it | 23:33 |
| *** wolverineav has quit IRC | 23:34 | |
| corvus | mordred: i feel like the k8s context should be part of the ansible inventory :/ | 23:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Configure .kube/config on bridge https://review.openstack.org/635358 | 23:43 |
| mordred | corvus: yeah. also - we should maybe put the k8s master and nodes into the inventory too | 23:43 |
| corvus | mordred, fungi: ^ | 23:43 |
| mordred | corvus: but probably in a group so that it only gets the base role / users ... I can work on that | 23:44 |
| *** kjackal has quit IRC | 23:44 | |
| mordred | (I feel like I hacked something together for making it run the user creation role before but that doesn't make sense with how it hangs together now) | 23:44 |
| *** wolverineav has joined #openstack-infra | 23:45 | |
| mgagne | fungi: service is back online in inap-mtl01. we had to restore a database backup and I think some Glance images are no longer referenced in the database, therefore Nodepool fails to create new server. Can you check? | 23:46 |
| clarkb | mordred: has to be in the puppet group to get puppeted. Maybe instead of special group for less setup we put the intermediate level of setup in a grouo and no group is most minimal? | 23:53 |
| clarkb | I hope that made sense | 23:53 |
| *** rh-jelabarre has quit IRC | 23:54 | |
| mordred | clarkb: yeah. in this case it definitely doesn't want to be puppeted - but the key is that we only want to run some of the minimal ansible | 23:55 |
| mordred | clarkb: so - yes, I agree with you | 23:55 |
| *** mriedem has quit IRC | 23:56 | |
| mordred | clarkb: or, at least, I do for now until I start writing yaml | 23:56 |
| mordred | let's see what winds up being the least ugly | 23:57 |
| fungi | mgagne: yep! openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://compute.api.mtl01.cloud.iweb.com/v2/addf341f4fa44528b1ac73e2f5c68a21/servers, Can not find requested image | 23:57 |
| fungi | i suppose we can delete them from nodepool and let it reupload | 23:58 |
| mgagne | fungi: can you reupload the missing images? I'm still doing some cleanup on my side. | 23:58 |
| mgagne | according to grafana there are no nodepool nodes left, is that right? | 23:58 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv: handle centos image-based builds https://review.openstack.org/635365 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!