| clarkb | I'll reply to the last email notice I sent out so that each of the main discuss lists gets a copy | 00:00 |
|---|---|---|
| fungi | unless you want to include a link to http://paste.openstack.org/show/749402/ in some announcements to cut down on the volume of messages we're bombarding them all with | 00:00 |
| clarkb | fungi: I can do that. Let me get an etherpad going so that I can be sure I say the right things for that | 00:01 |
| fungi | the only real individualization i had in mind was to let airship, starlingx and zuul know i'd identified their repos via cgit aliases in our gerrit/projects.yaml and in the case of those with repo prefixes (airship and stx) dropped the prefixes as part of that mapping | 00:01 |
| clarkb | fungi: https://etherpad.openstack.org/p/YzSHxshSyj | 00:02 |
| fungi | this should be no surprise to them as these details were communicated with their liaisons | 00:02 |
| clarkb | something like that? | 00:07 |
| fungi | lookin' | 00:08 |
| *** gyee has quit IRC | 00:08 | |
| fungi | lgtm, thanks!!! | 00:08 |
| clarkb | ok I'll get that out to the -discuss lists now | 00:08 |
| *** dangtrinhnt has joined #openstack-infra | 00:11 | |
| fungi | maybe -infra ml too? | 00:11 |
| fungi | that's the closest thing we have to a broadcast notification channel (for now) for unaffiliated projects hosting on opendev | 00:12 |
| clarkb | ++ | 00:12 |
| *** sthussey has quit IRC | 00:13 | |
| clarkb | ok emails sent | 00:14 |
| pabelanger | fungi: just to confirm, official projects like openstack/nova and openstack/glance, get outputted to another yaml file, right? This seems to be outside of the current pastebin | 00:14 |
| clarkb | pabelanger: openstack/nova and openstack/glance stay put | 00:15 |
| clarkb | pabelanger: the openstack/ namespace will become an actual oepnstack namespace | 00:15 |
| pabelanger | ah, okay. that makesense | 00:15 |
| *** jamesmcarthur has joined #openstack-infra | 00:15 | |
| pabelanger | not sure why my brain didn't register that | 00:16 |
| fungi | right, there will be a number of things transformed in the git repos during the maintenance, this is (some semblance of) the input which we'll use to determine which repos are moving into different namespaces (and possibly also losing some name prefixes) | 00:20 |
| fungi | all repos, even those not in the 825 listed there, will also be experiencing hostname moves which will involve some nonzero number of patches pushed into them | 00:21 |
| *** armax has quit IRC | 00:23 | |
| clarkb | fungi: were you going to bring up the system-config (and similar concerns) with the TC? | 00:26 |
| clarkb | I think for functional concerns the redirects will keep things working wherever they go | 00:26 |
| clarkb | system-config/modules.env for example should be served by the redirects | 00:26 |
| fungi | yeah, i think we should consider at least starting the process of moving some of those things out of governance but i'm on the fence for whether we ask to keep them in the openstack-infra namespace or move them somewhere new | 00:29 |
| *** jcoufal has joined #openstack-infra | 00:29 | |
| fungi | officially we'll need your +1 at least on whatever we do with official infra team repos | 00:29 |
| *** lseki has quit IRC | 00:30 | |
| fungi | but first, we need a list of which ones are that | 00:30 |
| *** jamesmcarthur has quit IRC | 00:31 | |
| clarkb | From a migration sanity standpoint it might be worth keeping stuff in openstack-infra/ just so that we have working deployment tools after :) | 00:31 |
| *** jamesmcarthur has joined #openstack-infra | 00:31 | |
| clarkb | but I expect most of them will still work fine after a rename due to the redirects so that is a small concern | 00:31 |
| *** jamesmcarthur has quit IRC | 00:35 | |
| fungi | i don't know that leaving things in the openstack-infra namespace necessarily buys us anything | 00:36 |
| clarkb | ya my biggest worry is some tools potentially won't follow redirects | 00:37 |
| fungi | and whatever moving them breaks, we're basically braced for mounds of potential work unbreaking other stuff anyway so it's just a few more | 00:37 |
| clarkb | and in those cases we'll have to fix whateverbreaks | 00:37 |
| *** diablo_rojo has quit IRC | 00:38 | |
| fungi | there's going to be stuff to untangle no matter what, and it's mostly the stuff we can't predict (because if we could predict it, we'd have already developed contingency plans) | 00:38 |
| clarkb | indeed | 00:38 |
| *** mriedem has quit IRC | 00:38 | |
| *** nicolasbock has quit IRC | 00:39 | |
| *** diablo_rojo has joined #openstack-infra | 00:43 | |
| *** michael-beaver has quit IRC | 00:48 | |
| *** ijw has quit IRC | 00:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Prune the intermediate registry before GCing https://review.openstack.org/653148 | 00:52 |
| clarkb | corvus: ^ I think that makes bashate happier (fwiw the thing it was mad about is a real issue so maybe it caught something worthwhile) | 00:53 |
| *** jcoufal has quit IRC | 00:55 | |
| *** ricolin has joined #openstack-infra | 00:56 | |
| *** jcoufal has joined #openstack-infra | 00:56 | |
| *** ijw has joined #openstack-infra | 01:04 | |
| clarkb | corvus mordred https://github.com/containers/skopeo/issues/631 I've filed that issue with skopeo | 01:06 |
| *** jcoufal has quit IRC | 01:07 | |
| *** ijw has quit IRC | 01:09 | |
| clarkb | and with that I've largely caught up on my todo list after resting today. Exceptions are nodepool + puppet-4 and starting the plan document for friday. I'll pick those up in the morning | 01:09 |
| *** rlandy|ruck has quit IRC | 01:11 | |
| *** bobh has joined #openstack-infra | 01:16 | |
| *** hamzy has joined #openstack-infra | 01:22 | |
| *** armax has joined #openstack-infra | 01:23 | |
| *** threestrands has joined #openstack-infra | 01:24 | |
| *** rosmaita has left #openstack-infra | 01:24 | |
| *** whoami-rajat has joined #openstack-infra | 01:31 | |
| *** hongbin has joined #openstack-infra | 01:37 | |
| *** bobh has quit IRC | 01:40 | |
| *** apetrich has quit IRC | 01:59 | |
| *** diablo_rojo has quit IRC | 02:05 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:08 | |
| *** mriedem has joined #openstack-infra | 02:11 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:14 | |
| *** bobh has joined #openstack-infra | 02:14 | |
| *** bhavikdbavishi has quit IRC | 02:14 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:14 | |
| *** bobh has quit IRC | 02:15 | |
| *** diablo_rojo has joined #openstack-infra | 02:46 | |
| *** bobh has joined #openstack-infra | 02:47 | |
| *** bobh has quit IRC | 02:58 | |
| *** dangtrinhnt has quit IRC | 02:59 | |
| *** dangtrinhnt has joined #openstack-infra | 02:59 | |
| anteaya | clarkb: did you want to update the email I sent to the -discuss list with the subject '[all] Gerrit will be offline April 19, 2019'? | 03:01 |
| anteaya | I had said that post would be updated with a start time | 03:01 |
| anteaya | I can do it tomorrow if you want | 03:01 |
| *** psachin has joined #openstack-infra | 03:12 | |
| *** aaronsheffield has quit IRC | 03:15 | |
| *** ykarel|away has joined #openstack-infra | 03:19 | |
| *** gregoryo has joined #openstack-infra | 03:30 | |
| *** hongbin has quit IRC | 03:38 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update grafana for new archive repo https://review.openstack.org/652443 | 03:38 |
| *** diablo_rojo has quit IRC | 03:44 | |
| *** diablo_rojo has joined #openstack-infra | 03:44 | |
| *** armax has quit IRC | 03:51 | |
| *** imacdonn has quit IRC | 04:07 | |
| *** imacdonn has joined #openstack-infra | 04:07 | |
| *** mriedem has quit IRC | 04:08 | |
| *** david-lyle has joined #openstack-infra | 04:13 | |
| *** manjeets_ has joined #openstack-infra | 04:14 | |
| *** dklyle has quit IRC | 04:16 | |
| *** manjeets has quit IRC | 04:16 | |
| *** hwoarang has quit IRC | 04:16 | |
| openstackgerrit | sean mooney proposed openstack/gertty master: replace MultiQueue with PQueue https://review.openstack.org/653275 | 04:18 |
| openstackgerrit | sean mooney proposed openstack/gertty master: use timers for periodic tasks https://review.openstack.org/653276 | 04:18 |
| *** hwoarang has joined #openstack-infra | 04:18 | |
| openstackgerrit | sean mooney proposed openstack/gertty master: use threadpool for syncing. https://review.openstack.org/653277 | 04:18 |
| *** ykarel|away has quit IRC | 04:27 | |
| *** e0ne has joined #openstack-infra | 04:35 | |
| *** udesale has joined #openstack-infra | 04:37 | |
| *** e0ne has quit IRC | 04:42 | |
| *** ykarel|away has joined #openstack-infra | 04:43 | |
| *** e0ne has joined #openstack-infra | 04:44 | |
| *** armax has joined #openstack-infra | 04:48 | |
| ianw | clarkb / cmurphy: 'If using an operating system of the Debian-based family, and the "repo" install_method, you will need to ensure that puppetlabs-apt version 4.x is installed.' | 04:48 |
| ianw | this is for grafana. the result seems to be that adding the gpg doesn't work with our version of puppetlabs-apt (2.something -- the last one that supports puppet3) | 04:49 |
| cmurphy | :( | 04:49 |
| cmurphy | i don't think we can update that one yet | 04:50 |
| ianw | to not waste time, i have just manually added the key on grafana, and things work now. in the very unlikely event we need to redo grafana we should be aware | 04:50 |
| ianw | cmurphy: yeah, so just a heads up it should probably be something we pull in after the puppet3 excise | 04:50 |
| *** ramishra has joined #openstack-infra | 04:50 | |
| cmurphy | ++ | 04:51 |
| *** yboaron has joined #openstack-infra | 04:52 | |
| ianw | #status log grafana02.openstack.org is now puppeting again with some manual intervention to install the grafana repo gpg key. this key should not change and can remain as is. will need a new version of puppetlabs-apt when puppet4 transition is complete to be fully automatic again. host running grafana 6.1.4 | 04:53 |
| openstackstatus | ianw: finished logging | 04:53 |
| *** armax has quit IRC | 04:55 | |
| *** zhurong has quit IRC | 05:00 | |
| *** armax has joined #openstack-infra | 05:07 | |
| *** zhurong has joined #openstack-infra | 05:08 | |
| *** ykarel|away is now known as ykarel | 05:10 | |
| *** mujahidali has joined #openstack-infra | 05:16 | |
| *** e0ne has quit IRC | 05:16 | |
| *** kjackal has joined #openstack-infra | 05:34 | |
| *** mattw4 has joined #openstack-infra | 05:37 | |
| *** e0ne has joined #openstack-infra | 05:40 | |
| *** _Cyclone_ has quit IRC | 05:46 | |
| *** _Cyclone_ has joined #openstack-infra | 05:47 | |
| *** quiquell|off is now known as quiquell|rover | 05:47 | |
| *** janki has joined #openstack-infra | 05:49 | |
| *** e0ne has quit IRC | 05:59 | |
| *** pcaruana has joined #openstack-infra | 06:11 | |
| *** mattw4 has quit IRC | 06:18 | |
| *** dpawlik has joined #openstack-infra | 06:31 | |
| *** udesale has quit IRC | 06:37 | |
| *** udesale has joined #openstack-infra | 06:38 | |
| *** udesale has quit IRC | 06:39 | |
| *** bhavikdbavishi has quit IRC | 06:40 | |
| *** udesale has joined #openstack-infra | 06:44 | |
| *** diablo_rojo has quit IRC | 06:45 | |
| *** udesale has quit IRC | 06:46 | |
| *** iurygregory has joined #openstack-infra | 06:49 | |
| *** udesale has joined #openstack-infra | 06:56 | |
| *** pgaxatte has joined #openstack-infra | 07:00 | |
| *** ginopc has joined #openstack-infra | 07:01 | |
| *** yolanda_ has joined #openstack-infra | 07:07 | |
| *** kopecmartin|off is now known as kopecmartin | 07:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 07:12 |
| *** rpittau|afk is now known as rpittau | 07:19 | |
| *** tosky has joined #openstack-infra | 07:29 | |
| *** yboaron has quit IRC | 07:31 | |
| *** dtantsur|afk is now known as dtantsur | 07:33 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:33 | |
| *** apetrich has joined #openstack-infra | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** e0ne has joined #openstack-infra | 07:51 | |
| *** kjackal has quit IRC | 07:55 | |
| *** kjackal_v2 has joined #openstack-infra | 07:55 | |
| *** Emine has joined #openstack-infra | 07:59 | |
| *** jpich has joined #openstack-infra | 08:01 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:05 | |
| frickler | fungi: jroll: not sure whether that's been mentioned in backlog already, but openstack/openstack -> x/openstack seems wrong, even if that's in line with the current criteria. also openstack/openstack-map -> x/openstack-map | 08:12 |
| *** rcernin has quit IRC | 08:13 | |
| *** yboaron has joined #openstack-infra | 08:14 | |
| AJaeger | frickler: want to reply to the email on openstack-infra mailing list? Good catch, thanks | 08:14 |
| *** gregoryo has quit IRC | 08:15 | |
| *** tkajinam has quit IRC | 08:17 | |
| *** yboaron_ has joined #openstack-infra | 08:19 | |
| *** lucasagomes has joined #openstack-infra | 08:19 | |
| *** yboaron has quit IRC | 08:21 | |
| frickler | AJaeger: oh, you already found one of those. replied, thx | 08:22 |
| *** rcernin has joined #openstack-infra | 08:27 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| openstackgerrit | Bernard Cafarelli proposed openstack-infra/project-config master: Update Grafana dashboards for stable Neutron releases https://review.openstack.org/653354 | 08:37 |
| *** e0ne has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:42 | |
| *** e0ne has quit IRC | 08:43 | |
| *** e0ne has joined #openstack-infra | 08:43 | |
| *** udesale has quit IRC | 08:51 | |
| *** ccamacho has joined #openstack-infra | 08:53 | |
| elod | hi infra team, i have a question regarding 'required project': stable periodic jobs on stable/rocky of networking-bagpipe fail as the jobs use the latest neutron instead the rocky version. I've tried to add neutron for all jobs as required projects but it still seems to use neutron from master ( https://review.openstack.org/#/c/653025/ ) | 08:57 |
| *** e0ne has quit IRC | 08:58 | |
| elod | or is the only solution to add an upper limit for neutron in networking-bagpipe/requirements.txt @ stable/rocky? | 08:58 |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** yboaron_ has quit IRC | 09:01 | |
| *** yboaron_ has joined #openstack-infra | 09:01 | |
| zbr | AJaeger: do you know why base-jobs linter is failing with ERROR! the role 'revoke-sudo' was not found in ... ? http://logs.openstack.org/37/652637/1/check/tox-linters/322cb06/job-output.txt.gz --- totally unrelated to the patch. | 09:02 |
| *** e0ne has quit IRC | 09:02 | |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** Lucas_Gray has joined #openstack-infra | 09:10 | |
| *** udesale has joined #openstack-infra | 09:20 | |
| AJaeger | zbr: might be configuration issue for opendev/base-jobs - best to ask infra-root on what's going on there | 09:21 |
| openstackgerrit | Ildiko Vancsa proposed openstack-infra/project-config master: Adding the OUI training VM repo under docs https://review.openstack.org/653364 | 09:22 |
| zbr | AJaeger: yeah, is had being like this for few days. I guess the big migration is taking most of the time :p | 09:22 |
| AJaeger | zbr: we need to fix this, I fear that otherwise we cannot merge anythign to opendev/base-jobs | 09:24 |
| zbr | i know | 09:24 |
| *** rascasoft has quit IRC | 09:27 | |
| *** snapiri has joined #openstack-infra | 09:28 | |
| *** rascasoft has joined #openstack-infra | 09:28 | |
| dtantsur | hi folks! we have a transient jobs failure, which seems dependent on which provider is used for a node. Do you have any ideas how to proceed/debug? | 09:29 |
| spotz | Hey all, who's doing stackytics patch changes these days? | 09:29 |
| dtantsur | tl;dr HTTP requests to <hostname>:8089 succeed or fail based on the provider | 09:30 |
| spotz | dtantsur: I'm assuming a specific provider? | 09:30 |
| dtantsur | spotz: I have quite limited statistics, but we seem to pass on rax and fail on limestone and vexxhost | 09:31 |
| spotz | dtantsur: Wondering if something didn't get pushed everywhere, though it could be the passes are behind or the failures:( | 09:32 |
| dtantsur | probably.. I have hard time imagining why it may happen | 09:33 |
| dtantsur | like, we're connecting from within a box to its own hostname, and we get CONNREFUSED. | 09:33 |
| dtantsur | provider-level firewalls should not kick in on this level, right? | 09:34 |
| spotz | dtantsur: No I would suspect they'd be open to allow testing from our systems anyways | 09:35 |
| *** threestrands has quit IRC | 09:36 | |
| spotz | Haven't seen mnaser yet this morning or guihermesp | 09:36 |
| *** e0ne has quit IRC | 09:37 | |
| spotz | limestone I think is logan and it's early for him too | 09:37 |
| dtantsur | ovh and inap seem working as well | 09:37 |
| dtantsur | okay, I'll re-ping people later | 09:37 |
| spotz | dtantsur: Give it another 3-5 hours for people to wake up:) | 09:40 |
| *** lpetrut has joined #openstack-infra | 09:40 | |
| *** e0ne has joined #openstack-infra | 09:40 | |
| dtantsur | right :) | 09:41 |
| *** electrofelix has joined #openstack-infra | 09:46 | |
| *** e0ne has quit IRC | 09:46 | |
| *** e0ne has joined #openstack-infra | 09:53 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:56 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.openstack.org/644557 | 09:56 |
| openstackgerrit | Merged openstack/ptgbot master: Preserve JSON dictionary order https://review.openstack.org/652685 | 10:00 |
| openstackgerrit | Merged openstack/ptgbot master: Remove last GitHub links for help https://review.openstack.org/652686 | 10:00 |
| *** strigazi has quit IRC | 10:09 | |
| *** jchhatbar has joined #openstack-infra | 10:14 | |
| *** janki has quit IRC | 10:16 | |
| *** janki has joined #openstack-infra | 10:23 | |
| *** jchhatbar has quit IRC | 10:25 | |
| *** e0ne has quit IRC | 10:27 | |
| *** e0ne has joined #openstack-infra | 10:31 | |
| *** nicolasbock has joined #openstack-infra | 10:38 | |
| frickler | zbr: AJaeger: I think we need to set up a variant of the tox-linters job that sets up ANSIBLE_ROLES_PATH, similar to this http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n1339 | 10:38 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/reviewstats master: Add vitrage projects https://review.openstack.org/653388 | 10:57 |
| *** AJaeger has quit IRC | 10:57 | |
| *** AJaeger has joined #openstack-infra | 11:00 | |
| jroll | frickler: yeah, the intent with sending the list was to find things we had missed :) | 11:13 |
| jroll | thanks for those | 11:13 |
| *** panda is now known as panda|lunch | 11:18 | |
| *** jchhatbar has joined #openstack-infra | 11:20 | |
| *** janki has quit IRC | 11:22 | |
| *** strigazi has joined #openstack-infra | 11:25 | |
| *** rh-jelabarre has joined #openstack-infra | 11:26 | |
| *** mujahidali has quit IRC | 11:26 | |
| *** rosmaita has joined #openstack-infra | 11:32 | |
| *** _erlon_ has joined #openstack-infra | 11:42 | |
| *** strigazi has quit IRC | 11:50 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Order JSON config import https://review.openstack.org/653411 | 11:50 |
| *** strigazi has joined #openstack-infra | 11:51 | |
| *** bobh has joined #openstack-infra | 11:51 | |
| *** strigazi has quit IRC | 11:52 | |
| *** strigazi has joined #openstack-infra | 11:52 | |
| *** bobh has quit IRC | 11:54 | |
| *** strigazi has quit IRC | 11:55 | |
| *** strigazi has joined #openstack-infra | 11:55 | |
| *** e0ne has quit IRC | 12:00 | |
| *** bobh has joined #openstack-infra | 12:01 | |
| *** bobh has quit IRC | 12:05 | |
| *** dtantsur is now known as dtantsur|brb | 12:07 | |
| *** e0ne has joined #openstack-infra | 12:10 | |
| openstackgerrit | Merged openstack/ptgbot master: Order JSON config import https://review.openstack.org/653411 | 12:19 |
| *** kgiusti has joined #openstack-infra | 12:20 | |
| *** rlandy has joined #openstack-infra | 12:20 | |
| *** zhurong has quit IRC | 12:20 | |
| *** Wryhder has joined #openstack-infra | 12:24 | |
| *** Lucas_Gray has quit IRC | 12:25 | |
| *** Wryhder is now known as Lucas_Gray | 12:25 | |
| *** rlandy is now known as rlandy|ruck | 12:27 | |
| *** pcaruana has quit IRC | 12:30 | |
| *** panda|lunch is now known as panda | 12:32 | |
| *** e0ne has quit IRC | 12:34 | |
| *** mujahidali has joined #openstack-infra | 12:36 | |
| pabelanger | AJaeger: zbr: linters job need to be setup to depends on openstack-infra/zuul-jobs, that is where revoke-sudo lives | 12:48 |
| pabelanger | like we do for openstack-infra/project-config | 12:48 |
| mujahidali | clarkb: I tried adding the simple-init in elements but getting the ssh timeout error. Here is my nodepool config file http://paste.openstack.org/show/749424/ | 12:48 |
| pabelanger | mujahidali: make sure openssh is getting installed, you might need to add openssh-server element | 12:51 |
| pabelanger | you can look at build logs and confirm | 12:52 |
| pabelanger | also make sure port 22/tcp is open on your provider network | 12:52 |
| *** aaronsheffield has joined #openstack-infra | 12:53 | |
| *** pcaruana has joined #openstack-infra | 12:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:56 | |
| *** sthussey has joined #openstack-infra | 12:56 | |
| *** mriedem has joined #openstack-infra | 12:57 | |
| *** bhavikdbavishi has quit IRC | 12:58 | |
| *** e0ne has joined #openstack-infra | 12:58 | |
| mujahidali | pabelanger: I need to add openssh-server and will rebuild the image. Provider's port is open. | 12:59 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated presentation Materials serializers https://review.openstack.org/653424 | 13:01 |
| mordred | dtantsur|brb: limestone is ipv6 only - vexxhost is dual-stack ipv4/ipv6 - I mention because the fact that it's only on some providers and not on others, and limestone is one of the issues - makes me wonder if there is an ipv6 issue | 13:03 |
| mordred | dtantsur|brb: like - maybe an on-host iptables setup (I'm assuming possibly incorrectly devstack related) might have opened 8089 on 127.0.0.1 but not on ::1 - and localhost might be resolving to the ipv6 address? | 13:05 |
| *** Goneri has joined #openstack-infra | 13:05 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated presentation Materials serializers https://review.openstack.org/653424 | 13:06 |
| *** liuyulong has joined #openstack-infra | 13:06 | |
| liuyulong | hi, what happened to zuul? test queued for 4 hours? | 13:06 |
| *** bhavikdbavishi has joined #openstack-infra | 13:06 | |
| *** dtantsur|brb is now known as dtantsur | 13:07 | |
| pabelanger | liuyulong: looks like gate has reset a few times, which tends to back things up | 13:07 |
| pabelanger | best to look why and fix those issues | 13:07 |
| *** yboaron_ has quit IRC | 13:07 | |
| dtantsur | mordred: hmm, possibly | 13:08 |
| pabelanger | liuyulong: we are also down a region in nodepool, so that will affect capacity | 13:09 |
| liuyulong | pabelanger, hi, OK, thanks, | 13:09 |
| *** ralonsoh has joined #openstack-infra | 13:11 | |
| mordred | pabelanger: you're smarter than me ... on https://review.openstack.org/#/c/652637 you said we need to set up tox-linters to depened on openstack-infra/zuul-jobs - but isn't tox-linters _in_ zuul-jobs? | 13:12 |
| zbr | pabelanger: are you referring to https://opendev.org/opendev/base-jobs/src/branch/master/zuul.yaml#L433 ? should I make tox-linters into a template and add role: zuul: ... ? | 13:12 |
| mordred | oh - wait. it's because the roles in zuul-jobs aren't available to the linter job ... | 13:13 |
| zbr | mordred: yep. this seems to be the issue. | 13:13 |
| mordred | ok. patch coming for that | 13:14 |
| pabelanger | mordred: zbr: http://git.openstack.org/cgit/openstack-infra/project-config/tree/tox.ini#n15 | 13:15 |
| pabelanger | I think we need to do that | 13:15 |
| zbr | pabelanger: this does not seem as something I should have to add to each tox.ini tested by tox-linters job? i hope not. if is a special case for base-jobs repo, is ok. | 13:17 |
| mordred | it's a special case | 13:17 |
| mordred | because the base-jobs repo has playbooks that use the roles from zuul-jobs | 13:17 |
| pabelanger | you'd need to do it in tox.ini or job stanza each time you want cross repo roles, outside of your project. So far, just base jobs I've seen doing it | 13:18 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Use zuul-jobs roles path for linters https://review.openstack.org/653429 | 13:18 |
| mordred | there ^^ that should fix the linters job | 13:18 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: [POC] Remove fetch-zuul-cloner from base job https://review.openstack.org/652637 | 13:18 |
| mordred | and there's your patch rebased on it | 13:18 |
| mordred | but - as pabelanger said also - we need to do a base-test patch first | 13:19 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Fix linters by defining ANSIBLE_ROLES_PATH in tox.ini https://review.openstack.org/653430 | 13:19 |
| mordred | zbr: sorry - beat you to it - also, you want those in setenv, not passenv | 13:19 |
| mordred | zbr: oh - actually - we both got something wrong | 13:20 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Fix linters by defining ANSIBLE_ROLES_PATH in tox.ini https://review.openstack.org/653430 | 13:21 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Use zuul-jobs roles path for linters https://review.openstack.org/653429 | 13:21 |
| mordred | zbr: ^^ we don't need openstack-zuul-jobs - and also the relative paths are wrong, this is opendev/base-jobs, so we need ../../openstack-infra/zuul-jobs | 13:22 |
| *** bobh has joined #openstack-infra | 13:22 | |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Fix paths for zuul org rename https://review.openstack.org/653431 | 13:22 |
| *** shardy has joined #openstack-infra | 13:23 | |
| mordred | pabelanger, fungi: ^^ there is a WIP followup patch - I'm not sure whether the rename script would catch this particular case of zuul org rename - is there a topic I should put on it so we know to include it in the rename patches to merge? | 13:23 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: [POC] Remove fetch-zuul-cloner from base job https://review.openstack.org/652637 | 13:24 |
| mordred | zbr: you wanna split that ^^ into 2 patches - one that applies the change to base-test - so we can land that and then test things against the base-test base job, and then a second that does the update to base? | 13:25 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add test_setup_reset_connection setting https://review.openstack.org/653130 | 13:27 |
| *** dikonoor has joined #openstack-infra | 13:27 | |
| *** e0ne has quit IRC | 13:30 | |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Remove fetch-zuul-cloner from base-test job 1/2 https://review.openstack.org/652637 | 13:32 |
| jroll | fungi: do you want to keep track of namespace suggestions from the list, or would you like me to? for unofficial things that people want to remain in the openstack namespace, do we want to just let them do so, have a TC conversation about it, or ask them to define another namespace? | 13:33 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Remove fetch-zuul-cloner from base job 2/2 https://review.openstack.org/653433 | 13:34 |
| *** ykarel is now known as ykarel|afk | 13:36 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 13:37 | |
| zbr | i still do not understand why the linters jobs is not running on a change that modified tox.ini file in base-jobs | 13:37 |
| *** bhavikdbavishi has quit IRC | 13:38 | |
| *** bhavikdbavishi1 has quit IRC | 13:41 | |
| *** ykarel|afk has quit IRC | 13:41 | |
| mordred | zbr: oh - maybe there is an exclusion? | 13:42 |
| mordred | hrm. no - no exclusions that I can see | 13:43 |
| zbr | something is weird there | 13:46 |
| mordred | zbr: the linters job is running on the tox.ini change | 13:53 |
| mriedem | fyi this doesn't hit much but it looks like maybe a bionic mirror issue https://bugs.launchpad.net/openstack-gate/+bug/1825190 | 13:57 |
| openstack | Launchpad bug 1825190 in OpenStack-Gate "Failed to fetch http://mirror.dfw.rax.openstack.org/ubuntu/dists/bionic-updates/universe/binary-amd64/Packages.gz File has unexpected size (1257053 != 1257062). Mirror sync in progress?" [Undecided,New] | 13:57 |
| *** quiquell|rover has quit IRC | 14:00 | |
| *** quiquell has joined #openstack-infra | 14:00 | |
| *** quiquell is now known as quiquell|off | 14:00 | |
| *** amansi26 has joined #openstack-infra | 14:01 | |
| *** lseki has joined #openstack-infra | 14:03 | |
| *** bobh_ has joined #openstack-infra | 14:05 | |
| fungi | mordred: yeah, it's going to be stuff like that which we won't be able to find easily in repos, so any of those sorts of relative path references (especially given this isn't even in zuul configuration or roles/playbooks it references) | 14:05 |
| fungi | topic:opendev-gerrit is what we've been using for the migration prep | 14:06 |
| *** bobh has quit IRC | 14:06 | |
| mordred | ok. cool. | 14:07 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Use zuul-jobs roles path for linters https://review.openstack.org/653429 | 14:08 |
| fungi | jroll: in the long run, openstack tc hat on, i think we need some way of tracking and authorizing those sorts of exceptions (like adding them to the tc repos list or the foundation board repos list or some similar new file). in the short term we can add them to the etherpad because the current script uses this precedence order airship > starlingx > openstack > zuul > ethercalc > openstack-legacy > | 14:09 |
| fungi | openstack-eviction > unchanged | 14:09 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Use zuul-jobs roles path for linters https://review.openstack.org/653429 | 14:10 |
| fungi | er, add them to the ethercalc i meant | 14:10 |
| mordred | corvus: ^^ can you tell me if I'm correct about the change between ps3 and ps4 there | 14:10 |
| jroll | fungi: looking again, networking-l2gw-tempest-plugin seems like the only miss (should be in projects.yaml). upstream-institute-virtual-environment has a governance change up to make it official, and the rest appear that they should remain independent | 14:12 |
| * jroll moves to -tc channel | 14:13 | |
| *** dpawlik has quit IRC | 14:19 | |
| corvus | mordred: makes sense to me! | 14:21 |
| *** ykarel|afk has joined #openstack-infra | 14:24 | |
| *** ykarel|afk is now known as ykarel | 14:25 | |
| *** liuyulong has quit IRC | 14:27 | |
| mordred | corvus: cool | 14:27 |
| *** liuyulong has joined #openstack-infra | 14:28 | |
| *** lpetrut has quit IRC | 14:28 | |
| fungi | and just to wrap up that loose end, the script worked fine, openstack/networking-l2gw-tempest-plugin was removed from reference/projects.yaml in governance a while back when the neutron stadium disbanded, but was not added to reference/legacy.yaml | 14:30 |
| corvus | fungi, mordred: what do you think of "ttygroup/gertty" ? | 14:32 |
| *** bobh_ has quit IRC | 14:33 | |
| mordred | corvus: ++ | 14:33 |
| *** markvoelker has joined #openstack-infra | 14:35 | |
| *** david-lyle is now known as dklyle | 14:35 | |
| *** yboaron_ has joined #openstack-infra | 14:36 | |
| dtantsur | mordred: it seems that IPv6 may be related. oslo_service.wsgi defaults to 0.0.0.0, the hostname may resolve to IPv6 though.. | 14:36 |
| fungi | corvus: i think it's ttyriffic! | 14:36 |
| mordred | dtantsur: awesome. that seems like a thing where we might want to fix oslo_service to default to :: instead | 14:36 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for mirror sync bug 1825190 https://review.openstack.org/653451 | 14:37 |
| openstack | bug 1825190 in OpenStack-Gate "Failed to fetch http://mirror.dfw.rax.openstack.org/ubuntu/dists/bionic-updates/universe/binary-amd64/Packages.gz File has unexpected size (1257053 != 1257062). Mirror sync in progress?" [Undecided,New] https://launchpad.net/bugs/1825190 | 14:37 |
| dtantsur | mordred: wouldn't it break IPv4-only hosts? | 14:37 |
| mordred | dtantsur: nope - :: should work for both things | 14:37 |
| fungi | unless the kernel is configured very strangely, :: binds to all available addresses, both v4 and v6 | 14:37 |
| corvus | fungi: that's awesome. | 14:37 |
| dtantsur | TIL! | 14:37 |
| *** michael-beaver has joined #openstack-infra | 14:37 | |
| mordred | yeah -0 these days :: should generally be used anywhere that 0.0.0.0 has been in the past | 14:37 |
| fungi | there *is* a sysctl opt to tell the kernel to use separate stacks for :: and 0.0.0.0 | 14:38 |
| fungi | but none of the major distros actually set that by default | 14:38 |
| fungi | mainly because it means that if you want your listening daemons to work in a dual-stack environment you end up needing to bind separately for v4 and v6 sockets | 14:39 |
| fungi | which is a royal pain | 14:39 |
| dtantsur | ++ | 14:40 |
| dtantsur | damn, one day I will learn all necessary things about IPv6.. one day... | 14:40 |
| fungi | i spent years as a professional network engineer, headed up v6 core and backbone connectivity at an isp, and still feel like i've barely scratched the surface with ipv6, so don't feel bad | 14:41 |
| *** mattw4 has joined #openstack-infra | 14:41 | |
| fungi | it's hard to overcome the inertia of decades of absorbed ipv4 knowledge | 14:42 |
| dtantsur | indeed :) | 14:43 |
| mnaser | dtantsur: both limestone and us run ipv6 apparnetly | 14:46 |
| mnaser | so that will most likely be your cause | 14:46 |
| fungi | i envy the first generation who gets to learn about ipv4 in history class | 14:46 |
| dtantsur | right, thanks fungi, mordred, mnaser | 14:46 |
| fungi | technically rackspace and ovh both provide ipv6, but do so in ways we can't consume in our gate jobs | 14:47 |
| AJaeger | fungi, jroll , I commented on the governance review - networking-l2gw-tempest-plugin was never under TC control AFAIU, couldn't find such a change | 14:48 |
| fungi | ovh requires you to look up the v6 address and manually configure it in the instance (not provided in any metadata), rackspace has it working if you use cloud-init but doesn't include it in the configdrive data | 14:48 |
| fungi | so we effectively run test nodes with only ipv4 in those providers | 14:48 |
| *** iurygregory has quit IRC | 14:49 | |
| fungi | AJaeger: i agree, i just keyword searched the entirety of the governance repo history and can confirm it never appeared in any patch (adding or removing) | 14:50 |
| *** mattw4 has quit IRC | 14:50 | |
| * AJaeger won't -1 that governance change - but that's for TC to decide... | 14:50 | |
| *** e0ne has joined #openstack-infra | 14:53 | |
| *** jchhatbar has quit IRC | 14:53 | |
| *** jchhatbar has joined #openstack-infra | 14:54 | |
| *** iurygregory has joined #openstack-infra | 14:54 | |
| *** dustinc has quit IRC | 14:56 | |
| *** dustinc has joined #openstack-infra | 14:56 | |
| *** dustinc is now known as dustinc_away | 14:56 | |
| *** weshay is now known as weshay|rover | 14:57 | |
| *** udesale has quit IRC | 14:57 | |
| *** udesale has joined #openstack-infra | 14:59 | |
| *** udesale has quit IRC | 14:59 | |
| *** udesale has joined #openstack-infra | 14:59 | |
| clarkb | mordred: dtantsur fungi the one issue with :: is if ipv6 is disabled for some reason | 15:00 |
| clarkb | I wrote that geard binding change to figure out what addrs are available safely | 15:00 |
| dtantsur | yeah.. it's probably the best we can do for a quick fix (it appears in our gate). | 15:00 |
| clarkb | I forget the details but its basically give me the full list of things I can bind to, then sort by family then take first option | 15:00 |
| *** lpetrut has joined #openstack-infra | 15:01 | |
| *** udesale has quit IRC | 15:06 | |
| clarkb | https://review.openstack.org/#/c/611920/2/gear/__init__.py | 15:07 |
| clarkb | then I think host is 'localhost' by default | 15:07 |
| *** sreejithp has joined #openstack-infra | 15:10 | |
| fungi | pretty sure ipv6 should never be disabled by default on any of our job nodes. there will at least be linklocal interfaces | 15:10 |
| *** bobh has joined #openstack-infra | 15:11 | |
| clarkb | for our nodes yes, but oslo is expected to run elsewhere too | 15:11 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for mirror sync bug 1825190 https://review.openstack.org/653451 | 15:12 |
| openstack | bug 1825190 in OpenStack-Gate "Failed to fetch http://mirror.dfw.rax.openstack.org/ubuntu/dists/bionic-updates/universe/binary-amd64/Packages.gz File has unexpected size (1257053 != 1257062). Mirror sync in progress?" [Undecided,New] https://launchpad.net/bugs/1825190 | 15:12 |
| fungi | clarkb: yeah, but defaulting to not listen on your ipv6 interface is at least as bad as failing to bind because you've disabled ipv6 in your kernel | 15:13 |
| AJaeger | mordred: https://review.openstack.org/653429 did not pass - is that not self-testing and thus needs a force merge? | 15:14 |
| fungi | neither default is perfect, but i feel like requiring special configuration to make it work with ipv6 is worse than requiring special configuration because you've gone to the trouble of turning off ipv6 in your system | 15:15 |
| clarkb | fungi: ya I'm not suggesting they don't bind to ipv6. I'm saying the code above is the safe way to do it over assuming :: works | 15:15 |
| mordred | AJaeger: it should be self-testing | 15:15 |
| mordred | oh - wait | 15:15 |
| clarkb | fungi: code above will lookup configured addresses for 'localhost' in all addressing families then pick the newest family addr | 15:15 |
| mordred | the required-projects line won't be | 15:15 |
| mordred | since it's a config repo | 15:16 |
| AJaeger | fun ;( | 15:16 |
| mordred | AJaeger: yes - I believe we're going to need to force-merge that - or we could split it, force-merge required projects (which won't fix the job but also won't break it more) - and then let the tox.ini run through the gate normally? | 15:17 |
| AJaeger | I'll +2... | 15:17 |
| mordred | AJaeger, clarkb, corvus: ^^ thoughts on which approach would be better | 15:17 |
| fungi | clarkb: i guess that looks at the loopback interface to see what address families are bound to it? because on a lot of systems "localhost" resolves only to 127.0.0.1 via /etc/hosts, and there's a separate ip6-localhost for ::1 | 15:17 |
| clarkb | fungi: yes. I think you can feed it '::' as the host too and it will give you back 0.0.0.0 if '::' isn't valid | 15:18 |
| AJaeger | mordred: I suggest to force-merge as is and gave a +2 on 653429 | 15:18 |
| corvus | mordred: ++ force merge | 15:18 |
| AJaeger | corvus: want to +2? | 15:20 |
| mordred | ok. I'll do the force-merge | 15:20 |
| openstackgerrit | Merged opendev/base-jobs master: Use zuul-jobs roles path for linters https://review.openstack.org/653429 | 15:21 |
| openstackgerrit | Monty Taylor proposed opendev/base-jobs master: Remove fetch-zuul-cloner from base-test job 1/2 https://review.openstack.org/652637 | 15:21 |
| *** pgaxatte has quit IRC | 15:21 | |
| mordred | rebased that - hopefully it comes back green | 15:21 |
| AJaeger | thanks, mordred | 15:23 |
| fungi | did we make a change to the ensure-twine role lately? trying to figure out what would have caused http://logs.openstack.org/19/19a7574237f44807b16c37e0983223ff57340ba3/release/release-openstack-python/769f856/job-output.txt.gz#_2019-04-17_02_21_43_528266 | 15:24 |
| clarkb | corvus: ttx: responded re the etherpad breakdown of projects and added a couple notes about some x/ projects that probably belong in openstack/. That plan seems reasonable to me. I think storyboard is the last big outstanding question in that list? | 15:24 |
| clarkb | fungi: ^ maybe you can weigh in on that as you've been most involved wtih storyboard recently | 15:24 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Use standard ansible-lint config file https://review.openstack.org/652708 | 15:26 |
| openstackgerrit | Sorin Sbarnea proposed opendev/base-jobs master: Remove fetch-zuul-cloner from base job 2/2 https://review.openstack.org/653433 | 15:27 |
| *** mujahidali has quit IRC | 15:29 | |
| corvus | clarkb: my classification of storyboard into its own namespace is just a gut feeling, i'm not tied to it. we should probably ask fungi, diablo_rojo, and SotK | 15:30 |
| clarkb | corvus: ya that was sort of how I read that section "this needs input" | 15:30 |
| corvus | mordred: where should openstack-infra/shade go? see https://etherpad.openstack.org/p/6CmVhW40m0 | 15:30 |
| clarkb | the proposal as is should be fine if we don't hear any objections | 15:31 |
| mordred | it shoudl go to openstack - it's a subproject of the openstacksdk team - will comment there | 15:31 |
| clarkb | jeepyb is another one that had me pause beacuse it is somewhat opinionated around the way we gerrit, but thinking about it more I think it would be better for that project to not be seen as an openstack/opendev specific tool | 15:32 |
| clarkb | so I like it under x/ | 15:32 |
| *** jchhatbar has quit IRC | 15:33 | |
| *** jchhatbar has joined #openstack-infra | 15:33 | |
| mordred | I don't have a problem with it under opendev/ - it's a tool that we wrote to manage projects in openstack-infra/opendev - and we still use it | 15:34 |
| sreejithp | Hello All, one of my friends ps is still not merged even after getting 2 +2 and workflow +1. when i checked zuul, there is no jobs running for this ps. trying to understand how the merge ps works | 15:34 |
| fungi | clarkb: what options do you want to present the storyboard team? openstack vs storyboard vs opendev? | 15:34 |
| sreejithp | this the ps https://review.openstack.org/#/c/652735/ | 15:34 |
| clarkb | sreejithp: can you link to the change? | 15:35 |
| openstackgerrit | Sorin Sbarnea proposed openstack-dev/pbr master: Add openstack-tox-py37 job https://review.openstack.org/610659 | 15:35 |
| clarkb | fungi: ya I think those three options are the ones that make sense. Considering the way storyboard has largely been independent I think storyboard/ is appropriate if storyboard wants to do that | 15:35 |
| sreejithp | clarkb: https://review.openstack.org/#/c/652735/ | 15:35 |
| mordred | clarkb: yeah - I think storyboard/ is nice for that one | 15:35 |
| fungi | sreejithp: looks like its git parent is https://review.openstack.org/651659 which isn't merged yet | 15:36 |
| *** jchhatbar has quit IRC | 15:36 | |
| clarkb | sreejithp: https://review.openstack.org/#/c/651659/ is the parent change of https://review.openstack.org/#/c/652735/ and because the parent change is not merging and cannot merge the child cannot either | 15:36 |
| *** jchhatbar has joined #openstack-infra | 15:36 | |
| AJaeger | sreejithp: check the right top corner "related changes" | 15:36 |
| fungi | clarkb: but we're okay if they are interested in going into the opendev namespace too, or is that a larger discussion? | 15:37 |
| sreejithp | oh ok, thanks for the info | 15:37 |
| clarkb | fungi: I'm ok with that as well. It is a service we intend to run for opendev and we've long been the "hosting" governance body | 15:37 |
| *** jchhatbar has quit IRC | 15:40 | |
| mordred | clarkb: re: jeepyb, git-review, git-restack, and glean - I kinda feel like those should be opendev/ - they're tools written by the opendev team to aid in either managing opendev systems or to aid in doing open development. if we were to start anyof those projects next week after the rename, I'd imagine we'd put them into opendev/ instead of x/ ... I don't feel STRONGLY about that - but they do feel | 15:41 |
| mordred | like tools we wrote, own and currently maintain in the context of our work on opendev | 15:41 |
| zbr | does any of you know why check jobs made on base-jobs never appear in zuul http://zuul.openstack.org/status ? i seen them run after a while but never seen them in zuul dashboard. | 15:41 |
| clarkb | zbr: because it is in a different tenant. Try zuul.opendev.org | 15:41 |
| clarkb | mordred: ya I can go either way on that. As you say they are tools we build for our use of the other tools, but they should also be useful elsewhere. | 15:42 |
| corvus | mordred: no objection there; i think my rubric was "is this used to operate our systems" but i'm fine taking a more generous line. | 15:42 |
| zbr | clarkb: ouch,... so I will need to update the zuul greasemonkey script to know about both... :p | 15:42 |
| *** lpetrut has quit IRC | 15:42 | |
| clarkb | mordred: I'm happy for them to be in opendev if that makes people happier :) | 15:42 |
| mordred | clarkb: totally - we've got pynotedb and gerritlib in opendev too though | 15:42 |
| clarkb | zbr: well you can view openstack tenant under opendev.org too so could just use that one domain | 15:43 |
| clarkb | mordred: thats a good point | 15:43 |
| corvus | i put project-config under openstack/ because i was thinking of all the zuul stuff in there... but it also has gerrit, irc bots, etc.... so maybe it should stay in opendev for now? | 15:46 |
| clarkb | corvus: opendev has a project-config though | 15:46 |
| corvus | okay | 15:46 |
| corvus | so we'll need to untangle that | 15:47 |
| clarkb | I think what will happen is the stuff in openstack/project-config will have to move into opendev/project-config over time | 15:47 |
| clarkb | but that will be independent of the repo reorg? | 15:47 |
| corvus | yeah | 15:47 |
| corvus | and we will want an openstack/project-config for the zuul.d content | 15:47 |
| clarkb | yup | 15:47 |
| mordred | yeah. although can our storyboard tooling handle two project-config repos? (do we still have things that expect the repo part of org/repo to be unique?) | 15:48 |
| * mordred agrees that we need a project-config for both opendev and openstack and that we'll need to move some things from openstack/project-config to opendev/project-config post-rename | 15:49 | |
| clarkb | devstack-gate expects that uniqueness but it shouldn't touch any project-config repos | 15:49 |
| ttx | Re: storyboard to me it depends whether we have external adoption or if it's just an opendev-specific thing | 15:49 |
| ttx | At this point I'd probably keep it under opendev/ | 15:50 |
| ttx | if only to encourage contribution to it | 15:50 |
| mordred | clarkb: k. cool | 15:50 |
| zbr | clarkb: it seems that I cannot just point the script to use http://zuul.opendev.org api because I need to mention which tenant to query, which kinda breaks it because it does not support multiple tenants. see https://github.com/openstack/coats/blob/master/coats/openstack_gerrit_zuul_status.user.js#L27 | 15:50 |
| corvus | mordred: afaik, storyboard uses full project names and devstack-gate (as clarkb says) is the only thing that makes that assumption. | 15:51 |
| *** adriant has quit IRC | 15:51 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Import Zuul job settings https://review.openstack.org/653133 | 15:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Load subproject data from governance https://review.openstack.org/653024 | 15:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Handle all exceptions loading pickled data https://review.openstack.org/653106 | 15:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config master: Remove reviewstats unit test job config https://review.openstack.org/653150 | 15:52 |
| *** adriant has joined #openstack-infra | 15:53 | |
| *** iurygregory has quit IRC | 15:53 | |
| fungi | mordred: storyboard projects are namespaced identically to gerrit/gitea so shouldn't be a problem. there's nothing in storyboard which uses a short repo name (or even understands that such a concept exists) | 15:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Import Zuul job settings https://review.openstack.org/653133 | 15:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Load subproject data from governance https://review.openstack.org/653024 | 15:55 |
| fungi | and i guess corvus said the same thing, only much faster than me ;) | 15:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Handle all exceptions loading pickled data https://review.openstack.org/653106 | 15:55 |
| *** gyee has joined #openstack-infra | 15:58 | |
| zbr | do we have a CORS policy between old and new domain because i think that lack of it would prevent me from getting the live build status. maybe waiting is better in this case. | 15:58 |
| clarkb | zbr: I don't think we do. you should be able to get it from the new domain directly without talking to another domain though | 15:59 |
| *** diablo_rojo has joined #openstack-infra | 15:59 | |
| *** bobh has quit IRC | 15:59 | |
| clarkb | zbr: http://zuul.opendev.org/t/openstack/status for example | 16:01 |
| *** ykarel is now known as ykarel|away | 16:05 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Prune the intermediate registry before GCing https://review.openstack.org/653148 | 16:05 |
| clarkb | corvus: ^ I think that fixes the ansible lint issue? that one seems less like a real concern than the bashate catch | 16:06 |
| zbr | clarkb: something is not working, what is the correct new API url for lets say curl https://zuul.openstack.org/api/status/change/652708,2 | 16:07 |
| *** dims has quit IRC | 16:07 | |
| zbr | i tried including /t/openstack/ but did not work. | 16:07 |
| *** lpetrut has joined #openstack-infra | 16:09 | |
| *** Lucas_Gray has quit IRC | 16:09 | |
| corvus | zbr: https://zuul.opendev.org/api/tenant/openstack/status/change/652708,2 | 16:09 |
| *** lucasagomes has quit IRC | 16:09 | |
| zbr | corvus: thanks. i am updating it now. Out of curiosity, can I infer the tenant from the project prefix? | 16:10 |
| corvus | zbr: no, there's no relation | 16:10 |
| fungi | nope. you can have the same projects (with the same prefixes) in different tenants | 16:10 |
| zbr | and I guess we do not have consolidated API support | 16:11 |
| corvus | zbr: correct, tenants are entirely separate | 16:11 |
| fungi | what do you mean by "consolidated"? | 16:11 |
| zbr | meaning making a single API call and get results from all tenants. like LDAP does. | 16:11 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 16:12 |
| fungi | but yes, the choice to put tenant early in the api path is so that deployments can choose to filter or authenticate specific tenants | 16:12 |
| fungi | which is easier than trying to bake an rbac into the status api | 16:12 |
| corvus | and among other reasons, a given project name (or change id) could actually appear in multiple tenants yet refer to a different project or change | 16:12 |
| zbr | i am trying to imagine how to make the zuul-status script with the new setup, in a way that would avoid hammering each zuul tenant. | 16:14 |
| corvus | also, just for the record since we're talking about it -- the zuul api is not yet stable and we don't yet make any backwards compatibility guarantees | 16:14 |
| clarkb | https://etherpad.openstack.org/p/the-great-opendev-git-migration I've started with a rough outline there | 16:16 |
| clarkb | please add details and things I have missed if you notice them | 16:16 |
| fungi | zbr: ahh, that's a good point... from within the context of our gerrit there's not a guaranteed 1:1 mapping from a change you're viewing to a specific status api tenant... you could have multiple pipelines in different tenants reporting into it | 16:16 |
| *** ccamacho has quit IRC | 16:16 | |
| zbr | exactly: the question is how does the change knows who to ping for updates. | 16:17 |
| clarkb | infra-root as a heads up I've approved the review.opendev.org dns change as well as puppet-4 upgrade for nodepool hosts | 16:18 |
| fungi | clarkb: one thing we're going to need is a traditional project rename change for merging to the project-config repo... handling acl renames is going to be the trickiest part. i've been noodling on ways to programmatically generate that from the renames mapping, the stickiest part is if we want to rename teh files for implicit acls and need to update any corresponding references as explicit acl file | 16:18 |
| fungi | entries in gerrit/projects.yaml (though i think i know a shortcut) | 16:18 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add review.opendev.org https://review.openstack.org/653108 | 16:19 |
| clarkb | fungi: if it is simpler maybe we add explicit acls paths then cleanup in a followup? | 16:19 |
| fungi | i think if we apply the same rename mapping to teh acl-file entries in projects.yaml as we apply to the project names themselves and then also compare that mapping against subpaths of gerrit/acls/ we can do the transformation cleanly in a single pass | 16:20 |
| *** jpich has quit IRC | 16:20 | |
| *** rpittau is now known as rpittau|afk | 16:20 | |
| fungi | i was also briefly worried about what this means for renaming retired repos since the project configs for them are read-only and manage-projects can't push updated acls into gerrit for them, but then realized that shouldn't be necessary if we're not actually altering the acl contents | 16:21 |
| fungi | that ought to just be a no-op | 16:21 |
| fungi | i don't anticipate renaming any gerrit groups as a part of the migration | 16:22 |
| clarkb | also we don't push retired acls anymore | 16:22 |
| *** ginopc has quit IRC | 16:22 | |
| fungi | cool | 16:23 |
| corvus | fungi: that sounds good | 16:23 |
| fungi | also still planning to update the rename mapping generator to output the yaml file our project rename playbooks ingest | 16:24 |
| corvus | clarkb, mordred: i found out what's going wrong with the registry thing -- thanks to tcpdump, i found that the registry is sending http redirects to the docker daemon telling it to fetch data from swift tempurls, but the url is invalid; example: 16:21 < fungi> that ought to just be a no-op | 16:24 |
| corvus | 16:22 < fungi> i don't anticipate renaming any gerrit groups as a part of the migration | 16:24 |
| corvus | 16:22 < clarkb> also we don't push retired acls anymore | 16:24 |
| corvus | gah | 16:24 |
| corvus | https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_622b11a1-5dfa-43b4-9f58-4ad3c6dbc4a0/intermediate_registry/files/docker/registry/v2/blobs/sha256/b0/b0299e0551df62b5f8061086ef5feb628f26949c1ba165ab7788e4e612072185/data?temp_url_sig=e0aba6eed50751107df81f64e85350a53045f07e&temp_url_expires=1555518472 | 16:24 |
| *** dtantsur is now known as dtantsur|afk | 16:24 | |
| corvus | clarkb, mordred: ^ | 16:24 |
| *** david-lyle has joined #openstack-infra | 16:24 | |
| *** dangtrinhnt_ has joined #openstack-infra | 16:25 | |
| *** igordc has joined #openstack-infra | 16:25 | |
| clarkb | corvus: for public access on rax you have to use their cdn setup which gives you the hashed domain name thing | 16:25 |
| fungi | is this related to the cdn proxy thing they use? | 16:25 |
| clarkb | corvus: I wonder if that is still required, but could explain the error | 16:25 |
| fungi | er, what clarkb said | 16:25 |
| corvus | i was under the impression that tempurls worked directly against swift | 16:25 |
| corvus | or rather, "cloud files" | 16:26 |
| *** manjeets__ has joined #openstack-infra | 16:26 | |
| *** ramishra has quit IRC | 16:26 | |
| fungi | that api endpoint is publicly reachable, at least | 16:27 |
| *** dangtrinhnt has quit IRC | 16:27 | |
| *** dklyle has quit IRC | 16:27 | |
| *** mattw4 has joined #openstack-infra | 16:27 | |
| *** manjeets_ has quit IRC | 16:27 | |
| zbr | i added https://review.openstack.org/#/c/653487/ -- but i will have to wait for the DNS to start responding. | 16:32 |
| corvus | clarkb, fungi: https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/ | 16:33 |
| clarkb | zbr: note that it won't be fully operational until friday | 16:33 |
| clarkb | zbr: since we have to update the gerrit config and apache | 16:33 |
| corvus | that's why i think this should work | 16:33 |
| jroll | hrm, temp urls used to work on cloud files :/ | 16:33 |
| clarkb | corvus: that does say it should work independent of teh cdn | 16:34 |
| corvus | maybe i should manually construct that tempurl and see if it matches | 16:34 |
| jroll | I've done this in the past, but it's been a while | 16:34 |
| mordred | corvus: so - the first thing that comes to mind is whether the shared secret key is correct | 16:35 |
| mordred | corvus: because it's going to be the swift driver for docker that's going to generate the temp_url_sig | 16:35 |
| corvus | mordred: yeah, manually constructing it would probably tell us that | 16:35 |
| corvus | here's the code for that, btw: https://github.com/ncw/swift/blob/master/swift.go#L1836 | 16:35 |
| mordred | so if it's doing that with the wrong key, the signature will be wrong | 16:35 |
| corvus | i will also verify that object exists :) | 16:36 |
| mordred | corvus: there were key-related swift driver config settings ... I'm assuming we added a key to the container or the account, yeah? | 16:37 |
| clarkb | fungi: does the gerrit/projects.yaml update fall under your scripting task or do I need to add a new task to the story for that? I just added a task for the gerrit apache config update | 16:37 |
| corvus | mordred: i did not, which, i aiui, means that the registry will generate one and set it. there is now a key on the account. | 16:38 |
| mordred | corvus: for debugging, it might be worth putting that key into the secretkey setting and see if the generated tempurls work | 16:38 |
| mordred | ah. ok. so there is an account key | 16:38 |
| mordred | well that should probably work then :) | 16:38 |
| corvus | welp, the web dashboard is returning an error when i look at cloud files in dfw. | 16:40 |
| mordred | awesome | 16:40 |
| fungi | clarkb: after thinking through it, i should be able to just include it in the same mass git change generator i'm already working on. the pattern matches in projects.yaml should be easy enough to integrate, which only leaves doing a git mv on a few acl files and making sure the parent dirs for them exist | 16:41 |
| clarkb | fungi: also the review.opendev.org cert you provisioned is for opendev.org only names or did we put the old openstack.org names in there too? Wondering if we need to manage two sets of certs or just one (I'm guessing two sets of certs) | 16:41 |
| clarkb | fungi: k | 16:41 |
| fungi | clarkb: it'll need separate certs if we put them on the same host | 16:41 |
| fungi | we might want to consider hosting the review.openstack.org redirect on files.o.o instead? | 16:41 |
| corvus | i think that would be fine; might be a little squirrely as we do the dns changes | 16:42 |
| clarkb | that is a good point, may simplify the config management update for review.o.o | 16:42 |
| fungi | we already have all the git redirects there so it's just one more | 16:42 |
| clarkb | I'm pretty sure current puppet for review doesn't know how to manage two certs so splitting that off is probably a good idea | 16:42 |
| corvus | ++ | 16:42 |
| *** eernst has joined #openstack-infra | 16:44 | |
| *** ijw has joined #openstack-infra | 16:45 | |
| corvus | the exmaple python code on that page is nonsensical | 16:45 |
| corvus | almost nothing about it is right, it's python3 with bare print, the variables don't match.... | 16:46 |
| mordred | corvus: the cloud files tempurl docs? | 16:48 |
| corvus | ya | 16:48 |
| corvus | however, once i fix it, it *does* produce a working tempurl | 16:48 |
| openstackgerrit | Merged openstack-infra/system-config master: Update nodepool servers to puppet 4 https://review.openstack.org/650379 | 16:48 |
| corvus | and that does not match the one from go | 16:48 |
| corvus | so i will dig into the algorithm now | 16:48 |
| mordred | corvus: awesome | 16:48 |
| clarkb | progress at least | 16:48 |
| mordred | corvus: https://docs.openstack.org/swift/latest/api/temporary_url_middleware.html is the upstream swift docs on the topic | 16:48 |
| clarkb | I'm going to monitor the puppet-4 upgrade to the nodepool servers. | 16:48 |
| *** eernst has quit IRC | 16:48 | |
| jroll | corvus: looks like the go code uses sha1, cloud files docs uses sha256 | 16:49 |
| clarkb | Does anyone want to work on the puppet updates for redirecting review.openstack.org to review.opendev.org as well as updating the canonical name and the logo? | 16:49 |
| *** ijw has quit IRC | 16:49 | |
| corvus | jroll: ah thanks! i changed the python script to use sha1 and it still produces a working url | 16:50 |
| corvus | so there must be some other difference | 16:51 |
| jroll | corvus: oh, interesting | 16:51 |
| *** amansi26 has quit IRC | 16:51 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update zuul servers to puppet 4 https://review.openstack.org/650380 | 16:53 |
| tosky | with the upcoming opendev change, will openstack-specific domains like governance., tarballs., docs., wiki.openstack.org stay as they are? Should I focus only on renaming the occurrences of git.openstack.org (and review.openstack.org outside the .gitreview file)? | 16:53 |
| *** dims has joined #openstack-infra | 16:54 | |
| mordred | corvus: do we know what c.StorageUrl is? and what then prefix.Path would be? like - maybe the wrong thing is getting passed in there | 16:54 |
| clarkb | tosky: the only domains that are changing on Friday are review.opendev.org and git.openstack.org becoming opendev.org | 16:54 |
| *** hwoarang has quit IRC | 16:54 | |
| clarkb | tosky: over time we may change others, but that will be separate efforts and we'll communicate them when they happen | 16:54 |
| tosky | thanks! | 16:55 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update lists.openstack.org to puppet 4 https://review.openstack.org/653489 | 16:55 |
| *** hwoarang has joined #openstack-infra | 16:56 | |
| fungi | well, also git.airshipit.org, git.starlingx.io and git.zuul-ci.org are also going to opendev.org on friday | 16:56 |
| corvus | mordred: here's the calling site: https://github.com/docker/distribution/blob/master/registry/storage/driver/swift/swift.go#L646 | 16:56 |
| clarkb | fungi: thanks just added that to the etherpad | 16:58 |
| *** dims has quit IRC | 16:59 | |
| corvus | mordred: i know from a log line that the "path" variable at the calling site is like this: /docker/registry/v2/blobs/sha256/b0/b0299e0551df62b5f8061086ef5feb628f26949c1ba165ab7788e4e612072185/data | 16:59 |
| corvus | mordred: (that's a different object than the example i pasted, but that's the format at least) | 16:59 |
| mordred | ok. so path is the docker object path | 16:59 |
| clarkb | fungi: the files.o.o change for the git.* redirects is already in place right? we simply need to update DNS records for that to take effect at this point? | 17:00 |
| corvus | mordred: i'm going to step through the code here: | 17:00 |
| corvus | mordred: https://etherpad.openstack.org/p/qCGtbe92ES | 17:01 |
| mordred | so d.swiftPath(path) should in _theory_ be sometihng like /v1/MossoCloudFS_622b11a1-5dfa-43b4-9f58-4ad3c6dbc4a0/intermediate_registry/files/{path} | 17:01 |
| *** e0ne has quit IRC | 17:01 | |
| fungi | clarkb: correct, you can even test it with an /etc/hosts entry currently | 17:01 |
| *** dims has joined #openstack-infra | 17:01 | |
| fungi | in case you want to make absolutely sure | 17:01 |
| clarkb | ok so we should be able to get the review.openstack.org redirect in place today and be in a similar spot for that redirect then we only need to update the cert values in hiera and the canonical hostname in the config | 17:01 |
| clarkb | I'm just mapping out my mental todo list then trying to get it onto the etherpad plan doc | 17:02 |
| *** kjackal_v2 has quit IRC | 17:03 | |
| *** kjackal has joined #openstack-infra | 17:03 | |
| clarkb | fungi: that redirect works with https too? (I assume it does just double checking things) | 17:04 |
| *** ricolin has quit IRC | 17:04 | |
| *** kopecmartin is now known as kopecmartin|off | 17:05 | |
| fungi | yes, we install the keys/certs on files.o.o with puppet | 17:05 |
| fungi | you should just be able to copy all the plumbing for the git sites | 17:06 |
| fungi | up through to hiera variables in the global site manifest | 17:06 |
| *** mattw4 has quit IRC | 17:06 | |
| clarkb | ok I think https://etherpad.openstack.org/p/the-great-opendev-git-migration is coming together. Basically just need to come up with the actual rename in gerrit and gitea plan | 17:07 |
| fungi | i'm starting to think i need to bump up the ethercalc in the rename generator's precedence, as i doubt we'll have time to get all of these things removed from openstack governance before friday | 17:19 |
| corvus | mordred: that was it! the secret key was wrong; starting the registry with it explicitly configured worked | 17:25 |
| mordred | corvus: woot! | 17:25 |
| corvus | that's very weird, because i obtained the secret key using, theoretically, the same method that the docker code is supposed to... but whatever. | 17:25 |
| mordred | corvus: well, that's an easy fix then | 17:25 |
| mordred | yeah | 17:26 |
| *** igordc has quit IRC | 17:26 | |
| corvus | clarkb, fungi, mordred: so i think i have everything i need to push up a change to the intermediate registry to use cloud files as a backend | 17:26 |
| mordred | corvus: woot! | 17:27 |
| clarkb | corvus: any ideaif garbage-collect is expected to work against that storage backend? | 17:27 |
| corvus | clarkb: based on my 50% understanding of how the registry and garbage collect works, i don't think it's any different | 17:27 |
| corvus | er, there may be unclarity here | 17:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Use zuul-ansible for ARA callback plugins https://review.openstack.org/653497 | 17:29 |
| corvus | clarkb: i think garbage-collect will work, but not work any differently (ie, registry still needs to be RO or offline) | 17:29 |
| corvus | clarkb: but i will verify at least that it works at all before i push up the change | 17:29 |
| clarkb | sounds good | 17:30 |
| *** markvoelker has quit IRC | 17:31 | |
| *** markvoelker has joined #openstack-infra | 17:31 | |
| corvus | clarkb: wow i spaced on https://review.openstack.org/653148 and forgot to plumb through the secrets; i've marked that WIP | 17:32 |
| *** markvoelker has quit IRC | 17:35 | |
| *** ralonsoh has quit IRC | 17:36 | |
| mnaser | clarkb: left a little suggestion | 17:37 |
| mnaser | and never mind | 17:39 |
| clarkb | mnaser: ya the redirect is expected to address that | 17:39 |
| mnaser | left my afterthought :P | 17:39 |
| mnaser | yup ok cool | 17:39 |
| tosky | uh, will zull provide compatibility links for cloned repositories like src/git.openstack.org/openstack/foo pointing to src/opendev.org/openstack/foo ? | 17:39 |
| fungi | tosky: i'm working on the change to patch references to those in everyone's job configs | 17:40 |
| fungi | which we'll push into all the repositories directly during the maintenance on friday | 17:40 |
| fungi | along with corrections to everyone's .gitreview files | 17:40 |
| corvus | clarkb: garbage-collect does work, but is *much* slower | 17:40 |
| tosky | fungi: ok, I will keep that out from my wip documentation patches | 17:41 |
| mnaser | fungi: it may be good to leave a note in the commit message to point to zuul docs, mentioning that you can actually avoid this entire thing if you used the zuul varaibles | 17:41 |
| fungi | mnaser: thanks, that's an excellent idea | 17:41 |
| corvus | mnaser: well... some of the zuul variables are keyed by canonical project name, which is changing in this case | 17:42 |
| tosky | but I will keep also an eye in case it's missed (we have a few of them in sahara roles and playbooks :) | 17:42 |
| corvus | (so in *some* cases it could be avoided, but i think not all) | 17:42 |
| mnaser | corvus: yep true, but for most things staying under openstack/ would have benefitted, but yes, I agree indeed | 17:42 |
| fungi | yeah, it doesn't cover all cases, but could have helped reduce the diff | 17:42 |
| mnaser | I know OSA is pretty bad but our run_tests.sh file is sync'd across all of our roles with openstack proposal bot | 17:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-diskimage_builder master: Use full gpg key fingerprint https://review.openstack.org/653499 | 17:43 |
| fungi | then again, this script probably isn't going to address all cases either, because not all of them will be findable programmatically | 17:43 |
| mnaser | so doing an o-a-t change would automatically push it for all of them | 17:43 |
| clarkb | infra-root ^ was the only thing that came out of puppet-4 on the nodepool hosts | 17:43 |
| corvus | we could probably make an ansible jinja filter plugin which does dynamic lookups, so you could do "zuul_project('nova')" and it would translate that into opendev.org/openstack/nova... or something similar. | 17:43 |
| mordred | corvus: yeah - using the zuul naming logic | 17:44 |
| mordred | corvus: that's probably not a terrible idea in general | 17:44 |
| *** igordc has joined #openstack-infra | 17:44 | |
| clarkb | corvus: we would be making a change for that too right? maybe best to just update to the new name in that case? | 17:44 |
| mordred | yeah - I think that's more "as a way to improve this for future cases" not "as a way to make friday easier" | 17:45 |
| clarkb | gotcha | 17:45 |
| * fungi is caught up on e-mail and irc and is going to be on silent running for a bit to better flesh out the git commit generator script for friday, though will be around for the storyboard meeting at 1900z (if there ends up being one) | 17:45 | |
| clarkb | I'm going to write the change for adding a review.openstack.org -> review.opendev.org redirect on files.o.o | 17:46 |
| clarkb | maybe someone else can pick up the update canonical hostname and logo change that is needed on the etherpad? | 17:46 |
| tosky | are you also going to backport the changes on all branches? (non-EM branches?) | 17:48 |
| fungi | tosky: not backport, no, the script is going to push commits to all open branches rewriting whatever it finds on them | 17:49 |
| mordred | clarkb: I'll pick up etherpad | 17:50 |
| tosky | fungi: oh, ok, direct push, but anyway all open branches will be fixed, thanks | 17:50 |
| fungi | tosky: those commits aren't going through code review, they'll be pushed directly on disk on the gerrit server | 17:50 |
| fungi | but yes | 17:50 |
| *** bobh has joined #openstack-infra | 17:50 | |
| fungi | "fixed" as best we can guess with a script, still possibly somewhat broken and in need of further patching | 17:51 |
| mordred | clarkb: how are you approaching the redirect on files? | 17:51 |
| fungi | mordred: clarkb: should just be able to copy the change i did to set up the git redirects on it | 17:52 |
| clarkb | mordred: I'm basically copying what fungi did for git.zuul-ci.org and friends. New vhosts for http and https that will redirect to review.opendev.org | 17:52 |
| mordred | ah - that makes sense | 17:52 |
| *** manjeets__ is now known as manjeets | 17:53 | |
| fungi | right, just need a vhost with the redirects baked into the vhost template, and a fistful of ssl files | 17:53 |
| *** arxcruz is now known as arxcruz|off|23 | 17:53 | |
| fungi | and then copy the existing ssl keys into group_vars/files.yaml on bridge | 17:53 |
| mordred | oh. hrm. where do I get those from? | 17:53 |
| mordred | (for etherpad) | 17:54 |
| * mordred is feeling dumb :) | 17:54 | |
| fungi | get them from group_vars/gerrit.yaml | 17:54 |
| corvus | mordred: what are you working on? | 17:54 |
| *** david-lyle is now known as dklyle | 17:55 | |
| mordred | OH WAIT | 17:55 |
| mordred | nevermind. nobody look at me | 17:56 |
| fungi | i saw nothing | 17:56 |
| corvus | nothing here | 17:56 |
| corvus | mordred: since you're available now, i think clarkb was saying there are some tasks on https://etherpad.openstack.org/p/the-great-opendev-git-migration that need doing | 17:56 |
| corvus | i was just going through that myself | 17:56 |
| mordred | corvus: yeah. I was starting to work on the apache vhost change ... but my brain mis-parsed what clarkb said earlier ... and I thought I was working on the vhost change _for_etherpad_ | 17:57 |
| corvus | that's what i was worried about :) | 17:57 |
| mordred | which is not actually the thing I need to be working on - so I'm now working on the actual needed thing | 17:57 |
| corvus | mordred: but i think clarkb is doing that vhost change? | 17:58 |
| mordred | he's doing the redirect change | 17:58 |
| corvus | ah gotcha | 17:58 |
| mordred | I'm doing update canonical hostname and logo | 17:58 |
| clarkb | yes | 17:58 |
| corvus | got it, i see all that in the etherpad now :) | 17:58 |
| mordred | clarkb: actually - since you're doing the redirect, doesn't that mean you're also adding the opendev vhost? | 17:59 |
| *** electrofelix has quit IRC | 18:00 | |
| clarkb | mordred: no the redirect is going on files.openstack.org | 18:02 |
| mordred | clarkb: kk | 18:03 |
| clarkb | and splitting it this way means we can test it ahead of time by merging my change whenever it is ready | 18:03 |
| clarkb | the change you are looking at should happen during the maintenance window | 18:03 |
| mordred | ah - gotit | 18:03 |
| *** Goneri has quit IRC | 18:06 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 18:07 |
| clarkb | infra-root if ^ that looks approximately correct feel free to +2 but not approve and I can approve when I've got hiera updated | 18:07 |
| fungi | another up-side to doing it on files.o.o is that it doesn't take effect until we update dns, so can merge and test it early | 18:08 |
| *** hwoarang has quit IRC | 18:08 | |
| fungi | if we tried to shoe-horn it onto the gerrit server that would be one more possibly complex and hard to test thing to coordinate during the maintenance | 18:09 |
| clarkb | hrm let me make sure mod alias is enabled | 18:09 |
| fungi | oh! | 18:10 |
| mordred | fungi: so - back to my question from earlier - the opendev vhost change needs review.opendev.org ssl certs - right? I only see one set of certs in group_vars/review.yaml | 18:10 |
| fungi | i just realized one reason we maybe can't do this | 18:10 |
| fungi | ssh api | 18:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 18:10 |
| mordred | fungi: oh right | 18:10 |
| fungi | :/ | 18:10 |
| mordred | yeah- I think we need to add the vhost on review.o.o itself because we need the dns to continue pointing there | 18:11 |
| clarkb | we are updating .gitreview files right? | 18:11 |
| clarkb | but ya maybe it does want to be there | 18:11 |
| mordred | we are - but that doesn't mean everyone is going to immediately update their remotes | 18:11 |
| fungi | we are, but not things like third-party ci systems listening on the stream | 18:11 |
| mordred | yeah | 18:11 |
| clarkb | in that case we'd incorporate what I've done into what mordred is doing (could be separate chagnes still) and update things such that a second vhost is set on that server | 18:12 |
| fungi | also, i don't think git-review notices changes in .gitreview (especially if you haven't pulled them in yet) so will need people to blow away their existing gerrit or push remotes too, right? | 18:12 |
| mordred | clarkb: yes - I agree | 18:12 |
| *** kjackal has quit IRC | 18:12 | |
| mordred | and I think it can be done like what you've go tthere - additional args to openstack_project::review - and just do a vhost there directly | 18:13 |
| clarkb | mordred: yup | 18:13 |
| clarkb | mordred: I'll move my stuff into review.pp | 18:13 |
| clarkb | then you can rebase on top of that | 18:13 |
| mordred | clarkb: cool. then I'll do mine on top of yours | 18:13 |
| clarkb | ++ | 18:13 |
| mordred | but - still ... where are certs for review.opendev.org ? | 18:13 |
| clarkb | mordred: /root/certs on bridge I think | 18:13 |
| *** psachin has quit IRC | 18:13 | |
| *** hwoarang has joined #openstack-infra | 18:14 | |
| fungi | basically workflow for folks will be something like... `git remote update; git checkout origin/master; git remote remove gerrit; git review -s` | 18:14 |
| mordred | clarkb: nope. only review.openstack there | 18:14 |
| clarkb | fungi: ^ any idea where you stashed them? | 18:14 |
| mordred | unless I can just use opendev.org - is that a wildcard cert? | 18:14 |
| clarkb | mordred: it is not | 18:14 |
| fungi | they're under a dated subdir | 18:14 |
| fungi | hold on | 18:14 |
| mordred | ah - I can find it | 18:14 |
| mordred | ./2019-03-26/review.opendev.org.crt yeah? | 18:15 |
| fungi | certs/2019-03-26/ | 18:15 |
| fungi | yeah | 18:15 |
| fungi | i did them at the same time as docs.opendev.org | 18:15 |
| mordred | cool. | 18:15 |
| *** kjackal has joined #openstack-infra | 18:16 | |
| *** mujahidali has joined #openstack-infra | 18:17 | |
| *** rfolco has quit IRC | 18:20 | |
| *** Goneri has joined #openstack-infra | 18:21 | |
| *** rfolco has joined #openstack-infra | 18:21 | |
| mordred | fungi, clarkb: I have put the review.opendev.org certs into group_vars as review_opendev_* | 18:22 |
| fungi | thanks! | 18:24 |
| mujahidali | clarkb: pabelanger: I have added simple-init openssh-server runtime-ssh-host-keys in the elemnts section but I am still getting the "Timeout waiting for ssh access" error. | 18:25 |
| mujahidali | Please have look at nodepool.yaml file http://paste.openstack.org/show/749442/ | 18:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Use zuul-ansible for ARA callback plugin detection https://review.openstack.org/653497 | 18:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo name mapping script https://review.openstack.org/653138 | 18:29 |
| fungi | okay, that ^ now also creates the repos.yaml input file for our rename_repos playbook | 18:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 18:29 |
| clarkb | mordred: fungi ^ how does that look? | 18:30 |
| fungi | also i reordered the matching so the ethercalc gets checked first, and as such we can use it to override any other decision later | 18:30 |
| clarkb | mordred: I think your change will then update the ssl_cert_contents and ssl_key_contents to point at new hiera keys as well as change vhost_name and set canonical hostname? | 18:30 |
| fungi | this way we can encode stuff in there to move to opendev even if we haven't gotten it excised from openstack governance first | 18:30 |
| clarkb | mujahidali: the next thing I would check is that the ssh key for the nodepool process is being created on the server image properly so that nodepool can ssh in | 18:32 |
| clarkb | mujahidali: you should be able to boot your image out of band setting your own ssh key then login as root | 18:32 |
| clarkb | mordred: and I think your change will actually have to go in first because we need to not conflict on the vhost name | 18:35 |
| clarkb | mordred: so feel free to rebase mine on top of yours | 18:35 |
| clarkb | skopeo says they don't want to diverge from docker | 18:35 |
| clarkb | mordred: or I can rebase when yours is up | 18:36 |
| mujahidali | clarkb: thank you, I will try out of band boot and see what comes next. | 18:37 |
| mordred | clarkb: yes on updating ssl_cert_contents to new keys | 18:38 |
| *** mattw4 has joined #openstack-infra | 18:40 | |
| mordred | clarkb: maybe we just want to squash them? I'm not sure either work or make sense without the other | 18:41 |
| clarkb | mordred: wfm | 18:41 |
| *** lpetrut has quit IRC | 18:41 | |
| fungi | infra-root: as a heads up, i'm rsyncnig the /home/gerrit2/review_site/git tree into /opt/fungi-test on review.o.o so i have a full set of the data there to test against. there is plenty of space in /opt | 18:41 |
| *** lpetrut has joined #openstack-infra | 18:42 | |
| fungi | i won't be running anything as root, so no chance i'll actually modify production data | 18:42 |
| fungi | huh, the contents of /home/gerrit2/review_site/git are only 11gib? for some reason i thought it was way larger. maybe that was just back when we were having git gc issues? | 18:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 18:44 |
| mordred | clarkb: how does that look? ^^ | 18:44 |
| clarkb | fungi: that number looks about right | 18:44 |
| clarkb | fungi: based on nodepool iamge sizes I think we are ~8gb of active repo data | 18:45 |
| mordred | fungi: also - when we first replicated into ceph and the block size was really big | 18:45 |
| fungi | ahh, yes i remember that now | 18:46 |
| clarkb | mordred: https://review.openstack.org/#/c/653507/4/modules/openstack_project/manifests/gerrit.pp line 71 I think we don't want to change that yet. gerrit will redirect you to "cgit" which will redirect you to opendev | 18:46 |
| clarkb | mordred: then as a followup we can update the rules in gerrit to directly point at opendev | 18:46 |
| mordred | clarkb: whyfore? | 18:46 |
| clarkb | mordred: you have to set up the rules to map to the gitea urls for that to work | 18:47 |
| mordred | clarkb: AH - gotcha | 18:47 |
| clarkb | also does git_http_url need similar treatment? | 18:48 |
| dmsimard | Just realized that the post pipeline doesn't trigger off of new tags which leaves a gap when expecting upload-git-mirror to run | 18:49 |
| mordred | clarkb: I thnk git_http_url is the http url for cloning | 18:49 |
| mordred | clarkb: so that shouldn't need any special deep-link handling | 18:49 |
| clarkb | gotcha | 18:49 |
| clarkb | dmsimard: there is a tag pipeline | 18:49 |
| mordred | clarkb: also - I updated review@openstack.org to review@opendev.org - is that also premature? | 18:49 |
| dmsimard | yeah, I suppose I'll add the job to it | 18:49 |
| clarkb | mordred: I think that should be ok though we may want spf records? | 18:50 |
| clarkb | I updated the etherpad with a note about adding spf records and updating the web repo url links to be direct links | 18:52 |
| fungi | i honestly don't know why we set up spf records for review.openstack.org in the first place. i think those predate my involvement in the community | 18:54 |
| fungi | oh, because it was for openstack.org not review.openstack.org | 18:55 |
| fungi | and openstack.org has spf records for osf reasons | 18:55 |
| clarkb | ya not actually sure if we need them but left a note so we don't forget to consider it | 18:55 |
| fungi | sending messages as anything at opendev.org is probably premature until we have some mta accepting e-mail for it | 18:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 18:58 |
| mordred | clarkb: k. updated | 18:58 |
| *** mujahidali has quit IRC | 18:58 | |
| clarkb | fungi: newer gerrit will accept email for itself but we aren't there yet | 18:58 |
| fungi | right | 18:59 |
| *** yboaron_ has quit IRC | 18:59 | |
| fungi | and then it will be something at review.opendev.org anyway, not just opendev.org | 18:59 |
| clarkb | ya | 18:59 |
| fungi | same for sb | 18:59 |
| fungi | once its smtp api gets developed anyway | 19:00 |
| *** ykarel|away has quit IRC | 19:02 | |
| clarkb | mordred: one thing given fungi's concern (missed email update cleanup) | 19:03 |
| clarkb | otherwise I think that looks pretty good | 19:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 19:06 |
| mordred | clarkb: whoops fixed | 19:06 |
| clarkb | mordred: that looks good to me. We should get people that aren't us to sanity check it though | 19:07 |
| mordred | clarkb: yes please :) | 19:08 |
| *** e0ne has joined #openstack-infra | 19:17 | |
| *** e0ne has quit IRC | 19:20 | |
| *** jcoufal has joined #openstack-infra | 19:21 | |
| *** Goneri has quit IRC | 19:21 | |
| *** e0ne has joined #openstack-infra | 19:22 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add support for yappi and objgraph output https://review.openstack.org/653541 | 19:30 |
| *** tjgresha has quit IRC | 19:33 | |
| *** tjgresha_nope has joined #openstack-infra | 19:33 | |
| *** Goneri has joined #openstack-infra | 19:34 | |
| *** tjgresha_nope has quit IRC | 19:40 | |
| *** lpetrut has quit IRC | 19:40 | |
| *** tjgresha has joined #openstack-infra | 19:44 | |
| prometheanfire | fungi: we are not using mailman2 right? the dmarc/dkim stuff was 'backported' to mm1? | 19:44 |
| clarkb | prometheanfire: we areusing 2 not 3 | 19:45 |
| *** rlandy|ruck is now known as rlandy|afk | 19:46 | |
| prometheanfire | right, I meant 3/2 not 2/1 | 19:47 |
| *** tjgresha has quit IRC | 19:51 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 19:55 |
| *** bobh has quit IRC | 19:56 | |
| clarkb | prometheanfire: is there context for the question? fwiw we are asking people to not change dkim moderation action settings on mailman | 20:04 |
| clarkb | (that was part of my email re the server upgrade) | 20:04 |
| prometheanfire | clarkb: no, it was a question based on some gentoo stuff | 20:06 |
| clarkb | ah. fungi has some work in progress exploring an upgrade to mailman3 which will give us a mroe friendly web ui (including posting from the ui aiui) | 20:07 |
| clarkb | but for now still on 2 | 20:07 |
| fungi | yeah, i have a (stale) poc up and running, and an etherpad listing the detailed steps i took to get it going, but aside from confirming it gets us out of our bolted-on whitelabeled mailman virtual hosts (now supports lists on multiple domains out of the box), i haven't done much else | 20:09 |
| fungi | also want to tear it down and try to do the same setup with containers so we're not stuck with an older version (mm3 is still under fairly heavy development) | 20:10 |
| fungi | https://etherpad.openstack.org/p/mm3poc | 20:10 |
| *** ijw has joined #openstack-infra | 20:11 | |
| *** ijw has joined #openstack-infra | 20:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 20:14 |
| clarkb | mordred: ^ changes so that tests can pass. | 20:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add support for yappi and objgraph output https://review.openstack.org/653541 | 20:15 |
| clarkb | infra-root reviews on 653507 so that we are ready for friday much appreciated | 20:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix for yappi output https://review.openstack.org/653549 | 20:17 |
| *** jcoufal has quit IRC | 20:22 | |
| *** jcoufal has joined #openstack-infra | 20:30 | |
| *** e0ne has quit IRC | 20:31 | |
| *** tjgresha has joined #openstack-infra | 20:34 | |
| *** tjgresha has quit IRC | 20:39 | |
| *** jcoufal has quit IRC | 20:39 | |
| *** Goneri has quit IRC | 20:40 | |
| *** pcaruana has quit IRC | 20:43 | |
| *** shardy has quit IRC | 20:45 | |
| corvus | clarkb: in 507, why the email address change? | 20:47 |
| rpioso | TheJulia: Should or perhaps must cherry picks have the same change-id footer as the source change? | 20:47 |
| clarkb | corvus: oh that came from mordred I think it is beacuse fungi pointed out we shouldn't use a domain until there is an mta running for it | 20:48 |
| corvus | clarkb: i'm pretty sure we've already created that account with that email address | 20:48 |
| clarkb | rpioso: I think the stable maintenance team asks that you do it for backports but it isn't strictly necessary in the general case | 20:48 |
| clarkb | corvus: ya we may not want to edit that one either | 20:48 |
| clarkb | corvus: I didn't write that half of the change, mordred did and I think it was due to fungi's feedback about about the lack of an mta | 20:48 |
| mordred | yeah. but I can remove that part of the chagne - fix coming | 20:49 |
| clarkb | mordred: don't forget to git review -d to pull down my updates | 20:49 |
| clarkb | ( I chagned some bits of the vhost stuff to make tests happier) | 20:49 |
| corvus | clarkb: since it's in production that way already, i feel like we should remove it; it will only cause production not to match our code :/ | 20:49 |
| clarkb | corvus: ya we can udpate it out of band if we decide it is necessary | 20:50 |
| clarkb | however gitea isn't very interactive so probably fine to leave as is | 20:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Redirect review.openstack.org to review.opendev.org https://review.openstack.org/653507 | 20:50 |
| clarkb | (eg gitea shouldn't send email) | 20:50 |
| corvus | clarkb: why would we update it -- it's already opendev.org? | 20:50 |
| mordred | fixed | 20:50 |
| corvus | i wrote it that way so we wouldn't have to change it... | 20:50 |
| rpioso | clarkb: Thank you. During our chat last week, I overlooked fungi's response -- http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-09.log.html#t2019-04-09T22:32:44 | 20:51 |
| * mordred needs to afk for a bit - biab | 20:51 | |
| * rpioso shrugs | 20:51 | |
| clarkb | corvus: aiui fungi doesn't want us to use domains that don't have an mta? I don't know if that is an actual concern | 20:51 |
| corvus | clarkb: i really don't understand | 20:51 |
| mordred | I think his concern was the use of an email address that wasn't an actual email address that mail could be sent to | 20:52 |
| clarkb | corvus: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-17.log.html#t2019-04-17T18:56:08 | 20:52 |
| corvus | how are gerrit@review.openstack.org and gerrit@review.opendev.org different in that respect? | 20:52 |
| clarkb | corvus: I think the "correct" fix would be to do @openstack.org according to fungi | 20:52 |
| clarkb | since that does have an mta | 20:52 |
| mordred | the actual review was about changes I'd made to update the gerrit emails to gerrit@review.opendev.org - I updated that playbook in error - I was mostly just trying to remove places where I'd made changes to the email addresses so that there would be no email address changes in the change | 20:53 |
| mordred | unfortunately, I undid one that I hadn't done in the first palce | 20:53 |
| mordred | and with that - I really do have to AFK for a bit now ... | 20:53 |
| corvus | mordred: thanks o/ | 20:54 |
| clarkb | I'm not particularly worried about the gitea case as gitea shouldn't be sending email | 20:54 |
| corvus | if everyone is sufficiently happy with the status quo there we can probably move on to something else now :) | 20:54 |
| *** tjgresha has joined #openstack-infra | 20:54 | |
| clarkb | gerrit on the other hand does send a bunch of email so maybe we leave that as is for no (which is what current change does) | 20:54 |
| *** tjgresha has quit IRC | 20:56 | |
| corvus | ya | 20:59 |
| *** whoami-rajat has quit IRC | 21:01 | |
| fungi | yes, i agree. i was mostly worried about high-volume gerrit e-mail | 21:01 |
| fungi | and in truth, i don't feel super strongly about not using an opendev.org address for gerrit too, but figure we'll start doing e-mail for that domain in the not too distant future and that might be a better time to change it | 21:08 |
| corvus | ya that all makes sense | 21:08 |
| *** rh-jelabarre has quit IRC | 21:09 | |
| *** rh-jelabarre has joined #openstack-infra | 21:09 | |
| corvus | is there a kolla swift image? | 21:14 |
| corvus | i found https://docs.openstack.org/kolla-ansible/pike/reference/swift-guide.html but not https://docs.openstack.org/kolla-ansible/stein/reference/swift-guide.html | 21:14 |
| tdasilva | corvus: are you looking for a swift-all-in-one container image? | 21:15 |
| corvus | let me ask a different question -- i'd like to be able to continue to test our intermediate registry after we switch to swift; that means i need a tiny little swift in order to do that; my first thought is a container image.... | 21:15 |
| corvus | tdasilva: yes! | 21:15 |
| tdasilva | corvus: try this: https://cloud.docker.com/u/swiftstack/repository/docker/swiftstack/picoswiftstack | 21:16 |
| *** slaweq has quit IRC | 21:17 | |
| corvus | tdasilva: thanks! | 21:17 |
| corvus | looks like https://hub.docker.com/r/swiftstack/picoswiftstack is the non-logged-in version of that url | 21:17 |
| corvus | tdasilva: any chance you know if you can provide credentials to that (eg via docker env settings) instead of just retrieving them like the docs describe? | 21:19 |
| *** slaweq has joined #openstack-infra | 21:20 | |
| tdasilva | corvus: looking | 21:22 |
| *** slaweq has quit IRC | 21:24 | |
| clarkb | infra-root based on the etherpad I've been trying to put together, https://etherpad.openstack.org/p/the-great-opendev-git-migration, and our story's task list I think the last remaining major items are the name update change generation from fungi and the need for a playbook to drive the whole thing | 21:26 |
| corvus | clarkb: i'll hack up the playbook | 21:27 |
| clarkb | corvus: thanks | 21:27 |
| clarkb | if anyone thinks of something else please add it to the etehrpad and/or the story list depending on what is most appropriate | 21:27 |
| corvus | tdasilva: i could also just use devstack i think | 21:28 |
| fungi | corvus: also, i think there's a proposed change or maybe merged which SotK wrote for storyboard to set up swift in a docker container for testing swift-backed story attachments, not sure if that helps | 21:28 |
| tdasilva | yeah, if you don't mind spinning up a vm, there are tools like devstack or https://github.com/thiagodasilva/ansible-saio | 21:29 |
| corvus | fungi: ah, it looks like that uses bouncestorage/swift-aio | 21:29 |
| *** Goneri has joined #openstack-infra | 21:29 | |
| weshay|rover | clarkb to the best of my knowledge.. zbr has killed the rechecks of our tripleo jobs, as requested | 21:29 |
| clarkb | weshay|rover: thank you! | 21:29 |
| weshay|rover | no prob man :) | 21:29 |
| tdasilva | corvus: bouncestorage/swift-aio is also a good option as that uses tempauth, so you just need to go inside container and set the auth as you would like | 21:29 |
| weshay|rover | thanks zbr! | 21:29 |
| clarkb | I'm going to pop out for a bit now. I've still got what I think are allergy related malaise so trying to not overdo it before friday. | 21:30 |
| corvus | tdasilva: that would probably work -- the main thing is that i need to be able to force the auth creds to be something known ahead of time (so that the part of the test which tests deployment is realistic) | 21:30 |
| clarkb | hard to tell if I'm getting properly sick or if all the plants have decided this is the week to kill me | 21:30 |
| tdasilva | clarkb: I feel like i'm under a combination of both | 21:31 |
| tdasilva | corvus: yeah, i think either one of those options will allow you to mock with the proxy config files and then you can just set the auth in tempauth filter | 21:32 |
| *** ijw has quit IRC | 21:34 | |
| mriedem | oh nice pip install seg fault http://logs.openstack.org/02/570202/8/check/grenade-py3/1625f50/logs/grenade.sh.txt.gz#_2019-04-16_18_13_00_689 | 21:34 |
| *** ijw has joined #openstack-infra | 21:34 | |
| *** logan- has quit IRC | 21:34 | |
| clarkb | mriedem: nice | 21:35 |
| mriedem | Apr 16 18:13:00 ubuntu-bionic-ovh-gra1-0005153120 kernel: pip3.6[11878]: segfault at 7f973189a000 ip 00000000004e8620 sp 00007ffc4b1c0370 error 4 in python3.6[400000+3b2000] | 21:35 |
| ianw | clarkb: though just then on removing the git.openstack.org cluster from replication; this will leave anyone hitting git:// targets with out-of-sync repos until it is completely shutdown? | 21:35 |
| *** Goneri has quit IRC | 21:36 | |
| fungi | ianw: yeah, maybe we stop haproxy on it | 21:36 |
| clarkb | ianw: git.openstack.org will be pointed to files.openstack.org for the redirection which means they will just break | 21:36 |
| fungi | oh, yep, perfect | 21:36 |
| clarkb | we can stop haproxy on it to be extra sure they break :) | 21:36 |
| *** logan- has joined #openstack-infra | 21:37 | |
| ianw | oh yeah, see that step, thanks | 21:37 |
| *** ijw has quit IRC | 21:40 | |
| *** slaweq has joined #openstack-infra | 21:41 | |
| *** ijw has joined #openstack-infra | 21:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use swift to back intermediate docker registry https://review.openstack.org/653613 | 21:41 |
| corvus | i haven't added swift to the tests for that yet -- it may or may not pass tests... | 21:42 |
| corvus | i expect that to take some more time, so, if it does pass tests, we might want to go ahead and merge it and add the testing later so we're not dealing with space issues | 21:42 |
| *** slaweq has quit IRC | 21:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add a one-time playbook for the opendev migration https://review.openstack.org/653614 | 21:45 |
| *** ijw has quit IRC | 21:48 | |
| *** Emine has quit IRC | 21:51 | |
| *** fdegir has quit IRC | 21:55 | |
| *** Goneri has joined #openstack-infra | 21:55 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for pip3.6 install seg fault bug 1825251 https://review.openstack.org/653617 | 21:58 |
| openstack | bug 1825251 in OpenStack-Gate "Segmentation fault during pip install on python3.6" [Undecided,Confirmed] https://launchpad.net/bugs/1825251 | 21:58 |
| *** kjackal has quit IRC | 22:01 | |
| *** kjackal has joined #openstack-infra | 22:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support Ansible 2.8 https://review.openstack.org/631933 | 22:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support Ansible 2.8 https://review.openstack.org/631933 | 22:06 |
| *** rh-jelabarre has quit IRC | 22:08 | |
| *** mriedem has quit IRC | 22:10 | |
| *** sreejithp has quit IRC | 22:15 | |
| *** ijw has joined #openstack-infra | 22:19 | |
| *** Goneri has quit IRC | 22:40 | |
| clarkb | I've updated the etherpad plan to disable the entire ansible cron instead of just putting review.o.o in the emergency file | 22:46 |
| clarkb | this way we can kick.sh on some representative hosts to ensure things are generally happy with the redirects | 22:46 |
| corvus | in 653613 tox-linters failed because ansible-lint is trying to lint a yaml file which is not ansible | 22:50 |
| corvus | should i attempt to fix it or remove ansible-lint? | 22:51 |
| *** kgiusti has left #openstack-infra | 22:51 | |
| clarkb | corvus: the issue is with our find I think | 22:52 |
| corvus | if so, we'd need to exclude "templates/*" or something | 22:52 |
| corvus | oh | 22:53 |
| corvus | or just make sure that .yaml is at the end | 22:53 |
| corvus | that might be easy | 22:53 |
| corvus | clarkb: how about this? -regex '.*.y[a]?ml$' | 22:53 |
| corvus | (just adding $ to the end there) | 22:53 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for pip3.6 install seg fault bug 1825251 https://review.openstack.org/653617 | 22:54 |
| openstack | bug 1825251 in OpenStack-Gate "Segmentation fault during pip install on python3.6" [Undecided,Confirmed] https://launchpad.net/bugs/1825251 | 22:54 |
| clarkb | won't that still match /home/zuul/src/git.openstack.org/openstack-infra/system-config/playbooks/roles/registry/templates/config.yml | 22:54 |
| corvus | also, i forgot to actually add the .j2 extension to the file i think | 22:54 |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| corvus | yeah, usually we do .foo.j2 in those dirs | 22:55 |
| corvus | hrm | 22:55 |
| corvus | i wonder if -regex already behaves like that and that's all i need to do | 22:55 |
| clarkb | oh I see ya if we rename the file to .j2 suffix and add the $ that should fix it I think | 22:55 |
| corvus | ah yep, -regex already works like that, so the only problem is lack of .j2 extension | 22:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use swift to back intermediate docker registry https://review.openstack.org/653613 | 22:57 |
| corvus | clarkb: i proposed that as an alternate to the /opt change (so everything will remain in /var afterwords) since it hasn't merged yet | 22:58 |
| clarkb | k | 22:58 |
| *** zigo has quit IRC | 22:58 | |
| clarkb | the migration playbook looks good too | 22:58 |
| *** auristor has quit IRC | 22:59 | |
| clarkb | tomorrow I'll add commands for doing db backups and the reindexing to the etherpad | 22:59 |
| corvus | /opt is at 55%, so i'll leave it alone for today | 22:59 |
| corvus | if we don't merge the swift thing tomorrow, i'll clear out the data | 22:59 |
| clarkb | things are coming together. Just need the rename all the things changes | 22:59 |
| corvus | clarkb: are we going with the etherpad for openstack-infra? | 23:00 |
| clarkb | corvus: I think so with the tweaks that mordred suggested for jeepy, git-review and glean | 23:01 |
| corvus | maybe i should take that and put it in the ethercalc? | 23:01 |
| clarkb | ++ | 23:01 |
| corvus | cool, will do that now | 23:01 |
| *** auristor has joined #openstack-infra | 23:01 | |
| clarkb | I'm reviewing the registry changes now | 23:02 |
| *** rlandy|afk is now known as rlandy|ruck | 23:03 | |
| clarkb | corvus: one comment on the registry change to make sure the cert config wasn't missed | 23:06 |
| clarkb | otherwise that lgtm | 23:06 |
| corvus | clarkb: fungi has some suggestions in https://etherpad.openstack.org/p/6CmVhW40m0 (groups especially, though i think trystack-site might be similar) | 23:07 |
| *** jamesmcarthur has joined #openstack-infra | 23:07 | |
| clarkb | looking | 23:07 |
| corvus | should i move those 3 to osf? | 23:07 |
| corvus | also odsreg | 23:09 |
| clarkb | ya if we are making an osf/ lets go ahead and move odsreg and groups* to there | 23:10 |
| corvus | and trystack-site? | 23:10 |
| clarkb | trystack was never an official thing was it? I think x/ might be most appropriate for that one now that I'm looking again | 23:10 |
| clarkb | either way it is being retired as a repo so probably doesn't matter too much if we pick oen or the other | 23:13 |
| clarkb | for infra-specs I'm ok with not renaming that one since it is still the opendev infra specs repo | 23:13 |
| corvus | clarkb: i put the first chunk into https://ethercalc.openstack.org/opendev-transition -- that look ok? (i added a note which should make it easy to see each of our chunks) | 23:14 |
| corvus | (rows 40-54) | 23:14 |
| corvus | oh, what should i do with bugdaystats? | 23:15 |
| clarkb | lets put that in openstack/ then openstack can decide if it wants to retire it or not | 23:15 |
| corvus | k | 23:16 |
| clarkb | and ya that first chunk lgtm | 23:16 |
| *** ijw has quit IRC | 23:16 | |
| *** ijw has joined #openstack-infra | 23:17 | |
| fungi | also, in case it was missed, i bumped the precedence for that ethercalc to the front of the renames generator, so we can just stuff all of this in the ethercalc and it will override any other possible later choices | 23:17 |
| corvus | fungi: yeah, i noted you mentioned that earlier, so that's what i'm doing now | 23:18 |
| fungi | spiff! | 23:18 |
| clarkb | corvus: oh wait the new name column should all be openstack/ in that first chunk right? | 23:18 |
| clarkb | not openstack-infra/ | 23:19 |
| fungi | i'm still futzing with popen calls to git commands | 23:19 |
| corvus | clarkb: yes, i copypastefailed | 23:19 |
| corvus | i got it right in the second chunk | 23:19 |
| clarkb | yup second chunk looking good so far | 23:19 |
| corvus | second chunk in place | 23:20 |
| corvus | oh, storyboard is opendev; i'll add that in | 23:20 |
| *** igordc has quit IRC | 23:21 | |
| *** aaronsheffield has quit IRC | 23:21 | |
| diablo_rojo | \o/ | 23:21 |
| *** jamesmcarthur has quit IRC | 23:23 | |
| corvus | given all the others, maybe we should move gear to opendev too? | 23:26 |
| clarkb | corvus: I figured I would defer to you on gear and restack. I'm happy for them to live in opendev as they have been somewhat important tools for us. But not necessary that they do if you prefer they live externally | 23:26 |
| clarkb | I do think that glean and git-review and jeepyb make a good home in opendev/ | 23:27 |
| corvus | ok, so from the x/ section, i'm moving git-restack git-review glean jeepyb gear | 23:27 |
| corvus | into opendev | 23:27 |
| clarkb | ++ | 23:28 |
| corvus | i have no idea what the two js- things are, nor what we should do with nose-html-output | 23:28 |
| fungi | seems like if git-review is opendev then git-restack probably is too | 23:29 |
| clarkb | https://git.openstack.org/cgit/openstack/swift/tree/test-requirements.txt#n10 swift uses nose-html-output (I don't think anyone else does) | 23:29 |
| clarkb | given that my vote for nose-html-output is openstack/ | 23:29 |
| fungi | i'd lean toward that choice too, yes | 23:29 |
| fungi | either openstack or x | 23:30 |
| fungi | since it's currently an official part of openstack, openstack makes more sense of the two | 23:30 |
| clarkb | also the infra team moved everyone off of nose years ago except swift which was happy with the status quo | 23:31 |
| clarkb | so I think openstack (swift) can own that and not us | 23:31 |
| clarkb | I'm looking up the js things now | 23:33 |
| corvus | ok i'll move nose-html-output to openstack | 23:33 |
| corvus | regarding x/ ... | 23:35 |
| corvus | is that a literal "x/" or is it a placeholder for "self-prefixed"? | 23:35 |
| clarkb | I've been using it as a literal 'x/' its a short less meaningful prefix than _/ or unknown/ | 23:36 |
| *** ijw has quit IRC | 23:36 | |
| clarkb | re the js things both are retired I think they are fine to be in x/ | 23:36 |
| corvus | i'm not sure "x/jenkins-job-builder" is what i would choose... | 23:37 |
| corvus | is thanh in irc? | 23:37 |
| clarkb | zxiiro-away: ^ | 23:37 |
| *** tosky has quit IRC | 23:38 | |
| corvus | i would suggest "jjb/jenkins-job-builder" for that... let's see what zxiiro-away says | 23:38 |
| corvus | i'll put that in the spreadsheet for now | 23:38 |
| clarkb | that wfm | 23:39 |
| fungi | the openstack tc recommended a namespace of "unknown" which had a somewhat negative connotation. "_" was suggested based on dockerhub's use of that as a meta-namespace, but there it means project explicitly vetted and managed by them (also "_" could get treated weirdly by some things maybe?), so "x" was suggested as being reasonably connotation-unencumbered and short | 23:39 |
| fungi | another alternative i suggested was "git" | 23:40 |
| fungi | basically we want something short and generic | 23:40 |
| corvus | zxiiro-away: here's some background: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html | 23:40 |
| *** ijw has joined #openstack-infra | 23:40 | |
| logan- | i'm going to temporarily disable the nodepool host aggregate on limestone and wait for the instances to drain to do some HV restarts. starting yesterday booting a nested virt instance instantly crashes the node. so i'm going to upgrade kernels on the HVs since that usually fixes it. | 23:42 |
| fungi | at least as a default https://opendev.org/x/foo is shorter than https://github.com/openstack/foo by 7 characters, so that's an incentive ;) | 23:43 |
| fungi | logan-: reports like that would be really helpful on the latest of the recurrent "why don't we use nested virt in jobs?" threads | 23:44 |
| corvus | clarkb, fungi: i believe i'm done with the ethercalc updates | 23:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support Ansible 2.8 https://review.openstack.org/631933 | 23:44 |
| clarkb | corvus: the 5 to opendev/ lgtm and jjb/jenkins-job-builder is a fine choice if zxiiro-away is happy with it | 23:45 |
| clarkb | that to say I think the list in ethercalc looks good | 23:45 |
| fungi | wfm | 23:46 |
| logan- | fungi: unfortunately for me our jobs use nested virt heavily and the jobs go from 20 minutes to 1+hr without nested KVM. so I have to keep it working for my sanity :( | 23:46 |
| fungi | i'm still hacking on loops within loops within loops within loops | 23:46 |
| corvus | fungi: i'm not sure why we keep calling it the opendev migration instead of the opendev inception | 23:46 |
| fungi | logan-: totally understand, we just keep hearing from, e.g., intel employees who are shocked and awed that nested virt isn't 100% stable and reliable | 23:47 |
| fungi | corvus: opendevification | 23:47 |
| clarkb | fungi: logan- right real world reports that it doesn't work 100% of the time in the real world is useful :) | 23:47 |
| logan- | fungi: ahh gotcha | 23:47 |
| *** mattw4 has quit IRC | 23:49 | |
| zxiiro-away | clarkb: corvus Just FYI I'm in Asia this week so my replies will be in a different timezone than usual. | 23:55 |
| zxiiro-away | clarkb: corvus I'm ok with the moving the namespace though having "openstack" prefixed probably no longer makes sense anyway. What's the proposed new namespace though is it jjb/* or opendev/* ? | 23:55 |
| clarkb | zxiiro-away: the current proposal for jenkins-job-builder is jjb/jenkins-job-builder | 23:56 |
| corvus | zxiiro-away: probably not opendev/ -- but you can have anything else you want, including jjb/ | 23:56 |
| zxiiro-away | clarkb: would python-jenkins go under there too? | 23:56 |
| clarkb | zxiiro-away: it can | 23:56 |
| zxiiro-away | cool yeah that works for me | 23:56 |
| zxiiro-away | jjb/python-jenkins and jjb/jenkins-job-builder | 23:56 |
| clarkb | but as corvus says you can pick somethign other than jjb/ unless you want jjb/ :) | 23:56 |
| corvus | that should be fine | 23:56 |
| clarkb | sounds great thnaks | 23:57 |
| corvus | is python-jenkins currently openstack/ ? | 23:57 |
| corvus | looks like yes | 23:57 |
| clarkb | yup | 23:57 |
| zxiiro-away | yep it's currently openstack/python-jenkins | 23:57 |
| corvus | ok, ethercalc updated with that (https://ethercalc.openstack.org/opendev-transition lines 196-197) | 23:58 |
| fungi | also, while it is some amount of work, the rename story after friday is somewhat less impactful for folks than previously given gitea's built-in redirecting | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!