| *** armstrong has quit IRC | 00:12 | |
| *** iurygregory has quit IRC | 01:16 | |
| *** iurygregory has joined #openstack-release | 01:17 | |
| *** iurygregory has quit IRC | 01:18 | |
| *** iurygregory has joined #openstack-release | 01:18 | |
| *** iurygregory has quit IRC | 02:08 | |
| *** iurygregory has joined #openstack-release | 02:09 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #openstack-release | 02:33 | |
| *** diablo_rojo has quit IRC | 02:40 | |
| *** armax has quit IRC | 02:55 | |
| *** armax has joined #openstack-release | 02:59 | |
| *** dulek has quit IRC | 03:09 | |
| *** dulek has joined #openstack-release | 03:19 | |
| *** armax has quit IRC | 03:57 | |
| *** ykarel|away has joined #openstack-release | 04:20 | |
| *** vishalmanchanda has joined #openstack-release | 04:30 | |
| *** ykarel|away is now known as ykarel | 04:39 | |
| hberaud | fungi: thanks | 05:07 |
|---|---|---|
| *** whoami-rajat has joined #openstack-release | 05:17 | |
| *** adriant has quit IRC | 05:43 | |
| *** slaweq has joined #openstack-release | 06:10 | |
| *** sboyron has joined #openstack-release | 06:21 | |
| *** eolivare has joined #openstack-release | 06:30 | |
| *** openstackgerrit has joined #openstack-release | 06:54 | |
| openstackgerrit | Hervé Beraud proposed openstack/releases master: Release ansible-role-atos-hsm for Wallaby https://review.opendev.org/c/openstack/releases/+/783908 | 06:54 |
| openstackgerrit | Hervé Beraud proposed openstack/releases master: Release ansible-role-thales-hsm for Wallaby https://review.opendev.org/c/openstack/releases/+/783905 | 06:55 |
| openstackgerrit | Takashi Kajinami proposed openstack/releases master: Puppet OpenStack Wallaby RC1 https://review.opendev.org/c/openstack/releases/+/784014 | 07:21 |
| *** tosky has joined #openstack-release | 07:33 | |
| openstackgerrit | Takashi Kajinami proposed openstack/releases master: Puppet OpenStack Wallaby RC1 https://review.opendev.org/c/openstack/releases/+/784014 | 07:34 |
| *** e0ne has joined #openstack-release | 07:35 | |
| openstackgerrit | Takashi Kajinami proposed openstack/releases master: Puppet OpenStack Wallaby RC1 https://review.opendev.org/c/openstack/releases/+/784014 | 07:39 |
| *** e0ne has quit IRC | 07:58 | |
| *** ykarel has quit IRC | 08:01 | |
| *** vishalmanchanda has quit IRC | 08:39 | |
| *** ykarel has joined #openstack-release | 08:43 | |
| *** ykarel is now known as ykarel|lunch | 08:43 | |
| openstackgerrit | Takashi Kajinami proposed openstack/releases master: Puppet OpenStack Wallaby RC1 https://review.opendev.org/c/openstack/releases/+/784014 | 08:58 |
| openstackgerrit | Takashi Kajinami proposed openstack/releases master: Puppet OpenStack Wallaby RC1 https://review.opendev.org/c/openstack/releases/+/784014 | 08:59 |
| *** vishalmanchanda has joined #openstack-release | 09:11 | |
| *** e0ne has joined #openstack-release | 09:14 | |
| openstackgerrit | Merged openstack/releases master: Release Bifrost 10.2.0 and create stable/wallaby https://review.opendev.org/c/openstack/releases/+/783665 | 09:16 |
| *** gibi is now known as gibi_away | 09:52 | |
| *** e0ne has quit IRC | 09:59 | |
| *** ykarel|lunch is now known as ykarel | 10:04 | |
| hberaud | ttx, elod: o/ Any idea concerning this error? => https://zuul.opendev.org/t/openstack/build/3c94d9fe7dbc41fb82030a7e5adbf88a/log/job-output.txt#4257 those both patches face the same problem => https://review.opendev.org/c/openstack/releases/+/783905 https://review.opendev.org/c/openstack/releases/+/783908 | 10:09 |
| hberaud | yet I think I've properly configuered templates => https://review.opendev.org/c/openstack/project-config/+/784011 | 10:10 |
| ttx | looking | 10:10 |
| ttx | Looks like rate limiting on Pypi | 10:11 |
| ttx | Not great news for us if it sticks | 10:13 |
| elod | :S | 10:14 |
| ttx | yeah, I suspect it's Pypi returning 429 (too many requests) on that XMLrpc requests | 10:15 |
| ttx | See https://github.com/pypa/warehouse/issues/8753 | 10:17 |
| ttx | we may have to introduce a sleep in there | 10:18 |
| ttx | or an incremental retry | 10:22 |
| ttx | or change the way we retrieve roles since XMLrpc seems to only be used for that | 10:24 |
| ttx | https://opendev.org/openstack/releases/src/branch/master/openstack_releases/pythonutils.py#L141 for those following at home | 10:25 |
| ttx | ah, wait | 10:25 |
| hberaud | hm ok... but then why other job pass? | 10:27 |
| hberaud | more or less at the same time | 10:27 |
| ttx | just a sec | 10:27 |
| hberaud | sure | 10:27 |
| ttx | testing an assumption here | 10:28 |
| ttx | https://zuul.opendev.org/t/openstack/build/3c94d9fe7dbc41fb82030a7e5adbf88a/log/job-output.txt#4227 shows that it retrieves roles twice | 10:30 |
| *** e0ne has joined #openstack-release | 10:31 | |
| ttx | so that explains why it hits the TooManyRequests limit | 10:31 |
| ttx | First request fails, and it tries again | 10:31 |
| ttx | For https://review.opendev.org/c/openstack/releases/+/783908 the error seems different? | 10:32 |
| elod | no, it's the same: https://zuul.opendev.org/t/openstack/build/f72a8875b48d4eef9b9f6f94ff5636c4/log/job-output.txt#4218-4219 | 10:36 |
| elod | I've checked a passing patch's log and it has only one 'retrieving roles for' debug log: https://zuul.opendev.org/t/openstack/build/fb8d23ea25c24ae6888d08c87f60adef/log/job-output.txt#4203 | 10:39 |
| elod | now the question is why we have two in the failing cases :) | 10:40 |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Avoid PyPI ratelimit on roles retrieval retry https://review.opendev.org/c/openstack/releases/+/784068 | 10:40 |
| ttx | It's because the package does not exist | 10:40 |
| ttx | https://pypi.org/project/ansible-role-atos-hsm/ | 10:40 |
| ttx | so it tries to retrieve roles, that fails, then it retries with the canonicalized version of the name (which in this case is the same) | 10:41 |
| hberaud | yes they were not released yet | 10:41 |
| elod | sounds logical | 10:41 |
| ttx | OK the patch above should fix it | 10:42 |
| hberaud | however that's not the first time that we created a new deliverable | 10:42 |
| ttx | Other validation runs do not hit that because they are created already | 10:42 |
| ttx | Sure, but the rate limit was added relatively recently | 10:42 |
| ttx | maybe our new host is fatser etc | 10:43 |
| hberaud | ah I see | 10:43 |
| hberaud | yes that make sense | 10:43 |
| ttx | A more comprehensive improvement would be to only retry if the canonicalized name is different from the name on the 1st try | 10:44 |
| ttx | but I guess retying can't hurt, maybe other transient failures can make it fail on 1st try | 10:44 |
| ttx | lunchtime! | 10:45 |
| hberaud | bon appetit | 10:46 |
| hberaud | For now let's observe if that help | 10:46 |
| elod | and I saw that xmlrpc is not recommended and json api would be better ( https://wiki.python.org/moin/PyPIXmlRpc ) | 10:49 |
| elod | but for now this fix hopefully solves our issue | 10:49 |
| hberaud | +1 | 10:49 |
| hberaud | improvements could comes with follow up patches | 10:49 |
| elod | and: Bon appétit! :) | 10:50 |
| hberaud | Merci | 10:50 |
| hberaud | :) | 10:50 |
| *** mugsie__ is now known as mugsie | 11:01 | |
| openstackgerrit | Merged openstack/releases master: Avoid PyPI ratelimit on roles retrieval retry https://review.opendev.org/c/openstack/releases/+/784068 | 11:15 |
| hberaud | well, retrying | 11:15 |
| elod | it works \o/ | 11:25 |
| hberaud | \o/ | 11:26 |
| hberaud | kudos ttx | 11:26 |
| elod | yes-yes, nice catch :) | 11:27 |
| hberaud | elod, ttx: so now these patches just need a final +2 https://review.opendev.org/c/openstack/releases/+/783908 https://review.opendev.org/c/openstack/releases/+/783905 | 11:31 |
| hberaud | after this point only 2 ironic* stable branches will still miss | 11:32 |
| hberaud | however, according to our process, I think that when those 2 patches ^ will be merged we will be able to branch requirements | 11:34 |
| *** dtantsur|afk is now known as dtantsur | 11:34 | |
| hberaud | The branching missing projects and the branching of requirements seem concurrent and independent tasks https://releases.openstack.org/reference/process.html#r-2-week | 11:36 |
| hberaud | s/branching missing/branching of missing/ | 11:37 |
| ttx | +2a | 11:43 |
| elod | oh, I was late. | 11:45 |
| elod | so the 1st one is now super-approved. | 11:45 |
| elod | :] | 11:45 |
| hberaud | thanks guys | 11:48 |
| fungi | catching up, looks like you were able to work out a likely cause for the pypi error. nothing needs to be rerun now? | 11:59 |
| hberaud | fungi: it was a false positive, https://review.opendev.org/c/openstack/releases/+/784068 . We don't have things to reenqueue | 12:05 |
| fungi | good, good... | 12:05 |
| hberaud | fungi: thanks for your diligence | 12:06 |
| fungi | just making sure as i try to work through priorities for my morning | 12:06 |
| fungi | no sweat, always happy to help | 12:06 |
| openstackgerrit | Merged openstack/releases master: Release ansible-role-atos-hsm for Wallaby https://review.opendev.org/c/openstack/releases/+/783908 | 12:07 |
| openstackgerrit | Merged openstack/releases master: Release ansible-role-thales-hsm for Wallaby https://review.opendev.org/c/openstack/releases/+/783905 | 12:08 |
| openstackgerrit | Merged openstack/releases master: Release manila-tempest-plugin for wallaby https://review.opendev.org/c/openstack/releases/+/783368 | 12:08 |
| hberaud | :) | 12:18 |
| *** nweinber has joined #openstack-release | 12:28 | |
| openstackgerrit | Merged openstack/releases master: Release barbican-tempest-plugin for wallaby https://review.opendev.org/c/openstack/releases/+/783341 | 12:30 |
| *** smcginnis has quit IRC | 12:53 | |
| hberaud | gmann: o/, How is the situation from the QA team perspective concerning your release duties? https://wiki.openstack.org/wiki/QA/releases | 12:59 |
| dtantsur | hberaud: hi! we're working towards proposing ironic and IPA releases by EOD | 13:13 |
| hberaud | dtantsur: excellent, thanks | 13:13 |
| *** armstrong has joined #openstack-release | 13:27 | |
| gmann | hberaud: are all projects branched? if so QA can start the process | 13:38 |
| hberaud | 2 projects remains unbranched for now ^ | 13:39 |
| hberaud | gmann: https://review.opendev.org/c/openstack/releases/+/783675 https://review.opendev.org/c/openstack/releases/+/783696 | 13:39 |
| hberaud | gmann: I'll ping you when they will be merged | 13:40 |
| gmann | hberaud: ok, I will keep eyes on these and start QA process accordingly, thanks | 13:40 |
| hberaud | That's an option | 13:40 |
| hberaud | thank you gmann | 13:40 |
| gmann | np! | 13:40 |
| armstrong | @hberaud I pinged the projects on IRC and one only one project have responded | 13:41 |
| hberaud | armstrong: ack thanks for the heads up | 13:41 |
| armstrong | np | 13:42 |
| hberaud | armstrong: I seen your private message | 13:42 |
| armstrong | is there something else I can do? | 13:42 |
| hberaud | not much for now | 13:43 |
| armstrong | ok | 13:43 |
| *** rpioso is now known as rpioso|afk | 13:55 | |
| openstackgerrit | Hervé Beraud proposed openstack/releases master: Improve R-2 documentation https://review.opendev.org/c/openstack/releases/+/784103 | 14:11 |
| *** armax has joined #openstack-release | 14:18 | |
| hberaud | elod, ttx: FYI I don't think that we need to expect PTL/liaison validation for this one https://review.opendev.org/c/openstack/releases/+/783365 https://zuul.openstack.org/build/7305e115025243779531bf323d8b171e | 14:32 |
| dtantsur | JFYI I'll leave earlier today, Julia will coordinate the final ironic releases | 14:33 |
| hberaud | ack | 14:33 |
| *** ykarel is now known as ykarel|away | 14:40 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/releases master: Creating stable wallaby for ironic-python-agent https://review.opendev.org/c/openstack/releases/+/783675 | 14:47 |
| ttx | ok approved | 14:48 |
| iurygregory | dtantsur, hberaud fyi ^ | 14:48 |
| hberaud | ack | 14:49 |
| hberaud | iurygregory: thanks | 14:49 |
| iurygregory | np | 14:50 |
| dtantsur | iurygregory, TheJulia, I wonder if https://docs.openstack.org/releasenotes/ironic-python-agent/unreleased.html#upgrade-notes and the vmedia change justify a major bump. | 14:50 |
| * iurygregory looks | 14:51 | |
| iurygregory | dtantsur, yeah I agree i can be a major bump to 7.0.0 | 14:54 |
| * iurygregory waits for TheJulia to confirm =) | 14:54 | |
| TheJulia | hmm | 14:55 |
| TheJulia | looking | 14:55 |
| ykarel|away | hberaud, looks like tarballs are not being pushed https://tarballs.opendev.org/openstack/barbican-tempest-plugin/?C=M;O=D | 14:55 |
| TheJulia | Yes, major version bump please | 14:55 |
| TheJulia | 7.0.0 for IPA | 14:55 |
| iurygregory | ack | 14:55 |
| ykarel|away | same for https://tarballs.opendev.org/openstack/manila-tempest-plugin/?C=M;O=D | 14:56 |
| ykarel|away | release jobs passed | 14:56 |
| ykarel|away | https://e263e188bf593a1accc9-15a9cdd27c1ff21de3e948d542da4662.ssl.cf1.rackcdn.com/d887d76c1f7d4e3a0630b5b9886c39c1bfb05263/release/release-openstack-python/7c42f4c/job-output.txt | 14:56 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/releases master: Creating stable wallaby for ironic-python-agent https://review.opendev.org/c/openstack/releases/+/783675 | 14:57 |
| openstackgerrit | Merged openstack/releases master: Release keystone-tempest-plugin for wallaby https://review.opendev.org/c/openstack/releases/+/783365 | 15:00 |
| fungi | ykarel|away: this was the build for that: https://zuul.opendev.org/t/openstack/build/7c42f4c7e93545918a321da1e91f941a | 15:01 |
| fungi | and /afs/.openstack.org/project/tarballs.opendev.org/openstack/barbican-tempest-plugin/barbican_tempest_plugin-1.3.0.tar.gz does exist, but we may be behind on syncing the read-only replica | 15:04 |
| fungi | i'll look into it | 15:04 |
| hberaud | We didn't get related notification about job failure so not sure that's a job failure http://lists.openstack.org/pipermail/release-job-failures/2021-March/thread.html | 15:05 |
| ykarel|away | fungi, Thanks | 15:05 |
| ykarel|away | hberaud, job passed but tarball not yet exists | 15:05 |
| hberaud | yes | 15:05 |
| hberaud | looks more a replication issue | 15:05 |
| fungi | hberaud: ykarel|away: yeah, i don't think there was a job failure. also the tarball does exist, it's just not being served by the tarballs site yet | 15:06 |
| fungi | it's in the read-write afs volume | 15:06 |
| fungi | yep, confirmed, i think we didn't consider that our decision to add a new replica on afs01.ord for better redundancy would delay volume updates for the other replicas | 15:08 |
| fungi | so nothing's "broken" i don't think, this is a self-imposed delay in the publication process | 15:09 |
| hberaud | ack | 15:10 |
| *** noonedeadpunk has quit IRC | 15:27 | |
| *** noonedeadpunk has joined #openstack-release | 15:28 | |
| *** ykarel|away has quit IRC | 15:37 | |
| *** mtreinish has joined #openstack-release | 15:53 | |
| prometheanfire | reqs branch waiting on IPA and the like? | 15:56 |
| *** rpioso|afk is now known as rpioso | 16:00 | |
| hberaud | prometheanfire: yes | 16:00 |
| hberaud | IPA is on the rails but it still miss the other ironic patch https://review.opendev.org/c/openstack/releases/+/783696 | 16:01 |
| prometheanfire | kk | 16:01 |
| hberaud | prometheanfire: and normally we also need to wait for the QA team duties before branching reqs, however they also wait for the ironic patches | 16:02 |
| hberaud | elod, ttx: Please can one of you can validate this patch => https://review.opendev.org/c/openstack/releases/+/783675/ | 16:04 |
| *** diablo_rojo has joined #openstack-release | 16:06 | |
| elod | hberaud: +2+W'd | 16:22 |
| hberaud | elod: thanks | 16:22 |
| elod | np | 16:22 |
| iurygregory | hberaud, re ironic patch, we are waiting for some changes to merge on master and I will update | 16:29 |
| hberaud | ack | 16:29 |
| hberaud | iurygregory: which patches? | 16:29 |
| hberaud | s/patches/changes/ | 16:29 |
| iurygregory | 1min let me get the links | 16:29 |
| iurygregory | I think they are on gate already | 16:30 |
| hberaud | ok | 16:30 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/744117 - just got merged | https://review.opendev.org/c/openstack/ironic/+/744117 - on gate | 16:31 |
| hberaud | ok thanks | 16:31 |
| iurygregory | I'm reviewing to see if there is something else | 16:31 |
| hberaud | both patches are the same ^ | 16:32 |
| iurygregory | enr XD | 16:33 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/721593 | 16:36 |
| iurygregory | and probably https://review.opendev.org/c/openstack/ironic/+/783759 | 16:36 |
| hberaud | iurygregory: ok. Do you think it will be possible to merge the last one ASAP? | 16:38 |
| hberaud | https://review.opendev.org/c/openstack/ironic/+/783759 | 16:38 |
| iurygregory | yeah CI should be happy | 16:38 |
| hberaud | ok | 16:38 |
| iurygregory | I ask Julia =) | 16:38 |
| iurygregory | I will* | 16:38 |
| hberaud | thanks for your feedback | 16:38 |
| hberaud | ok | 16:38 |
| openstackgerrit | Marios Andreou proposed openstack/releases master: WIP Make tripleo repos release for stable/ussuri branch https://review.opendev.org/c/openstack/releases/+/784124 | 16:40 |
| openstackgerrit | Merged openstack/releases master: Creating stable wallaby for ironic-python-agent https://review.opendev.org/c/openstack/releases/+/783675 | 16:42 |
| *** eolivare has quit IRC | 16:43 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/releases master: Create stable/wallaby for devstack https://review.opendev.org/c/openstack/releases/+/784132 | 17:00 |
| openstackgerrit | Ghanshyam Mann proposed openstack/releases master: Create stable/wallaby for grenade https://review.opendev.org/c/openstack/releases/+/784134 | 17:03 |
| gmann | hberaud: ^^ pushed devstack/grenade branch with depends-on on Ironic patch. once that is merged these are all good to go | 17:04 |
| hberaud | gmann: ak | 17:04 |
| hberaud | ack | 17:04 |
| hberaud | thanks | 17:04 |
| *** dtantsur is now known as dtantsur|afk | 17:04 | |
| openstackgerrit | Hervé Beraud proposed openstack/releases master: add branch for openstack requirements https://review.opendev.org/c/openstack/releases/+/779832 | 17:10 |
| hberaud | prometheanfire: I rebased the requirements branch on HEAD => https://opendev.org/openstack/requirements/commit/040e24a73f93c5f1c81fc2312a3552f4cb972cc5 and I added two depends-on gmann is patches ^ | 17:11 |
| hberaud | prometheanfire: these patches themselves have a depends-on on => https://review.opendev.org/c/openstack/releases/+/783696 (the missing branch) | 17:13 |
| * hberaud AFK for awhile, be back later | 17:13 | |
| openstackgerrit | Michael Johnson proposed openstack/releases master: Release Octavia Wallaby RC2 https://review.opendev.org/c/openstack/releases/+/784139 | 17:21 |
| openstackgerrit | Ghanshyam Mann proposed openstack/releases master: Create stable/wallaby for devstack https://review.opendev.org/c/openstack/releases/+/784132 | 17:33 |
| openstackgerrit | Ghanshyam Mann proposed openstack/releases master: Create stable/wallaby for grenade https://review.opendev.org/c/openstack/releases/+/784134 | 17:34 |
| *** vishalmanchanda has quit IRC | 17:51 | |
| prometheanfire | hberaud: ok | 19:03 |
| hberaud | iurygregory, TheJulia: The CI seems happy here => https://review.opendev.org/c/openstack/ironic/+/783759 and the other patch is now merged | 19:20 |
| TheJulia | hberaud: help me understand real quick, is the pressure because you guys want to branch requirements and see us being branched first as a requirement to that? | 19:23 |
| hberaud | TheJulia: We need the ironic's wallaby stable branch to cut requirements | 19:25 |
| TheJulia | is ironic in requirements? | 19:26 |
| TheJulia | I mean, its kind because traditionally QA/release teams have just branch and broken devstack right out from under us at the end of the cycle | 19:26 |
| TheJulia | I'm just confused why now | 19:26 |
| TheJulia | I'm surely missing something | 19:27 |
| hberaud | I don't think that ironic is in requs, concerning myself is more that our process ask for all stable branches before cutting reqs => https://releases.openstack.org/reference/process.html#r-2-week | 19:28 |
| hberaud | but maybe you are right and we don't really need this one... and process are outdated or misleading | 19:28 |
| hberaud | prometheanfire: any opinion? ^ | 19:29 |
| TheJulia | I mean, that is openstack at large's process, doesn't mean everyone must conform precisely to it. | 19:29 |
| TheJulia | Granted, I want to get the release out and lay down and sleep for a day | 19:29 |
| TheJulia | but yeah | 19:29 |
| hberaud | To drive the coordinated releases we need referential to organize each steps and our existing process (https://releases.openstack.org/reference/process.html) is my referential for now | 19:31 |
| TheJulia | I've got a patch that has to be backported in the queue now, regardless, so its not a big deal to cut without that, and I've kind of given a contributor a little bit of time to fix a lack of an api version check we spotted at the last minute. | 19:32 |
| prometheanfire | hberaud: it's more that the projects can start pointing at the stable-reqs branch once it's cut I think | 19:32 |
| TheJulia | prometheanfire: that was my understanding as well, fwiw | 19:32 |
| TheJulia | hberaud: i guess this is your first time driving the process... | 19:33 |
| hberaud | TheJulia: exactly | 19:33 |
| hberaud | I think that we can wait until tomorrow or early on Friday | 19:33 |
| TheJulia | Sean would have just done it and ignored us which explains why I feel more frustrated about releases this year | 19:33 |
| TheJulia | well, cycle | 19:33 |
| TheJulia | hberaud: yeah, my hope is by EOD at this momen | 19:34 |
| prometheanfire | ya, not a huge deal, but I'd like to unfreeze soon | 19:34 |
| TheJulia | I basically said "has to pass on first go" to the contributor | 19:34 |
| TheJulia | prometheanfire: ++ | 19:34 |
| hberaud | Ok let's wait a bit more | 19:34 |
| TheJulia | you guys are free to go ahead and branch requirements I think, its super unlikely to break us immediately | 19:34 |
| TheJulia | I guess what I'm saying is "we're used to getting broken, it won't come as surprising to us at the last moment" | 19:35 |
| TheJulia | espescialy when things ran longer than we expected | 19:36 |
| * TheJulia sees a 2nd attempt appear and starts getting worried about ci | 19:40 | |
| prometheanfire | we've backported stuff to stable branches before as well | 19:45 |
| prometheanfire | to branch reqs faster | 19:45 |
| TheJulia | we had to do that with like 16 patches for ironic once | 19:45 |
| TheJulia | it was a nightmare | 19:45 |
| prometheanfire | :D | 19:45 |
| *** sboyron has quit IRC | 20:12 | |
| *** nweinber has quit IRC | 20:14 | |
| johnsom | Just adding $0.02. Octavia also had a hard time with all of the ci outages (ddos, caches down, memory issues, DNS problems, etc.) over the last few weeks trying to get in a few very important bugs. It is the nature of our release cycle. Yeah, it's not pleasant to have to again try to get the backports in for another release, but that is what we have decided is in the best interest of the stakeholders. | 20:36 |
| johnsom | Octavia has been proposed to not be a part of a release in the past because we couldn't get a patch through the gates for a stable cut. | 20:36 |
| clarkb | johnsom: what were the caches, memory, and dns issues? (I was out last week and aware of the gitea troulbes, but the others are not immediately familiar) | 20:37 |
| johnsom | We reluctantly decided we needed to move forward with RC1 on schedule and planned for RC2 (proposed today). | 20:37 |
| clarkb | oh zuul memory issues? | 20:38 |
| johnsom | clarkb Yes. | 20:38 |
| johnsom | ddos was someone going direct to opendev git for a deployment, killing git checkouts | 20:38 |
| johnsom | memory issues took down parts of zuul causing jobs to time out | 20:39 |
| clarkb | ya I know that one. Still trying ti girue out what caches and dns problems were | 20:39 |
| clarkb | more accurately the zuul memory leak causes the jobs to restart but has a similar impact | 20:39 |
| johnsom | cache issues were package not found | 20:39 |
| johnsom | That one might have been tracked back to a cdn outage | 20:40 |
| johnsom | DNS was the resolution over IPv6 was not resolving the mirror addresses (no answer flag in unbound log). | 20:40 |
| fungi | there was an observed job failure where a dns lookup in ovh returned a pypi lookup failure, logs collected from the node indicated that unbound had gotten a valid record and handed it back to the requesting process, so unclear why it seemed to think the name was not found (suspicion is that the error reported by pip was misleading) | 20:41 |
| johnsom | The one I saw unbound didn't get an answer to the query | 20:41 |
| fungi | also not sure why that job was going directly to pypi instead of our mirror servers | 20:41 |
| fungi | johnsom: oh, the second one, yeah that was in limestone and looked more like local network connectivity failure in the provider's network | 20:42 |
| johnsom | Yeah, it happens, we have seen it before | 20:42 |
| johnsom | Basically the trail of tears is captured on this patch: https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/774157 and it's pair in the octavia repo. | 20:42 |
| johnsom | Anyway, this is why we have a few weeks after the stable cut to get any critical patches in. | 20:44 |
| clarkb | ok, just making sure we're trying to stay on top of things. I know fungi worked through gitea last week and we're poking at zuul now to improve that. Sounds like the other issue are less persistent and may be issues adjacent to us but not directly in our control | 20:44 |
| johnsom | Yeah, the DNS issue is one you and I have looked at before as well. | 20:45 |
| fungi | we may want to consider turning off our use of the limestone cloud if connectivity continues to be spotty in there | 20:46 |
| *** whoami-rajat has quit IRC | 20:47 | |
| johnsom | If you want to run it all to ground I can extract the important bits for each issue into an etherpad. | 20:47 |
| clarkb | the focus is definitely on the zuul memory issues right now, but ya I can try and take a look at it again when the othre issues settle a bit | 20:48 |
| johnsom | The gotcha is the logs may expire. But if you want I'm happy to try to extract the highlights. | 20:49 |
| *** e0ne has quit IRC | 21:00 | |
| *** dhellmann_ has joined #openstack-release | 21:12 | |
| *** dhellmann has quit IRC | 21:12 | |
| *** dhellmann_ is now known as dhellmann | 21:14 | |
| *** dhellmann has quit IRC | 21:16 | |
| *** dhellmann has joined #openstack-release | 21:17 | |
| *** dhellmann has quit IRC | 21:23 | |
| *** dhellmann has joined #openstack-release | 21:25 | |
| *** dansmith has quit IRC | 21:33 | |
| *** dansmith has joined #openstack-release | 21:38 | |
| *** e0ne has joined #openstack-release | 21:40 | |
| *** e0ne has quit IRC | 21:51 | |
| johnsom | clarkb fungi Here are the ones that were easy to find: https://etherpad.opendev.org/p/wallaby-RC1-ci-challenges | 22:02 |
| clarkb | thanks! | 22:02 |
| *** tosky has quit IRC | 23:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!