| opendevreview | Dale Smith proposed openstack/election master: [2024.2] Propose dalees candidacy for Adjutant https://review.opendev.org/c/openstack/election/+/909803 | 00:05 |
|---|---|---|
| *** diablo_rojo is now known as Guest500 | 03:09 | |
| *** enick_952 is now known as diablo_rojo | 03:09 | |
| spotz[m] | Hey all: New topic for discussion from the ML - https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/UTA7RJCNFVT52EUUGNELDLDNVOHAFCGZ/ | 13:21 |
| opendevreview | Martin Kopec proposed openstack/election master: Add Martin Kopec candidacy for QA 2024.2 PTL https://review.opendev.org/c/openstack/election/+/909871 | 14:28 |
| opendevreview | Axel Vanzaghi proposed openstack/election master: Adding Axel Vanzaghi candidacy for Mistral https://review.opendev.org/c/openstack/election/+/909901 | 15:38 |
| frickler | JayF: according to https://github.com/rthalley/dnspython/issues/1050 there is a known issue with latest dnspython and eventlet, would have been nice to tell the reqs team | 18:51 |
| JayF | I am not actively working on eventlet, more doing some early coordination and now shifting to trying to get the migration plan done | 18:52 |
| JayF | This is the first knowledge I have of any of this | 18:52 |
| JayF | dtantsur: ^ could this be related to the issue you saw this morning | 18:53 |
| frickler | hmm, seems a fix is already in eventlet, just waiting for a release https://github.com/eventlet/eventlet/issues/913 :-/ | 18:59 |
| JayF | getting *that* done more quickly; that I can do :D | 18:59 |
| frickler | bit of bad timing with reqs freeze next week, but can't blame external projects for it I gues | 19:00 |
| JayF | #919 says 0.35.2 is out, fixing that issue | 19:00 |
| JayF | https://pypi.org/project/eventlet/0.35.2/ confirmed | 19:00 |
| clarkb | that release is from yesterday. | 19:00 |
| clarkb | perhaps just race in communication and testing? | 19:00 |
| JayF | was there some newer break? The issue frickler linked was 2 days ago last updated | 19:01 |
| JayF | (the dnspython issue) | 19:01 |
| frickler | no, the bot will see new eventlet only on saturday | 19:01 |
| frickler | I'll propose a manual bump instead of the dnspython revert https://review.opendev.org/c/openstack/requirements/+/909923 | 19:01 |
| dansmith | safest thing to do is lower dnspython to known-working, and then test eventlet and new dnspython together right? | 19:02 |
| dansmith | new eventlet could have any number of other impacts | 19:02 |
| dansmith | this is blocking merges right now | 19:02 |
| frickler | dansmith: that's true, will put both together on top of the revert | 19:02 |
| gmann | ++, agree let's get back to 2.5.0 working one first | 19:03 |
| dansmith | frickler: thanks, if we weren't in FF rush right now it might be different but I'd rather go for quickest resolution first :) | 19:03 |
| gmann | dansmith: just wondering, it is not caught by the cross project (glance) job in requirement gate right? those are unit/functional only | 19:04 |
| dansmith | gmann: actually glance jobs aren't failing IIUC, it's nova while talking to glance | 19:05 |
| dansmith | but they run our job and it's only the ceph-multistore job, | 19:06 |
| dansmith | maybe because ceph involves more network IO at image upload time or something | 19:06 |
| gmann | ohk, nova-ceph-multistore | 19:06 |
| gmann | i see | 19:06 |
| dansmith | abhi and cyril have been debugging in -glance all day so they have the deets | 19:06 |
| dansmith | gmann: redhat is on holiday tomorrow so we're all starting to disappear, but I assume someone will be around to get this pushed right? | 19:08 |
| frickler | reqs only runs a single tempest-full job, not sure it would make sense to add more complicated, likely more unstable tests | 19:08 |
| dansmith | abhishekk_ has been debugging all day on the eve of his holiday so he wants to cut loose of course :) | 19:08 |
| abhishekk_ | :D | 19:08 |
| abhishekk_ | If this gets cleared, I can use sometime of weekend to recheck glance patches | 19:09 |
| gmann | frickler: agree, but how about the experimental and those can be run on demand if we see multiple external deps bump | 19:09 |
| gmann | but agree that we cannot cover all the cases | 19:09 |
| frickler | hmm, the error in that bug doesn't actually look related to eventlet IMO | 19:09 |
| frickler | gmann: oh, some experimental jobs that we can run manually on bot patches, that's a good idea | 19:10 |
| gmann | dansmith: pinged prometheanfire in requirement channel, he is pretty fast on review, let's see | 19:10 |
| gmann | frickler: yeah | 19:10 |
| dansmith | gmann: ah thanks I looked for him in -qa and didn't see him so I figured he wasn't around | 19:11 |
| dansmith | oh frickler got it | 19:11 |
| gmann | ++ thanks frickler | 19:11 |
| dansmith | abhishekk_: go to sleep :) | 19:11 |
| JayF | My only concern with a rollback; I know 2.6.1 was a CVE fix | 19:11 |
| JayF | that impacted only 2.6.0, right? /me checks | 19:11 |
| dansmith | JayF: yeah it fixed the security hole all right :) | 19:11 |
| abhishekk_ | ack o/ | 19:12 |
| dansmith | airgapped is very secure | 19:12 |
| JayF | service iptables panic # in library form (don't run this on old(?) rhel, it blocks everything) | 19:12 |
| JayF | https://github.com/rthalley/dnspython/commit/f66e25b5f549acf66d1fb6ead13eb3cff7d09af3 is what we lose, it looks like it'd impact 2.5.x as well but per https://ubuntu.com/security/CVE-2023-29483 and https://github.com/rthalley/dnspython/issues/1051#issuecomment-1949383928 imply it's not that big of a deal | 19:14 |
| frickler | hmm, so we shouldn't have been using 2.6.0 anyway? | 19:14 |
| JayF | 2.6.0 had the CVE fix | 19:14 |
| dansmith | well, if the eventlet bump fixes it we can roll to it asap | 19:14 |
| JayF | the reason 2.6.1 released is to fix an issue in that CVE fix afaict | 19:14 |
| JayF | lol | 19:14 |
| JayF | yeah, that's mainly what I'm saying: roll back is OK as long as we don't trap ourselves on the vuln version for Caracal | 19:15 |
| JayF | ^ that opinion is very weakly held fwiw | 19:15 |
| dansmith | sure, I'm just saying: not working at all is not a whole lot better than "has a mild CVE" | 19:15 |
| JayF | ++ | 19:15 |
| JayF | but we do have working versions of dnspython and eventlet in pypi right now | 19:15 |
| dansmith | ..we think :) | 19:16 |
| JayF | so it should be rollback to preserve, roll forward PR pushes we test in it and roll forward | 19:16 |
| JayF | some things are trust, but verify. Eventlet+DNSPython is suspiciousness+verification LOL | 19:16 |
| prometheanfire | gmann: yep, reviewed, but frickler beat me to it | 19:16 |
| gmann | prometheanfire: yeah. thanks | 19:17 |
| frickler | proposed https://review.opendev.org/c/openstack/requirements/+/909925 + 26 now for latest eventlet and dnspython, so these can get tested independently. likely a depends-on patch in glance would be the best test? | 19:19 |
| dansmith | or nova | 19:20 |
| dansmith | actually, I have a DNM nova patch in the glance stack they were working on, let me use that | 19:21 |
| dansmith | frickler: https://review.opendev.org/c/openstack/nova/+/891207 | 19:22 |
| frickler | dansmith: ack, will watch that, thx | 19:26 |
| frickler | hrm, huge stack of reqs patches in gate, that can take some time. also a cinder gate failure, if someone wants to take a look https://zuul.opendev.org/t/openstack/build/0ac810bc000b412bb1149f2457b2e841 | 19:55 |
| frickler | aand a second one. wonder if that eventlet issue might only be triggered sporadically there or whether there's another timeout issue https://zuul.opendev.org/t/openstack/build/378ff209f6fd420aa11bf11767d290e1 | 20:01 |
| frickler | anyway I'm out now, will see what has happened further tomorrow | 20:02 |
| dansmith | frickler: the nova patch failed the same way.. is depends-on a requirements patch good enough to make sure it is honored? | 21:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!