14:00:52 <amoralej> #startmeeting RDO meeting - 2023-11-22
14:00:52 <opendevmeet> Meeting started Wed Nov 22 14:00:52 2023 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:52 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:52 <opendevmeet> The meeting name has been set to 'rdo_meeting___2023_11_22'
14:01:27 <amoralej> #topic roll call
14:01:57 <jcapitao[m]> o/
14:03:48 <amoralej> #chair jcapitao[m]
14:03:48 <opendevmeet> Current chairs: amoralej jcapitao[m]
14:06:08 <amoralej> let's start with the first topic
14:06:35 <amoralej> #topic Status of train, ussuri,  victoria. Can we remove the content from trunk.r.o ?
14:06:55 <amoralej> i was wondering if we may remove the content
14:07:46 <amoralej> rlandy_, ^ are upgrades jobs from train still running? do you still need train repos available?
14:08:43 <amoralej> we also need https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/29878 merged, right
14:09:41 <jcapitao[m]> I just added another infra guy for review
14:10:11 <rdogerrit> Joan Francesc Gilabert proposed rdo-jobs master: [dataplane adoption]Collect container logs  https://review.rdoproject.org/r/c/rdo-jobs/+/50865
14:10:12 <spotz[m]> \o/ ran out to the yard to get eggs:)
14:10:35 <amoralej> #chair spotz[m]
14:10:35 <opendevmeet> Current chairs: amoralej jcapitao[m] spotz[m]
14:10:40 <jcapitao[m]> :)
14:13:13 <spotz[m]> rlandy might be on PTO, US holiday tomorrow and Friday
14:13:25 <amoralej> ah, ok, let's sync with her next week then
14:13:42 <amoralej> #action amoralej to ask rlandy about the status of upgrade jobs from train
14:14:16 <jcapitao[m]> from RDO POW, I think we can remove the content, as nobody replied to the mail we sent about it
14:14:40 <amoralej> yes, i think once we get the ack from tripleo ci team, we can get rid of it
14:14:58 <amoralej> let's move to next topic
14:15:00 <amoralej> #topic RDO deployments on CS8 are broken with SSL enabled
14:15:37 <amoralej> #info yoga and wallaby jobs on CS8 during the last weeks
14:16:08 <amoralej> i've been doing some debugging and apparently the problem is related to a recent python3-libs update in CS8 which contains a CVE fis
14:16:33 <amoralej> #info root cause seems https://gitlab.com/redhat/centos-stream/rpms/python3/-/commit/b1da2b26d2c7066b271a3bdd9d508e97bc38da81
14:17:42 <amoralej> i'm working on it today but i'm not sure i will find a fix or i will need to report it to centos
14:18:31 <amoralej> it's related to eventlet + ssl, so it only fails if endpoint is configured with ssl enabled in the service itself
14:19:02 <spotz[m]> THere's an evenlet thread on the openstack-discuss list if you've missed it
14:19:50 <amoralej> yes, i've read it, i think it's unrelated to the issue reported in the thread, which is about support for 3.12
14:19:52 <jcapitao[m]> in the ML it's related to Eventlet supporting on py3.12
14:20:45 <jcapitao[m]> so it's python3 being patched
14:20:47 <spotz[m]> And out of date and unsupported, I think we need to patch it and keep using it until an alternative is found and in place
14:21:12 <amoralej> yeah, it will not be easy, eventlet is used everywhere ...
14:21:40 <amoralej> actually, i was thinking that it does not affect cs9 but now i have doubts if we enable ssl in p-o-i jobs in cs9
14:21:41 <amoralej> mmm
14:22:49 <amoralej> yes, we do
14:23:41 <amoralej> so, it's only affecting cs8 even with the same version of eventlet , and python 3.9 has the same patch which has been backporte to 3.6
14:24:10 <jcapitao[m]> :O
14:24:12 <amoralej> we'll see how can we fix it, i'm a bit blocked now
14:25:00 <spotz[m]> Reach out on CentOS-Dev or Openstack-discuss if you need to
14:26:06 <jcapitao[m]> I see current python3 in CS8 is python36-3.6.8-38.module_el8.5.0+895+a459eca8.x86_64
14:26:41 <amoralej> the problem comes from python3-libs-3.6.8-56.el8.x86_64
14:26:58 <amoralej> i think the other is the modularized one
14:27:12 <jcapitao[m]> ah right
14:28:04 <amoralej> i'm traying to create a minimal reproducer
14:28:12 <amoralej> but couldn't get it done yet
14:30:49 <amoralej> i think that was it from my side
14:30:55 <spotz[m]> I wonder if we could EPEL a newer python if that helps
14:30:55 <jcapitao[m]> maybe run the tests living in Lib/test/test_ssl.py
14:31:04 <jcapitao[m]> the ones from https://gitlab.com/redhat/centos-stream/rpms/python3/-/blob/c8s/00404-cve-2023-40217.patch?ref_type=heads
14:32:00 <jcapitao[m]> no forget
14:32:05 <amoralej> i tried to run unit tests with no luck so far
14:32:27 <amoralej> i got different issues
14:32:34 <amoralej> althought, i got an idea now
14:33:21 <spotz[m]> Sweet
14:34:29 <jcapitao[m]> I think we won't be the only ones hitting that
14:34:41 <amoralej> one never knows ... :)
14:34:43 <spotz[m]> Probably not
14:35:13 <amoralej> i think most people using https is terminating ssh with haproxy or httpd or similar
14:35:22 <amoralej> not using eventlet
14:35:24 <amoralej> i'd say ...
14:35:46 <amoralej> yeah, i've just reproduced the issue with unit tests! thanks
14:36:02 <jcapitao[m]> nice !
14:36:10 <amoralej> i think that's a step ahe
14:36:12 <amoralej> ahed
14:36:14 <amoralej> ahead
14:38:07 <amoralej> i'll keep working after the mtg and i'll see if i can run the tests in cs9 / python 3.9, if so i will report it as a bug
14:38:16 <amoralej> in cs8
14:38:30 <spotz[m]> +1
14:39:10 <jcapitao[m]> ok
14:39:32 <amoralej> if there are no more agenda topics, i'll open the open floor
14:39:39 <amoralej> #topic open floor
14:39:47 <amoralej> volunteers to chair next week ?
14:40:08 <jcapitao[m]> but I still think some CI/CD pipelines are hitting that issue too as eventlet is heavily used in unittests
14:40:32 <spotz[m]> Open Floor - SWAG order process started!. I'm trying to find out if they have fitted shirts if not I've reached out to everyone who had prefered that style
14:40:33 <amoralej> but if they are not using ssl ...
14:41:20 <spotz[m]> I should be able to lead, I don't think I have the conflict anymore due to the time change
14:41:36 <spotz[m]> If something changes I'll reach out
14:41:49 <jcapitao[m]> if they are not using SSL then it's not a real unit test/integration test :)
14:42:04 <amoralej> it passes unit tests in cs9
14:42:13 <amoralej> right :)
14:42:36 <amoralej> and the other thing is the version of python, unit tests are running in ubuntu upstream
14:43:04 <jcapitao[m]> ah right
14:43:18 <jcapitao[m]> which might not be patched
14:43:50 <amoralej> yeah, or being a newer version of python which seems not to fail according to my tests with python 3.9.18 in cs9
14:43:54 <jcapitao[m]> GH actions are using ubuntu as python base AFAIK
14:44:14 <amoralej> the backport may be incomplete or something
14:45:32 <jcapitao[m]> yeah
14:46:13 <amoralej> #info SWAG order process started!. I'm trying to find out if they have fitted shirts if not I've reached out to everyone who had prefered that style
14:46:31 <amoralej> spotz[m], anything else wrt swag or events ?
14:46:54 <jcapitao[m]> spotz: I'm remembering now the bagpack.. is it still orderable ??
14:52:44 <amoralej> i asked for volunteers :)
14:53:11 <jcapitao[m]> ah right !
14:53:34 <jcapitao[m]> I can take it
14:53:49 <amoralej> #action jcapitao will chair next week
14:54:14 <amoralej> if there are no more topics for today, i will close the mtg
14:54:34 <jcapitao[m]> nothing else from me
14:55:13 <spotz[m]> I volunteered and mentioned SWAG.
14:55:31 <spotz[m]> Um We are a go for the shared booth at FOSDEM and Connect would be our next event
14:55:51 <amoralej> ah, sorry spotz[m] ! i missed your message about chairing
14:55:52 <spotz[m]> OpenInfra plans to have an OpenInfra Summit in  Asia, several OpenInfra Day events in Europe (Sweden, Paris, Germany for sure and possibly London), and are working on an OpenInfra Day in NA and possibly LATAM
14:55:54 <amoralej> #undo
14:55:54 <opendevmeet> Removing item from minutes: #action jcapitao will chair next week
14:56:04 <spotz[m]> And OpenInfra plans to have a night event at FOSDEM
14:56:04 <amoralej> #action spotz will chair next week
14:56:10 <spotz[m]> That's it for me:)
14:56:19 <amoralej> thanks !
14:56:47 <amoralej> i'm closing the mtg, thanks both for joining!
14:56:53 <amoralej> #endmeeting