| *** mhen_ is now known as mhen | 01:15 | |
| opendevreview | melanie witt proposed openstack/tempest master: WIP Add tests for vTPM live migration https://review.opendev.org/c/openstack/tempest/+/957475 | 03:30 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/tempest master: WIP Add tests for vTPM live migration https://review.opendev.org/c/openstack/tempest/+/957475 | 03:37 |
| opendevreview | Ghanshyam proposed openstack/devstack master: Configure glance user in cinder conf https://review.opendev.org/c/openstack/devstack/+/958718 | 04:07 |
| gthiemonge | Hi folks, octavia-grenade is broken on 2025.1, it appears that it's related to the setuptools version installed in 2024.2 and some new python modules installed during the update | 07:36 |
| gthiemonge | I added more details in https://bugs.launchpad.net/octavia/+bug/2121578 | 07:36 |
| gthiemonge | (designate-grenade-* is also affected) | 07:37 |
| gthiemonge | any ideas? shouldn't we update setuptools in grenade? | 07:37 |
| frickler | in general it would be good if upgrades can work with only updating openstack code, keeping dependencies untouched. not sure if that is possible with what setuptools maintainers are doing | 08:05 |
| *** ralonsoh_ is now known as ralonsoh | 08:43 | |
| oschwart | Hey folks, the same errors that broke grenade in Octavia break Designate grenade jobs too | 09:08 |
| oschwart | I am not sure, but maybe this patch could fix those (?) | 09:09 |
| oschwart | https://review.opendev.org/c/openstack/keystonemiddleware/+/954018 | 09:09 |
| gthiemonge | oschwart: pip is already pulling oslo.cache 3.10.2, so I don't think it will fix it | 09:19 |
| oschwart | It looks like grenade gets a Segmentation fault when it runs sudo usermod -a -G libvirt stack | 12:26 |
| oschwart | From a quick search I found instances of that error on tempest, devstack, neutron, designate, cinder, manila, and others | 12:26 |
| clarkb | gthiemonge: frickler: we shouldn't be pinning setuptools at all... | 14:25 |
| clarkb | I'm not sure I understand the "setuptools version installed in 2024.2" assertion. It should use the latest version of setuptools supported by the python version there | 14:25 |
| clarkb | this changes if you start using pyproject.toml everywhere to control the setuptools version but until openstack does that setuptools isn't something we can really manage the version of reliably so we've tried to stick with the latest | 14:26 |
| gthiemonge | clarkb: ack, I will check which setuptools version is installed in the initial deployment | 14:29 |
| clarkb | oschwart: any idea if the segfault is in sudo or usermod? I would probably start there to determine if sudo or usermod is the source of the bug | 14:32 |
| clarkb | do you have a link to the failure? the link about has passing tests | 14:34 |
| clarkb | *the link above | 14:35 |
| gthiemonge | clarkb: during the deployment, devstack does something like "pip install -U pip 'setuptools[core]' -c https://raw.githubusercontent.com/openstack/requirements/refs/heads/stable/2024.2/upper-constraints.txt" which pulls setuptools 75.3.2, this version appears to be broken with recent modules, and we don't update it before/during grenade | 14:36 |
| clarkb | wow I would personalyl consider that a bug in devstack. For this very reason | 14:37 |
| clarkb | as a general rule trying to control setuptools is a bad idea unless you can control the entire build environment | 14:37 |
| clarkb | (in this case I'm guessing that PBR is too new for that old setuptools but I'm not certain) | 14:38 |
| clarkb | so by only controlling a portion of the build environment there is a risk of these sorts of failures | 14:38 |
| oschwart | clarkb here is an example: https://zuul.opendev.org/t/openstack/build/ece7d0bfffd34c1e90cdec378dc7f943 | 15:50 |
| clarkb | oschwart: https://zuul.opendev.org/t/openstack/build/ece7d0bfffd34c1e90cdec378dc7f943/log/controller/logs/syslog.txt I think this indicates that usermod is segfaulting not sudo. | 15:52 |
| oschwart | hmm right, I see it now | 15:54 |
| clarkb | libc6 (the ld-linux-x86-64.so.2 source) doesn't seem to haev been updated since May. And passwd (where usermod comes from) is even older | 15:59 |
| clarkb | its also curious that nothing else seems to be tripping it (everything links to libc so you'd expect a lot more fallout) | 16:00 |
| clarkb | I can put a hold request in so that you can inspect the node directly. Maybe the package is corrupted or something? | 16:00 |
| clarkb | just let me know what change and job you'd like to do that for if you want ot do that | 16:01 |
| clarkb | but then you can run it under gdb and check the files match what you see upstream in the repos | 16:03 |
| oschwart | That same error happened also to tempest, devstack, neutron, cinder, manila | 16:08 |
| oschwart | I don't think it is related to a specific patch or to designate | 16:08 |
| clarkb | right its happening in usermod and ld-linux | 16:11 |
| tkajinam | is anyone aware of the problem that all of the "installed" flake8 plugins are now enabled regardless of flake8:local-plugins ? | 16:24 |
| opendevreview | melanie witt proposed openstack/tempest master: WIP Add tests for vTPM live migration https://review.opendev.org/c/openstack/tempest/+/957475 | 17:54 |
| sean-k-mooney | anyone know how depenent the rpm disto support is on rdo in devstack. i know we used to use eple in the past but started depenign on rdo more over time | 19:51 |
| clarkb | I think that packges like ovs come from rdo | 19:52 |
| sean-k-mooney | on c9s at leat we coudl have pulled form form the centos fastdatapath sig i think | 19:53 |
| sean-k-mooney | but ok | 19:53 |
| clarkb | in devstack all of the openstack components should be from source. Then we get things like the database and libvirt from the distro. On centos/rhel ovs did not exist in the base os and it comes from rdo | 19:53 |
| clarkb | so I think its basically whatever non openstack systems aren't available in the os (ovs, ceph, rabbit? etc) | 19:54 |
| sean-k-mooney | https://sigs.centos.org/nfv/#packages-and-centos-stream-versions-matrix | 19:54 |
| clarkb | you can probably get a sense for those items by listing packges in rdo and removing all the openstack stuff | 19:54 |
| sean-k-mooney | ya the reaon im asking is redhat has massivly pulled back it investment in rdo of late and no one has really stepped in to supprot it in the rdo comunity | 19:55 |
| sean-k-mooney | its not as if there is 0 maintaince | 19:56 |
| sean-k-mooney | but long term im not sure whats going to happen there | 19:56 |
| sean-k-mooney | so im wondering if we could pull more form upstream distos ranter then deping on rdo in the future | 19:57 |
| sean-k-mooney | to be fair its many of the same people inovled in the centos sigs and rdo | 19:57 |
| sean-k-mooney | clarkb: ceph we take form upstream anyway and installe ti with cephadm as contaienr i think | 19:58 |
| sean-k-mooney | at elast i dont think we use distro ceph as our defaults even if its an option in the devstack plugin | 19:59 |
| sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/cephadm#L32 | 20:00 |
| sean-k-mooney | i would be very surpised if rabbit or mysql is not alos packged elsewhere | 20:00 |
| sean-k-mooney | im not going to start looking into this now | 20:01 |
| sean-k-mooney | but i am going to wait and see if RDO does an offical 2025.2 release | 20:01 |
| sean-k-mooney | and if so what distos they release packages for | 20:01 |
| clarkb | ya I don't know concrete details just giving examples of what may be | 20:03 |
| clarkb | basically its the tertiary things that are not part of centos/rhel proper | 20:03 |
| sean-k-mooney | worst case we could continue down the ceph route and use containers if there were missing native packages at least for rabbit ectra | 20:04 |
| sean-k-mooney | although it woudl calll into qustion the utility of testing with centos at all | 20:05 |
| opendevreview | Merged openstack/devstack master: Configure nova 'service' role in tempest https://review.opendev.org/c/openstack/devstack/+/958612 | 20:05 |
| sean-k-mooney | i see takasi asked the same thing https://meetings.opendev.org/irclogs/%23rdo/%23rdo.2025-08-22.log.html and | 20:08 |
| sean-k-mooney | no responce | 20:09 |
| sean-k-mooney | the mailing list have been dead sicne 2021 | 20:09 |
| opendevreview | melanie witt proposed openstack/tempest master: WIP Add tests for vTPM live migration https://review.opendev.org/c/openstack/tempest/+/957475 | 20:53 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!