| *** bauzas_ is now known as bauzas | 00:12 | |
| *** bauzas_ is now known as bauzas | 01:44 | |
| gboutry | Hello Nova, I've tried upgrading libvirt/qemu in a dev env, and I'm unable to live migrate anymore. | 06:44 |
|---|---|---|
| gboutry | I'm getting "virDomainMigrateToURI3() failed" "unsupported configuration: Target CPU feature count 44 does not match source 109". | 06:45 |
| gboutry | The 2 hosts are VMs created the same way, with one having 8 cores and the other one having 4 cores, but same CPU features. | 06:45 |
| gboutry | This error does not appear with libvirt 8.0.0 / QEMU 6.2.0 (ubuntu 22.04 versions), but do appear with libvirt 10.0.0 / QEMU 8.2.1 (ubuntu 24.04 versions). | 06:45 |
| gboutry | These are pretty big version jumps I know, do you have any insight on how to investigate this? | 06:45 |
| *** efried1 is now known as efried | 07:03 | |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | 07:39 |
| *** bauzas_ is now known as bauzas | 08:07 | |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | 08:25 |
| opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | 08:25 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | 08:43 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | 08:43 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | 11:03 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | 11:03 |
| *** bauzas_ is now known as bauzas | 12:08 | |
| *** haleyb|out is now known as haleyb | 13:34 | |
| frickler | bauzas: did you see the "3 RLock(s) were not greened" messages e.g. at the start of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_42a/922140/13/experimental/nova-mtty/42a251f/controller/logs/screen-n-super-cond.txt ? JayF mentioned something similar yesterday in the ironic channel, not sure if that may actually be relevant? | 13:59 |
| frickler | seems to be related to running on py3.12 at least | 14:00 |
| JayF | I asked my eventlet expert downstream about it. I haven't gotten any response yet. I suspect it maybe that the way they are detecting those locks are getting false positives. | 14:04 |
| frickler | ack, otherwise one may want to insert more debugging output into that warning | 14:25 |
| JayF | There isn't much more to give. Basically it's just inquiring from the garbage collector if it's tracking any rlock objects. It can't really reliably /do/ anything with those objects if detected. | 14:48 |
| JayF | I printed the information on the objects, and nonsurprisingly, it wasn't useful. I suspect it might be possible to get deeper with a debugger + a lower level knowledge of python than I have at hand | 14:49 |
| *** ykarel_ is now known as ykarel | 14:59 | |
| *** ykarel is now known as ykarel|away | 15:00 | |
| *** Guest10177 is now known as dasm | 15:15 | |
| opendevreview | Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421 | 16:28 |
| melwitt | frickler, JayF: I looked up that log message in eventlet and there's a code comment about how in python < 3.10 the warning won't be logged, so that at least explains why we start seeing it in newer python https://github.com/eventlet/eventlet/blob/ec6c0fffccc129bea468a5badb03034d83f8cc5e/eventlet/patcher.py#L406 | 16:51 |
| JayF | melwitt: well, I think there's a behavior change in 3.12 | 16:52 |
| JayF | melwitt: because chasing down + eliminating those warnings was part of getting eventlet happier when the changes started | 16:52 |
| melwitt | yeah, the code comment explains the behavior change | 16:53 |
| JayF | it explains behavior change from 3.9 -> 3.10, not >=3.10 changing | 16:53 |
| JayF | I'm going to test on 3.10 with my ironic reproducer and see | 16:54 |
| melwitt | "On older Pythons (< 3.10), gc.get_objects() won't return any RLock instances, so this warning won't get logged on older Pythons." that isn't explaining what happens different in >= 3.10? | 16:54 |
| JayF | I'm saying that at no point have I said *3.10* is the point where behavior changed. I'm seeing it in 3.12 and absolutely chased down + elminated many of these dangling locks in Ironic on =>3.10 at some point | 16:55 |
| JayF | so either: 1) some non-python-version behavior changed to give us ungreened locks (really just locks that were made before we monkey patched) or 2) python version is the culprit | 16:55 |
| JayF | I don't know which yet, or even if it matters in reality, but it's not the kinda thing I wanna ignore | 16:56 |
| melwitt | ok, well, my response was to "seems to be related to running on py3.12" and I found the code comment so I thought that was the relation | 16:56 |
| JayF | Confirmed: in Ironic python 3.10 does not give the warning, in python 3.12 it gives the warning with 2 locks | 16:57 |
| JayF | I'll test with 3.11 to ensure it's 3.11 -> 3.12 behavior change as hypothesized | 16:57 |
| JayF | melwitt: frickler: FWIW I've confirmed with some certainty the behavior change is python-version-related and not anything else. With Ironic as the example; python 3.11: no ungreened locks, python 3.12: 2 ungreened locks. Only library difference is pyayncore and setuptools installed on py3.12 and installing those under py3.11 doesn't cause the issue. | 17:14 |
| JayF | if nova is seeing something similar, I may file an upstream eventlet issue | 17:15 |
| frickler | JayF: well the log I linked to above was from bauzas testing nova on Noble, so running with py3.12 too. I'd need to double check that it doesn't happen on py3.11 | 17:31 |
| frickler | hmm, opensearch does find some hits in toy-py311 jobs, too, like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce6/922421/1/check/openstack-tox-py311/ce64a27/job-output.txt | 17:34 |
| JayF | they almost always appear in unit tests | 17:34 |
| sean-k-mooney | frickler: i was able to stack on noble fine yestereday | 17:34 |
| JayF | they are only relevant for real runs | 17:34 |
| sean-k-mooney | i have that runnign in a vm currently | 17:34 |
| sean-k-mooney | but i didn try ot use the deploy openstack | 17:34 |
| sean-k-mooney | frickler: i can boot a vm and do a console log show | 17:38 |
| frickler | JayF: ah, o.k., the number found in those unit tests seems to go from 1 to 4 though from 3.11 to 3.12. also not sure neutron-functional job falls https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cec/917430/8/check/neutron-fullstack-with-uwsgi/cec5565/controller/logs/services.txt | 17:39 |
| frickler | other than that the only hit is the new noble job | 17:39 |
| sean-k-mooney | im sure there are proably bugs but at first glance things appear to work fin on noble and python 3.12 | 17:41 |
| frickler | sean-k-mooney: yes, what triggered me to look at the logs were the tempest failures in the mtty job. but they seem to be related to the actual mtty stuff afaict | 17:41 |
| JayF | My current working hypothesis is it's a reporting issue, not a functional one. | 17:41 |
| sean-k-mooney | frickler: ah ok well i htink there is a bug in sylvains nova patch | 17:41 |
| sean-k-mooney | or somehting related | 17:41 |
| frickler | JayF: yes, at least I have no data that would say otherwise, but would be great if we could be sure of that | 17:42 |
| sean-k-mooney | the tempet test are failign because nova is not reporting an resouce provider inventireos for the mtty devices | 17:42 |
| sean-k-mooney | frickler: the only thing i had to do was pip install -U setuptools in the requiremend and global venv | 17:42 |
| sean-k-mooney | i started on a patch to do that in devstack yesterday but didnt get back to it | 17:43 |
| sean-k-mooney | has anyoen proposed a noble job to the devstack repo yet? | 17:43 |
| sean-k-mooney | if not i can maybe do that this week | 17:43 |
| frickler | ah, yes, I still wanted to add some unit test for edit-constraints to the reqs fix | 17:45 |
| *** jph5 is now known as jph | 17:46 | |
| opendevreview | sean mooney proposed openstack/nova master: [WIP] split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425 | 17:49 |
| *** bauzas_ is now known as bauzas | 18:02 | |
| sean-k-mooney | frickler: JayF https://review.opendev.org/c/openstack/devstack/+/922427 should work? | 18:08 |
| JayF | nice, I can hook that up to ironic once it shows it's working elsewhere | 18:09 |
| JayF | would be interesting to see what's busted | 18:09 |
| sean-k-mooney | maybe | 18:09 |
| sean-k-mooney | i just got back to the eventlet stuff after 4 weeks or so | 18:09 |
| sean-k-mooney | so right now im trying to load back up the context on what i was doing | 18:10 |
| sean-k-mooney | JayF: ill generally be moving most of my dev envs over to noble or c10s over the next few weeks, both will be using python 3.12 | 18:12 |
| JayF | Honestly, it's just a fluke I caught this. 3.12 is my default python, I was testing a local dev thing for Ironic, and just saw it | 18:13 |
| sean-k-mooney | i feel like that how we catch most things | 18:13 |
| *** bauzas_ is now known as bauzas | 18:14 | |
| sean-k-mooney | one of use runs a ill advicsed setup (i.e. the latest and gratest of whatever) and fine odd bugs | 18:14 |
| sean-k-mooney | in anycase that should give use a full tempest-full run on noble without any other chagnes if it stacks properly | 18:15 |
| frickler | sean-k-mooney: while just testing you may also want to drop all the other jobs in the check pipeline | 18:15 |
| JayF | I was about to make some snarky comment about being offended you find my gentoo laptop ill advised ... then I realized you're *probably* actually right :D | 18:15 |
| sean-k-mooney | whihc shoudl be antoher datapoint to if this is an actul problem or jsut a logging issue | 18:15 |
| sean-k-mooney | frickler: yep i remmebered that after i pushed | 18:15 |
| sean-k-mooney | frickler: ill do it if i respin | 18:15 |
| frickler | ack | 18:16 |
| sean-k-mooney | pbr still depends on distutils by the way | 18:19 |
| sean-k-mooney | https://opendev.org/openstack/pbr/src/branch/master/pbr/core.py#L51 | 18:19 |
| sean-k-mooney | so until we drop that or we start adding seutuptools > whatever | 18:20 |
| sean-k-mooney | to the relevent project that depend on distutils | 18:20 |
| sean-k-mooney | we will need to keep the workarouncx in devstack | 18:20 |
| opendevreview | Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421 | 18:33 |
| opendevreview | Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421 | 18:35 |
| clarkb | sean-k-mooney: PBR also depends on setuptools (which caries distutils these days) so you're depending on both really | 18:38 |
| sean-k-mooney | clarkb: i didnt see the requrieemtns file for pbr in the repo | 18:39 |
| sean-k-mooney | did i miss it or is that depenciy specified elsewher | 18:39 |
| clarkb | sean-k-mooney: its implicit as pbr is a setuptools setup_requires hook or whatever its called | 18:39 |
| sean-k-mooney | there are no deps in https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/setup.cfg or https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/setup.py | 18:39 |
| clarkb | you can't list it as a requirment because of chicke and egg issues. The pyproject.toml documentation does explicitly list setuptools though | 18:40 |
| sean-k-mooney | clarkb: ok but without an explict depency we can specify a min version | 18:40 |
| sean-k-mooney | right here https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/pyproject.toml.future#L7 | 18:40 |
| clarkb | pbr is expected to run with a large range of setuptools versions | 18:40 |
| clarkb | there probably is some min version but its going to be old | 18:40 |
| sean-k-mooney | we basically need to say if python 3.12 reuqire setuptools >= the one with distutils | 18:41 |
| sean-k-mooney | which we would normaly do in a requriemtes.txt | 18:41 |
| sean-k-mooney | with the version spericifc marker | 18:41 |
| clarkb | right but you can't because of the chicken and egg problem. The only solution to that is to use pyproject.toml instead | 18:42 |
| clarkb | but no projects are doing that yet as far as I know | 18:42 |
| sean-k-mooney | i.e. setuptools>=70;python_version='3.12' | 18:42 |
| sean-k-mooney | clarkb: well i started adding pyproject.toml support | 18:43 |
| sean-k-mooney | but it need the wsgi_srcipts to be replaced | 18:43 |
| sean-k-mooney | so that work kind of stalled | 18:43 |
| clarkb | basically the dependency may exist but you can't encode that in a requirements file in a way that works/makes sense. You can encode it in project.toml build requires though | 18:43 |
| sean-k-mooney | well for now we can just have devstack install it | 18:44 |
| sean-k-mooney | for porject other then pbr if they have a dep on distutils they can sepciry the requireemnt in the normal way | 18:44 |
| clarkb | yes, the workaround there is appropriate. I just wanted to call out that it isn't just for distutils. You also need setuptools proper which isn't included by default in modern python either | 18:45 |
| sean-k-mooney | ack ya thats also true | 18:45 |
| sean-k-mooney | looking at codesearch/hound there are more project then i would like importing distutils | 18:47 |
| sean-k-mooney | .... inlcuing oslo.versionedobjects | 18:48 |
| sean-k-mooney | we proably should look to remove that dep and replace it with somethign else | 18:48 |
| sean-k-mooney | mostly we seam to be suing it for version parsing | 18:49 |
| sean-k-mooney | which we can replace with microversion_parse | 18:49 |
| sean-k-mooney | frickler: the noble run is currently on tempet https://zuul.openstack.org/stream/0b61b3b596554e4093acd6e17f61b26c?logfile=console.log | 18:51 |
| sean-k-mooney | so thats looking promising | 18:51 |
| sean-k-mooney | im going to leave it there for today | 18:51 |
| opendevreview | sean mooney proposed openstack/nova master: split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425 | 22:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!