| opendevreview | Takashi Kajinami proposed openstack/placement master: Remove the job with Ubuntu 22.04 https://review.opendev.org/c/openstack/placement/+/958820 | 01:40 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/placement master: Remove the job with Ubuntu 22.04 https://review.opendev.org/c/openstack/placement/+/958820 | 03:02 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Follow-up of AMD SEV-ES support https://review.opendev.org/c/openstack/nova/+/958822 | 03:54 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Follow-up of AMD SEV-ES support https://review.opendev.org/c/openstack/nova/+/958822 | 03:59 |
| opendevreview | Merged openstack/nova master: Ask for pre-prod testing for native threading https://review.opendev.org/c/openstack/nova/+/957424 | 04:35 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Follow-up of AMD SEV-ES support https://review.opendev.org/c/openstack/nova/+/958822 | 04:54 |
| tkajinam | I've proposed a follow-up of SEV-ES support, which addresses most of the comments https://review.opendev.org/c/openstack/nova/+/958822 . Could you please have a look when you have time ? | 04:55 |
| tkajinam | bauzas, sean-k-mooney ^^^ | 04:55 |
| opendevreview | Merged openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436 | 04:57 |
| opendevreview | Merged openstack/placement master: Remove Python 3.9 support https://review.opendev.org/c/openstack/placement/+/953367 | 05:13 |
| bauzas | tkajinam: sure, will look | 07:20 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/2025.1: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/958834 | 07:40 |
| opendevreview | Tobias Urdin proposed openstack/nova master: api-ref: Add new lines not allowed for server description https://review.opendev.org/c/openstack/nova/+/928758 | 08:01 |
| opendevreview | Tobias Urdin proposed openstack/nova master: libvirt: update description for live_migration_completion_timeout https://review.opendev.org/c/openstack/nova/+/874083 | 08:02 |
| opendevreview | Tobias Urdin proposed openstack/nova master: libvirt: set remaining to 0 when no disk to migrate https://review.opendev.org/c/openstack/nova/+/873846 | 08:03 |
| opendevreview | Tobias Urdin proposed openstack/nova master: Remove libvirt tunnelled migration https://review.opendev.org/c/openstack/nova/+/879021 | 08:05 |
| opendevreview | Tobias Urdin proposed openstack/nova master: Remove libvirt tunnelled migration https://review.opendev.org/c/openstack/nova/+/879021 | 08:06 |
| opendevreview | Tobias Urdin proposed openstack/nova master: Retry more volume API functions on server error https://review.opendev.org/c/openstack/nova/+/942981 | 08:08 |
| opendevreview | Tobias Urdin proposed openstack/nova master: Remove gabbi from test-requirements.txt https://review.opendev.org/c/openstack/nova/+/953578 | 08:10 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for volumes APIs https://review.opendev.org/c/openstack/nova/+/952348 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for snapshots APIs https://review.opendev.org/c/openstack/nova/+/952349 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for volume attachments APIs https://review.opendev.org/c/openstack/nova/+/952350 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for floating IP APIs https://review.opendev.org/c/openstack/nova/+/952972 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for security group APIs https://review.opendev.org/c/openstack/nova/+/952973 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for services APIs https://review.opendev.org/c/openstack/nova/+/953196 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server usage audit log APIs https://review.opendev.org/c/openstack/nova/+/953209 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for tenant network APIs https://review.opendev.org/c/openstack/nova/+/956088 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for simple tenant usage APIs https://review.opendev.org/c/openstack/nova/+/956096 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for limits API https://review.opendev.org/c/openstack/nova/+/956139 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota class sets API https://review.opendev.org/c/openstack/nova/+/956140 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for quota sets API https://review.opendev.org/c/openstack/nova/+/956141 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for migrations API https://review.opendev.org/c/openstack/nova/+/956142 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server migrations API https://review.opendev.org/c/openstack/nova/+/956143 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server tags API https://review.opendev.org/c/openstack/nova/+/956144 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server metadata APIs https://review.opendev.org/c/openstack/nova/+/956145 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify servers views (1/3) https://review.opendev.org/c/openstack/nova/+/956231 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify servers views (2/3) https://review.opendev.org/c/openstack/nova/+/956232 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify servers views (3/3) https://review.opendev.org/c/openstack/nova/+/956233 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (1/6) https://review.opendev.org/c/openstack/nova/+/956234 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (2/6) https://review.opendev.org/c/openstack/nova/+/956236 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (3/6) https://review.opendev.org/c/openstack/nova/+/956237 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (4/6) https://review.opendev.org/c/openstack/nova/+/956238 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (5/6) https://review.opendev.org/c/openstack/nova/+/956239 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for servers APIs (6/6) https://review.opendev.org/c/openstack/nova/+/956240 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server shares APIs https://review.opendev.org/c/openstack/nova/+/956266 | 09:17 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Invert validation check https://review.opendev.org/c/openstack/nova/+/956241 | 09:17 |
| opendevreview | Merged openstack/placement master: Reproduce bug 2104040: allocate in over-capacity https://review.opendev.org/c/openstack/placement/+/945464 | 10:04 |
| tkajinam | o/ anyone has a min to review https://review.opendev.org/c/openstack/placement/+/958820 to remove remaining Jammy job ? | 10:06 |
| gibi | tkajinam: +2 bauzas ^^ easy win | 10:08 |
| bauzas | yup | 10:08 |
| bauzas | tkajinam: you deserve some rest, you know | 10:09 |
| tkajinam | bauzas, gibi thanks ! | 10:09 |
| tkajinam | I eventually took PTO today so had a good rest during the day :-) | 10:10 |
| bauzas | well, you actually worked two days in a day yesterday so indeed, take PTO :) | 10:11 |
| * bauzas when I saw tkajinam's ping in my already unreasonable time : wow. | 10:11 | |
| opendevreview | Merged openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | 10:45 |
| gibi | dansmith: ^^ \o/ | 10:51 |
| stephenfin | gmaan: gibi: bauzas: As sean-k-mooney said I've been out sick with a bug the last two days. Sorry for pushing that series so late: (a) I didn't realise FF was yesterday (I thought I'd another week) and (b) I couldn't push earlier as there would have been horrific conflicts with the (much larger) OpenAPI series. I'm happy to wait until early in G to get that in and appreciate the reviews | 10:59 |
| stephenfin | I'm getting both it and the OpenAPI series tidied up so that we can hopefully get them in asap once we branch | 10:59 |
| gibi | stephenfin: hope you are feeling better now. Thanks for working on these improvements | 11:01 |
| stephenfin | nw | 11:03 |
| gibi | sean-k-mooney: how do you feel about https://review.opendev.org/c/openstack/nova/+/936093 ? I think we saw the issue now downstream as well (see slack). I reviewed the patch and I think this is a good small fix. | 11:34 |
| sean-k-mooney | gibi: ill need to load context as it looks liek i have not reviewd it befreo | 11:35 |
| sean-k-mooney | we do kwno that there is a bug in mtu chagne on live migration but we need to ensure that it never changes across the migrtion | 11:36 |
| gibi | tldr: mtu is not in our network cache and that can cause live migration issue later as libvirt don't allow MTU change during live migration but we wrongly used the nil from the cache | 11:36 |
| sean-k-mooney | right | 11:36 |
| sean-k-mooney | so ill readd the patch but how we palnned to fix that was to make sure we dont modify the xml by saving the orginal mtu if its present | 11:36 |
| sean-k-mooney | we also do not incldue the mtu intetionally in teh netork metadlata if the neutron network has dhcp enabled | 11:37 |
| gibi | OK. So this is more complicated than I anticipated | 11:37 |
| sean-k-mooney | so we need to be careful not to change that | 11:37 |
| sean-k-mooney | gibi: so the issue sometime come up with people going form ml2/ovs to ml2/ovn | 11:38 |
| sean-k-mooney | nova does nto actully supprot changing the mtu on a network | 11:38 |
| sean-k-mooney | there was an optional extention to allow that in neturon | 11:38 |
| sean-k-mooney | btu it was never supproted on the nova side | 11:39 |
| sean-k-mooney | so the mtu should never change on a live migration | 11:39 |
| sean-k-mooney | howver when you do the db hacks to migrate form ml2/ovs to ml2/ovn you cahnge from vxlan to geneve and that reduces the mtu | 11:39 |
| gibi | in this case the source mtu is undefined, that converted to 0 and then of course that does not match the dest mtu that is set | 11:40 |
| sean-k-mooney | well the is an other rinkel | 11:41 |
| sean-k-mooney | there are 2 formats for virtio feature flags | 11:41 |
| sean-k-mooney | 32bit and 64bit | 11:41 |
| sean-k-mooney | if you spcify the mtu in the xml it changes form 32bit to 64bit | 11:41 |
| sean-k-mooney | which is also nto supproted by qemu during a live migration | 11:41 |
| sean-k-mooney | so if the xml does nto have an mtu we cant add it durign a live migration | 11:42 |
| gibi | I see, so this is the case ^^ | 11:42 |
| gibi | nova tries to add it | 11:42 |
| sean-k-mooney | that why we orginally started down the line of grab the mtu form the port and store it then generate the new config and restore the mtu | 11:43 |
| sean-k-mooney | gibi: yep so we fixed this once before after i acindeltely broke vexhost | 11:43 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/553072 | 11:45 |
| sean-k-mooney | that was the change to set the mtu in the first palace (its also neede for jumbo frames) | 11:45 |
| sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1747496 | 11:46 |
| sean-k-mooney | that shoudl result in ports being attached with an mtu as well | 11:47 |
| sean-k-mooney | gibi: the issue they are reproting in https://bugs.launchpad.net/nova/+bug/2080531 | 11:47 |
| opendevreview | Merged openstack/placement master: Remove the job with Ubuntu 22.04 https://review.opendev.org/c/openstack/placement/+/958820 | 11:47 |
| sean-k-mooney | i think artom had satred to fxi a while ago | 11:47 |
| sean-k-mooney | let me see if i can find that | 11:47 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Move regex helpers to utils https://review.opendev.org/c/openstack/nova/+/958745 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Clean up flavors tests https://review.opendev.org/c/openstack/nova/+/958746 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify API version check for flavor description https://review.opendev.org/c/openstack/nova/+/958747 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add ability to filter flavors by name https://review.opendev.org/c/openstack/nova/+/958748 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove dead fields from flavors response https://review.opendev.org/c/openstack/nova/+/958749 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Restrict additional query string arguments https://review.opendev.org/c/openstack/nova/+/958750 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Fix typo https://review.opendev.org/c/openstack/nova/+/958855 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Add runtime check for query additionalProperties https://review.opendev.org/c/openstack/nova/+/958856 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Add missing test coverage https://review.opendev.org/c/openstack/nova/+/958857 | 11:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: WIP: api: Add runtime check for general additionalProperties https://review.opendev.org/c/openstack/nova/+/958858 | 11:48 |
| sean-k-mooney | gibi: https://review.opendev.org/c/openstack/nova/+/791553/1 | 11:48 |
| sean-k-mooney | so we have a regression test https://review.opendev.org/c/openstack/nova/+/791235/3 | 11:49 |
| sean-k-mooney | then then the poc to fix it | 11:49 |
| sean-k-mooney | and there as a seocnd attept to fix it in https://review.opendev.org/c/openstack/nova/+/852367/1 | 11:49 |
| sean-k-mooney | gibi: these all kind of stalled out partly because of review banwith | 11:50 |
| gibi | I'm not sure if simply checking the MTU does not change will help if we don't even have the MTU on the source host to begin with. So probably multiple fixes are needed. Anyhow I will let the downstream reported file a bug on us to get motivation to fix it | 11:51 |
| gibi | s/reported/reporter/ | 11:52 |
| sean-k-mooney | ack | 11:52 |
| sean-k-mooney | ya so there are a few issues | 11:52 |
| sean-k-mooney | i might actilly have an old pathc for the mtu not being set | 11:52 |
| sean-k-mooney | i think the best way forward is 1 letg create some functional repoduces for the diffent senairos 2 then fix the mtu setting on attach and then on migrate as followups | 11:54 |
| gibi | yepp I agree | 11:55 |
| sean-k-mooney | the patch you linked migth fix part of it i woudl jsut need to trace throught he code to see why | 11:55 |
| sean-k-mooney | its not obvious that it should | 11:56 |
| sean-k-mooney | by the way we also knwo that there is a bug related to this that applie to sriov port as well | 11:58 |
| sean-k-mooney | basiclly wehn testing seriov live mgirtion i belvie we have to use 2 sriov prot (one mac vtap and one direct) to make it work instaed of ovs and macvtapo | 11:59 |
| sean-k-mooney | i dont exactly recall the details but it has somethign to do with the mtu i think as well but we could check with james parker and they might have a link | 12:00 |
| sean-k-mooney | we had ot work aroudn it in our ci | 12:00 |
| gibi | uuu a workarond in CI sounds interesting | 12:02 |
| gibi | the new report upstream involves SRIOV ports | 12:03 |
| gibi | I guess | 12:03 |
| sean-k-mooney | oh so ya | 12:03 |
| sean-k-mooney | am if you have an ovs prot and a sriov prot in direct mode | 12:03 |
| sean-k-mooney | live migration does not work because of somethign ot do with the mtu it hkn | 12:04 |
| sean-k-mooney | i debuged it once but again dint have tiem to go fix it | 12:04 |
| sean-k-mooney | ok reading the bug again ya this is the same issue we see in ci | 12:06 |
| sean-k-mooney | but the netwrok data in the metadata api is not alwasy the same as in the network info cache | 12:07 |
| sean-k-mooney | ill try and look at this review again properly next week | 12:08 |
| sean-k-mooney | this is the first item i have seen that bug or patch | 12:09 |
| gibi | OK. thanks | 12:11 |
| sean-k-mooney | im partly nerd snips because i was going to start on sumething new but you pinged me before i start on it | 12:13 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Correct version for rebuild response https://review.opendev.org/c/openstack/nova/+/958862 | 12:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | 12:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | 12:14 |
| opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose remove-os-volumes_boot-api https://review.opendev.org/c/openstack/nova-specs/+/958865 | 12:14 |
| gibi | sean-k-mooney: sorry for that. This is not urgent so feel free to punt it for a while | 12:14 |
| opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose flavor-search-by-name spec https://review.opendev.org/c/openstack/nova-specs/+/958866 | 12:15 |
| sean-k-mooney | im not complaining but im going to spend 15 mins considerign the patch then move on. ill come back to it on monday | 12:16 |
| sean-k-mooney | i think the change they are propsoing is safe to proceed with | 12:17 |
| sean-k-mooney | i am not conviced it will fix all 2-3 of the related cases i am vagely aware of but i dont think it will make any of those harder to fix later | 12:18 |
| opendevreview | Merged openstack/nova-specs master: Create specs directory for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/956166 | 12:19 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Disable VMCoreInfo device for SEV-encrypted instances https://review.opendev.org/c/openstack/nova/+/958868 | 12:19 |
| sean-k-mooney | gibi: i feel like it need a bit more testing but im tempeted to pull this into one of my devstacks and try it and maybe jsut take a good enough is better then perfect approch in this case | 12:20 |
| sean-k-mooney | gibi: in other words we can proably proceed with this before rc1 but i want to poke at it a bit before then | 12:21 |
| opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose openapi spec for Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/958869 | 12:22 |
| gibi | sean-k-mooney: OK. Thanks | 12:33 |
| stephenfin | gmaan: For next cycle, but you'll probably like https://review.opendev.org/c/openstack/nova/+/958856/ and https://review.opendev.org/c/openstack/nova/+/958858/ | 12:35 |
| stephenfin | one less thing for you to worry about during reviews 😄 | 12:35 |
| sean-k-mooney | gibi: i basiclly want to see if i can replciate this with 2 ovs network with differnt mtu or if it actully need ovs + sriov to see if we can maybe test this in tempest and if it obviously fixes the issue for me locally | 12:37 |
| sean-k-mooney | gibi: i dont curertnly have sriov hardware avaibel to thise this simiply so i need to create 2 devstack vms using igb and set it up for live migration ectra | 12:39 |
| sean-k-mooney | so i wont get to thtat today | 12:39 |
| opendevreview | Takashi Kajinami proposed openstack/nova stable/2025.1: libvirt: Disable VMCoreInfo device for SEV-encrypted instances https://review.opendev.org/c/openstack/nova/+/958874 | 12:39 |
| gibi | sean-k-mooney: sync with Arnau from neutron they migh have envs as this was reported from running neutron's whitebox against antelope | 12:44 |
| sean-k-mooney | ack i mostly want to see if we can test this edge case in the 1st party ci going forward via tempest without the sriov depency described in the bug | 13:03 |
| sean-k-mooney | if it can happen anytime you have a vm with 2 networks with diffent mtus for exampel we can easilly add a senario test in neutron to do that | 13:04 |
| sean-k-mooney | like in theory if usign the neutron whitebox tempest plugin found it downstream that same test should have repodcuded it on master right | 13:05 |
| gibi | yepp that theory make sense | 13:08 |
| gibi | there was also recent changes in that repo about the exact same test case so it might pass on master but not on older whitebox versions | 13:09 |
| dansmith | gibi: woohoo, thanks | 13:42 |
| amorin | hey team, the call to attach a volume on nova is done through rpc. During this, the wsgi worker keep the connection open (it's a synchronous call). Do you know why it's a sync call and not an async one? | 13:43 |
| amorin | see here https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5127 | 13:43 |
| amorin | and there https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5150 | 13:43 |
| dansmith | amorin: probably so that we delete the BDM if the attach fails | 13:45 |
| amorin | hum, damn | 13:47 |
| amorin | what if we send the attach action to the compute, answer to the API request, and then delegate all the hard work to conductor / compute? | 13:49 |
| dansmith | I mean, anything is possible, but this used to be a situation where you got the device name back when you attached, which is not possible otherwise | 13:54 |
| dansmith | why are you asking? I'm guessing you've got attach requests stacked up or something? | 13:54 |
| amorin | exactly :) | 13:54 |
| amorin | exhausting my apache2 workers | 13:54 |
| amorin | and finally nova was answering 503 | 13:55 |
| amorin | due to the fact that one of my ceph cluster was not healthy | 13:55 |
| amorin | I am trying to figure out a way to avoid such behavior | 13:55 |
| dansmith | in that case the attach is going to fail regardless right? | 13:56 |
| dansmith | is the problem that compute gets totally hung on ceph, or that it waits too long for ceph? | 13:56 |
| amorin | yes, it fails after the RPC timeout | 13:57 |
| dansmith | because part of the eventlet stuff is hopefully going to make us less stupid about talking to (and specifically timing out on) ceph in the compute node | 13:57 |
| dansmith | so having a short recoverable timeout there would help your situation I think, yeah? | 13:57 |
| amorin | I think so | 13:58 |
| amorin | something under 10s | 13:59 |
| amorin | at least that would help | 13:59 |
| dansmith | gibi: are you seeing this? ^ | 14:00 |
| amorin | otherwize, I was thinking to have dedicated apache workers for attach volume requests, so at least 503 answers are given only to people trying to attach, but other calls succeed, do you know if some other operators are already doing this? | 14:01 |
| dansmith | amorin: speaking of that in general, are you positioned to provide some pre-prod feedback for us on 2025.2 with scheduler and api in non-eventlet mode? we really need some real-world testing of those as we work on the other services | 14:01 |
| gibi | dansmith: not hanging on ceph for ever, sure. :) | 14:02 |
| dansmith | amorin: no, I'm not sure, but that's definitely something I'd consider yeah | 14:02 |
| dansmith | gibi: :P | 14:02 |
| amorin | dansmith: about preprod, maybe I can talk to the team about this | 14:03 |
| dansmith | amorin: that would be super helpful (hopefully for both of us :) ) | 14:04 |
| gibi | amorin: if you hit some walls with non-eventlet services then feel free to ping me as well with the issue I was the one that commites most of the crimes there | 14:06 |
| dansmith | amorin: unless everything goes well, in which case it was a team effort :D | 14:07 |
| jkulik | attachments and synchronous calls reminds me of the bug report: https://bugs.launchpad.net/nova/+bug/1930406 | 14:08 |
| amorin | ack gibi | 14:08 |
| gibi | dansmith: btw, just verified that with the futurist fix https://review.opendev.org/c/openstack/futurist/+/958689 aggregate image caching works well in threding mode now. The futurist fix will not be in Flamingo due to req freeze, but at least we can land nova-conductor change early in G | 14:08 |
| dansmith | gibi: excellent | 14:08 |
| gibi | dansmith: yepp yepp, failure is on my, success is on the team :) | 14:08 |
| gibi | s/on my/on me/ | 14:09 |
| amorin | jkulik: sounds like my exact issue | 14:09 |
| dansmith | jkulik: yep | 14:09 |
| jkulik | maybe we can get that mentioned spec back on track? :D | 14:09 |
| jkulik | since eventlet will go away, we also need a solution for this problem. currently, we're still on the eventlet-based webserver for only this reason :/ | 14:11 |
| dansmith | we could just make this an async operation in a later microversion | 14:11 |
| amorin | +1 | 14:11 |
| dansmith | easiest would be to delegate to conductor and keep compute the same | 14:11 |
| dansmith | that's the least upgrade-impact too | 14:11 |
| jkulik | > simply an approximation by most virt drivers as their underlying hypervisors cannot guarantee how the device will eventually be presented. | 14:12 |
| jkulik | my understanding was, that we cannot guarantee the device name anyways | 14:12 |
| dansmith | but it might require blocking earlier microversions to prevent people from triggering it | 14:12 |
| jkulik | so the whole call is ... senseless | 14:12 |
| dansmith | jkulik: on libvirt it's fairly pointless, but it was not on other hypervisors | 14:12 |
| jkulik | I can say it's also pointless on vmware | 14:12 |
| dansmith | gibi: so maybe we should make this under the eventlet removal umbrella.. moving this to an async task on conductor | 14:15 |
| jkulik | https://github.com/sapcc/nova/commit/e4e8e6276ab0324704f31d6a3cc4e47f061d25ca this is how we currently try to get relieve on the wait time for the lock, FYI | 14:15 |
| dansmith | the spec (as skimmed) looks far more complex than it needs to be if we just kick this to conductor in a newer microversion | 14:15 |
| gibi | dansmith: maybe. I'm a bit affraid of increasing the scope. But I understand that it is related | 14:15 |
| dansmith | well, if jkulik is sticking on eventlet because of this that tells me that (a) people might be less likely to try threaded until they're forced and (b) a lot more people will hit this once they move | 14:16 |
| dansmith | and tbh, it's probably worth a security hardening classification since this is sort of DoSable | 14:16 |
| jkulik | I agree to ^ | 14:17 |
| jkulik | DoS waiting to happen | 14:17 |
| dansmith | since it's friday maybe I'll at least write this up in a spec rough draft to capture it while it is in my head | 14:19 |
| jkulik | please CC me on that. I'm quite interested in the topic :) | 14:19 |
| amorin | CC me as well :) | 14:20 |
| dansmith | are either of you interested in working on the implementation? | 14:20 |
| jkulik | yes, I can probably do that | 14:21 |
| amorin | not sure I would have a lot of time very soon, but we I probably help (or someone from our team) | 14:22 |
| dansmith | cool | 14:23 |
| amorin | but I try to motivate our management, as this is affecting us quite badly | 14:23 |
| gibi | I can be pulled in as reviewer for sure. If we decide that this should be fixed before the n-cpu gets threading support then I can be pulled in as author as well | 14:23 |
| gibi | dansmith: thank for writing up a spec skeleton. My friday brain is already pretty useless anyhow | 14:24 |
| dansmith | I don't think this is going to be very big and I don't think it needs to be serialized against n-cpu work | 14:25 |
| gibi | I just try to limit the number of things open in parallel waiting for me | 14:26 |
| dansmith | ack | 14:26 |
| opendevreview | Dan Smith proposed openstack/nova-specs master: WIP: Spec for async volume attach https://review.opendev.org/c/openstack/nova-specs/+/958900 | 15:10 |
| dansmith | jkulik: amorin ^ .. just rough brain dump of what we talked about | 15:10 |
| amorin | yay! | 15:11 |
| ratailor | sean-k-mooney, gibi could you please review this https://review.opendev.org/q/Ifc20894801f723627726e3c9bed7076144542660 and provide your suggestion on approach here https://review.opendev.org/c/openstack/nova/+/954460 | 15:11 |
| dansmith | I'm sure I saw the bug years ago when it was filed, but this is the sort of latent thing I don't want to lose track of as we push for eventlet deprecation, | 15:12 |
| dansmith | where we think "yeah everything works in devstack" but we have lots of smaller reasons like this that prevent people from actually being able to move | 15:13 |
| dansmith | this isn't even the class of things that are highest concern (i.e. general threading problems, deadlocks, etc) but they are just as much of a problem for getting people transitioned | 15:13 |
| ratailor | stephenfin, submitted fix for microversion v2.100 as well https://review.opendev.org/c/openstack/python-openstackclient/+/958829 and backported v2.96 to relevant stable/releases https://review.opendev.org/q/I1e398bb3379fa6443b0a44db76baaf6241a945e7 | 15:19 |
| gmaan | stephenfin: perfect, that is really helpful. thanks. ack about targeting for next cycle | 15:43 |
| gmaan | sean-k-mooney: glancer service role fix is ready for review now https://review.opendev.org/q/topic:%22glance-service-api%22 | 19:23 |
| *** gmaan is now known as gmaan_afk | 19:30 | |
| opendevreview | Merged openstack/nova master: api: Correct expected errors https://review.opendev.org/c/openstack/nova/+/951640 | 21:12 |
| *** gmaan_afk is now known as gmaan | 23:20 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!