| *** NobodyCam_ is now known as NobodyCam | 03:50 | |
| *** vishalmanchanda_ is now known as vishalmanchanda | 03:51 | |
| hgy | 1111 | 05:31 |
|---|---|---|
| bauzas | good morning Nova | 08:08 |
| gibi | bauzas: good morning | 08:16 |
| bauzas | gibi: hey ;) | 08:16 |
| brinzhang | bauzas, gibi: we cannot open the ethercalc, it report Service Unavailable(503), about the nova-cyborg interaction vGPU topic you can talked and pasting the decision on the etherpad, and we will look it ASAP | 08:36 |
| brinzhang | bauzas, gibi: thanks | 08:36 |
| gibi | bauzas: fyi, reported a new gate failure bug https://bugs.launchpad.net/nova/+bug/1947687 | 08:43 |
| gibi | brinzhang: we will have that session today from 13:00 UTC | 08:44 |
| bauzas | brinzhang: the cyborg-nova meeting will happen at 1pm UTC | 08:47 |
| bauzas | gibi: ack | 08:48 |
| gibi | bauzas: I'm not jumping to investigative mode for tha bug now. | 08:48 |
| brinzhang | ack,but I cannot open https://ethercalc.openstack.org/8tum5yl1bx43 this page | 08:51 |
| brinzhang | report 503 now, in cyborg PTG meeting we were usd tencent meeting instead of zoom | 08:52 |
| gibi | brinzhang: could you open https://ptg.opendev.org/ptg.html ? | 08:58 |
| gibi | brinzhang: if yes, then try to click on the nova cell in the 13:00 UTC column | 08:59 |
| gibi | brinzhang: that should bring you in the zoom call | 08:59 |
| gibi | (probably nobody there at the momement) | 08:59 |
| bauzas | brinzhang: what gibi said | 09:00 |
| bauzas | they moved from ethercalc AFAIK | 09:00 |
| brinzhang | ack | 09:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova-specs master: Re-propose QoS minimum guaranteed packet rate https://review.opendev.org/c/openstack/nova-specs/+/814531 | 09:03 |
| stephenfin | gibi: bauzas: Have you noticed many failures on the 'nova.tests.unit.db.main.test_migrations.TestModelsSyncMySQL.test_models_sync' and 'TestModelsLegacySyncMySQL.test_models_sync' tests? | 09:16 |
| gibi | stephenfin: I'm not | 09:16 |
| stephenfin | Ah, okay, I'm seeing a lot of failures on the bp/remove-sqlalchemy-migrate series and trying to figure out if I've already broken master slightly or if its one of the unmerged patches that's breaking things, e.g. https://review.opendev.org/c/openstack/nova/+/812144 | 09:18 |
| songwenping | gibi,bauzas: when i stop nova_compute service and evacuate the vm on the host, then i delete the vm when it is evacuating, the vm is local deleted by nova-api, but the libvirt process is not terminated, and the resources like GPU are still used by the vm process, how can we handle the vm process? | 09:24 |
| gibi | songwenping: you delete the VM while it is being evacuated? | 09:33 |
| songwenping | yes | 09:34 |
| gibi | and then you start the source compute back up | 09:35 |
| songwenping | yeah | 09:35 |
| gibi | and the the source compute does not clean up the instance? | 09:36 |
| songwenping | yeah | 09:36 |
| gibi | I think the source compute should detect that there was a local delete while it was down and clean up the instance | 09:36 |
| gibi | so it is probably a bug | 09:36 |
| songwenping | i am looking into the init_host process. | 09:37 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Increase timeout for migration tests https://review.opendev.org/c/openstack/nova/+/814547 | 10:14 |
| stephenfin | gibi: bauzas: I'm not sure how to test that, but I'm pretty sure we're going to need it to at least fix the migrations issues we've been seen in the ARM CI jobs ^ | 10:14 |
| stephenfin | I guess I can mash recheck | 10:15 |
| gibi | stephenfin: ack | 10:16 |
| gibi | is it only affects ARM? | 10:16 |
| stephenfin | I don't think so. I've seen failures on x86 also, but ARM is particularly badly affected | 10:34 |
| stephenfin | example: https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7 | 10:34 |
| stephenfin | vs (aarch64) https://zuul.opendev.org/t/openstack/build/f693ac69815f4b92b75f7f8dc5ec3803 | 10:34 |
| stephenfin | gibi: ^ | 10:34 |
| gibi | stephenfin: when it passes it is also slow just not too slow to hit the timeout? | 10:40 |
| gibi | I see this runs | 10:42 |
| gibi | nova.tests.unit.db.main.test_migrations.TestModelsSyncMySQL.test_models_sync 11.154 | 10:42 |
| gibi | where it is actully fast | 10:42 |
| gibi | https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b/log/job-output.txt | 10:43 |
| gibi | anyhow I'm not against extending the timeout | 10:44 |
| gibi | I just feel that we still not fully grok what can be the problems with this db tests | 10:44 |
| opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | 10:49 |
| stephenfin | gibi: Yeah, I see about 7-8 seconds runtime on my localhost but it _seems_ very much dependent on the environment. Perhaps it's because some VMs are using spinning rust and others are SSD based? | 11:09 |
| stephenfin | We can compare these two builds https://zuul.opendev.org/t/openstack/build/00197f336d1249e4a55255769f24faf7 (iWeb) and https://zuul.opendev.org/t/openstack/build/8c4f8a81770e4dd1a523e86690ce340b (OVH) | 11:09 |
| stephenfin | gibi: Yeah, crazy differences in the MySQL tests https://paste.opendev.org/show/810073/ | 11:14 |
| gibi | I remember I saw this craxy differences before alembic too, and I had not explanation for that | 11:15 |
| songwenping | gibi: the evacuation status is failed, and this filter[1] cannot get the vm. [1]https://review.opendev.org/plugins/gitiles/openstack/nova/+/refs/heads/master/nova/compute/manager.py#703 | 11:16 |
| songwenping | so the vm process is not destory when the compute service restore. | 11:17 |
| gibi | songwenping: the problem with failed evacuation itself is that it does not imply the the VM is deleted (and hence the evac failed). If the evac fail for other reasons then keeping the VM on the source host after recovery is logical | 11:18 |
| gibi | so I think you have to make some logic changes where the failed evacuations are also queried but only if the instance is already marked deleted | 11:19 |
| songwenping | to report a bug? | 11:22 |
| stephenfin | Looks like https://review.opendev.org/c/openstack/nova/+/814547/ is working at least, since the aarch64 jobs are passing. I left a comment for bauzas | 11:24 |
| sean-k-mooney | stephenfin: i do have an arm laptop, my pinebook pro i can proably try and run it or power it up and let you ssh into it an use it although i proably will have to reinstall the os before its really useful | 11:39 |
| stephenfin | sean-k-mooney: I don't think it's specific to the architecture. I think it's the host in general. Bad I/O performance or something similar | 11:40 |
| stephenfin | but that's just a theory | 11:41 |
| sean-k-mooney | ah ok | 11:41 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add notes about IoOpsWeigher, BuildFailureWeigher https://review.opendev.org/c/openstack/nova/+/814561 | 11:52 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add a new cells v2 document https://review.opendev.org/c/openstack/nova/+/814562 | 11:52 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add new architecture guide https://review.opendev.org/c/openstack/nova/+/814563 | 11:52 |
| stephenfin | bauzas: you're probably interested in https://review.opendev.org/c/openstack/nova/+/814561 also when you're back | 11:54 |
| gibi | songwenping: sure, a bug is needed too | 12:02 |
| gibi | stephenfin: yep it seems to work. It runs the test longer than before | 12:03 |
| gibi | nova.tests.unit.db.main.test_migrations.TestMigrationsWalkMySQL.test_walk_versions [372.936004s] | 12:04 |
| gibi | that is a lot of time :) | 12:04 |
| sean-k-mooney | gibi: stephenfin im proably goingt to add https://review.opendev.org/c/openstack/nova/+/805649 to the ptg adgenda at the end but i think directionally this is the correct approch to take and once the testing is built out and a release note added its close to a complete | 12:06 |
| sean-k-mooney | if ye have time could ye take a look and or set review priority on it if you think it should have it | 12:07 |
| gibi | sean-k-mooney: ack, does it have a bp attached? I'm not 100% sure it is a bug per se | 12:08 |
| sean-k-mooney | tl;dr its adressign numa blancing and pci affintiy my making prefer actully prefer the numa node | 12:08 |
| sean-k-mooney | gibi: right i think it a blueprint not a bug | 12:08 |
| sean-k-mooney | so no it currently does not | 12:09 |
| gibi | sean-k-mooney: also it worth discussing on the ptg. I think we need a consensus on what are the inputs we are sorting the NUMA cells on | 12:10 |
| sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1893121 was my orginal bug | 12:10 |
| sean-k-mooney | which when we lass discussed it we said was a blueprint | 12:11 |
| sean-k-mooney | gibi: yes that is why i was going to add it | 12:11 |
| sean-k-mooney | to make sure we agreed on the algoritim | 12:11 |
| gibi | cool | 12:16 |
| gibi | bauzas: have you seen melwitt's comment on L83 on the ptg etherpad? | 12:20 |
| gibi | bauzas: I guess we can re-visit oslo.limits later in the agenda | 12:20 |
| sean-k-mooney | we are going to have a followup unifed limits session | 12:24 |
| sean-k-mooney | so perhaps we can ask the oslo folk to join that | 12:24 |
| sean-k-mooney | e.g. in the cross project seesion we can let them know we will be having a topic on unifed limits later and ask them who form the oslo.limits side would be good to ping to attend | 12:25 |
| gibi | yeah that is a good idea | 12:29 |
| bauzas | sorry folks, I was off during this lunch for the gym | 12:29 |
| bauzas | gibi: looking at the etherpad | 12:30 |
| * gibi needs to drop for 30 mins, wil be back for the ptg | 12:30 | |
| bauzas | yeah, let's discuss with the oslo folks about this | 12:31 |
| sean-k-mooney | ok so we are starting with cyborg then oslo then rbac correct | 12:31 |
| sean-k-mooney | so all cross project sessions today | 12:31 |
| sean-k-mooney | we wont be starting nova only adgenday until tommoro after the neutron session | 12:32 |
| sean-k-mooney | i guess i better get a headset ready although i might just used my wired one i dont know if my wireless one is chaged since its in my backpack instead of plugged in on my desk | 12:33 |
| bauzas | sean-k-mooney: if we have time, we can start a few | 12:44 |
| bauzas | sean-k-mooney: given we will stop to discuss for the rbac at 4pm while we still have a slot until 5pm | 12:45 |
| bauzas | (UTC of course) | 12:45 |
| bauzas | starting the meetings in 3 mins | 12:56 |
| sean-k-mooney | ah ya that makes sense | 13:00 |
| songwenping_ | mdbooth: i have meet the problem last week, and have to reinstall my devstack | 13:08 |
| opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646 | 13:26 |
| opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646 | 13:55 |
| *** lbragstad_ is now known as lbragstad | 14:15 | |
| * bauzas goes rebooting as I had mic issues :) | 14:23 | |
| bauzas | hopefully will be back at time ! | 14:23 |
| gibi | bauzas: belmoreira asked about when the vncproxy topic will be discussed so that he can join | 14:26 |
| bauzas | gibi: we can start discussing the vncproxy topic at first after the break, wdyt ? | 14:28 |
| gibi | as it is a cell issue I would rather wait for dansmith / melwitt to join for it | 14:29 |
| gibi | at least I have not much to add there | 14:29 |
| dansmith | have glance things, but planning to be in nova for the rbac stuff | 14:29 |
| gibi | bauzas: and I think belmoreira is OK to have it tomorrow as well, so we don't need to rush, he just needs some timing so that he can join just for that discussion | 14:30 |
| bauzas | gibi: fair point about the cell impact | 14:31 |
| bauzas | gibi: let's target the vncproxy thing on tomorrow 15:30pm UTC | 14:32 |
| bauzas | I'll amend the etherpad so | 14:32 |
| gibi | belmoreira: is it OK for you ^^ ? | 14:32 |
| bauzas | I'll put unified limits and vncproxy as a bulk for tomorrow late | 14:33 |
| gibi | ack | 14:34 |
| opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/812646 | 14:35 |
| belmoreira | thanks gibi bauzas. I will be able to join between 13:00 and 15:00 utc. Just ping me here. I can also join in other timeslots but you will need to tell me in advance | 14:35 |
| bauzas | belmoreira: the only problem is that we also want to wait for melwitt to be there | 14:41 |
| bauzas | belmoreira: so that's why we'd prefer to discuss the vncproxy stuff at 15:00 UTC if you don't mind | 14:42 |
| bauzas | 15:10pm if we break before | 14:42 |
| belmoreira | thanks bauzas. I will join today at 15:10 utc | 14:43 |
| bauzas | belmoreira: thanks, appreciated a lot. | 14:46 |
| bauzas | belmoreira: wait, sorry, it's 3:40pm | 14:47 |
| bauzas | my bad | 14:47 |
| bauzas | we have a interop discussion before | 14:47 |
| belmoreira | ok, thanks | 14:47 |
| dansmith | rbac session coming up in nova yeah? | 14:53 |
| gibi | dansmith: yes | 14:53 |
| gibi | dansmith: at the top of the hour | 14:53 |
| hgy | This is my first session | 14:55 |
| dansmith | I don't see a link to join in the etherpad, can someone paste it here? gibi ? | 14:56 |
| gibi | dansmith: https://www.openstack.org/ptg/rooms/newton | 14:56 |
| gibi | that will redirect | 14:56 |
| gibi | to a zoom call | 14:56 |
| dansmith | thanks, can we put that in the etherpad or is it secret? | 14:57 |
| gibi | sure | 14:57 |
| gibi | I can | 14:57 |
| dansmith | thanks | 14:57 |
| gibi | but it is already on the https://ptg.opendev.org/ptg.html page | 14:57 |
| dansmith | okay, it's just easier to jump back and forth if the etherpads have them | 14:57 |
| gibi | sure added it L49 | 14:58 |
| dansmith | sweet thanks | 14:59 |
| hgy | Is it begin? | 15:00 |
| gibi | hgy: you are in the good place if you are for the rbac discussion | 15:00 |
| gibi | hgy: we will switch to that topic soon | 15:00 |
| hgy | I almost missed it. I thought it was 16:00 | 15:01 |
| hgy | Thank you | 15:01 |
| hgy | 1 | 15:25 |
| gibi | .11 | 15:26 |
| bauzas | I'm lost in translation too | 15:31 |
| bauzas | wondering what to start with the rbac thing :) | 15:32 |
| stephenfin | bauzas: can you mute? | 15:51 |
| dansmith | bauzas: mute | 15:51 |
| bauzas | done and sorry | 15:52 |
| stephenfin | ta | 15:52 |
| bauzas | I was about to mention we have 8 mins and we somehow need to wrap up | 15:52 |
| bauzas | 8 min left* | 15:52 |
| dansmith | bauzas: yeah I have to go to another session at the top of the hour now | 15:53 |
| bauzas | I'll use my stick then | 15:53 |
| artom | "super interesting discussion" | 15:54 |
| artom | gdi the contempt is just dripping | 15:55 |
| artom | :D | 15:55 |
| belmoreira | bauzas it's getting late for me... I will need to leave in 5 min. I see that melwitt is OK with the change in the vncproxy. Thank you all. | 16:18 |
| bauzas | belmoreira: cool | 16:18 |
| belmoreira | regarding the instance move between projects, I guess it will be difficult... there is a lot of complexity | 16:19 |
| belmoreira | see you tomorrow | 16:19 |
| ralonsoh | sean-k-mooney, hi! https://review.opendev.org/c/openstack/nova/+/807359 | 16:21 |
| ralonsoh | if you have time, trivial patch | 16:21 |
| sean-k-mooney | ya i can take a look it looks correct | 16:22 |
| ralonsoh | thanks! | 16:22 |
| artom | It's going to be hard to concentrate on the substance with that backdrop! | 16:25 |
| gibi | bauzas: I lost zoom | 16:36 |
| stephenfin | bauzas: did you kill Zoom again? | 16:36 |
| bauzas | NOT ME ! | 16:36 |
| sean-k-mooney | sure it wasnt :P | 16:36 |
| stephenfin | oh, it's gmann | 16:36 |
| artom | bauzas, there are better ways to censor me ;) | 16:36 |
| stephenfin | he was host | 16:36 |
| melwitt | it said "the host left" | 16:36 |
| bauzas | nah it was gmann :) | 16:36 |
| sean-k-mooney | so can we restart the same one | 16:37 |
| gibi | it is back | 16:37 |
| stephenfin | https://www.openstack.org/ptg/rooms/newton | 16:37 |
| stephenfin | for anyone rejoining (atom, melwitt, ...?) ^ | 16:37 |
| gmann | oh, did i end meeting? | 16:40 |
| gmann | bauzas: so sorry about that. i mean to leave meeting and join horizon. sorry for mesh up | 16:41 |
| bauzas | gmann: heh no worries at all :p | 16:41 |
| bauzas | gmann: you helped me shutting artom's mouth when he was having an argument with me, appreciated :p | 16:41 |
| stephenfin | bauzas: you're unmuted again | 16:41 |
| bauzas | (just kidding) | 16:41 |
| stephenfin | and smiling like a maniac :P | 16:42 |
| gmann | :) | 16:42 |
| stephenfin | stop teasing gmann :P | 16:42 |
| -opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again | 17:01 | |
| opendevreview | Jan Hartkopf proposed openstack/nova master: ensure samples folder exists for microversion https://review.opendev.org/c/openstack/nova/+/813672 | 17:31 |
| melwitt | sean-k-mooney: sorry if I missed it but do the review priority labels prevent someone from +1ing or +2ing their own patch? I'm thinking of gibi's point that +1 shouldn't be used as a ping | 17:33 |
| melwitt | (that's how the old manual etherpad process worked, you can only raise attention to someone else's patch) | 17:34 |
| gmann | melwitt: sean-k-mooney i am wondering how to do that in project-config acl. I have not seen exclusive list we can n any label. or you are saying just to document that as guidlines ? | 17:38 |
| gmann | *can add in any label | 17:38 |
| melwitt | gmann: I don't know if it's possible to exclude self in a label, that's what I was asking sean :) is (1) is it possible? and (2) would we want to do that? | 17:41 |
| gmann | ok | 17:42 |
| opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | 19:36 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!