| *** erlon2 is now known as erlon | 02:23 | |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 03:03 |
|---|---|---|
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572 | 03:57 |
| *** jlejeune5 is now known as jlejeune | 03:57 | |
| gmaan | gibi: i saw the same notification test failing again, reported the bug with trackback and link https://bugs.launchpad.net/nova/+bug/2164606 | 04:00 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572 | 04:01 |
| dansmith | gouthamr: I've restarted m-shr and no reconcile has happened. I've captured the logs for the service during the restart, a dump of the manila database, and a capture of what osc shows me | 13:38 |
| dansmith | gouthamr: anything else you want me to capture (other service logs) before I force fix the share and move on? | 13:38 |
| gouthamr | dansmith: o/ ty, looking | 14:21 |
| opendevreview | Brian Haley proposed openstack/nova master: Change neutron API calls to use project_id https://review.opendev.org/c/openstack/nova/+/994321 | 14:57 |
| gouthamr | dansmith: in meetings and got distracted; did you have the log posted somewhere? i can take a look at it and i'm thinking we have an eventlet bug.. I don't need anyhing else though, so yes.. force-fix the share with "access delete --unrestrict" as admin | 15:22 |
| dansmith | gouthamr: no, I was waiting to see if you wanted me to capture anything else.. I'll post what I have in a sec then | 15:22 |
| gouthamr | dansmith: i reported this https://bugs.launchpad.net/manila/+bug/2164609 and started investigating.. my hunch is this has to do with the eventlet backend (default today) | 15:23 |
| gouthamr | dansmith: tyty | 15:23 |
| dansmith | gouthamr: the logs and things could go in the bug.. do you want the DB dump there as well (or not at all?) | 15:23 |
| dansmith | oh, is that bug for something more specific and thus not quite appropriate? | 15:26 |
| dansmith | FWIW, it does seem like restarting the service takes a long time, like it's waiting for something | 15:26 |
| gmaan | dansmith: btw, this is ready. CI is flaky over other unrelated failure so zuul -1 https://review.opendev.org/c/openstack/nova/+/1000211/14 | 15:28 |
| dansmith | ack | 15:28 |
| gmaan | thanks | 15:28 |
| dansmith | gouthamr: the log is too big to paste, as is the sql dump of course, so I emailed them to you and you can decide what to do witht hem | 15:44 |
| gouthamr | ah! thank you dansmith, got that email - this is super helpful | 15:48 |
| dansmith | gouthamr: using --unrestrict let me delete the stuck access rule. attempting to start the instance after that re-created the rule and the instance started (for the first time ever) | 15:49 |
| dansmith | so that's good, I was able to get it working again with just admin/api stuff and no surgery.. IIRC, this would all have required surgery to resolve before you added the idempotent give-myself-access-always-on-start behavior | 15:50 |
| dansmith | so a definite win for robustness, IMHO | 15:50 |
| gouthamr | wow 😅 | 15:50 |
| gouthamr | dansmith: ty for pushing for that robustness.. this looks a lot better than i initially intended it | 16:06 |
| gouthamr | will get back to you about the m-shr issues | 16:06 |
| * dansmith nods | 16:07 | |
| opendevreview | Brian Haley proposed openstack/nova master: Change neutron API calls to use project_id https://review.opendev.org/c/openstack/nova/+/994321 | 17:40 |
| gmaan | another bug (most prob tempest schema) happening in gate, reported https://bugs.launchpad.net/nova/+bug/2164677 | 18:06 |
| gmaan | currently I am seeing these many failure happening in CI, not sure we have etherpad to track them ( Uggla may knows ) but listing here just in case anyone else want it to check it before recheck | 18:08 |
| gmaan | https://bugs.launchpad.net/nova/+bug/2164606 | 18:08 |
| gmaan | https://bugs.launchpad.net/nova/+bug/2163552 | 18:08 |
| gmaan | https://bugs.launchpad.net/nova/+bug/2163448 | 18:08 |
| gmaan | https://bugs.launchpad.net/nova/+bug/2160254 | 18:08 |
| gmaan | https://bugs.launchpad.net/nova/+bug/2164677 | 18:08 |
| melwitt | gmaan: I thought a schema validation failure would be 100% fail rate, so I'm a bit confused for this. the example linked in https://bugs.launchpad.net/nova/+bug/2164677 patch is also not touching any API area so it's not that either | 18:20 |
| gmaan | melwitt: indeed. it is a clear error in tempest schema but not sure why it was/is not failing 100% there is 'instance_uuid' in required field instead of 'server_uuid' field https://github.com/openstack/tempest/blob/master/tempest/lib/api_schema/response/compute/v2_1/servers.py#L555 | 18:22 |
| melwitt | gmaan: hm, it looks like it's /servers/{server_id}/migrations API has the 'server_uuid' | 18:26 |
| melwitt | https://github.com/openstack/nova/blob/ec47550235877e82f529f6a8c99f32dd9ea9c1a8/nova/api/openstack/compute/schemas/server_migrations.py#L85 | 18:27 |
| gmaan | melwitt: yeah, i think i got it. this schema is for list server in-progress live migration and most of time it may be empty so passing but this time it catch the in-progress live migration and hit the schema bug | 18:27 |
| melwitt | ahhh | 18:27 |
| melwitt | makes sense | 18:28 |
| gmaan | here https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_live_migration.py#L477C9-L477C31 | 18:28 |
| gmaan | will fix it but happy that this test is catching some in-progress live migration :) | 18:28 |
| melwitt | yeah agreed | 18:29 |
| melwitt | so yeah looks like in that case the tempest schema is wrong ... it should be server_uuid for the migrations API | 18:29 |
| gmaan | yeah fixing https://review.opendev.org/c/openstack/tempest/+/1001722 | 18:32 |
| gmaan | btw added these gate failures in this etherpad https://etherpad.opendev.org/p/nova-ci-failures-minimal | 19:12 |
| melwitt | ++ | 19:18 |
| melwitt | (aside: looks like i need to clean up some really old stuff on there) | 19:19 |
| opendevreview | Brian Haley proposed openstack/nova master: Change neutron API calls to use project_id https://review.opendev.org/c/openstack/nova/+/994321 | 20:16 |
| gmaan | Gmaan@openrh28 | 20:32 |
| gmaan | oops :) | 20:33 |
| opendevreview | Merged openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | 21:11 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | 21:13 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!