| opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | 04:01 |
|---|---|---|
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 04:22 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 05:07 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | 05:07 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Generalize memory encryption config https://review.opendev.org/c/openstack/nova/+/998466 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add memory encryption config for TDX https://review.opendev.org/c/openstack/nova/+/998608 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Launch security Intel TDX https://review.opendev.org/c/openstack/nova/+/999249 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | 07:25 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Check only x86_64 arch for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000805 | 07:57 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Add test for MTU overwrite during live migration https://review.opendev.org/c/openstack/nova/+/995803 | 08:33 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Avoid setting MTU during live migration if different https://review.opendev.org/c/openstack/nova/+/995809 | 08:33 |
| opendevreview | Lajos Katona proposed openstack/nova master: Add regression test to repoduce bug 1854844 https://review.opendev.org/c/openstack/nova/+/991772 | 08:33 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Preserve queue_size during live migration https://review.opendev.org/c/openstack/nova/+/995841 | 08:33 |
| RomanHros[m] | Hello, please someone look at the https://launchpad.net/bugs/2016173. It looks like very beneficial patch but seems to be forgotten... | 08:38 |
| frickler | melwitt: sean-k-mooney[m]: ^^ any reason this is stuck except lack of focus? https://review.opendev.org/c/openstack/nova/+/880399 | 09:42 |
| sean-k-mooney[m] | nothing comes to mind | 09:45 |
| sean-k-mooney[m] | im still waking up but i can take a look properly when im at my desk | 09:48 |
| opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Fix outstanding flake8 issues https://review.opendev.org/c/openstack/nova/+/999714 | 10:39 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | 10:39 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Enable G (flake8-logging-format) rules https://review.opendev.org/c/openstack/nova/+/998557 | 10:39 |
| gibi | antia: tkajinam: after pulling down the latest version of the TDX series to the same machine I used before for testing I libvirt rejects the VM boot with | 13:24 |
| gibi | Aug 13 16:23:40 beast01.rhos.lab.eng.brq2.redhat.com nova-compute[1108220]: ERROR nova.compute.manager [instance: e034aa57-fdca-4ea2-831f-41bcf51eb1d3] libvirt.libvirtError: operation failed: Unable to find 'efi' firmware that is compatible with the current configuration | 13:24 |
| gibi | so I guess this is connected to the secure loader / secure firmware change | 13:25 |
| antia | gibi: okay interesting | 13:25 |
| gibi | it is on centos 10 stream | 13:25 |
| antia | what firmware files does it have? | 13:27 |
| gibi | https://paste.openstack.org/show/bT6yOXgAEr7XuYjlHCvI/ | 13:27 |
| gibi | this is the xml | 13:27 |
| antia | that forces no secure boot, does it have a no secure boot TDX firmware? | 13:28 |
| gibi | collecting... | 13:28 |
| gibi | https://paste.openstack.org/show/bSKoS2kOht69y2yUygFX/ | 13:29 |
| gibi | these are the descriptors | 13:29 |
| gibi | it seems that I have only one tdx descriptor and that points to a secureboot firmware | 13:30 |
| antia | yup that makes sense, I have the same on my system | 13:30 |
| gibi | OK so I guess I can boot if I request secureboot as well | 13:31 |
| antia | or set it to 'optional' | 13:31 |
| sean-k-mooney | optinal effectivly mean off | 13:31 |
| gibi | I have OVMF.inteltdx.secboot.fd and OVMF.inteltdx.fd so I can also create a new descriptor | 13:32 |
| sean-k-mooney | it means libvirt decies which migh tactully work now | 13:32 |
| sean-k-mooney | beofre it used ot more or less be off but worht a try | 13:32 |
| antia | I think in this case libvirt has no other choice than to give an enabled one, but I could be wrong | 13:32 |
| gibi | sean-k-mooney: given I have only firmware descriptor for tdx + secure boot libvirt will select that if optional | 13:32 |
| gibi | I will test it :) | 13:32 |
| antia | I created a no secure boot firmware from my available firmware | 13:33 |
| antia | https://paste.openstack.org/show/bdbleKLLworBpma70kXB/ | 13:33 |
| antia | I suppose it works differently on centos | 13:34 |
| gibi | yepp optional works and the VM uses the secure boot firmware | 13:36 |
| gibi | as that is the only one with tdx now | 13:36 |
| gibi | I will try antia's nosb descriptor now :) | 13:36 |
| antia | :D | 13:37 |
| gibi | ...adapted to my env | 13:39 |
| sean-k-mooney | ya its likely just a packaging issue | 13:40 |
| sean-k-mooney | thats not a blocker for the feature | 13:41 |
| gibi | yeah it seems to work | 13:42 |
| gibi | so I'm OK too | 13:42 |
| sean-k-mooney | that will get fixed in a future ovmf version eventually | 13:43 |
| sean-k-mooney | we could add a note or just recommen using optional | 13:44 |
| sean-k-mooney | i.e. in the feature release note | 13:44 |
| sean-k-mooney | or docs | 13:44 |
| gibi | yepp | 13:44 |
| antia | yes can add that to docs | 13:45 |
| sean-k-mooney | it would be nice to have a tmepest test for this eventually by the way , in whitebox or main tempest but since we cant run it upstream i would not proritze that before FF | 13:46 |
| gibi | yeah we discussed that in case of SNP and TDX as an optional step to provide whitebox coverage | 13:46 |
| sean-k-mooney | we just need to creat the flavor/image with the approate metadata to enabel the feature and spawn | 13:47 |
| gibi | I think both tkajinam for SNP and antia for TDX said they will try to add it | 13:47 |
| gibi | yepp that is and in whitebox we can look at the xml created | 13:47 |
| sean-k-mooney | true | 13:47 |
| sean-k-mooney | while we do not have the ablity to report upstream form our downstrema host we do have the ablity to runa devstack job there in thory | 13:48 |
| sean-k-mooney | so may be we can figoure out a way to do that testing later | 13:48 |
| sean-k-mooney | evertually we might eb able to make that public but we would have to sanitze the logs | 13:48 |
| sean-k-mooney | that the mian blocker currenlty if its a post merge ci | 13:49 |
| sean-k-mooney | i.e. a weekly perodic | 13:49 |
| gibi | I noted the os_secure_boot=optional thing in the doc patch in the TDX series | 13:50 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | 13:55 |
| gibi | OK I'm cool with the TDX series now. I run some local tests as well with the latest series. | 13:56 |
| gibi | /run/ran/ | 13:57 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | 14:29 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | 14:29 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | 14:29 |
| opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Check only x86_64 arch for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000805 | 14:29 |
| dansmith | unit tests are giving me "no such option enforce_scope in group [oslo_policy]" errors on several.. gmaan that was a recent change right? tried rebuilding venv, etc but still get it.. what am I missing? | 15:32 |
| gmaan | dansmith: yes, this option is removed. i think recreate venv should fix it but is it nova giving error? | 15:34 |
| gmaan | most of projects are fixed with this but not all. nova/placement etc are fixed alreadt | 15:34 |
| dansmith | yeah, venv rebuild seemed to not.. I figured it was a library version thing | 15:39 |
| dansmith | hrm | 15:40 |
| dansmith | gmaan: is it a version of oslo.policy I need to have or something? | 15:41 |
| gmaan | dansmith: oslo.policy 6.0.0 removed it but nova should not be using it anywhere so it should work any version of oslo.policy | 15:42 |
| dansmith | gmaan: this is an example (one of many) https://paste.opendev.org/show/bdPguy48EU0zoQuirlrk/ | 15:43 |
| dansmith | oslo.policy in the venv is indeed 6.0.0 | 15:44 |
| gmaan | dansmith: ohk and nova seems old one not master https://github.com/openstack/nova/blob/master/nova/tests/unit/policies/base.py#L66-L70 | 15:44 |
| gmaan | is it master and you updated the repo or stable branch? | 15:45 |
| gmaan | old nova with oslo.policy 6.0.0 will not work | 15:45 |
| dansmith | I'm on 3c23390cc8 | 15:45 |
| dansmith | wait a damn second | 15:45 |
| dansmith | lol, okay I did a local clone for a new working tree and I think it doesn't know what actual origin/master is so I'm based on something way old.. wtf | 15:46 |
| gmaan | dansmith: yeah that's too old Jan 26 | 15:47 |
| dansmith | yeah, stupid local shallow clone | 15:47 |
| dansmith | sorry for the noise | 15:47 |
| gmaan | np! I used to have same issue in my multiple local copies and forgot to rebase before change and finds it during git review which adds more work. I tried to stick to have 1 or 2 local working repo now :) | 15:50 |
| dansmith | yeah I almost always work out of one repo, but for something destructive I'm using another.. I did a local clone and so it's (long stale) master branch becomes origin/master in the new clone so I thought I was rebased on master but wasn't | 15:52 |
| dansmith | which is stupid rookie mistake, I just rarely do this so I was on autopilot I guess | 15:52 |
| gmaan | gibi: i updated the thread pool executor refactoring, i replied to this comment, I think there are more chance that different executors creation can happen in parallel compare to before this change where most of them were created during init https://review.opendev.org/c/openstack/nova/+/998571/comment/0f4554fd_8780afaf/ | 17:04 |
| gmaan | that is why I think a another lock is worth here to make them parallel but i do not have much performance data/impact so we can leave that with single lock (serial creation) and can change if any issue we see. let me know and i can update it accordingly | 17:05 |
| opendevreview | melanie witt proposed openstack/nova master: Reproducer for bug 2016173 https://review.opendev.org/c/openstack/nova/+/946222 | 18:18 |
| opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | 18:18 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 19:40 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | 19:55 |
| gmaan | melwitt: i saw you +w on the 2nd change but this base change is also ready, just in case you missed it, but if you are still reviewing that, please ignore this ping :) https://review.opendev.org/c/openstack/nova/+/996299/ | 20:03 |
| melwitt | gmaan: yes, sorry I am mid review still. thanks for checking :) | 20:04 |
| gmaan | melwitt: ohk, thanks | 20:04 |
| gmaan | dansmith: there are other two changes are also ready to complete the graceful shutdown series, https://review.opendev.org/c/openstack/nova/+/1000211 and doc/releasenotes on top of it. | 20:08 |
| gmaan | dansmith: Those are dependent on thread pool executor refactoring which makes the executors shudown easy to handle during graceful shutdown, you are welcome to review the same but just to clarify that it is part of eventlet-removal and not directly to graceful shutdown BP https://review.opendev.org/c/openstack/nova/+/998571 | 20:09 |
| gmaan | that is why i set the different topic for those even they are in same series | 20:10 |
| dansmith | gmaan: yeah I thought gibi had a -1 on the next patch.. is that resolved? | 20:10 |
| gmaan | dansmith: I updated it but gibi have not checked the updated one yet | 20:11 |
| gmaan | i resolved his feedback except one which i replied and waiting for him to check before we need any update there | 20:11 |
| dansmith | ack | 20:18 |
| melwitt | seems like the nova-graceful-shutdown job fails intermittently "Timed out waiting for compute service on npb4f32edc7a464 to be inactive (current: failed)" https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275 but I don't see any gate-failure bugs open about it | 21:37 |
| gmaan | melwitt: i have one but that is race between some test stopping service and other need it but this looks new to me, checking | 21:39 |
| gmaan | it seems two issue here 1. graceful shutdown is timeout andit seems lot of periodic tasks were running, this is solved in task tracking system where we stop running any new periodic tasks during shutdown | 22:03 |
| gmaan | 2, when timeout it seems privsep-helper still running | 22:04 |
| gmaan | Aug 13 19:11:12.515025 npb4f32edc7a464 nova-compute[92418]: INFO oslo_service.backend._threading.service [None req-e086152e-d33a-491b-8053-3d300bc9a11e None None] Graceful shutdown timeout exceeded, instantaneous exiting | 22:04 |
| gmaan | Aug 13 19:11:12.543995 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE | 22:04 |
| gmaan | Aug 13 19:11:12.544060 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'. | 22:04 |
| gmaan | Aug 13 19:11:12.544388 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 93426 (privsep-helper) remains running after unit stopped. | 22:04 |
| gmaan | Aug 13 19:11:12.544562 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 94748 (privsep-helper) remains running after unit stopped. | 22:04 |
| gmaan | Aug 13 19:11:12.545602 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Consumed 12.751s CPU time, 277.1M memory peak, 0B memory swap peak. | 22:04 |
| gmaan | this seems something new | 22:04 |
| gmaan | thats does not seems the issue, i can see "Unit process 82937 (privsep-helper) remains running after unit stopped." in successful case also | 22:06 |
| gmaan | not sure why service status is updated in DB, because after 180 sec of graceful shutdown, test wait for another 180 sec timeout for serviec to be inactive | 22:11 |
| gmaan | i see, I think its the status update interval time report_interval and service_down_time is racing with test timeout. report_interval = 120 service_down_time = 720 | 22:17 |
| gmaan | https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275/log/controller/logs/screen-n-cpu.txt#101-115 | 22:17 |
| gmaan | i think i need to update the test script to consider the report_interval and service_down_time for service status update timeout | 22:17 |
| gmaan | melwitt: reported the bug, I will propose fix sometime tomorrow otherwise next week https://bugs.launchpad.net/nova/+bug/2163448 | 22:28 |
| melwitt | thanks for doing that gmaan | 22:44 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!