| opendevreview | Ghanshyam proposed openstack/nova master: Bump openstack-placement version in functional tox env https://review.opendev.org/c/openstack/nova/+/871011 | 00:41 |
|---|---|---|
| *** dasm is now known as dasm|off | 01:10 | |
| gmann | gibi: bauzas: placement is released and bumping the version in tox.ini https://review.opendev.org/c/openstack/nova/+/871011 | 01:27 |
| gibi | gmann: thanks | 08:33 |
| bauzas | gmann: ack, looking | 08:37 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Clean up after ImportModulePoisonFixture https://review.opendev.org/c/openstack/nova/+/870993 | 09:19 |
| gibi | sean-k-mooney: good point about __init__ vs setUp() in the fixture, I fixed it up ^^ | 09:19 |
| gibi | sean-k-mooney: we have python test executor crashes in functional but we don't have syslog saved in functional so I cannot confirm that it is due to OOM. I see python crashes locally with functional run couple of weeks back and that wasnt OOM but segfault from the interpreter | 09:21 |
| zigo | Hi there! | 09:21 |
| zigo | We wrote this: | 09:21 |
| zigo | https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/ | 09:21 |
| zigo | which works well, but crashes here: | 09:21 |
| zigo | https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/-/blob/debian/zed/ceilometer_instance_poller/instance_poller.py#L133 | 09:21 |
| zigo | if we're using the Ceph backend. | 09:21 |
| zigo | Does anyone of you know how to set the libvirt Ceph secret so then libvirt/libguestfs knows how to use the Ceph backend? | 09:21 |
| bauzas | zigo: sorry, not a expert at all on how we integrate with ceph | 09:37 |
| bauzas | maybe look at how nova attaches rdb disks | 09:37 |
| bauzas | rbd* | 09:37 |
| zigo | Yeah, it's smoething like that! :) | 09:38 |
| zigo | Got to find out how to tell libvirt/libguestfs what username and secret to use. | 09:38 |
| zigo | Obviously, I've been search for a long time before asking here ... | 09:38 |
| bauzas | gibi: weirdo | 09:40 |
| bauzas | gibi: I find we only take 30secs for grabbing the image which sizes 85 | 09:40 |
| bauzas | 2023-01-18 16:24:58.358 127779 INFO tempest.api.compute.admin.test_aaa_volume [-] DNM bauzas: Time elapsed: 31.457598802999883 2023-01-18 16:24:58.379 127779 INFO tempest.api.compute.admin.test_aaa_volume [-] DNM bauzas: Size of custom img: 85 | 09:40 |
| bauzas | and in this run, it took 93 secs | 09:41 |
| bauzas | have we changed the size of the default image ? | 09:41 |
| bauzas | mmm, yeah | 09:42 |
| bauzas | 2023-01-18 16:22:09.379365 | controller | ++ lib/tempest:configure_tempest:434 : iniset /opt/stack/tempest/etc/tempest.conf compute image_ref 7bac73d8-9596-43ae-900f-546a3d53de36 2023-01-18 16:22:09.411839 | controller | ++ lib/tempest:configure_tempest:435 : iniset /opt/stack/tempest/etc/tempest.conf compute image_ref_alt 7bac73d8-9596-43ae-900f-546a3d53de36 | 09:42 |
| bauzas | oh, nevermind | 09:44 |
| bauzas | we return the image_id hence the very small size | 09:45 |
| opendevreview | Merged openstack/nova master: Bump openstack-placement version in functional tox env https://review.opendev.org/c/openstack/nova/+/871011 | 10:11 |
| gibi | I cannot reproduce the functional failure locally with the same way I did in the past https://bugs.launchpad.net/nova/+bug/1946339 so I'm stuck a bit there | 10:19 |
| bauzas | gibi: I'll try to run it | 10:26 |
| bauzas | gibi: have you tried to run the functional tests by using the subunit ? | 10:27 |
| bauzas | gibi: using $ stestr run --load-list FILENAME | 10:27 |
| bauzas | gibi: lemme try to do it | 10:28 |
| bauzas | but after this, I'll need to go off until end of the day as I need to help my daughters due to the french strike | 10:30 |
| gibi | bauzas: yepp | 10:31 |
| gibi | bauzas: I tried the list of test from the failed worker from the recent gate failure and also tried the list from my past reproducers in that bug report | 10:32 |
| gibi | neither triggered the problem after couple of hours of continuous run | 10:32 |
| gibi | now simply run the test with --until-failure and --random | 10:33 |
| gibi | but this is a long shot | 10:33 |
| bauzas | gibi: ok, I'll try to run it the same | 10:33 |
| bauzas | gibi: I guess you know about https://stestr.readthedocs.io/en/latest/MANUAL.html#automated-test-isolation-bisection ? | 10:34 |
| gibi | i think that only works after we can reproduce the fauilure | 10:34 |
| gibi | and it needs a stable reproduction not a stohastic one | 10:35 |
| bauzas | gibi: I'll try to look at that | 10:35 |
| bauzas | folks, as I also said downstream, I'll be off until 1630UTC | 10:35 |
| bauzas | gibi: so I loaded the subunit | 10:39 |
| bauzas | into stestr | 10:39 |
| bauzas | (functional-py38) [sbauza@sbauza nova]$ stestr load < /tmp/zuul-logs.bNmdDA/testrepository.subunit | 10:39 |
| bauzas | I should now be able to run the failing test | 10:40 |
| bauzas | anyway, I need to disappear :( | 10:41 |
| gokhanis | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem and how can I solve this issue ? | 10:55 |
| sean-k-mooney | gokhanis: what kind fo cinder backedn are you using | 11:06 |
| sean-k-mooney | is this nfs based? | 11:06 |
| gokhanis | sean-k-mooney, yes I am using cinder nfs driver. | 11:08 |
| sean-k-mooney | this does not sound like it actuly a nova issue but rather and issue with how you have deployed it. ie. like you are hitting a ulimit or similar in the env | 11:08 |
| sean-k-mooney | gokhanis: ok is it nfs3 or nfs4 | 11:08 |
| sean-k-mooney | nfs3 has locking issues and we dont really support it anymore | 11:08 |
| sean-k-mooney | ideally you should use nfs 4.2 or newer | 11:09 |
| sean-k-mooney | gokhanis: i think your hitiing this https://bugzilla.redhat.com/show_bug.cgi?id=1547095 | 11:10 |
| sean-k-mooney | gokhanis: apprently you can "fix" it by using the nolock mount option for nfsv3 | 11:11 |
| sean-k-mooney | gokhanis: also relevent https://access.redhat.com/solutions/2780381 | 11:12 |
| gokhanis | sean-k-mooney, I deployed my env with Openstack Ansible. I have create nfs share on zfs storage and I used it as cinder nfs backend. | 11:12 |
| sean-k-mooney | gokhanis: form a downstream perspetive redhat discontinued support for nfsv3 in our porudct because of these locking issues | 11:12 |
| sean-k-mooney | gokhanis: yep so on the zfs culster you need to either ensure that the zfs pool is exported as nfsv4, nfsv3 with the nfsv4 lock manager or you have to ensure the nfs share is mounted with the nolock option on the compute hosts | 11:14 |
| sean-k-mooney | gokhanis: you can use https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options to set teh nolock option i think | 11:15 |
| sean-k-mooney | but if the fileserver supprot v4 then vers=4.2,lookupcache=pos | 11:15 |
| sean-k-mooney | woudl be better then nolock | 11:15 |
| sean-k-mooney | gokhanis: some more details on nolock https://github.com/oVirt/vdsm/commit/8aad4ebad10bab15c46ff743c105b6c2cba68d4e | 11:17 |
| gokhanis | thanks sean-k-mooney I will check nfs side in my env and add options you have said. | 11:17 |
| opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | 12:39 |
| opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | 12:44 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/861738 | 12:45 |
| opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround https://review.opendev.org/c/openstack/nova/+/870794 | 12:57 |
| kashyap | gibi: --^ Fixed the test, please have a re-look if you get a chance today | 12:57 |
| *** dasm|off is now known as dasm | 13:08 | |
| opendevreview | Merged openstack/nova master: Clean up after ImportModulePoisonFixture https://review.opendev.org/c/openstack/nova/+/870993 | 13:39 |
| gibi | bauzas, dansmith: the dstat log in https://review.opendev.org/c/openstack/nova/+/870950 shows that without that one test case that loads the image data there is no other memory hogger test case. At the highest memory load there is still 800MB free RAM in the VM. | 13:43 |
| gibi | kashyap: +2, thanks | 13:51 |
| sean-k-mooney | oh something merged thats a shock | 13:51 |
| sean-k-mooney | its because test only change do not run the tempest jobs | 13:52 |
| sean-k-mooney | similar for docs onlychanges | 13:52 |
| kashyap | gibi: Thank you; (I fixed your other points too; sure you've seen it) | 13:53 |
| kashyap | gibi: Is the nit worth respinning? | 13:54 |
| sean-k-mooney | gibi: can you look at https://review.opendev.org/c/openstack/nova/+/869950 | 13:54 |
| kashyap | sean-k-mooney: melwitt: or anyone else: I'm chaing a second +2/+W: please have a look at this workaround - https://review.opendev.org/c/openstack/nova/+/870794 | 13:54 |
| sean-k-mooney | i have it open but i need to leave in 5 mins to drop back the tempory car i have and get back before my next meeting | 13:55 |
| kashyap | gibi: sean-k-mooney: On _top_ of this workaround I'll add in the "replace old API with new API" patch - https://review.opendev.org/c/openstack/nova/+/869950 -- libvirt: Replace usage of compareCPU() with compareHypervisorCPU() | 13:55 |
| sean-k-mooney | so it might be when i get back | 13:55 |
| kashyap | Sure. | 13:55 |
| sean-k-mooney | yep i want to merge both although i also think we should simplfy it so we only do one check | 13:56 |
| gibi | kashyap: that nit from me does not worth a respin | 13:56 |
| sean-k-mooney | e.g. only check with model+extra_flags | 13:56 |
| kashyap | Yep; ie. remove the workaround-code | 13:57 |
| kashyap | gibi: Nod; thx | 13:57 |
| kashyap | (Remove later, i.e. not now :)) | 13:57 |
| sean-k-mooney | anyway ill be back hopefully in an hour | 13:57 |
| gibi | kashyap: so https://review.opendev.org/c/openstack/nova/+/869950 will be top of https://review.opendev.org/c/openstack/nova/+/870794 ? | 13:58 |
| kashyap | gibi: Right | 13:58 |
| kashyap | Makes sense to you? | 13:58 |
| sean-k-mooney | well the workaround right now is disabling both check right | 13:58 |
| gibi | kashyap: yeah that make sens | 13:58 |
| sean-k-mooney | i have not looked at that patch in a few revisions | 13:58 |
| kashyap | sean-k-mooney: Yes, only if you enable the workaround, obv. | 13:58 |
| kashyap | sean-k-mooney: It's dead-simple - if workaround -> just skip the first _compare_cpu() call. | 13:59 |
| kashyap | sean-k-mooney: And thanks, I see you've also +2ed the API-replacement patch, but I need to rebase it on top of the workaroud patch | 13:59 |
| sean-k-mooney | and we still do the second call in init wiht the flags | 14:00 |
| sean-k-mooney | ok am ill leave this for now and come back to it when i have time to review both again | 14:00 |
| kashyap | Yeah, the second call correctly succeeds as it also takes into account the flags | 14:01 |
| sean-k-mooney | yep so as long as that is alwasy there i was ok with removing the first call | 14:02 |
| sean-k-mooney | i was not ok with removeing both of them and that why i wanted the workaround | 14:02 |
| sean-k-mooney | anyway got to go be back later | 14:02 |
| opendevreview | Sofia Enriquez proposed openstack/nova master: Implement is_luks_inside_qcow2 funtion https://review.opendev.org/c/openstack/nova/+/854030 | 14:05 |
| opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: Replace usage of compareCPU() with compareHypervisorCPU() https://review.opendev.org/c/openstack/nova/+/869950 | 14:22 |
| kashyap | gibi: --^ (No hurry) Done, rebased it on top of the workaround. | 14:23 |
| gibi | kashyap: done :) | 14:33 |
| kashyap | (シ_ _)シ | 14:34 |
| kashyap | (It's a Japanese bow) | 14:34 |
| gibi | ahh, I see :) | 14:47 |
| dansmith | gibi: nice | 15:00 |
| dansmith | gibi: I'm working on the real fix, but it's giving me a hard time.. lots of layers to cut through | 15:00 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: fixtures: Handle binding of first port https://review.opendev.org/c/openstack/nova/+/871048 | 15:06 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Neutron fixture: don't clobber profile and vif_details if empty https://review.opendev.org/c/openstack/nova/+/871049 | 15:08 |
| dansmith | ...aaand I got it working. sleep helps. | 15:08 |
| dansmith | now to clean up the giant mess I've made | 15:08 |
| gibi | dansmith: sounds good | 15:10 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: fixtures: Handle binding of first port https://review.opendev.org/c/openstack/nova/+/796905 | 15:28 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Neutron fixture: don't clobber profile and vif_details if empty https://review.opendev.org/c/openstack/nova/+/796906 | 15:28 |
| *** dasm is now known as Guest1846 | 15:29 | |
| *** Guest1846 is now known as dasm | 15:30 | |
| bauzas | gibi: all good to know | 17:45 |
| bauzas | hopefully https://review.opendev.org/c/openstack/tempest/+/870974 will be merged tonight | 17:45 |
| opendevreview | Sofia Enriquez proposed openstack/nova master: WIP: Implement encryption on backingStore https://review.opendev.org/c/openstack/nova/+/870012 | 20:50 |
| opendevreview | Sofia Enriquez proposed openstack/nova master: WIP: Implement encryption on backingStore https://review.opendev.org/c/openstack/nova/+/870012 | 21:11 |
| *** dasm is now known as dasm|off | 23:13 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!