opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126 | 00:06 |
---|---|---|
*** bauzas_ is now known as bauzas | 01:26 | |
*** bauzas_ is now known as bauzas | 06:15 | |
*** elodilles_pto is now known as elodilles | 07:24 | |
opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.1: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926556 | 07:39 |
opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.1: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926557 | 07:39 |
bauzas | morning nova | 08:17 |
frickler | good morning bauzas, in case you missed it last week: please check the os-vif release patch https://review.opendev.org/c/openstack/releases/+/926768 , we're now past the deadline already | 08:25 |
bauzas | frickler: ack, I was on PTO for the previous 3 weeks but I'll take a look at it very shortly | 08:26 |
frickler | thx, just wanted to make sure that you were aware of it with all the flapping that's still happening with your account :) | 08:28 |
frickler | (I made my client ignore joins and quits, but not name changes, so these kind of stand out to me) | 08:29 |
*** ykarel_ is now known as ykarel | 08:36 | |
opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | 08:36 |
opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | 08:36 |
auniyal | bauzas, frickler, yes, so I was not sure of https://review.opendev.org/c/openstack/releases/+/926768 | 08:38 |
auniyal | so was waiting for bauzas | 08:38 |
bauzas | no worries | 09:01 |
*** bauzas_ is now known as bauzas | 10:44 | |
gibi | It seems we have a gate block on stable/2024.1 | 10:46 |
gibi | curl -f -O https://download.ceph.com/rpm-/el9/noarch/cephadm returns 404 making the ceph job fail | 10:46 |
gibi | example https://zuul.opendev.org/t/openstack/build/3be807e5dbf843ff995fbe1fdcb7fed2 | 10:46 |
gibi | elodilles: is it something known? | 10:46 |
elodilles | gibi: well, i haven't seen that yet :-o | 10:47 |
gibi | sorry not 2024.1 but 2023.2 | 10:48 |
elodilles | the link is definitely missing something after "rpm-" | 10:48 |
gibi | hm, good point | 10:49 |
elodilles | and it doesn't seem to be a constant error, at least if we look at the job history: https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack%2Fnova&branch=stable%2F2023.2&skip=0 | 10:50 |
sean-k-mooney | it should be https://download.ceph.com/rpm-quincy/ | 10:51 |
sean-k-mooney | or https://download.ceph.com/rpm-reef/ | 10:51 |
elodilles | it looks like a CEPH_RELEASE (?) variable or something like that was not set | 10:51 |
sean-k-mooney | yep | 10:51 |
sean-k-mooney | although that shoudl not block the gate just the c9s jobs | 10:51 |
sean-k-mooney | which should not be voting | 10:52 |
gibi | https://review.opendev.org/c/openstack/nova/+/926555 this reproduced the error twice in a row | 10:52 |
gibi | it happening in the nova-multi-cell jo | 10:52 |
gibi | b | 10:52 |
sean-k-mooney | why is that not using ubuntu? | 10:53 |
sean-k-mooney | oh it is | 10:53 |
sean-k-mooney | Ubuntu 22.04 | 10:54 |
sean-k-mooney | so for some reason its trying to pull the rpm version when it should be pulling the deb package | 10:54 |
gibi | hm there was error in the script before that | 10:56 |
gibi | Error: writing blob: storing blob to file "/var/tmp/storage2783954232/1": happened during read: unexpected EOF | 10:56 |
gibi | Error: writing blob: storing blob to file "/var/tmp/storage2783954232/1": happened during read: unexpected EOF | 10:56 |
gibi | https://zuul.opendev.org/t/openstack/build/3be807e5dbf843ff995fbe1fdcb7fed2/log/job-output.txt#11462 | 10:56 |
elodilles | gibi: note that the 1st error is from nova-ceph-multistore and the second error is from nova-multi-cell job | 10:57 |
elodilles | gibi: and the nova-multi-cell job failure is 2 test case failure | 10:57 |
elodilles | so it's not the same | 10:58 |
sean-k-mooney | ok cause i was looking at multi cell and not seeing any references to ceph :) | 10:58 |
elodilles | :] | 10:59 |
sean-k-mooney | oh it is | 11:00 |
sean-k-mooney | so looking at the multi store josb local.conf https://3e402c0e76741e83fc60-d00ff4f1a74cdbc5ea9d8044145b77c0.ssl.cf2.rackcdn.com/926555/2/check/nova-ceph-multistore/3be807e/controller/logs/_.localrc_auto.txt | 11:01 |
sean-k-mooney | we do not seam to be settign a release version in the job so we are using the defaults form the plugin | 11:01 |
sean-k-mooney | which should be ok | 11:01 |
sean-k-mooney | that is defiend here https://github.com/openstack/devstack-plugin-ceph/blob/stable/2023.2/devstack/lib/ceph#L31 | 11:04 |
sean-k-mooney | this is in the lib module meaing that it for internal use only otherwise it shoudl be in settings at the top of the plugin | 11:04 |
sean-k-mooney | it should still be set but lateter so its probaly fine as is | 11:05 |
sean-k-mooney | so this is just trhe podman pull that is failing an di have seen that fail at least once in the last 2 weeks | 11:06 |
sean-k-mooney | but i didnt look into why so im not sure this is jobs specific | 11:07 |
sean-k-mooney | i think the way this is coded we download the container that ceph adm would use | 11:08 |
sean-k-mooney | then we parse teh version form that and download the corresponding cephadm binary directly | 11:08 |
sean-k-mooney | and then use that | 11:08 |
sean-k-mooney | so the fact its using el9 on ubuntu does not matter https://github.com/openstack/devstack-plugin-ceph/blob/stable/2023.2/devstack/lib/cephadm#L132 | 11:09 |
sean-k-mooney | so ya this look like a generic podman pull issue with quay.io | 11:10 |
sean-k-mooney | we could perhaps retry that pull to mitigate it | 11:11 |
sean-k-mooney | although i think ideally we would mirror that or precache the container and cephadm in the base image at least for the default release | 11:12 |
frickler | iiuc we do have a caching proxy for quay.io, so someone just would have to make that job use it (cc gouthamr) | 11:25 |
sean-k-mooney | ah ok is that by configuing http_proxy ? or something else? | 11:25 |
sean-k-mooney | we have a generic setup mirrors role in the openstack-ci-jobs repo i think | 11:26 |
sean-k-mooney | i kind of assumed that also configured the http proxies | 11:26 |
frickler | there was some discussion about this in some other channel last week, iirc that work never completed. but kolla does somehow use that proxy already | 11:27 |
sean-k-mooney | ah i was thinking of https://github.com/openstack/openstack-zuul-jobs/tree/master/roles but i guess the morrior setup is in the generic zuul-jobs repo | 11:27 |
gibi | sorry for the mixup | 11:28 |
sean-k-mooney | i guess that does not handel the systm proxies | 11:28 |
sean-k-mooney | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/defaults/main.yaml | 11:29 |
frickler | sean-k-mooney: yes, afaict this is still missing for general zuul jobs. cf. https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/templates/globals-default.j2#L82-L88 | 11:30 |
sean-k-mooney | so that could be a good change to put in the base devstack job | 11:30 |
sean-k-mooney | do we know why that has not been done before? | 11:30 |
sean-k-mooney | oh this is doing it via explicly using a difent registry | 11:32 |
frickler | sean-k-mooney: iiuc this was started by some tripleo folks but never finished. https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2024-08-21.log.html#t2024-08-21T11:06:16 was the discussion I was referring to | 11:32 |
sean-k-mooney | right i tought this might be settign the generic http proxy but its acatully podman/docker speciric config | 11:32 |
frickler | yes, we cannot do a generic proxy since we could not properly secure it | 11:33 |
sean-k-mooney | i think the kolla approch need to have all usage of podman be parmatiarised by the registry | 11:33 |
sean-k-mooney | so we would need to modify the ceph devstack plugin | 11:33 |
sean-k-mooney | but that should not be hard | 11:33 |
frickler | yes, I don't think podman is currently being used anywhere else in devstack? even when it was, that should not be a blocker | 11:34 |
sean-k-mooney | it might have been used for something else but we can always do it one plugin at a time | 11:35 |
sean-k-mooney | i dont think we use it in core devstack | 11:35 |
opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.1: hardware: Correct log https://review.opendev.org/c/openstack/nova/+/927110 | 11:41 |
sean-k-mooney | frickler: looks like we can either configure a mirror which will take precidnece and then fall back or we can remap prefixes | 11:50 |
sean-k-mooney | https://www.redhat.com/sysadmin/manage-container-registries | 11:50 |
sean-k-mooney | i think the mirror approch is prefered as it will try the direct connection if the mirror does not work | 11:50 |
sean-k-mooney | i can see if i can find time to code that up in core devstack | 11:51 |
*** bauzas_ is now known as bauzas | 13:27 | |
opendevreview | Dan Smith proposed openstack/nova master: Use format_inspector from oslo https://review.opendev.org/c/openstack/nova/+/925025 | 14:22 |
opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | 14:55 |
opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 14:55 |
opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | 14:56 |
opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | 14:56 |
dansmith | bauzas: hopefully this will be quick, but here's the patch to deprecate AMI as we discussed.. people wanted to wait for you to return/approve: https://review.opendev.org/c/openstack/nova/+/926144 | 15:10 |
bauzas | dansmith: all good, will look at it now | 15:10 |
*** bauzas_ is now known as bauzas | 16:20 | |
sean-k-mooney | gibi: so as i was saying in the qa channel thanks for the reivews. i wonder wich is a higher priority to make progress on the healthch code or the eventlet removal | 16:48 |
sean-k-mooney | i put the healthcheck partly on top of the eventlet serise to avoid a merge conflcit but perhaps i should split them | 16:48 |
sean-k-mooney | im not convince we will have time to merge both or either of them so im open to input on which to proceed with first | 16:49 |
melwitt | sean-k-mooney: I just looked at the review comments you mention and yeah, that's the same conclusion that was reached with my patch is that there is no way to cancel a native thread (from the main thread). so it's either let it keep running (and assume it exits on its own eventually) or switch to processes. or like you said somehow run a timer in the child thread to exception exit itself, if there is a way we can do that | 16:58 |
sean-k-mooney | we might be able to do that with funtools.partil and have it create a time before callign the orginal funciton | 16:59 |
melwitt | the code as-is in the patch will just move on after the timer expires and leave the worker thread | 16:59 |
sean-k-mooney | but that feels even more hacky | 16:59 |
sean-k-mooney | i could use a process pool but that feels a bit heavy weight. if people are ok with that its easy to do i think | 16:59 |
sean-k-mooney | process pools are ok provide the funciton does not try to modify any shared sate | 17:00 |
sean-k-mooney | i dont think this code does but thats somethign we would need to be careful of | 17:00 |
melwitt | yeah. with my patch the review just kept going around in circles with nobody happy with anything so I hopefully you will have better luck | 17:00 |
sean-k-mooney | im feelign like i might be better off moving that patch to later in the series | 17:01 |
sean-k-mooney | however i see value in makeing the api eventlet free first | 17:01 |
sean-k-mooney | since otherwise its the simplest service to do that with | 17:02 |
*** bauzas_ is now known as bauzas | 17:23 | |
opendevreview | melanie witt proposed openstack/nova master: DNM testing https://review.opendev.org/c/openstack/nova/+/925635 | 19:17 |
opendevreview | melanie witt proposed openstack/nova master: DNM testing https://review.opendev.org/c/openstack/nova/+/925635 | 19:19 |
sean-k-mooney | melwitt: by the way this is somehting i was playing with https://github.com/SeanMooney/os-vol | 19:20 |
melwitt | sean-k-mooney: that is very different :) | 19:30 |
sean-k-mooney | yep kind of intentionally | 19:30 |
sean-k-mooney | i was playing with the idea of storage backends abstracting how the storage is provisioned and allocating volumed which are then consumed | 19:30 |
sean-k-mooney | with the connection between a volume and a backend indirect though a storage pool | 19:31 |
sean-k-mooney | so you would not create a file or rbd volume | 19:32 |
sean-k-mooney | you would ask the storage pool to allocate a volume of a given size from its backend | 19:32 |
sean-k-mooney | that volume object is the logical handel to the backing storage | 19:33 |
melwitt | hm ok | 19:33 |
sean-k-mooney | my idea was you could have multiple backend and have adaptor that can take data form diffen sources and import it into a volume | 19:34 |
sean-k-mooney | i have not really fleshed out the interface much as i said i was just playing with that desgin to see what it looked like | 19:35 |
melwitt | that's cool | 19:35 |
sean-k-mooney | im not saying you should not consinue with your experiments by the way, the design im going for might be overly generic for what we want | 19:36 |
sean-k-mooney | on the other hand i kind of like the idea of having this in a self contianed lib or module with a stict inteface | 19:37 |
sean-k-mooney | the reason i have host_attach in the interafce for example is so we can allocate a file (lvm volume, rbd volume) | 19:37 |
sean-k-mooney | attach it to the host | 19:38 |
sean-k-mooney | call makefs on it and unmount it | 19:38 |
sean-k-mooney | without having to care where the data is stored | 19:38 |
melwitt | sure. where I was coming from is I wondered if we can pare down/simplify imagebackend as a baby step and whether or not it could make image format tracking easier in some way | 19:39 |
sean-k-mooney | yep | 19:39 |
sean-k-mooney | i think that would be valueable to do in anycase | 19:39 |
melwitt | I was thinking priority one is getting the format tracking done and I thought maybe removing some of the confusing layers of indirection could help facilitate that | 19:40 |
sean-k-mooney | yep i agree | 19:40 |
sean-k-mooney | i dont know if you looked at some of the tests i created https://github.com/SeanMooney/os-vol/blob/master/os_vol_tests/unit/backends/test_flat_file.py#L38-L63 but as i said this is just something im toying with form time to time | 19:42 |
sean-k-mooney | i have not considered how you could use this in nova really | 19:42 |
sean-k-mooney | having a simplifed code path would help if we were ever going to consider going this way | 19:43 |
melwitt | dansmith: I tried a minimalistic refactor of imagebackend things with the goal of helping somehow with the image format tracking we want to do https://review.opendev.org/c/openstack/nova/+/925635 I dunno if something like that could help with the ideas you had for the format tracking | 19:44 |
dansmith | melwitt: ah cool, I'll queue up a look | 19:45 |
melwitt | sean-k-mooney: ah yeah, I didn't see the tests. that does make it easier to understand | 19:46 |
sean-k-mooney | i defintly have some bugs but im going to keep playing wiht it on and off and see if i can build it out to be something more useful but its just a toy for now. | 19:47 |
opendevreview | Merged openstack/nova stable/2024.1: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926556 | 21:01 |
opendevreview | Merged openstack/nova stable/2024.1: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926557 | 21:07 |
*** bauzas_ is now known as bauzas | 21:11 | |
opendevreview | Merged openstack/nova master: Deprecate AMI image formats https://review.opendev.org/c/openstack/nova/+/926144 | 21:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!