| *** tosky has quit IRC | 00:09 | |
| *** swp20 has joined #openstack-nova | 01:33 | |
| *** macz_ has joined #openstack-nova | 02:01 | |
| *** macz_ has quit IRC | 02:06 | |
| *** jhesketh has joined #openstack-nova | 02:34 | |
| *** rcernin has quit IRC | 02:42 | |
| *** rcernin has joined #openstack-nova | 02:54 | |
| *** k_mouza has joined #openstack-nova | 03:08 | |
| *** khomesh24 has joined #openstack-nova | 03:09 | |
| *** k_mouza has quit IRC | 03:12 | |
| *** psachin has joined #openstack-nova | 03:42 | |
| *** vishalmanchanda has joined #openstack-nova | 04:22 | |
| *** ratailor has joined #openstack-nova | 04:38 | |
| *** rcernin has quit IRC | 04:51 | |
| *** rcernin has joined #openstack-nova | 04:56 | |
| *** vdrok has quit IRC | 04:58 | |
| *** johnsom has quit IRC | 04:58 | |
| *** flaviof has quit IRC | 04:58 | |
| *** TheJulia has quit IRC | 04:58 | |
| *** masayukig has quit IRC | 04:58 | |
| *** johnsom has joined #openstack-nova | 04:58 | |
| *** vdrok has joined #openstack-nova | 04:58 | |
| *** flaviof has joined #openstack-nova | 04:58 | |
| *** TheJulia has joined #openstack-nova | 04:58 | |
| *** masayukig has joined #openstack-nova | 04:58 | |
| *** sapd1 has quit IRC | 05:31 | |
| *** whoami-rajat__ has joined #openstack-nova | 05:32 | |
| *** k_mouza has joined #openstack-nova | 05:34 | |
| *** k_mouza has quit IRC | 05:38 | |
| *** k_mouza has joined #openstack-nova | 05:45 | |
| *** k_mouza has quit IRC | 05:50 | |
| *** rcernin has quit IRC | 07:16 | |
| *** rcernin has joined #openstack-nova | 07:19 | |
| *** ralonsoh has joined #openstack-nova | 07:22 | |
| *** rcernin has quit IRC | 07:23 | |
| *** rcernin has joined #openstack-nova | 07:36 | |
| *** links has joined #openstack-nova | 07:47 | |
| *** zoharm has joined #openstack-nova | 07:47 | |
| *** k_mouza has joined #openstack-nova | 07:50 | |
| *** k_mouza has quit IRC | 07:54 | |
| *** supamatt has quit IRC | 08:01 | |
| *** luksky has joined #openstack-nova | 08:07 | |
| *** prometheanfire has quit IRC | 08:10 | |
| *** andrewbonney has joined #openstack-nova | 08:22 | |
| *** k_mouza has joined #openstack-nova | 08:29 | |
| *** rcernin has quit IRC | 08:32 | |
| *** k_mouza has quit IRC | 08:33 | |
| *** jraju__ has joined #openstack-nova | 08:34 | |
| *** links has quit IRC | 08:35 | |
| *** ygk_12345 has joined #openstack-nova | 08:36 | |
| *** ygk_12345 has left #openstack-nova | 08:36 | |
| *** prometheanfire has joined #openstack-nova | 08:42 | |
| *** xek has joined #openstack-nova | 08:43 | |
| *** prometheanfire has quit IRC | 08:47 | |
| *** martinkennelly has joined #openstack-nova | 08:49 | |
| *** prometheanfire has joined #openstack-nova | 08:49 | |
| *** prometheanfire has quit IRC | 09:02 | |
| *** rpittau|afk is now known as rpittau | 09:03 | |
| kashyap | Morning, folks | 09:03 |
|---|---|---|
| kashyap | Anyone else hitting these two Nova live migration job failures: | 09:04 |
| kashyap | - tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_iscsi_volume | 09:05 |
| kashyap | - {0} tearDownClass (tempest.api.compute.admin.test_live_migration.LiveMigrationTest) | 09:05 |
| kashyap | Both the failures seem to be unrelated to my patch: https://zuul.opendev.org/t/openstack/build/2d510b007eb14485a5492c51efdc7318 | 09:06 |
| kashyap | Looks like timeouts | 09:09 |
| lyarwood | it's a detach timeout | 09:10 |
| kashyap | Right: | 09:11 |
| kashyap | Details: volume d461731f-26eb-419d-8b43-e0fd7a691abd failed to reach available status (current detaching) within the required time (196 s). | 09:11 |
| lyarwood | https://zuul.opendev.org/t/openstack/build/2d510b007eb14485a5492c51efdc7318/log/controller/logs/screen-n-cpu.txt#11065 | 09:11 |
| lyarwood | the failure to delete the volume is because n-cpu couldn't detach the actual device from the instance | 09:11 |
| *** prometheanfire has joined #openstack-nova | 09:11 | |
| kashyap | lyarwood: I see; what are options now? 'recheck'? | 09:12 |
| kashyap | Morning, BTW | 09:12 |
| kashyap | lyarwood: gibi: If you get time this week, can you have a gander at the simple patch below it too (a partial revert of an earlier one), when you can? -- https://review.opendev.org/c/openstack/nova/+/775431 (libvirt: Don't drop CPU flags with policy='disable' from guest XML) | 09:13 |
| *** ociuhandu has joined #openstack-nova | 09:13 | |
| kashyap | Sorry for the lengthy commit message; it is due to the example. | 09:14 |
| lyarwood | kashyap: morning :) yeah recheck | 09:14 |
| kashyap | lyarwood: Okay; will do. | 09:15 |
| lyarwood | kk | 09:15 |
| kashyap | Next week I'm on PTO, trying to see if I can get these across the line this week. | 09:15 |
| *** hemna has quit IRC | 09:20 | |
| *** mtreinish has quit IRC | 09:20 | |
| *** hemna has joined #openstack-nova | 09:20 | |
| *** zenkuro has joined #openstack-nova | 09:21 | |
| gibi | kashyap: will look | 09:21 |
| *** ociuhandu has quit IRC | 09:22 | |
| *** ociuhandu has joined #openstack-nova | 09:22 | |
| kashyap | Thank you | 09:23 |
| *** tosky has joined #openstack-nova | 09:24 | |
| *** rcernin has joined #openstack-nova | 09:28 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** dtantsur|afk is now known as dtantsur | 09:36 | |
| *** derekh has joined #openstack-nova | 09:56 | |
| *** k_mouza has joined #openstack-nova | 10:00 | |
| *** rcernin has quit IRC | 10:02 | |
| *** ociuhandu has joined #openstack-nova | 10:04 | |
| *** ociuhandu has quit IRC | 10:09 | |
| gibi | lyarwood: thanks for the reply in https://review.opendev.org/c/openstack/nova/+/767533 now it is lot clearer to me | 10:12 |
| *** hemanth_n has joined #openstack-nova | 10:12 | |
| lyarwood | gibi: np, just rewrote the commit to make it clear and working through some failures in the nova-next job using q35 at the tip of that series | 10:13 |
| lyarwood | brb | 10:13 |
| hemanth_n | lyarwood: sean-k-mooney: elod: zuul build issues are resolved now on nova stable/rocky, can you review this nova backport to rocky when you get time https://review.opendev.org/c/openstack/nova/+/761824 | 10:15 |
| sean-k-mooney | ah the pci stat change | 10:16 |
| sean-k-mooney | am sure just finishing an email | 10:17 |
| *** ociuhandu has joined #openstack-nova | 10:18 | |
| *** derekh has quit IRC | 10:19 | |
| *** derekh has joined #openstack-nova | 10:24 | |
| *** ociuhandu has quit IRC | 10:28 | |
| *** ociuhandu has joined #openstack-nova | 10:29 | |
| *** jangutter_ has quit IRC | 10:33 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-next: Drop NOVA_USE_SERVICE_TOKEN as it is now True by default https://review.opendev.org/c/openstack/nova/+/775575 | 10:33 |
| lyarwood | hemanth_n: ack looking | 10:34 |
| *** ociuhandu has quit IRC | 10:34 | |
| *** jangutter has joined #openstack-nova | 10:34 | |
| *** ociuhandu has joined #openstack-nova | 10:38 | |
| lyarwood | kashyap: any idea why blkid wouldn't work within an instance using the q35 machine type? | 10:43 |
| * kashyap thinks | 10:45 | |
| kashyap | lyarwood: It rings a faint bell; but my tired brain cannot seem to recall ... let me see if I can rediscover it | 10:46 |
| sean-k-mooney | lyarwood: i have used it before | 10:46 |
| lyarwood | kashyap: np, the context is https://review.opendev.org/c/openstack/nova/+/708701 where the only failures are due to the IDE bus not being present and these blkid failures to find the config drive | 10:47 |
| sean-k-mooney | lyarwood: it normally works | 10:47 |
| lyarwood | hmm ah wait I think I know what it is | 10:47 |
| lyarwood | just rubber duck'd myself there | 10:47 |
| lyarwood | the config drive is normally IDE | 10:47 |
| sean-k-mooney | ide + q35 | 10:47 |
| lyarwood | really? | 10:47 |
| kashyap | lyarwood: Aaah | 10:47 |
| sean-k-mooney | ya its sata | 10:47 |
| sean-k-mooney | for q35 | 10:47 |
| sean-k-mooney | so different name | 10:47 |
| lyarwood | yeah so it's now SATA so blkid is going to be different | 10:47 |
| lyarwood | yeah | 10:47 |
| lyarwood | fun | 10:47 |
| kashyap | lyarwood: Yes, for Q35 it is SATA -- you did the patch yourself :) | 10:48 |
| lyarwood | yeah I knew the IDE thing | 10:48 |
| lyarwood | I just didn't connect the dots with the config drive | 10:48 |
| lyarwood | I thought the label would be the same tbh | 10:48 |
| lyarwood | hmm | 10:48 |
| sean-k-mooney | the filesystem lable shoudl be | 10:49 |
| * lyarwood runs lsblk during a test | 10:51 | |
| *** ociuhandu has quit IRC | 10:55 | |
| *** ociuhandu has joined #openstack-nova | 10:56 | |
| *** macz_ has joined #openstack-nova | 11:01 | |
| *** ociuhandu has quit IRC | 11:01 | |
| *** jangutter_ has joined #openstack-nova | 11:05 | |
| *** macz_ has quit IRC | 11:06 | |
| *** dviroel has joined #openstack-nova | 11:07 | |
| *** jangutter has quit IRC | 11:09 | |
| *** zenkuro has quit IRC | 11:09 | |
| *** zenkuro has joined #openstack-nova | 11:09 | |
| lyarwood | hmm can we kill the ovsdbapp.backend.ovs_idl.vlog DEBUG logs in devstack@n-cpu | 11:20 |
| sean-k-mooney | ... yes i kind of ment to do that a while ago | 11:20 |
| sean-k-mooney | its form the lib we use in os-vif | 11:21 |
| lyarwood | ah | 11:21 |
| sean-k-mooney | that said its not that annoying just a bit noisy | 11:21 |
| sean-k-mooney | normally i dont notice it at this point but i perfonally have filtered it out in my head | 11:21 |
| sean-k-mooney | we can register a log filter in the os-vif init | 11:22 |
| sean-k-mooney | or do it in nova | 11:22 |
| lyarwood | yeah it's just killing my devstack env and I assume isn't helpful for our log indexing | 11:22 |
| sean-k-mooney | lyarwood: you are seeing this more recently because we now default to the ptyhon libvary | 11:23 |
| sean-k-mooney | neutron should also have that log in teh l2 agent | 11:23 |
| sean-k-mooney | if you feel like writing a patch it shoudl proably get setup here https://github.com/openstack/os-vif/blob/master/os_vif/__init__.py#L24-L49 | 11:24 |
| lyarwood | yup I can look at that later today | 11:25 |
| * lyarwood also has the os-brick patch to write | 11:25 | |
| lyarwood | kashyap: any idea how I can tell if q35 has a SATA bus? | 11:27 |
| kashyap | lyarwood: 'dmesg' in the guest? | 11:28 |
| sean-k-mooney | well it will the way we generate the xml by default | 11:28 |
| lyarwood | oh wait I wonder if support is actually missing in the cirros image | 11:28 |
| lyarwood | everything looks fine in the domain | 11:28 |
| lyarwood | the device is there and attached as a SATA cdrom etc | 11:29 |
| lyarwood | it's just not there within the guestOS | 11:29 |
| kashyap | lyarwood: Very good point; it's the CirrOS image | 11:29 |
| sean-k-mooney | well the cirros image usess a ubuntu 18.04 kernel | 11:29 |
| sean-k-mooney | so it will have sata support | 11:29 |
| lyarwood | you would think | 11:29 |
| lyarwood | don't they recompile the kernel now? | 11:30 |
| kashyap | Yeah, don't assume. | 11:30 |
| sean-k-mooney | im not | 11:30 |
| sean-k-mooney | i have use it with sata before | 11:30 |
| sean-k-mooney | although i dont know if i have used it with sata and q35 | 11:31 |
| sean-k-mooney | i have used it with sata and pc | 11:31 |
| sean-k-mooney | lyarwood: are you lookign at a gate failure by the way | 11:32 |
| lyarwood | sean-k-mooney: yeah the tip of my q35 series switches nova-next over to using q35 and some tests that rely on config drive (device tagging etc) are failing | 11:33 |
| sean-k-mooney | ah ok | 11:33 |
| lyarwood | sean-k-mooney: have a local reproducer anyway and I can't see anything related to SATA in dmesg within the instance | 11:33 |
| sean-k-mooney | ya its strang | 11:37 |
| sean-k-mooney | i booted it too | 11:37 |
| sean-k-mooney | so it boots fine with sata | 11:37 |
| sean-k-mooney | but lsblk and blkid are empty | 11:37 |
| * lyarwood tilts head | 11:38 | |
| lyarwood | f33 finds the device fine | 11:38 |
| lyarwood | weird, I could make the cdrom bus host configurable? | 11:39 |
| lyarwood | as a workaround | 11:39 |
| sean-k-mooney | well its configurable via the image alredy | 11:39 |
| sean-k-mooney | whats odd its ist not in /sys/block | 11:39 |
| kashyap | [root@rawhide-1 ~]# dmesg | grep -i sata | head -1 | 11:40 |
| kashyap | [ 1.724456] ahci 0000:00:1f.2: AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode | 11:40 |
| sean-k-mooney | i dont see it in /dev either | 11:41 |
| sean-k-mooney | kashyap empty | 11:41 |
| lyarwood | I wonder if SCSI would be a better bus for this anyway | 11:43 |
| kashyap | sean-k-mooney: Look for `dmesg | grep -i sata` | 11:43 |
| kashyap | What gues did you boot? | 11:43 |
| lyarwood | ah I guess we don't know if we have the controller setup | 11:43 |
| sean-k-mooney | kashyap: i did and it gives nothing | 11:43 |
| kashyap | What guest? | 11:44 |
| lyarwood | yeah same for me with cirros | 11:44 |
| sean-k-mooney | cirros | 11:44 |
| kashyap | Right; my guest is Fedora ("Rawhide" - rolling) | 11:45 |
| sean-k-mooney | yes which is not what we are trying to debug | 11:45 |
| sean-k-mooney | its odd because im not seeing anyting in /dev | 11:46 |
| sean-k-mooney | or /sys for the disk | 11:46 |
| sean-k-mooney | im really confused how the root filestyem is currenlty mounted | 11:46 |
| kashyap | sean-k-mooney: I was posting it for info; not saying Fedora is what we're debugging here | 11:47 |
| kashyap | lyarwood: sean-k-mooney: I see some forums online say CirrOS does not support SATA interface | 11:47 |
| sean-k-mooney | unless we use the vfat config driver format that woudl be a problem | 11:48 |
| sean-k-mooney | we cant really assuem scsi will work and virtio-blk is not avalid for a cdrom | 11:49 |
| lyarwood | sean-k-mooney: ah check /etc/modules | 11:50 |
| sean-k-mooney | just deleted them to try scsi | 11:50 |
| lyarwood | scsi should work but I think we can get sata to also work by updating /etc/modules | 11:50 |
| sean-k-mooney | that woudl requrie cirros to be updated | 11:51 |
| sean-k-mooney | given its botting it is able to use sata somehow | 11:52 |
| sean-k-mooney | /etc/modules is just the set of addtional moduels to load | 11:52 |
| sean-k-mooney | by the way why i said scsi isnt really an option is windows does not have teh virtio-scsi drivers by default | 11:53 |
| sean-k-mooney | so we could not change the default to scsi | 11:53 |
| sean-k-mooney | we chose sata becasue it shoudl work on all operating systmes | 11:53 |
| lyarwood | is scsi == virtio-scsi? | 11:53 |
| lyarwood | you have to set the model to virtio-scsi for that to be the case right? | 11:53 |
| sean-k-mooney | yes you do | 11:54 |
| sean-k-mooney | the other scsi contoller are all quite old | 11:54 |
| sean-k-mooney | im not sure we would want to use any of those by default | 11:54 |
| lyarwood | well at least Windows would support it then | 11:54 |
| lyarwood | but yeah | 11:54 |
| sean-k-mooney | im not sure if it would it might | 11:54 |
| sean-k-mooney | using scsi works by the way for cirros | 11:55 |
| sean-k-mooney | show up as normal | 11:55 |
| lyarwood | not for me as the config cdrom drive | 11:56 |
| lyarwood | so I think there's something still missing there | 11:56 |
| lyarwood | I found https://bugs.launchpad.net/cirros/+bug/1715009 from a while ago | 11:56 |
| openstack | Launchpad bug 1715009 in CirrOS "Missing modules for scsi cdrom config_drive in ppc64le" [Medium,Fix committed] | 11:56 |
| lyarwood | so I wonder if there's additional things we need on x86_64 | 11:56 |
| sean-k-mooney | so looking at /lib/modules/... | 11:57 |
| sean-k-mooney | cirros hs sfcsi and virtio drivers | 11:57 |
| sean-k-mooney | but not sata | 11:57 |
| sean-k-mooney | it only has the virtio scsi drivers by the way | 11:57 |
| sean-k-mooney | im guessing the init ramfs has the sata drivers | 11:58 |
| sean-k-mooney | or it fell back to an in kernel driver | 11:58 |
| sean-k-mooney | rahter then a module | 11:58 |
| *** nightmare_unreal has joined #openstack-nova | 11:59 | |
| *** ociuhandu has joined #openstack-nova | 12:00 | |
| sean-k-mooney | lyarwood: by the way we have a bug with bfv | 12:02 |
| sean-k-mooney | lyarwood: shocking i know but bfv does not select the correct bus when you use sata | 12:02 |
| sean-k-mooney | it will use scsi instead | 12:03 |
| sean-k-mooney | i fixed this years ago for non bfv | 12:03 |
| sean-k-mooney | but it looks like it back or was never fixed for bfv | 12:03 |
| kashyap | Sorry for my choppiness here. Struggling with something today. | 12:03 |
| openstackgerrit | Hemanth N proposed openstack/nova stable/rocky: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761824 | 12:08 |
| *** jraju__ has quit IRC | 12:09 | |
| *** ociuhandu has quit IRC | 12:22 | |
| *** zenkuro has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-nova | 12:23 | |
| *** zenkuro has joined #openstack-nova | 12:23 | |
| lyarwood | sean-k-mooney: sorry had to go afk, with hw_disk_bus=sata? | 12:26 |
| *** ociuhandu has quit IRC | 12:27 | |
| sean-k-mooney | lyarwood: ya if you hw_disk_bus=sata with bfv on ussuri then you get scsi | 12:30 |
| *** sapd1 has joined #openstack-nova | 12:34 | |
| *** ociuhandu has joined #openstack-nova | 12:39 | |
| *** k_mouza has quit IRC | 12:39 | |
| *** k_mouza has joined #openstack-nova | 12:39 | |
| *** khomesh24 has quit IRC | 12:44 | |
| *** martinkennelly has quit IRC | 12:45 | |
| *** martinkennelly has joined #openstack-nova | 12:47 | |
| *** Luzi has joined #openstack-nova | 12:55 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add a config update workflow test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898 | 13:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899 | 13:02 |
| *** macz_ has joined #openstack-nova | 13:02 | |
| *** macz_ has quit IRC | 13:07 | |
| *** khomesh24 has joined #openstack-nova | 13:08 | |
| *** xek has quit IRC | 13:22 | |
| *** xek has joined #openstack-nova | 13:23 | |
| *** artom has joined #openstack-nova | 13:23 | |
| *** xek has quit IRC | 13:26 | |
| *** xek_ has joined #openstack-nova | 13:26 | |
| *** hemanth_n has quit IRC | 13:29 | |
| *** khomesh24 has quit IRC | 13:39 | |
| *** khomesh24 has joined #openstack-nova | 13:55 | |
| *** macz_ has joined #openstack-nova | 13:56 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/c/openstack/nova/+/708701 | 13:57 |
| *** khomesh24 has quit IRC | 13:57 | |
| *** khomesh24 has joined #openstack-nova | 13:58 | |
| *** macz_ has quit IRC | 14:00 | |
| *** ratailor has quit IRC | 14:03 | |
| *** ociuhandu has quit IRC | 14:12 | |
| *** zenkuro has quit IRC | 14:14 | |
| *** ociuhandu has joined #openstack-nova | 14:14 | |
| *** zenkuro has joined #openstack-nova | 14:14 | |
| *** Luzi has quit IRC | 14:34 | |
| *** tosky has quit IRC | 14:55 | |
| *** zenkuro has quit IRC | 14:56 | |
| *** zenkuro has joined #openstack-nova | 14:56 | |
| *** ociuhandu has quit IRC | 14:56 | |
| *** ociuhandu has joined #openstack-nova | 14:57 | |
| *** efried has joined #openstack-nova | 15:13 | |
| *** psachin has quit IRC | 15:17 | |
| *** vishalmanchanda has quit IRC | 15:22 | |
| *** sapd1 has quit IRC | 15:27 | |
| *** iurygregory_ has joined #openstack-nova | 15:29 | |
| *** iurygregory has quit IRC | 15:30 | |
| *** iurygregory_ is now known as iurygregory | 15:30 | |
| lyarwood | stephenfin: thanks for the review, would you mind if I hold off on that respin until the rest of the series has been looked at? ./me doesn't want to hammer CI | 15:36 |
| stephenfin | yup, fine by me. Working through it atm | 15:36 |
| lyarwood | stephenfin: excellent thanks | 15:37 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/c/openstack/nova/+/748226 | 15:38 |
| *** ociuhandu has quit IRC | 15:39 | |
| *** ociuhandu has joined #openstack-nova | 15:40 | |
| *** xek_ is now known as xek | 15:44 | |
| *** ociuhandu has quit IRC | 15:44 | |
| *** ociuhandu has joined #openstack-nova | 15:46 | |
| *** khomesh24 has quit IRC | 15:46 | |
| *** macz_ has joined #openstack-nova | 15:48 | |
| *** dklyle has quit IRC | 15:48 | |
| *** ociuhandu has quit IRC | 15:51 | |
| *** ociuhandu has joined #openstack-nova | 15:52 | |
| kukacz | hi. having queens, nova disk with images_type=raw, when doing server resize I'm ending up with unbootable instance. I noticed that new (resized up) boot disk is being created as qcow, not raw. Is this a bug or am I rather missing some configuration detail? | 15:53 |
| lyarwood | kukacz: that sounds like a bug, can you open one and share some logs of the resize? | 15:57 |
| * lyarwood isn't sure we actually have any coverage in the gate for images_type=raw and resizing root disks | 15:58 | |
| lyarwood | https://launchpad.net/nova/+bug for the bug btw if you didn't know | 15:58 |
| *** tosky has joined #openstack-nova | 15:59 | |
| *** ociuhandu has quit IRC | 15:59 | |
| *** tosky has quit IRC | 15:59 | |
| *** ociuhandu has joined #openstack-nova | 16:00 | |
| *** tosky has joined #openstack-nova | 16:02 | |
| kukacz | lyarwood: sure, thanks, I'll collect logs+details into the bug. I'll need to switch the server to different configuration (lvm backend) for production, most probably not being able to collect more details or tests into the bug when required for some weeks. is it reasonable to file the bug despite such constraint? | 16:06 |
| lyarwood | kukacz: yup I might try to reproduce this tomorrow anyway | 16:08 |
| * lyarwood has downstream customers on queens using images_type=raw | 16:08 | |
| kukacz | lyarwood: perfect! I'll do that then | 16:09 |
| kukacz | btw. what is the current status of LVM backen resize support in Nova? I did a similar test with it and was (correctly) refused, stating it in logs that it's not supported. has that changed in newer releases? | 16:10 |
| lyarwood | kukacz: I don't think so but let me check | 16:11 |
| lyarwood | yeah I don't think it is sorry | 16:19 |
| *** ociuhandu has quit IRC | 16:29 | |
| *** ociuhandu has joined #openstack-nova | 16:29 | |
| kukacz | no worries, thanks for checking that! | 16:34 |
| *** prometheanfire has quit IRC | 16:34 | |
| *** ociuhandu has quit IRC | 16:34 | |
| *** macz_ has quit IRC | 16:36 | |
| *** ociuhandu has joined #openstack-nova | 16:37 | |
| *** prometheanfire has joined #openstack-nova | 16:39 | |
| kukacz | lyarwood: is the LVM backend in a good shape otherwise? I am planning to deploy "local disk" compute nodes utilizing LVM as image type after doing own evaluation, comparing it to qcow2/raw options. | 16:40 |
| lyarwood | kukacz: It doesn't get as much love as the default qcow2 backend and isn't something we support downstream as a result | 16:41 |
| *** ralonsoh has quit IRC | 16:44 | |
| *** ralonsoh has joined #openstack-nova | 16:44 | |
| kukacz | lyarwood: hmm, I see. I've found LVM superior both in performance characteristics (qcow2 on ext4 was badly stealing most of NVMe random pattern iops) and availability-operations capabililites though | 16:53 |
| kukacz | lyarwood: when writing "we support ..." you meant a particular vendor or Nova project? | 16:55 |
| lyarwood | kukacz: Vendor sorry, Red Hat. | 16:55 |
| kukacz | I am considering putting this into production, therefore I am so curious :-) | 16:56 |
| kukacz | ok, it might even mean both in that case ;-) | 16:56 |
| kukacz | I might still be missing something. especially I wonder if that performance hit on qcow2 (on default ext4 on 1-2 disk mdraid level 0) is common or there's an factor I've ignored so far | 16:59 |
| *** ociuhandu has quit IRC | 17:07 | |
| *** ociuhandu has joined #openstack-nova | 17:08 | |
| *** ociuhandu has quit IRC | 17:12 | |
| *** ociuhandu has joined #openstack-nova | 17:15 | |
| *** ociuhandu_ has joined #openstack-nova | 17:22 | |
| *** ociuhandu_ has quit IRC | 17:23 | |
| *** ociuhandu has quit IRC | 17:24 | |
| *** rpittau is now known as rpittau|afk | 17:47 | |
| *** mlavalle has joined #openstack-nova | 17:52 | |
| *** rcernin has joined #openstack-nova | 17:59 | |
| sean-k-mooney | kukacz: qcow2 tends not to have a large perfromnce hit if you preallocate the file by the way | 17:59 |
| sean-k-mooney | at least not on an xfs host filesystem | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| sean-k-mooney | im not sure you really rant to run raid 0 | 18:00 |
| sean-k-mooney | raid 10 maybe but unless you hate your customers data raid 0 is proably not what you want to use in production | 18:01 |
| *** johnsom has quit IRC | 18:01 | |
| *** rpittau|afk has quit IRC | 18:01 | |
| *** johnsom has joined #openstack-nova | 18:02 | |
| *** rpittau|afk has joined #openstack-nova | 18:04 | |
| *** rcernin has quit IRC | 18:04 | |
| kukacz | sean-k-mooney: I had the preallocation enabled while testing. the performance was good in sequential operations (>3300MiB/s), not in random though. in random, the 750k IOPS seen in lvm or raw are dropping to around 30k. tested on precreated 80GiB large fio file, running concurrently 6 instances | 18:05 |
| kukacz | I was using ext4 without extra parameters except -m0. same as with raw, though | 18:06 |
| sean-k-mooney | kukacz: try turing off the atime? i think | 18:07 |
| kukacz | hmm, I believe fio keeps the single testfile opened for all the time of test. but I might try that anyway | 18:07 |
| sean-k-mooney | for xfs i do defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio but you will proably want something similar for ext4 | 18:08 |
| sean-k-mooney | oh i have /var/lib/docker ext4 defaults,noatime 0 0 | 18:08 |
| sean-k-mooney | kukacz: historically | 18:09 |
| sean-k-mooney | kukacz: lvm had better io performacne but it took longer to deploy vms and used more space | 18:09 |
| sean-k-mooney | basically becasue even when using thing providioning it need to write all the data for the base iamge every time a vm boots | 18:10 |
| sean-k-mooney | we dont do any kind of snapshoting in the lvm driver to merged multiple thin provisioned images | 18:10 |
| sean-k-mooney | for the qcow backedn we share a backing disk and jsut create a thin snapshot | 18:10 |
| kukacz | I was not enabling thin provisioning with LVM, if you mean that nova.conf parameter | 18:11 |
| sean-k-mooney | so if you have long running vms lvm will likely preferm betere even if its less maintianed | 18:11 |
| sean-k-mooney | if you have short lived vms the qcow is proably going to be better. | 18:12 |
| kukacz | I wanted the extents to be preallocate in a predictable way - priorizing volumes to be placed on same disk, if it fits by size | 18:12 |
| kukacz | hmm, the provisioning time was nothing I have even noticed with LVM, but I was using quite smallish but typical ubuntu 20 image (around 5GB) | 18:14 |
| sean-k-mooney | are you on flash | 18:15 |
| sean-k-mooney | if you have ssd then you wont notice it too much | 18:15 |
| sean-k-mooney | if you have HDD and you have multiple vms starting you will | 18:15 |
| kukacz | raid0 was intentional here - focus on performance. if customer is using local disk, they assume that the single instance might be lost anyway. targetting cloud-native replicated applications | 18:15 |
| kukacz | yes, flash, NVMe SSDs | 18:16 |
| sean-k-mooney | kukacz: well jus tmake sure they are aware of that | 18:16 |
| sean-k-mooney | i assume you will be providing them with a ha cinder solution? ceph? | 18:16 |
| sean-k-mooney | so tha they can store all there main data elsewhere | 18:17 |
| kukacz | of course, they're having other options for more traditional network storage | 18:17 |
| kukacz | yes, ceph as cinder backend | 18:17 |
| sean-k-mooney | so local lvm is for low latency local storage | 18:17 |
| sean-k-mooney | for root files system/scratch space | 18:18 |
| sean-k-mooney | it used to work resonbably well for that | 18:18 |
| sean-k-mooney | i is tempthing to just use ceph for everything if you have it | 18:18 |
| sean-k-mooney | possible deploying osd on the compute nodes too | 18:19 |
| sean-k-mooney | if you find the perfomce of lvm is not worth it the rbd driver would be worht considering | 18:19 |
| kukacz | we've had used rbd driver so far as nova backend | 18:20 |
| kukacz | still, ceph is a shared network storage. if it has an issue, everyone in the cloud has an issue. very binding for the operators | 18:21 |
| kukacz | while there's an amount of workload which does not need the features of shared storage. also, they replicate blocks by themselves already, and storing those on ceph, which creates additional copies is counter-productive | 18:22 |
| kukacz | so the idea is to offer a storage layer for these workloads. it would get much better latencies, much higher iops, not being exposed to risk of central storage issue | 18:24 |
| sean-k-mooney | it a valid concern and ya that is what many doo | 18:26 |
| sean-k-mooney | if you can live with it being shared storage you can also modify the crush map to create an ssd only pool with no replication and expsoe that as a seperate cinder volume type | 18:27 |
| sean-k-mooney | anyway if you want best performce then you shoudl use eivehr lvm or raw | 18:27 |
| sean-k-mooney | qcow will be slower the both the other but it has some advnatges if customer will use thing like snapshopt | 18:28 |
| kukacz | so far, I've been quite excited with the LVM experience in this usage. only issue not being able to resize | 18:29 |
| *** macz_ has joined #openstack-nova | 18:30 | |
| kukacz | I acn even lose a drive from linear VG, only dropping the volumes stored on that particular drive. the rest of workload on same VG keeps running. which is also an advantage compared to raid-0 | 18:31 |
| kukacz | I can ... | 18:31 |
| *** dtantsur is now known as dtantsur|afk | 18:32 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/c/openstack/nova/+/673790 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Auto-detect UEFI, secure boot support https://review.opendev.org/c/openstack/nova/+/682627 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Methods to handle request for Secure Boot & non-Q35 machine types https://review.opendev.org/c/openstack/nova/+/682628 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add secure boot loader config support https://review.opendev.org/c/openstack/nova/+/775687 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add missing hints https://review.opendev.org/c/openstack/nova/+/775688 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps https://review.opendev.org/c/openstack/nova/+/775689 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler https://review.opendev.org/c/openstack/nova/+/775690 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Switch to libvirt's firmware auto-selection https://review.opendev.org/c/openstack/nova/+/775691 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Store UEFI'ness of guest as attribute https://review.opendev.org/c/openstack/nova/+/775692 | 18:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Stop checking non-host architectures for SEV https://review.opendev.org/c/openstack/nova/+/775693 | 18:34 |
| *** andrewbonney has quit IRC | 18:34 | |
| *** macz_ has quit IRC | 18:35 | |
| *** openstackgerrit has quit IRC | 18:38 | |
| *** martinkennelly has quit IRC | 18:39 | |
| kukacz | sean-k-mooney: I might retry the perf with xfs. thank you for the parameters. I don't target for top performance comparing lvm-raw-qcow. say 20% difference is perfectly ok if there is better implementation of qcow. just the huge drop (700k -> 30k) I'm seeing is too much already. I have to recheck the formatting/mount options. | 18:39 |
| sean-k-mooney | raw shoudl basically give you the same performace as lvm with a small amount of overhead for ext4/xfs | 18:41 |
| sean-k-mooney | but you can resize ectra | 18:41 |
| *** zoharm has quit IRC | 18:43 | |
| kukacz | yes, with raw I'm getting good numbers. resize did not work for me today though, perhaps I hit a bug - resizing an instance with raw disk in queens created an unbootable qcow image | 18:45 |
| sean-k-mooney | did you have the for raw disk set | 18:45 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.force_raw_images | 18:45 |
| kukacz | yes, I have that set to True | 18:46 |
| sean-k-mooney | and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.use_cow_images set to false | 18:47 |
| kukacz | no, I have use_cow_images unset, which probably renders true | 18:49 |
| sean-k-mooney | true is the default | 18:49 |
| aarents | kukacz: seems odd, we are heavy user of raw and resize are fine, we have use_cow_images to false as sean suggest in newton and stein | 18:50 |
| *** macz_ has joined #openstack-nova | 18:51 | |
| kukacz | aarents: interesting. I'll try that configuration | 18:51 |
| kukacz | though I did not have other issues with raw image deployments and backing files. only issue was this resize | 18:51 |
| sean-k-mooney | so we removed teh raw backend a while ago and folded it into the flat backend https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L1286 | 18:52 |
| sean-k-mooney | and it gets is resize form ate dynamically https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L593-L595 | 18:53 |
| sean-k-mooney | i think this is what decideds https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L537-L545 | 18:54 |
| sean-k-mooney | kukacz: perhaps qemu-image info was retruning something other then raw? | 18:55 |
| *** macz_ has quit IRC | 18:55 | |
| kukacz | sean-k-mooney: yes, file tool identified the disk as "QEMU QCOW Image (v3)" | 18:57 |
| sean-k-mooney | before or after the resize | 18:58 |
| kukacz | now I have retried with `use_cow_images = false` and that works | 18:59 |
| *** rcernin has joined #openstack-nova | 18:59 | |
| *** nightmare_unreal has quit IRC | 18:59 | |
| sean-k-mooney | did you explictly set teh images_type to raw? | 18:59 |
| kukacz | sean-k-mooney: that was after resize. original disk is identified as "DOS/MBR boot sector" = raw | 18:59 |
| kukacz | sean-k-mooney: yes, explicitely `images_type=raw` | 19:00 |
| sean-k-mooney | i suspect tha twehe we merged raw into flat the behavior changed slightly and now you neeed to set use_cow_iamges=false | 19:00 |
| sean-k-mooney | ya its not an alais for flat which wallows qcow images | 19:00 |
| sean-k-mooney | it just does not crate a backing files when it uses qcow image like the normal qcow one those | 19:01 |
| *** k_mouza has quit IRC | 19:03 | |
| sean-k-mooney | stephenfin: bauzas so i figured out my fixture issue. i was expecting to use the libvirtneutron fixture but i was using the base one. | 19:07 |
| sean-k-mooney | once i fixed that my func tests started passing. | 19:07 |
| *** openstackgerrit has joined #openstack-nova | 19:12 | |
| openstackgerrit | sean mooney proposed openstack/nova master: support per port numa policies with sriov https://review.opendev.org/c/openstack/nova/+/773792 | 19:12 |
| *** rcernin has quit IRC | 19:12 | |
| kukacz | sean-k-mooney, aarents: thank you! the resize issue is resolved. I did couple more tests and seems ok now. | 19:14 |
| sean-k-mooney | cool | 19:14 |
| sean-k-mooney | raw should be very very close to lvm so if that works for you then it will give you the advanate of better testing and more features too | 19:15 |
| kukacz | sean-k-mooney: hmm, that sounds good. so lvm is the least tested option compared to qcow/raw? not being used in production deployments as often? | 19:19 |
| sean-k-mooney | that is my understnading yes most peopel that dont use qcow for local sotrage use raw then lvm is less common after that | 19:25 |
| kukacz | there's another factor I've registered: with LVM I could expand capacity instantly just by adding a PV into VG. with qcow2/raw, I was using mdadm level 0, which needs to redistribute blocks while adding a disk. that was such a performance impacting and lengthy operation, that I could not be doing this online - with production workload running. | 19:27 |
| sean-k-mooney | well you can just put the raw files on an lvm volumn | 19:28 |
| kukacz | but I could move the LVM to OS level here, format, mount to /var/lib/nova and keep the disk management benefits perhaps, while utilitizing better integrated qcow or raw backends | 19:28 |
| sean-k-mooney | and expand it the same way | 19:28 |
| kukacz | sean-k-mooney: exactly :-) | 19:30 |
| sean-k-mooney | using raw/qcow does not percluded you mounting the nova state dir on an lvm volumn seperatly | 19:30 |
| kukacz | sean-k-mooney: you are right, it does not. I have just defined myself different test scenarios originally, not including this one | 19:33 |
| sean-k-mooney | my home openstack has my nova directory in an lvm vg on a raid 6 disk array with an optane ssd as a cache via bcache | 19:34 |
| kukacz | with the current knowledge, it seems the best option. supposing I can get to slightly comparable performance | 19:35 |
| sean-k-mooney | in your case you dont need bcach since its all ssd but for me its nice to hide some of the latency of HDDs | 19:36 |
| sean-k-mooney | anyway o/ | 19:36 |
| kukacz | of course. I'm also trying to keep the configuration as simple as possible. avoiding any extra layers. that's why I'm so enthusiastic of the idea of direct LVM usage | 19:40 |
| kukacz | if I put filesystem (/var/lib/nova/instances) on top of LVM, it will not partially-tolerate disk failure anymore. that could only be achieved with direct Nova-LVM integration | 19:42 |
| kukacz | seems that I'll need to accept a compromise | 19:43 |
| *** rcernin has joined #openstack-nova | 19:48 | |
| *** macz_ has joined #openstack-nova | 19:51 | |
| *** macz_ has quit IRC | 19:56 | |
| *** luksky has quit IRC | 19:58 | |
| *** luksky has joined #openstack-nova | 19:58 | |
| *** rcernin has quit IRC | 20:05 | |
| *** openstackgerrit has quit IRC | 20:06 | |
| *** luksky has quit IRC | 20:17 | |
| *** rcernin has joined #openstack-nova | 20:21 | |
| *** whoami-rajat__ has quit IRC | 20:22 | |
| *** luksky has joined #openstack-nova | 20:30 | |
| *** prometheanfire has quit IRC | 20:46 | |
| *** prometheanfire has joined #openstack-nova | 20:52 | |
| *** rcernin has quit IRC | 20:57 | |
| *** luksky has quit IRC | 21:00 | |
| *** prometheanfire has quit IRC | 21:06 | |
| *** prometheanfire has joined #openstack-nova | 21:12 | |
| *** luksky has joined #openstack-nova | 21:12 | |
| *** jkulik has quit IRC | 21:17 | |
| *** jkulik has joined #openstack-nova | 21:18 | |
| *** hamalq has joined #openstack-nova | 21:22 | |
| *** rcernin has joined #openstack-nova | 21:23 | |
| *** rcernin has quit IRC | 21:38 | |
| *** rcernin has joined #openstack-nova | 21:38 | |
| *** xek has quit IRC | 21:48 | |
| *** bbowen has quit IRC | 22:04 | |
| *** lxkong has joined #openstack-nova | 22:06 | |
| *** slaweq has quit IRC | 22:18 | |
| *** noonedeadpunk has quit IRC | 22:24 | |
| *** noonedeadpunk has joined #openstack-nova | 22:30 | |
| *** mlavalle has quit IRC | 23:02 | |
| *** luksky has quit IRC | 23:17 | |
| *** macz_ has joined #openstack-nova | 23:25 | |
| *** macz_ has quit IRC | 23:30 | |
| *** efried has quit IRC | 23:41 | |
| *** bbowen has joined #openstack-nova | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!