| opendevreview | Dr. Jens Harbott proposed openstack/grenade master: Make octavia-grenade job voting https://review.opendev.org/c/openstack/grenade/+/896867 | 02:22 |
|---|---|---|
| opendevreview | Ghanshyam proposed openstack/grenade master: Update grenade setting on current master https://review.opendev.org/c/openstack/grenade/+/896831 | 04:02 |
| opendevreview | Ghanshyam proposed openstack/grenade master: Make octavia-grenade job voting https://review.opendev.org/c/openstack/grenade/+/896867 | 04:02 |
| opendevreview | Dr. Jens Harbott proposed openstack/grenade stable/2023.2: stable-only: drop grenade-skip-level job definition https://review.opendev.org/c/openstack/grenade/+/897381 | 05:43 |
| opendevreview | Dr. Jens Harbott proposed openstack/grenade master: Update grenade setting on current master https://review.opendev.org/c/openstack/grenade/+/896831 | 05:44 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup for object_storage tests https://review.opendev.org/c/openstack/tempest/+/896011 | 07:17 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup for object_storage tests https://review.opendev.org/c/openstack/tempest/+/896011 | 07:21 |
| opendevreview | Lukas Piwowarski proposed openstack/devstack master: Add support volume backup_driver config option https://review.opendev.org/c/openstack/devstack/+/897402 | 08:30 |
| opendevreview | Dr. Jens Harbott proposed openstack/grenade master: Make octavia-grenade job voting https://review.opendev.org/c/openstack/grenade/+/896867 | 08:46 |
| yadnesh | Can someone please review https://review.opendev.org/c/openstack/devstack/+/896215 & https://review.opendev.org/c/openstack/devstack/+/896762 | 09:38 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup for object_storage tests https://review.opendev.org/c/openstack/tempest/+/896011 | 09:57 |
| yadnesh | kopecmartin++ thanks for the review | 12:25 |
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: Fix cleanup for object_storage tests https://review.opendev.org/c/openstack/tempest/+/896011 | 12:44 |
| frickler | kopecmartin: please also check https://review.opendev.org/c/openstack/grenade/+/897381 , it is the fix needed for the grenade update on master | 14:23 |
| opendevreview | Merged openstack/devstack stable/xena: Use RDO official CloudSIG mirrors for C9S deployments https://review.opendev.org/c/openstack/devstack/+/896215 | 14:31 |
| opendevreview | Lukas Piwowarski proposed openstack/devstack master: Add support volume backup_driver config option https://review.opendev.org/c/openstack/devstack/+/897402 | 15:28 |
| opendevreview | Merged openstack/whitebox-tempest-plugin master: Enable LM back and forth flag for SRIOV https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/889338 | 15:43 |
| gmann | kopecmartin: you around? | 16:30 |
| opendevreview | Merged openstack/grenade stable/2023.2: stable-only: drop grenade-skip-level job definition https://review.opendev.org/c/openstack/grenade/+/897381 | 16:40 |
| sean-k-mooney | frickler: fungi: just want to pick your brains on the writhe way to approch something when you have time | 17:52 |
| sean-k-mooney | frickler: fungi: if we wanted to test something with a kernel module that is not packaged. what would ye recommend | 17:53 |
| sean-k-mooney | in the past i have dont that by having a devstack plugin compile it (we also did that for ovs in the past) | 17:54 |
| sean-k-mooney | i could also see doing that in a disk-image builder context | 17:54 |
| sean-k-mooney | i assume we dont have anything generic that can can compile an addtional ubuntu kernel moduel for the current kernel write | 17:54 |
| sean-k-mooney | the reason im asking is currently we supprot generic mdevs in nova but we dont have any testing of them | 17:55 |
| sean-k-mooney | the kernel has a couple of sable mdev modules that we could use for testign but they are not packaged on ubuntu or debian | 17:56 |
| sean-k-mooney | these ones | 17:56 |
| sean-k-mooney | im playing with it in a vm at the moment to see if i could use the mtty or mdpy sample model with a nova vm | 17:57 |
| fungi | sean-k-mooney: are there separate (binary or dkms) packages for them maybe? | 17:57 |
| sean-k-mooney | just to see if its posibel to use one of those sample modules for testing | 17:57 |
| sean-k-mooney | fungi: i was hoping they woudl be in kernel-moduels-extra | 17:57 |
| fungi | what are the lkm filenames? | 17:57 |
| sean-k-mooney | do you mean /samples/vfio-mdev/mdpy.c | 17:58 |
| fungi | i'll take a quick poke around with apt-file and see if they're in some other package already | 17:58 |
| fungi | yeah | 17:58 |
| sean-k-mooney | in the past i have just manually compiled them but i dont erally want to do that in a job if its avoidable | 17:59 |
| sean-k-mooney | so if there is a package that would be nice | 17:59 |
| sean-k-mooney | im trying to see if there is anything we can do to impove test coverage | 17:59 |
| sean-k-mooney | if the anser is now thats fine | 17:59 |
| sean-k-mooney | *no | 18:00 |
| fungi | like once they're compiled, you'll have a something.ko file | 18:00 |
| fungi | and are these under block? net? virt? | 18:00 |
| sean-k-mooney | ya so its been about 2 years since i last did this but when i last did it i comiipled a .ko and used instmod to load it | 18:01 |
| sean-k-mooney | sample | 18:01 |
| sean-k-mooney | well sampels | 18:01 |
| sean-k-mooney | i have a compile runing currenlty ill check if it has produced them yet | 18:01 |
| fungi | right, i'm just asking whether you know the names of the .ko files you're wanting to load | 18:02 |
| fungi | i can search the package database for those files, in case they're already packaged under some other grouping | 18:02 |
| clarkb | one thing we have tried to move away from in our test env is special sauce that creates confusion when things don't work outside our test env | 18:03 |
| fungi | and they'll generally be installed somewhere like /lib/modules/6.5.0-1-amd64/kernel/samples/ then? | 18:03 |
| clarkb | put another way this means we should not add that to our kernels in dib | 18:03 |
| fungi | yeah, we definitely have other options, an "easy" one might be to just package it in a ppa | 18:04 |
| clarkb | depending on the module it might only take a few seconds to compile as well | 18:04 |
| sean-k-mooney | ya | 18:04 |
| fungi | right, and there may even be a dkms recipe/package for it too | 18:05 |
| sean-k-mooney | so right now im doing the ultra dumb approch of tweaking the ubuntu kernel cofnig and rebuilding it | 18:05 |
| sean-k-mooney | but in the past i have built just that module | 18:05 |
| sean-k-mooney | and that is not that long | 18:05 |
| fungi | sure, install kernel headers, run the module build, load it | 18:05 |
| JayF | If it's not in the ubuntu/debian kernel packages, and doesn't have a dkms package, have you considered requesting upstream package it? | 18:05 |
| sean-k-mooney | for ovs-dpdk in the past we compile ovs and dpdk form source in devstack | 18:05 |
| fungi | as long as it doesn't need to be loaded at boot, should work fine in a job | 18:05 |
| sean-k-mooney | it does not | 18:06 |
| JayF | We have Canonical employees/contributors in OpenStack, that might solve a "moving forward" problem even if this works for now. | 18:06 |
| sean-k-mooney | i guess the real question is "are the sameple kernel moduels" packaged anywhere | 18:06 |
| sean-k-mooney | my google foo said no | 18:06 |
| fungi | yeah, JayF makes an excellent point, even if it may not happen for the current version we're testing on it could become simpler in 2024.1 lts | 18:06 |
| fungi | sean-k-mooney: do you happen to know what makes them "samples" rather than normal drivers? | 18:07 |
| sean-k-mooney | before i ask anyone to package anything | 18:07 |
| sean-k-mooney | i want to locally confirm we coudl use them for testing | 18:07 |
| fungi | of course | 18:07 |
| fungi | it may even be that we ask the kernel devs to move one or more of those samples so that they're considered actually supported drivers too | 18:08 |
| sean-k-mooney | fungi: baicaly tehse are fake driver used for testing the kernel subsystem and as a refernce to other to implemente a real hardware driver | 18:08 |
| sean-k-mooney | so mtty create a fake tty that is exposed as an mdev that just loop back any data you write to it | 18:08 |
| sean-k-mooney | mdpy create a fake vgpu | 18:09 |
| fungi | aha, so it may make sense to create a sample drivers package with those too | 18:09 |
| fungi | "driver fakes/stubs for use in testing" or whatever | 18:09 |
| sean-k-mooney | ya | 18:09 |
| sean-k-mooney | so mdpy and mbochs are two fake vGPU impelmetaions | 18:10 |
| sean-k-mooney | that we coudl actull use to test vGPU without fancy nviaia gpus | 18:10 |
| sean-k-mooney | at least in theory | 18:10 |
| JayF | sean-k-mooney: you make me slightly jealous :D | 18:10 |
| * JayF wants a kernel-module fake DPU to test Ironic with /s ;) | 18:10 | |
| sean-k-mooney | well this is all still in theory | 18:11 |
| JayF | Instead we get to write fake-hardware-to-libvirt abstraction layers and keep a stick around to prevent users from going "I can make my own nova with Ironic!" which they love to do | 18:11 |
| sean-k-mooney | in other news qemu added emulation of intesl igb nic recently | 18:12 |
| sean-k-mooney | what makes that interesting is it supprot sriov | 18:12 |
| fungi | oh, so we could do virtual sriov testing? | 18:13 |
| sean-k-mooney | meaning its not possibel to create a libvirt/qemu vm that can be used for sriov | 18:13 |
| sean-k-mooney | yep | 18:13 |
| fungi | awesome | 18:13 |
| sean-k-mooney | i have not fully got it working but i started trying that on monday | 18:13 |
| sean-k-mooney | libvirt in the vm sees the vfs perfectly | 18:13 |
| sean-k-mooney | but i hit a bug in nova | 18:13 |
| sean-k-mooney | which i guess i now need to go fix | 18:13 |
| fungi | at least you know some people who know some people who can make that happen | 18:14 |
| fungi | (or you can just make that happen) | 18:14 |
| sean-k-mooney | ya so i actully dont know why its not working but its failing in nova resouce tracker | 18:14 |
| sean-k-mooney | but nova detected the vfs and put them in the db right so i dont knwo why its now not happy with asigining one to a vm | 18:15 |
| sean-k-mooney | anyway that next weeks me's problem | 18:15 |
| sean-k-mooney | fungi: once i can confimr this work logall i plan to look at enableing it in nova too. | 18:16 |
| sean-k-mooney | by that i mena just addtint igb to the list of emulated nic module nova supprots so that you can ask for a vm with a neutron port and have that port emulated as a device that capablie of sriov | 18:18 |
| fungi | sean-k-mooney: so anyway, if the sample gpu fake drivers have "mdpy" or "mbochs" in their .ko filenames, i'm not finding any evidence they're packaged in current ubuntu lts, but i'd hit up jamespage or tinwood in #openstack-infra with questions about getting them officially packaged in the future. in the meantime, local module compiles as part of the job are probably your fastest path | 18:22 |
| fungi | forward | 18:22 |
| sean-k-mooney | thanks thats what i assumed | 18:23 |
| fungi | sounds like they'd be pretty quick ansible tasks anyway | 18:23 |
| sean-k-mooney | basically i wanted to confirm if this was possibel and then ask in the ptg is this something we would like to do | 18:23 |
| fungi | makes sense | 18:23 |
| fungi | more hardware enablement is good from my perspective, but there are lots of other stakeholders | 18:24 |
| sean-k-mooney | vgpus are kind of in a weired place | 18:25 |
| sean-k-mooney | in the past we used to require hardware vendors to provide third party ci | 18:25 |
| fungi | particularly with the licensing requirements right now, yeah | 18:25 |
| sean-k-mooney | but for vgpus nvidia had no deirsre to to that yet we had operators/user of openstack askign for them | 18:26 |
| sean-k-mooney | so right now i dont think we have any first/thirdpary testing of vgpus in nova or cyborg | 18:26 |
| sean-k-mooney | we have unit/functional tests but well they only go so far | 18:26 |
| clarkb | fwiw nothing says nvidia has to run the third party ci | 18:27 |
| sean-k-mooney | yes but the issue is licesning | 18:28 |
| sean-k-mooney | i really wish there was an opensouce mdev capable card | 18:28 |
| sean-k-mooney | so we coudl test without there proprity drivers and lieciening server | 18:29 |
| sean-k-mooney | that partly why im interested in the fake kernel moduels | 18:29 |
| clarkb | yes whoever runs it would need to pay for the licenses assuming nvidia would issue them for that purpose | 18:29 |
| sean-k-mooney | i was exicted but disapointed by intels gpus | 18:30 |
| JayF | sean-k-mooney: Ironic is having a third-party-CI meeting sorta along those lines during PTG; with lessened investment in OpenStack by large vendors & improved standardizataion it's getting harder to justify a hardline stance | 18:30 |
| sean-k-mooney | the actully supprot vgpu on one of there integrated gpus on there cpus | 18:31 |
| fungi | with the collapse of blockchain hype/ponzi schemes, unless machine learning picks up the slack, nvidia will be revisiting its temporary gpu empire | 18:31 |
| sean-k-mooney | but not on there pcie add in cards | 18:31 |
| * sean-k-mooney waits for the slack based chatGPT bot wars to start | 18:32 | |
| sean-k-mooney | you know someone has pitch slaping chatGPT into slack somwhere | 18:33 |
| sean-k-mooney | and nivida will be happy to sell gpus to power it | 18:34 |
| sean-k-mooney | its something i really dont know that much about (how much computational power does an large Langue model use once its traied) | 18:35 |
| fungi | not a lot from what i understand, it's the training processes which use a lot of processor power (and have been tuned in many cases to offload that onto gpu coprocessors) | 18:38 |
| fungi | but definitely not my field of expertise, i only know of these technologies third-hand | 18:39 |
| sean-k-mooney | i have been using github codepiolt a littel but the designer of it would cry if they knwo i mostly use it for writing commnets | 18:40 |
| sean-k-mooney | it can get close but form my experice of usign it so far unless you are very familar with the codebase in question | 18:41 |
| fungi | from what i understand, it's pretty good at describing functions | 18:41 |
| fungi | just not as good at writing them | 18:42 |
| sean-k-mooney | you wont notice that the perferctly reasonable function name it invented does not actully exist | 18:42 |
| sean-k-mooney | ya | 18:42 |
| sean-k-mooney | so the code it generates looks correct | 18:42 |
| sean-k-mooney | that does not mean any of it works altogh its often close | 18:42 |
| fungi | that's the curse of every llm. what it creates will *look* correct | 18:43 |
| fungi | whether it's source code or answers to technical questions | 18:43 |
| sean-k-mooney | ... BTF [M] drivers/net/ethernet/brocade/bna/bna.ko | 18:44 |
| sean-k-mooney | x86_64-linux-gnu-objcopy: drivers/gpu/drm/amd/amdgpu/stY7KMFU: No space left on device | 18:44 |
| sean-k-mooney | it litraly just finsite dteh compile and was writeing all the ko files | 18:45 |
| sean-k-mooney | ok fungi clarkb thanks for the input ill try again tomorrow | 18:45 |
| fungi | ouch | 18:47 |
| fungi | yeah, i have a separate scratch lv i mount just for doing kernel recompiles, for that very reason | 18:47 |
| sean-k-mooney | its a devstack vm so its fine | 18:48 |
| sean-k-mooney | it ran out because i have devstack also deployed in it | 18:48 |
| sean-k-mooney | and cinder/swift loop files are using much of the space | 18:49 |
| sean-k-mooney | -rw-r--r-- 1 sean sean 30G Oct 2 17:58 stack-volumes-lvmdriver-1-backing-file | 18:50 |
| sean-k-mooney | given my root is 35G that would defeinlly do it | 18:51 |
| opendevreview | Merged openstack/grenade master: Update grenade setting on current master https://review.opendev.org/c/openstack/grenade/+/896831 | 19:33 |
| kopecmartin | gmann: what's up? | 21:14 |
| * kopecmartin gonna get some sleep | 21:41 | |
| gmann | kopecmartin: once you are online in morning, this is ready depends-on is merged openstack | 21:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!