| *** k_mouza has joined #openstack-nova | 00:05 | |
| *** k_mouza has quit IRC | 00:10 | |
| *** k_mouza has joined #openstack-nova | 00:16 | |
| *** k_mouza has quit IRC | 00:20 | |
| *** k_mouza has joined #openstack-nova | 00:48 | |
| *** martinkennelly has quit IRC | 00:49 | |
| *** k_mouza has quit IRC | 00:52 | |
| *** damien_r has quit IRC | 01:03 | |
| *** damien_r has joined #openstack-nova | 01:04 | |
| *** ChanServ has quit IRC | 01:05 | |
| *** damien_r has quit IRC | 01:05 | |
| *** boxiang_ has quit IRC | 01:05 | |
| *** LinPeiWen has quit IRC | 01:12 | |
| *** damien_r has joined #openstack-nova | 01:15 | |
| *** boxiang_ has joined #openstack-nova | 01:15 | |
| *** ChanServ has joined #openstack-nova | 01:15 | |
| *** weber.freenode.net sets mode: +o ChanServ | 01:15 | |
| *** LinPeiWen has joined #openstack-nova | 01:37 | |
| *** boxiang_ has quit IRC | 02:08 | |
| *** boxiang_ has joined #openstack-nova | 02:08 | |
| *** mkrai has joined #openstack-nova | 03:01 | |
| *** derekh has quit IRC | 03:16 | |
| *** derekh has joined #openstack-nova | 03:22 | |
| *** k_mouza has joined #openstack-nova | 04:10 | |
| *** k_mouza has quit IRC | 04:11 | |
| *** k_mouza_ has joined #openstack-nova | 04:11 | |
| *** k_mouza_ has quit IRC | 04:15 | |
| *** ricolin has joined #openstack-nova | 04:18 | |
| *** ratailor has joined #openstack-nova | 04:45 | |
| *** ratailor_ has joined #openstack-nova | 04:56 | |
| *** ratailor has quit IRC | 04:59 | |
| *** links has joined #openstack-nova | 05:51 | |
| *** slaweq has joined #openstack-nova | 06:03 | |
| *** icey has joined #openstack-nova | 06:10 | |
| icey | hey - any chance of getting a review on https://review.opendev.org/c/openstack/nova/+/761824/ ? It got a +1 from Zuul a while back but has been sitting in limbo since February | 06:11 |
|---|---|---|
| *** ralonsoh has joined #openstack-nova | 06:19 | |
| *** ratailor_ has quit IRC | 06:26 | |
| *** ratailor has joined #openstack-nova | 06:29 | |
| *** gyee has quit IRC | 06:29 | |
| *** zzzeek has quit IRC | 06:34 | |
| *** zzzeek has joined #openstack-nova | 06:35 | |
| *** dklyle has quit IRC | 06:51 | |
| *** vishalmanchanda has joined #openstack-nova | 06:54 | |
| *** mkrai_ has joined #openstack-nova | 07:04 | |
| *** mkrai has quit IRC | 07:07 | |
| *** jawad_axd has joined #openstack-nova | 07:16 | |
| *** andrewbonney has joined #openstack-nova | 07:17 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| *** tosky has joined #openstack-nova | 07:22 | |
| *** mkrai_ has quit IRC | 07:27 | |
| *** ratailor_ has joined #openstack-nova | 07:36 | |
| *** ratailor has quit IRC | 07:39 | |
| lyarwood | icey: Apologies, I'll take a look now | 07:43 |
| *** priteau has joined #openstack-nova | 07:44 | |
| bauzas | morning folks | 07:46 |
| * bauzas starts his specs review day | 07:47 | |
| bauzas | if folks want me to look at some spec, ping me | 07:47 |
| *** benj_ has quit IRC | 07:52 | |
| *** openstackgerrit has joined #openstack-nova | 07:53 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/790660 | 07:53 |
| *** benj_ has joined #openstack-nova | 07:56 | |
| openstackgerrit | Merged openstack/nova-specs master: Allow migrating PMEM's data https://review.opendev.org/c/openstack/nova-specs/+/785563 | 07:58 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/wallaby: Add regression test for bug #1928063 https://review.opendev.org/c/openstack/nova/+/792902 | 08:01 |
| openstack | bug 1928063 in OpenStack Compute (nova) "SEV enabled instance unable to hard reboot" [Medium,In progress] https://launchpad.net/bugs/1928063 - Assigned to Lee Yarwood (lyarwood) | 08:01 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: Add regression test for bug #1928063 https://review.opendev.org/c/openstack/nova/+/792903 | 08:03 |
| openstack | bug 1928063 in OpenStack Compute (nova) "SEV enabled instance unable to hard reboot" [Medium,In progress] https://launchpad.net/bugs/1928063 - Assigned to Lee Yarwood (lyarwood) | 08:03 |
| *** jawad_axd has quit IRC | 08:04 | |
| *** lucasagomes has joined #openstack-nova | 08:04 | |
| *** jawad_axd has joined #openstack-nova | 08:05 | |
| *** benj_ has quit IRC | 08:09 | |
| icey | lyarwood: thanks, it's appreciated! I'm hopimng to get it into Ubuntu, and the cloud archive, for Queens (and, incidentally, Rocky, to ensure that upgrades don't break) | 08:10 |
| lyarwood | ack np elod ^ if you could also review icey's change above that would be great | 08:11 |
| *** vishalmanchanda has quit IRC | 08:11 | |
| icey | lyarwood: sadly, not quite my change, but I do get to hear the desire from the patch's author to get it in :) | 08:11 |
| *** coreycb has quit IRC | 08:12 | |
| elod | lyarwood icey : yes, it is on my todo for a long time, I'll try to review it now :S | 08:13 |
| lyarwood | many thanks :) | 08:13 |
| icey | elod: thanks! there's a cherry-pick to Queens as well, it currently has a -1 because it wasn't picked from Rocky; I'd appreciate advice ont he best path forward on that one as well :) | 08:14 |
| *** mnasiadka has quit IRC | 08:17 | |
| *** coreycb has joined #openstack-nova | 08:17 | |
| *** whoami-rajat_ has joined #openstack-nova | 08:19 | |
| *** vishalmanchanda has joined #openstack-nova | 08:22 | |
| *** mkrai_ has joined #openstack-nova | 08:23 | |
| *** mnasiadka has joined #openstack-nova | 08:26 | |
| openstackgerrit | Wenping Song proposed openstack/nova-specs master: Repropose vGPU support spec https://review.opendev.org/c/openstack/nova-specs/+/780452 | 08:31 |
| *** dtantsur has quit IRC | 08:34 | |
| *** benj_ has joined #openstack-nova | 08:36 | |
| *** martinkennelly has joined #openstack-nova | 08:38 | |
| *** dtantsur has joined #openstack-nova | 08:39 | |
| *** ociuhandu has joined #openstack-nova | 08:41 | |
| *** whoami-rajat_ is now known as whoami-rajat | 08:42 | |
| *** stephenfin has left #openstack-nova | 08:43 | |
| *** stephenfin has joined #openstack-nova | 08:46 | |
| stephenfin | lyarwood: have you ever used the 'cinder attachment-*' commands? | 08:51 |
| *** sapd1 has joined #openstack-nova | 08:51 | |
| *** ociuhandu has quit IRC | 08:51 | |
| stephenfin | particularly 'cinder attachment-create' | 08:51 |
| lyarwood | yes | 08:51 |
| lyarwood | oh | 08:51 |
| lyarwood | well only to reserve the attachment | 08:51 |
| lyarwood | never with an actual connector | 08:51 |
| *** benj_ has quit IRC | 08:51 | |
| stephenfin | yeah, I'm not sure how that's supposed to work | 08:51 |
| lyarwood | yeah providing an actual connector is awkward as hell AFAICT | 08:52 |
| *** ociuhandu has joined #openstack-nova | 08:52 | |
| stephenfin | So it would seem. The docs suggest that you don't need to provide much of it and nova and cinder will fill in the blanks between them | 08:52 |
| stephenfin | but I can't get it to move from attaching to attached. Not sure if it's me or my DevStack deploy or what | 08:53 |
| stephenfin | the cinder docs are significantly worse than even the nova docs :( | 08:53 |
| lyarwood | oh there's a final API call to cinder to complete the attachment | 08:54 |
| lyarwood | that moves it from attaching to attached | 08:54 |
| lyarwood | I'm not sure if there's a command for that in cinderclient tbh | 08:54 |
| lyarwood | https://docs.openstack.org/api-ref/block-storage/v3/index.html#complete-attachment | 08:55 |
| * stephenfin wonders what good is attachment-create so :D | 08:55 | |
| stephenfin | oh, that exists | 08:55 |
| stephenfin | cinder attachment-complete | 08:55 |
| lyarwood | cool, use that and it should move | 08:55 |
| lyarwood | well create on it's own is useful when we just need to reserve (create without a connector) I guess | 08:56 |
| lyarwood | but if you want to do the entire workflow it's a PITA at the moment | 08:56 |
| lyarwood | a higher level command that does all three would be cool | 08:56 |
| lyarwood | create, update and complete | 08:56 |
| lyarwood | but then that doesn't actually attach it on the nova side | 08:56 |
| *** sapd1 has quit IRC | 08:57 | |
| lyarwood | so it's only going to be useful in terms of mapping a volume to a host | 08:57 |
| *** dtantsur has quit IRC | 08:57 | |
| *** dtantsur has joined #openstack-nova | 08:57 | |
| *** sapd1 has joined #openstack-nova | 08:58 | |
| stephenfin | aha, complete did the trick alright | 08:59 |
| stephenfin | I assume there's no way to trigger things on the nova side without invoking the whole flow though? | 08:59 |
| * stephenfin apologises for his lack of knowledge in this area | 08:59 | |
| lyarwood | not at the moment, this is why I've wanted to write some nova-manage commands so we can update our BDMs with the attachment_id and connection_info you've just created here | 09:00 |
| lyarwood | for SHUTDOWN instances etc | 09:00 |
| lyarwood | and/or to have nova-manage orchestrate the entire flow to refresh things | 09:00 |
| lyarwood | stephenfin: https://docs.openstack.org/nova/latest/reference/attach-volume.html FWIW | 09:04 |
| stephenfin | lyarwood: is it fair to say that 'cinder attachment-create' doesn't really need to be a user-facing command so? | 09:06 |
| stephenfin | it's a machine API, effectively | 09:06 |
| stephenfin | context being we're implementing this in OSC and I'm trying to decide if we should since I can't figure out how to use it | 09:06 |
| *** benj_ has joined #openstack-nova | 09:06 | |
| lyarwood | As I said above, I've used it a few times just to reserve things so I wouldn't want to remove it unless I had another command | 09:07 |
| lyarwood | for context, we reserve attachments when shelving | 09:07 |
| lyarwood | and in the past when that has failed for whatever reason I've found it useful to create fresh attachments and update Nova's BDMs | 09:07 |
| lyarwood | so without a nova-manage or cinder-manage command that could do the same I think there's still a place for it | 09:08 |
| stephenfin | so perhaps keep it but have a big warning that you probably don't want to use it unless you know what you're doing? | 09:08 |
| lyarwood | Yup fair | 09:08 |
| *** ociuhandu has quit IRC | 09:12 | |
| *** sapd1 has quit IRC | 09:17 | |
| *** sapd1 has joined #openstack-nova | 09:20 | |
| *** sapd1 has quit IRC | 09:25 | |
| *** ociuhandu has joined #openstack-nova | 09:28 | |
| openstackgerrit | Tobias Urdin proposed openstack/nova master: When Nova-Api runs behind the load balancer or Reverse proxy, Loadbalancer IP is getting logged in nova_api.log instead of end user source ip by RequestLog https://review.opendev.org/c/openstack/nova/+/786766 | 09:33 |
| *** ociuhandu has quit IRC | 09:33 | |
| *** ociuhandu has joined #openstack-nova | 09:35 | |
| openstackgerrit | Wenping Song proposed openstack/nova-specs master: Repropose vGPU support spec https://review.opendev.org/c/openstack/nova-specs/+/780452 | 09:39 |
| *** ociuhandu has quit IRC | 09:40 | |
| sean-k-mooney | stephenfin: given we have started adding admin commands for nova i would personally add the attachment manament commands to osc | 09:52 |
| sean-k-mooney | stephenfin: in that same vain if you are adding commands i have been meaning to add the ablitiy to list, create, activate and delete neutron port bindings | 09:53 |
| sean-k-mooney | stephenfin: lyarwood also going back to cinder attachments you can use cinder standalone. i had an isci volume mounted on my gaming pc at home | 09:54 |
| sean-k-mooney | if i remeber correctly i had to use the cinder clinet instead of osc to do that | 09:54 |
| sean-k-mooney | so if i could avoid cinder clint next time that would be nice | 09:54 |
| lyarwood | yeah that's fair, I almost think we need another osc command capable of pulling the hosts connector JSON blob togther that we can feed into it | 09:55 |
| lyarwood | something like | 09:55 |
| sean-k-mooney | like the tooling we have around bdm creation | 09:56 |
| lyarwood | openstack volume attachment create --connector $(openstack volume connector fetch) --volume $volume-id | 09:56 |
| sean-k-mooney | ya that would be nice | 09:56 |
| lyarwood | but that would mean calling into os-brick | 09:57 |
| sean-k-mooney | cant you get the connector info form cinder api | 09:57 |
| lyarwood | the volume connector is a dict detailing the local hosts attributes like IQNs etc | 09:58 |
| lyarwood | Nova just asks os-brick to generate it | 09:58 |
| sean-k-mooney | ah right | 09:59 |
| lyarwood | that's passed to c-api during attachment create and/or update to then map the volume to that specific host | 09:59 |
| sean-k-mooney | cinder client has flags for all the fields | 09:59 |
| sean-k-mooney | e.g. --iqn ectra | 09:59 |
| lyarwood | I'm not sure if it has all the fields but tbh it would just be cleaner to have a blob generated by another command we can then pass in | 09:59 |
| sean-k-mooney | for the nova usecase maybe | 10:00 |
| sean-k-mooney | but for standalone you need both | 10:00 |
| lyarwood | for true standalone yeah that's fair | 10:00 |
| lyarwood | where you don't want os-brick on the host | 10:00 |
| sean-k-mooney | https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-create | 10:01 |
| lyarwood | but tbh with os-brick we could even connect the volumes | 10:01 |
| sean-k-mooney | so the iqn is --initiator | 10:01 |
| sean-k-mooney | you can also se the host/ip exctra | 10:01 |
| sean-k-mooney | andn if you pass --connect | 10:01 |
| sean-k-mooney | it generates teh connoctor info | 10:02 |
| sean-k-mooney | other then a lack of docsn which i was ment to go write ... teh cinder cli parmaters seams to work well | 10:02 |
| lyarwood | right my point was that the connector dict we get from os-brick can contain alot more info that the options there | 10:03 |
| sean-k-mooney | oh ok | 10:03 |
| lyarwood | that's just the basics for iSCSI etc | 10:03 |
| lyarwood | it doesn't cover FC | 10:04 |
| lyarwood | NVMeOF etc | 10:04 |
| sean-k-mooney | well the initiator is generic it does not have to be an iscsi one but ya i get your point not everything is exposed | 10:04 |
| sean-k-mooney | osc could call into os-brick | 10:04 |
| sean-k-mooney | but may os-brick should call the sdk | 10:05 |
| sean-k-mooney | and we sould move the code for the connector generation | 10:05 |
| sean-k-mooney | then osc can call the sdk without needing os-brick | 10:05 |
| sean-k-mooney | lyarwood: i think osc currently calls python-client | 10:06 |
| sean-k-mooney | *cinder-client | 10:06 |
| sean-k-mooney | does cinder clinet depend on os-brick for its local-attch command | 10:06 |
| sean-k-mooney | ah no | 10:08 |
| sean-k-mooney | thats packaged seperatly in python-brick-cinderclient-ext | 10:08 |
| sean-k-mooney | https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/use-cinder-without-nova.html | 10:08 |
| *** masayukig has quit IRC | 10:08 | |
| *** tinwood has quit IRC | 10:08 | |
| *** ttx has quit IRC | 10:08 | |
| *** DinaBelova has quit IRC | 10:08 | |
| *** rm_work has quit IRC | 10:08 | |
| *** masayukig has joined #openstack-nova | 10:09 | |
| *** tinwood has joined #openstack-nova | 10:09 | |
| *** ttx has joined #openstack-nova | 10:09 | |
| *** rm_work has joined #openstack-nova | 10:09 | |
| *** DinaBelova has joined #openstack-nova | 10:09 | |
| *** lyarwood has quit IRC | 10:12 | |
| *** k_mouza has joined #openstack-nova | 10:12 | |
| *** pmannidi has quit IRC | 10:12 | |
| *** fungi has quit IRC | 10:13 | |
| *** arne_wiebalck has quit IRC | 10:13 | |
| *** janno has quit IRC | 10:13 | |
| *** gary_perkins_ has quit IRC | 10:13 | |
| *** mugsie has quit IRC | 10:13 | |
| *** fungi has joined #openstack-nova | 10:14 | |
| *** arne_wiebalck has joined #openstack-nova | 10:14 | |
| *** janno has joined #openstack-nova | 10:14 | |
| *** gary_perkins_ has joined #openstack-nova | 10:14 | |
| *** mugsie has joined #openstack-nova | 10:14 | |
| *** benj_ has quit IRC | 10:14 | |
| *** mkrai_ has quit IRC | 10:14 | |
| *** ratailor_ has quit IRC | 10:14 | |
| *** ralonsoh has quit IRC | 10:14 | |
| *** slaweq has quit IRC | 10:14 | |
| *** ricolin has quit IRC | 10:14 | |
| *** openstackgerrit has quit IRC | 10:14 | |
| *** f0o has quit IRC | 10:14 | |
| *** dansmith has quit IRC | 10:14 | |
| *** bhagyashris has quit IRC | 10:14 | |
| *** gryf has quit IRC | 10:14 | |
| *** ociuhandu has joined #openstack-nova | 10:16 | |
| *** benj_ has joined #openstack-nova | 10:16 | |
| *** mkrai_ has joined #openstack-nova | 10:16 | |
| *** openstackgerrit has joined #openstack-nova | 10:16 | |
| *** ratailor_ has joined #openstack-nova | 10:16 | |
| *** ralonsoh has joined #openstack-nova | 10:16 | |
| *** slaweq has joined #openstack-nova | 10:16 | |
| *** ricolin has joined #openstack-nova | 10:16 | |
| *** f0o has joined #openstack-nova | 10:16 | |
| *** dansmith has joined #openstack-nova | 10:16 | |
| *** bhagyashris has joined #openstack-nova | 10:16 | |
| *** gryf has joined #openstack-nova | 10:16 | |
| *** benj_ has quit IRC | 10:16 | |
| *** mnaser has quit IRC | 10:16 | |
| *** rnoriega_ has quit IRC | 10:16 | |
| *** bnemec has quit IRC | 10:16 | |
| *** icey has quit IRC | 10:16 | |
| *** amotoki has quit IRC | 10:16 | |
| *** mjturek has quit IRC | 10:16 | |
| *** mdbooth has quit IRC | 10:16 | |
| *** k_mouza has quit IRC | 10:17 | |
| *** ganso has quit IRC | 10:17 | |
| *** benj_ has joined #openstack-nova | 10:17 | |
| *** pmannidi has joined #openstack-nova | 10:18 | |
| *** dasp has quit IRC | 10:18 | |
| *** CeeMac has quit IRC | 10:18 | |
| *** jrosser has quit IRC | 10:18 | |
| *** fnordahl has quit IRC | 10:18 | |
| *** lemko has quit IRC | 10:18 | |
| *** supamatt has quit IRC | 10:18 | |
| *** hoonetorg has quit IRC | 10:18 | |
| *** markmcclain has quit IRC | 10:18 | |
| *** Dmitrii-Sh has quit IRC | 10:18 | |
| *** sean-k-mooney has quit IRC | 10:18 | |
| *** ebbex has quit IRC | 10:18 | |
| *** smcginnis has quit IRC | 10:18 | |
| *** purplerbot has quit IRC | 10:18 | |
| *** andreaf has quit IRC | 10:18 | |
| *** mloza has quit IRC | 10:18 | |
| *** Corwin has quit IRC | 10:18 | |
| *** dtantsur has quit IRC | 10:18 | |
| *** iurygregory has quit IRC | 10:18 | |
| *** tobiash has quit IRC | 10:18 | |
| *** lifeless has quit IRC | 10:18 | |
| *** Anticimex has quit IRC | 10:18 | |
| *** vishalmanchanda has quit IRC | 10:18 | |
| *** zzzeek has quit IRC | 10:18 | |
| *** owalsh has quit IRC | 10:18 | |
| *** hemna has quit IRC | 10:18 | |
| *** eandersson has quit IRC | 10:18 | |
| *** Jeffrey4l has quit IRC | 10:18 | |
| *** johanssone has quit IRC | 10:18 | |
| *** tobias-urdin has quit IRC | 10:18 | |
| *** pots has quit IRC | 10:18 | |
| *** obre has quit IRC | 10:18 | |
| *** dtantsur has joined #openstack-nova | 10:20 | |
| *** vishalmanchanda has joined #openstack-nova | 10:20 | |
| *** zzzeek has joined #openstack-nova | 10:20 | |
| *** owalsh has joined #openstack-nova | 10:20 | |
| *** lemko has joined #openstack-nova | 10:20 | |
| *** hoonetorg has joined #openstack-nova | 10:20 | |
| *** supamatt has joined #openstack-nova | 10:20 | |
| *** fnordahl has joined #openstack-nova | 10:20 | |
| *** jrosser has joined #openstack-nova | 10:20 | |
| *** CeeMac has joined #openstack-nova | 10:20 | |
| *** dasp has joined #openstack-nova | 10:20 | |
| *** markmcclain has joined #openstack-nova | 10:20 | |
| *** iurygregory has joined #openstack-nova | 10:20 | |
| *** Dmitrii-Sh has joined #openstack-nova | 10:20 | |
| *** sean-k-mooney has joined #openstack-nova | 10:20 | |
| *** hemna has joined #openstack-nova | 10:20 | |
| *** eandersson has joined #openstack-nova | 10:20 | |
| *** Jeffrey4l has joined #openstack-nova | 10:20 | |
| *** ebbex has joined #openstack-nova | 10:20 | |
| *** tobiash has joined #openstack-nova | 10:20 | |
| *** Corwin has joined #openstack-nova | 10:20 | |
| *** mloza has joined #openstack-nova | 10:20 | |
| *** andreaf has joined #openstack-nova | 10:20 | |
| *** purplerbot has joined #openstack-nova | 10:20 | |
| *** smcginnis has joined #openstack-nova | 10:20 | |
| *** johanssone has joined #openstack-nova | 10:20 | |
| *** lifeless has joined #openstack-nova | 10:20 | |
| *** Anticimex has joined #openstack-nova | 10:20 | |
| *** tobias-urdin has joined #openstack-nova | 10:20 | |
| *** obre has joined #openstack-nova | 10:20 | |
| *** pots has joined #openstack-nova | 10:20 | |
| *** icey has joined #openstack-nova | 10:21 | |
| *** mnaser has joined #openstack-nova | 10:21 | |
| *** rnoriega_ has joined #openstack-nova | 10:21 | |
| *** bnemec has joined #openstack-nova | 10:21 | |
| *** mjturek has joined #openstack-nova | 10:21 | |
| *** amotoki has joined #openstack-nova | 10:21 | |
| *** mdbooth has joined #openstack-nova | 10:21 | |
| *** mnasiadka has quit IRC | 10:21 | |
| *** priteau has quit IRC | 10:21 | |
| *** ChanServ has quit IRC | 10:21 | |
| *** martinkennelly has quit IRC | 10:21 | |
| *** coreycb has quit IRC | 10:21 | |
| *** links has quit IRC | 10:21 | |
| *** jawad_axd has quit IRC | 10:21 | |
| *** boxiang_ has quit IRC | 10:21 | |
| *** damien_r has quit IRC | 10:21 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761824 | 10:24 |
| *** ganso has joined #openstack-nova | 10:24 | |
| *** martinkennelly has joined #openstack-nova | 10:24 | |
| *** mnasiadka has joined #openstack-nova | 10:24 | |
| *** coreycb has joined #openstack-nova | 10:24 | |
| *** jawad_axd has joined #openstack-nova | 10:24 | |
| *** priteau has joined #openstack-nova | 10:24 | |
| *** links has joined #openstack-nova | 10:24 | |
| *** boxiang_ has joined #openstack-nova | 10:24 | |
| *** damien_r has joined #openstack-nova | 10:24 | |
| *** ChanServ has joined #openstack-nova | 10:24 | |
| *** weber.freenode.net sets mode: +o ChanServ | 10:24 | |
| *** ociuhandu has quit IRC | 10:24 | |
| *** trident has quit IRC | 10:27 | |
| *** ianw has quit IRC | 10:27 | |
| *** yankcrime has quit IRC | 10:27 | |
| *** trident has joined #openstack-nova | 10:27 | |
| *** ianw has joined #openstack-nova | 10:27 | |
| *** ociuhandu has joined #openstack-nova | 10:29 | |
| *** k_mouza has joined #openstack-nova | 10:29 | |
| *** Luzi has joined #openstack-nova | 10:34 | |
| *** ociuhandu has quit IRC | 10:37 | |
| *** ociuhandu has joined #openstack-nova | 10:37 | |
| openstackgerrit | Hemanth N proposed openstack/nova stable/queens: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761825 | 10:44 |
| openstackgerrit | Hemanth N proposed openstack/nova stable/queens: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761825 | 10:47 |
| *** mkrai_ has quit IRC | 10:48 | |
| *** whoami-rajat has quit IRC | 10:49 | |
| *** lyarwood has joined #openstack-nova | 10:54 | |
| gibi | bauzas: I've left questions in https://review.opendev.org/c/openstack/nova-specs/+/792796 feel free to ping me if you want real time discussion | 10:56 |
| *** smcginni1 has joined #openstack-nova | 10:57 | |
| *** supamatt has quit IRC | 10:57 | |
| *** smcginnis has quit IRC | 10:57 | |
| *** Corwin has quit IRC | 10:57 | |
| *** Corwin has joined #openstack-nova | 10:58 | |
| *** hoonetorg has quit IRC | 10:58 | |
| *** Dmitrii-Sh has quit IRC | 11:00 | |
| *** Dmitrii-Sh8 has joined #openstack-nova | 11:00 | |
| *** markmcclain has quit IRC | 11:02 | |
| sean-k-mooney | lyarwood: i responded to the open question on https://review.opendev.org/c/openstack/nova-specs/+/783806 but have not done a rereview but i removed my -1 since you adress my previous concerns | 11:02 |
| sean-k-mooney | lyarwood: ill try an review that again today | 11:02 |
| sean-k-mooney | lyarwood: are you currently wroking on a new reviewion? | 11:02 |
| sean-k-mooney | *revision | 11:02 |
| lyarwood | sean-k-mooney: I was trying to but tbh I'm totally distracted waiting for a phone call from the hospital, I'll wait for your review and respin it later today if that's okay | 11:03 |
| lyarwood | /me has been waiting for the call since 8:30am, damn NHS. | 11:03 |
| *** lemko7 has joined #openstack-nova | 11:05 | |
| *** ociuhandu has quit IRC | 11:05 | |
| *** lemko has quit IRC | 11:06 | |
| *** mloza has quit IRC | 11:06 | |
| *** lemko7 is now known as lemko | 11:06 | |
| sean-k-mooney | lyarwood: ok i dont know if i have any real changes that i want you to make there are thing we discussed previosly but i expect any comment i add to be relitvly minor | 11:08 |
| lyarwood | ack kk | 11:08 |
| *** sapd1 has joined #openstack-nova | 11:10 | |
| *** sapd1_x has quit IRC | 11:11 | |
| *** jawad_ax_ has joined #openstack-nova | 11:12 | |
| *** dtantsur is now known as dtantsur|brb | 11:12 | |
| *** k_mouza has quit IRC | 11:13 | |
| *** k_mouza_ has joined #openstack-nova | 11:13 | |
| *** jawad_axd has quit IRC | 11:15 | |
| *** sapd1 has quit IRC | 11:16 | |
| *** sapd1 has joined #openstack-nova | 11:17 | |
| *** ociuhandu has joined #openstack-nova | 11:18 | |
| *** ociuhandu has quit IRC | 11:18 | |
| *** whoami-rajat has joined #openstack-nova | 11:35 | |
| *** masterpe has joined #openstack-nova | 11:35 | |
| *** ociuhandu has joined #openstack-nova | 11:40 | |
| *** masterpe has left #openstack-nova | 11:46 | |
| *** masterpe has joined #openstack-nova | 11:48 | |
| *** ociuhandu has quit IRC | 11:58 | |
| *** ociuhandu has joined #openstack-nova | 12:04 | |
| masterpe | Hi, In train when I use the command ./nova-manage db archive_deleted_rows with --before it will a long time before it completed the command. Even tho I use a --max_rows 1000. Without the --before option it is a lot faster. When I do a SHOW INDEX FROM nova.instances; on the database I think I don't see any indexes on the deleted_at column. | 12:05 |
| *** tosky has quit IRC | 12:08 | |
| sean-k-mooney | masterpe: --before is intedned to be run requently as in daily | 12:08 |
| *** ociuhandu has quit IRC | 12:09 | |
| sean-k-mooney | so typicaly it would be used to delete rows over say 90 days | 12:09 |
| sean-k-mooney | and you would do that daily so its only removeing 1 days worth at a time | 12:09 |
| *** damien_r has quit IRC | 12:10 | |
| masterpe | Without before it is doing everything? | 12:10 |
| sean-k-mooney | if you dont set anything i think so | 12:11 |
| sean-k-mooney | its slighly painful to do the initall catch up but once you get it to the time period you want to retain --before is generally our recomended way to do this | 12:12 |
| *** ociuhandu has joined #openstack-nova | 12:22 | |
| *** ratailor_ has quit IRC | 12:31 | |
| *** ociuhandu has quit IRC | 12:34 | |
| bauzas | gibi: sorry, I was afk | 12:35 |
| sean-k-mooney | bauzas: im making my way through your mdev spec currently too by the way | 12:35 |
| sean-k-mooney | mainly grammer nits/word chocices so far nothing technical | 12:36 |
| bauzas | sean-k-mooney: okay, then we could be discussing about this spec once you look at it | 12:36 |
| sean-k-mooney | actully just got to a technical point | 12:36 |
| sean-k-mooney | i do not belive we should limit the resouce class | 12:36 |
| sean-k-mooney | and i do not belive we should have a singel generic mdev class | 12:37 |
| sean-k-mooney | we should support Custom_ resouce classes | 12:37 |
| sean-k-mooney | im undecieded if we shoudl supprot standard ones or not | 12:37 |
| masterpe | But I see that "nova-manage db archive_deleted_rows" command uses a filter on the deleted_at column when using the --before option. But on that column is no index. | 12:38 |
| *** yankcrime has joined #openstack-nova | 12:38 | |
| bauzas | sean-k-mooney: well, then we would need to ask the operator to provide which custom RC they'd like to get | 12:40 |
| sean-k-mooney | bauzas: yes | 12:40 |
| sean-k-mooney | so must remove choices form mdev_class | 12:40 |
| sean-k-mooney | and allow it to be just a sting | 12:40 |
| sean-k-mooney | we can desided to limit it in python to Custom_ if we want too or not | 12:41 |
| gibi | bauzas: no worries. I'm on a call, but I will try to multitask :) | 12:41 |
| *** alex_xu has joined #openstack-nova | 12:41 | |
| *** ociuhandu has joined #openstack-nova | 12:42 | |
| *** ociuhandu has quit IRC | 12:47 | |
| admin0 | lyarwood, do we save any version info or migration info in the nova database .. i have a nova database which is fairly old ..and i need to figure out if its from newton, ocata or pike | 12:53 |
| *** dtantsur|brb is now known as dtantsur | 12:58 | |
| bauzas | gibi: I soft+1d again your PUT RP spec | 12:58 |
| bauzas | gibi: with another API alternative | 12:58 |
| *** ociuhandu has joined #openstack-nova | 12:58 | |
| bauzas | in my mind, QS parameters are just another way to pass attributes without using data | 12:59 |
| bauzas | so, generally those params should be identical to the resource attributes | 12:59 |
| gibi | QS are not part of the object representation, while the body is. So whatever we add to the body we make it part of the object as per REST | 12:59 |
| bauzas | gibi: I proposed to use a sub-resource for this | 13:00 |
| *** sapd1_x has joined #openstack-nova | 13:00 | |
| bauzas | /force is maybe a bad name | 13:00 |
| gibi | resource_providers/{uuid}/force feels like an action API that is also not that restful | 13:01 |
| sean-k-mooney | bauzas: ok just pushed my comments on your spec | 13:02 |
| gibi | bauzas: so you feel that we cannot add param to the QS that is not part of the body? | 13:02 |
| *** sapd1 has quit IRC | 13:02 | |
| sean-k-mooney | admin0: i belive we do have a version for the schema yes | 13:02 |
| sean-k-mooney | admin0: i think there is a nova manage command to get it | 13:03 |
| gibi | admin0: mysql> select * from migrate_version; | 13:03 |
| gibi | +---------------+-------------------------------------------------+---------+ | 13:03 |
| gibi | | repository_id | repository_path | version | | 13:03 |
| gibi | +---------------+-------------------------------------------------+---------+ | 13:03 |
| sean-k-mooney | admin0: nova-manage db version | 13:03 |
| gibi | | nova | /opt/stack/nova/nova/db/sqlalchemy/migrate_repo | 422 | | 13:03 |
| gibi | +---------------+-------------------------------------------------+---------+ | 13:03 |
| gibi | admin0: or wha sean-k-mooney suggessts | 13:03 |
| sean-k-mooney | ^ is teh db table its reading form | 13:04 |
| sean-k-mooney | so if you have a backup you have not loaded gibi's way might be quicker to grep for | 13:04 |
| sean-k-mooney | there is also nova-manage api_db version | 13:05 |
| bauzas | gibi: as I said in my last comment, the current API is not RESTful either way | 13:05 |
| sean-k-mooney | nova-manage db version tells you the cell db version | 13:05 |
| sean-k-mooney | bauzas: the resful way to update the parent uuid woudl be jsut to update it | 13:06 |
| sean-k-mooney | so a put to resource_providers/{uuid} | 13:06 |
| sean-k-mooney | with the uuid updated in the body | 13:06 |
| bauzas | sean-k-mooney: technically, put would be for updating the whole resource | 13:07 |
| sean-k-mooney | yes | 13:07 |
| sean-k-mooney | if you wanted to do a partial update we woudl use patch | 13:07 |
| bauzas | but we're nitpicking | 13:07 |
| sean-k-mooney | but we dont use patch in openstack | 13:07 |
| bauzas | my biggest concern comes to the fact I feel a qs param is just hacky | 13:07 |
| sean-k-mooney | it often comes up but we keep saying no to patch | 13:07 |
| bauzas | just for the idea of "oh shit, we need to do *something*" | 13:08 |
| sean-k-mooney | bauzas: well i was orginaly suggesting just allowing this by defult | 13:08 |
| *** ociuhandu has quit IRC | 13:08 | |
| bauzas | sean-k-mooney: I proposed three alternatives : one be PATCH, one be adding a new HTTP header | 13:08 |
| bauzas | and the last being to have a specific subresource | 13:09 |
| sean-k-mooney | im not conviced we need to have a flag of any kind to opt into it but im not against having one | 13:09 |
| bauzas | sean-k-mooney: see my comments on PS2, I was thinking like you | 13:09 |
| bauzas | but looks like we agreed at the PTG to sanity check this | 13:09 |
| sean-k-mooney | ya the header occured to me but not sure we have precendent for that | 13:09 |
| sean-k-mooney | we did | 13:09 |
| sean-k-mooney | i did not have stong feels for or against added a sanity check with a slight one for not doign that | 13:10 |
| sean-k-mooney | ill go read your comment and catch up | 13:10 |
| gibi | I'm on the side of either not add a check (but we agreed on the PTG to add it) or if we add then for me the extra parameters is better fits to the QS than to header or body or subresource | 13:11 |
| bauzas | gibi: hence my +1 :) | 13:11 |
| sean-k-mooney | bauzas: if we were to have a subresouce i actully woudl go with PUT /resource-providers/{uuid}/uuid/{new uuid} | 13:11 |
| bauzas | gibi: I don't wanna hold on this, but I don't appreciate the QS param outcome | 13:12 |
| sean-k-mooney | with an empty body | 13:12 |
| sean-k-mooney | sorry | 13:13 |
| sean-k-mooney | put /resource-providers/{uuid}/parent_uuid/{parent} | 13:13 |
| sean-k-mooney | with an empty body | 13:13 |
| sean-k-mooney | that basically woudl work like patch without using it | 13:13 |
| bauzas | this sounds quite good to me | 13:14 |
| bauzas | but again, I'm just one | 13:14 |
| sean-k-mooney | you also have to expressly opt into that url | 13:14 |
| gibi | bauzas: sorry I don't get why the current PUT is not RESTful | 13:16 |
| gibi | you have to PUT all the attributes | 13:16 |
| gibi | that you POSTed before | 13:16 |
| bauzas | I guess the universe entropy is probably smaller than a discussion of 3 engineers about specing an API endpoint | 13:16 |
| bauzas | count a 4th and you'll wait for the big crunch | 13:17 |
| bauzas | gibi: it's not RESTful in the sense you can't update all the attributes as of now | 13:17 |
| gibi | bauzas: you have restriction of the value of a field yes, but you have to still list all the fields in the PUT | 13:18 |
| sean-k-mooney | bauzas: i think gibi was saying you would get teh current state update teh parent field and then put back the updated state | 13:18 |
| gibi | sean-k-mooney: yes | 13:18 |
| gibi | for me this is the restful put | 13:18 |
| sean-k-mooney | if you do a update of the full resouce yes | 13:18 |
| bauzas | this | 13:18 |
| bauzas | if you just PUT a name, this isn't RESTful | 13:19 |
| gibi | but the current PUT is not partial and I'm not suggesting any partial PUT | 13:19 |
| gibi | either | 13:19 |
| bauzas | and if you PUT a parent uuid which is not None while the resource parent is not None, then it's not RESFul either | 13:19 |
| bauzas | because you depend on the state of the resource | 13:20 |
| bauzas | but we're bikeshedding I guess | 13:20 |
| sean-k-mooney | gibi: right i think bauzas was assuming you wanted to change that | 13:20 |
| sean-k-mooney | bauzas: were you assuming gibi was suggeting using put for a partial update | 13:20 |
| gibi | so today, PUT takes 3 paramters, name, uuid, and parent_uuid. After my change PUT takes the same parameters | 13:20 |
| bauzas | actually, I was wrong | 13:21 |
| gibi | just the value restriciotn of parent_uuid is relaxed | 13:21 |
| sean-k-mooney | gibi: maybe we coudl put json exmaple into the spec to make that clear | 13:21 |
| bauzas | name and uuid aren't optional | 13:21 |
| bauzas | but parent_uuid is | 13:21 |
| gibi | I assume parent_uuid field is not optional, it just allow taking the value of None | 13:21 |
| bauzas | gibi: tbc, I'd be OK with a parent_uuid QS param name | 13:21 |
| sean-k-mooney | gibi: right that is how i think of that too | 13:22 |
| gibi | the API ref says optional, but I'm not sure that it is just due to the microversion dependency | 13:22 |
| bauzas | but I'm not OK with some allow_reparenting name | 13:22 |
| sean-k-mooney | the parent is an intrinic part of the resouce but None is a valid and default value | 13:22 |
| sean-k-mooney | bauzas: you can change the name whenever you like | 13:23 |
| sean-k-mooney | oh sorry | 13:23 |
| sean-k-mooney | misread that | 13:23 |
| sean-k-mooney | bauzas: you ment you are not ok with teh allow_reparenting query arg name | 13:23 |
| sean-k-mooney | bauzas: i would prefer allow_reparenting over force | 13:24 |
| *** ricolin has quit IRC | 13:28 | |
| *** ociuhandu has joined #openstack-nova | 13:28 | |
| *** ricolin has joined #openstack-nova | 13:30 | |
| openstackgerrit | Merged openstack/nova-specs master: Set minversion of tox to 3.18.0 https://review.opendev.org/c/openstack/nova-specs/+/791975 | 13:33 |
| sean-k-mooney | gibi: ill get back to the reparenting spec later today. left my toughts on this as comments but i have no stong feeling agaisnt the current query arg approch nesssarly. | 13:33 |
| gibi | bauzas, sean-k-mooney: thanks | 13:33 |
| *** ociuhandu has quit IRC | 13:35 | |
| *** artom has joined #openstack-nova | 13:36 | |
| *** ociuhandu has joined #openstack-nova | 13:38 | |
| *** ociuhandu has quit IRC | 13:43 | |
| masterpe | I have placed indexes on the deleted_at column on all the tables on the databases nova, nova_api and nova_cell0, this speeds this nova-manage db archive_deleted_rows with multiple minutes per run. First it was 19 minutes now it is 19 seconds. | 13:46 |
| *** Luzi has quit IRC | 13:46 | |
| masterpe | So I think this is a bug. | 13:46 |
| gibi | masterpe: please file a bug. and feel free to propose a patch that adds the index | 13:47 |
| masterpe | Is bugs.launchpad.net/ still used as bug tracker? | 13:47 |
| gibi | for nova, yes | 13:47 |
| gibi | https://bugs.launchpad.net/nova/+filebug | 13:48 |
| masterpe | thanks | 13:48 |
| gibi | thank you for reporting | 13:49 |
| sean-k-mooney | masterpe: sound like a quick fix. althoguh we dont always index everythin that shoudl be indexed by default | 13:49 |
| sean-k-mooney | deleted should already be indexed yes | 13:50 |
| sean-k-mooney | deleted_at woudl only be imporant if you are using --before | 13:50 |
| sean-k-mooney | i assume we just missed that usage chagne wehn we added --before | 13:50 |
| sean-k-mooney | prior to that deleted_at would be very rarely used in queired as a column we filtered on | 13:51 |
| *** ociuhandu has joined #openstack-nova | 13:51 | |
| sean-k-mooney | i think the soft delete code would have been the only thing that checked it really bar a direct user request for deleted instances | 13:52 |
| *** ociuhandu has quit IRC | 13:56 | |
| *** mlavalle has joined #openstack-nova | 13:59 | |
| masterpe | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1929563 | 13:59 |
| openstack | Launchpad bug 1929563 in OpenStack Compute (nova) "Missing database index deleted_at column" [Undecided,New] | 13:59 |
| masterpe | Do I only need to add the new indexes to ./db/sqlalchemy/models.py ? | 14:00 |
| sean-k-mooney | masterpe: you also need to add a migration to add the indexs i think | 14:02 |
| masterpe | I need to remember how to make commits, so probably it is easyer one someone else does that ;) | 14:03 |
| sean-k-mooney | well if you want to start with the model change then you could | 14:04 |
| sean-k-mooney | and we could pick it up but stephenfin is also looking at swaping out sqlalcamey_migrate for alembic this cycle | 14:04 |
| sean-k-mooney | so how you do that will cahnge soon ish | 14:04 |
| *** ociuhandu has joined #openstack-nova | 14:05 | |
| masterpe | That will get into the master branch and not Train ? | 14:05 |
| gibi | bauzas: you are right, PUT /resource_providers/{uuid} is not RESTFul as it allows partial update today. This is very unfortunate :/ | 14:06 |
| gibi | the parent_uuid not need to be passed, and the uuid field is not part of the body just part of the url | 14:06 |
| gibi | the current PUT is very pretty close to PATCH now | 14:07 |
| *** jangutter_ has joined #openstack-nova | 14:17 | |
| *** ociuhandu has quit IRC | 14:19 | |
| *** ociuhandu has joined #openstack-nova | 14:20 | |
| *** jangutter has quit IRC | 14:20 | |
| gibi | stephenfin: we need you on https://review.opendev.org/c/openstack/nova-specs/+/783827 | 14:23 |
| gibi | :0 | 14:23 |
| gibi | :) | 14:23 |
| stephenfin | I'm needed? How wonderful | 14:23 |
| * stephenfin looks | 14:24 | |
| gibi | it is your spec where sean-k-mooney has some comments making it pending | 14:24 |
| *** k_mouza_ has quit IRC | 14:27 | |
| *** k_mouza has joined #openstack-nova | 14:28 | |
| *** jangutter has joined #openstack-nova | 14:32 | |
| sean-k-mooney | stephenfin: my main question is do we need OS-EXT-SRV-ATTR:hostname anymore and shoudl we just have hostname instead | 14:36 |
| *** jangutter_ has quit IRC | 14:36 | |
| stephenfin | ah, that point | 14:36 |
| stephenfin | I did see that and thought I had replied | 14:37 |
| stephenfin | I'm easy. Changing it is more work, both on the server side (code and docs) and client (SDK, OSC, novaclient) side | 14:37 |
| sean-k-mooney | we could adress that point in a follow up which is why im currently +1 | 14:37 |
| stephenfin | but on the other hand, it's certainly a saner response | 14:38 |
| sean-k-mooney | it would be but i think it would be a nice UX improvemnt | 14:38 |
| sean-k-mooney | changing it in the futrue woudl need another micorversion bump | 14:38 |
| sean-k-mooney | so basically i we ever want to change it i think it would be nice to do it now ihn this change | 14:38 |
| stephenfin | what about the rest of those prefixed options though? | 14:39 |
| stephenfin | if we do one, shouldn't we do them all | 14:39 |
| stephenfin | ? | 14:39 |
| sean-k-mooney | well im open to that also | 14:39 |
| sean-k-mooney | im ok wiht elevating my +1 to a +w and we can dicuss that in a followup patch? although it really was just hostname that i wanted to chagne in this case | 14:40 |
| sean-k-mooney | some of the other prefixed fiels are admin only | 14:40 |
| sean-k-mooney | so im not really sure it there is merrit in renaming those form a UX point of view | 14:41 |
| *** mloza has joined #openstack-nova | 14:43 | |
| stephenfin | I would be tempted to do all or nothing, personally | 14:44 |
| sean-k-mooney | let me look at the list. not sure what the otehrs are fully | 14:44 |
| sean-k-mooney | the prefix was form when we had extentions | 14:44 |
| gibi | is this prefixing comes from the time when we had api plugins? | 14:44 |
| sean-k-mooney | to show that they were optional and could not be replied on to be in all clouds | 14:44 |
| gmann | sean-k-mooney: stephenfin left comment there. agree to improve the name | 14:44 |
| sean-k-mooney | gibi: yes | 14:45 |
| gibi | so they can be relied on now | 14:45 |
| sean-k-mooney | yep | 14:45 |
| gibi | so we can drop all the prefix :) | 14:45 |
| gibi | lets do that in a separate microversion. :) | 14:45 |
| sean-k-mooney | we can. that sound like a seperate mini spec so | 14:45 |
| gmann | in past we left these prefix as it is not worth to do with all prefix rename but as in some other microvesion change we should definitely improver them | 14:45 |
| gmann | gibi: we abandon that idea in past | 14:45 |
| gmann | let me check the spec | 14:46 |
| gibi | bottom line, I'm OK with stephenfin's spec as is, and we can have a separate discussion if and when we drop all the prefixes | 14:46 |
| stephenfin | yes, I do think we should treat this separately. Microversions are (relatively) cheap | 14:46 |
| sean-k-mooney | ok in that case ill upgrade my +1 to +w and refrence this irc log | 14:47 |
| sean-k-mooney | unless gibi gmann you want me to wait? | 14:47 |
| openstackgerrit | masterpe proposed openstack/nova master: WIP: Fixes missing indexes on column deleted_at at the different tables. https://review.opendev.org/c/openstack/nova/+/792989 | 14:47 |
| gibi | sean-k-mooney: go for it | 14:47 |
| gibi | but wait for gmann | 14:47 |
| gibi | as he said he reads the spec | 14:47 |
| gmann | gibi: stephenfin sean-k-mooney item#3 in https://etherpad.opendev.org/p/nova-api-cleanup | 14:48 |
| *** jangutter_ has joined #openstack-nova | 14:48 | |
| sean-k-mooney | ok in that case ill leave teh +w to gmann | 14:48 |
| sean-k-mooney | ah yes i tought that had come up before now | 14:49 |
| gibi | gmann: ack. microversion make us safe against breaking clients. | 14:49 |
| gibi | gmann: but I'm fine both ways about the prefixes | 14:49 |
| gibi | gmann: are you OK with the rest of stephenfin's proposal? | 14:49 |
| *** dklyle has joined #openstack-nova | 14:49 | |
| gmann | gibi: you mean stephenfin spec or proposal of removing all prefix with one microversion ? | 14:50 |
| gibi | spec | 14:50 |
| gibi | the prefix removal an be taken separately from the spec | 14:50 |
| sean-k-mooney | gmann: i think gibi was refering to stephenfin current spec | 14:51 |
| *** jangutter has quit IRC | 14:51 | |
| gibi | yepp, this one https://review.opendev.org/c/openstack/nova-specs/+/783827 | 14:52 |
| gibi | I want to get it merged :) | 14:52 |
| gibi | gmann: is it OK to you we we skip the renaming now/ | 14:52 |
| gibi | ? | 14:52 |
| gmann | gibi: stephenfin sean-k-mooney sorry, give me some time, stick in two meeting in parallel | 14:55 |
| *** sapd1_x has quit IRC | 14:55 | |
| sean-k-mooney | gmann: no worries, we have time | 14:56 |
| gibi | gmann: sure, no worries | 14:56 |
| *** sapd1_x has joined #openstack-nova | 14:56 | |
| bauzas | sean-k-mooney: can't see your comments on https://review.opendev.org/c/openstack/nova-specs/+/792796/1/specs/xena/approved/generic-mdevs.rst | 14:57 |
| bauzas | replying for the moment to gibi | 14:57 |
| sean-k-mooney | you cant? | 14:57 |
| sean-k-mooney | can you refersh? | 14:58 |
| *** macz_ has joined #openstack-nova | 14:58 | |
| sean-k-mooney | that is strang i cant see you replies | 14:58 |
| sean-k-mooney | i see i my comments on mulitipel browser even when im logged out so i assume you are looking at a cached copy? | 14:59 |
| gibi | I see sean-k-mooney's comments too | 15:00 |
| sean-k-mooney | if you go to the commit message and go back tothe spec you should see them too | 15:00 |
| *** ociuhandu has quit IRC | 15:05 | |
| *** ociuhandu has joined #openstack-nova | 15:05 | |
| *** sapd1_x has quit IRC | 15:05 | |
| *** sapd1_x has joined #openstack-nova | 15:06 | |
| bauzas | sean-k-mooney: gibi: oh, now I see those | 15:08 |
| *** ozzzo has joined #openstack-nova | 15:08 | |
| bauzas | weirdo | 15:08 |
| gmann | gibi: stephenfin sean-k-mooney on https://review.opendev.org/c/openstack/nova-specs/+/783827 | 15:08 |
| ozzzo | after upgrading compute hosts from centos 7-8, I can't migrate VMs to them. This is the error in the scheduler log: | 15:09 |
| ozzzo | https://paste.ubuntu.com/p/5vqpFnb4ZZ/ | 15:09 |
| ozzzo | When I look at the hypervisors, I see different cpu_info on the upgraded ones | 15:09 |
| sean-k-mooney | bauzas: i just left 1 or 30 :) most are minor nits/word choice issues though so dont be scared by the number | 15:09 |
| gmann | gibi: stephenfin sean-k-mooney it seems it is related to decide on prefix as we will end up different name for single field in request and response. ``hostname`` in request and ``OS-EXT-SRV-ATTR:hostname`` in response | 15:09 |
| ozzzo | the old ones have "| cpu_info | {"vendor": "Intel", "model": "Skylake-Server-IBRS", " | 15:09 |
| ozzzo | the new ones have "| cpu_info | {"arch": "x86_64", "model": "Cascadelake-Server-noTSX", " | 15:09 |
| ozzzo | is that why VMs won't migrate? How can I change it? | 15:09 |
| *** ociuhandu has quit IRC | 15:09 | |
| gmann | if we want to leave the prefix in response now then OS-EXT-SRV-ATTR:hostname in request make it consistent | 15:09 |
| sean-k-mooney | ozzzo: i assume you are not pinning to a specific cpu_model today | 15:10 |
| ozzzo | I have this in the flavors: | properties | hw:cpu_policy='dedicated', hw:cpu_sockets='1', hw:numa_nodes='1' | | 15:10 |
| sean-k-mooney | ozzzo: if the live migration was rejected it typically means the destination is missign a cpu flag that is present on the source node but not the dest | 15:10 |
| gmann | and removing all prefix together has negative feedback from team users in train PTG so i am not 100% sure if that can happen - http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005824.html | 15:10 |
| sean-k-mooney | ozzzo: the cpu model is set in the nova.conf not the flavor | 15:11 |
| sean-k-mooney | ozzzo: i would gess the issue is TSX | 15:11 |
| ozzzo | I kept the same nova.conf | 15:11 |
| ozzzo | Is it possible that centos 8 recognizes the CPU differently from 7? | 15:11 |
| sean-k-mooney | did it have cascade lake cpus before the upgrade | 15:12 |
| sean-k-mooney | if so then host-model would have selected the clost named model which could have been Skylake-Server-IBRS | 15:12 |
| gibi | gmann: so you see an asymetry between the request and the response due the the prefix in the response. | 15:12 |
| sean-k-mooney | but as part of the upgrade if you cahnge the cpu/libvirt verion then it could now have a new cascadelake model | 15:13 |
| gibi | gmann: that is a good point | 15:13 |
| gmann | gibi: yeah | 15:13 |
| gibi | gmann: but if changing all the prefixes was rejected then can we still change prefixes one by one? | 15:13 |
| ozzzo | sean-k-mooney: I think so, yes, but I don't think I changed anything in the config during the upgrade | 15:13 |
| ozzzo | I just removed computes from the cluster, installed centos 8, and then deployed kolla-ansible from the same playbook | 15:13 |
| sean-k-mooney | ozzzo: can you checnk what you have in the libvirt section | 15:13 |
| gmann | gibi: I think yes, if we improve that API/filed somewhere then it is very valid improvement. let me find example, i think we did it in past | 15:14 |
| sean-k-mooney | ozzzo: oh you are using kolla | 15:14 |
| sean-k-mooney | ozzzo: can you show me your config override for nova | 15:14 |
| ozzzo | this is my /etc/kolla/config/nova.conf: | 15:15 |
| ozzzo | https://paste.ubuntu.com/p/p5rCWRZt7n/ | 15:15 |
| gibi | gmann: OK. then stephenfin would you be against removing the prefix from the hostname in the response to make the req/rsp symmetric? | 15:16 |
| sean-k-mooney | ozzzo: ok then its using the default since cpu_mode and cpu_model is not set which is host-model | 15:16 |
| ozzzo | this is /etc/kolla/config/nova/nova-compute.conf: | 15:16 |
| stephenfin | gibi: no issues with it from my end | 15:16 |
| ozzzo | https://paste.ubuntu.com/p/nTW2s45Xxk/ | 15:16 |
| sean-k-mooney | ozzzo: so the model that is used depend on the qemu version | 15:16 |
| gibi | stephenfin: then lets go that direction. gmann convinced me | 15:16 |
| sean-k-mooney | ozzzo: so to fix this you need to pin the cpu model in the nova.conf | 15:17 |
| gibi | sorry for the noise | 15:17 |
| sean-k-mooney | ozzzo: ill show you and exmple one sec | 15:17 |
| ozzzo | ok ty | 15:17 |
| sean-k-mooney | ozzzo: http://paste.openstack.org/show/805703/ assume all you servers are the same you can do that intially to allow migration to work and then later you can change it | 15:18 |
| ozzzo | sean-k-mooney: I'll try that, ty! | 15:19 |
| sean-k-mooney | ozzzo: there is also a replacemnt for cpu_model called cpu_models which is a preferentially ordered list. | 15:19 |
| sean-k-mooney | so you could do cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX | 15:19 |
| ozzzo | ok | 15:20 |
| *** jawad_ax_ has quit IRC | 15:21 | |
| gmann | gibi: stephenfin if it was different attribute then I would have fine in either way but it is same param we are ending in different name | 15:21 |
| sean-k-mooney | ozzzo: http://paste.openstack.org/show/805704/ the cpu model you had enabled before however had tsx enabled | 15:22 |
| sean-k-mooney | ozzzo: so the reason its failing is likely because by default TSX is now disabled in the upstream kerenl | 15:22 |
| sean-k-mooney | ozzzo: so you likely will need to re enabled it. | 15:22 |
| ozzzo | righton | 15:22 |
| sean-k-mooney | with TSX=on | 15:23 |
| ozzzo | you mean, with "cpu_models" ? | 15:23 |
| sean-k-mooney | no on the grub commandline | 15:23 |
| ozzzo | oic ok | 15:23 |
| sean-k-mooney | in which case i would actully use cpu_models=Cascadelake-Server-noTSX,Skylake-Server-IBRS | 15:23 |
| sean-k-mooney | that way new vms will boot without using tsx but you shoudl still be able to migrate your exsting instnaces | 15:24 |
| ozzzo | ok I'll try that | 15:24 |
| gibi | fyi, the ovs -> ovn default switch have been merged (again) | 15:28 |
| gibi | https://review.opendev.org/c/openstack/devstack/+/791436 | 15:28 |
| *** jangutter_ has quit IRC | 15:33 | |
| *** jangutter has joined #openstack-nova | 15:33 | |
| *** ociuhandu has joined #openstack-nova | 15:36 | |
| *** ociuhandu has quit IRC | 15:48 | |
| gibi | nova weekly meeting starts in 10 minutes on #openstack-meeting-3 | 15:49 |
| gibi | yes it is on Tuesday now :) | 15:49 |
| *** ociuhandu has joined #openstack-nova | 15:50 | |
| sean-k-mooney | ah yes it is | 15:52 |
| sean-k-mooney | ill just open that tab now just incase | 15:52 |
| gibi | :) | 15:56 |
| *** lucasagomes has quit IRC | 15:58 | |
| *** sapd1 has joined #openstack-nova | 16:01 | |
| *** tosky has joined #openstack-nova | 16:14 | |
| *** ociuhandu has quit IRC | 16:17 | |
| *** ociuhandu has joined #openstack-nova | 16:17 | |
| *** jangutter_ has joined #openstack-nova | 16:24 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Address TODO https://review.opendev.org/c/openstack/nova/+/792363 | 16:26 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add 'hw:vif_multiqueue_enabled' flavor extra spec https://review.opendev.org/c/openstack/nova/+/792356 | 16:26 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document virtio-net multiqueue https://review.opendev.org/c/openstack/nova/+/792362 | 16:26 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Move 'hw:pmu', 'hw_pmu' parsing to nova.virt.hardware https://review.opendev.org/c/openstack/nova/+/792364 | 16:26 |
| *** jangutter has quit IRC | 16:27 | |
| *** ociuhandu has quit IRC | 16:27 | |
| *** ociuhandu has joined #openstack-nova | 16:29 | |
| *** ociuhandu has quit IRC | 16:30 | |
| *** ociuhandu has joined #openstack-nova | 16:30 | |
| *** mlavalle has quit IRC | 16:32 | |
| *** ociuhandu has quit IRC | 16:35 | |
| *** k_mouza has quit IRC | 16:37 | |
| *** gyee has joined #openstack-nova | 16:39 | |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| *** k_mouza has joined #openstack-nova | 16:43 | |
| *** k_mouza has quit IRC | 16:44 | |
| *** k_mouza has joined #openstack-nova | 16:45 | |
| *** k_mouza has quit IRC | 16:48 | |
| *** k_mouza has joined #openstack-nova | 16:49 | |
| stephenfin | bauzas: I know it's the end of your day, but could you pick up https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open again tomorrow? | 16:50 |
| stephenfin | (well past the end of your day, in fact) | 16:50 |
| bauzas | stephenfin: I could | 16:50 |
| bauzas | stephenfin: thanks for working on it, btw. | 16:51 |
| stephenfin | nw | 16:51 |
| sean-k-mooney | dansmith: gibi lyarwood i think i see the isseu with lyarwood gate bug | 16:51 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12243 | 16:51 |
| sean-k-mooney | May 24 10:23:19.977224 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 27 {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:263}} | 16:52 |
| sean-k-mooney | May 24 10:23:24.981354 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=107012) __log_wakeup /usr/lib/python3/dist-packages/ovs/poller.py:248}} | 16:52 |
| sean-k-mooney | May 24 10:23:24.982747 ubuntu-focal-inap-mtl01-0024794054 nova-compute[107012]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: idle 5001 ms, sending inactivity probe {{(pid=107012) run /usr/lib/python3/dist-packages/ovs/reconnect.py:103}} | 16:52 |
| sean-k-mooney | the gap in logs lins up to those ovs timeouts | 16:52 |
| *** ociuhandu has joined #openstack-nova | 16:53 | |
| sean-k-mooney | so it look like this is possible causeing the compute agent to lock up | 16:54 |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| sean-k-mooney | we dont need to do that polling in os-vif for nova use | 16:54 |
| sean-k-mooney | that functionality in ovsdbapp is there for neutron to monitor the ovs bridge for the addtion and removal of ports | 16:55 |
| sean-k-mooney | so i wonder if we can just turn that off | 16:55 |
| dansmith | ah good catch | 16:56 |
| sean-k-mooney | ill take a look at ovsdbapp and see if we can change that if not maybe i can kick it into a differen thread | 16:56 |
| *** ociuhandu has quit IRC | 16:58 | |
| sean-k-mooney | actully its in a thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L60-L91 but its not a pthread | 16:58 |
| sean-k-mooney | since we monkey patch in nova | 16:58 |
| *** alex_xu has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:03 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | 17:03 |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | 17:03 |
| openstackgerrit | Ghanshyam proposed openstack/nova-specs master: Allow project admin to list hypervisors https://review.opendev.org/c/openstack/nova-specs/+/793011 | 17:03 |
| stephenfin | gibi: gmann: bauzas: sean-k-mooney: proposed a follow-up on that spec so we'd have a better history of the changes ^ | 17:04 |
| stephenfin | I think it's okay to send the bulk of it on its way now? | 17:04 |
| stephenfin | i.e. +w | 17:05 |
| stephenfin | I'll rework the patches tomorrow | 17:05 |
| stephenfin | and review some specs, since I only got to one today :( | 17:05 |
| gmann | stephenfin: ack. | 17:05 |
| gmann | stephenfin: dansmith gibi sean-k-mooney spec for project admin to create server on host (what we discussed in PTG) - https://review.opendev.org/c/openstack/nova-specs/+/793011 | 17:06 |
| *** sapd1 has quit IRC | 17:06 | |
| sean-k-mooney | oh white space errors remind me again that a -3 right :) ill review it shortly it look pretty short and too the point | 17:07 |
| sean-k-mooney | actully im being summoned to go for a walk so when i get back | 17:07 |
| gmann | ok, I was searching whitespace in stephenfin spec :P | 17:09 |
| sean-k-mooney | hehe no in yours https://review.opendev.org/c/openstack/nova-specs/+/793011/1/specs/xena/approved/allow-project-admin-list-hypervisors.rst#39 | 17:10 |
| gmann | yeah :) | 17:10 |
| gmann | stephenfin: sean-k-mooney gibi bauzas I have +W this https://review.opendev.org/c/openstack/nova-specs/+/783827/2 and +2 on follow up 793010 | 17:11 |
| gibi | gmann: ack on both count | 17:12 |
| openstackgerrit | Merged openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | 17:23 |
| *** hoonetorg has joined #openstack-nova | 17:25 | |
| *** andrewbonney has quit IRC | 17:36 | |
| ozzzo | sean-k-mooney: I set cpu_models = Cascadelake-Server-noTSX,Skylake-Server-IBRS in nova.conf and also: GRUB_CMDLINE_LINUX="crashkernel=auto tsx=on" in /etc/default/grub, and rebooted, and then redeployed the compute, but I still get "ERROR oslo_service.service nova.exception.InvalidCPUInfo: Configured CPU model: Skylake-Server-IBRS is not compatible with host CPU." in nova-compute.log | 17:42 |
| *** links has quit IRC | 17:46 | |
| openstackgerrit | Ghanshyam proposed openstack/nova-specs master: Allow project admin to list hypervisors https://review.opendev.org/c/openstack/nova-specs/+/793011 | 17:48 |
| *** stephenfin has quit IRC | 18:00 | |
| *** mlavalle has joined #openstack-nova | 18:03 | |
| ozzzo | looks like I might need to use grubby; reading man page | 18:09 |
| ozzzo | sean-k-mooney: that fixed it, ty! | 18:17 |
| *** ralonsoh has quit IRC | 18:25 | |
| *** mnaser has quit IRC | 18:30 | |
| *** rnoriega_ has quit IRC | 18:30 | |
| *** bnemec has quit IRC | 18:30 | |
| *** icey has quit IRC | 18:30 | |
| *** amotoki has quit IRC | 18:30 | |
| *** mjturek has quit IRC | 18:30 | |
| *** mdbooth has quit IRC | 18:30 | |
| *** mnaser has joined #openstack-nova | 18:30 | |
| *** icey has joined #openstack-nova | 18:30 | |
| *** mdbooth has joined #openstack-nova | 18:31 | |
| *** amotoki has joined #openstack-nova | 18:31 | |
| *** bnemec has joined #openstack-nova | 18:31 | |
| *** rnoriega_ has joined #openstack-nova | 18:33 | |
| ozzzo | sean-k-mooney: that fix it in the sense that I can now start the nova-compute service with the new settings, but VMs still refuse to migrate with the same " Host filter ignoring hosts:" error | 18:34 |
| ozzzo | sean-k-mooney: when I check the hypervisor now, cpu_info has changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" | 18:59 |
| ozzzo | so it still isn't matching "Skylake-Server-IBRS" | 18:59 |
| ozzzo | Trying "cpu_models=Skylake-Server-IBRS" in "/etc/kolla/config/nova.conf" | 19:00 |
| ozzzo | that didn't make a difference. | 19:07 |
| ozzzo | I looked inside the container, and I see "cpu_models = Skylake-Server-IBRS" in /etc/nova/nova.conf | 19:08 |
| ozzzo | but the hypervisor show still says "Cascadelake-Server" | 19:09 |
| ozzzo | what am I missing? | 19:09 |
| *** k_mouza has quit IRC | 19:09 | |
| *** k_mouza has joined #openstack-nova | 19:10 | |
| *** k_mouza has quit IRC | 19:15 | |
| masterpe | I had the same | 19:32 |
| masterpe | at my place the commands says Cascadelake-Server-noTSX. This happend when I upgrade to qemu(1:4.2-3ubuntu6.13) or libvirt(6.0.0-0ubuntu8.8) | 19:37 |
| masterpe | To solve this I restarted the instances and accepted that I can not live migrate those instances. | 19:38 |
| ozzzo | that won't work for me. When I upgrade production, we have to be able to migrate instances there | 19:41 |
| masterpe | ozzzo: I can Imagine, I was able to live migrate from a machine that was no upgraded to a node that is upgraded. | 19:44 |
| ozzzo | masterpe: did you try the grub trick with tsx=on? | 19:44 |
| masterpe | ozzzo: no I did not. | 19:45 |
| ozzzo | I did that, and that allowed me to start nova_compute with "cpu_models=Skylake-Server-IBRS" but it looks like I'm still missing something | 19:45 |
| masterpe | But that instance further live migraten to a second machine was not possible. | 19:45 |
| masterpe | but Skylake- is older then Cascadelake | 19:48 |
| masterpe | What does /usr/bin/virsh -r capabilities|grep \<model\>|head -1|cut -d '>' -f2|cut -d '<' -f1 | 19:49 |
| masterpe | say ? | 19:49 |
| masterpe | and libvirt and qemu version are you and also what processor do you have? | 19:50 |
| ozzzo | my computes don't seem to have virsh installed | 19:51 |
| ozzzo | I'm running kolla | 19:52 |
| ozzzo | everything is in containers | 19:52 |
| ozzzo | Skylake-Server-IBRS is how centos 7 recognized my compute CPUs | 19:52 |
| ozzzo | centos 8 recognizes as "Cascadelake-Server-noTSX" | 19:53 |
| ozzzo | and after the grub trick, it recogizes as "Cascadelake-Server" | 19:53 |
| masterpe | I use the "Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz" as CPU. | 19:54 |
| ozzzo | my computes have Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 19:55 |
| masterpe | thanks | 19:55 |
| masterpe | Is this helpfull? https://bugzilla.redhat.com/show_bug.cgi?id=1678381 ? | 19:59 |
| openstack | bugzilla.redhat.com bug 1678381 in libvirt "[RFE] - libvirt support for cascade lake cpu - Fast Train" [Unspecified,Closed: errata] - Assigned to jdenemar | 19:59 |
| masterpe | and or https://bugzilla.redhat.com/show_bug.cgi?id=1839926 ? | 20:01 |
| openstack | bugzilla.redhat.com bug 1839926 in libvirt "5 more -noTSX x86 CPU models are newly introduced" [Unspecified,Closed: errata] - Assigned to jdenemar | 20:01 |
| ozzzo | masterpe: it looks like the first bug was 8.1 and I'm on 8.3 | 20:01 |
| ozzzo | reading 2nd | 20:01 |
| *** k_mouza has joined #openstack-nova | 20:04 | |
| *** k_mouza has quit IRC | 20:10 | |
| ozzzo | masterpe: I think the different CPU might be related to the 2nd one, but I think sean's grub trick should have worked around that | 20:25 |
| ozzzo | I must be missing something | 20:25 |
| masterpe | I saw on https://bugzilla.redhat.com/show_bug.cgi?id=1839926#c7 the hle and rtm set to disabled. | 20:27 |
| openstack | bugzilla.redhat.com bug 1839926 in libvirt "5 more -noTSX x86 CPU models are newly introduced" [Unspecified,Closed: errata] - Assigned to jdenemar | 20:27 |
| ozzzo | Do I need to set something in cpu_model_extra_flags? | 20:36 |
| ozzzo | I see it mentioned here: https://docs.openstack.org/nova/train/admin/configuration/hypervisor-kvm.html | 20:36 |
| *** slaweq has quit IRC | 20:43 | |
| *** elod has quit IRC | 21:16 | |
| *** elod has joined #openstack-nova | 21:18 | |
| *** pmannidi has quit IRC | 21:21 | |
| *** pmannidi has joined #openstack-nova | 21:22 | |
| *** k_mouza has joined #openstack-nova | 21:36 | |
| *** k_mouza has quit IRC | 21:41 | |
| *** ociuhandu has joined #openstack-nova | 22:07 | |
| *** ociuhandu has quit IRC | 22:11 | |
| *** vishalmanchanda has quit IRC | 22:13 | |
| *** admin0 has left #openstack-nova | 22:18 | |
| *** tobiash has quit IRC | 22:24 | |
| *** tosky has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | 23:23 |
| *** macz_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!