openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Nova object changes for forbidden aggregates request filter https://review.opendev.org/671072 | 00:03 |
---|---|---|
openstackgerrit | Shilpa Devharakar proposed openstack/nova master: DB API changes to get non-matching aggregates from metadata https://review.opendev.org/671074 | 00:05 |
openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Add a new request filter to isolate aggregates https://review.opendev.org/671075 | 00:05 |
efried | dansmith, melwitt: If you're still around, this iteration of that fixture passed zuul: https://review.opendev.org/#/c/676495/ | 00:06 |
efried | would be nice to have it merged by morning to unwedge https://review.opendev.org/#/c/676457/ | 00:06 |
openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Enable request filter isolate_aggregates https://review.opendev.org/667952 | 00:06 |
*** ociuhandu has joined #openstack-nova | 00:16 | |
*** markvoelker has joined #openstack-nova | 00:24 | |
melwitt | efried: looking | 00:26 |
efried | thanks melwitt | 00:28 |
melwitt | guh, I don't know what I'm looking at here | 00:31 |
efried | melwitt: I can splain if you got a minute | 00:31 |
melwitt | ok. I see mordred said earlier that he thinks it looks good, if he could +1 it again that would be helpful too | 00:32 |
efried | Without any fixturing, unit and functional tests fail because many of them hit paths that try to use placement, and placement now uses sdk. | 00:36 |
efried | When we try to load up sdk, it normally does some discovery for us, things like checking the service catalog and asking for version documents. | 00:36 |
efried | And in unit/functional tests, we don't care about any of that. | 00:36 |
efried | The problem is that the discovery stuff is done through ksa - and we have other things (glance, cinder, ironic...) that rely on those ksa-isms either not being mocked at all, or being mocked a certain way in their own tests. | 00:36 |
efried | So I can't just mock away the ksa stuff - which was my first PS on this fix - because then those all puke. | 00:36 |
efried | So instead I'm mocking the thing in the sdk that triggers that discovery. And I'm making it do almost all of what it would do anyway - just skipping that one part that hits ksa. | 00:36 |
efried | The linked patch shows the code I cribbed from. | 00:36 |
melwitt | ok, thanks | 00:37 |
efried | As before, this is just making sure that unit/func tests that only care about a default no-op-ish ksa Adapter actually get something that looks like an Adapter. | 00:38 |
efried | The problem is that sdk is a moving target right now, which is why I had to change it; but hopefully this version will stick for a little while longer until we can get real fixtures from sdk. | 00:38 |
melwitt | yeah, I was looking through wondering why mocking get_endpoint didn't work | 00:39 |
*** gyee has quit IRC | 00:40 | |
melwitt | like, you're skipping around the get_endpoint call in the new mock, but yet mocking get_endpoint itself didn't work. and I wasn't seeing what/how this avoids the problem | 00:40 |
efried | Right - if you look at the PS1 failures, you can see where things like glance and cinder are relying on get_endpoint to "work" and be un-mocked. | 00:40 |
efried | So whereas mocking get_endpoint gets around the paths where *placement* tries to create an Adapter, it makes those other ones puke. | 00:41 |
melwitt | efried: could you have mocked get_endpoint conditionally if self.service_type is placement to get the same thing or no? | 00:44 |
efried | melwitt: I think then in tests that hit placement *and* one of the other things, it would still get mocked - possibly at an unpredictable time in the test | 00:46 |
melwitt | ok. sorry for all the questions, this is confusing to me | 00:47 |
efried | the right solution, long-term, and mordred is working on this, is for openstacksdk to publish fixtures that we can just import and use. Because openstacksdk will always know what it's doing internally and how. | 00:47 |
efried | This - like the previous iteration of the fixture that mocked Adapter.get_major_api_version - is just a stopgap to let us move forward until that happens. | 00:48 |
efried | and it's only affecting unit/func tests. The real proof that sdk is working for us is in the fact that all the tempest tests - which all use placement, which uses sdk - pass. | 00:49 |
melwitt | yeah, I get that part, I just didn't want to fast approve something I had zero understanding of. I'll look at it some more and if I can grok it, I'll send it on | 00:49 |
melwitt | thanks for explaining all that | 00:50 |
efried | Thanks for looking. | 00:50 |
efried | I think alex_xu might be arriving on the scene soon, so if you prefer not to +W it, he might be able to do so while we sleep. | 00:50 |
*** markvoelker has quit IRC | 00:51 | |
*** markvoelker has joined #openstack-nova | 00:52 | |
melwitt | it might just take me a bit longer to get this | 00:52 |
efried | It's not the end of the world if it has to wait for morning. | 00:53 |
efried | It would just be nice to be able to recheck the u-c patch and have it merged by morning so I can move on with the stuff that builds on it. | 00:53 |
melwitt | yeah, I understand. I'll keep looking at it, just need a bit more time | 00:54 |
efried | thanks melwitt | 00:55 |
*** brinzhang has joined #openstack-nova | 00:56 | |
*** hongbin has joined #openstack-nova | 00:58 | |
*** markvoelker has quit IRC | 01:01 | |
*** ociuhandu has quit IRC | 01:02 | |
*** ricolin has joined #openstack-nova | 01:03 | |
*** markvoelker has joined #openstack-nova | 01:05 | |
*** markvoelker has quit IRC | 01:11 | |
*** markvoelker has joined #openstack-nova | 01:16 | |
*** bhagyashris has joined #openstack-nova | 01:25 | |
*** spsurya has joined #openstack-nova | 01:28 | |
*** takashin has joined #openstack-nova | 01:35 | |
openstackgerrit | Dongcan Ye proposed openstack/nova master: Libvirt: Rename _guest_add_memory_balloon https://review.opendev.org/676534 | 01:43 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix 'has_calls' method calls in unit tests https://review.opendev.org/676536 | 01:45 |
*** Tianhao_Hu has joined #openstack-nova | 01:48 | |
openstackgerrit | Mohammed Naser proposed openstack/nova master: cleanup: remove osapi_v21.project_id_regex https://review.opendev.org/676537 | 01:53 |
melwitt | efried: +W (finally) | 01:58 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix 'has_calls' method calls in unit tests https://review.opendev.org/676536 | 02:02 |
yonglihe | melwitt: Hi, I hope the comments address your concern, very nice to have you review that, thank. https://review.opendev.org/#/c/627765/ | 02:06 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: config: remove deprecated checksum options https://review.opendev.org/676539 | 02:14 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: Remove nova.image.download.modules extension point https://review.opendev.org/676540 | 02:29 |
melwitt | yonglihe: I'll try to take a look tomorrow, sorry for the delay | 02:32 |
*** BjoernT has joined #openstack-nova | 02:36 | |
*** BjoernT_ has joined #openstack-nova | 02:43 | |
*** BjoernT has quit IRC | 02:44 | |
*** takashin has left #openstack-nova | 03:01 | |
*** markvoelker has quit IRC | 03:10 | |
openstackgerrit | CHENGLEI proposed openstack/nova stable/pike: Insert the description of the change. https://review.opendev.org/676541 | 03:13 |
*** altlogbot_0 has quit IRC | 03:16 | |
*** altlogbot_2 has joined #openstack-nova | 03:17 | |
*** brinzhang has quit IRC | 03:21 | |
*** brinzhang has joined #openstack-nova | 03:22 | |
*** hongbin has quit IRC | 03:27 | |
*** markvoelker has joined #openstack-nova | 03:55 | |
*** markvoelker has quit IRC | 04:00 | |
*** BjoernT_ has quit IRC | 04:12 | |
*** dave-mccowan has joined #openstack-nova | 04:26 | |
*** ociuhandu has joined #openstack-nova | 04:30 | |
*** dave-mccowan has quit IRC | 04:31 | |
*** ociuhandu has quit IRC | 04:35 | |
*** brinzhang has quit IRC | 04:38 | |
*** brinzhang has joined #openstack-nova | 04:39 | |
*** brinzhang has quit IRC | 04:44 | |
*** brinzhang has joined #openstack-nova | 04:45 | |
*** markvoelker has joined #openstack-nova | 04:50 | |
*** tkajinam has quit IRC | 04:53 | |
*** tkajinam has joined #openstack-nova | 04:54 | |
*** markvoelker has quit IRC | 04:54 | |
*** brinzhang has quit IRC | 04:55 | |
*** brinzhang has joined #openstack-nova | 04:55 | |
*** brinzhang has quit IRC | 04:56 | |
*** tkajinam has quit IRC | 05:00 | |
*** tkajinam has joined #openstack-nova | 05:05 | |
*** Tianhao_Hu has quit IRC | 05:12 | |
*** jaosorior has joined #openstack-nova | 05:16 | |
openstackgerrit | ZhengMa proposed openstack/nova master: Check container_format on instance creation https://review.opendev.org/673407 | 05:17 |
*** tkajinam has quit IRC | 05:31 | |
*** markvoelker has joined #openstack-nova | 05:35 | |
*** markvoelker has quit IRC | 05:41 | |
*** brinzhang has joined #openstack-nova | 05:45 | |
*** Tianhao_Hu has joined #openstack-nova | 05:49 | |
*** mgoddard has quit IRC | 05:54 | |
*** mgoddard has joined #openstack-nova | 06:01 | |
*** tkajinam has joined #openstack-nova | 06:04 | |
*** hongbin has joined #openstack-nova | 06:28 | |
*** hongbin has quit IRC | 06:32 | |
*** tkajinam has quit IRC | 06:49 | |
*** belmoreira has joined #openstack-nova | 06:55 | |
*** tkajinam has joined #openstack-nova | 07:00 | |
*** ricolin_ has joined #openstack-nova | 07:03 | |
*** ricolin has quit IRC | 07:05 | |
*** Tianhao_Hu has quit IRC | 07:06 | |
*** d34dh0r53 has quit IRC | 07:07 | |
*** d34dh0r53 has joined #openstack-nova | 07:10 | |
*** alemgeta has joined #openstack-nova | 07:19 | |
*** jaosorior has quit IRC | 07:23 | |
*** jawad_axd has joined #openstack-nova | 07:26 | |
*** tssurya has joined #openstack-nova | 07:32 | |
*** ricolin_ is now known as ricolin | 07:33 | |
*** helenafm has joined #openstack-nova | 07:40 | |
gibi | sean-k-mooney: thanks for the devstack conf with the mixed sriov ovs setup | 07:52 |
*** ivve has joined #openstack-nova | 07:53 | |
*** boxiang has joined #openstack-nova | 07:58 | |
*** boxiang has quit IRC | 08:00 | |
*** boxiang has joined #openstack-nova | 08:00 | |
gibi | sean-k-mooney: did you set up connectivity between br-ex and enp1s0f1 on the physical host? | 08:11 |
*** jaosorior has joined #openstack-nova | 08:17 | |
*** markvoelker has joined #openstack-nova | 08:20 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass RequestContext to oslo_policy https://review.opendev.org/674038 | 08:24 |
*** markvoelker has quit IRC | 08:25 | |
openstackgerrit | ZhengMa proposed openstack/nova master: Check container_format on instance creation https://review.opendev.org/673407 | 08:28 |
sean-k-mooney | gibi: that config does it automatically | 08:28 |
sean-k-mooney | it add enp1sof1 to the br-ex | 08:28 |
sean-k-mooney | *adds | 08:29 |
sean-k-mooney | i also put the echo to allocate the vf in the local.conf and set the pci whitelist based on the vendor id and procut id of the vf so it automaticl allows the correect vfs | 08:30 |
sean-k-mooney | when stacking compltes you are basiaclly ready to boot a vm with an sriov port and it will get a dhcp adress for the dhcp agent attached to ovs | 08:31 |
sean-k-mooney | i say basiacally ready as cirros may not have your nic driver so you might need to upload a new image first | 08:32 |
*** derekh has joined #openstack-nova | 08:40 | |
*** gibi has quit IRC | 08:42 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies https://review.opendev.org/669181 | 08:45 |
*** tkajinam has quit IRC | 08:47 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies https://review.opendev.org/669181 | 08:49 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-services https://review.opendev.org/645427 | 08:50 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 08:51 |
*** cdent has joined #openstack-nova | 08:52 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | 08:54 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | 08:54 |
*** gibi has joined #openstack-nova | 08:59 | |
gibi | sean-k-mooney: thanks | 09:03 |
*** gibi has quit IRC | 09:05 | |
*** gibi has joined #openstack-nova | 09:06 | |
*** ricolin has quit IRC | 09:07 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/669578 | 09:14 |
gibi | gmann: hi! If you have time, could you give your oppinion in https://review.opendev.org/#/c/675371/ ? | 09:19 |
openstackgerrit | Merged openstack/nova master: Enhance SDK fixture for 0.34.0 https://review.opendev.org/676495 | 09:23 |
*** jawad_axd has quit IRC | 09:29 | |
*** zbr|flu is now known as zbr | 09:33 | |
*** alemgeta has quit IRC | 09:36 | |
gmann | gibi: sure, i will check. | 09:39 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles and mapping in policy base class https://review.opendev.org/645452 | 09:42 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 09:42 |
*** sapd1_x has joined #openstack-nova | 09:45 | |
*** dpawlik has joined #openstack-nova | 09:48 | |
*** bhagyashris has quit IRC | 09:49 | |
*** jangutter has joined #openstack-nova | 09:51 | |
*** hamzy_ has joined #openstack-nova | 09:51 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | 09:54 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | 09:54 |
*** hamzy has quit IRC | 09:54 | |
*** belmoreira has quit IRC | 10:03 | |
*** helenafm has quit IRC | 10:04 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823 | 10:14 |
*** helenafm has joined #openstack-nova | 10:14 | |
openstackgerrit | Guo Jingyu proposed openstack/nova master: define and catch InstanceGuestAgentNotRunning for set-password failure https://review.opendev.org/676667 | 10:19 |
*** hongbin has joined #openstack-nova | 10:22 | |
*** markvoelker has joined #openstack-nova | 10:36 | |
openstackgerrit | Guo Jingyu proposed openstack/nova master: define and catch InstanceAgentNotRunning for set-password failure https://review.opendev.org/676667 | 10:39 |
*** jangutter has quit IRC | 10:41 | |
*** markvoelker has quit IRC | 10:41 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests https://review.opendev.org/676670 | 10:42 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 10:44 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 10:44 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 10:48 |
openstackgerrit | Chris Dent proposed openstack/nova master: Add a "Caveats" section to the eventlet profiling docs https://review.opendev.org/676672 | 10:53 |
sean-k-mooney | what time does matt normally log on? about 1-2 hours form now? | 10:56 |
*** jangutter has joined #openstack-nova | 10:57 | |
stephenfin | sean-k-mooney: We have a conflict with cpu-resources | 10:57 |
sean-k-mooney | i started early today so it feels weird to have been online for a while and not see the us folks about | 10:57 |
sean-k-mooney | we do? | 10:57 |
sean-k-mooney | with what | 10:57 |
stephenfin | There's a workaround option to disable VCPU -> PCPU flavor translation and we've decided to report both PCPUs and VCPUs for compute nodes without explicit CPU pinset configuration | 10:58 |
sean-k-mooney | yes | 10:58 |
sean-k-mooney | i was talking about this yesterday | 10:58 |
stephenfin | If I reshape, I move every allocation for pinned instances on that host from VCPUs to PCPUs | 10:58 |
sean-k-mooney | right | 10:58 |
sean-k-mooney | which is why i raised that workaround option as an issue | 10:59 |
stephenfin | But if you enable the workaround option (so your pinned instances continue to request VCPUs), more pinned instances will land on that host that use VCPUs instead of PCPUs | 10:59 |
sean-k-mooney | yes | 10:59 |
stephenfin | so those two things can't coexist | 10:59 |
sean-k-mooney | yes you cant reshap until you disabel the workaround | 11:00 |
stephenfin | yup, and the workaround needs to be set on every single host, and toggled on every host at the same time | 11:00 |
sean-k-mooney | well currently its only set on the schduler | 11:00 |
stephenfin | compute nodes and controllers (well, scheduler hosts) | 11:00 |
sean-k-mooney | but it would have to be if you were to use | 11:00 |
sean-k-mooney | it to contol the reshape | 11:00 |
stephenfin | I'm tempted to drop the config option, to be honest | 11:01 |
*** jangutter has quit IRC | 11:01 | |
sean-k-mooney | well the translation is on by default | 11:01 |
stephenfin | Yup. I'd rather we didn't support turning it off again | 11:01 |
sean-k-mooney | so im ok with saying that it should nver be set on a new host | 11:01 |
stephenfin | You just need to make sure you upgrade sufficient hosts at once or you'll see degraded scheduling performance due to lack of viable hosts | 11:02 |
stephenfin | *a sufficient number of compute nodes at once | 11:02 |
sean-k-mooney | e.g. you should only use it if and only if you want to first upgrade to train then when all node are on train enable this feature | 11:02 |
stephenfin | so you'd rather keep the config option and make it global? | 11:03 |
sean-k-mooney | we could remove it but it there for host with vms that cant be migrated | 11:03 |
stephenfin | why not? The hosts just have to be updated | 11:03 |
sean-k-mooney | it cant really be global unless you mean set in ever host | 11:03 |
stephenfin | I do. I think that's the only way this could work | 11:03 |
sean-k-mooney | ya | 11:04 |
sean-k-mooney | so if we keep it i think it needs to be set on all host | 11:04 |
sean-k-mooney | and your reshap code needs to check it | 11:04 |
stephenfin | It would need to toggle (a) flavor translation, (b) whether we report PCPUs (since that's a heuristic we rely on to detect if reshapes are needed) and (c) whether we reshape | 11:04 |
stephenfin | Is it worth the effort? | 11:04 |
sean-k-mooney | and we need to document that it must be set too the same value on all hosts | 11:04 |
stephenfin | Yup | 11:04 |
sean-k-mooney | can you support inplacem upgrade without live migration without it | 11:05 |
stephenfin | and turned off again on the controller first | 11:05 |
stephenfin | otherwise the compute nodes will have reshaped everything but the scheduler will still be making requests for VCPUs | 11:06 |
sean-k-mooney | the workflow you be. set workatound on all hosts-> upgrade->disbale workaound on all hosts. and yes it would have to toggel all those things | 11:06 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 11:07 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 11:07 |
stephenfin | "can you support inplacem upgrade without live migration without it" what do you mean? | 11:07 |
sean-k-mooney | in which case we shoudl renable it to disable_pcpus_in_placment or something | 11:07 |
sean-k-mooney | i mean for edge deployment | 11:07 |
sean-k-mooney | where we may not ever be able to cold or live migrate instances | 11:08 |
sean-k-mooney | for those deployments wee need to supprot inplace updates | 11:08 |
sean-k-mooney | and we need to support updating differen edge cells at differnet times | 11:08 |
*** belmoreira has joined #openstack-nova | 11:09 | |
sean-k-mooney | does that make sense? | 11:09 |
*** belmoreira has quit IRC | 11:11 | |
openstackgerrit | YumengBao proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | 11:12 |
stephenfin | sean-k-mooney: Why would I want to live migrate an instance as part of the upgrade though? | 11:12 |
stephenfin | I pull in the latest packages, I restart the controllers followed by compute nodes, the controllers start requesting PCPUs instead of VCPUs and the compute nodes reshape their allocations so pinned instances are consuming PCPUs instead of VCPUs | 11:14 |
sean-k-mooney | you dont but we cant assume that you will upgrade part of the host first. allow them to reshape and then upgrade the rest | 11:14 |
stephenfin | Again, not sure how that's an issue | 11:14 |
*** belmoreira has joined #openstack-nova | 11:14 | |
sean-k-mooney | in edge deployment that is not a thing as we dont have spare hosts and we cant assume we can do a rolling upgrade of subset of hosts within an edge site | 11:14 |
stephenfin | what would they be doing instead of a rolling upgrade? | 11:15 |
sean-k-mooney | the upgrade all nodes within an edge site together and fail over traffic to a different edgesite | 11:15 |
sean-k-mooney | but that does not mean that all edge site will be upstad at the same time. | 11:16 |
stephenfin | Why would that be an issue? | 11:16 |
stephenfin | Oh, that on the other hand, yea | 11:16 |
stephenfin | *yeah | 11:16 |
*** hongbin has quit IRC | 11:16 | |
sean-k-mooney | they do a rolling update at the site level rather then host level at least in some cases | 11:16 |
sean-k-mooney | so for an edge deployment you would have to update all edge sites to disabel the workaround | 11:17 |
stephenfin | Do those different sites share controllers or is there a controller per site? | 11:17 |
sean-k-mooney | i think that depends. i think each site is a different cell but they all share the same schduler | 11:17 |
stephenfin | *scheduler | 11:17 |
stephenfin | then we're kind of stuck whatever way we go | 11:18 |
*** jaosorior has quit IRC | 11:18 | |
stephenfin | if we have a config option, then we need to make sure every site has the config option toggled at the same time or things break | 11:18 |
sean-k-mooney | well the 3 step update with the config option will work | 11:18 |
sean-k-mooney | yes | 11:18 |
stephenfin | well, things don't break but you won't be able to schedule instances to hosts that still have the config option toggled | 11:19 |
sean-k-mooney | you could do it another way | 11:19 |
stephenfin | because they won't be reporting PCPU inventory | 11:19 |
stephenfin | shoot | 11:19 |
sean-k-mooney | we could use host_aggreate metadata to contol it instead | 11:19 |
*** ircuser-1 has quit IRC | 11:21 | |
stephenfin | Hmm, not sure how that helps | 11:21 |
sean-k-mooney | that would requrie the traslation to be moved to a prefilter proably | 11:21 |
sean-k-mooney | it would make the traslation non global. | 11:22 |
sean-k-mooney | its not ideal however | 11:22 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480 | 11:22 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | 11:23 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | 11:23 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823 | 11:23 |
sean-k-mooney | stephenfin: i would be good to get dansmith opipion on how much of an issue the edge deplopment case is likely to be | 11:24 |
stephenfin | Yeah, we need to have a discussion on this. I'll start a mailing list post about it | 11:24 |
sean-k-mooney | even in an edge deployment the n=>n+1 version compatiable still applies so we would expect edge sites to be updated at the same cadance as teh main datacenter | 11:26 |
openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677 | 11:27 |
sean-k-mooney | belmoreira: you work at cern correct? or am i mixing you your irc nic? | 11:27 |
belmoreira | sean-k-mooney: yes I do | 11:28 |
sean-k-mooney | good :) am ye have a hevaily multi cell deployment. how do ye go about upgradeign cells | 11:29 |
sean-k-mooney | do ye do upgrades cell by cell or do ye do it some other way | 11:29 |
sean-k-mooney | im just wondering how long it takes ye to go form all host on version N to all on N+1 and if multiple cells alters that | 11:30 |
sean-k-mooney | i dont know if ye have done a major version update since ye moved to cellsv2 but it might be good input for stephenfin cpu pinning in plamcent issue | 11:31 |
belmoreira | I upgrade all the control plane! nova-api, nova-scheduler, nova-conductor "super", nova-conductor "cells", ... (This has downtime! ) later upgrade compute-nodes | 11:31 |
sean-k-mooney | ok but you do all the contol plane including the cell conductors all in one go | 11:32 |
sean-k-mooney | then upgrade the computes after | 11:32 |
belmoreira | yes | 11:32 |
sean-k-mooney | and would setting a config uption on each host before the upgrade be problematic for you in that case? | 11:32 |
sean-k-mooney | i guess it would be annoying but it should be managable? | 11:33 |
belmoreira | what do you mean? | 11:33 |
sean-k-mooney | for cpu pinning in placment there is currently a workaound config option to disable using the new resouce class | 11:33 |
sean-k-mooney | if we keep it then you would have to set that on all hosts. upgrade to train and only disable it after all host are on train | 11:34 |
openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: Stabilize unshelve notification sample tests https://review.opendev.org/676677 | 11:34 |
belmoreira | that looks reasonable. I don't have a big issue with that | 11:35 |
sean-k-mooney | roughly how long do your upgrades take end to end after you have done contol plane | 11:36 |
belmoreira | I'm still not familiar with the changes for the cpu pinning for train | 11:36 |
sean-k-mooney | would it be days/week or are we talking a few months | 11:36 |
sean-k-mooney | belmoreira: they allow pinned and unpinned guest on the same host and move counting pinned core to placement | 11:37 |
openstackgerrit | Chris Dent proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.opendev.org/623557 | 11:37 |
openstackgerrit | Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558 | 11:37 |
openstackgerrit | Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810 | 11:37 |
belmoreira | in my case the control plane upgrade is less that ~1 hour, then all the compute nodes are upgraded during the next 24h. | 11:38 |
sean-k-mooney | belmoreira: to allow pinned and unpinned guest on the same host we need to have two resouce classes in placement instead of 1 vCPUs(shared) and pCPUs(dedicated) and as part of that we need too update exsiting allcoation to the new type | 11:38 |
sean-k-mooney | cool | 11:38 |
sean-k-mooney | so ya that updating to the new type is what i was discussing with stephenfin | 11:39 |
sean-k-mooney | and what teh config option would be contolling | 11:39 |
sean-k-mooney | stephenfin: weel there is one data point but still good to bring this up on the list and bug dansmith about when he is online | 11:40 |
belmoreira | and what you are suggesting is to keep it disabled (config option) and then only enable when everything is upgraded | 11:40 |
sean-k-mooney | yes | 11:40 |
*** markvoelker has joined #openstack-nova | 11:40 | |
belmoreira | in that case I may suggest that the default of the config is "disabled" just in case the operator is not aware | 11:41 |
sean-k-mooney | you may. we were hoping to have the feature enable by defualt so that it just worked for new deployment but i guess we could handel that in installers | 11:45 |
sean-k-mooney | it would simplify things | 11:46 |
sean-k-mooney | if it was disabel by default | 11:46 |
sean-k-mooney | as you would just upgrade normally | 11:46 |
sean-k-mooney | the set teh config option if you wanted the feature. | 11:47 |
sean-k-mooney | i guess we could also cahge the default in U and remove the option in V | 11:47 |
sean-k-mooney | so it give people 3 cycle to switch instead of 1 which is what we were planning | 11:47 |
sean-k-mooney | stephenfin: ^ thoughts? | 11:48 |
* stephenfin looks | 11:48 | |
stephenfin | Let me finish that email and then I'll think about this | 11:48 |
sean-k-mooney | tl;dr global config pcpus_in_placement=False by default in train change to true in U and remove in V. which completely disables the feature | 11:49 |
sean-k-mooney | once you set it to true you do it globally and cant set it back to false | 11:50 |
sean-k-mooney | in the V release we drop all support fo deploying without it enabled. in U its on by default and train its off by default | 11:50 |
sean-k-mooney | then we modify installer to enable it by defualt for new installs in train | 11:51 |
*** tbachman has quit IRC | 11:52 | |
mnaser | if i am not wrong, it seems like we're installing `psycopg2` in test-requirements.txt for no reason? | 11:58 |
mnaser | i cant find any psycopg2 references except in test-requirements? | 11:58 |
*** jaosorior has joined #openstack-nova | 11:58 | |
sean-k-mooney | it used if you are using postgres via sqlalcambe for you db i think | 11:59 |
stephenfin | Do we still support that? There was talk of dropping it some time back, though cfriesen had objections | 11:59 |
mnaser | right but you'd have to install that yourself (i.e. it shouldn't be inside test-requirements.txt in openstack/nova ?) | 11:59 |
mnaser | oh i found this https://review.opendev.org/#/c/296604/ | 11:59 |
stephenfin | What's the opportunistic testing he refers to? | 12:00 |
mnaser | this is 3 years ago, but i think there's a tc patch saying we dont support postgres. | 12:00 |
sean-k-mooney | we have a db mixing that will use mysql or postgress instead of sqlitght if it avioable i think | 12:01 |
cdent | mnaser: that patch never landed in full | 12:01 |
sean-k-mooney | mnaser: there was discussion or it yes | 12:01 |
mnaser | ah, ok. | 12:01 |
cdent | instead we decided that we wouldn't explicitly disallow it | 12:01 |
mnaser | i'm just annoyed by it because i just had tox fail on me cause i didnt have postgres development headers.. | 12:02 |
cdent | instead projects could try to use sqlachemy in the way it was intended if they like | 12:02 |
sean-k-mooney | mnaser: are you just annoyed by the fact that requires use to install the postgres client and complie it | 12:02 |
sean-k-mooney | and that breaks on stable | 12:02 |
sean-k-mooney | ya | 12:02 |
mnaser | i have to get the postgres client to run tox locally (its inside test-requirements, not requirements) | 12:02 |
sean-k-mooney | we could replacy it with psycopg2-binary | 12:02 |
mnaser | oh thats a reasonable suggestion | 12:03 |
sean-k-mooney | "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.£ | 12:03 |
cdent | mnaser: yeah, in placement we did this: https://review.opendev.org/#/c/671249/ which may be a method for possible hacking around things | 12:03 |
sean-k-mooney | we are using it for testing so i think it makes sense | 12:03 |
mnaser | oh thats a nice workaround too | 12:04 |
mnaser | TIL about sys_platform | 12:04 |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611 | 12:04 |
sean-k-mooney | cdent: so when you develop on macos you dont run all the db test and leave the gate check it | 12:04 |
sean-k-mooney | unfortunetly i dont think we will be allowed do the same on linux | 12:05 |
sean-k-mooney | maybe i should go back ot using windows | 12:05 |
cdent | sean-k-mooney: the db tests are opportunistic so the migrations are testing on sqlite but not msyql/postgresql | 12:05 |
cdent | i'm not suggesting that we do exactly what's done in that patch, rather that there might be _some_ strategy for making those requirements optional | 12:06 |
sean-k-mooney | ya but back to the binary package is there any reason not to jsut use that instead | 12:06 |
sean-k-mooney | cdent: well we could make it a tox siblig too | 12:06 |
sean-k-mooney | rather then have it in test requirements | 12:07 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682 | 12:12 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: config: remove deprecated checksum options https://review.opendev.org/676539 | 12:18 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: Remove nova.image.download.modules extension point https://review.opendev.org/676540 | 12:18 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: requirements: move to using psycopg2-binary https://review.opendev.org/676684 | 12:18 |
*** luksky has joined #openstack-nova | 12:22 | |
stephenfin | cdent: That's easy - 'extras' | 12:25 |
stephenfin | e.g. https://github.com/openstack/oslo.messaging/blob/master/setup.cfg#L22-L29 | 12:25 |
sean-k-mooney | stephenfin ah ya that is what i was tihnking of when i said tox siblings | 12:26 |
sean-k-mooney | but we would have to be able to use tox -e py37[mysql,postgres] or whatever in the upstream gate | 12:27 |
sean-k-mooney | if we could not run the tox env with the extras installed then it wont help | 12:27 |
*** nweinber__ has joined #openstack-nova | 12:28 | |
stephenfin | sean-k-mooney: Not necessarily | 12:28 |
stephenfin | sean-k-mooney: https://tox.readthedocs.io/en/latest/config.html#conf-extras | 12:29 |
sean-k-mooney | the goal is to not need to install postgres header locally to be able to run tox | 12:30 |
sean-k-mooney | but to still enable all the testing with the optional deps in the gate | 12:30 |
sean-k-mooney | so im not sure that actully helps | 12:31 |
*** smcginnis is now known as smcginnis|pto | 12:34 | |
*** nweinber__ has quit IRC | 12:36 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688 | 12:36 |
*** ricolin has joined #openstack-nova | 12:37 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823 | 12:38 |
*** dpawlik has quit IRC | 12:38 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682 | 12:39 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095 | 12:39 |
*** hongbin has joined #openstack-nova | 12:43 | |
*** hongbin has quit IRC | 12:43 | |
*** dpawlik has joined #openstack-nova | 12:50 | |
*** tbachman has joined #openstack-nova | 12:51 | |
*** dpawlik has quit IRC | 12:55 | |
*** eharney has joined #openstack-nova | 12:55 | |
*** tbachman has quit IRC | 13:01 | |
*** helenafm has quit IRC | 13:02 | |
*** BjoernT has joined #openstack-nova | 13:03 | |
*** tssurya has quit IRC | 13:03 | |
*** tssurya has joined #openstack-nova | 13:04 | |
*** tbachman has joined #openstack-nova | 13:08 | |
*** tbachman_ has joined #openstack-nova | 13:14 | |
*** tbachman has quit IRC | 13:15 | |
*** tbachman_ is now known as tbachman | 13:15 | |
*** tbachman has quit IRC | 13:17 | |
*** mriedem has joined #openstack-nova | 13:18 | |
*** tbachman has joined #openstack-nova | 13:19 | |
gmann | melwitt: can you review this. This is followup fix for policy default refresh spec - https://review.opendev.org/#/c/669196/ | 13:20 |
gmann | melwitt: johnthetubaguy also I have prepared the first set of changes and looking for feedback before I proceed on other policy changes. | 13:21 |
gmann | melwitt: johnthetubaguy this and its base patches are the complete set of os-services policy change - https://review.opendev.org/#/c/676688/ and | 13:22 |
gmann | sent the review guide on ML also http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008504.html | 13:22 |
*** noonedeadpunk has left #openstack-nova | 13:28 | |
*** luksky has quit IRC | 13:33 | |
*** priteau has joined #openstack-nova | 13:38 | |
mriedem | gibi: sorry i didn't address these before rebasing yesterday, but replied now https://review.opendev.org/#/c/635080/38 | 13:40 |
sean-k-mooney | mriedem: o/ | 13:42 |
sean-k-mooney | mriedem: i played around with lxc a bit last night | 13:42 |
sean-k-mooney | so i managed to boot a contianer but there are issues | 13:42 |
sean-k-mooney | i was able to boot with the .img file in https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.tar.gz | 13:43 |
sean-k-mooney | that is a img file create from a raw ext4 partion with the full file system | 13:44 |
sean-k-mooney | i confimed the cirros image seam to use /init instead of /sbin/init and we hardcode the path | 13:44 |
*** belmoreira has quit IRC | 13:45 | |
sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5055-L5057 | 13:45 |
sean-k-mooney | but i also found we do not resize the image and as a result cloud init does not run | 13:45 |
sean-k-mooney | * we might resize the image but we dont resize the filesystem in the image | 13:46 |
gmann | alex_xu: cc for ^^ policy default refresh changes review and early feedback- http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008504.html | 13:47 |
gmann | mriedem: can you check this now, added functional tests - https://review.opendev.org/#/c/676275/ | 13:47 |
sean-k-mooney | mriedem: so tl;dr if you have a image preprepared with enough space for cloud-init to run we might be oke but i dont know if cloud inti can resize the file system when dealing with lxc | 13:48 |
*** sapd1_x has quit IRC | 13:50 | |
*** belmoreira has joined #openstack-nova | 13:51 | |
openstackgerrit | Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810 | 13:55 |
*** sapd1_x has joined #openstack-nova | 13:56 | |
*** sapd1_x has quit IRC | 13:57 | |
yonglihe | mriedem: I got Alex's +2, you recently reviewed this patch, Hope you had bandwidth to review it again: https://review.opendev.org/#/c/621476/ | 14:02 |
efried | melwitt: Thank you for sending that. The u-c bump merged as well. That opens up several improvements in nova. | 14:02 |
mordred | efried: did your mock.patch thing wind up working? | 14:12 |
efried | yes! | 14:12 |
gibi | mriedem: ack, I try to look at it today | 14:12 |
efried | mordred: which excites me not because it's a lovely solution, but because it lets me use all those shiny new features in nova now. | 14:13 |
*** tbachman has quit IRC | 14:13 | |
sean-k-mooney | efried: you are not useing mock.patch outside of tests are you? | 14:13 |
efried | which I will start doing just as soooon as I wake up. | 14:13 |
efried | sean-k-mooney: Why, would that be a bad thing? | 14:14 |
efried | it's exactly the same as temporary_mutation | 14:14 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: config: remove deprecated checksum options https://review.opendev.org/676539 | 14:14 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: Remove nova.image.download.modules extension point https://review.opendev.org/676540 | 14:14 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: requirements: move to using psycopg2-binary https://review.opendev.org/676684 | 14:14 |
efried | sean-k-mooney: I'm yanking your chain. We're talking about this https://review.opendev.org/676495 | 14:14 |
sean-k-mooney | because you are monkeypatchin other code at runtime which can fail in really hard to debug ways | 14:14 |
sean-k-mooney | :) | 14:15 |
sean-k-mooney | ok that is less scary | 14:15 |
cdent | sigh: the nova functional tests are _still_ reading /etc/nova/nova.conf (if it exists). I thought we fixed that? | 14:16 |
sean-k-mooney | what test? | 14:17 |
sean-k-mooney | its proably missing a config fixture | 14:17 |
* aspiers has been wrestling with mock.PropertyMock for two hours now, and is still losing :-( | 14:17 | |
aspiers | AFAICS it just doesn't do what it claims to | 14:18 |
tssurya | mriedem, dansmith: hmm I was thinking for the UnexpectedTaskStateError exception for the power-update event, should we also handle the faults/error notifications in case its actually another action that won the race ? | 14:18 |
tssurya | as of now if we simply return the event is considered as a successful attempt | 14:18 |
dansmith | tssurya: I'm not sure what you're asking | 14:19 |
openstackgerrit | Chris Dent proposed openstack/nova master: single pass instance info fetch in host manager https://review.opendev.org/623558 | 14:20 |
mriedem | yonglihe: i can review again based on my previous comments but i likely won't be approving that change, because (1) we're trying to land tssurya's change for 2.76 since there is an approved ironic change that depends on the nova change (see the ML) and (2) i can't vote on the low-level contents of what's going into the numa topology response, i think dansmith or others that were more involved in the spec approval should revie | 14:20 |
mriedem | ose details. | 14:20 |
mriedem | sean-k-mooney: ack on the lxc thing; we can define an image to use in the job, but if that hard-coding in the libvirt driver is a problem we'll need to do something about that. | 14:21 |
mriedem | sean-k-mooney: i'm not sure how much we care about cloud-init in the lxc job, but someone does have some patches up for making cloud-init work with lxc (so saying that, it might not work today anyway) | 14:21 |
*** sapd1_x has joined #openstack-nova | 14:21 | |
tssurya | dansmith: ah nvm, I got the answer | 14:21 |
mriedem | sean-k-mooney: yeah this https://review.opendev.org/#/c/667976/ | 14:21 |
tssurya | :D sorry for the noise | 14:22 |
openstackgerrit | Eric Fried proposed openstack/nova master: Bump minimum ksa (3.16.0) and sdk (0.34.0) https://review.opendev.org/676707 | 14:22 |
dansmith | tssurya: heh, okay, sorry | 14:22 |
cdent | mriedem: I reckon that single pass instance info thing ^ is a real winner. I've removed some dead code (I think jay had future plans) but I'm a bit lost on what to do test wise, so if you have ideas, I've got ears. | 14:22 |
sean-k-mooney | "/sbin/init" has been the location of the init script for decades so i think cirros just need to add a symlink | 14:22 |
dansmith | sean-k-mooney: I have that open in a tab, working on trying to focus on it for a bit... is there a change coming? | 14:22 |
sean-k-mooney | im more worried about the symlink | 14:22 |
sean-k-mooney | *filesystem | 14:22 |
sean-k-mooney | e.g not resizing it before starting the container | 14:22 |
sean-k-mooney | i dont know if you can resize it inside the container and a spent a while trying to do it last night and failed | 14:23 |
efried | aspiers: what are you trying to do with PropertyMock? | 14:23 |
aspiers | trying to patch objects.Instance.image_meta to return my image_meta fixture | 14:23 |
mriedem | cdent: fwiw track_instance_changes might be true in grenade jobs, would need to check, but grenade jobs don't run in superconductor mode | 14:24 |
aspiers | but it just returns a mock object instead | 14:24 |
sean-k-mooney | dansmith: sorry context are you refering to stephens chagnes? | 14:24 |
sean-k-mooney | or something else | 14:24 |
dansmith | sean-k-mooney: gah, sorry that was supposed to be aimed at tssurya | 14:24 |
sean-k-mooney | ah | 14:24 |
aspiers | calling instance.image_meta() returns the fixture I want, but not without the () | 14:24 |
tssurya | dansmith: ok no wait I didn't get the answer :( . So in the API we do the action_start thing which creates the instance-action event, however when we specially handle the UnexpectedTaskStateError (so that we can continue to process other events) in the manager, the "compute_power_update' event is recorded as a success | 14:24 |
cdent | mriedem: there's something in devstack comments about that, lemme refresh memory | 14:25 |
aspiers | it's infuriating, been staring for hours and can't see what I'm doing wrong | 14:25 |
tssurya | which is wrong right ? | 14:25 |
dansmith | tssurya: ah, yeah, can we record that as fail instead in the exception handler? | 14:25 |
mriedem | cdent: having said that, i'm crunched for time today, leaving this afternoon for a long weekend | 14:25 |
sean-k-mooney | mriedem: ill look at the cloud-init patch but without it we are not doing the ssh key injection or creating a user so you would have to build all that into the image | 14:25 |
cdent | i don't think there's any rush on this, just think it is definitely worth bring back from the dead | 14:26 |
tssurya | dansmith: yea probably, let me check what does that | 14:26 |
*** mlavalle has joined #openstack-nova | 14:26 | |
dansmith | tssurya: it would be super good if we can get mriedem's ack on that patch before he leaves and then you leave | 14:26 |
tssurya | wait mriedem is also going on vacation soon ? | 14:26 |
dansmith | for a day yeah | 14:27 |
tssurya | hmm, ok I am just a bit confused with instance faults/actions/error notification handling for the UnexpectedTaskStateError part | 14:27 |
mriedem | if you're trapping that error, there will be no fault | 14:28 |
mriedem | or error notification | 14:28 |
dansmith | oh | 14:28 |
dansmith | yeah, mriedem just answered | 14:28 |
mriedem | you'll only record a fault or send an error notification if something raises, like NotImplementedError | 14:28 |
mriedem | @wrap_instance_event will record a start and finish event on entry and exit of the method regardless of how it exits | 14:28 |
dansmith | you can log and re-raise maybe? | 14:29 |
mriedem | the event gets record on the instance action (Created in the api) keyed by instance uuid and context.request_id | 14:29 |
mriedem | *recorded | 14:29 |
tssurya | mriedem: yea but since I do a return from the preemption if any it records a success I guess ? | 14:29 |
mriedem | yes, it will say it was successful | 14:30 |
mriedem | https://github.com/openstack/nova/blob/872a823d9a02f31e266882bfb499673c51fb4075/nova/compute/utils.py#L1302 | 14:30 |
mriedem | https://github.com/openstack/nova/blob/872a823d9a02f31e266882bfb499673c51fb4075/nova/objects/instance_action.py#L184 | 14:31 |
aspiers | efried: *finally* figured it. patch(..., new_callable=mock.PropertyMock), *not* patch(..., new_callable=mock.PropertyMock(name='fake_prop')) | 14:31 |
mriedem | i think i mentioned something in my earlier big round of comments that the functional test should assert the events are recorded as expected for the compute_power_update event | 14:31 |
tssurya | mriedem: yea I added that | 14:32 |
tssurya | but this should not be a success for the task_state race error right ? | 14:32 |
mriedem | idk | 14:32 |
mriedem | yesterday i thought we said that we'd handle UnexpectedTaskStateError, log and return | 14:32 |
mriedem | if you want to record the action as failed, then you'd log and re-raise | 14:33 |
yonglihe | mriedem: ok if i need to bump up minor version, i gonna do that, but thanks. | 14:33 |
tssurya | ok yea | 14:33 |
tssurya | I'll do that | 14:33 |
dansmith | mriedem: I didn't grasp that detail about letting it go | 14:33 |
mriedem | well, what does dansmith think | 14:33 |
mriedem | oh | 14:33 |
mriedem | note that ironic at this point has got a response and moved on | 14:33 |
dansmith | ironic doesn't care, | 14:33 |
mriedem | so this is only useful for an admin debugging that power-update exchange and if it worked | 14:33 |
dansmith | this is more like what a user would see | 14:33 |
dansmith | or the user itself right? | 14:34 |
tssurya | yea | 14:34 |
mriedem | the user can see the actions on their server yeah | 14:34 |
tssurya | I am also concerned because of the instance-action CLI show thing | 14:34 |
efried | aspiers: That's a facet of new= vs new_callable= | 14:34 |
mriedem | and the events after a certain microversion (but not traceback details unless they are admin) | 14:34 |
aspiers | efried: yeah, I realise that *now*, after two hours :-( | 14:34 |
yonglihe | dansmith: I need to bother you with it then. https://review.opendev.org/#/c/621476/ | 14:34 |
dansmith | I can see it both ways: | 14:34 |
aspiers | efried: I was misled by some existing nova tests which use it incorrectly | 14:35 |
dansmith | either they see an action that started/finished, but nothing happened because it was basically dropped, which might be weird, | 14:35 |
efried | aspiers: If you need to customize your PropertyMock, you would use new=PropertyMock(...) | 14:35 |
*** spatel has joined #openstack-nova | 14:35 | |
dansmith | and they might see an action start/fail and think something is wrong | 14:35 |
dansmith | so I dunno | 14:35 |
* aspiers blames everyone else | 14:35 | |
dansmith | mriedem: tssurya ^ | 14:35 |
aspiers | it can't possibly be me just being stupid | 14:35 |
efried | aspiers: Feel free to propose a patch to fix those bogus usages | 14:35 |
tssurya | dansmith: yea let me check how it is for other actions like pause | 14:35 |
aspiers | efried: I certainly will | 14:35 |
efried | I see three-ish | 14:35 |
tssurya | which are not implemented in ironic driver | 14:35 |
tssurya | wonder if that records a success action | 14:36 |
openstackgerrit | Chris Dent proposed openstack/nova master: DNM: add profiling to scheduler https://review.opendev.org/674810 | 14:36 |
mriedem | tssurya: it wouldn't, | 14:36 |
mriedem | https://github.com/openstack/nova/blob/872a823d9a02f31e266882bfb499673c51fb4075/nova/compute/manager.py#L4977 | 14:36 |
tssurya | yea it doesn't | 14:36 |
mriedem | it would record a failure b/c of NotImplementedError | 14:36 |
tssurya | yea | 14:37 |
tssurya | dansmith, mriedem: if you are both ok with the whole success and return thing I can do that | 14:38 |
tssurya | but if you feel its confusing for the users I can reraise and log and ignore raise | 14:38 |
dansmith | tssurya: I dunno, that I have a strong feeling about which is right or not | 14:38 |
tssurya | dansmith: ack | 14:39 |
mriedem | gmann: then why have these unit tests still? https://review.opendev.org/#/c/676275/4/novaclient/tests/unit/v2/test_shell.py | 14:39 |
mriedem | they are just asserting what the stub is doing https://review.opendev.org/#/c/676275/4/novaclient/tests/unit/v2/fakes.py | 14:39 |
mriedem | which might be wrong if there is a bug in the stub | 14:39 |
mriedem | idk either - the user is going to see a stop or start action with a compute_power_update event, they likely won't know what that event means. does the user know that someone powered on or off their bm instance out-of-band? | 14:41 |
*** mlavalle has quit IRC | 14:41 | |
mriedem | and since this isn't an action initiated by the user (at least directly in the compute api) but by ironic, is it just best effort and we shouldn't record a failed event if that operatoin lost a race? | 14:41 |
*** mlavalle has joined #openstack-nova | 14:42 | |
mriedem | if it's best effort and we don't consider losing a race a failure, then we shouldn't record the overall action as failed, but just "processed" | 14:42 |
dansmith | mriedem: yeah, that's a valid way to look at it I think | 14:42 |
dansmith | it's best effort, a hint so to speak, so failure isn't really failure | 14:43 |
tssurya | fair enough | 14:43 |
tssurya | and its easier that way for me :) | 14:43 |
dansmith | just put "it's easier for tssurya" in the spec and then you're good ;) | 14:43 |
tssurya | I'll stick to what we discussed yesterday then | 14:43 |
tssurya | dansmith: hehe | 14:44 |
mriedem | thinking about volume-extended events, we definitely want those to be failed so the user sees because it's initiated by the user resizing their volume in the cinder api | 14:44 |
tssurya | mriedem: this was a another doubt I had | 14:44 |
tssurya | that means we block the rest of the events in the queue no ? | 14:44 |
tssurya | because one volume-extend event broke ? | 14:44 |
tssurya | and you raise | 14:44 |
mriedem | we wouldn't get a volume-extended and power-update event in the same request | 14:45 |
mriedem | unless someone sends that manually | 14:45 |
mriedem | there might be a bug in the volume-extended event handling in compute that's not handling that properly, but that's a separate bug | 14:45 |
tssurya | mriedem: got it | 14:45 |
mriedem | tssurya: how much longer are you going to be around today? | 14:46 |
tssurya | mriedem: do you have a sample functional test that you have written for such races | 14:46 |
tssurya | ? | 14:46 |
tssurya | I couldn't find a way to hack the task_state in between my code | 14:47 |
tssurya | for the functional teste | 14:47 |
tssurya | mriedem: i'll be around only for 2 more hours | 14:47 |
mriedem | you'd have to stub/mock something in between to change the task_state after the compute manager sets it and the driver code gets it | 14:47 |
mriedem | i'm not sure we need the functional test to cover that though | 14:47 |
mriedem | that's pretty unit-y test level to me | 14:48 |
tssurya | but its ok, I'll be online and if you have comments will update them during vacation | 14:48 |
tssurya | mriedem: ok then I'll just do a unit test | 14:48 |
dansmith | yep, unit test makes sense to me | 14:49 |
dansmith | just to make sure we catch it and do the right thing | 14:49 |
tssurya | dansmith: yea I'll do that | 14:49 |
mriedem | tssurya: but i won't be around to review the updates until monday... and people are held up on this microversion, so i'll review it in detail now before you make another push | 14:51 |
tssurya | mriedem: ok sounds good | 14:52 |
tssurya | even if I am on vacation I do intend to work on this patch to get it in, so I hope to not block everyone | 14:52 |
tssurya | s/not block/unblock soon | 14:53 |
atmark | hello, I have mix setup of local disk computes and rbd backed computes. In nova.conf of local disk computes, I have images_type=raw and in the rbd backed compute image_type=rbd. I can spin up VMs fine in both setup however the when RESIZE a local disk VM, I get 'boot failed: not a bootable disk' from the VM's console | 14:55 |
atmark | resizing a RBD VM works. Resizing local disk VMs fails | 14:56 |
*** gyee has joined #openstack-nova | 14:57 | |
mriedem | tssurya: doesn't test_external_instance_event_power_update_unexpected_task_state already cover the race you were asking about testing? | 15:12 |
tssurya | mriedem: yes | 15:13 |
tssurya | that's the unit test I have | 15:13 |
mriedem | then we don't need a functional test wrinkle for that | 15:13 |
tssurya | I didn't have a functional test hence asked | 15:13 |
tssurya | ah ok | 15:13 |
tssurya | cool | 15:13 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test heal port allocations in nova-next https://review.opendev.org/669879 | 15:14 |
*** READ10 has joined #openstack-nova | 15:15 | |
mriedem | tssurya: dansmith: comments posted https://review.opendev.org/#/c/645611/ | 15:18 |
*** artom has joined #openstack-nova | 15:18 | |
tssurya | thanks | 15:18 |
mriedem | if you run out of time, post what you have and i'll fix the rest and +2 | 15:20 |
*** dave-mccowan has joined #openstack-nova | 15:22 | |
*** READ10 has quit IRC | 15:24 | |
*** belmoreira has quit IRC | 15:26 | |
*** macz has joined #openstack-nova | 15:26 | |
*** belmoreira has joined #openstack-nova | 15:28 | |
*** jamesdenton has quit IRC | 15:29 | |
*** dave-mccowan has quit IRC | 15:30 | |
*** macz has quit IRC | 15:31 | |
*** ivve has quit IRC | 15:35 | |
*** belmoreira has quit IRC | 15:38 | |
openstackgerrit | Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899 | 15:39 |
openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | 15:39 |
openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | 15:39 |
openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | 15:39 |
openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691 | 15:39 |
openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213 | 15:39 |
efried | dustinc: Rebased and with requirements bump to sdk 0.34.0 ^ | 15:39 |
dustinc | thanks | 15:39 |
efried | that should fix `fields`. I'm hoping it also fixes the remaining patches | 15:39 |
dustinc | 🤞 | 15:40 |
efried | cause I also added a dep to the devstack patch | 15:40 |
efried | hmm | 15:40 |
efried | Will the second patch still observe the dep from the first? | 15:40 |
mriedem | yes | 15:42 |
openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Limit get_sdk_adapter to requested service type https://review.opendev.org/674678 | 15:44 |
*** markvoelker has quit IRC | 15:44 | |
efried | cool | 15:45 |
*** efried has quit IRC | 15:46 | |
mriedem | i guess forbidden aggregates is never getting out of a runway slot | 15:48 |
*** markvoelker has joined #openstack-nova | 15:48 | |
*** efried has joined #openstack-nova | 15:50 | |
mriedem | sean-k-mooney: if you can get a new patch set for https://review.opendev.org/#/c/671338/ today it would be good as well, since it was pretty close | 15:51 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: config: remove deprecated checksum options https://review.opendev.org/676539 | 15:51 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: Remove nova.image.download.modules extension point https://review.opendev.org/676540 | 15:51 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: requirements: move to using psycopg2-binary https://review.opendev.org/676684 | 15:51 |
sean-k-mooney | ok i was doing som sriov testing but ill priorities that and get it updated today | 15:52 |
sean-k-mooney | mriedem: ill ping you when its done | 15:52 |
*** cdent has quit IRC | 15:53 | |
*** mlavalle has quit IRC | 15:53 | |
openstackgerrit | Mohammed Naser proposed openstack/nova master: constraints: bump psycopg2 to 2.8 https://review.opendev.org/676725 | 15:53 |
mriedem | brinzhang: i'm going to move https://blueprints.launchpad.net/nova/+spec/support-specifying-az-when-restore-shelved-server into the open runway slot though you'll likely have to rebase it again once 2.76 is taken by the power-update change | 15:54 |
*** luksky has joined #openstack-nova | 15:55 | |
*** mlavalle has joined #openstack-nova | 15:57 | |
*** sapd1_x has quit IRC | 15:58 | |
melwitt | tssurya: what do you think about multi-cell archive deleted rows skipping disabled cells? or would you rather expect it to try each cell and move on if can't connect instead | 15:59 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Mock libvirt'y things globally https://review.opendev.org/676727 | 15:59 |
tssurya | melwitt: you mean down cells ? | 16:00 |
tssurya | disabled cells should be archived without distinction | 16:00 |
mriedem | disabled | 16:00 |
mriedem | wouldn't one want to disable a cell if it's down? | 16:00 |
*** slaweq has joined #openstack-nova | 16:00 | |
mriedem | so the scheduler doesn't try to pull computes from it and timeout? | 16:00 |
tssurya | yea but its only a workaround for scheduling right ? | 16:01 |
tssurya | people could also have disabled the cells since its new | 16:01 |
melwitt | tssurya: ok. that's kinda what I was thinking, not to assume because disabled to skip. because otherwise you (operator) would have to enable any disabled cells in order to archive and I thought maybe that would be annoying | 16:01 |
tssurya | not necessarily down | 16:01 |
mriedem | melwitt: well, you could just archive them directly | 16:01 |
mriedem | like you do today | 16:01 |
mriedem | anyway, it was a question - feel free to ignore it until it's a problem | 16:01 |
tssurya | melwitt: right yea | 16:01 |
*** markvoelker has quit IRC | 16:02 | |
melwitt | ok, just wanted to take the opportunity to ask tssurya since I saw her around | 16:02 |
mriedem | could add a --skip-disabled later if people want that | 16:03 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 16:03 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Follow up for I491eb7abe1405b27c552cdef142c688a46e914f2 https://review.opendev.org/675589 | 16:03 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 16:03 |
melwitt | aye | 16:03 |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional regression recreate test for bug 1839560 https://review.opendev.org/676507 | 16:03 |
openstack | bug 1839560 in OpenStack Compute (nova) stein "ironic: moving node to maintenance makes it unusable afterwards" [High,In progress] https://launchpad.net/bugs/1839560 - Assigned to Matt Riedemann (mriedem) | 16:03 |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Restore soft-deleted compute node with same uuid https://review.opendev.org/676509 | 16:03 |
mriedem | mnaser: ^ functional test should be fixed now | 16:04 |
stephenfin | mriedem, efried: When you've a chance, could you look at my two TODOs here to make sure I'm understanding what placement is doing correctly? (it's for the VCPU -> PCPU reshape) https://review.opendev.org/#/c/674895/8/nova/virt/libvirt/driver.py@7373 https://review.opendev.org/#/c/674895/8/nova/virt/libvirt/driver.py@7389 | 16:06 |
openstackgerrit | Akihiro Motoki proposed openstack/nova master: DNM: WIP: PDF documentation build https://review.opendev.org/676730 | 16:07 |
openstackgerrit | Merged openstack/nova stable/queens: Fix misuse of nova.objects.base.obj_equal_prims https://review.opendev.org/676291 | 16:09 |
*** markvoelker has joined #openstack-nova | 16:11 | |
efried | stephenfin: It is true that *today* the compute node provider is the only provider of *CPU resources. So adding new code with that assumption ingrained is safe, but adds to the tech debt we have to pay off when we model NUMA nodes. | 16:11 |
stephenfin | efried: Is that a fair assumption for the reshape operation though | 16:11 |
stephenfin | ? | 16:11 |
openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611 | 16:12 |
stephenfin | Given that in order to get a NUMA topology, we'd have to had reshaped first | 16:12 |
efried | stephenfin: What's the upgrade path? Would there be a way to FFU past Train and straight into U? | 16:13 |
efried | How are we enforcing the order of different reshapes? | 16:13 |
efried | Because that's what we're talking about here. | 16:13 |
tssurya | mriedem, dansmith: ^ thanks for all the help | 16:13 |
openstackgerrit | Akihiro Motoki proposed openstack/nova master: DNM: WIP: PDF documentation build https://review.opendev.org/676730 | 16:13 |
stephenfin | I have no idea. sean-k-mooney and I have been discussing that | 16:13 |
dansmith | melwitt: I agree with tssurya that archive should not look at the disabled flag | 16:13 |
stephenfin | I'm not even sure how reshapes and FFU work in general | 16:14 |
efried | stephenfin: My point, when you write the NUMA reshaper, is it also going to account for VCPU->PCPU-ness? | 16:14 |
stephenfin | Ah, gotcha | 16:14 |
efried | well... they don't, yet. | 16:14 |
*** slaweq has quit IRC | 16:14 | |
stephenfin | I guess we could just do it in order? | 16:14 |
efried | yeah. Though we don't yet have any kind of infrastructure set up for "do these reshapes in this order". | 16:15 |
efried | Like what happens for db migrations (right?) | 16:15 |
stephenfin | So as we add reshapers, we follow the pattern started with vgpu and keep adding new '_update_provider_tree_for_foo' functions | 16:15 |
stephenfin | i.e. https://review.opendev.org/#/c/674895/8/nova/virt/libvirt/driver.py@6926 | 16:15 |
mriedem | stephenfin: replied | 16:15 |
stephenfin | *adding new functions below the existing functions | 16:15 |
melwitt | dansmith: ack, thanks | 16:16 |
stephenfin | mriedem: ta | 16:16 |
*** ricolin has quit IRC | 16:17 | |
efried | stephenfin: Yes, that's the idea. I suppose it would be nice to have it more codified. Especially since we're going to want some way to invoke these generically (i.e. from a non-virt-driver-specific engine) for FFU. | 16:17 |
* mriedem gets out his red pen just for tssurya | 16:17 | |
*** tssurya has quit IRC | 16:18 | |
melwitt | and..... she left | 16:18 |
efried | stephenfin: I draw your attention to http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004204.html | 16:18 |
mriedem | efried: the FFU support for VGPU never happened right? | 16:18 |
efried | mriedem: jinx | 16:18 |
mriedem | wonder if anyone has hit the reshape for vgpu in stein - mnaser, did vexxhost have vgpu resources in rocky? | 16:18 |
mnaser | no, we do not use vgpus, sorry | 16:18 |
mriedem | or only physical gpu passthrough | 16:19 |
mriedem | ok | 16:19 |
mnaser | only physical passthrough | 16:19 |
mriedem | have you ever heard of anyone providing vgpus? | 16:19 |
mnaser | someone contributed a patch to OSA to enable vgpu support | 16:19 |
mnaser | and it seemed like they knew what they were doing in it, maybe i can find them | 16:19 |
mnaser | (osa does the whole mdev lookup and stuff) | 16:19 |
mriedem | i think i remember hearing cern didn't use them b/c they didn't get the performance needed compared to physical passthorugh | 16:19 |
*** eharney has quit IRC | 16:20 | |
mriedem | though people have been pushing the patches to hide the hypervisor in the guest to hack around nvidia licensing stuff | 16:20 |
mnaser | https://review.opendev.org/#/c/597183/ | 16:20 |
mnaser | looks like that person was testing it with the i915 gvt stuff | 16:20 |
mriedem | maybe that was OVH? | 16:20 |
mnaser | according to the review | 16:20 |
mnaser | Valentin Boucher <valentin.boucher@kontron.com> | 16:21 |
mriedem | ok i'm thinking ovh was checking out vgpus | 16:21 |
mriedem | but not sure how to ask, amorin ^? | 16:21 |
mnaser | oh look at that | 16:21 |
mnaser | valentin lives in montreal | 16:21 |
mriedem | *who | 16:21 |
mriedem | mnaser: i may be 'merican but i know montreal != france :) | 16:21 |
melwitt | dansmith, mriedem: what do you think about moving onto the next cell if anything fails (down cells)? on the surface, it seemed like a good idea to me but now I'm wondering if that could possibly mislead anyone by making them think archive "completed" when it didn't? | 16:21 |
mnaser | oh no i was just surprised he's a local :) | 16:22 |
mriedem | melwitt: can we avoid handling down cells in that already too big patch? | 16:22 |
melwitt | sure | 16:22 |
mriedem | e.g. leave a todo in the code where a down cell thing could be handled, which is either log and continue or fail | 16:22 |
dansmith | agree | 16:22 |
dansmith | but also, yes I think skip/warn unreachable cells | 16:23 |
melwitt | yeah, I was thinking just "except Exception" log continue | 16:24 |
mriedem | stephenfin: i'm guessing i just ruined your day with the "what about pending resizes and migration allocation consumers" thing | 16:26 |
mriedem | you're welcome | 16:26 |
stephenfin | yeah, way to bum me out | 16:26 |
*** slaweq has joined #openstack-nova | 16:31 | |
mriedem | dansmith: before i address my comments in https://review.opendev.org/#/c/645611/14 do you want to make a pass? | 16:34 |
mriedem | s/do you want to/will you please/ | 16:35 |
dansmith | mriedem: I read your comments before and commented on that physical thing, are there other questions? | 16:35 |
mriedem | not questions no, | 16:36 |
dansmith | because tbh, it's fine enough for me and I'd just merge it to claim the version and then address the nits and otherthings later | 16:36 |
mriedem | but if you have comments on the diff itself, make them now before i make changes | 16:36 |
mriedem | ok i'll start updating then | 16:36 |
dansmith | oh are you pushing the next verison? | 16:36 |
mriedem | yes | 16:36 |
mriedem | surya is gone | 16:36 |
dansmith | oh, sorry, she pushed again, my bad | 16:36 |
dansmith | been distracted | 16:37 |
*** slaweq has quit IRC | 16:38 | |
mriedem | efried: do you want me to respin for that unrelated change in https://review.opendev.org/#/c/675218/4 ? | 16:39 |
dansmith | mriedem: there was one legit test failure in the last run, IIRC, but I didn't look into it.. is that fixed? | 16:39 |
dansmith | otherwise, yeah, I'm fine with current plus your planned changes | 16:39 |
efried | mriedem: Might as well, it doesn't seem worth a whole fup. I'm here to re+2 it quick. | 16:40 |
mriedem | dansmith: idk, but i can test locally https://logs.opendev.org/11/645611/13/check/openstack-tox-py27/7deb014/testr_results.html.gz | 16:40 |
mriedem | efried: ack will do | 16:40 |
efried | coo | 16:40 |
dansmith | ookie | 16:41 |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional regression recreate test for bug 1839560 https://review.opendev.org/676513 | 16:43 |
openstack | bug 1839560 in OpenStack Compute (nova) stein "ironic: moving node to maintenance makes it unusable afterwards" [High,In progress] https://launchpad.net/bugs/1839560 - Assigned to Matt Riedemann (mriedem) | 16:43 |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Restore soft-deleted compute node with same uuid https://review.opendev.org/676514 | 16:43 |
*** slaweq has joined #openstack-nova | 16:48 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: Re-use DB MetaData during archive_deleted_rows https://review.opendev.org/675218 | 16:50 |
artom | dansmith, mriedem, hey, you guys were the main reviewers of NUMA LM in Stein, will have bandwidth to do the same in Train, or should I be sucking up to other cores? ;) | 16:57 |
artom | *will you have | 16:57 |
dansmith | I will try | 16:58 |
artom | dansmith, ack, thanks :) | 16:58 |
openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: add support for vPMU configuration. https://review.opendev.org/671338 | 16:58 |
mriedem | artom: not today | 17:01 |
mriedem | artom: you should also beat down the starlingx people's door, | 17:02 |
artom | mriedem, I wasn't really expecting today :) | 17:02 |
mriedem | because at the ptg and last summit they were saying this was the big ticket thing they needed | 17:02 |
artom | I meant in general | 17:02 |
mriedem | so if this is so critical for starlingx they should have people reviewing and testing it | 17:02 |
artom | mriedem, yeah, I emailed the ML, and dtroyer and Gerry personally | 17:02 |
artom | Radio silence :/ | 17:02 |
mriedem | cfriesen is probably the best starlingx person to review the series but he's been out of nova for awhile now | 17:03 |
sean-k-mooney | well that and sriov numa policies was on the list too. | 17:03 |
sean-k-mooney | i think we almost have everything this cycle? | 17:03 |
artom | Gerry Kopec (not upstream as far as I can tell, though he did leave some reviews) tested it | 17:03 |
artom | And found some problems | 17:03 |
artom | Was hoping he'd step up again | 17:03 |
artom | Radio silence so far. Which is weird, because Dean actually emailed me first | 17:04 |
artom | Asking how it was going | 17:04 |
sean-k-mooney | artom: i need to set up a second node but when im done testign a sriov thing i plan to test your numa live migration code with stephens cpu pinning stuff and my sriov migration code and report back | 17:05 |
sean-k-mooney | eta on that is sometime next week | 17:05 |
*** derekh has quit IRC | 17:05 | |
artom | sean-k-mooney, my series has conflicts with stephenfin's, so not sure how you'll test those together | 17:06 |
artom | Well, resolve manually, I suppose | 17:06 |
mriedem | cpu pinning stuff == PCPU? | 17:06 |
sean-k-mooney | ya i was thinking of doign that but if there are a lot ill test them seperatly | 17:06 |
sean-k-mooney | yes | 17:06 |
sean-k-mooney | i would also like to test gibis bandwith stuff | 17:06 |
mriedem | if it were me, i'd test artom's numa live migration stuff w/o trying to wrangle the PCPU stuff in | 17:07 |
mriedem | jesus | 17:07 |
mriedem | so yo'ud like to actually not get anything done :) | 17:07 |
sean-k-mooney | well ill test them seperatly first | 17:07 |
artom | He'll become a master at git rebasing and conflict resolution though :) | 17:07 |
sean-k-mooney | but i want to actully test those 4 features in 1 vm in train | 17:08 |
sean-k-mooney | i guess it better to wait untill some/all have merged to do that | 17:08 |
mriedem | well i applaud your efforts for at least trying to test this stuff | 17:09 |
mriedem | since no one else is | 17:09 |
mriedem | together i mean | 17:09 |
sean-k-mooney | well people will want to combine those so i want to test them before i get the bug report | 17:09 |
mriedem | don't worry, the bug report will be in 18+ months | 17:10 |
mriedem | you might not even be working on openstack at that point | 17:10 |
*** cdent has joined #openstack-nova | 17:10 | |
sean-k-mooney | ya.... if we are lucky and people deploy train by then | 17:10 |
artom | I still don't think testing them together before they're even merged is a good idea | 17:11 |
sean-k-mooney | TIL we still have at least 1 customer with a suppor exception for liberty until 2021 ... | 17:11 |
artom | Laudable, but not good ;) | 17:11 |
*** prometheanfire has joined #openstack-nova | 17:11 | |
prometheanfire | nova still has an issue with the latest libvirt https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_16/675816/5/check/cross-nova-py36/a6c71f8/testr_results.html.gz | 17:11 |
sean-k-mooney | artom: well in that case ill just test them 1 by one | 17:11 |
artom | But yeah, I totally get mriedem's gripe about the RH dev/QE lag | 17:12 |
*** tbachman has joined #openstack-nova | 17:12 | |
openstackgerrit | Adam Spiers proposed openstack/nova master: Fix use of mock.patch with new_callable=PropertyMock https://review.opendev.org/676738 | 17:12 |
aspiers | efried: ^^^ hope that's right, still trying to wrap my head around some of this stuff | 17:12 |
artom | It's been half-assedly brought up internally, but it takes a lot of work (literally - force over a distance) for a long time to change that | 17:13 |
mriedem | artom: well it's not really only RH, it's any distro that EOLs an old series and customers are forced to move up | 17:13 |
aspiers | any other mock experts wanna review that? | 17:13 |
mriedem | which is why we get all the RH queens bug reports bubblin gup now | 17:13 |
artom | mriedem, are there other significant distros besides OSP at this point? Honest question | 17:14 |
cdent | efried: I assume you probably know this already and we're merely following protocol: but I didn't write that "single pass instance info fetch in host manager". My ressurection of it was solely based on the profiling benefits. So a) I'll have to dig deeper to respond to the comment, b) the comments make that digging easier. | 17:14 |
artom | Commercial, I mean | 17:14 |
mriedem | canonical? VIO? suse? idk ask their devs | 17:14 |
sean-k-mooney | i think ubuntu still has more deplopyment then osp | 17:14 |
efried | aspiers: lgtm, thank you. | 17:14 |
artom | Ah, SUSE, right | 17:15 |
artom | (With apologies to aspiers) | 17:15 |
mriedem | artom: you can ask aspiers how suse is ramping up for rolling out rocky with SEV forks :) | 17:15 |
sean-k-mooney | and canonical openstack | 17:15 |
aspiers | artom: what are you apologising for? :) | 17:15 |
artom | aspiers, forgetting you exist, essentially ;) | 17:15 |
aspiers | haha | 17:15 |
aspiers | easily done | 17:15 |
aspiers | mriedem: yeah sadly that's probably gonna happen | 17:16 |
efried | cdent: ack. If we want to land this pretty much as-is, we just need unit/func test coverage in this patch. The rest can be FUPed. | 17:16 |
aspiers | Not 100% sure though. We have a policy of trying to stay as close to upstream as humanly possible, even on stable branches. | 17:16 |
aspiers | artom: let me know if I can help with anything from the distro angle | 17:16 |
sean-k-mooney | so do we but PMs have a different opipion on that when there are $$$$ | 17:17 |
artom | aspiers, thank you :) Though we were just discussing dev/QE lag | 17:17 |
artom | aspiers, and how the RH QE is lagging behind upstream | 17:17 |
artom | So we'll do an RFE, and like a year later, we find out if never worked | 17:17 |
artom | (Only slightly exaggerating) | 17:18 |
aspiers | artom: well at least you do QE on upstream :) | 17:18 |
artom | aspiers, we don't, hence the lag | 17:18 |
aspiers | I mean eventually | 17:18 |
artom | By the time stuff's in our builds, it's been months, if not years | 17:18 |
*** nweinber has joined #openstack-nova | 17:18 | |
aspiers | lag implies it happens eventually :) | 17:18 |
aspiers | If it never happens, that's infinite lag | 17:18 |
aspiers | although maybe I kinda just contradicted myself | 17:19 |
*** cdent has quit IRC | 17:19 | |
openstackgerrit | Matt Riedemann proposed openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611 | 17:20 |
mriedem | dansmith: ^ | 17:20 |
* mriedem gets lunch | 17:20 | |
*** eharney has joined #openstack-nova | 17:32 | |
dansmith | mriedem: looks good | 17:35 |
*** priteau has quit IRC | 17:39 | |
*** bbowen__ has quit IRC | 17:52 | |
*** nweinber has quit IRC | 17:55 | |
*** nweinber has joined #openstack-nova | 17:55 | |
*** nweinber has quit IRC | 17:55 | |
melwitt | argh why does it do "An auth plugin is required to determine endpoint URL" when I'm using the PlacementFixture | 18:03 |
efried | melwitt: You need to use the sdk fixture :) | 18:04 |
melwitt | this is what happens when I don't use ServersTestBase | 18:04 |
melwitt | oh. ok | 18:04 |
efried | (as well as PlacementFixture, if what you're trying to do involves real placement calls) | 18:04 |
melwitt | the sdk fixture is in test.TestCase | 18:05 |
efried | melwitt: are you getting the error locally or in zuul? | 18:06 |
efried | locally => upgrade your ksa and sdk in your virtualenv | 18:06 |
efried | zuul => rebase and recheck, since the u-c bump landed overnight. | 18:06 |
melwitt | locally. I'm trying to change from deriving from ServersTestCase to just TestCase | 18:06 |
melwitt | ok | 18:07 |
efried | yeah, I bet it's because ksa and/or sdk in your venv is downlevel OR those are latest but you don't have the new fixture you merged last night. | 18:07 |
melwitt | I had just recreated my venv. but yeah let me just rebase again in case that helps. I can never remember how this works with pulling u-c | 18:10 |
*** ivve has joined #openstack-nova | 18:10 | |
efried | melwitt: If you recreated your venv without https://review.opendev.org/676707 then you might still have the old libs, I'm not actually sure. | 18:12 |
efried | You can check with | 18:12 |
efried | .tox/$env/bin/pip show openstacksdk keystoneauth1 | 18:12 |
efried | ...and of course you'll want your change rebased against latest master to pick up last night's fixture change to accommodate that. | 18:13 |
melwitt | 0.34.0 and 3.16.0 | 18:13 |
efried | those are latest, good; so you'll have a problem if your change is based earlier than the fixture commit from yesterday. | 18:13 |
efried | ...and if all that is up to date, then it's a legit code problem :) | 18:14 |
melwitt | it is. thanks, rebasing now | 18:14 |
*** boxiang has quit IRC | 18:14 | |
efried | okay, cool, lmk if you still get errors of this ilk, I'm tracking closely. | 18:14 |
melwitt | thanks | 18:15 |
*** boxiang has joined #openstack-nova | 18:15 | |
openstackgerrit | sean mooney proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | 18:16 |
sean-k-mooney | melwitt: care to take a look at ^ | 18:16 |
*** Sundar has joined #openstack-nova | 18:16 | |
melwitt | no | 18:17 |
melwitt | jk | 18:17 |
sean-k-mooney | i would not blame you :P | 18:18 |
sean-k-mooney | but you more or less deprecated them with your mail post like 18 months ago but we never actully did | 18:18 |
melwitt | I feel attacked | 18:18 |
melwitt | yeah, I know. I'll look at it | 18:19 |
sean-k-mooney | cool im going to finish one last test and then call it a day | 18:19 |
sean-k-mooney | thanks | 18:19 |
mriedem | "no" was the correct answer | 18:20 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py https://review.opendev.org/636318 | 18:23 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 18:23 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 18:23 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Allow assertXmlEqual() to pass options to matchers.XMLMatches https://review.opendev.org/674628 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Fix libvirt driver tests to use LibvirtConfigCapsGuest instances https://review.opendev.org/674629 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types https://review.opendev.org/673151 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py https://review.opendev.org/636318 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 18:32 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 18:32 |
mriedem | sean-k-mooney: melwitt: i reviewed that deprecation patch (again) | 18:43 |
mriedem | more docs fun | 18:43 |
mriedem | melwitt: btw i missed that you split this all up https://review.opendev.org/#/c/640898/ - bug me next week to review that series; i wanted to play with it in devstack while reviewing, which i won't have time for today | 18:44 |
sean-k-mooney | mriedem: jsut looked ya i can fix up those issues and repin ill likely do it tomrrow however | 18:45 |
melwitt | mriedem: cool, will do. I'm gonna respin it to address tetsuro's comments, then I'll bug you next week | 18:46 |
melwitt | thanks | 18:46 |
mriedem | it's only ~8pm, you've got 5 more hours to put in sean | 18:46 |
sean-k-mooney | for once i started at 9 this moring | 18:46 |
sean-k-mooney | well 9 ish | 18:46 |
*** nweinber has joined #openstack-nova | 18:50 | |
*** spsurya has quit IRC | 18:53 | |
*** ociuhandu has joined #openstack-nova | 18:56 | |
mriedem | what time do you usually start? | 18:58 |
sean-k-mooney | around 10:30 or 11 | 18:58 |
mriedem | cripes | 18:58 |
*** ociuhandu has quit IRC | 18:58 | |
mriedem | i could watch the price is right every day | 18:58 |
sean-k-mooney | and i usually take an hour or two break between around 6:30 ish and come back on line around 7 or 8 with the intent of finsih up before 8 | 18:59 |
sean-k-mooney | *10 | 18:59 |
sean-k-mooney | i often dont but that in theory is the plan | 19:00 |
mriedem | sean-k-mooney: vpmu comments as well https://review.opendev.org/#/c/671338/ | 19:11 |
artom | He waved us good night on the internal channel | 19:12 |
mriedem | bah | 19:12 |
mriedem | i know he can't quite this early | 19:12 |
mriedem | https://www.youtube.com/watch?v=nsmY0tFJEIE | 19:14 |
*** gregwork has quit IRC | 19:16 | |
*** gregwork has joined #openstack-nova | 19:17 | |
melwitt | yessssss func test "OK" | 19:17 |
artom | mriedem, that's a very unhealthy relationship to have with a piece of software ;) | 19:17 |
artom | I suppose that could change in the future once AI becomes better | 19:18 |
mriedem | artom: i'll let you make the firmware joke here | 19:18 |
artom | You have to flash it. | 19:19 |
mriedem | HI-O | 19:19 |
artom | *finger guns* | 19:20 |
*** nweinber has quit IRC | 19:22 | |
*** nweinber has joined #openstack-nova | 19:22 | |
efried | mriedem: grenade jobs have been failing a lot today, any ideas? | 19:26 |
*** nweinber has quit IRC | 19:27 | |
mriedem | linke me | 19:28 |
mriedem | *link | 19:28 |
mriedem | artom: trying to push the starlingx folk publicly as well | 19:28 |
artom | mriedem, you're asking me to push them, or saying you're doing it? | 19:29 |
mriedem | i called them out in the ML | 19:29 |
mriedem | artom: i'm assuming starlingx has a weekly meeting, it would probably be good if you can get your thing on their agenda as a reminder | 19:30 |
mriedem | https://wiki.openstack.org/wiki/Starlingx/Meetings | 19:30 |
*** vesper11 has quit IRC | 19:30 | |
mriedem | "Bi-weekly Marketing Call" hells yeah | 19:31 |
mriedem | i have no idea which sub-project meeting this would come up in though | 19:31 |
mriedem | i guess, "Weekly Community Call" | 19:31 |
efried | mriedem: https://review.opendev.org/#/c/676539/ and its successor in the series | 19:33 |
artom | mriedem, dtroyer is in #openstack-qa, we can just ping him | 19:34 |
mriedem | https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_39/676539/4/check/nova-grenade-live-migration/bd182b5/logs/grenade.sh.txt.gz | 19:34 |
mriedem | die 198 'SSH to the client did not work, something very wrong' | 19:34 |
mriedem | ssh fail | 19:34 |
mriedem | efried: so, that's like any other random ssh fail, and likely coming from the stein side where devstack doesn't run n-api-meta with caching | 19:38 |
mriedem | so recheck | 19:38 |
efried | okay, I just saw like four or five runs in a row come in with grenade failures, didn't want you to bugger off on a bear-baiting trip if there was something actually wrong. | 19:39 |
mriedem | i've got a call in 20 minutes anyway so i'm stuck here for a bit | 19:39 |
mriedem | but after that it's pants off dance off | 19:39 |
artom | Rock out with your, err, rooster out? | 19:40 |
* artom is getting *nothing* done :( | 19:41 | |
artom | Sick puns tho | 19:42 |
mriedem | so just a normal day huh | 19:42 |
artom | *ZING* | 19:42 |
* mriedem calls the fire dept | 19:42 | |
artom | https://en.wikipedia.org/wiki/List_of_burn_centres_in_Canada#Quebec | 19:44 |
artom | Oh damn, there's only 1 for adults | 19:44 |
openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | 19:44 |
openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Public method to retrieve custom resource providers https://review.opendev.org/676029 | 19:44 |
openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Load the custom resource providers to resource tracker https://review.opendev.org/676522 | 19:44 |
mriedem | efried: you can recheck on bug 1840355 | 19:59 |
openstack | bug 1840355 in grenade "cinder server ping check intermittently fails in the gate with "SSH to the client did not work, something very wrong"" [Undecided,Confirmed] https://launchpad.net/bugs/1840355 | 19:59 |
*** gouthamr is now known as gouthamr|brb | 20:02 | |
*** eharney has quit IRC | 20:08 | |
*** phasespace has joined #openstack-nova | 20:13 | |
*** nweinber has joined #openstack-nova | 20:16 | |
*** hemna has quit IRC | 20:24 | |
openstackgerrit | melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486 | 20:27 |
openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | 20:27 |
*** spatel has quit IRC | 20:27 | |
melwitt | mriedem: hopefully got all your comments addressed ^ I added another unit test for the until_complete=True case. and I will have to update the post test hook patch later | 20:28 |
mriedem | melwitt: ok i won't be able to look until next week | 20:28 |
melwitt | ok, cool | 20:29 |
melwitt | I have to run for now. will miss the nova meeting | 20:29 |
mriedem | same | 20:29 |
*** gouthamr|brb is now known as gouthamr | 20:30 | |
*** hemna has joined #openstack-nova | 20:36 | |
efried | thanks mriedem | 20:40 |
*** nweinber has quit IRC | 20:41 | |
*** spatel has joined #openstack-nova | 20:48 | |
*** takashin has joined #openstack-nova | 20:48 | |
*** slaweq has quit IRC | 20:50 | |
efried | nova meeting in 9 minutes in #openstack-meeting | 20:51 |
efried | sounds like it may be a sparse one... | 20:52 |
*** spatel has quit IRC | 20:53 | |
*** vesper11 has joined #openstack-nova | 20:53 | |
*** Sundar has quit IRC | 20:59 | |
*** mriedem has quit IRC | 21:00 | |
edleafe | "sparse" is putting it lightly :) | 21:09 |
*** slaweq has joined #openstack-nova | 21:11 | |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Deprecate the '--config-drive' option https://review.opendev.org/653683 | 21:15 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020 | 21:16 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | 21:16 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | 21:16 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | 21:17 |
*** slaweq has quit IRC | 21:17 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | 21:17 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | 21:17 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676 | 21:18 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689 | 21:18 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709 | 21:18 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | 21:19 |
openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Replace glance command with openstack command https://review.opendev.org/637060 | 21:19 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.opendev.org/641200 | 21:19 |
openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix non-existent method of Mock https://review.opendev.org/675041 | 21:20 |
*** BjoernT has quit IRC | 21:28 | |
*** markvoelker has quit IRC | 21:29 | |
*** artom has quit IRC | 21:30 | |
*** luksky has quit IRC | 22:01 | |
*** trident has quit IRC | 22:03 | |
*** markvoelker has joined #openstack-nova | 22:05 | |
*** markvoelker has quit IRC | 22:09 | |
*** trident has joined #openstack-nova | 22:10 | |
*** slaweq has joined #openstack-nova | 22:11 | |
*** markvoelker has joined #openstack-nova | 22:11 | |
*** slaweq has quit IRC | 22:16 | |
openstackgerrit | Merged openstack/nova stable/pike: Fix misuse of nova.objects.base.obj_equal_prims https://review.opendev.org/676292 | 22:40 |
*** markvoelker has quit IRC | 22:40 | |
openstackgerrit | Adam Spiers proposed openstack/nova master: Track libvirt host/domain capabilities for multiple machine types https://review.opendev.org/673151 | 22:40 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | 22:41 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | 22:41 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | 22:41 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py https://review.opendev.org/636318 | 22:41 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 22:41 |
openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 22:41 |
*** prometheanfire has quit IRC | 22:42 | |
*** ivve has quit IRC | 22:46 | |
*** tkajinam has joined #openstack-nova | 23:00 | |
*** slaweq has joined #openstack-nova | 23:11 | |
*** slaweq has quit IRC | 23:16 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova stable/stein: Fix non-existent method of Mock https://review.opendev.org/676838 | 23:28 |
openstackgerrit | Merged openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611 | 23:39 |
openstackgerrit | Merged openstack/nova master: Fix non-existent method of Mock https://review.opendev.org/675041 | 23:40 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: requirements: move to using psycopg2-binary https://review.opendev.org/676684 | 23:44 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: constraints: bump psycopg2 to 2.8 https://review.opendev.org/676725 | 23:44 |
openstackgerrit | Mohammed Naser proposed openstack/nova master: constraints: bump psycopg2 to 2.8 https://review.opendev.org/676725 | 23:44 |
*** markvoelker has joined #openstack-nova | 23:45 | |
*** igordc has joined #openstack-nova | 23:46 | |
*** spatel has joined #openstack-nova | 23:48 | |
brinzhang | mriedem: it's not matter, thanks. | 23:49 |
*** markvoelker has quit IRC | 23:50 | |
*** mlavalle has quit IRC | 23:51 | |
*** spatel has quit IRC | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!