| *** esberglu has joined #openstack-powervm | 00:51 | |
| *** esberglu has quit IRC | 01:00 | |
| *** esberglu has joined #openstack-powervm | 01:05 | |
| *** esberglu has quit IRC | 01:10 | |
| *** esberglu has joined #openstack-powervm | 01:35 | |
| *** esberglu has quit IRC | 01:40 | |
| *** esberglu has joined #openstack-powervm | 01:45 | |
| *** esberglu has quit IRC | 01:50 | |
| *** esberglu has joined #openstack-powervm | 02:15 | |
| *** esberglu has quit IRC | 02:19 | |
| *** esberglu has joined #openstack-powervm | 02:50 | |
| *** esberglu has quit IRC | 02:55 | |
| *** esberglu has joined #openstack-powervm | 03:05 | |
| *** esberglu has quit IRC | 03:10 | |
| *** esberglu has joined #openstack-powervm | 03:55 | |
| *** esberglu has quit IRC | 04:00 | |
| *** esberglu has joined #openstack-powervm | 04:50 | |
| *** esberglu has quit IRC | 04:55 | |
| *** esberglu has joined #openstack-powervm | 05:15 | |
| *** esberglu has quit IRC | 05:20 | |
| *** esberglu has joined #openstack-powervm | 05:35 | |
| *** esberglu has quit IRC | 05:40 | |
| *** esberglu has joined #openstack-powervm | 06:00 | |
| *** esberglu has quit IRC | 06:05 | |
| *** esberglu has joined #openstack-powervm | 06:25 | |
| *** esberglu has quit IRC | 06:30 | |
| *** esberglu has joined #openstack-powervm | 07:00 | |
| *** esberglu has quit IRC | 07:05 | |
| *** esberglu has joined #openstack-powervm | 07:25 | |
| *** esberglu has quit IRC | 07:30 | |
| *** esberglu has joined #openstack-powervm | 07:40 | |
| *** esberglu has quit IRC | 07:46 | |
| *** esberglu has joined #openstack-powervm | 08:25 | |
| *** esberglu has quit IRC | 08:29 | |
| *** esberglu has joined #openstack-powervm | 08:35 | |
| *** esberglu has quit IRC | 08:40 | |
| *** esberglu has joined #openstack-powervm | 08:55 | |
| *** esberglu has quit IRC | 09:00 | |
| *** esberglu has joined #openstack-powervm | 09:25 | |
| *** esberglu has quit IRC | 09:31 | |
| *** edmondsw has joined #openstack-powervm | 12:45 | |
| *** esberglu has joined #openstack-powervm | 13:23 | |
| *** edmondsw has quit IRC | 13:29 | |
| *** edmondsw has joined #openstack-powervm | 13:36 | |
| *** edmondsw has quit IRC | 13:41 | |
| *** edmondsw has joined #openstack-powervm | 13:43 | |
| efried | edmondsw: You gonna propose nova-powervm RC1 today? | 16:52 |
|---|---|---|
| efried | *-powervm I guess | 16:52 |
| edmondsw | efried yes | 16:54 |
| edmondsw | efried I'm trying to get the docs fixed first | 17:09 |
| efried | ight | 17:10 |
| efried | just add me to 'em and/or ping me when ready. | 17:10 |
| edmondsw | efried https://review.openstack.org/#/c/590444/ | 17:27 |
| efried | edmondsw: Oh, we're giving up (deferring) integrating docs build? | 17:28 |
| edmondsw | not necessarily, but I wanted to throw this up first anyway in case we can't get the rest done | 17:29 |
| efried | do you have quick links where I can validate that these numbers are right, and that openstackci is properly registered? | 17:30 |
| efried | and/or does this patch somehow build and post results where I can see the rtd content properly generated? Or do I have to take it on faith? | 17:31 |
| efried | edmondsw: Okay, got it all sussed. +1. | 17:47 |
| edmondsw | efried I've figured out how to tell rtd not to build pdf, so trying that now | 17:57 |
| efried | edmondsw: By commenting out the latex section in doc/source/conf.py? | 17:57 |
| efried | or something you can configure on rtd itself? | 17:57 |
| edmondsw | no... unchecking something in rtd | 17:57 |
| efried | edmondsw: Well, I tried to follow the build process locally and didn't get far. | 18:11 |
| edmondsw | my rtd build just passed | 18:12 |
| edmondsw | so unchecking the boxes for building pdfs and epubs fixed it | 18:12 |
| efried | is that an acceptable solution? | 18:12 |
| edmondsw | well, and changing it to use doc/requirements instead of test-requirements.txt | 18:12 |
| edmondsw | I don't think we need pdf and epub | 18:12 |
| edmondsw | I wouldn't even know where to find them | 18:12 |
| efried | edmondsw: It what? It should have been doing that already. | 18:13 |
| edmondsw | rtd was not updated when we changed our code to split out doc/requirements | 18:13 |
| efried | If you look at the 7th entry in https://readthedocs.org/projects/nova-powervm/builds/7615137/ | 18:13 |
| edmondsw | efried that was one of the builds I just ran where I had changed that | 18:14 |
| efried | could it be that that was the whole problem? | 18:14 |
| efried | oh, I guess not then. | 18:14 |
| efried | Well, at least short term I'm fine with nixing pdf/epub. Would be interested to know whether other projects are doing that. | 18:14 |
| edmondsw | so it's a two-fold problem... | 18:15 |
| edmondsw | 1) apparently the recent doc changes we/stephenfin made don't work for pdfs, so we had to disable that | 18:15 |
| edmondsw | 2) when he split out doc/requirements.txt we needed to update rtd to match | 18:15 |
| edmondsw | the downside of #2 is that rtd doesn't let you specify that per branch... so by fixing it for master we break it for all our stable branches | 18:15 |
| edmondsw | but we don't update docs for stable branches very often | 18:16 |
| edmondsw | if/when we do, we'll need to remember to temporarily change things back to test-requirements.txt, run a build, and then change it back for the sake of master | 18:16 |
| efried | edmondsw: https://readthedocs.org/projects/ceilometer-powervm/downloads/ is where the pdf would be, if there was one. | 18:16 |
| efried | and epub | 18:16 |
| edmondsw | efried most users wouldn't have access to that, though, I don't think... requires login, right? | 18:17 |
| edmondsw | maybe not, I don't know | 18:17 |
| efried | I was about to ask. But yes, you should still be able to access it without login. | 18:17 |
| efried | So, what other projects exist in rtd that we can go look at? | 18:17 |
| edmondsw | yeah, I can still see that link after I logout | 18:18 |
| edmondsw | efried I'm not sure what other projects use it | 18:18 |
| edmondsw | I can check the projects yaml, hold on | 18:18 |
| efried | https://readthedocs.org/projects/nova-zvm/downloads/ | 18:18 |
| efried | their latest build was 5mo ago though. | 18:18 |
| efried | https://readthedocs.org/projects/ - there's only about 83,600-ish projects. | 18:20 |
| edmondsw | efried https://readthedocs.org/projects/ironic-staging-drivers/downloads/ | 18:20 |
| efried | okay, so theirs works. | 18:20 |
| efried | pdflatex sounds like a bowel disorder | 18:21 |
| efried | Hah, especially with -interaction=nonstopmode | 18:21 |
| edmondsw | efried lol | 18:22 |
| edmondsw | efried I suspect they are not using openstackdocstheme? | 18:22 |
| edmondsw | our problems started when we moved to that | 18:22 |
| edmondsw | if i remember correctly | 18:22 |
| efried | edmondsw: True story, assuming openstackdocstheme would be a thing listed in doc/source/conf.py | 18:23 |
| efried | ironic-staging-drivers' conf.py does not list it. | 18:23 |
| efried | so | 18:24 |
| efried | what do we do? | 18:24 |
| edmondsw | for the moment I think we live without PDF and EPub | 18:25 |
| edmondsw | and then long term we try to get off RTD entirely | 18:26 |
| efried | wfm. So does that mean we're ready to cut RC? | 18:27 |
| efried | yeah, so the zuul output from https://review.openstack.org/#/c/590444/ isn't worth anything for this. | 18:29 |
| edmondsw | efried https://review.openstack.org/#/c/590475/ | 18:33 |
| efried | edmondsw: Why does nova-powervm have tarball_base but the others don't? | 18:35 |
| edmondsw | tarball_base? | 18:36 |
| edmondsw | oh... | 18:36 |
| edmondsw | because y'all misnamed it in... let me find it... | 18:36 |
| edmondsw | setup.cfg | 18:37 |
| efried | eh? | 18:37 |
| edmondsw | _ when it should have been - | 18:37 |
| edmondsw | going to eat | 18:38 |
| efried | okay. I still don't understand ^ but no matter. ttyl. | 18:38 |
| efried | edmondsw: ? | 19:56 |
| edmondsw | efried yes? | 19:57 |
| efried | edmondsw: I've set up a session where I can fiddle with a flavor and see what lpar xml comes out the other side. | 19:58 |
| efried | If you're busy, I can just hack at it myself. But if you're available to bounce stuff off of... | 19:58 |
| edmondsw | pretty busy, but I'll push it aside if you need me | 19:58 |
| efried | edmondsw: okay, don't worry about it then. I think I'll just dump stuff in here for the sake of posterity, and you can read it if you care, whenever you get a chance. If I need your brain I'll tag you. Cool? | 20:01 |
| edmondsw | sounds good | 20:01 |
| efried | First run through, I set flavor.vcpus=4 and no extra specs, and ended up with | 20:03 |
| efried | <uom:DesiredProcessingUnits>0.40</uom:DesiredProcessingUnits> | 20:03 |
| efried | <uom:DesiredVirtualProcessors>4</uom:DesiredVirtualProcessors> | 20:03 |
| efried | <uom:MaximumProcessingUnits>0.40</uom:MaximumProcessingUnits> | 20:03 |
| efried | <uom:MaximumVirtualProcessors>4</uom:MaximumVirtualProcessors> | 20:03 |
| efried | <uom:MinimumProcessingUnits>0.40</uom:MinimumProcessingUnits> | 20:03 |
| efried | <uom:MinimumVirtualProcessors>4</uom:MinimumVirtualProcessors> | 20:03 |
| efried | This is in SharedProcessorConfiguration | 20:03 |
| efried | General note: the actual inventory count we're going to consume from placement is always going to be what's in flavor.vcpus. We have no say over that, and that's going to be a problem. | 20:04 |
| efried | In this particular case, the actual resources we're consuming from the system: 0.4 of a physical proc; so if we had an allocation ratio of 10.0 (1/CONF.powervm.proc_units_factor) then we would be dead on. | 20:05 |
| efried | However... | 20:05 |
| efried | when I set extra_specs['powervm:proc_units']=5, all the 0.40s change to 5.00s. So we're still asking the LPAR for 4 VCPUs (meaning I suppose that the guest OS will see four procs) and we're still consuming 4 VCPUs as far as placement is concerned, but we're actually consuming five full procs from the system. | 20:06 |
| efried | TL;DR: setting proc_units totally borks our allocation ratio calculation. | 20:07 |
| efried | edmondsw: So this is where it might be useful to know things about pvc and real deployments. Does pvc use proc_units? All the time, most of the time, rarely? And can end users affect whether it's used or not? | 20:07 |
| edmondsw | yes, PowerVC specifies proc_units in every flavor | 20:08 |
| edmondsw | an admin (flavor creation is restricted to admin) can set max and min as well, optionaly, but must at least set desired proc units | 20:09 |
| efried | wow | 20:09 |
| edmondsw | tell me again what this allocation ratio is? | 20:11 |
| edmondsw | proc units is one of the strengths of the PowerVM architecture, allowing you to get much better proc usage than you can on x86, where processing power is going to waste | 20:12 |
| edmondsw | because x86 doesn't let you get that granular | 20:12 |
| edmondsw | in your example above, when you got 0.4 it was because the default conf setting for proc_units_factor is 0.1 and then multiple that by 4 vcpus to get 0.4 proc units | 20:14 |
| efried | yes yes, but you still have full control over the number of proc units by combining proc_units_factor and the default behavior of the VCPUs field. | 20:15 |
| edmondsw | and then when you started specifying proc_units you get exactly what you ask for... no multiplcation | 20:15 |
| efried | so it may be wonderful for talking directly to power, but for openstack it sucks ass. | 20:15 |
| efried | because accounting is just totally out the window. | 20:15 |
| edmondsw | why? I'm not following | 20:15 |
| efried | Next data point: proc_units is only used for shared proc config. | 20:15 |
| efried | ignored when dedicated_proc is True. | 20:16 |
| edmondsw | right... why would you want a dedicated proc and then not use the whole thing? | 20:16 |
| edmondsw | since it's dedicated, anything you don't use is going to waste | 20:16 |
| efried | And I'm assuming that | 20:16 |
| efried | <uom:DesiredProcessors>4</uom:DesiredProcessors> | 20:16 |
| efried | <uom:MaximumProcessors>4</uom:MaximumProcessors> | 20:16 |
| efried | <uom:MinimumProcessors>4</uom:MinimumProcessors> | 20:16 |
| efried | means four whole procs. | 20:16 |
| efried | yeah, I guess I'm confusing "dedicated" with the minimum on sharing proc config. | 20:16 |
| edmondsw | I'm not sure if uom:DesiredProcessors is num vcpus or num proc units | 20:17 |
| edmondsw | I would have thought vcpus | 20:17 |
| efried | I am, that's why I'm experimenting. Yes, it's VCPUs. proc units is ignored. | 20:17 |
| edmondsw | but I don't keep that in my head | 20:17 |
| efried | Anyway, the effect on accounting is that, when dedicated_proc=True, allocation_ratio is exactly 1. | 20:18 |
| efried | edmondsw: does pvc allow dedicated mode? | 20:18 |
| edmondsw | yes | 20:19 |
| efried | in which case someone may be surprised that proc_units is being ignored :) Or maybe you've got the software set up properly to hide the fact that the underlying field is actually proc_units, so it does the right thing anyway. | 20:19 |
| efried | or... maybe this is a bug | 20:19 |
| efried | so | 20:20 |
| efried | I started off thinking we were going to be able to calculate a sane allocation ratio for procs and set it dynamically from the driver. | 20:20 |
| edmondsw | the PowerVC GUI won't let you specify proc_units if you select the dedicated procs option | 20:20 |
| efried | okay, that's good then. Somebody figured this out. | 20:21 |
| efried | ...Now I'm thinking we just freaking use the same thing nova uses, which is a conf setting for allocation ratio. | 20:21 |
| efried | Let the admin set it based on the flavors they think they'll be using mostly | 20:21 |
| edmondsw | I think allocation ratio is a very x86-specific concept from what I'm reading here: https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html | 20:21 |
| edmondsw | I don't think this form of overcommit exists for PowerVM | 20:22 |
| efried | well. | 20:22 |
| edmondsw | it's my understanding that on PowerVM, if you have a proc_unit then you have a proc_unit... nobody else can be using that proc_unit | 20:23 |
| edmondsw | mdrabe that right? | 20:23 |
| efried | for sharing, there's min, desired, and max. | 20:23 |
| edmondsw | let's talk min and max for a second | 20:23 |
| efried | min means nobody else can have 'em. | 20:23 |
| efried | max - min is where it floats depending on how busy the system is. | 20:24 |
| edmondsw | no min is how much you could DLPAR (live resize) down to | 20:24 |
| mdrabe | edmondsw: I think it depends on if the LPAR is configured for shared procs or not | 20:24 |
| mdrabe | If it's in dedicated mode, then the LPAR should be entitled to the full proc unit | 20:24 |
| efried | there's still desired, min, and max for dedicated though | 20:25 |
| mdrabe | min and max are for DLPAR | 20:25 |
| mdrabe | active resize | 20:25 |
| edmondsw | there is one other usage of min... if a proc dies and the system has to try to keep as many LPARs running as it can, it may run you at your min to free up resources for other things | 20:25 |
| edmondsw | but it's not an overcommit thing in the x86 sense | 20:25 |
| efried | this doesn't sound right. I thought the point of shared procs was that the system can dynamically change how much compute power each vcpu has under the covers according to the load on the system. | 20:26 |
| edmondsw | max is how much you can DLPAR up to, and I believe it's also a cap of how much additional CPU you might be allowed to use if there is free CPU on the system | 20:27 |
| edmondsw | that's where the sharing comes in | 20:27 |
| efried | you still have four VCPUs, but if your min proc units is 0.4 and your max is 4.0 it means each VCPU could have anywhere from 0.1 to 1.0 worth of real proc moxy at any given time. | 20:27 |
| edmondsw | no, I don't think that's how it works | 20:28 |
| efried | there are six fields for sharing. desired/min/max * proc_units/VCPUs | 20:28 |
| efried | I thought the proc_units part is the dynamic under-the-covers thing; and the VCPUs part is the DLPAR thing. | 20:29 |
| edmondsw | efried these are some links I bookmarked... see if they help: | 20:29 |
| edmondsw | https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/Virtualization+Concepts | 20:29 |
| edmondsw | http://ibmsystemsmag.com/ibmi/administrator/lpar/lpar_settings/ | 20:29 |
| edmondsw | DLPAR can be used to change either VCPUs or proc units or both | 20:30 |
| efried | that's changing the actual lpar profile. | 20:30 |
| efried | What do the six fields mean in the context of a single lpar profile? | 20:30 |
| edmondsw | they mean what I was saying above | 20:31 |
| efried | (sorry, I shouldn't say profile, I don't mean actual LPAR profile; I just mean settings) | 20:31 |
| edmondsw | desired is what you get. | 20:31 |
| edmondsw | min in is what you could dlpar down to and how much you're saying you have to have at min to stay running in an emergency where a CPU dies. | 20:31 |
| edmondsw | max is what you could dlpar up to and you might be allowed to use more up to that cap without using DLPAR if there are free cycles | 20:32 |
| edmondsw | ^ is for proc units | 20:33 |
| efried | yup, that gels with what I was saying. | 20:33 |
| edmondsw | for vcpus, desired is what you get, min and max are what you could dlpar to | 20:33 |
| edmondsw | s/to/between/ | 20:33 |
| efried | so back to allocation ratio... | 20:34 |
| efried | the only sane way to think about actual resource usage in shared mode is via desired proc units. | 20:35 |
| efried | BUT | 20:35 |
| efried | the only thing placement knows about is desired vcpus. | 20:35 |
| efried | in dedicated mode it's desired procs and alloc ratio of 1 | 20:36 |
| efried | BUT | 20:36 |
| efried | placement doesn't know about shared vs. dedicated. | 20:36 |
| edmondsw | efried where do you actually specify allocation ratio? | 20:38 |
| edmondsw | https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html doens't say (bug?) | 20:39 |
| efried | Today, it's done via CONF.cpu_allocation_ratio | 20:39 |
| efried | which defaults to 16.0 | 20:39 |
| efried | but not via the conf option default | 20:39 |
| edmondsw | I think placement would need different logic for powervm... ignoring CONF.cpu_allocation_ratio and using proc units | 20:41 |
| efried | but allocation ratio is a static(ish) thing on the inventory. It doesn't change for each instance or whatever. | 20:42 |
| edmondsw | doesn't the virt driver tell placement how much cpu and proc_units it has free? And then placement should match both of those up with the flavor? | 20:42 |
| edmondsw | I'm not saying allocation ratio should change per instance... I'm saying it shouldn't be used for powervm hosts | 20:43 |
| efried | so | 20:43 |
| efried | we could just use #physical procs / 0.05 as our VCPU inventory count, and an allocation ratio of 1. | 20:44 |
| efried | or make sure the real allocation is always for the number of proc units | 20:44 |
| efried | the problem isn't that we couldn't do the math. The problem is that we don't *get* to do the math. | 20:44 |
| efried | it's out of our hands - in the compute manager / placement, not in the virt driver. | 20:45 |
| edmondsw | somehow it needs to be put back in our hands unless they want to have powervm-specific logic in nova proper rather than in the virt driver | 20:45 |
| edmondsw | I don't see another way | 20:45 |
| efried | placement allocation is done based on flavor.vcpus, end of story. And it uses allocation ratio to determine when inventory is "full" when trying to schedule. | 20:46 |
| efried | well, the other way is for us to completely change how we interpret flavors. | 20:47 |
| edmondsw | ? | 20:48 |
| efried | - Use the conf option for allocation ratio to inform the micropartitioning size (e.g. 20.0 => 0.05) | 20:50 |
| efried | - VCPUs will always give you a number of physical procs corresponding to #VCPUs * that micropartitioning size | 20:50 |
| efried | - If you flip on 'dedicated', then VCPUs / allocation_ratio had better be a whole number or we'll error | 20:50 |
| edmondsw | that won't work | 20:53 |
| edmondsw | wouldn't allow you to have some VMs using dedicated and other using shared | 20:53 |
| efried | eh, sure it would. You can still use the powervm:dedicated_proc in the flavor | 20:53 |
| edmondsw | wouldn't allow you to have some VMs using more proc units per vcpu than others | 20:53 |
| edmondsw | etc | 20:53 |
| efried | yes, true story. I'm not sure why that's a problem, but taking it on faith that it is... | 20:54 |
| efried | we should therefore have the override for the number of guest VCPUs be part of the extra specs instead of coming from the outer flavor.vcpus | 20:54 |
| efried | that ought to do the trick. | 20:55 |
| edmondsw | I expect that would cause other problems... | 20:56 |
| efried | for code, or for users? | 20:56 |
| edmondsw | yes | 20:56 |
| edmondsw | separate topic if I can interject... #openstack-release is saying we need to create branches | 20:57 |
| edmondsw | I thought that happened automatically based on the rc1 commit I threw up, but no? | 20:57 |
| edmondsw | do you know how to create branches? | 20:57 |
| efried | oh, crap. | 20:58 |
| edmondsw | nm, I think I know what I need to do | 20:59 |
| edmondsw | I just don't have that commit right yet | 20:59 |
| edmondsw | when it's right, it will create the branchees | 20:59 |
| efried | um, branches are just branches, but I think we may need to tag. esberglu is here, maybe he can advise. (And we can put it in a doc for future ref) | 20:59 |
| edmondsw | I remembered / was reminded... I think the commit is updated to do that now | 21:03 |
| edmondsw | https://review.openstack.org/#/c/590475/ | 21:04 |
| esberglu | LGTM | 21:07 |
| efried | +1 | 21:08 |
| efried | so | 21:09 |
| efried | other than making users understand the new way to use extra specs, how would it be problematic? | 21:09 |
| efried | the other option to consider is dynamically rebalancing our inventory based on existing instances | 21:10 |
| efried | which is something I was going to bring up anyway for calculating reserved values... | 21:11 |
| edmondsw | it would be problematic in that you lose capabilities like I described above | 21:18 |
| efried | no, I addressed that | 21:18 |
| edmondsw | with something that doesn't work... | 21:19 |
| edmondsw | maybe if you were clearer on what you're suggesting | 21:19 |
| edmondsw | efried interjecting again... how do I use pdb in a tox run since we moved to stestr? | 21:21 |
| efried | Okay, with: | 21:22 |
| efried | CONF.cpu_allocation_ratio = 20.0 <== corresponds to micropartitioning granularity of 0.05 | 21:22 |
| efried | flavor.vcpus = 40 <== multiplied by 0.05 means you will get 2.0 proc units from the platform *and* the allocation will reduce your inventory by 40 | 21:22 |
| efried | and if you need: | 21:22 |
| efried | extra_specs['powervm:how_many_vcpus_do_you_want_the_vm_to_see'] = 2 <== so each vcpu will correspond to 1.0 processor unit | 21:22 |
| efried | edmondsw: I use remote_pdb these days. | 21:22 |
| edmondsw | efried that won't work on a lot of levels. It's not backward compatible. Nova is going to think you have 40 vcpus when you really have 2, etc. | 21:24 |
| efried | Well, that latter thing is the whole point of doing it this way. | 21:25 |
| edmondsw | the problem here is placement... allocation ratios is an x86-specific concept and needs to be replaced with something that is not x86-specific | 21:25 |
| efried | with an allocation ratio of 20.0 and a system with 8 physical procs, nova/placement will allow you to create allocations with VCPU total of 8 x 20 = 160. | 21:26 |
| edmondsw | nobody wants a VM with 160 VCPUs | 21:26 |
| efried | Great, so they can specify that extra spec. | 21:27 |
| edmondsw | which will not help... nova will still say you have 160 VCPUs | 21:27 |
| edmondsw | well it helps, but not enough | 21:27 |
| efried | Yeah, it's still holey; I guess you're deciding which part you want to lie about. | 21:27 |
| edmondsw | you're trying to cram a square peg into a round hole | 21:27 |
| efried | cause right now, there's no way NOT to lie about inventory/usage. | 21:28 |
| edmondsw | because allocation ratio is an x86-specific concept (and not a good one even there) that needs to die | 21:28 |
| edmondsw | when I say x86-specific, I mean it's a nova+x86-specific concept | 21:28 |
| edmondsw | it's not purely x86 | 21:29 |
| edmondsw | there is no reason that nova should have to have that concept for x86 | 21:29 |
| edmondsw | x86 users should have more flexibility than they have with the current allocation ratio system | 21:29 |
| edmondsw | where they're limited by a conf value that controls the whole system instead of the ability to ask for what they want on a per-vm basis via flavor | 21:29 |
| efried | so a few things there. | 21:30 |
| efried | The Power perspective on this is very not-cloudy. | 21:30 |
| edmondsw | lol | 21:30 |
| edmondsw | no it's not | 21:30 |
| edmondsw | and I mean that in disagreement | 21:30 |
| edmondsw | it's perfectly cloudy | 21:31 |
| efried | It's pet-not-cattle | 21:31 |
| edmondsw | no, it has nothing to do with pet/cattle | 21:31 |
| edmondsw | nothing whatsoever | 21:31 |
| efried | the allocation ratio as it's being used on x86 is allowing the admin to set up in one broad stroke how much moxy she wants a VCPU to have. | 21:32 |
| efried | 1.0 means on physical proc to one VCPU. | 21:32 |
| efried | s/on/one/ | 21:32 |
| edmondsw | remember that admins control flavors... regular users will have no control over what they're getting in any model | 21:32 |
| efried | except by picking a flavor. | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ceilometer-powervm stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/590527 | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ceilometer-powervm stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590528 | 21:34 |
| edmondsw | which an admin created... so the admin controls what your options are | 21:34 |
| edmondsw | that is perfectly cloudy | 21:34 |
| efried | right | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/networking-powervm stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/590529 | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/networking-powervm stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590530 | 21:34 |
| efried | no | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova-powervm stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/590531 | 21:34 |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova-powervm stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590532 | 21:34 |
| efried | the not cloudy part is the part where you're saying a precise number of proc units you want. On one system that's a lot of moxy, on a different system, less. | 21:35 |
| efried | allocation ratio is a way to normalize that picture across your cloud. | 21:36 |
| efried | i.e. the ratio of all your allocation ratios makes all your hosts "equal" in terms of the moxy of one "VCPU" | 21:36 |
| edmondsw | if that's not cloudy, then the whole concept of flavors is not cloudy. Every VM should have the same amount of mem and cpu and disk | 21:37 |
| efried | swhat I'm saying. If you deploy a flavor on one host, it should have the same amount of moxy as same flavor deployed on a different host. | 21:38 |
| edmondsw | and it does in any model | 21:38 |
| edmondsw | we're not talking about per-host changes here | 21:38 |
| efried | uh, not with proc_units | 21:38 |
| edmondsw | yes, with proc_units | 21:38 |
| edmondsw | there's nothing host-specific here | 21:38 |
| edmondsw | you're saying you don't like our flavor extra-specs | 21:38 |
| edmondsw | not a host setting | 21:38 |
| efried | proc_units=2 on a P7 is going to give you a weaker VM than proc_units=2 on a P9 | 21:39 |
| efried | or whatever. | 21:39 |
| edmondsw | vcpus =1 on a skylake is oging to give you less power than vcpus=1 on the previous generation | 21:39 |
| efried | not if you've set your allocation ratios properly. | 21:39 |
| efried | is exactly my point. | 21:39 |
| edmondsw | wrong | 21:39 |
| edmondsw | there is no way to control that with allocation ratios even for x86 | 21:40 |
| edmondsw | a skylake core != a broadwell core | 21:40 |
| efried | so if a proc on host X has twice as much power as a proc on host Y, set the allocation ratio on Y to twice what it is on X. | 21:40 |
| edmondsw | it doesn't work like that | 21:41 |
| efried | Then when you ask for one VCPU, you're getting twice as many proc units on host X as you are on host Y, ergo the same amount of moxy. | 21:41 |
| efried | no, it sure doesn't today. Swhat I'm saying, we could make it so it does. | 21:41 |
| efried | and, btw, that is (ish) how it works with the x86 allocation ratio. It only really comes into play on a crowded system, but at that point it's effectively as I've described. | 21:42 |
| edmondsw | there is no way to calculate that a proc on host X has twice as much power as a proc on host Y... it's not that cut and dry. It will be better at some things than at others in different proportions, etc... | 21:42 |
| efried | yeah, I get that it's not an exact science, but it's up to the admin to figure out a close enough approximation and set the alloc ratios accordingly. | 21:42 |
| edmondsw | and I really don't think that's what allocation_ratio is used for from what I read | 21:43 |
| edmondsw | nova docs tout it as an overcommit feature, which would be an entirely different usage from what you're saying | 21:43 |
| efried | As it stands today, if I've specified proc_units in my flavor and I deploy it on a heterogeneous Power cloud, I have no idea how much actual compute power I'm going to get. So if I'm deploying a workload that needs to perform to a certain level, I need to tie it to a specific host. <= not cloudy. | 21:44 |
| * efried opens link again to make sure reading same doc as edmondsw | 21:44 | |
| efried | "increase the number of instances running on your cloud at the cost of reducing the performance of the instances" | 21:45 |
| efried | difference is that in the x86 model, that degradation only hits when you get crowded. (though on power, with uncapped shared procs, it would be similar) | 21:46 |
| efried | this is pretty academic, though. Because in the near term, we are not going to make major changes to how we interpret flavors for power; and we are not going to get nova to make changes to how allocations are calculated for our "boutique compute driver". | 21:48 |
| efried | So we just need to decide what we're going to do with the allocation ratio and reserved amounts right now, given the existing framework. | 21:48 |
| efried | My vote is to keep the behavior as it is now, which is: | 21:55 |
| edmondsw | I believe the typical answer to what you're talking about is to put different types of systems in different cells and then request a specific cell | 21:55 |
| efried | - For VCPU, use CONF.cpu_allocation_ratio and default it to 16.0 if unspecified. | 21:55 |
| efried | - MEMORY_MB: CONF.ram_allocation_ratio, default to 1.5 | 21:55 |
| efried | - DISK_GB: CONF.disk_allocation_ratio, default to 1.0 | 21:55 |
| efried | and we can try to improve upon that in the future. | 21:55 |
| edmondsw | there are other ways | 21:56 |
| edmondsw | "use" in what sense? | 21:56 |
| efried | in the sense of specifying it to placement in the inventory record's `allocation_ratio` field. | 21:57 |
| efried | because that's what happens today | 21:57 |
| efried | because the resource tracker uses get_available_resource plus those conf values to generate placement inventory records for us | 21:58 |
| efried | because we have not implemented get_inventory or update_provider_tree yet. | 21:58 |
| edmondsw | so... change nothing? | 21:58 |
| efried | once we implement those, that automatic generation of inventory records goes away; we're responsible for doing it. | 21:58 |
| efried | yeah, change nothing behaviorally by duplicating the existing logic in our update_provider_tree impl. | 21:59 |
| efried | i.e. start explicitly lying about inventory in the same way as we were implicitly lying about it before. | 21:59 |
| efried | which seemed worse to me, which is why I wanted to have this conversation. | 21:59 |
| edmondsw | you've got me worried now that the existing logic is fatally flawed | 21:59 |
| efried | yeah. It is. It's completely bogus. Unequivocally. | 21:59 |
| efried | Today if that conf option is not specified, we are allowing the nova scheduler to attempt to schedule instances to a host for a total of (# physical procs * 16) VCPUs. | 22:00 |
| efried | which is completely arbitrary and guaranteed to be wrong almost no matter what kind of Power-ish flavor is being used. | 22:01 |
| edmondsw | change that to (# physical procs * 20) VCPUs? | 22:01 |
| efried | now, if we go over *actual* capacity, we'll fail the deploy from within spawn(). Which will cause a reschedule. Which, as of I think Queens, will retry up to two other hosts in the same cell before dying. | 22:02 |
| edmondsw | otherwise someone using 0.05 proc units per VCPU is going to start getting issues where the scheduler thinks it's full and it's not | 22:02 |
| edmondsw | efried correct | 22:02 |
| efried | why change it? But if we're going to change it, proc_units_factor defaults to 0.1, so we should change it to 1 / proc_units_factor. | 22:02 |
| efried | Which will at least make us correct if no flavors use dedicated procs or proc_units. | 22:03 |
| edmondsw | "defaults", not "min" | 22:03 |
| edmondsw | min is 0.05 | 22:03 |
| efried | right, but we'll never use 0.05 | 22:03 |
| edmondsw | yeah we would... why don't you think so? | 22:03 |
| efried | unless proc_units_factor is 0.05 *and* the flavor doesn't say dedicated_proc=True or proc_units=X | 22:04 |
| edmondsw | remember proc_units_factor isn't used if you specify proc_units in the flavor extraspec | 22:04 |
| efried | exactly my point. | 22:04 |
| edmondsw | so you can't base on proc_units_factor | 22:04 |
| edmondsw | because it may not be used | 22:04 |
| efried | If dedicated_proc=True and/or proc_units is specified in the flavor, all bets are off and we have no way to calculate a reasonable allocation ratio at all. | 22:04 |
| efried | using proc_units_factor is the only thing we could possibly attempt to do dynamically (without "Solution Rebalance" - discussion for later). | 22:05 |
| edmondsw | my point of using 0.05 is that it's the min | 22:05 |
| edmondsw | so scheduler is never saying "you're full" when you might not be | 22:05 |
| edmondsw | I guess if we use proc_units_factor we can tell folks to change that to 0.05 to avoid that issue | 22:07 |
| efried | this on the off chance that you've said proc_units=0.05 somewhere? | 22:07 |
| edmondsw | annoying as it is to have to change on every host | 22:07 |
| edmondsw | I'd say it's more than an off chance, but yes | 22:07 |
| efried | Look, right now with the default of 16.0, that corresponds to a micropartition of 0.0625, which is close enough to 0.05 for government work. | 22:08 |
| efried | Unless you really envision a scenario where *all* the VMs deployed on a host are going to be using 1VCPU=0.05pproc *and* we're going to get real close to filling that host up, that seems adequate for me. | 22:09 |
| efried | Cause we also need to talk about reserved. Which again, nova is setting for us based on conf values, and which again have zero relation to what we should actually be reserving (for the nvl/VIOSes and any OOB LPARs). | 22:11 |
| efried | rn those conf values default to 0 for disk and cpu, 512 for mem. | 22:14 |
| efried | and btw, they reserve cpu equal to the amount of that conf value, not equal to that x allocation_ratio, yet the doc states it's a physical proc. For us, we would have to figure the calculation according to the number of actual physical procs, factored against whatever allocation ratio we set. | 22:16 |
| edmondsw | why aren't they reserving based on what is actually used? | 22:20 |
| edmondsw | maybe they mean something different for reserved than I'm thinking? | 22:21 |
| efried | "They" who? | 22:24 |
| efried | The reserved amount is supposed to represent the amount of processing power the "hypervisor" is using. | 22:25 |
| edmondsw | ok, that's what I was thinking it should mean | 22:25 |
| efried | In the libvirt model, the hypervisor is just... running. | 22:25 |
| efried | it's using whatever proc/mem it can get its hands on. sharing it all with the VMs. | 22:25 |
| edmondsw | true | 22:25 |
| efried | So by setting a reserved value you're essentially limiting the number of VMs so that there's so many of them that it prevents the hypervisor stuff from running properly. | 22:25 |
| edmondsw | yep | 22:26 |
| edmondsw | same applies for PowerVM | 22:26 |
| edmondsw | of course we have the NovaLink LPAR to factor into the equation, but there is still phyp beyond that | 22:26 |
| efried | hm, I thought whatever phyp uses doesn't count. | 22:27 |
| edmondsw | why wouldn't it count? | 22:27 |
| efried | like, I thought whatever phyp exposes to us is already whatever's left over after it has taken what it needs. | 22:27 |
| edmondsw | and of course there may be VIOS as well | 22:27 |
| efried | So like, I thought we were just gonna add up the nvl and VIOSes and OOB partitions. | 22:27 |
| edmondsw | that would be news to me... maybe | 22:28 |
| efried | anyway, for now I think we should again just duplicate the existing bogosity and leave TODOs to do things properly later. | 22:28 |
| edmondsw | efried I've pip installed and pip3 installed remote_pdb, and even tox -r, but I'm still getting import errors on it... | 22:31 |
| edmondsw | any ideas? | 22:31 |
| efried | You installed it in your venv? | 22:32 |
| efried | when you tox -r you would be blowing it away. | 22:32 |
| edmondsw | I can't find a venv... I assumed tox must be doing that under the covers now | 22:32 |
| efried | Here's how I do it: | 22:32 |
| edmondsw | there used to be a .venv directory, but no more | 22:32 |
| efried | bash # create a new shell so I can ^D back to sanity | 22:33 |
| efried | . .tox/py27/bin/activate # enter the venv | 22:33 |
| efried | pip install remote_pdb | 22:33 |
| efried | ^D | 22:33 |
| efried | what repository are you working in? | 22:33 |
| edmondsw | nova | 22:33 |
| edmondsw | ok, I think you told me what I needed | 22:33 |
| edmondsw | activate is in .tox/py27/bin | 22:33 |
| efried | yes. But I think we did combine the venvs recently, so depending what actual commit you're on, it could be s/py27/something else/ | 22:37 |
| edmondsw | that worked, tx | 22:40 |
| edmondsw | signing off... | 22:40 |
| efried | later | 22:43 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: update_provider_tree with device exposure https://review.openstack.org/589668 | 22:57 |
| efried | edmondsw: FYI ^ | 22:57 |
| efried | ttyl | 22:57 |
| *** efried has quit IRC | 23:31 | |
| *** efried has joined #openstack-powervm | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!