14:00:43 <n0ano> #startmeeting nova-scheduler
14:00:44 <openstack> Meeting started Mon Feb 15 14:00:43 2016 UTC and is due to finish in 60 minutes.  The chair is n0ano. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:48 <openstack> The meeting name has been set to 'nova_scheduler'
14:00:49 <edleafe> \o
14:00:53 <n0ano> anybody here to talk about the scheduler?
14:00:53 <Yingxin> o/
14:01:26 <cdent> o-o
14:01:31 <bauzas> \....o
14:01:38 <_gryf> o_O
14:02:50 * jaypipes in another meeting for 10 or so minutes...
14:02:54 <jaypipes> maybe 15.. :(
14:03:00 * n0ano too early to decipher the various I'm here tags :-)
14:03:35 <n0ano> anyway...
14:03:42 <edleafe> n0ano: just use the pythonic bool()
14:03:47 <bauzas> jaypipes: we can swap our topics
14:03:52 <jaypipes> bauzas: yes please
14:03:59 <bauzas> n0ano ?
14:04:15 <bauzas> so bugs... :)
14:04:41 <n0ano> #topic Bugs - https://bugs.launchpad.net/nova/+bugs?field.tag=scheduler
14:04:51 <n0ano> (sorry, my keyboard locked up)
14:05:11 <n0ano> I checked the bug list and we're unchanged at 39
14:05:17 <bauzas> yup, AFAIK
14:05:51 <n0ano> is anybody here actively working on any of them?
14:06:08 * _gryf has started on that one: https://bugs.launchpad.net/nova/+bug/1442024
14:06:08 <openstack> Launchpad bug 1442024 in OpenStack Compute (nova) "AvailabilityZoneFilter does not filter when doing live migration" [Medium,Confirmed]
14:06:16 <cdent> I had been planning to, but resource-* is eating the world
14:06:30 <bauzas> I'm also a bit concerned by https://bugs.launchpad.net/nova/+bug/1542491
14:06:30 <openstack> Launchpad bug 1542491 in Ubuntu "Scheduler update_aggregates race causes incorrect aggregate information" [Undecided,New]
14:06:36 <n0ano> _gryf, good to hear, let us know if you need anything
14:06:47 <Yingxin> I'll work on some of them because of the bug-smash festival
14:06:53 <n0ano> cdent, understood completely
14:07:09 <bauzas> Yingxin: cool, which ones are you planning to look at ?
14:07:26 <_gryf> n0ano, sure thing.
14:07:45 <Yingxin> bauzas: I'm still looking, maybe my issued bugs first
14:08:22 <n0ano> I would encourage everyone to look for some easy ones, just to get the list down
14:08:42 <n0ano> I'm guessing some of the bugs can be closed as not valid after closer inspection
14:08:55 <bauzas> Yingxin: ack
14:09:11 <bauzas> again, that one is making me torn https://bugs.launchpad.net/nova/+bug/1542491
14:09:11 <openstack> Launchpad bug 1542491 in Ubuntu "Scheduler update_aggregates race causes incorrect aggregate information" [Undecided,New]
14:10:07 <bauzas> I'll mark it Incomplete because I need a version, but that looks like the rpc fanout is not as fast as I thought
14:10:49 <n0ano> bauzas, tell me that doesn't mean it'll be `fixed` by increasing a timeout (I `hate` timeouts)
14:11:14 <bauzas> n0ano: that's weirdo, because the scheduler should get the update eventually
14:11:28 <bauzas> n0ano: the guy wouldn't need to restart the scheduler processz
14:11:31 <bauzas> anyway
14:11:48 <bauzas> I don't want to discuss that now, just raising a concern
14:11:55 <n0ano> bauzas, +1
14:12:31 <n0ano> OK, we're thinking about bugs, that's good, let's move on
14:12:38 <n0ano> #topic Patches/Reviews – https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
14:13:15 <n0ano> looks like we've got a few that need review
14:13:36 <cdent> resource provider objects is ready for review: https://review.openstack.org/#/c/277466/
14:13:52 <cdent> a parent is failing unit tests, though, so it is currently -1 by jenkins
14:14:36 <n0ano> cdent, bummer, note that I have a tendency to ignore patches that fail jenkins, never know what changes will be needed to pass
14:14:59 <cdent> yeah, I need to get with dansmith to find out what's going on there
14:15:15 <cdent> (its his patch that has gone wonky, but it might have been my rebase)
14:15:29 <bauzas> yeah, we need to clean-up the list of patches
14:15:32 <bauzas> for resource-*
14:15:35 <bauzas> some have merged
14:16:19 <bauzas> oh snap
14:16:23 <bauzas> :p
14:16:46 <n0ano> and magically the etherpad is updated :-)
14:16:51 <bauzas> :)
14:17:09 <bauzas> jaypipes: still stuck with people not us?
14:17:38 <edleafe> bauzas: what are we, chopped liver? :)
14:17:53 <bauzas> hah
14:17:55 <jaypipes> bauzas: 1 more min
14:18:23 <bauzas> so, waiting jay for the resource-* stuff
14:18:32 <n0ano> then, while waiting, I'll point out that I'll be on a plane next Mon., someone else will have to chair
14:18:57 <bauzas> just wanted to put lights on the 2 other bits mentioned in the etherpad :)
14:19:06 <jaypipes> ok, so...
14:19:08 <n0ano> bauzas, go ahead
14:19:09 <edleafe> n0ano: no worries, I'll be around
14:19:22 <n0ano> edleafe, a volunteer, you win
14:19:37 <edleafe> \o/
14:19:38 <bauzas> the disk allocation ratio series is kinda fuzzy, because /me forgot to provide it when he did the move for RAM and CPU
14:19:42 <bauzas> call me stupid
14:20:13 <bauzas> aaaand funny, not fuzzy of course :)
14:20:45 <jaypipes> ok, I'm done.
14:20:46 <bauzas> speaking of https://review.openstack.org/245619 et al.
14:20:48 <jaypipes> sorry guys
14:21:17 <bauzas> jaypipes: np, you'll just be charged for the whole hour, sorry
14:21:24 <jaypipes> figurd as much :)
14:21:41 <jaypipes> anyway, can we talk about resource-providers status or have you already done that?
14:21:59 <bauzas> jaypipes: I was just waiting for you, speaking of https://review.openstack.org/245619 meanwhile :)
14:22:23 <bauzas> anyway
14:22:30 <bauzas> moving on to the resource-* stuff
14:22:33 <n0ano> jaypipes, according to the priorities etherpad there;s only 1 resources change outstanding
14:22:58 <n0ano> if that's correct that's great news
14:22:58 <jaypipes> yup, I'll review that shortly. no problems with moving the remaining allocation ratio to the DB.
14:23:37 <cdent> n0ano: it's not quite accurate as there is at least one pending spec
14:23:40 <cdent> maybe two
14:24:00 <n0ano> cdent, I was afraid of that, it's be good to get those reflected in the pad
14:24:35 <cdent> we had a race collision on who was going to do that, I'll just go ahead and do it now
14:24:42 <jaypipes> n0ano: I'll work with cdent on updating the pad.
14:24:58 <n0ano> cdent, jaypipes tnx
14:25:39 <bauzas> so, about the resource-scheduler spec (which we agreed to not merge in Mitaka anyway), I know that jaypipes is planning to update it by splitting it in two
14:25:52 <n0ano> to specifics, bauzas did you want to bring up your disk issues?
14:26:16 <jaypipes> bauzas: correct. one part for the pushing join conditions to the DB, the other for the claims in scheduler part.
14:26:16 <bauzas> n0ano: not really, it was rather for consuming time waiting for jay :)
14:26:40 <n0ano> bauzas, NP, sounds like it's progressing properly then
14:27:13 <Yingxin> join conditions?
14:27:15 <bauzas> so, I left some comments in the spec, but I'd like to see how Yingxin's spec about shared-state scheduler could benefit from jaypipes and cdent's work on resource-*
14:27:54 <bauzas> Yingxin: the ability to select hosts with enough RAM and disk by filtering out in the query directly instead of filters
14:28:08 <Yingxin> I'll finish my review on resource-providers-scheduler after the meeting
14:28:19 <Yingxin> bauzas: got it
14:28:55 <jaypipes> thx Yingxin
14:29:09 <bauzas> so, my guys on that is that operators probably want both possibilities
14:29:27 <cdent> We still seem to have a fair amount of difference of opinion on where the truth lives. And it would be great to make that _not_ optional.
14:29:37 <cdent> It should live in one place. Doesn't matter where (to me).
14:29:39 <Yingxin> bauzas: jaypipes: it looks very diffierent from my approach
14:29:53 <jaypipes> Yingxin: yes, sorry, my db background showing through a bit there..  pushing join conditions is about not running filters in the Python code and instead doing that on the DB side :)
14:29:55 <cdent> choice is bad, mmmkay?
14:30:10 <bauzas> well, that's actually a very good open question
14:30:36 <bauzas> MHO is that we should provide an ubiquitous scheduler, but we can have different drivers
14:30:36 <edleafe> jaypipes: you're trying to minimize the number of hosts constantly pulled from the db.
14:30:41 <jaypipes> Yingxin: not a problem. let's have data inform our decisions on the scheduler design. I'm more than happy to let the better approach win :)
14:30:45 <edleafe> So is Yingxin
14:31:25 <edleafe> just not in a db-centric way
14:31:29 <bauzas> jaypipes: what I like with Yingxin's approach is that it's close to what I dreamed with https://etherpad.openstack.org/p/liberty-nova-scalable-scheduler
14:31:47 <bauzas> and I'm less concerned cell-ish
14:32:03 <bauzas> but like I said, I wonder how all of that can integrate
14:32:22 <Yingxin> looks like a performance competition :)
14:32:25 <bauzas> atm, we have 3 drivers
14:32:41 <jaypipes> like I said, I'm very open to all ideas. let's just get some data to tell us when one approach is better than the other. and, be aware, one approach may be better for some scenarios and not others.
14:32:48 <bauzas> the ChanceScheduler, for things like functional tests that don't really care of filtering
14:32:57 <bauzas> the FilterScheduler
14:33:12 <bauzas> and the CachingScheduler, which is just a FilterScheduler with some caching
14:33:19 <edleafe> jaypipes: that's what always causes resistence
14:33:23 <jaypipes> bauzas: some racy caching :)
14:33:26 <cdent> I think we need to be careful not to prioritize performance over being useful with a variety of resources, maintainability and extractability from Nova
14:33:42 <edleafe> jaypipes: solve for the 80% (or even 90%), and you'll have complaining from the rest
14:33:42 <bauzas> jaypipes: totally agreed, but that's a reasonable tradeoff for large operators
14:33:50 <jaypipes> cdent: agreed. the simplicity and debuggability of a design also matters.
14:33:58 <cdent> performance is all well and good but if we end up with some kind of magic box that makes everyone scratch their head that's a bummer
14:34:01 <n0ano> well, better in different cases is why multiple drivers is good
14:34:12 <bauzas> hence my point
14:34:13 <edleafe> cdent: we also shouldn't sacrifice performance to perpetuate a poor design
14:34:25 <jaypipes> edleafe: well, we already have that (solved for 80%) and we've had a continual stream of complaints for 4+ years :P
14:34:30 <cdent> edleafe: I definitely want _change_
14:34:42 <bauzas> well, having an optimistic scheduler is not really a bad choice IMHO
14:35:06 <bauzas> and I explained that in https://review.openstack.org/271823
14:35:11 <edleafe> cdent: hmmm... change in and of itself is not a good thing
14:35:25 <edleafe> cdent: but I understand the feeling
14:35:34 <cdent> edleafe: are you just being contrary for the sake of it I think we're mostly agreeing :)
14:35:39 <jaypipes> I'd like to make it clear that the resource-providers framework doesn't affect the scheduler much at all until the resource-providers-scheduler blueprint, which isn't coming until Newton.
14:35:55 <cdent> ++
14:36:08 <Yingxin> ++
14:36:15 <bauzas> jaypipes: I'm totally on board with you
14:36:17 <cdent> It does feel, however, like we need to gain a more shared goal
14:36:28 <jaypipes> dammit my phone is buzzing off the hook. damn you $work!
14:37:01 <bauzas> anyway, we're not about to solve our problem in that meeting
14:37:11 <jaypipes> cdent: I agree completely. but at the same time, I'd like to prove out the various design ideas and show where each shines and each doesn't.
14:37:24 <cdent> yeah, definitely, no disagreement on that point
14:37:40 <jaypipes> cdent: my benchmarking framework is an attempt to prove out various strategies.
14:37:46 <cdent> I just worry that sometimes we're not all talking about the same thing. It's my superpowwer to notice when that happens.
14:37:49 <bauzas> my take is just to raise all initiatives (which is great) and try to figure a complete path for success :)
14:37:52 <cdent> code++
14:37:53 <jaypipes> cdent: :)
14:39:00 <bauzas> Yingxin: are you planning to attend Austin ?
14:39:18 <jaypipes> Yingxin: I will review your code and ideas this week. I will try to emulate your design into my placement-bench project, ok?
14:39:38 <cdent> Austin Scheduler Cage Match
14:39:42 <Yingxin> Yeah, if the session is elected then I'll definitly be there.
14:40:04 <n0ano> cdent, we have a rich history of Scheduler Cage Matches :-)
14:40:09 <Yingxin> jaypipes: ok thanks very much
14:40:12 <cdent> s/Scheduler/Placement/
14:40:17 <bauzas> jaypipes: that would be very gentle
14:40:30 <jaypipes> bauzas: as always, your change requests would be most welcome. feel free to peruse the placement-bench code and ask me questions on it. I'll be adding the legacy (existing Nova) schema for compute nodes soon to show difference between placement requests using old and new resource providers schemas.
14:40:50 <jaypipes> n0ano: s/Scheduler//g
14:40:50 <bauzas> jaypipes: you mean, PRs on the bench system ?
14:40:56 <jaypipes> bauzas: yes, sorry :)
14:41:07 <Yingxin> jaypipes: the installation instructions are in the commit message of https://review.openstack.org/#/c/280047
14:41:16 <bauzas> jaypipes: ack, will try to see how I can setup all of that
14:41:30 <jaypipes> Yingxin: well, I won't be installing it :) I'll be essentially emulating the key parts of the design.
14:41:32 <bauzas> I have 2 mini-dells currently sleeping
14:41:50 <Yingxin> jaypipes: well, magic
14:41:52 <jaypipes> Yingxin: have a look at https://github.com/jaypipes/placement-bench and you'll see what I mean :)
14:42:44 <Yingxin> jaypipes: I'm still using the dumb way to test schedulers using devstack :P
14:42:52 <Yingxin> jaypipes: will upgrade my weapon
14:43:02 <jaypipes> Yingxin: that's not dumb at all :) just not as isolated...
14:43:05 <jaypipes> Yingxin: hehe
14:43:41 <jaypipes> Yingxin, bauzas: again, this was only 3 days work... I'll refine it this week
14:44:29 <edleafe> Yingxin: for some of us, that's the only testing method available
14:44:43 * edleafe would love to have racks of hardware to play with...
14:45:14 <n0ano> edleafe, Intel is offering 1000, you just have to have a specific proposal for them
14:45:33 <edleafe> n0ano: yeah, that's the exact opposite of 'playing'
14:45:47 <n0ano> edleafe, good point :-)
14:46:05 <bauzas> jaypipes: eh, no worries, I just need to understand your testbed
14:46:20 <bauzas> jaypipes: but that looks very promising
14:46:56 <jaypipes> ok, we all wrapped up here?
14:47:04 <Yingxin> I'm still figuring out how to test scheduler (performance) using existing functional test framework.
14:47:35 <n0ano> getting close to the hour
14:47:41 <n0ano> #topic opens
14:47:47 <n0ano> anything new?
14:48:30 <Yingxin> one question
14:48:35 <n0ano> Yingxin, go ahead
14:48:46 <Yingxin> Why there can be multiple RTs and compute nodes in the same host?
14:48:56 <Yingxin> do those RTs have the same resource view?
14:49:47 <bauzas> Yingxin: you have one RT per compute node
14:49:58 <bauzas> Yingxin: the compute-manager service can hold N nodes
14:50:24 <bauzas> Yingxin: because some drivers (ironic and hyper-v) are reporting more than one node per compute
14:50:46 <Yingxin> bauzas: so their compute node resources(CPU/RAM/DISK) are different?
14:50:51 <bauzas> what we call a "ComputeNode" object is one node
14:50:56 <bauzas> Yingxin: yeaj
14:50:57 <bauzas> yeah
14:51:06 <bauzas> but that's some behaviour we'd like to change
14:51:11 <bauzas> at least for ironic
14:51:35 <bauzas> tl;dr: ironic is reporting the list of ironic nodes it managers thru a single nova-compute
14:51:41 <bauzas> it manages*
14:52:01 <bauzas> each compute node being an ironic node
14:52:48 <edleafe> Yingxin: the original model assumed VMs. The current design of multiple nodes per host was an attempt to fit ironic and vmware into that same VM model
14:53:49 <Yingxin> bauzas: edleafe: My question is are the resources of the host shared or allocated separatedly to different compute nodes?
14:54:36 <bauzas> Yingxin: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6329-L6331
14:55:08 <bauzas> Yingxin: the resource usage is given by the drivre
14:55:09 <bauzas> driver
14:55:19 <bauzas> Yingxin: for each node it knows
14:55:33 <Yingxin> bauzas: thanks its clear
14:55:36 <bauzas> Yingxin: so each 'driver node' is reporting differently
14:55:52 <bauzas> there is no aggregation
14:56:09 <n0ano> any other last minute items?
14:56:18 <Yingxin> bauzas: yeah makes sense
14:56:34 <bauzas> and the scheduler doesn't know that the 2 corresponding nodes are on the same host
14:56:45 <bauzas> that's 2 computes for it
14:57:02 <_gryf> n0ano, one more from me - I'm open to work on the resources-* - if there is anything I can take, it would be great.
14:57:09 <bauzas> heh
14:57:29 <bauzas> _gryf: reviewing is certainly a good option :)
14:57:32 <n0ano> _gryf, I'd coordinate with cdent & jaypipes , I'm sure they can give you something to do
14:57:34 <Yingxin> bauzas: but the scheduler can only send RPC messages to a host
14:57:37 <edleafe> bauzas: I'm taking a crack at the scheduler utils stuff
14:57:42 <_gryf> bauzas, I'm already doing that :)
14:57:58 <bauzas> Yingxin: let's take that offline in -nova right after the meeting if it's not too late for you, okay.
14:57:59 <bauzas> ?
14:58:05 <Yingxin> ok
14:58:05 <edleafe> bauzas: I may have questions along the way for you
14:58:18 <bauzas> I'm all ears in -nova :)
14:58:28 <n0ano> indeed, I think it's time people, tnx and most of us will talk again next week
14:58:33 <n0ano> #endmeeting