17:00:29 #startmeeting nova_cells 17:00:29 like DBZ ? 17:00:30 Meeting started Wed Dec 3 17:00:29 2014 UTC and is due to finish in 60 minutes. The chair is alaski. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:31 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:34 The meeting name has been set to 'nova_cells' 17:00:39 30 years of Dragon Ball ! 17:00:43 Hi everyone 17:00:44 Hello cellz 17:00:49 nothing like DBZ 17:00:52 hi 17:00:52 hi 17:00:54 \o 17:01:11 hi 17:01:14 #topic cells manifesto 17:01:21 https://etherpad.openstack.org/p/kilo-nova-cells-manifesto 17:01:30 LGTM 17:01:43 cool 17:01:45 oo nice 17:01:48 * mriedem needs to read it 17:01:52 it wasn't clear if anyone had looked at it yet 17:01:57 o/ 17:02:05 yeah, EU people have an advantage because we were not eating Turkey :) 17:02:14 I think I have a few changes I'd like to put in for discussion, but due to a short week haven't gotten to it 17:02:16 alaski: I'm +1 on it 17:02:19 that's a disadvantage right? 17:02:32 seems like it to me 17:02:50 alaski: so once we're good with the etherpad, where does that go? 17:02:55 devref? blog? other? beuler? 17:03:09 devref IMHO 17:03:15 + blog 17:03:15 I think it needs to hit the mailing list, but not in etherpad form 17:03:18 mriedem: devref ultimately, also an operator list post, and a blog would be good as well 17:03:40 ok, yeah, ML would be good (and not in etherpad so random joe can mark it up) 17:03:48 do we want to just propose that as a devref thing, perhaps with alaski's diagrams, 17:03:50 and send that to the ML? 17:03:56 i'd be good with that 17:04:11 yeah, sounds good 17:04:11 +1 17:04:14 if i'm going to read it, i might as well do it in gerrit 17:04:24 dansmith: +1 17:04:32 dansmith: you want to take that? 17:04:34 mriedem : +1 17:04:43 alaski: heh, was just about to make you do it 17:04:48 heh 17:04:50 but sure, we'll do it somehow 17:05:06 yeah, I don't mind proposing it 17:05:12 show your diagrams 17:05:17 #action propose manifesto to devref 17:05:27 http://paste.openstack.org/show/144068/ 17:05:35 I forgot to add this to the agenda 17:05:40 but I drew a few pictures 17:05:56 I would love feedback 17:05:58 good ascii diagrams are like geek porn 17:06:00 alaski: nice 17:06:13 and want to include them in appropriate specs, and the manifesto if that helps 17:06:26 I think they should go with the manifesto as well, yeah 17:06:32 alaski: volunteering for creating yet another etherpad for diags ? 17:06:40 bauzas: I tried that... 17:06:52 alaski: oh, fonts are getting you mad ? 17:06:52 the pads aren't monospaced 17:07:13 alaski: eh 17:07:15 it's a client side setting to change it 17:07:29 so I'm happy to put them in a pad, with instructions on how to actually view them 17:07:39 alaski: agreed 17:08:01 well, 17:08:04 devref can link to images right? 17:08:18 once we're happy with the diags we could just put into an image file and store with the docs 17:08:20 devref can include these eaasily 17:08:23 bauzas: https://etherpad.openstack.org/p/nova-cells-flow-diagram I need to add the last one 17:08:26 because they're ascii 17:08:30 mriedem: IIRC, there are even diagrams in devref 17:08:39 bauzas: yeah for the rpc stuff 17:08:58 mriedem: they would need some updated btw. :d 17:08:58 alaski: etherpad just broke your diagram 17:09:07 *updates 17:09:12 bauzas: yeah, devref doesn't mention objects i'm pretty sure... 17:09:23 mriedem: think about betting... 17:09:52 what's next? 17:09:52 alaski: shall we move on? 17:09:56 alaski: that sounds acceptable for reviewing, I mean the client change in the etherpad 17:10:12 yep, distracted by etherpad 17:10:15 #topic testing 17:10:20 https://etherpad.openstack.org/p/nova-cells-testing 17:10:30 I've added a bunch of info there 17:10:31 ooh nice 17:10:48 there are some reviews linked with fixes as well 17:11:05 helping to categorize/fix the remaining failures would be a huge help 17:11:23 since we are on testing, has anyone looked into this? https://blueprints.launchpad.net/nova/+spec/cells-tempest 17:11:28 quite an old BP 17:11:45 what about it? 17:11:55 heh, that's basically what we're doing 17:12:17 but realistically that should be an infra bp if anything 17:12:24 qa 17:12:29 dansmith, alaski: yes exactly... aren't we repeating it.. I mean that BP was basically abandoned 17:12:29 yeah qa 17:12:30 or that... 17:12:31 or tempest or whatever, but yeah 17:12:34 alaski: can we organize that into a list of reviews that are: merged, ready for review, wip? 17:12:38 but they've been fine with the patches so far it seems 17:12:49 vineetmenon_: it was never started 17:12:52 alaski: very nice info 17:13:05 sdague: yes, I can do that 17:13:06 sdague: yeah i was going through that list now 17:13:20 alaski: thanks 17:13:29 #action alaskiorganize testing etherpad into reviews that are: merged, ready for review, wip 17:14:11 vineetmenon_: I think that bp is fine being abandoned. if we need a new one we'll create it 17:14:21 https://review.openstack.org/#/c/135285/ is going through the gate 17:14:23 which will take awhile 17:14:30 will definitely help on getting them reviewed in a more timely manner if we have a concise list. It would also be really nice to call out which repo the reviews are against so people will know if it's something they have +2 on before following the link 17:14:36 alaski: sure.. I just thought mentioning it here.. 17:15:13 vineetmenon_: appreciated. if it was a qa bp I would feel different, but most of the work isn't in nova 17:15:20 aye 17:15:48 sdague: cool, we can do that too 17:15:56 anything else on testing? 17:16:06 how much are we waiting on https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/flavor-from-sysmeta-to-blob,n,z now? 17:16:34 a little 17:16:51 I'm eager for it, but there are plent of other things to tackle still 17:17:00 ok, just figuring out nova review priorities since the one nova patch in there is approved, just needs gate babysitting 17:17:12 i've sucked at nova reviews since the break 17:17:38 flavors have been keeping me from reviews too 17:17:39 once a couple of these get in, we can go back and check test results again 17:17:40 it will become more urgent as the list of failing tests dwindles 17:17:43 I need to just put it down for a day 17:17:52 moving on? 17:17:59 #topic analysis of tables 17:18:03 https://etherpad.openstack.org/p/nova-cells-table-analysis 17:18:11 okay, big disclaimer: 17:18:11 dansmith put some lists together 17:18:29 this is just a first, uninformed pass at sorting things into global/cell scope 17:18:46 just one comment, could we maybe put all the etherpads in a wikipage ? 17:18:46 maybe you want to put that into the etherpad :) 17:18:47 there are a bunch I don't know what to do with, mostly because I need to look at how they're used, but some are opinion calls 17:18:53 bauzas: was thinking the same 17:18:58 global cell == global wiki page 17:19:04 mriedem: agreed 17:19:07 bauzas: yeah 17:19:13 time for opening a wikipage, eh ? 17:19:13 I should also be linking all of these... 17:19:18 #link https://etherpad.openstack.org/p/nova-cells-table-analysis 17:19:33 alaski: yeah, I'm pretty worried of loosing focus on all etherpads 17:19:38 I can't track them like in Gerrit 17:19:58 and I'm pretty bad in making shortlinks 17:20:05 bauzas: +1 17:20:08 ok, I'll corral everything into a wiki 17:20:13 alaski: awesome 17:20:28 as far as the tables go 17:20:32 alaski: feel free to delegate a bit :) if you want i can do the wiki tedium 17:20:51 :) 17:21:11 \o too 17:21:15 it would be good to get some feedback on the preliminary organization, and thoughts on what to do with the uncategorized one 17:21:36 yeah, the unsure section needs debating a bit IMHO 17:21:41 alaski do we put our comments in the etherpad itself? 17:21:48 #action mriedem bauzas set up a wiki page to corral etherpad links for cells work 17:22:00 woah, 2 people on that heavy task :) 17:22:11 dheeraj-gupta-4: comments about the table split you mean? 17:22:23 yes for the table split 17:22:37 I think this is going to be a worksheet, so lets see how it goes with people putting comments in here 17:22:48 just please try to keep the list readable by people as well 17:22:50 yeah, I'm just concerned about grouping some tables 17:22:59 like aggregates and instancegroups 17:23:05 comments in the etherpad seems like a good place to start 17:23:05 both of them are related to scheduler 17:23:14 alaski: agree 17:23:23 comments on any etherpad content in general maybe, make comments at the bottom or something? 17:23:36 yes comments section is a good idea 17:23:39 melwitt: sure, or at the bottom of a section 17:23:40 this is very preliminary so we'll likely have to adjust how we approach this, but this is a good starting place 17:23:59 bauzas alaski: for heated tables, I'm afraid etherpad would may become illegible.. 17:24:01 yeah, we can do both inline and at the end of the doc, sure 17:24:15 vineetmenon_: that's why people need to be on their best behavior and keep the doc readable 17:24:24 if you want to argue about something, etherpad isn't the place anyway 17:24:30 maybe just call out controversial tables 17:24:39 and we can discuss here or in another medium 17:24:49 after we make a few passes at this, we could do another devref and argue the minutia in gerrit 17:25:14 the biggest problem I see here is actually the nova-network tables 17:25:14 dansmith: +1 17:25:15 alaski: well, even the 2nd table about a global CellsV2 DB is debatable IMHO (compute_nodes) 17:25:30 bauzas: +1 17:25:37 because I don't want to spend a bunch of time making nova-network split-able just to deprecate it 17:25:40 bauzas: https://wiki.openstack.org/wiki/Nova-Cells-v2 17:25:40 bauzas: what does that mean? 17:26:08 dansmith: I mean that I'm not sure that compute_nodes should be global if we consider a scheduler per cell 17:26:25 scheduling is a whole other can of worms right now 17:26:38 alaski: yeah, hence my worries 17:26:40 bauzas: ah, well, I don't know that I think scheduling should be per-cell anyway, which is why I put it there :) 17:26:44 but I think we should consider it orthogonal for now 17:26:46 dansmith: nova-newtwork may be a requirement... depends in neutron migration progress 17:26:54 alaski: agreed 17:26:57 belmoreira: I know, that's my point 17:27:06 how about this 17:27:15 alaski: but in that case aggregates should be orthogonal, and here comes the discussion 17:27:22 if there are things in one or the other section, lets move them to a "contentious" section and explain why 17:27:31 unsure will continue to be "needs more investigation" 17:27:42 dansmith: +1 17:27:50 anyway, let's put our thoughts on the etherpad, for sure 17:28:28 it will be nice to get a list of low hanging fruit tables, like flavors and keypairs 17:28:39 from there we can move into the contentious ones 17:28:48 yep 17:28:57 alaski: I think the sections are trying to do this 17:29:09 alaski: maybe some of the items need adjustment 17:29:14 dansmith is already done with low handing fruit tables, right 17:29:32 vineetmenon_: this is just a proposal, it needs review and sign-off 17:29:40 bauzas: sure, lets capture that on the pad for now 17:29:42 dansmith: ack 17:29:44 like, compute_nodes was one I assumed was low hanging, but it's not 17:29:55 so let's see next week if people have more issues with these 17:30:00 after thinking/reading 17:30:12 +1 17:30:17 agreed 17:30:18 and it would be great if folks could look at the unsure ones and try to evaluate where they belong, or at least add comments about the implications 17:30:20 +1 17:30:55 moving on 17:30:56 i need to read back through alaski's 2 specs first i think 17:31:07 #topic scheduling requirements 17:31:12 * dansmith runs 17:31:22 #link https://etherpad.openstack.org/p/nova-cells-scheduling-requirements 17:31:40 dansmith: you're off the hook for this one 17:31:46 woot 17:31:59 johnthetubaguy couldn't make it today, but he put some thoughts down 17:32:31 he added some possible solutions at the bottom, but what I'm more interested in currently is what are the needs 17:32:57 so, belmoreira should add some things? :) 17:33:12 I know what Rackspace needs, but I don't know about CERN/Nectar and non cells users 17:33:22 dansmith: right :) 17:33:30 so let's specifically call out the holes 17:33:56 dansmith: I need to read it... 17:34:09 mmm, that still relates to the discussion about a global scheduler or not 17:34:38 bauzas: it will inform that discussion 17:34:39 I agree with johnthetubaguy about the scaling concerns 17:34:57 alaski: agreed 17:35:00 bauzas: it does, but we need to know what the current at-scale folks need/want/think is important 17:35:03 belmoreira: sure, we'll touch back on this next week or after 17:35:10 what about some people at scale not using cells? 17:35:14 dansmith: yeah, I totally understand 17:35:15 like phil or paul? 17:35:39 yeah, that would be great info 17:35:43 dansmith: I discussed with john about some scaling issues they had with the existing nova scheduiler 17:35:46 alaski: for next weeks can you publish the links before the meeting? 17:36:01 belmoreira: yes 17:36:16 I'll add links to the agenda 17:36:22 dansmith: that's basically IO problems with context switching the greenthreads 17:36:29 #action add links to the agenda before a meeting 17:36:34 dansmith: when they were accessing the DB IMHO 17:36:39 bauzas: yeah 17:36:42 wiki is here now: https://wiki.openstack.org/wiki/Nova-Cells-v2 17:36:42 s/IMHO/IIRC 17:36:44 alaski: ^ 17:36:55 so we could just throw that in the meeting agenda 17:36:58 mriedem: ahh, good point. I can add them there too 17:37:07 dansmith: so I'm a little concerned about a "global scheduler" wishlist 17:37:07 #link https://wiki.openstack.org/wiki/Nova-Cells-v2 17:37:45 bauzas: what's your concern? 17:37:53 bauzas: I put "as it relates to cells" in the page, but I think we need to hear this 17:38:00 alaski: I just want to make sure that the scheduler can scale :) 17:38:03 this shouldn't be a wishlist, but a list of what the current usage is 17:38:35 alaski: hence the CachingScheduler that john provided 17:38:37 bauzas: so do I :) 17:38:37 bauzas: we don't have to do everything that is posted here, and certainly not things that aren't achievable or compatible with the overall goal :) 17:38:55 alaski: but I don't want to speak on behalf of him :) 17:39:13 dansmith: \o/ 17:39:32 sounds like the voice of reason 17:39:37 next? 17:39:53 yep, had to find my agenda tab... 17:39:57 #topic Connection info in database or config file? 17:39:57 opens ? 17:39:58 * dansmith cracks the whip 17:40:00 oh 17:40:06 #link https://review.openstack.org/#/c/135424/ 17:40:11 thanks for having raised it 17:40:17 so, I definitely think that the connection info needs to be in the DB 17:40:25 and I agree with that 17:40:32 eh eh 17:40:38 next then ? :) 17:40:47 :) 17:40:55 nah, just saying it was a configuration problem 17:40:56 I'd like to understand why a config would be useful 17:40:59 then are you planning every config to reside in DB? or something in DB and something in file? 17:41:21 alaski: I'm just thinking of something that is operator-related 17:41:36 vineetmenon_: all of it in the db 17:41:43 vineetmenon_: regarding what's in the spec 17:41:59 so it will need complete overhaul of existing things as well, right 17:42:06 vineetmenon_: not all of nova.conf of course, just this tiny piece 17:42:10 alaski: atm, connection URIs are stored in a conffile 17:42:11 bauzas: I think a config makes operation more difficult as it can't be done online 17:42:27 alaski: well, it requires a service restart I agree 17:42:32 dansmith, alaski: ack 17:42:45 alaski: do we consider autoscaling for cells ? 17:43:03 bauzas: and for cells v1, we do store the cell relationships in the db and the mq connection info 17:43:04 alaski: I still need to remember my little knowledge about Heat resources 17:43:22 alaski: but you agree that's only for cells, not for Nova ? 17:43:25 in general I mean 17:43:52 bauzas: I'm not sure autoscaling and cells go well together 17:43:57 bauzas: what do you mean "only for cells" ? 17:44:02 because there's a physical component involved 17:44:06 alaski: there is also the file option 17:44:07 alaski: so is the service restart really a problem ? 17:44:16 bauzas: yes 17:44:21 bauzas: well, nova is part of cell, in one way or another.. So, I guess both 17:44:24 belmoreira: that's what I'm mentioning as conf file 17:44:44 but OK, I don't want to bikeshed things 17:44:57 seems like there is a large majority 17:45:06 bauzas: yes, this is only for cells 17:45:20 and that's something manageable thru a nova-manage thing 17:45:28 alaski: but every nova deployment will have at least one of these in the table 17:45:39 dansmith: right 17:45:43 using configuration management tools is much more easy to manage configuration files 17:45:47 which is why I don't want to say things like "only for cells" 17:45:48 so, I'll change my vote and ask for something manageable, that's it 17:45:49 If we move the connection and DB to a file, we probably won't need a cells table :) 17:45:56 dansmith: fair enough 17:46:28 dheeraj-gupta-4: sure, but then I have to restart my apis to add a new cell 17:46:56 or to delete one 17:46:57 alaski: actually policy files are not requiring a service update 17:47:10 alaski: so, that's maybe an implementation detail ? 17:47:25 s/service update/service restart 17:47:36 * bauzas dammit 17:47:51 i think being able to refresh config options on the fly is a wishlist for everyone for a long time 17:47:52 we could do it in a config file without requiring restarts, of course 17:47:53 belmoreira: if you're currently using config I don't want to break that for you. perhaps we can look into a way to make them both work 17:48:15 I don't really see the point, TBH 17:48:18 belmoreira: but operationally I think a db is easier 17:48:19 we could read the config file only for the cells group and reload the options in memory 17:48:25 yeah 17:48:38 mriedem: everything is doable, nothing about a config file requires static-ness 17:48:47 yeah... 17:48:48 the thing is, however, 17:48:52 if you want an API to add a cell 17:49:00 so the question is, what's better for cells operators ? 17:49:01 doing it in the config file means you really can't have that 17:49:08 what I'm thinking is that we could load from a config into a db on startup 17:49:20 dansmith: that's really my question, do we envisage something orchestrating cells ? 17:49:21 but have the db be what's actually used 17:49:30 * bauzas not saying the word "Heat" 17:49:37 alaski: I think that we should go only for one solution 17:49:40 alaski: slightly more potentially confusing I think 17:49:40 bauzas: I don't 17:49:42 yeah 17:49:50 okay 17:50:08 we can change this later without too much trouble if it turns out to be a huge problem 17:50:17 belmoreira: can you comment on the spec again with your thoughts 17:50:37 alaski: sure 17:51:03 thanks 17:51:29 anything left? 17:51:31 let's bring the discussion back to the review, but maybe we're a bit closer now to consensus 17:51:46 #topic open discussion 17:51:50 alaski: yeah, as said, I'll amend my comment 17:51:56 bauzas: thanks 17:52:14 anything for open discussion? 17:52:31 i'm hangry 17:52:36 alaski: do you have anything to do for minions (me) ? specifically in testing? 17:52:37 let's close 17:52:50 or specs, either 17:52:51 vineetmenon: scour through the testing etherpad 17:53:09 vineetmenon: alaski calls out the need for investigation into other failing tests 17:53:09 vineetmenon_: mainly we could use help understanding the current failures 17:53:30 there's a review linked with the full logs 17:53:45 alaski, mriedem: okay 17:53:54 or I can put up some info on getting cells running in devstack 17:53:57 to run locally 17:54:19 yeah, some instructions would be awesome 17:54:26 alaski: don't you have a wip for that or something in devstack? 17:54:33 alaski: that I am able to, with local cell 17:54:41 everytime I'm changing something in the cell namespace, I'm crossing fingers... :/ 17:54:46 vineetmenon_: cool 17:54:53 mriedem: it's just some localrc settings 17:54:58 region and child, right 17:54:59 devstack does the rest already 17:55:12 alaski: nova-net ? 17:55:26 alaski: I beg your pardon, my knowledge is poor 17:55:30 bauzas: yes 17:55:36 belmoreira: ack 17:55:53 alright, closing up early 17:55:59 #endmeeting