21:01:24 <alaski> #startmeeting nova_cells
21:01:24 <openstack> Meeting started Wed Apr 20 21:01:24 2016 UTC and is due to finish in 60 minutes.  The chair is alaski. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:29 <openstack> The meeting name has been set to 'nova_cells'
21:01:29 <alaski> anyone around today?
21:01:36 <mriedem> o/
21:01:38 <mriedem> sort of
21:01:38 <melwitt> o/
21:01:51 <ccarmack1> o/
21:01:54 <bauzas> \o
21:02:05 <alaski> #topic testing
21:02:36 <alaski> haven't heard of any gate failures, so things are sailing smoothly these days
21:02:45 <alaski> ccarmack1 had a thing to discuss though
21:02:50 <ccarmack1> I put some thoughts about testing in https://etherpad.openstack.org/p/nova-cells-testing
21:03:10 <ccarmack1> please add any comments
21:03:21 <alaski> at the bottom it looks like
21:03:24 <melwitt> ah, my old friend that etherpad
21:03:29 <ccarmack1> yea, sorry at the bottom
21:03:47 <alaski> does anyone object to me removing the big glob in the middle?
21:04:00 <ccarmack1> I was thinking we need something that approximates a real multi node cells env
21:04:22 <melwitt> alaski: not me. I totally forgot about this etherpad
21:04:35 <alaski> melwitt: I had repressed it as well :)
21:04:40 <melwitt> haha :)
21:04:54 <alaski> ccarmack1: yes, that would be ideal
21:04:55 <ccarmack1> amazing what google will find
21:05:07 <ccarmack1> how found that etherpad
21:05:16 <melwitt> impressive
21:05:40 <bauzas> orly?
21:05:57 <alaski> so the question I have for multi-node is where will cell0 live
21:06:11 <alaski> I would probably put it on the api node
21:06:25 <melwitt> that's what I was thinking. that would be like real life anyway, right?
21:06:33 <ccarmack1> I was thinking of a devstack with api + cell1
21:06:33 <alaski> for a lot of people, yeah
21:06:42 <ccarmack1> and cell0
21:07:15 <alaski> on one node?
21:07:15 <melwitt> for multi-node I thought we would have 2 cells, no?
21:07:31 <melwitt> api + cell1 + cell2
21:07:31 <ccarmack1> I was concerned about a 3 host multinode
21:07:40 <ccarmack1> but yes that would be better
21:07:56 <ccarmack1> does the gate support 3 hosts?
21:08:02 <bauzas> why not 2 nodes with (api, cell0, cell1) and (cell2) ?
21:08:11 <ccarmack1> thats where I left it
21:08:17 <bauzas> like an AIO-+1
21:08:22 <alaski> ccarmack1: not sure. that's probably a sdague or -infra question
21:08:42 <alaski> I was actually thinking (api, cell0), then cell1 separate
21:08:55 <alaski> and eventually another node for cell2
21:09:01 <alaski> but we're not quite up to a cell2 yet
21:09:10 <ccarmack1> thats what I have in the doc.. but I can ask infra about having 3 hosts in a multi-node test
21:09:25 <ccarmack1> not quite up to cell2 ?
21:09:40 <melwitt> yeah, I was thinking future, I guess I assumed multi-node means multi-cell
21:10:05 <alaski> ccarmack1: we don't have all the pieces in place to make that work yet
21:10:27 <alaski> but melwitt has a patch up right now with a big chunk of it
21:10:42 <ccarmack1> I think this is planning for testing when everything is working
21:11:02 <alaski> okay
21:11:27 <ccarmack1> melwitt's patch in the rpc switching?
21:11:29 <melwitt> okay, so are we thinking we have one multi-node job now that has api + cell0 + cell1 and then in the future another multi-node job that has 2 cells or just amend the existing job to have 2 cells?
21:11:30 <alaski> yeah
21:11:55 <alaski> melwitt: I was thinking the latter
21:12:12 <ccarmack1> and each cell has 1 host?
21:12:25 <ccarmack1> aka node
21:12:32 <alaski> yes, if infra can provide that
21:12:39 <ccarmack1> ok, I'll ask
21:12:54 <alaski> we could do (api, cell0, cell1) and (cell2) as a reasonable test bed as well I think
21:13:16 <ccarmack1> that's 2 hosts, if infra can't do 3
21:13:17 <bauzas> I'm just thinking of people using All-in-One
21:13:29 <alaski> ccarmack1: yeah. a fallback
21:13:30 <bauzas> given that, adding a new cell
21:13:49 <melwitt> wait, are we waiting until multi-cell to do multi-node or are we thinking of multi-node now with everything talking to the same db/mq?
21:13:58 <bauzas> hence 2 nodes, first with (api, cell0, cell1) and the other with only a cell
21:14:15 <bauzas> 2 MQs
21:14:22 <bauzas> melwitt: and different DBs
21:14:32 <bauzas> but within the same infra node
21:14:57 <ccarmack1> melwitt: I'm thinking of multi-cell, when its ready
21:15:12 <alaski> well, I was thinking of multi-node with one cell, and then expand to two when ready
21:15:35 <alaski> but if everyone wants to wait we can do that
21:15:49 <ccarmack1> alaski: would that env test anything specific to cells?
21:16:18 <ccarmack1> it would be a non-AIO
21:16:30 <alaski> it wouldn't
21:16:42 <alaski> so maybe it's not important now
21:17:27 <alaski> okay, so multi-node will be multi-cell
21:17:37 <alaski> and it may be a while until that works
21:17:53 <bauzas> I was basically thinking that a cell DB could be close to the compute
21:17:57 <ccarmack1> I was thinking waiting for multi-cell and then have a config as we are saying  — node1 has api, cell0, cell2 .. node2 has cell1
21:17:58 <melwitt> it would be figuring out the legwork now instead of later I guess. or rather setting it up now once to be expanded instead of potentially having to redo a lot to expand?
21:17:58 <bauzas> but yeah...
21:18:13 <alaski> melwitt: yeah, that was my thinking
21:18:32 <bauzas> melwitt: alaski: well, that's just config stuff right?
21:18:53 <bauzas> meaning that we cut our deployment with the manage calls we want
21:18:59 <alaski> ccarmack1: that works, but it won't pass anything now so the question is are you planning to work on it now, and should it run on all reviews?
21:19:03 <bauzas> not sure changing that could be difficult
21:20:05 <ccarmack1> alaski: I was planning to do some leg work to set up the jobs, etc
21:20:31 <alaski> ccarmack1: okay. that would be great, but it probably needs to start in the experimental queue
21:20:39 <ccarmack1> yup
21:21:04 <alaski> bauzas: you're thinking of doing two cells in AIO?
21:21:10 <ccarmack1> bauzas: I was thinking of having each cell's db on the same host has compute
21:21:17 <bauzas> alaski: nope
21:21:24 <bauzas> alaski: one cell in AIO
21:21:41 <bauzas> alaski: and one other cell in a separate node with another compute
21:21:52 <alaski> bauzas: gotcha
21:21:53 <bauzas> alaski: but that means a cell DB could be close to the compute node
21:22:21 <alaski> yeah, that's fine though
21:22:23 <bauzas> which could be a bit sad for performance... but we use AIO too :)
21:22:27 <alaski> yep
21:22:40 <bauzas> so, maybe later a large-ops one
21:23:35 <alaski> yeah
21:23:41 <alaski> okay, I think we're all on the same page now
21:23:55 <ccarmack1> could you all comment in the etherpad
21:24:04 <ccarmack1> so I can chew on this later?
21:24:14 <ccarmack1> also, I'm not sure about the grenade stuff
21:24:19 <alaski> #action all comment on https://etherpad.openstack.org/p/nova-cells-testing
21:24:52 <ccarmack1> alaski: is multi-tier cells in the future?
21:25:03 <bauzas> erh
21:25:05 <ccarmack1> or not for newton?
21:25:08 <alaski> I think the only upgrade thing different between cells and non for grenade is testing the update from non cells to cells
21:25:13 <alaski> ccarmack1: not planned at all
21:25:17 <bauzas> I think we agreed to only have one parent/child
21:25:42 <bauzas> horizontally, not vertically
21:25:47 <ccarmack1> alaski: and a cellsv1 to cellsv2 upgrade?
21:25:55 <alaski> yeah, multi-tier would be cellsv3, or v2.5 or something. not being considered for this effort
21:26:02 <ccarmack1> ok
21:26:15 <melwitt> multi-tier means nested, right?
21:26:17 <alaski> ccarmack1: yes, and we don't really have c1->v2 sorted yet
21:26:25 <ccarmack1> yes melwitt
21:26:28 <alaski> ccarmack1: we kind of need multi-cell support first
21:26:42 <melwitt> okay. yeah, nested cells hasn't been a thing so far and maybe never will
21:26:51 <ccarmack1> ok
21:27:31 <alaski> any more on testing?
21:27:38 <ccarmack1> I'm done
21:27:45 <alaski> cool
21:27:51 <alaski> #topic Open Reviews
21:27:52 <bauzas> ccarmack1: cellv1 to cellv2 path is clear now
21:27:59 <alaski> https://etherpad.openstack.org/p/newton-nova-priorities-tracking
21:28:07 <ccarmack1> bauza: yes I saw your doc update
21:28:17 <ccarmack1> I think you did that…
21:28:33 <bauzas> multi cellsv1 can be only one cellv2 by default
21:28:40 <alaski> for reviews just check out the etherpad
21:28:45 <alaski> #topic Open Discussion
21:29:24 <alaski> yeah, we have a plan for v1->v2, but without multicell support no one is realistically going to upgrade
21:29:27 <alaski> and we can't fully test it
21:29:59 <ccarmack1> we can't it fully be tested?
21:30:13 <ccarmack1> why,,,
21:30:35 <bauzas> alaski: yup, agreed
21:30:38 <alaski> just that we can't migrate two v1 cells to v2, which is really what we want to check
21:31:43 <melwitt> I was wondering how are we doing on the 'boot an instance' with a single cell v2. are all of the patches up for that or is there anything else still needed?
21:31:59 <alaski> there's a few things still needed
21:32:18 <alaski> I have a local WIP which moves instance creation to the conductor, that's the last big piece
21:32:40 <alaski> there's also handling listing instances from a BuildRequest
21:32:46 <alaski> which has no work on it yet
21:32:49 <melwitt> okay, cool. just wanted to make sure is it a matter of just reviews or if there's more to come
21:32:59 <alaski> mostly those two things I think
21:33:13 <rlrossit> aren't there still API DB migrations needed too?
21:33:47 <alaski> I consider those more of the multi-cell story
21:33:59 <alaski> but yes, plenty of those still needed
21:34:23 <rlrossit> yeah I figured those weren't done yet :)
21:34:40 <alaski> heh, not even close
21:34:44 <melwitt> what else for multi-cell? API DB migrations, is there anything more with the scheduling interaction or is that covered under the single cell work?
21:34:51 <alaski> doffm did get some specs up though
21:35:15 <rlrossit> Yep and they've come up on my objects reviews dashboard, but I've been ignoring them
21:35:21 <alaski> melwitt: that should be covered under the single cell work, unless there's something being overlooked
21:35:28 <melwitt> alaski: great
21:36:02 <melwitt> alaski: well, I suppose it will be a matter of putting target_cell in the right place for the switching too
21:36:12 <alaski> right
21:36:15 <melwitt> alaski: in my local testing I put it in conductor
21:36:49 <alaski> using switching for db and rpc and db migrations. I think that's mostly what's left after current WIP is done
21:36:57 <melwitt> cool
21:37:19 <alaski> and testing, and upgrade path for v1
21:37:37 <melwitt> *nod*
21:38:06 <alaski> it probably goes without saying, but I will not be running this meeting next week
21:38:19 <alaski> if you want to talk cells come find me in Austin :)
21:38:56 <melwitt> yeah, I assumed no meetings next week
21:39:31 <alaski> also, hopefully everyone knows by now that I am not a locquacious person, so please come ready to help with the cells summit sessions
21:39:57 <melwitt> I'm not either but I will try :)
21:40:09 * rlrossit googles locquacious
21:40:24 <alaski> dangit, spelled it wrong
21:40:41 <alaski> melwitt: between the two of us we're almost a talkative person
21:40:52 <melwitt> alaski: lol, so true
21:41:12 <alaski> that's all I have for this week
21:41:15 <alaski> anyone have a topic?
21:41:24 <ccarmack1> nooope
21:41:27 <rlrossit> see y'all in austin
21:41:32 <rlrossit> getting my accent ready
21:41:41 <alaski> great! see y'all there
21:41:42 <melwitt> great!
21:41:43 <alaski> :)
21:41:45 <melwitt> o/
21:41:47 <alaski> #endmeeting