17:00:22 <alaski> #startmeeting nova_cells
17:00:22 <openstack> Meeting started Wed Aug 26 17:00:22 2015 UTC and is due to finish in 60 minutes.  The chair is alaski. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:26 <openstack> The meeting name has been set to 'nova_cells'
17:00:42 <alaski> anyone around today?
17:00:48 <dheeraj> o/
17:00:48 <melwitt> o/
17:00:50 <vineetmenon> o/
17:00:57 <lalitdagre> o/
17:01:00 <alaski> awesome
17:01:07 <alaski> #topic Tempest testing
17:01:22 <alaski> it appears that elastic search isn't responding to me at the moment
17:01:29 <alaski> but melwitt, any news?
17:01:52 <melwitt> I've noticed it's been like this the past few weeks at least, for the 7d choice. if I change it to 48h I can get results
17:02:07 <alaski> ahh, that worked
17:02:59 <melwitt> things seem okay, yesterday there were 8 failures, one was a bad patch. the others are the instance not found problem in test_list_servers_by_admin_with_all_tenants
17:03:32 <alaski> okay, great
17:03:50 <alaski> mriedem was working on that one
17:03:58 <melwitt> oh, cool
17:04:16 <alaski> https://review.openstack.org/#/c/215859/ I think
17:04:34 <alaski> #topic Open Reviews
17:05:02 <alaski> there are a few reviews that could always use feedback
17:05:11 <alaski> I'll just link https://wiki.openstack.org/wiki/Meetings/NovaCellsv2 which has them all listed
17:05:53 <dheeraj> alaski: about the flavor port one (http://review.openstack.org/201606)
17:05:55 <alaski> if there are any missing feel free to add them there and to the priority etherpad
17:06:14 <dheeraj> we dropped soft-delete for now because too many tempest and unit tests were failing
17:06:24 <dheeraj> I mean we keep soft-delete option
17:06:44 <alaski> dheeraj: tempest tests on that patch, or the following one?
17:06:52 <dheeraj> on the following one
17:07:07 <dheeraj> when we actually tried to write code that used new Flavor models
17:07:48 <alaski> I commented on that in the other review
17:08:18 <alaski> I think you should keep the object the same, i.e. don't remove deleted/deleted_at
17:08:19 <dheeraj> alaski: yeah, so we figured it was simpler to keep deleted columns for now
17:08:27 <dheeraj> about the dependent patch (https://review.openstack.org/#/c/213041/)
17:09:03 <dheeraj> a lot of code looks at .id (the auto-generated id in the table) and when flavors will be moved, the id will obviously change
17:09:03 <alaski> it's simpler now, but we'll need to deal with it later
17:09:12 <dheeraj> alaski: agreed
17:09:16 <vineetmenon> +!
17:09:42 <lalitdagre> +1
17:10:10 <alaski> dheeraj: okay, that sounds like another issue different from the deleted one
17:10:23 <dheeraj> so id is a sticking point. Right now the WIP patch doesn't implement all the stuff (it is more focussed on get_*), but we would like to get more feedback about it
17:10:36 <alaski> okay, I'll look at that and comment
17:10:56 <dheeraj> alaski:: thanks
17:11:12 <alaski> for soft-delete I think it can be handled reasonably well by having compatibility code in the db methods for the nova_api db
17:11:31 <alaski> if the object passes in deleted/deleted_at just ignore it, and add those fields in to the db return
17:12:03 <dheeraj> alaski: some db code also looks at deleted columns
17:12:41 <alaski> dheeraj: yes.  but any new db code shouldn't.  so db code for nova db will, but nova_api db won't
17:12:56 <alaski> as long as those don't get mixed it should be fine
17:13:29 <melwitt> I have some questions from working on https://review.openstack.org/161906 db connection switch
17:13:43 <dheeraj> alaski: ok. we'll take a look
17:13:53 <dheeraj> maybe have a dirty patch and see how tempest holds
17:14:12 <alaski> dheeraj: okay.  the preference would be to not have soft-delete for the new db, but if it's overly complex right now it's not a blocker
17:14:23 <alaski> melwitt: go for it
17:14:30 <dheeraj> alaski: cool
17:15:39 <melwitt> I've been trying (unsuccessfully) to write a functional test and ran into wondering, how is it supposed to work with for example, Instance object contains a SecurityGroupList field, when the security groups reside in a different database
17:16:50 <melwitt> like when FK will span across databases
17:17:04 <alaski> melwitt: I think we need to assume it will be in the same db
17:17:11 <alaski> unless it's something in the api and cell db
17:17:44 <alaski> and in that case we may need to think about composing the objects differently
17:17:53 <melwitt> okay. I was using http://docs.openstack.org/developer/nova/cells.html#global-api-level-tables as a guide on how to think about it and currently we have security_groups in the api db, or maybe it will be both?
17:18:23 <alaski> melwitt: do we write to security groups outside of the api, do you know?
17:19:17 <melwitt> alaski: in general I don't think so but I'd have to check to be sure. what happens now is during instance_create, it will create the default sec group if it doesn't exist
17:20:51 <alaski> okay, I trying to think through cases here
17:20:55 <alaski> *I'm
17:21:29 <alaski> I think my first question is does changing a security group apply to all instances with that group?
17:21:54 <alaski> I feel like we should treat it like a flavor, and write it to instance_extra with the instance in the cell db
17:22:09 <alaski> but if changing the group is supposed to propagate that won't work
17:22:44 <melwitt> I'm not sure how it works but SecurityGroup has an orm relationship with SecurityGroupInstanceAssociation which has a FK of instance.uuid
17:23:50 <melwitt> *orm relationship Instance, SecurityGroupInstanceAssociation
17:24:04 <alaski> I hate to say it, but we might need to look at breaking that
17:25:06 <alaski> in general I'd like to move towards the instance picking up attributes as it moves through the system and storing those itself.  like with flavor
17:25:36 <alaski> but at this point I'm not sure I have a great answer
17:26:08 <melwitt> okay. there might be other things too but in my test I started with doing an instance.create() and saw it going to the sec groups so I stumbled over it
17:26:27 <melwitt> yeah, something to think about
17:26:43 <alaski> for now it would be good to have a list of everything the instance object relies on having access to
17:27:18 <alaski> we'll probably have to go case by case and come up with solutions
17:27:58 <alaski> #topic Open discussion
17:28:10 <melwitt> okay, I'll go through and make a list. do you think we only have to be concerned with Instance?
17:28:34 <alaski> melwitt: probably not
17:28:40 <alaski> but for now it's the big concern
17:28:58 <alaski> actually, it might just be instance
17:29:39 <dheeraj> alaski: feature freeze is next week, so does that mean all code has to be merged before that or is there an exception procedure?
17:29:55 <alaski> dheeraj: there is an exception process
17:30:07 <alaski> but I'm not sure any of our stuff will qualify
17:30:23 <alaski> mainly because it doesn't complete anything
17:30:36 <dheeraj> alaski: yeah. In that case we should probably just roll it over to M
17:30:36 <melwitt> johnthetubaguy commented on the review also mentioning about, what will be the default database? I had been thinking "default" is whatever is local to the code querying, that is, each database will be configured for its local database. but then that wouldn't work to let it go to the default if it's say, querying for security groups. does that mean we'll need to actually have for example, get_api_session() for all security group related calls
17:30:37 <melwitt> inside the db api?
17:31:24 <alaski> melwitt: my thinking has been that anything we know is in the api db shoudl use get_api_session
17:31:37 <melwitt> alaski: okay, that makes sense
17:31:41 <alaski> and the context switching is primarily for things that could be in one of many cells
17:31:53 <dheeraj> alaski: maybe compute cells will need to know about the api cell
17:32:09 <alaski> dheeraj: that's seeming more and more likely
17:32:18 <melwitt> alaski: yeah, I think that would make it clearer to understand what's going on anyway too
17:32:24 <alaski> dheeraj: but something I would like to avoid as much as possible
17:32:30 <alaski> melwitt: cool
17:32:35 <vineetmenon> alaski: do we have to redo commits if we roll over to M ?
17:32:43 <vineetmenon> specs and code commits?
17:33:01 <alaski> vineetmenon: for specs, maybe.  that was unclear at this point.  code shouldn't need to
17:33:42 <alaski> vineetmenon: but if specs need to be redone it's a trivial approval process
17:33:49 <melwitt> I need to dig into how the sqlite test database stuff works because I find if I create two EngineFacade intended to point to the same database, it doesn't work. one of them errors saying no tables exist. (I'm trying to verify that different ones get used depending on context, throwing away the "connection" kwarg before letting it be created so they go to the same database really)
17:34:46 <melwitt> or should I instead be looking at actually making separate test databases and doing it that way?
17:35:22 <melwitt> (for the functional test under nova/tests/functional/db)
17:35:22 <alaski> I vaguely recall running into something like that
17:35:47 <alaski> from what I remember the sqlite db is created in memory so each connection ends up with it's own view
17:35:59 <alaski> so you might need to run migrations on each
17:36:36 <melwitt> that's definitely what I'm seeing. I know from some doc or something I found somewhere that it's indeed an in-memory sqlite db
17:37:58 <alaski> you might need to precreate the facades and run migrations on them, rather than letting them be lazily created in the tests
17:38:05 <alaski> if I'm understanding correctly
17:38:12 <melwitt> what I don't understand is how "database" exists, migration.db_sync(database=self.database) where self.database is just "main"
17:38:34 <melwitt> from the Database fixture
17:38:47 <dheeraj> melwitt: it runs API migrations if database="api" and nova DB mingrations if database="main"
17:38:59 <melwitt> is "main" indicating the EngineFacade or something else?
17:39:48 <melwitt> like how does it know from just the string "main" how to get to a thing to run migrations on
17:40:09 <dheeraj> It is coded somewhere in fixtures
17:40:17 <dheeraj> I think alaski committed that code
17:40:39 <alaski> it's all in https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migration.py#L90
17:41:37 <melwitt> okay, so it'll create its own new migrate repo if it doesn't exist
17:42:03 <melwitt> I think I understand enough to go that direction. thanks
17:42:14 <alaski> it looks like only 'api' has meaning, and anything else just does the old default
17:42:42 <dheeraj> alaski: yes
17:42:54 <dheeraj> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migration.py#L1146
17:43:46 <alaski> melwitt: sorry, I can't vouch for the prettiness of the code, only that I got it to work
17:43:48 <melwitt> I found that sync step actually creates the EngineFacade
17:43:56 <alaski> ahh
17:44:12 <alaski> oh, it calls get_engien
17:44:29 <melwitt> oh, right. that's what would do it
17:45:54 <melwitt> alaski: it's just new to me, didn't know what was happening :)
17:46:28 <alaski> melwitt: heh, no worries.  I remember it taking me some time to understand enough to do that migration piece
17:46:55 <alaski> and it's mostly disconnected, except for that get_engine call
17:47:29 <melwitt> yeah, I was wondering how does the EngineFacade connect to the seemingly connectionless testing sqlite database
17:48:10 <alaski> yeah, and the in-memory piece makes sqlite interesting compared to the other backends
17:48:22 <alaski> anything else for today?
17:49:17 <alaski> have a great day everyone
17:49:20 <alaski> #endmeeting