14:00:29 #startmeeting Nova Live Migration 14:00:30 Meeting started Tue Mar 29 14:00:29 2016 UTC and is due to finish in 60 minutes. The chair is PaulMurray. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:32 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:34 The meeting name has been set to 'nova_live_migration' 14:00:40 o/ 14:00:43 o/ 14:00:46 o/ 14:01:25 hi - I'm surprised to find people here - its been so quite today 14:01:45 * davidgiluk was here an hour ago, but then I noticed this is a UTC meeting :-) 14:01:46 o/ 14:01:51 Agenda: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration 14:02:10 lets wait one more minute in case of late arrivals 14:02:15 o/ 14:02:43 yeah, at the very beginning I wanted to say thanks for remainding about time changes, i'd fail hard today ;) 14:03:11 PaulMurray: ^ 14:03:18 #topic Bugs 14:03:32 https://bugs.launchpad.net/nova/+bug/1550250 14:03:32 Launchpad bug 1550250 in OpenStack Compute (nova) "migrate in-use status volume, the volume's "delete_on_termination" flag lost" [High,In progress] - Assigned to YaoZheng_ZTE (zheng-yao1) 14:03:51 markus_z sent out a mail about this one 14:04:25 #link mail thread about "delete_on_termination" bug http://lists.openstack.org/pipermail/openstack-dev/2016-March/090684.html 14:04:43 Does anyoe have an opinion on this bug ? 14:05:17 it seems volumes lose their delete_on_termination flag on live migration 14:05:25 is this somehow related to live migration? 14:05:35 thought that this is cinder migration, not nova's live migration 14:05:52 4.run cinder migrate volume 14:06:01 (one of provided steps) 14:06:07 Yup, that's not nova live migration 14:06:21 ah, your right - I was given the impression it was nova, but it looks like cinder ? 14:07:03 ok - moving on 14:07:09 (embarrased) 14:07:37 are there any bugs anyone wants to bring up? 14:08:12 #topic Summit sessions 14:08:48 Just a reminder to add anything you want considered to the etherpad 14:09:06 #link summit sessions: https://etherpad.openstack.org/p/newton-nova-summit-ideas 14:09:41 #topic Update libvirt domain xml interface section on live migration 14:09:54 this added by scheuran 14:09:57 Hi 14:10:01 scheuran, do you want to take this 14:10:04 yep 14:10:30 so my goal is to update the interface section of the domain.xml before live migration starts 14:10:59 I have 2 use cases 14:11:03 scheuran: On the destination I assume? 14:11:17 during pre_live migration 14:11:33 but update the xml with the destination information, right 14:11:49 #1 Live Migration with newly added neutron macvtap agent 14:11:57 #2 live migration cross neutron agents 14:12:14 e.g. migrate from a host with linuxbridge agent to a host with ovs agent 14:12:38 what I need for this is the vif information for the destinatnion host 14:12:58 neutron generates this when the update_binding_host_id call is made in post live migration 14:13:09 is this something that you can take only from the destination? I mean, this vif information 14:13:23 pkoniszewski, from the neutron server 14:13:30 and only from the neutron server 14:13:37 okay 14:13:57 so in pre_live migration i need to call neutron, and ask for the vif information 14:14:15 but this information gets updated in post live migration today, like described before 14:14:38 is there any incomatibility between agents? or can you live migration from any to any? 14:14:43 so what I want to discuss is, if we can make the udpate_binding_host_id call in pre_live migration insteaed of post_live migration 14:15:03 pkoniszewski, today you cannot migrate between agents 14:15:09 scheuran, at the moment the port binding update triggers creating networking on dest 14:15:11 i mean from neutron perspective 14:15:16 as the on the target, nova plugs always the old vif type 14:15:28 yeah, that's right 14:15:32 there are a few neutron-nova bugs that want network setup in pre_live_migrate 14:15:36 pkoniszewski, no, nothing from Neutron 14:15:46 okay 14:16:04 PaulMurray, right, I'm aware of them - but they all do not solve this issue 14:16:22 #link https://review.openstack.org/#/c/297100/ 14:16:27 this is the prototype I did 14:16:36 it's working for the good cases 14:17:06 however I'm still looking for a way how to rollback the portbinding if migration failed... 14:17:24 don't you have old domain XML on source? 14:17:30 yes I have 14:18:00 and I need to update it with the new vif information 14:18:10 Probably stupid question as I know very little about networking: I assume it's possible to have these different agents on the same segment as presented to the vm? 14:19:04 mdbooth, yes, those agents can serve the same network segment 14:19:20 but not on the same host of course 14:19:51 but you could have a mixed cloud, running linuxbridge and running ovs and everybody could talk to each other.. 14:20:03 scheuran, does updating the port binding affect the source networking ? 14:20:16 PaulMurray, no 14:20:28 it's just a database operation 14:20:52 I saw you removed the migreate_instance_finish() - what does that do ? 14:21:47 scheuran: so, you are asking about rollback, maybe a stupid question, isn't rollback_live_migration in compute manager enough? if source networking is unaffected it should work 14:21:57 PaulMurray, did I? In which file? 14:22:13 in compute manager 14:22:14 https://review.openstack.org/#/c/297100/2/nova/compute/manager.py 14:22:21 its commented out 14:22:23 https://review.openstack.org/#/c/297100/2/nova/compute/manager.py@5562 14:22:34 PaulMurray, a right 14:22:52 the only purpose of this method is updating the binding_host_id 14:23:02 today 14:23:37 so for production code we could move things around a bit that this hook is still present 14:24:03 pkoniszewski, is rollback_live_migration executed in every case? 14:24:20 if something after pre_live_migration fails - yes 14:24:36 including pre_live_migration step 14:24:51 pkoniszewski, yeah - the problem is, that today won't fail in pre livmigration 14:25:02 but during migration operation 14:25:10 when libvirt tries to define the xml on the destination 14:25:12 it's okay, LM monitor will trigger rollback 14:25:20 and the requested devices are not present... 14:25:47 you mean when LM is already finished from hypervisor perspective? 14:26:19 ah ok, got it 14:26:29 pkoniszewski, not sure - when libvirt is executing the migration 14:26:42 pkoniszewski, not sure if nova treats this as already finished... 14:27:10 this is fine, live_migration_operation goes in seperate thread 14:27:20 we will still start monitor 14:27:23 that will call rollback 14:27:36 pkoniszewski, ok, so I'll give it a try! 14:28:05 just to name the alternatives.. 14:28:38 it would be a new neutron api, that returns that vif information for the destination - without persisting it in the database... 14:28:56 or allow a port to be bound to 2 nodes (source & target) 14:29:21 but just doing the portbinding in pre_live migration seemed to be the most easiest way 14:29:55 we used the last approach that you mentioned for volumes 14:30:10 we use * 14:30:29 pkoniszewski, so that during migration both hosts own the volume? 14:30:44 scheuran, volumes are attached to both hosts 14:30:50 during migration 14:30:57 we have connection open to both hosts, even if nova fails during LM, instance will keep operating 14:31:08 pkoniszewski, ok I see 14:31:16 sounds like the most secure approach to me 14:31:24 but in the database the owner is still the source, until migration finished, right? 14:31:41 the owner is an instance which does not change 14:31:41 but its slightly different - volumes can be used from both hosts 14:31:50 with networking we want only one to get packets 14:31:52 right, cause for neutron its just a database problem 14:32:12 physcially I wouldn't do anything other than today 14:32:32 it's "just" about updating the database record 14:33:00 scheuran, do you have a spec for this ? 14:33:08 so during migration, the database says that port is bound to destinatnion, although it is still active on source 14:33:17 PaulMurray, not yet 14:33:28 PaulMurray, I first wanted to get a feeling which approach is the best one 14:33:43 I understand 14:33:56 specs are a good way to get wider opinion as well 14:34:06 PaulMurray, ok 14:34:06 so when you think you have an idea 14:34:13 its good to write it down 14:34:18 also creating a blueprint? 14:34:22 yes 14:34:37 blueprints are really only used fro tracking 14:34:47 but the spec will get reviewed and you get feedback 14:34:53 PaulMurray, ok, than this is my todo until next week.. 14:34:57 *then 14:35:30 I'm a Neutron guy - so not very familar with the nova process.. 14:35:40 so spec + bp, perfect 14:35:41 no worries, we're friendly 14:35:45 :) 14:36:09 scheuran: also if you have spec, you can discuss it during nova unconference session on summit 14:36:24 pkoniszewski, good point 14:36:40 I already added this topic to the nova-neutron topics (in the neutron etherpad) 14:36:51 which is actually the best way to clarify things that can be implemented different ways 14:37:21 ok. so to summarize - I'll try out the rollback stuff 14:37:29 and come up with a bp + spec until next week 14:37:49 good - thanks for bringing this to our attention 14:38:00 #topic Open Discussion 14:38:00 yes, thank you guys! 14:38:16 does anyone have anything else to brin gup ? 14:39:20 ok - thanks for coming 14:39:24 Hi! 14:39:39 luis5tb: Hi Luis 14:39:43 I would like to know if someone is taking a look at including post-copy live migration 14:40:08 luis5tb, are you interested in that ? 14:40:16 I've been working on including it (but for JUNO version) and would like to know if that would be of interest 14:40:45 what do you mean by "for Juno version" ? 14:40:55 yep, I want to take a look at the latest migration code and try to adapt it (many things have changed since then) 14:41:23 I integrated post-copy into nova (OpenstacK Juno release) a year ago 14:41:42 I think there is interest 14:41:50 there is a list here: https://etherpad.openstack.org/p/newton-nova-live-migration 14:42:02 it is on the list - you could add yourself 14:42:11 or rather anything you want to add as information 14:42:12 I saw points 6 and 7 14:42:21 but if I understand the next step, is that it would be to write a spec? 14:42:46 davidgiluk, yes, took the words out of my mouth 14:42:57 I was actually wondering if someone is already doing that 14:43:00 ? 14:43:08 I not then please do 14:43:15 * davidgiluk doesn't know of anyone writing a spec 14:43:26 ok, just wondering if someone else already took a look into it, or is in the to do list for the future 14:43:46 ok 14:44:01 so, the libvirt change is not in yet 14:44:08 pkoniszewski: Oh yes it is! 14:44:11 luis5tb, i think we put it off before (as a group) because we had a lot to do 14:44:13 yep, it is 14:44:18 oh, i missed it then 14:44:22 also there was the 2.6 change 14:44:42 pkoniszewski: Got merged last week 14:44:59 good news 14:45:02 so may be a good time to move on with it 14:45:07 yeah 14:45:13 great 14:45:49 luis5tb, don't worry about waiting for others - if someone else wanted to do it they can get together with you 14:46:23 ok, I'll try to write a spec regarding work item 6 14:46:41 ok, I'll try to write a spec regarding work item 6/ 14:46:42 ok, I'll try to write a spec regarding work item 6/7 14:46:48 great - that's a big help 14:46:49 yup, i will be interested in helping there 14:46:50 thanks 14:47:03 pkoniszewski: Great 14:47:16 great 14:47:38 please add it to the list on the etherpad too 14:48:18 ok - anything else 14:48:29 * PaulMurray will give a big long pause this tiem 14:49:32 thanks for coming 14:49:36 #endmeeting