14:00:16 <PaulMurray> #startmeeting Nova Live Migration
14:00:17 <openstack> Meeting started Tue Jan 12 14:00:16 2016 UTC and is due to finish in 60 minutes.  The chair is PaulMurray. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:20 <openstack> The meeting name has been set to 'nova_live_migration'
14:00:29 <tdurakov> hi
14:00:30 <andrearosa> hi
14:00:39 <shaohe_feng> hi
14:00:42 <PaulMurray> Hi all
14:01:08 <PaulMurray> just wait around a minute to see if any else comes
14:01:12 <davidgiluk> hi
14:01:54 <eliqiao> o/
14:02:06 <PaulMurray> The agenda is on the meeting page: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration
14:02:13 <jlanoux_> o/
14:02:48 <PaulMurray> I have been ill so I only got round to putting the agenda up a while ago
14:03:09 <tdurakov> PaulMurray, could you add ci for agenda please
14:03:27 <PaulMurray> tdurakov, yes, certainly
14:03:44 <PaulMurray> In fact lets do that first
14:03:47 <PaulMurray> #topic CI
14:04:06 <tdurakov> current status: https://review.openstack.org/#/c/252283/
14:04:17 <tdurakov> this patch add ceph for hook
14:04:23 <tdurakov> review and merge needed
14:05:38 <PaulMurray> #action ALL please review  https://review.openstack.org/#/c/252283/
14:06:26 <eliqiao> seems all logic are in run_tests.sh, would that be better to split it out?
14:06:27 * PaulMurray the internet is being a little slow for me today
14:06:50 <tdurakov> eliqiao, sounds reasonable
14:07:09 <tdurakov> eliqiao, check Sebastien comment also
14:07:42 <eliqiao> oh, gold, already version 73,  I am so late
14:08:02 * eliqiao :(s/gold/god
14:08:52 <PaulMurray> I was going to mention this in the next topic: https://review.openstack.org/#/c/227278/ failed the live migration CI
14:08:54 <pkoniszewski> o/
14:09:24 <tdurakov> eliqiao, it could be splitted, Sebastien talking about reusing ceph from devstack/lib
14:09:39 <pkoniszewski> PaulMurray: its already fixed
14:09:43 <eliqiao> tdurakov: cool.
14:09:59 <PaulMurray> pkoniszewski, was that the live migration job or the review at fault?
14:10:27 <tdurakov> PaulMurray, what ps number?
14:10:48 <PaulMurray> 25
14:11:48 <PaulMurray> pkoniszewski, from your comment:
14:11:51 <PaulMurray> I'm currently investigating it, but so far it looks like this fail is not related to my changes. Tests without NFS worked correctly:
14:11:51 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration [66.216203s] ... ok
14:11:51 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused [19.978373s] ... ok
14:11:52 <PaulMurray> With NFS one test passed and another one failed:
14:11:56 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration [15.971409s] ... FAILED
14:11:59 <PaulMurray> tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused [18.673453s] ... ok
14:12:02 <PaulMurray> Let's see the result on BLM with config drive patch
14:12:19 <pkoniszewski> PaulMurray: looks to me like it was a test issue, but Matt found another issue in another patch, no idea right now which one caused this
14:12:36 <pkoniszewski> I can see in my environment that everything works fine now
14:12:42 <pkoniszewski> so I pushed another patch
14:13:05 <PaulMurray> I saw that the config drive patch passed
14:13:42 <PaulMurray> so as we are discussing it.....
14:13:48 <PaulMurray> #topic Priority reviews
14:14:02 <PaulMurray> pkoniszewski, how is that series going?
14:14:12 <PaulMurray> you got one merged
14:15:05 <pkoniszewski> There is one concern from Daniel, but its just naming issue in old dicts, I need to dump data and respond to his review
14:15:51 <pkoniszewski> config drive patch is ready
14:16:01 <PaulMurray> #link https://review.openstack.org/#/q/topic:bp/block-live-migrate-with-attached-volumes
14:16:33 <PaulMurray> ok - we'll keep looking to review it
14:16:49 <PaulMurray> The next one I wanted to call out was
14:17:02 <PaulMurray> #link https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration
14:17:42 <pkoniszewski> so I need to go through code carefuly to see how andrearosa's proposition would work
14:17:55 <pkoniszewski> probably they are right and we can move logic down to drivers
14:18:09 <andrearosa> pkoniszewski: feel free to ping me if my comments are not clear
14:18:15 <pkoniszewski> my idea was to keep this at a high level (compute api) to be able to throw correct instance action
14:19:18 <pkoniszewski> this is async so there is no other option to notify about which action is taken at a lower level, or I don't know something about nova's notifications
14:23:42 <pkoniszewski> PaulMurray timed out :<
14:23:48 <eliqiao> :(
14:24:09 <pkoniszewski> let's move forward, not sure we will be able to finish the meeting or add links and so on
14:24:18 <pkoniszewski> #topic Bugs
14:24:29 <pkoniszewski> nah, it doesn't listen :<
14:24:57 <pkoniszewski> so just a kindly reminder that we are tracking bugs in etherpad - https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
14:25:15 <pkoniszewski> most of them need reviews
14:26:08 <eliqiao> yeah, hard work
14:26:10 <pkoniszewski> there is a new series to deprecate migration flags config
14:26:29 <kashyap> pkoniszewski: Randomly chiming in, yeah, it avoids a lot of confusion
14:26:39 <eliqiao> pkoniszewski: I see, Dansims gives +2 on them.
14:27:01 <kashyap> Because, there are a lot of flags and tunables that libvirt allows, unless one really has tested all them, it's hard to remember which combination is suitable.
14:27:21 <pkoniszewski> so every review right now is appreciated, we are getting close to midcycle, I believe that we will be able to work on some changes and merge some of them at the midcycle
14:27:42 <eliqiao> pkoniszewski: seems they are all in high prority, get cores' eyes already.
14:28:17 <pkoniszewski> kashyap: exactly, this series will remove a lot of confusions around LM configuration and will give nova more flexibility to configure it on correctly
14:28:55 <pkoniszewski> a lot of patches are very close
14:29:20 <kashyap> Yeah, the bug that triggered it all - https://bugs.launchpad.net/nova/+bug/1441054
14:29:21 <openstack> Launchpad bug 1441054 in OpenStack Compute (nova) "live-migration --block-migrate fails with default libvirt flags" [Medium,Incomplete] - Assigned to Mathieu Rohon (mathieu-rohon)
14:29:54 <pkoniszewski> andrearosa: do you know what Paul wanted to say about midcycle?
14:29:55 <kashyap> In the time the bug was reported, and now, the said issue seems to be fixed there.
14:30:15 <pkoniszewski> it starts in a less than 2 weeks
14:30:27 <andrearosa> pkoniszewski: nope he is not in the office, I guess he just want to start to discuss about some topics we want to put in the agenda for the meetup
14:31:24 <pkoniszewski> okay, so here is the etherpad for midcycle - https://etherpad.openstack.org/p/mitaka-nova-midcycle
14:32:16 <eliqiao> pkoniszewski: thx, we should file something for LM
14:32:22 <pkoniszewski> live migration will be discussed on Tuesday
14:32:44 * andrearosa is trying to reach PaulMurray on different channels
14:32:46 <PaulMurray> Hi, am I back in?
14:32:49 <pkoniszewski> PaulMurray:  is bad
14:32:52 <pkoniszewski> back *
14:32:52 <pkoniszewski> sorry
14:32:53 <pkoniszewski> :D
14:33:10 <eliqiao> pkoniszewski: will you attend mid-cycle?
14:33:14 <PaulMurray> Seems I got disconnected
14:33:15 <pkoniszewski> PaulMurray: we moved forward to midcycle, just started this topic and mentioned about etherpad
14:33:30 <PaulMurray> cool
14:33:46 <pkoniszewski> eliqiao: yes, I will participate
14:34:20 <eliqiao> pkoniszewski: cool.
14:34:28 <PaulMurray> tdurakov, did you get your visa?
14:34:45 <tdurakov> PaulMurray, yep, will attend mid-cycle
14:35:05 <PaulMurray> great - there's a reasonable turn out all round
14:35:24 <PaulMurray> there will be several people who work on libvirt and qemu there
14:35:43 <PaulMurray> so its a good time to bring up anything needed there
14:35:49 <tdurakov> great!
14:36:41 <tdurakov> pkoniszewski, could we discuss your patch and ci failure after this meeting?
14:36:55 <pkoniszewski> tdurakov: sure!
14:37:07 <PaulMurray> There are a few virt layer items at the bottom of our eatherpad
14:37:20 <PaulMurray> please add there if there is anything that I haven't included
14:37:43 <PaulMurray> see: https://etherpad.openstack.org/p/mitaka-live-migration
14:37:56 <PaulMurray> Live Migration Libvirt / qemu Work Items
14:38:13 <PaulMurray> This is to do with lining up work we may depend on in the future
14:38:27 <PaulMurray> the people coming may be able to take some on
14:39:10 <PaulMurray> #topic open discussion
14:39:23 <PaulMurray> Anything anyone wants to discuss?
14:39:47 <pkoniszewski> I see that there is series about deprecating migration flags in open discussion
14:40:14 <pkoniszewski> is owner of the series there?
14:40:26 <PaulMurray> I don't think so
14:40:51 <PaulMurray> its markmc
14:42:00 <PaulMurray> pkoniszewski, did you follow the thread?
14:42:04 <tdurakov> do we really need such constraint for operators?
14:42:08 <pkoniszewski> yes, I did
14:43:06 <pkoniszewski> tdurakov: what do you mean?
14:43:27 <eliqiao> hiden flgs from admin
14:43:35 <tdurakov> eliqiao, yep
14:44:05 <pkoniszewski> well, I don't think that there is reason to expose all these flags that we currently expose to operators
14:44:21 <pkoniszewski> we allow them, e.g., to misconfigure nova
14:44:22 <eliqiao> I commented to have a log for flags.
14:44:49 <tdurakov> from my sight it should be enough to have warning in logs
14:44:55 <eliqiao> markmc told it's hard to convert from bit to string.. and also havs warings..
14:45:16 <eliqiao> tdurakov: hmm.. that would be not easy to findout all flags
14:45:24 <tdurakov> e.x. what about post-copy mode for live-migration?
14:46:12 <pkoniszewski> depends how it will impact performance of different workloads
14:46:35 <pkoniszewski> imo there should be another way to trigger post copy live migration, but I don't know much about its impact yet
14:46:45 <davidgiluk> pkoniszewski: What would you like to know
14:46:46 <eliqiao> Add a new switch flag to indicate to use post-copy or not, that what I can image onl.
14:46:48 <PaulMurray> ok - got back in
14:47:07 <tdurakov> pkoniszewski, afair we need to provide flag
14:47:33 <tdurakov> so if wee hide flags from operator - we need to provide some flag in cli/rest-api
14:47:50 <tdurakov> do we really need this?)
14:47:57 <PaulMurray> #chair andrearosa
14:47:58 <openstack> Current chairs: PaulMurray andrearosa
14:48:00 <eliqiao> ops. a new REST api ?
14:48:10 <pkoniszewski> eliqiao: just a flag
14:48:31 <tdurakov> eliqiao, if there is no way to change flags you need to change api
14:48:46 <pkoniszewski> tdurakov: I just don't know in which case you would want to trigger only post-copy live migrations
14:48:58 <pkoniszewski> I'm not a fan of all these static configurations in nova.conf
14:49:13 <eliqiao> pkoniszewski: add new options to live-migration CLI to passing flags, it make some scense to me.
14:49:15 <pkoniszewski> they are way too static for different kind of workloads and different sizes of VMs
14:49:25 <tdurakov> eliqiao, yep, new options
14:50:16 <tdurakov> but we recently going to remove disk-over-commit option from cli, no?
14:50:29 <eliqiao> tdurakov: cool, that's a good suggestion I think, (but maybe some one will say we don't need to expose low level details)
14:50:31 <pkoniszewski> its not used anymore afaik
14:50:47 <eliqiao> tdurakov: yeah, that's what I am working on
14:51:02 <tdurakov> it looks inconsistent imho
14:51:22 <tdurakov> to hide one flag and then add new one
14:51:27 <pkoniszewski> davidgiluk: I would like to know how post-copy affects databases and comparison to pre-copy
14:51:39 <tdurakov> but yes, this need to be discussed more
14:52:20 <davidgiluk> pkoniszewski: OK, PM for a discussion about performance and stuff, but the 'howto use' it is you must select it before the start of migration
14:52:42 <davidgiluk> pkoniszewski: The simple answer is that with postcopy the migration always completes, irrespective of the workload
14:53:04 <eliqiao> davidgiluk: do you konw the progress post-copy migration supported in libivrt, I don't find any patch about it.
14:53:14 <pkoniszewski> that's true, it ends in a finite time
14:53:21 <tdurakov> davidgiluk, tradeoff is no way to rollback it, right?
14:53:32 <eliqiao> yes, can not revert
14:53:34 <davidgiluk> eliqiao: jdenemar is currently tidying it up for posting, I need to sort out a couple of things for him on the qemu side
14:53:40 <PaulMurray> also slower vm if have to fetch
14:53:43 <pkoniszewski> tdurakov: yes, and in case of network failure you will need to reboot VM
14:53:54 <PaulMurray> alot
14:53:58 <davidgiluk> tdurakov: Right, once it's in postcopy mode you can't abandon the migrate since the destination has started writing to disk and sending packets etc
14:54:34 <davidgiluk> PaulMurray: Right, although normally the postcopy phase is only a short part of it, so that finishes pretty quickly
14:55:44 <eliqiao> davidgiluk: seems we are far away to support post-migration since it's not merged in libvirt yet?
14:56:03 <tdurakov> from my sight post-copy is one of examples where we need to decide, re we going to hide flags from conf and add extra cli/api params, or leave conf as-is
14:56:24 <davidgiluk> eliqiao: not merged in libvirt yet, but reasonably close; there's also a demo that Umea uni have of an implementation on openstack
14:56:37 <pkoniszewski> eliqiao: per my discussion with Daniel its very close and it's real to get it done in N/O-release of OpenStack
14:57:03 <eliqiao> pkoniszewski: do you mean in libvirt?
14:57:15 <pkoniszewski> tdurakov: we can just say in nova.conf that all live migrations will be post-copy or all live migrations will be pre-copy
14:57:19 <pkoniszewski> tdurakov: it's way too static
14:57:29 <davidgiluk> pkoniszewski: Yeh that's way too static
14:57:48 <pkoniszewski> tdurakov: so I strongly prefer to remove these flags from nova.conf, we need better approach
14:57:51 <andrearosa> pkoniszewski: you meant we can't
14:58:01 <pkoniszewski> andrearosa: yes, I meant that we can :D
14:58:04 <pkoniszewski> can't...
14:58:07 <tdurakov> :)
14:58:12 <andrearosa> ok, thanks
14:58:21 <eliqiao> almost done, sorry to interupt, can anyone help to review this topic #link https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/making-live-migration-api-friendly
14:58:25 <PaulMurray> We are coming to the end of time
14:58:37 <pkoniszewski> PaulMurray: just one more question
14:58:44 <PaulMurray> yes
14:58:51 <pkoniszewski> PaulMurray: will we discuss LM agenda for midcycle before midcycle?
14:58:55 <eliqiao> PaulMurray: that one is not discuess last week(I missed), just hight light
14:58:57 <pkoniszewski> like, next week or on a mailing list?
14:59:19 <PaulMurray> I tried to discuss it today when I got dropped
14:59:50 <PaulMurray> I'll put an eatherpad link on the live migration link
15:00:00 <PaulMurray> and we can list items there
15:00:05 <PaulMurray> then go over next week
15:00:09 <PaulMurray> sound ok?
15:00:14 <PaulMurray> I'll send an email
15:00:25 <pkoniszewski> sounds good!
15:00:27 <pkoniszewski> thanks
15:00:32 <PaulMurray> #endmeeting