14:00:31 <PaulMurray> #startmeeting Nova Live Migration 14:00:32 <openstack> Meeting started Tue Jun 7 14:00:31 2016 UTC and is due to finish in 60 minutes. The chair is PaulMurray. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:35 <openstack> The meeting name has been set to 'nova_live_migration' 14:00:43 <mriedem> o/ 14:00:45 <diana_clarke> o/ 14:00:47 <claudiub> o/ 14:00:50 <paul-carlton2> o/ 14:00:50 <tdurakov> o/ 14:00:55 <luis5tb> o/ 14:01:02 <mdbooth> o/ 14:01:03 <PaulMurray> Agenda: https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration 14:01:16 <PaulMurray> I slightly changed the agenda 5 minutes ago 14:01:17 <pkoniszewski> o/ 14:01:36 <andrearosa> hi 14:01:40 <PaulMurray> #topic Approved non-priority specs 14:02:00 <PaulMurray> I thought it would be a good idea to quickly go over the approved specs 14:02:14 <PaulMurray> #link https://review.openstack.org/#/c/306561 Automatic Live Migration Completion (paul-carlton) 14:02:33 <PaulMurray> paul-carlton2, has a request for implementers for this 14:02:43 <PaulMurray> luis5tb, did you want to be involved ? 14:02:52 <luis5tb> Yes, I'm happy to help with that! 14:02:54 <davidgiluk> o/ 14:03:23 <pkoniszewski> i can help too, as i have patch with flags up for review (abandoned right now as it was waiting for newer ubuntu in gate) 14:03:50 * andrearosa is available as well 14:03:58 <PaulMurray> davidgiluk, were you just waving or volunteering? 14:04:24 <davidgiluk> PaulMurray: A bit of both; I don't know the innards of openstack but I'm happy to help from the qemu side if anyone gets stuck with the postcopy or similar 14:05:08 <PaulMurray> The spec was a combination of luis5tb, paul-carlton2 and pkoniszewski specs I think, so makes sense for luis5tb and pkoniszewski to lead it I think 14:05:14 <PaulMurray> did I get that right ? 14:05:33 <pkoniszewski> yeah, we can go that way i think 14:05:33 <PaulMurray> andrearosa, is ready to help 14:05:52 <PaulMurray> so please get together and organise as you see fit 14:06:11 <PaulMurray> next 14:06:15 <PaulMurray> #link https://review.openstack.org/#/c/307131 Live Migration of Rescued Instances (paul-carlton) 14:06:27 <PaulMurray> I think you have this pretty much done paul-carlton2 ? 14:07:01 <paul-carlton2> Yep, first part of implementation is ready for review https://review.openstack.org/308198 14:07:10 <PaulMurray> this is in the reviews section on the agendahttps://review.openstack.org/#/c/308198 14:07:14 <PaulMurray> oops 14:07:26 <paul-carlton2> I need to do a tempest test and implement xenAPI driver too 14:07:37 <PaulMurray> #link rescue patch https://review.openstack.org/#/c/308198 14:07:56 <PaulMurray> good - thanks 14:08:10 <PaulMurray> #link https://review.openstack.org/#/c/283421/: Modify live_migration API response content (eliqiao) 14:08:23 <PaulMurray> eliqiao, are you set for that one ? 14:09:06 <PaulMurray> I see alex_xu is the primary on that spec 14:09:07 <tdurakov> PaulMurray: I thought it just update for implemented spec, no? 14:09:25 <PaulMurray> ah -oops 14:09:33 <PaulMurray> yes thanks, I knew there was one of those 14:09:41 <PaulMurray> I got my list mixed up - ignore 14:09:45 <PaulMurray> #undo 14:09:45 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x7fe237707ad0> 14:10:11 <PaulMurray> #link https://review.openstack.org/#/c/296408/: Re-Proposes to check destination on migrations (bauzas) 14:10:15 <alex_xu> yea, it is a update 14:10:56 <PaulMurray> I will catch up with bauzas later - I expect this is being tracked under scheduler 14:10:58 <bauzas> PaulMurray: it's now implemented and merged 14:11:13 <bauzas> PaulMurray: it's just missing the novaclient changes, that's it 14:11:21 <PaulMurray> cool - thanks - I'll mark that down as lm sub group progress :) 14:11:35 <PaulMurray> bauzas, do you need any help with that ? 14:12:01 <bauzas> PaulMurray: nope, it's fine :) 14:12:11 <bauzas> I should upload the changes by the end of this week 14:12:19 <PaulMurray> good, thank you 14:12:26 <PaulMurray> #link https://review.openstack.org/#/c/292271/: Remove compute-compute communication in live-migration (tdurakov) 14:12:30 <PaulMurray> and 14:12:41 <PaulMurray> https://review.openstack.org/#/c/320416/: Make checks before live-migration async (tdurakov) 14:12:48 <PaulMurray> I think these go together ? 14:13:07 <PaulMurray> tdurakov, how is it going ? 14:13:10 <tdurakov> it's mine, first one is in progress, going to submit new patchset tommorow, second one need some review https://review.openstack.org/#/c/314932/ 14:13:43 <PaulMurray> good - thanks 14:14:30 <PaulMurray> Please put reviews on https://etherpad.openstack.org/p/newton-nova-priorities-tracking as you go and keep up to date 14:14:43 <tdurakov> sure 14:14:53 <PaulMurray> That's for everyone ^^^ 14:15:10 <mriedem> bauzas: is there any way to test this in tempest? https://review.openstack.org/#/c/318334/ 14:15:22 <PaulMurray> Also remember the FF is quite short for non-priority 14:15:40 <PaulMurray> So please all help each other out with reviews 14:15:42 <mriedem> we can't test evacuate in the gate right now, but otherwise 14:15:46 <bauzas> mriedem: well, testing evacuate in Tempest could be terrible, but maybe for live-migrate, yes 14:16:12 <bauzas> mriedem: tbh, I don't know yet on how live-mig works good for Tempest in the gate 14:16:12 <mriedem> it's just making sure the instance migrates to the host you specify right? 14:16:17 <bauzas> yup 14:16:22 <mriedem> although with a 2 node setup that doesn't test much 14:16:24 <mriedem> :) 14:16:42 <mriedem> ok, nvm then. i just wanted to check before marking the bp complete. 14:17:18 <PaulMurray> So, moving on 14:17:24 <PaulMurray> on the subject of... 14:17:27 <PaulMurray> #topic CI 14:17:49 <PaulMurray> mriedem, to change an existing job to use raw instead of qcow2 14:17:55 <PaulMurray> mriedem, ? 14:18:06 <mriedem> PaulMurray: as noted in the agenda, that's hairier than i expected 14:18:23 <PaulMurray> #info devstack change was shot down ( https://review.openstack.org/#/c/318262/ ) - need to use post-config in devstack-gate to set libvirt.images_type 14:18:29 <mriedem> devstack people want the config to happen via post-config in devstack-gate, which isn't really streamlined yet 14:18:41 <mriedem> i haven't looked into devstack-gate yet to see if there is an easy way to hack that in 14:19:13 <mriedem> there is like a 400+ LOC change to d-g to add post-config / local.conf support, but i don't have the stomach to review that 14:19:42 <mriedem> mdbooth: have you been running the experimental queue on the refactor series to test with the lvm job? 14:19:55 <diana_clarke> mriedem: we're running it now 14:20:01 <mriedem> ok, cool 14:20:19 <mriedem> PaulMurray: so i wouldn't hold out hope for the raw backend job change anytime soon 14:20:24 <mriedem> since it's become less trivial 14:20:55 <PaulMurray> does anyone know anything about that post-config stuff mriedem is talking about ^^ 14:20:59 <PaulMurray> I have no idea 14:21:09 <mriedem> sdague: and mtreinish would 14:21:18 <mriedem> it's probably just a matter of asking around in -qa 14:21:33 <tdurakov> mriedem: post hook or post_config? 14:21:44 <sdague> mriedem: where is the local.conf patch, I'll review it today 14:21:49 <mriedem> i'd have to find it 14:22:05 <mriedem> https://review.openstack.org/#/c/311668/ ? 14:22:11 <sdague> I was spending my live migration energies fixing the cpu map issue 14:22:24 * mriedem hands sdague a cliff bar 14:22:32 <PaulMurray> +630, -1 14:23:08 <PaulMurray> are you ok taking that review sdague 14:23:17 <mriedem> so i guess the idea is nova would have a local.conf in tree for the job? i'm not really sure how this works yet 14:23:27 <mriedem> but i was planning on updating the postgres job to use the raw image backend 14:23:53 <mriedem> anyway, can be worked out after the meeting 14:24:00 <sdague> right, the point is that localrc is deprecated in devstack, d-g should use local.conf, that would allow jobs to pass arbitrary config options to project 14:24:42 <kashyap> mriedem: That above change says "Not current", but the Jenkins message says 'merged' 14:24:53 <kashyap> mriedem: "above change" == 311668 14:25:12 <mriedem> looks fine to me 14:25:34 <pkoniszewski> looks fine to me too 14:25:42 <PaulMurray> kashyap, go to latest patchset 14:25:43 <kashyap> Okay, disregard me then 14:26:10 <PaulMurray> well, there is a plan for now - i.e. make up plan after meeting when sdague has reviewed the review 14:26:19 <kashyap> sdague: While you're spending energies in that area, mind +Wing this? It had your ACK, but lost during a new patch set https://review.openstack.org/#/c/323970/ - "functions-libvirt: Add log filter to capture CPU driver errors" 14:27:17 <PaulMurray> There are a couple of comments about Live Migration job status in the agenda 14:27:26 <tdurakov> PaulMurray, mriedem, sdague is it ok: This change doesn't yet support grenade mode and multinode.? 14:27:29 <mriedem> sorry everyone, i have to run to other meetings 14:27:57 <sdague> tdurakov: no one is saying that change is right yet 14:28:05 <sdague> what we're saying is that is the path forward 14:28:11 <kashyap> PaulMurray: Yes, you're right. (Going to the latest patchset says it's merged, of course.) 14:28:18 <tdurakov> ok 14:28:33 <sdague> we've pushed back on every project that you can't just add new random shell variables to set 1 conf option 14:29:03 <sdague> because it makes everyone learn a new DSL, that's devstack specific, instead of knowing how anything is configured, and you end up with the lib/neutron-legacy that is unreadable by everyone 14:29:50 <sdague> I also don't think this is the top blocking issue by any regard, this just eliminates one possible part of the test matrix 14:30:16 <sdague> getting everything else in the live migration test matrix working should be further up the list 14:31:14 <PaulMurray> lets go onto the next item 14:31:24 <PaulMurray> tdurakov, anything to say ? 14:31:37 <PaulMurray> re: Currently blocked on bug https://bugs.launchpad.net/nova/+bug/1589591 - tdurakov has a patch up: https://review.openstack.org/#/c/326111/ 14:31:37 <openstack> Launchpad bug 1576093 in OpenStack Compute (nova) "duplicate for #1589591 block migration fail with libvirt since version 1.2.17" [High,In progress] - Assigned to Eli Qiao (taget-9) 14:31:54 <tdurakov> PaulMurray: It turns out to be a duplicate 14:31:58 <pkoniszewski> this is correct patch https://review.openstack.org/#/c/310707 14:33:03 <tdurakov> ^ this one is good, waiting for experimental job results and danpb aprove 14:33:33 <PaulMurray> interesting - I saw this on RHEL, but didn't realise it was in standard libvirt 14:33:55 <PaulMurray> so nothing extra needed then ? 14:34:11 <tdurakov> I think so 14:34:19 <pkoniszewski> we need to wait for experimental job anyway 14:34:34 <PaulMurray> is danpb going to look ? 14:34:47 <PaulMurray> i.e have you spoken to him about it ? 14:34:48 <pkoniszewski> doesn't look like he is online today 14:34:56 <kashyap> pkoniszewski: He is on -nova 14:35:04 <mdbooth> diana_clarke: The top of the refactor stack just passed the lvm job :) 14:35:13 <kashyap> (And a bunch of other places on OFTC, I've seen today) 14:35:18 <pkoniszewski> yeah, but he is not responding 14:35:19 <pkoniszewski> kashyap: ^ 14:35:39 <kashyap> pkoniszewski: Ah, then I don't know. Probably he's occupied with other stuff. 14:36:38 <PaulMurray> we'll need someone else anyway, so lets give it a few reviews ourselves 14:36:40 <tdurakov> experimental job results is top priority for now imo 14:37:34 <PaulMurray> put it at the top of the list on the tracking page 14:37:35 <tdurakov> PaulMurray: maybe action to review https://review.openstack.org/#/c/310707 ? 14:37:50 <PaulMurray> #action to review https://review.openstack.org/#/c/310707 14:37:56 <PaulMurray> like that ? 14:38:02 <tdurakov> yes:) 14:38:20 <PaulMurray> anything else on CI ? 14:38:30 <tdurakov> update on nfs 14:38:37 <tdurakov> I've set up xenial locally 14:38:46 <tdurakov> and start suspecting this line 14:38:47 <tdurakov> https://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/nfs.sh#L24 14:39:00 <tdurakov> i mean it supposed to be useless for xenial node 14:39:06 <tdurakov> s/node/nodes 14:40:03 <tdurakov> there are some update from 14.04 on idmapd service and it's dependency to nfs-kernel-server 14:40:12 <tdurakov> researching this 14:40:53 <tdurakov> that's all for me on CI 14:41:08 <PaulMurray> ok... not sure I get what you mean but as long as your working on it.... 14:41:35 <PaulMurray> #topic Libvirt Storage Pools 14:42:03 <PaulMurray> mdbooth, diana_clarke paul-carlton2 I think we got some update from you interspersed above 14:42:09 <diana_clarke> The new image backend methods are ready for review (and in the etherpad): https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:libvirt-instance-storage 14:42:18 <diana_clarke> The tempest tests are now passing for the patch that uses these new image backend methods, and 'check experimental' just passed too :) 14:42:40 <PaulMurray> fantastic 14:43:23 <PaulMurray> https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:libvirt-instance-storage 14:43:50 <PaulMurray> #action all keep up the reviews on libvirt instance storage 14:43:54 <paul-carlton2> specs https://review.openstack.org/#/c/310505/ and https://review.openstack.org/#/c/310538/ are still awating review 14:44:51 <paul-carlton2> I have a -1 on one but will address that when I'm done with live migration of rescued instances implementation 14:45:16 <paul-carlton2> Would appreciate feedback from others on these specs 14:45:24 <mdbooth> FYI, dansmith has asked us for an approach to the refactor series which makes the backend updates used immediately they merge. There might be a way to achieve that, so there may be a bit more churn in there yet. 14:45:32 <PaulMurray> This one https://review.openstack.org/#/c/310505/ (use libvirt storage pools) has questions from max - could do with some other feedback 14:46:13 <mdbooth> It might come to nothing, though, esp if it looks like too much work. 14:46:35 <paul-carlton2> I need to standup a ploop devstack to address max's comment 14:47:08 <paul-carlton2> will get to this in next week or so 14:47:15 <PaulMurray> mdbooth, have you seen the comments about ploop - would be good to get your opinion 14:47:34 <mdbooth> Reading. 14:48:07 <PaulMurray> mdbooth, might need to do offline 14:48:20 <mdbooth> PaulMurray: Yup, will comment in review. 14:48:26 <PaulMurray> thanks 14:48:50 <PaulMurray> Any thing else in this topic ? 14:49:25 <PaulMurray> #topic Review requests 14:49:40 <PaulMurray> this one in the agenda https://review.openstack.org/#/c/308198 14:49:47 <PaulMurray> mentioned above already 14:50:03 <PaulMurray> about lm for rescued instances 14:50:47 <PaulMurray> Anything else anyone wants to bring up for review ? 14:51:20 <PaulMurray> #topic Open Discussion 14:51:25 <PaulMurray> anything here ? 14:52:04 <PaulMurray> I think we're done for the week 14:52:08 <PaulMurray> thank you for coming 14:52:15 <PaulMurray> #endmeeting