14:00:15 <mriedem> #startmeeting nova
14:00:15 <openstack> Meeting started Thu Jun 30 14:00:15 2016 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:20 <openstack> The meeting name has been set to 'nova'
14:00:22 <alaski> o/
14:00:23 <markus_z> o/
14:00:24 <diana_clarke> o/
14:00:25 <rlrossit> o/
14:00:27 <lyarwood> o/
14:00:41 <edleafe> \o
14:01:21 <lbeliveau> o/
14:01:22 <woodster_> o/
14:01:28 <andrearosa> hi
14:01:29 <takashin> o/
14:01:30 <dansmith> o/
14:01:46 <johnthetubaguy> o/
14:01:49 <mriedem> a flurry of activity in -nova
14:01:58 <auggy> O/
14:01:59 <mriedem> let's get started
14:02:03 <mriedem> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova
14:02:13 <mriedem> #topic release news
14:02:22 <mriedem> #link Newton release schedule: https://wiki.openstack.org/wiki/Nova/Newton_Release_Schedule
14:02:27 <mriedem> #info June 30: non-priority feature freeze (end of day today)
14:02:28 * alex_xu_ waves late
14:02:29 <andreykurilin> \o
14:02:35 <lxsli> o/
14:02:55 <PaulMurray> o/
14:02:59 <mriedem> I've already started deferring blueprints that were approved but never got started, or are blocked and won't make it.
14:03:18 <mriedem> i think that dropped us from 100 to ~90 approved blueprints :)
14:03:36 <mriedem> We will probably have a small number of exceptions for things that are  mostly merged, very close and already have core reviewer attention.  Everything else will be deferred to Ocata.
14:03:43 <johnthetubaguy> I just re-ran my old blueprint script, incase this helps: http://5885fef486164bb8596d-41634d3e64ee11f37e8658ed1b4d12ec.r44.cf3.rackcdn.com/release_status.html
14:03:51 <mnestratov> o/
14:04:40 <mriedem> johnthetubaguy: ok, i've been just doing it the old fashioned way on LP
14:04:45 <mriedem> with ink and candel
14:04:48 <mriedem> *candle
14:05:10 <mriedem> so we have 26 computed blueprints
14:05:17 <johnthetubaguy> sorry, I forgot that thing existed
14:05:19 <mriedem> soon to be a couple more completed this morning i think
14:05:26 <mriedem> *completed blueprints
14:05:45 <mriedem> so my informal plan was to really just see what gets done today
14:05:59 <mriedem> and when the dust settles, start talking about exceptions tomorrow among the core team
14:06:20 <mriedem> questions?
14:06:24 <dansmith> mriedem: that sounds good to me, I'm just goign to review all day
14:06:50 <mriedem> #topic bugs
14:06:50 <jaypipes> mriedem: I will do my best to follow orders from you and dansmith for reviewing.
14:06:51 <johnthetubaguy> +1 from me
14:06:55 <danpb> mriedem: the deadline is tonight, right ?
14:07:00 <mriedem> danpb: yeah
14:07:12 <mriedem> danpb: more or less EOD for dansmith in pacific time
14:07:18 <dansmith> woot
14:07:22 * dansmith holds the wand of doom
14:07:30 <danpb> i guess we should focus on non-priority BPs reviews really too
14:07:32 <mriedem> dansmith and melwitt i should say :)
14:07:42 <mriedem> danpb: that's all i've focused on really for the last 2 weeks
14:07:44 <mriedem> so yeah
14:07:48 <dansmith> same
14:08:07 <mriedem> if review is just starting on a thing today, then it's probably not going to make it
14:08:11 <mriedem> unless it's super trivial
14:08:34 <mriedem> there were some one-off hyper-v bp changes i think when i looked last night
14:08:35 <mriedem> anyway, bugs
14:08:48 <mriedem> https://bugs.launchpad.net/nova/+bug/1597770
14:08:48 <openstack> Launchpad bug 1597770 in Ironic "Ironic grenade gate fails: Failed to allocate the network(s)" [Critical,Confirmed]
14:08:52 <mriedem> ^ came up this morning
14:09:01 <mriedem> looks like a change that went in yesterday is breaking grenade for ironic
14:09:08 <mriedem> there are some debug patches up
14:09:14 <mriedem> to get a better idea of what's failing
14:09:20 <mriedem> i think i know what's failing, but not sure why
14:09:38 <dansmith> mriedem: nothing to do with my change I hope?
14:09:50 <mriedem> dansmith: that's the one
14:09:59 <mriedem> https://github.com/openstack/nova/commit/e2eb6659a426bae05a54b93ebf31ff3247125258
14:10:00 <dansmith> really?
14:10:10 <mriedem> yeah, vif.create fails in neutronv2 api
14:10:16 <mriedem> probably on a unique constraint failure with duplicate macs
14:10:23 <mriedem> i have no idea why though
14:10:27 <dansmith> hrm
14:10:33 <markus_z> The quobyte CI has also troubles: https://bugs.launchpad.net/nova/+bug/1597644
14:10:33 <openstack> Launchpad bug 1597644 in OpenStack Compute (nova) "Permissioned denied on console.log during instance startup" [Undecided,New]
14:10:36 <mriedem> we wouldn't have seen it since it's in the ironic grenade job, which we don't run
14:11:01 <dansmith> okay
14:11:28 <mriedem> markus_z: volume shelve test, sounds new, and sounds like something PaulMurray or paul-carlton or andrearosa might know about
14:11:35 <johnthetubaguy> yeah, I wonder if the ironic grenade job is worthing running instead of our existing ironic job, longer term?
14:12:28 <mriedem> johnthetubaguy: idk, don't have an answer for that right now
14:12:30 <mriedem> it's voting at least
14:12:34 <mriedem> the one we run isn't voting
14:12:41 <johnthetubaguy> hmm, true
14:13:15 <mriedem> there was also a regression / race in the gate yesterday
14:13:17 <mriedem> which we reverted
14:13:37 <mriedem> melwitt was poking at the fix on that one
14:13:54 <mriedem> markus_z: are there any other critical bugs?
14:14:05 <markus_z> nope, nothing else
14:14:10 <vladikr> Don't know if it's critical, but there is a bug that we've introduced with the config drive attach patch, I've suggested a fix: https://review.openstack.org/#/c/335676
14:14:21 <johnthetubaguy> mriedem: hang on, we have a global unique constraint on mac address? or am I reading that wrongly?
14:14:24 <markus_z> A lot of new reports in the last 24 hours
14:14:40 <mriedem> johnthetubaguy: your eyes do not deceive you
14:14:55 <johnthetubaguy> mriedem: so that has to be rubbish, different networks but same mac is totally fine
14:15:06 <dansmith> vladikr: not critical I think, but definitely one we should fix
14:15:14 <dansmith> johnthetubaguy: right, which is really strange to me
14:15:19 <lyarwood> https://bugs.launchpad.net/nova/+bug/1597669 also came up yesterday after vladikr's config drive change but it appears it's been like this for a while.
14:15:19 <openstack> Launchpad bug 1597669 in OpenStack Compute (nova) "[libvirt] disk.config.rescue is not used when rescuing an instance" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood)
14:15:22 <dansmith> johnthetubaguy: why we'd be failing there
14:15:42 <johnthetubaguy> dansmith: true, will have to wait for those logs
14:15:46 <danpb> vladikr: bug fixes aren't critical wrt freeze - we've plenty of time to get them in
14:16:02 <jaypipes> johnthetubaguy: confirmed. there is a uniq_virtual_interfaces0address0deleted index.
14:16:18 <dansmith> that's strange
14:16:26 <dansmith> maybe n-net had such a requirement
14:16:26 <dansmith> ?
14:16:33 <johnthetubaguy> yeah, address, network, deleted might be closer to allowed, even then...
14:16:41 <jaypipes> dansmith: not sure :(
14:16:52 <mriedem> yeah keep in mind we literally haven't used VIF objects for neutron until yesterday
14:16:52 <dansmith> but the constraint is not covering network right?
14:16:57 <mriedem> it's not
14:16:59 <jaypipes> dansmith: no
14:17:01 <mriedem> and we don't set network_id on the vif for neutron
14:17:08 <dansmith> mriedem: we do
14:17:08 <mriedem> b/c network_id is the nova-net network.id
14:17:13 <johnthetubaguy> dansmith: well, nova-network never really got used with multiple physical networks
14:17:18 <mriedem> dansmith: i'm pretty sure we don't
14:17:22 <dansmith> johnthetubaguy: I guess
14:17:25 <mriedem> b/c neutron's network id is a uuid
14:17:26 <mriedem> not an int
14:17:37 <dansmith> mriedem: okay I guess you're right
14:17:43 <mriedem> \o/
14:17:49 <jaypipes> heh
14:17:51 <dansmith> I thought I did
14:17:55 <dansmith> but I'm probably thinking about the tests
14:17:56 <mriedem> i remember looking
14:18:00 <dansmith> okay
14:18:02 <mriedem> anywho
14:18:04 <jaypipes> dansmith: network_id is a nullable int.
14:18:05 <mriedem> so, we have some bugs
14:18:11 <dansmith> jaypipes: yeah
14:18:16 <mriedem> we'll talk about those in -nova after the meeting
14:18:18 <johnthetubaguy> yeah, there be bugs to work on
14:18:55 <mriedem> #topic reminders
14:19:02 <mriedem> #link Newton review focus list: https://etherpad.openstack.org/p/newton-nova-priorities-tracking
14:19:21 <mriedem> as noted, today reviews should really focus on getting non-priority bp's in that are close already
14:19:41 <mriedem> #help https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty Volunteers for 1 week of bug skimming duty?
14:19:50 <mriedem> It would be good to get someone from the core team to own bug triage for a week (next week?). ^
14:20:05 <mriedem> if we do some FFEs next week, core team people doing bug triage might have to wait a week
14:20:32 <mriedem> but in general, more cores helping with bug triage would be appreciated
14:20:42 <mriedem> #topic Stable branch status: https://etherpad.openstack.org/p/stable-tracker
14:20:54 <mriedem> i think the stable gate is ok
14:21:00 <mriedem> i haven't heard of any explosions there lately
14:21:13 <mriedem> there is a cve fix to get out in liberty
14:21:25 <mriedem> #topic subteam highlights
14:21:33 <mriedem> there was no cells v2 meeting yesterday
14:21:42 <mriedem> edleafe: scheduler meeting highlights?
14:21:44 <alaski> yep, nothing to report
14:21:52 <edleafe> Still no agreement on the best way to handle interaction of external systems with scheduler
14:21:55 <edleafe> E.g., Watcher wanting to validate a potential live migration target before calling live mig
14:21:58 <edleafe> Agreed to discuss on the spec itself, and at the midcycle if still not resolved.
14:22:01 <edleafe> Also agreed to hold off on any other attempts to interact with the scheduler until we have a public API for that.
14:22:04 <edleafe> Separating qualitative parts of a request is generally agreed to be the right way, but not enough feedback yet on the best implementation.
14:22:07 <edleafe> Definitely a midcycle topic.
14:22:13 <edleafe> Yingxin's eventually consistent host state proposal has a lot more performance data: http://lists.openstack.org/pipermail/openstack
14:22:17 <edleafe> -dev/2016-June/098202.html
14:22:36 <edleafe> (can you tell I typed that ahead of time?)
14:22:45 <mriedem> yes
14:22:50 <mriedem> way to be prepared
14:22:55 <edleafe> for once
14:23:14 <mriedem> and jaypipes wants review on https://review.openstack.org/#/c/300177/
14:23:15 <patchbot> mriedem: patch 300177 - nova-specs - resource-providers: Populate allocation fields
14:24:18 <mriedem> edleafe: anything else?
14:24:26 <edleafe> nope
14:24:55 <mriedem> live migration meeting, PaulMurray?
14:24:55 <alaski> there's a question for jaypipes on that review, otherwise I'm +2
14:25:08 <PaulMurray> There was more talk on CI
14:25:15 <PaulMurray> live migration job is in check queue
14:25:23 <PaulMurray> some tests seem stable
14:25:41 <PaulMurray> still getting problems with the top two ive migration bugs in recheck
14:25:53 <PaulMurray> mriedem, did you figure out if they are still happening in the new job?
14:26:30 <mriedem> PaulMurray: from logstash the volume-backed abort one was the failure on master in the live migration job
14:26:49 <mriedem> this bug https://bugs.launchpad.net/nova/+bug/1524898
14:26:49 <openstack> Launchpad bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,Confirmed]
14:27:06 <mriedem> danpb: had a test patch up to run nova with latest libvirt/qemu i think
14:27:11 <mriedem> danpb: did that work?
14:27:38 <mriedem> PaulMurray: anything else?
14:27:38 <PaulMurray> I think I saw a failure on at least one of his patches
14:27:38 <danpb> mriedem: yes & no - the hack to run a qemu with extra debug worked
14:28:01 <danpb> mriedem: but i only replaced qemu on the xenial based nodes, and of course those are only showing 4 failures in 7 days, so i've not reproduced it yet
14:28:15 <danpb> mriedem: i'm getting builds of hacked qemu done for trusty too, where its easier to reproduce
14:28:35 <PaulMurray> mriedem, one more item
14:28:38 <mriedem> ok, i think the multinode full job runs on trusty still
14:28:44 <mriedem> the live migration job runs on xenial
14:29:00 <danpb> yeah,  multinode-full is on trusty and that's where its easily reproducable
14:29:19 <danpb> my hack only touched multinode-live-migration where its much rarer on xenial
14:29:40 <danpb> should have the hack for trusty ready within next day though
14:30:17 <mriedem> cool
14:30:19 <PaulMurray> mriedem, last thing
14:30:22 <PaulMurray> Cross neutron/nova live migration change coming that needs coordination:
14:30:22 <PaulMurray> https://blueprints.launchpad.net/nova/+spec/migration-use-target-vif
14:30:26 <mriedem> i've confirmed in logstash that we aren't seeing https://bugs.launchpad.net/nova/+bug/1539271 in the xenial job
14:30:26 <openstack> Launchpad bug 1539271 in OpenStack Compute (nova) "Libvirt live block migration migration stalls" [High,Confirmed]
14:30:59 <mriedem> PaulMurray: yeah there is a dev list thread on that now and it's going to be discussed at the midcycle on wednesday afternoon
14:31:07 <danpb> mriedem: good, i thought that would be fixed with newer builds
14:31:07 <mriedem> carl_baldwin and armax will be at the midcycle on wednesday
14:31:27 <PaulMurray> thought armax was a maybe
14:31:45 <armax> mriedem, PaulMurray I am finalizing plans
14:31:59 <armax> mriedem, PaulMurray hopefully I’ll be able to attend the 20th
14:32:16 <mriedem> carl_baldwin is going to need some neutron backup
14:32:37 <mriedem> anyway, there is also the dev list thread
14:32:53 <mriedem> ok, moving on
14:32:56 <mriedem> thanks paul
14:33:00 <mriedem> alex_xu_: api meeting highlights?
14:33:32 <mriedem> maybe alex isn't here,
14:33:40 <alex_xu_> i'mhere
14:34:02 <alex_xu_> we have good progress on proxy discovery, proxy api/extension deprecation.
14:34:19 <alex_xu_> The api-ref still needs help, welcome people to help on review and patch
14:34:53 <alex_xu_> and there is discussion on about sync update_at of instance_actions http://lists.openstack.org/pipermail/openstack-dev/2016-June/098299.html
14:35:29 <alex_xu_> that is all we have
14:35:47 <mriedem> yup, ok, thanks
14:35:54 <mriedem> i think sdague is around next week
14:36:03 <mriedem> and then off for like 2 weeks
14:36:07 <mriedem> but i'm not sure
14:36:18 <alex_xu_> yea, that is another import thing, we need sdague!
14:36:28 <mriedem> :)
14:36:36 <mriedem> moshele isn't here
14:36:40 <lbeliveau> moshele couldn't attend and asked me to fill in for him
14:36:44 <mriedem> ok
14:36:56 <lbeliveau> good progress on sriov
14:37:02 <lbeliveau> there are two patches ready for core reviews for fixing migration with sriov
14:37:07 <lbeliveau> https://review.openstack.org/#/c/242573
14:37:11 <diga> alex_xu_: I have setup ready. if you need help, I can work on fixing
14:37:12 <lbeliveau> https://review.openstack.org/#/c/328983
14:37:33 <diga> bugs/patch
14:37:54 <lbeliveau> also Mellanox is still working on multi-node CI
14:38:11 <mriedem> lbeliveau: will ci with resize tests on a single node test any of that?
14:38:18 <alex_xu_> diga: cool, thanks, we can talk that off the meeting
14:38:20 <mriedem> or we still need multinode sriov/pci ci?
14:38:29 <diga> alex_xu_: sure
14:39:01 <lbeliveau> this one should be tested by current CI https://review.openstack.org/#/c/242573
14:40:03 <lbeliveau> I'll validate and ping you later today
14:40:06 <mriedem> lbeliveau: with this test? tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_resize
14:40:10 <mriedem> http://13.69.151.247/73/242573/36/check-nova/Nova-ML2-Sriov/67c190c/testr_results.html.gz
14:40:14 <lbeliveau> yes
14:40:22 <mriedem> anyway, yeah, we can take after the meeting
14:40:43 <lbeliveau> that's it
14:41:04 <mriedem> thanks
14:41:18 <mriedem> gibi_: was there a notifications meeting
14:41:19 <mriedem> ?
14:41:22 <gibi_> mriedem: yes
14:41:46 <gibi_> so the scope of the transformation spec in newton is really close to merge
14:42:01 <gibi_> additional easy transformations are also up and some already has +2 so there is chance that some gets merged
14:42:24 <gibi_> first patch of json schema generation is merged in ovo
14:42:57 <gibi_> so we might get a chance to add a tool to nova to generate json schema for the notification payloads
14:42:58 <mriedem> gibi_: the goals for newton again are versioned notifications for instance.delete and instance.update and create?
14:43:13 <gibi_> mriedem: that is the remaining yes
14:43:15 <mriedem> i want to keep the target in mind for newton
14:43:16 <mriedem> ok
14:43:44 <gibi_> after the freeze we have some follow up patches to do
14:43:50 <gibi_> doc generation improvement
14:43:54 <gibi_> test improvements
14:44:21 <mriedem> ok
14:44:29 <mriedem> anything else?
14:44:36 <gibi_> swap_volume notification patches queued up after instance.update
14:44:49 <gibi_> that is a separate bp
14:44:56 <mriedem> yeah there were like 3 separate notification bps
14:45:02 <mriedem> hypervisors, swap volume and keypairs
14:45:11 <mriedem> at least one of those didn't have code up so i deferred it last night
14:45:19 <mriedem> anyway
14:45:21 <gibi_> hypervisor stuck a bit on discussion but has code
14:45:27 <gibi_> swap_volume looks good
14:45:34 <gibi_> keypair deferred I think
14:45:37 <gibi_> that is all
14:45:43 <johnthetubaguy> is swap_volume in a separate bp?
14:45:51 <takashin> Yes
14:45:54 <johnthetubaguy> ah, I see it now
14:46:01 <gibi_> johnthetubaguy: yes as that is a new notification not a transformation
14:46:10 <johnthetubaguy> gotcha
14:46:11 <mriedem> yeah those were all new notifications
14:46:18 <mriedem> dependent on the versioned notification stuf
14:46:29 <gibi_> yes
14:46:32 <mriedem> ok let's move on
14:46:36 <mriedem> #topic stuck reviews
14:46:44 <mriedem> there wasn't anything on the agenda
14:47:00 <mriedem> paging for instance actions is stuck on what we decide to do in http://lists.openstack.org/pipermail/openstack-dev/2016-June/098503.html
14:47:07 <mriedem> which alex_xu_ already brought up
14:47:21 <mriedem> #topic open discussion
14:47:27 <mriedem> #info If you RSVP'ed for the meetup but know you won't be coming, please let me know so I can get an accurate headcount.
14:47:41 <mriedem> anything else for open discussion?
14:48:25 <gjayavelu> mriedem: need some help on reviewing vmware sub-team patches https://etherpad.openstack.org/p/newton-nova-priorities-tracking
14:48:30 <diga> mriedem: as discussed last time, if we can setup live stream for remote joinees will be good
14:48:42 <johnthetubaguy> mriedem: we did have BPs up for the instance actions stuff, I missed that discussion the other day and on ML, will go read that
14:48:46 <diga> mriedem: mid-cycle meetup
14:49:16 <mriedem> diga: unless there is a specific thing that we need to do a hangout for, they usually don't work
14:49:23 <mriedem> there isn't a microphone for everyone
14:49:36 <mriedem> so watching me drink coffee with no sound isn't great
14:49:46 <edleafe> mriedem: I beg to differ
14:49:53 <mriedem> we will have a nova/cinder hangout on wednesday morning
14:50:14 <auggy> mriedem: that's what the moustache addons are for
14:51:06 <mriedem> ok, wrapping up
14:51:10 <mriedem> thanks everyone
14:51:12 <mriedem> #endmeeting