21:00:12 <mriedem> #startmeeting nova
21:00:12 <openstack> Meeting started Thu Jul  7 21:00:12 2016 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:15 <openstack> The meeting name has been set to 'nova'
21:00:22 <takashin> o/
21:00:27 <dansmith> o/
21:00:29 <edleafe> \o
21:00:48 <cdent> /o\
21:00:50 <woodster_> o/
21:01:15 <alaski> o/
21:01:23 * Vek waves
21:01:48 <jroll> \o
21:01:56 * bauzas celebrates
21:02:02 <mriedem> ok let's get started
21:02:08 <diana_clarke> o/
21:02:10 <mriedem> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
21:02:20 <mriedem> #topic release news
21:02:27 <mriedem> #link Newton release schedule: https://wiki.openstack.org/wiki/Nova/Newton_Release_Schedule
21:02:47 <mriedem> we're past the non-priority feature freeze
21:02:57 <mriedem> there were some FFEs http://lists.openstack.org/pipermail/openstack-dev/2016-July/098896.html
21:03:10 <mriedem> the core team should be focusing on the list in ^ this week
21:03:21 <mriedem> since the deadline on those is next wednesday 7/13
21:03:24 <bauzas> ack
21:03:47 <mriedem> Friday 7/15 is n-2
21:04:16 <mriedem> and then we have the midcycle the week after that
21:04:42 <mriedem> if you haven't booked a hotel yet for the midcycle, fyi the price on the holiday inn has gone up
21:04:48 <mriedem> i updated the wiki page for the midcycle
21:05:07 <mriedem> any questions on release stuff?
21:05:12 <dansmith> and I charge by the sqft for sleeping bags
21:05:22 <Vek> haha :)
21:05:23 <dansmith> or sqm for europeans
21:05:41 <mriedem> #topic bugs
21:05:52 <mriedem> we have some kind of regression
21:06:00 <mriedem> #help gate-tempest-dsvm-postgres-full has been failing with SSH timeouts since 7/4 or 7/5
21:06:10 <mriedem> Swanson was going to open a bug
21:06:12 <mriedem> The postgres job runs with nova-network but gate-tempest-dsvm-full isn't  failing at the same rate. The postgres job also runs with the  n-api-meta service, which might be contributing somehow to the failures,  but it's not obvious.
21:06:30 <mriedem> i compared the failure rates between the gp job and gate-tempest-dsvm-full and they are different,
21:06:45 <mriedem> and logstash shows the pg job being by far the worst with these ssh timeout failures since ~7/4
21:06:58 <mriedem> i checked nova/devstack/tempest commits going back to 7/3 and nothing is jumping out at me
21:07:10 <mriedem> not sure it would be node provider either
21:07:37 <anteaya> are all the failures on a single cloud provider?
21:07:39 <mriedem> anyway, if you want to land code, help out
21:08:06 <mriedem> http://logstash.openstack.org/#dashboard/file/logstash.json?query=%28message%3A\%22TestVolumeBootPattern%3Atest_volume_boot_pattern\%22%20OR\n%20message%3A\%22TestVolumeBootPatternV2%3Atest_volume_boot_pattern\%22%29%20AND\nmessage%3A\%22Initializing%20SSH%20connection%20to\%22%20AND%20message%3A\%22via%20SSH%20timed%20out.\%22%20AND%20tags%3A\%22tempest.txt\%22%20AND%20voting%3A1&from=7d
21:08:24 <mriedem> anteaya: no
21:08:45 <anteaya> thanks
21:09:14 <mriedem> as noted the pg job's main wrinkle is it runs the metadata api
21:09:22 <mriedem> but i don't really know why that would be screwing something up here
21:09:27 <mtreinish> mriedem: we've had issues with the metadata service in the past
21:09:35 <dansmith> well, instances won't get IPs
21:09:40 <dansmith> that's one reason it would affect them
21:09:53 <mtreinish> which is what prompted us to default using config drive everywhere else
21:10:11 <mriedem> ok that's something i'm not familiar with
21:10:16 <mriedem> obviously
21:10:36 <jroll> dansmith: or ssh keys
21:10:42 <mtreinish> dansmith: that or the ssh keys
21:10:44 <dansmith> jroll: yeah this is timeout I think
21:10:48 <jroll> I guess IP is the main thing with hanging
21:10:50 <mtreinish> jroll: you win :)
21:10:50 <jroll> yeah
21:10:55 <jroll> heh
21:11:04 * mtreinish blames airplane wifi
21:11:38 <mriedem> that's the only critical bug i'm aware of
21:11:44 <anteaya> Swanson: has offered #link https://bugs.launchpad.net/nova/+bug/1600005
21:11:44 <openstack> Launchpad bug 1600005 in OpenStack Compute (nova) "gate-tempest-dsvm-postgres-full failing on node ovh " [Undecided,New]
21:12:07 <anteaya> and feels he could use some constructive feedback on filing more bug reports in future
21:12:10 <anteaya> thanks Swanson
21:12:32 <mriedem> #topic reminderes
21:12:39 <mriedem> yikes typo
21:12:46 <mriedem> #link Newton review focus list: https://etherpad.openstack.org/p/newton-nova-priorities-tracking
21:13:01 <mriedem> but really it's FFEs this week
21:13:11 <mriedem> #help https://wiki.openstack.org/wiki/Nova/BugTriage#Weekly_bug_skimming_duty Volunteers for 1 week of bug skimming duty?
21:13:31 <mriedem> we have 36 untriaged bugs
21:14:00 <mriedem> #topic stable branch status
21:14:13 <mriedem> not much news here, still working on sorting out a cve to release in liberty
21:14:28 <mriedem> lots of mitaka reviews https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/mitaka,n,z
21:14:43 <mriedem> we'll probably do a mitaka review push after the midcycle
21:14:47 <mriedem> and a release
21:15:08 <mriedem> #topic subteam highlights
21:15:13 <mriedem> alaski: cells v2 meeting?
21:15:24 <alaski> not much to report this week
21:15:31 <alaski> most of us were focused on FF
21:15:40 <alaski> hopefully things will spin up a bit more now
21:15:49 <alaski> reviews welcome
21:15:52 <alaski> that's all
21:16:10 <mriedem> thanks
21:16:14 <mriedem> edleafe: scheduler meeting?
21:16:18 * edleafe copies, and then...
21:16:23 <edleafe> No meeting this week b/c of US holiday
21:16:23 <edleafe> A couple of points of focus:
21:16:23 <edleafe> Placement API: https://review.openstack.org/329149
21:16:23 <edleafe> RP population: https://review.openstack.org/300177
21:16:23 <edleafe> Updates to remove 'nodename' from RT: series beginning with https://review.openstack.org/#/c/334031/
21:16:31 <edleafe> that's it
21:17:33 <mriedem> https://blueprints.launchpad.net/openstack/?searchtext=ironic-multiple-compute-part-deuce ?
21:18:06 <bauzas> I don't think we have a blueprint for
21:18:16 <mriedem> the rest of the series is for generic-resource-pools
21:18:40 <bauzas> well, not really I think
21:18:48 <bauzas> not really related*
21:18:59 <mriedem> well, that's what the commit messages and topic say
21:19:05 <mriedem> but jaybird isn't here
21:19:09 <dansmith> yeah, I don't think it is required
21:19:15 <mriedem> isn't there some spec amendment for this?
21:19:18 <dansmith> I think he'd like to get those in, but not required for g-r-p
21:19:21 <dansmith> that I know of
21:19:31 <mriedem> hmm
21:19:34 <jroll> what's up?
21:19:38 <mriedem> then it sounds like a distraction
21:20:01 <mriedem> #action mriedem to talk to jaypipes about https://review.openstack.org/#/c/334031/
21:20:15 <mriedem> PaulMurray isn't here for live migration
21:20:25 <mriedem> mdbooth: diana_clarke: was there a meeting this week?
21:20:54 <diana_clarke> Yup, I can dig up some logs.
21:21:18 <mriedem> diana_clarke: i'll come back
21:21:21 <mriedem> alex_xu: are you around?
21:21:29 <jroll> mriedem: late, but what you were looking for is https://blueprints.launchpad.net/nova/+spec/ironic-multiple-compute-hosts
21:21:52 <mriedem> jroll: yeah i know about that, but that's not the bp link in jay's bottom change
21:22:10 <bauzas> is that something we agreed as a prio ?
21:22:16 <mriedem> g-r-p is the means to the ironic multiple compute end
21:22:19 <mriedem> bauzas: it's not
21:22:22 <jroll> mriedem: right, ok, just making sure
21:22:39 <mriedem> ok so no api people around
21:22:50 <mriedem> i don't think there was an sriov meeting this week
21:22:56 <mriedem> gibi_: was there a notifications meeting?
21:23:01 * mriedem guesses not
21:23:03 <edleafe> mriedem: the meeting was just me and johnthetubaguy
21:23:09 <mriedem> edleafe: which one?
21:23:17 <edleafe> sorry, nova api
21:23:25 <mriedem> versioned notification transformation is basically frozen for newton
21:23:35 <mriedem> there are some approved patches going through rebases, but that's it
21:23:42 <mriedem> i created a bp for ocata for that work though
21:23:56 <mriedem> but would like to see the changes actually stacked up because they merge conflict on every approved change
21:23:59 <mriedem> which is super annoying
21:24:24 <mriedem> edleafe: did you and johnthetubaguy resolve all api issues?
21:24:40 <edleafe> mriedem: yes, and then had a party to celebrate
21:24:48 <mriedem> good
21:25:00 <mriedem> diana_clarke: anything for live migration?
21:25:07 <diana_clarke> Here are the logs from the live migration meeting. http://eavesdrop.openstack.org/meetings/nova_live_migration/2016/nova_live_migration.2016-07-05-14.01.log.html
21:25:24 <diana_clarke> Otherwise, I don't have much to add since I haven't really been working on the image backend stuff recently.
21:25:32 <mriedem> http://eavesdrop.openstack.org/meetings/nova_live_migration/2016/nova_live_migration.2016-07-05-14.01.html no action items
21:25:36 <mriedem> so ok
21:25:48 <mriedem> #topic stuck reviews
21:25:55 <mriedem> there was nothing on the agenda
21:26:07 <mriedem> #topic open discussion
21:26:14 <mriedem> #info: no meeting on July 21st due to the midcycle
21:26:23 <mriedem> #info: dims is proposing to retire nova-docker: http://lists.openstack.org/pipermail/openstack-dev/2016-July/098940.html
21:26:28 <dansmith> +1000
21:26:33 <dims> yay
21:26:53 <mriedem> last thing
21:26:58 <mriedem> Fail build request if we can't inject files? http://lists.openstack.org/pipermail/openstack-dev/2016-July/098703.html
21:27:09 <mriedem> i'm looking for some input on how to handle this
21:27:19 <mriedem> it turns out, w/o host capabilities, our api kind of sucks :)
21:27:24 <mriedem> i'm not sure if anyone realized that
21:27:53 <mriedem> basically there is a neutron ssh job that is non-voting but it's the only job that will test the 2.32 microversion that does device tagging
21:27:58 <mriedem> so we want that job running on nova changes
21:28:26 <mriedem> i debugged why it was failing, and it was because it was running a file injection test and then ssh'ing into the guest to assert the files existed, which they didn't b/c we disable file injection by default
21:28:34 <mriedem> the test doesn't check config drive or the metadata service
21:28:53 <mriedem> it doesn't create the instance with config drive, and the metadata service is optional
21:29:04 <mriedem> and the code today just silently ignores if you requested file injection but we don't do it
21:29:34 <mriedem> so we can't really fail the build if you're using config drive or the metadata api, but i can't check from the virt driver if the metadata service is running
21:29:58 <mriedem> so....we're kind of left with a debug log message and possibly a note in the api-ref for the personality extension
21:30:03 <mriedem> unless someone else has ideas
21:30:31 <alaski> have we deprecated that extension? I know it was discussed a while back
21:30:41 <mriedem> alaski: it's now folded into the servers extension
21:30:48 <mriedem> so you can't disable it via policy either
21:30:51 <mriedem> anymore anyway
21:31:04 <alaski> okay, but we're still saying that file injection is a thing people should use?
21:31:35 * dansmith has an idea
21:31:47 <mriedem> we do'nt have anything in the api-ref saying you shouldn't use it
21:32:07 <mriedem> yeah so we could just delete it
21:32:45 <mriedem> which is a backward incompatible api change
21:32:49 <alaski> yeah
21:32:56 <mriedem> we could microversion it out, like the api proxies
21:33:13 <mriedem> but...that doesn't really work either
21:33:20 <mriedem> because you can inject the files via md or config drive
21:33:22 <alaski> so on the one hand I feel like it should cause a build failure, anything else you request during an instance boot causes a failure if it doesn't work
21:33:43 <alaski> but on the other hand making a guarantee about fiddling with the guest is hard
21:33:57 <alaski> so it feels like an exception to that
21:34:26 <mriedem> which is why i think i'm basically stuck with debug log that we're ignoring the files in the virt driver, and warning in the api-ref
21:34:42 <mriedem> that tihs is entirely dependent on the compute config and may not work, and may not fail
21:34:46 <mriedem> so use md or config drive
21:35:10 <alaski> yeah, we really want to funnel people towards config-drive
21:35:25 <mriedem> btw, file injection has been disabled by default for libvirt since icehouse
21:35:26 <mriedem> :)
21:35:52 <mriedem> anyway, if you have other thoughts, please speak up in the thread,
21:36:03 <mriedem> when i get a chance i'll probably update my nova change to just do the logging/warning docs stuff
21:36:18 <mriedem> i do have a change to the neutron job and devstack to actually make those tests work properly
21:36:30 <mriedem> and i got a patch to disable file injection testing in tempest by default
21:36:33 <mriedem> since nova disables it by default
21:36:36 <mriedem> that was the easy fix
21:36:47 <mriedem> alright, any other open discussion?
21:36:59 <dansmith> move to adjourn
21:37:03 <mriedem> denied!
21:37:06 <mriedem> second?
21:37:12 <alaski> seconded
21:37:15 <mriedem> hear hear
21:37:22 <mriedem> #endmeeting