16:01:16 <adrian_otto> #startmeeting containers
16:01:17 <openstack> Meeting started Tue Mar  8 16:01:16 2016 UTC and is due to finish in 60 minutes.  The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:20 <openstack> The meeting name has been set to 'containers'
16:01:51 <adrian_otto> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2016-03-08_1600_UTC Our Agenda
16:01:55 <adrian_otto> #topic Roll Call
16:01:58 <adrian_otto> Adrian Otto
16:02:00 <coreyob> Corey O'Brien
16:02:01 <juggler> o/
16:02:03 <levi_b> Levi Blackstone
16:02:04 <dane_leblanc> o/
16:02:04 <hongbin> o/
16:02:04 <thomasem> Thomas Maddox
16:02:06 <rpothier> Rob Pothier
16:02:07 <jward> John Ward
16:02:08 <Tango> Ton Ngo
16:02:10 <sidx64> Siddharth Shanbhogue
16:02:17 <strigazi> o/ Spyros Trigazis
16:02:44 <muralia> murali allada
16:02:50 <adrian_otto> hello coreyob juggler levi_b dane_leblanc hongbin thomasem rpothier jward Tango sidx64 strigazi and muralia
16:02:58 <muralia> hi all
16:03:09 <sidx64> Hi everyone
16:03:19 <juggler> hello
16:03:21 <eghobo> o/
16:03:43 <fgimenez> hello everyone o/
16:03:45 <adrian_otto> #topic Announcements
16:04:01 <adrian_otto> 1) Feature Freeze and String Freeze
16:04:36 <adrian_otto> although we don't have a ton of features landing right now, we should set expectations for when we cut Mitaka
16:05:01 <adrian_otto> traditionally Feature freeze means bugs can be fixed, but new features can not be added without an approval from the PTL
16:05:18 <adrian_otto> the reason for this is to allow coordination across projects
16:05:59 <adrian_otto> does anyone have any objections to setting the feature freeze date to Monday, March 14th?
16:06:45 <adrian_otto> at that point we will fork the repo, and patches against trunk will need to be resubmitted against the new branch
16:07:02 <adrian_otto> in order to add that work to Mitaka
16:07:14 <adrian_otto> otherwise it will be part of the subsequent release
16:07:37 <adrian_otto> I'm not hearing any objections, so I'm going to record this as an #agreed
16:07:48 <adrian_otto> thoughts to share first?
16:08:50 <adrian_otto> #agreed Magnum Mitaka Feature Freeze shall commence upon the creation of the Mitaka code branch from master on March 14th.
16:09:00 <adrian_otto> now, similar topic for string freeze
16:09:20 <adrian_otto> this has to do with restricting what string literals can be changed.
16:09:31 <adrian_otto> the purpose for this is to allow for translations into alternate locales
16:09:56 <Tango> Does this include docs?
16:09:57 <adrian_otto> this will apply to our documentation, which is still incomplete, and set to Essential for Mitaka.
16:10:01 <Tango> ah ok
16:11:09 <adrian_otto> Hard string freeze is also scheduled to be on March 14th
16:11:22 <adrian_otto> any objections to that cutoff date?
16:11:57 <adrian_otto> any objections or discussion prior to recording this as an #agreed?
16:12:23 <thomasem> Sounds like agreement to me
16:12:27 <adrian_otto> #agreed Magnum's hard string freeze will commence on March 14th 2016
16:13:12 <adrian_otto> 3) Magnum will not be having a dedicated OpenStack Summit talk in the main track, as it had in two prior summits, however, several magnum related talks are scheduled in the Containers talk.
16:13:24 <adrian_otto> congratulations to those contributors who had talks selected
16:13:41 <adrian_otto> we will still have a bunch of spots in the Design Summit
16:14:07 <adrian_otto> s/Containers talk/Containers track/
16:14:44 <adrian_otto> any other announcements from team members?
16:16:37 <adrian_otto> ok, let's review ai's
16:16:43 <adrian_otto> #topic Review Action Items
16:17:05 <adrian_otto> 1) thomasem to update team on progress on libvirt-lxc work
16:17:09 <adrian_otto> status?
16:17:21 <thomasem> bug opened about one of the issues https://bugs.launchpad.net/nova/+bug/1552740
16:17:21 <openstack> Launchpad bug 1552740 in OpenStack Compute (nova) "Nova hard reboot fails to mount logical volume (LVM + libvirt-lxc)" [Medium,Confirmed] - Assigned to Thomas Maddox (thomas-maddox)
16:17:31 <thomasem> Had a few production issues that I had to divert my attention to after that.
16:17:36 <thomasem> So, didn't get to work on it any more.
16:18:18 <adrian_otto> ok
16:18:31 <adrian_otto> is that the only known defect?
16:18:45 <thomasem> That and this: https://review.openstack.org/#/c/240611/1
16:18:52 <thomasem> Sorry, let me get the bug associated
16:19:02 <thomasem> https://bugs.launchpad.net/nova/+bug/1191960
16:19:02 <openstack> Launchpad bug 1191960 in OpenStack Compute (nova) "force-delete of cinder volume errors with Can\'t remove open logical volume " [Medium,In progress] - Assigned to Thomas Maddox (thomas-maddox)
16:19:22 <thomasem> lvremove sometimes also fails
16:19:26 <thomasem> with device being busy
16:19:50 <adrian_otto> because there are processes holding references to inodes on that volume?
16:19:54 <thomasem> I'm wondering if there's a Libvirt bug or two underneath all of this.
16:20:07 <thomasem> It seems like it, however fuser/lsof don't really give me any indication as to what process.
16:20:35 <thomasem> The fd just seemed to eventually die off over time
16:20:40 <adrian_otto> do you find entries in lsof that relate to that volume at all?
16:20:44 <thomasem> nope
16:20:58 <thomasem> It's like something orphaned in another namespace
16:20:59 <thomasem> or something else
16:21:00 <adrian_otto> oh, so probably something lower level than filesystem access
16:21:06 <thomasem> yeah, that's what I'm worried about
16:21:17 <thomasem> It is a fairly old kernel now, fwiw.
16:21:34 <thomasem> I haven't had this issue at all on 3.18.21
16:22:06 <adrian_otto> ok
16:22:16 <thomasem> Wondering if moving to another kernel in gate (new Ubuntu, or some other distro) might magically make these go away.
16:22:22 <adrian_otto> is there anything that other Magnum contributors may be able to assist you with, thomasem
16:22:37 <thomasem> Not unless one wants to dig in on either of these
16:22:48 <adrian_otto> thomasem: trying a different kernel in the gate sounds worth it to me
16:22:48 <thomasem> But they are Nova problems, specifically, with Libvirt/LXC
16:23:01 <thomasem> Yeah, I've seen far better behavior on upstream 3.x kernels
16:23:04 <thomasem> than the Ubuntu ones
16:23:13 <thomasem> Even at the same version
16:23:27 <adrian_otto> thanks for the update thomasem.
16:23:31 <thomasem> Sure thing
16:23:35 <adrian_otto> that was our only action item from last week
16:23:44 <adrian_otto> #topic Blueprint Review
16:23:52 <adrian_otto> Essential Blueprint Review
16:24:03 <adrian_otto> #link https://blueprints.launchpad.net/magnum/mitaka Mitaka Blueprints
16:24:28 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/magnum-troubleshooting-guide Troubleshooting Guide
16:24:45 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/user-guide User Guide
16:24:47 <Tango> We got the patch on etcd troubleshooting merged, thanks everyone for the review.
16:25:02 <Tango> There is a new patch on image management for review
16:25:07 <adrian_otto> great!
16:25:18 <Tango> wangqun has volunteered to cover the section on storage in the user guide
16:25:39 <Tango> She has been implementing most of that feature
16:26:05 <Tango> That's all from me
16:26:23 <adrian_otto> thanks Tango
16:27:05 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/resource-quota Resource Qhota
16:27:26 <adrian_otto> vilobh11 had been working on this
16:27:41 <adrian_otto> it might make sense to push this into the next release cycle
16:28:05 <adrian_otto> thoughts on this?
16:29:09 <adrian_otto> ok, I removed the targeting from Mitaka
16:29:37 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/magnum-installation-guide Magnum Install Guide
16:30:04 <adrian_otto> Spyros was working on this one
16:30:05 <strigazi> I'm working on it, and I pushed a patch-set https://review.openstack.org/#/c/288580
16:30:35 <strigazi> I also created a blueprint under openstack-manuals
16:30:48 <adrian_otto> thanks for your work on this, strigazi
16:31:09 <strigazi> this also needs review https://review.openstack.org/#/c/289994/
16:31:51 <adrian_otto> so these are in two separate repos
16:32:05 <strigazi> yes
16:32:09 <adrian_otto> which means that we have two places to edit when we want to adjust this
16:32:27 <adrian_otto> might it make sense to make the one within the magnum repo a pointer back to the other?
16:32:40 <adrian_otto> for DRY
16:33:11 <strigazi> I'm not sure, but to have something in openstack-manuals it must be in docs-specs too
16:33:16 <adrian_otto> what have other projects done?
16:33:34 <strigazi> there is something similar for manila
16:34:32 <strigazi> eg https://blueprints.launchpad.net/openstack-manuals/+spec/create-manila-install-guide
16:35:03 <strigazi> the guide's specs is in docs-specs
16:35:19 <adrian_otto> ok, so reviewers will need to take note that if future patches surface against the install docs, that the copy over in openstack-manuals will also need equivalent updates.
16:35:33 <adrian_otto> are our reviewers okay with that?
16:36:02 <Tango> We can put a note in the BP
16:36:15 <adrian_otto> but that will get closed out upon release
16:36:32 <adrian_otto> I'm thinking forward to when today is a distant memory
16:36:44 <Tango> Comment in the doc?
16:36:48 <vilobhmm11> how about a note is added in commit message will that help
16:36:59 <adrian_otto> to avoid drift between those two references… probably a comment in the doc.
16:37:26 <strigazi> ok, I'll check what they did in manila as well
16:37:36 <adrian_otto> and maybe a tech-debt bug to remove our copy and replace it with a link to openstack-manuals upon mutually agreed maturity of the text (slow rate of change)
16:38:22 <adrian_otto> thanks strigazi
16:39:08 <strigazi> you're wellcome
16:39:23 <adrian_otto> ok, that concludes the essential blueprints
16:39:34 <adrian_otto> Blueprints, Bugs, Specs, and other work items to be discussed as a team
16:39:47 <adrian_otto> last week we touched on 1) swarm related heat templates (for storage support) https://review.openstack.org/#/c/275034/
16:40:00 <adrian_otto> there is news here
16:40:17 <adrian_otto> docker 1.10.x has a patch that eliminates the bug requiring selinux to be disabled
16:40:39 <adrian_otto> so this patch can be reworked to use that fixed code and I will be able to lift my -2 vote
16:41:02 <adrian_otto> special thanks to Dan Walsh from Redhat for helping us
16:41:31 <adrian_otto> any other comments on this one?
16:41:45 <coreyob> fyi, I think se_linux got disable agend
16:41:47 <coreyob> again
16:41:48 <coreyob> https://review.openstack.org/#/c/289626/
16:42:02 <vilobhmm11> yup thats right
16:42:04 <juggler> should I keep pressing forward with my efforts, or is that a different topic?
16:42:25 <adrian_otto> juggler: I will regroup with you after
16:42:32 <juggler> got it
16:42:36 <eghobo> it looks like most Kub deployments prefer to stay with 1.8.2
16:43:27 <hongbin> It looks the k8s bay is not functioning if selinux is enabled. The root cause is unknown
16:43:42 <Tango> eghobo: Is it because of stability in later release?
16:43:49 <adrian_otto> which version of docker is in that non-functioning config?
16:44:12 <hongbin> Not sure exactly. It is from the *-5 image
16:44:52 <Tango> we have docker 1.8.1
16:44:56 <eghobo> no, networking. docker is running dns server inside now ;). in theory everything it should work, but i never tried
16:45:30 <adrian_otto> ok, lets form a tiger team to burn this problem down.
16:45:43 <adrian_otto> I volunteer to help
16:45:55 <adrian_otto> juggler: do you have time to work on it?
16:45:58 <hongbin> For reference, here is the bug
16:46:02 <hongbin> #link https://launchpad.net/bugs/1551648
16:46:03 <openstack> Launchpad bug 1551648 in Magnum "k8s podmaster doesn't work" [Critical,Fix released] - Assigned to hongbin (hongbin034)
16:46:05 <adrian_otto> thanks hongbin
16:46:52 <adrian_otto> so hongbin, let's start up an ML thread explaining the problem
16:46:59 <hongbin> sure
16:47:04 <adrian_otto> and do some initial troublehsooting together
16:47:14 <juggler> aotto: we can chat about it during the regroup
16:47:50 <adrian_otto> we can pull in help as needed. I asked juggler to investigate ways to avoid disabling selinux by policy building
16:47:54 <adrian_otto> thanks juggler
16:48:33 <adrian_otto> the easiest way to troubleshoot this is to put the host into permissive mode, and look at the /var/log/audit.log file
16:48:43 <adrian_otto> and find out what the action policy violations are
16:49:14 <adrian_otto> 3) thomasem to get Libvirt/LXC Nova gate test to voting
16:49:21 <adrian_otto> you updated us earlier on the work in nova
16:49:27 <adrian_otto> any work to report within Magnum?
16:49:59 <thomasem> I believe coreyob tried to get Magnum func tests working with Libvirt/LXC and had significant image issues
16:50:35 <adrian_otto> coreyob: what's happening there?
16:50:47 <thomasem> He could give you more details, but in order to accomplish that we'll need to roll a custom image atm.
16:50:49 <coreyob> yeah I looked into a bit and stopped at the point of having to convert the atomic images to something that woudl work with the lxc driver
16:50:52 <thomasem> Not sure if that's desired.
16:51:20 <coreyob> the atomic images are multi-partition with lvm and not ext4 which are in conflict with the lxc driver's expectations
16:51:31 <adrian_otto> I see
16:51:46 <coreyob> pretty sure we can work around that, but I didn't actually do the work to prove that
16:51:48 <adrian_otto> is this somethign you have tabled, or are you planning to keep on with it?
16:52:13 <coreyob> tabling it for now until the team is confident enough with the lxc driver to move forward with that idea
16:52:52 <adrian_otto> I don't remember any other viable options to get our functional gate tests to execute quickly enough
16:53:13 <adrian_otto> the only other alternative I can think of is to use the 3rd party CI facility to to the func tests on bare metal hosts
16:53:16 <coreyob> unless infra gives us bigger VMs :)
16:53:35 <adrian_otto> it's not so much the size but the speed of them… particularly the IO performance
16:53:51 <coreyob> last time we discussed it there were some hesitations about using the libvirt/lxc driver from nova because it wasn't as maintained
16:53:57 <adrian_otto> we are short on time, so I will take this to the ML as well, and pull in openstack-infra
16:54:02 <coreyob> sounds good
16:54:36 <adrian_otto> #action adrian_otto to start an ML thread about Magnum functional gate tests taking forever to execute in OpenStack CI.
16:55:09 <adrian_otto> we have a topic on the agenda: Bay node OS support in Magnum, but I am reluctant to raise this for discussion because we don't have time today to cover it adequately
16:55:24 <adrian_otto> so I am planning to table that until next meeting, and move it higher into the agenda
16:55:36 <thomasem> Sounds good
16:55:44 <adrian_otto> #topic Open Discussion
16:55:49 <sidx64_> Hey all, I'm working on documenting my attempts at configuring and deploying magnum on RDO with mesos and marathon. Can I help with this in regards to the magnum install guide? Also, wondering if I could reach someone who has worked on this set up (magnum with mesos marathon)
16:55:53 <juggler> x2 with thomasem
16:56:08 <adrian_otto> sidx64_: yes, we welcome your participation
16:57:25 <adrian_otto> sidx64_: I know hongbin has a ton on his plate, but he may be able to offer a pointer or two there.
16:57:40 <hongbin> sidx64_: Yes, feel free to ping me in the IRC
16:57:55 <adrian_otto> in #openstack-containers
16:58:02 <hongbin> yes
16:58:16 <sidx64_> Thank you adrian_otto and Hong bin. Thanks a ton.
16:58:22 <adrian_otto> and let's do our best to capture the insight that sidx64_ learns, and codify that into our docs
16:58:41 <adrian_otto> sidx64_: we would really appreciate your help in making it easier for those who come after you
16:59:21 <adrian_otto> ok, we are approaching the end of our scheduled meeting time
16:59:30 <sidx64_> I'll share my documentation with you as soon as I have it up. I have shared a rough draft for setting up magnum with mvelten once.but that needs extensive changes now.
16:59:32 <sidx64_> Thank you
16:59:54 <adrian_otto> our next team meeting will be 2016-03-15 at 1600 UTC here in #openstack-meeting-alt
17:00:01 <adrian_otto> see you all then!
17:00:05 <adrian_otto> thanks for attending
17:00:10 <adrian_otto> #endmeeting