16:01:08 <inc0> #startmeeting kolla
16:01:09 <openstack> Meeting started Wed Oct 12 16:01:08 2016 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:13 <openstack> The meeting name has been set to 'kolla'
16:01:14 <akwasnie> o/
16:01:15 <pbourke> woot
16:01:19 <Jeffrey4l_> 0/
16:01:20 <egonzalez90> o/
16:01:25 <duonghq> o/
16:01:25 <qwang> o/
16:01:27 <vhosakot> o/
16:01:28 <berendt> o/
16:01:29 <jascott1> o/
16:01:30 <hrito> o/
16:01:31 <coolsvap> o/
16:01:33 <duonghq> no roll call topic?
16:01:34 <inc0> hold on
16:01:40 <inc0> #topic rollcall - woot for kolla
16:01:42 <sdake> o/
16:01:44 <rhallisey> lul
16:01:49 <Jeffrey4l_> \o/
16:01:54 <inc0> woot guys plz;)
16:01:55 <egonzalez90> woot
16:01:57 <duonghq> woot
16:01:58 <berendt> woot
16:02:04 <sdake> i do not consent ;)
16:02:13 <vhosakot> \o/00T
16:02:14 <pbourke> woot again
16:02:16 <akwasnie> woot
16:02:19 <Jeffrey4l_> woot
16:02:21 <rhallisey> vhosakot, nice
16:02:26 <jascott1> woot
16:02:27 <vhosakot> ;)
16:02:37 <zhubingbing> \o/
16:02:47 <inc0> ok, cores please -2 sdakes changes until he submits to our internal rituals
16:02:58 <sdake> your pain not mine :)
16:03:08 <inc0> (kidding ofc, we'll deal with it phusically in summit if he wont)
16:03:18 <inc0> #topic announcements
16:03:21 <sdake> buy me lots o beers
16:03:21 <sdake> i'm broke ;)
16:03:42 <inc0> 1. RC2 is tagging today, we need to open stable/newton branch
16:03:58 <inc0> we will have rc3 due to critical bugs outstanding
16:04:08 <inc0> it's going to be Oct18
16:04:23 <inc0> 2. remind - summit schedule is up for review
16:04:41 <inc0> any announcements from community>
16:05:18 <inc0> guess not
16:05:25 <inc0> #topic newton release
16:05:27 <inc0> sdake, you're up
16:05:47 <sdake> https://launchpad.net/kolla/+milestone/newton-rc2
16:06:00 <sdake> the summary there is there are 7 critical bugs that make kolla unusable
16:06:13 <sdake> we have a slew of high bugs which make a particular service have some kind of defect
16:06:25 <wirehead_> o/
16:06:32 <sdake> on the plus side, we have fixed 66(!) bugs in rc2
16:06:44 <sdake> we also have a slew of bugs in the "INPROGRESS STATE"
16:06:55 <sdake> if there is a bug in INPROGRESS that can be merged, lets get it merged
16:07:11 <sdake> after we tag rc2, only critical bugs will be backported to stable/newton
16:07:21 <sdake> a few mechanics on the tagging/branching
16:07:41 <sdake> rc2 tags today 23:45 UTC -> the release team tags when they get to it
16:07:51 <sdake> the branch gets created when the release team gets to it
16:07:51 <sdake> they are not slow
16:07:59 <sdake> but don't expect a branch immediately when the tag occurs
16:08:13 <sdake> for rc3, we are carrying 7 bugs
16:08:21 <sdake> these need critical attention
16:08:27 <sdake> there are some other bugs in that list that are in triaged
16:08:32 <sdake> that are critical
16:08:56 <sdake> if someone could get them out of the triaged state, that would be fantastic (by confrming, marking invalid, etc)
16:09:10 <sdake> i will be carrying over triaged bugs that are critical as well as confirmed bugs
16:09:39 <sdake> if they are marked critical - I assume whoever triaged it understands the bug impacts all of kolla
16:09:51 <sdake> now, what to do if you have a high severity bug that you think should be critical?
16:09:54 <sdake> change it to critical
16:10:02 <sdake> (before 23:45)
16:10:21 <sdake> the workflow seems to be going well from last meeting
16:10:42 <sdake> it would be great if we could finish the job  on those inprogress bugs rather then having to backport them, so i'd suggest fixing those today :)
16:10:46 <sdake> or reviewing them
16:10:51 <sdake> etc
16:10:58 <sdake> if its critical, just assume it will be fixed starting tomorrow
16:11:04 <sdake> I need 1 more week out of everyone :)
16:11:12 <sdake> and then we have free time for some time :)
16:11:23 <sdake> (tag is tuesday for rc3)
16:11:32 <inc0> 3 days, then summit:)
16:11:46 <sdake> shhhh :)
16:12:03 <sdake> after summit - you will have plenty o time to recover :)
16:12:40 <sdake> inc0 you wanted a status update on each bug?
16:12:52 <sdake> rather each critical
16:12:53 <inc0> yes, let's do this, for critical at least
16:13:04 <sdake> cool take it away - have to do something rq like :)
16:13:08 <inc0> #topic critical bugs review
16:13:26 <inc0> let's spend few moments to discuss each critical bug
16:13:37 <inc0> to help fixing it
16:13:53 <inc0> 1 https://bugs.launchpad.net/kolla/+bug/1564773
16:13:53 <openstack> Launchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]
16:14:55 <inc0> hmm, anyone have steps to reproduce this one?
16:15:34 <Jeffrey4l_> i followed this bug. but got nothing about it. :(
16:15:58 <inc0> dug
16:15:59 <inc0> duh
16:16:02 <zhubingbing> Bug report time is too long.
16:16:13 <inc0> yes, but comments are fres
16:16:14 <inc0> h
16:16:20 <inc0> so it's still a thing as it seems
16:16:25 <Jeffrey4l_> yes. the error message is different too.
16:16:34 <duonghq> tomorrow I'll try to triaged this bug, today the epel repo is broken for awhile
16:16:37 <berendt> sean-k-mooney noted it happens on maste, can he support?
16:16:42 <sdake> pbourke conirmed it
16:16:47 <sdake> i changed it to critical as a result
16:17:43 <sdake> we need to circle back with pbourke on this issue and see what his defintiion of confirmed is
16:17:51 <sdake> i'll do tha ttomororw morning
16:18:11 <pbourke> one moment I'll check now
16:18:21 <sdake> oops ssorry pbourke didn't know you were here
16:18:51 <pbourke> i confirmed it as there are at least 3 separate people in in the comments saying they see it
16:18:57 <pbourke> as well as our own sean-k-mooney
16:19:15 <sdake> pbourke well sean should know how the software works
16:19:22 <sdake> can a core take on confirming this problem
16:19:33 <inc0> pbourke, can you check with Sean tomorrow to and try to triage it?
16:19:38 <pbourke> will do
16:19:39 <sdake> and fixing it
16:19:42 <inc0> thanks
16:19:57 <inc0> https://bugs.launchpad.net/kolla/+bug/1616268
16:19:57 <openstack> Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] - Assigned to Jeffrey Zhang (jeffrey4l)
16:20:13 <inc0> Jeffrey4l_, I understand you have theory and you're going to fix it?
16:20:27 <sdake> pretty sure Jeffrey4l_ is on the money on it
16:20:32 <Jeffrey4l_> inc0, yes. i am working on it. will push a PS later today.
16:20:39 <inc0> great
16:20:40 <inc0> thanks
16:20:52 <inc0> https://bugs.launchpad.net/bugs/1617334
16:20:53 <openstack> Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Critical,Triaged]
16:21:12 <sdake> this is marked critical because if reconfigure is busted, thats not good
16:21:21 <sdake> its not marked confirmed because it hasn't been confirmed in the bug log
16:21:27 <sdake> with several core reviewers trying iirc
16:21:51 <sdake> i use openvswitch all the time
16:21:53 <sdake> and it wfm
16:21:56 <sdake> even reconfigure
16:22:01 <berendt> same here
16:22:14 <sdake> i'm surprised linuxbridge works for htis fellow and not ovs
16:22:30 <sdake> a good course of action would be eto request more info and mark it incomplete
16:22:42 <sdake> determine what info to request is difficult ...
16:23:52 <inc0> but it's not triaged as well right?
16:24:01 <sdake> its been triaged
16:24:06 <sdake> it hasn't been confirmed
16:24:25 <sdake> triage = setting milestone and priority
16:24:51 <sdake> Jeffrey4l_ for htis bug, you requested his inventory file
16:25:09 <Jeffrey4l_> sdake, yep. but still get nothing idea now.
16:25:10 <sdake> Jeffrey4l_ so hard for us to mark it incomplete without actually looking at his inventory file
16:25:21 <sdake> Jeffrey4l_ ok - i've got this one
16:25:24 <sdake> go ahead and move on inc0
16:25:25 <inc0> nothing too special there
16:25:33 <inc0> https://bugs.launchpad.net/kolla/+bug/1625648
16:25:33 <openstack> Launchpad bug 1625648 in kolla "horizon login fails with TLS enabled" [Critical,Triaged]
16:25:55 <inc0> bjolo promised to try to reproduce
16:26:30 <zhubingbing> i don't reproduce this bug
16:26:56 <inc0> let's wait for bjolo
16:27:00 <inc0> https://bugs.launchpad.net/kolla/+bug/1631072
16:27:00 <openstack> Launchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [High,Triaged]
16:27:24 <sdake> are we going throug hthe high bugs too?
16:27:37 <Jeffrey4l_> this is a ubuntu bug.
16:27:38 <inc0> ahh copyfail sorry
16:27:57 <inc0> https://bugs.launchpad.net/kolla/+bug/1623013
16:27:57 <openstack> Launchpad bug 1623013 in kolla "keystone-fernet: rsync: Failed to exec ssh: No such file or directory (2)" [Critical,In progress] - Assigned to Christian Berendt (berendt)
16:28:04 <sdake> this must be fixed
16:28:05 <Jeffrey4l_> see #4 and #6
16:28:21 <sdake> berendt any updates? :)
16:28:33 <Jeffrey4l_> this is under review i think.
16:28:35 <berendt> i thought Jeffrey4l_ took this one
16:28:43 <sdake> i see
16:28:51 <Jeffrey4l_> https://review.openstack.org/#/c/369418
16:28:51 <sdake> well - seems like whos on first to me  :)
16:28:59 <Jeffrey4l_> review is welcome ;)
16:29:21 <inc0> ok, make this review priority folks please
16:29:24 <sdake> groan
16:29:32 <sdake> so - one thing i failed to mention in the announcements
16:29:34 <sdake> the gate is busted
16:29:37 <berendt> how to proceed? with or without ssh?
16:29:39 <sdake> because ceph.com is down
16:29:51 <inc0> yeah, but we can still review code
16:29:53 <inc0> without merging it
16:29:53 <sdake> a really dont want to merge anything complex before rc2
16:29:56 <sdake> right
16:30:05 <sdake> so review away - pls dontmerge anythign that isn't totally obvious :)
16:30:07 <Jeffrey4l_> we can not other ceph mirror.
16:30:10 <jascott1> ceph is back i thought
16:30:10 <inc0> so let's not merge until gates gets up, but review it
16:30:15 <Jeffrey4l_> we can use other ceph mirror
16:30:26 <Jeffrey4l_> jascott1, cool.
16:30:32 <berendt> jascott1 no it is still down
16:30:37 <berendt> at least on my site
16:30:56 <sdake> digitalocean has an outage
16:31:03 <sdake> ironically enough caused by ceph :)
16:31:14 <sdake> i think thats the place hosting this
16:31:21 <sdake> or maybe it was somewhere else
16:31:22 <sdake> anyway - its down for me too
16:31:24 <inc0> so last bug
16:31:28 <inc0> https://bugs.launchpad.net/kolla/+bug/1631503
16:31:28 <openstack> Launchpad bug 1631503 in kolla "inconsistent UID in rabbitmq results in inability to upgrade" [Critical,In progress] - Assigned to Steven Dake (sdake)
16:31:33 <jascott1> its dreamhost
16:31:38 <sdake> jascott1 right thanks
16:31:46 <Jeffrey4l_> fix is here: https://review.openstack.org/384598
16:32:03 <Jeffrey4l_> i will update the commit message.
16:32:04 <inc0> for rabbitmq
16:32:15 <Jeffrey4l_> but the fix logical is done.
16:32:15 <inc0> we need to figure out other affected services
16:32:24 <sdake> inc0 anything with a named volume
16:32:38 <inc0> which is almost everything
16:32:51 <Jeffrey4l_> right. so i think we need another bug to describe the detail rather then centos rabbitmq upgrade issue.
16:33:00 <inc0> Jeffrey4l_, can you make you review partial-bug please?
16:33:07 <inc0> or that
16:33:10 <sdake> change th topic of the bug eff
16:33:11 <sdake> Jeffrey4l_
16:33:21 <sdake> click yellow button next to bug topic
16:33:22 <Jeffrey4l_> inc0, np.
16:33:51 <Jeffrey4l_> sdake, yep. change it to ?
16:34:51 <sdake> Jeffrey4l_ make something up? :)
16:35:11 <sdake> "upgrades broken because of inconsistent UIDs" should do it
16:35:30 <Jeffrey4l_> done.
16:35:30 <inc0> I'd rather not create 10 bugs for each service, but once Jeffreys patch is merged we need to do same thing for all the services
16:35:34 <inc0> so we can spread work
16:35:48 <inc0> let's make sure to use same topic in review
16:35:56 <inc0> https://review.openstack.org/#/q/topic:bug/1631503
16:36:04 <inc0> to avoid duplication of work
16:36:15 <inc0> and reviewers, please keep eye on this link
16:36:29 <sdake> inc0 an etherpad would come in handy here i think - to figure out who is doing which part if you intend to spread load
16:36:54 <Jeffrey4l_> we need some thing to trick all the container which need fix.  should we use bugs/work items or anything else.
16:37:02 <sdake> 6 days to 3.0.0 - spreading work will be more difficult lI think
16:37:12 <Jeffrey4l_> sdake, agreed to use etherpad.
16:37:13 <inc0> well, one person won't handle it all
16:37:45 <inc0> #link https://etherpad.openstack.org/p/kolla-bug-1631503
16:37:58 <sdake> cool so if spreading work lets use etherpad - Jeffrey4l_ can you link that in the bug so devs ca nfind it
16:38:08 <Jeffrey4l_> no.
16:38:15 <Jeffrey4l_> sorry. np.
16:38:48 <inc0> http://s2.quickmeme.com/img/48/483ef0911f5e27073a015b45aee7a288b9c8d3bfa104f8bfe6625572f97cfa52.jpg sorry, couldn't help myself
16:39:20 <Jeffrey4l_> it is a typo. ;(
16:39:38 <inc0> I know, don't worry;)
16:40:02 <inc0> Jeffrey4l_, I changed topic on your ps to bug/1631503
16:40:18 <Jeffrey4l_> ok.
16:40:29 <inc0> so between etherpad and this gerrit review queue, let's fix this
16:40:53 <Jeffrey4l_> should we update the bug?  it do not just fix rabbitmq upgrade in centos now.
16:41:11 <sdake> Jeffrey4l_ we are doing partial-fix on each component i think
16:41:26 <Jeffrey4l_> but the bug title is bad: inconsistent UID in rabbitmq results in inability to upgrade
16:41:32 <inc0> https://bugs.launchpad.net/kolla/+bug/1631503 <= changed name
16:41:32 <openstack> Launchpad bug 1631503 in kolla "inconsistent UID in named volume results in inability to upgrade" [Critical,In progress] - Assigned to Steven Dake (sdake)
16:41:44 <sdake> Jeffrey4l_ i gave you a title in scrollback to use for bug description
16:41:46 <inc0> no mention of centos there
16:41:46 <Jeffrey4l_> actually it is: fix the uid issue during upgrade
16:42:33 <Jeffrey4l_> hmm. drop the "rabbitmq" word?
16:42:38 <inc0> I did:)
16:42:50 <Jeffrey4l_> cool
16:42:51 <inc0> replaced it with named voluem
16:43:04 <Jeffrey4l_> another think is: should we backport this to mitaka?
16:43:13 <sdake> one note on this bug
16:43:23 <inc0> so again:
16:43:23 <sdake> folks - the approach ahs been bikeshedded to death
16:43:27 <inc0> #link https://review.openstack.org/#/q/topic:bug/1631503
16:43:29 <sdake> lets just merge the design as is
16:43:38 <inc0> this link has to be reviewed, keep an eye on it
16:44:03 <sdake> by bikeshed to death - i've played telephone for atleast 8 hours on this specific issue
16:44:34 <inc0> ok, I think we stressed the importance enough:)
16:45:10 <inc0> so, we have status update on all critical bugs
16:45:16 <inc0> let's get them fixed in following week
16:45:30 <inc0> anything else to add?
16:45:48 <inc0> guess not
16:45:55 <inc0> #topic open discussion
16:46:12 <rhallisey> inc0, can we re order the summit sessions
16:46:12 <inc0> so one thing, you guys think we need meeting next week?
16:46:33 <duonghq> do we have semi-fixed summit sessions?
16:46:35 <inc0> yeah, let me do it now
16:46:35 <sdake> yes meeting next week - last minute summit prep is critical
16:46:43 <inc0> ok
16:46:47 <inc0> meeting stays then
16:46:54 <zhubingbing> About Heka we have any good alternative？
16:47:30 <sdake> zhubingbing we need an honest evaluation of the alternatives
16:47:37 <sdake> of which there are two that I am aware of
16:48:03 <duonghq> iirc there is solution already use in kolla-k8s?
16:48:05 <inc0> zhubingbing, I see 3 now, filebeat and fluentd are there, snap is meaning to grow this feature in near future (we should be ok with timing)
16:48:43 <sdake> snap sounds like heka 2.0 :)
16:49:00 <inc0> rhallisey, sdake what were your rescheduling issues? I'm switching osic review with ci
16:49:01 <zhubingbing> －）
16:49:10 <inc0> sdake, yeah, snap is exactly that
16:49:19 <rhallisey> inc0, can we move the road map later
16:49:20 <inc0> heka was cool, until mozilla dropped it
16:49:29 <rhallisey> I want to change around a bunch of things
16:49:36 <rhallisey> disagree with the current order
16:49:55 <rhallisey> I do have some conflicts, but I think the overall order needs to be changed
16:49:57 <pbourke> should look at logstash again also
16:50:08 <sdake> pbourke logstash-forwarder is EOL bro ;)
16:50:11 <rhallisey> imo
16:50:13 <jascott1> Mozilla  mention this yesterday failed to get kk8s up due to ceph.com outage so worked on ansible playbooks, reviewed code. Today create cinder ansible commit. Blockers: ceph.com is still down:(
16:50:23 <berendt> pbourke filebeat.. not logstash
16:50:23 <jascott1> whoa epic fail
16:50:25 <pbourke> sdake: you sure?
16:50:31 <jascott1> https://github.com/trink/hindsight)
16:50:35 <sdake> pbourke yup look at their repo
16:50:48 <duonghq> logstash-forwarder is replaced by Beat iirc
16:50:53 <inc0> rhallisey, whats wrong with it?;)
16:50:57 <duonghq> seem promise?
16:50:57 <sdake> pbourke elastic EOL'ed logstash-forwarder and introduced filebeat instead
16:51:15 <Jeffrey4l_> inc0, could u check this again https://review.openstack.org/378762  i perfer use the same timezone for all containers.
16:51:16 <sdake> why?  don't know ;)
16:51:18 <duonghq> sdake: Beats, indeed
16:51:32 <rhallisey> inc0, let's get into it after the meeting
16:51:44 <pbourke> whats the difference between logstash and logstash forwarder :/
16:51:56 <inc0> ok rhallisey
16:51:58 <sdake> logstash collects logs and sends them to elasticsearch
16:52:07 <zhubingbing> iirc what's mean ? i don;t know ;)
16:52:10 <sdake> logstachforwarder sends logs to logstash (or elasticsearch)
16:52:11 <inc0> Jeffrey4l_, ok, I just wanted more context
16:52:13 <sdake> we don't need logstash itself
16:52:27 <duonghq> pbourke: Elastic introdue Beats as protocol for log shipping purpose
16:52:35 <jascott1> is mozilla's own recommeded replacement not suitable? https://github.com/trink/hindsight
16:52:37 <Jeffrey4l_> what happen to logstash? isn't it widely used?
16:53:00 <sdake> jascott1 we need an honest eval of the different solutions
16:53:07 <sdake> rather then picking one randomly :)
16:53:12 <pbourke> jascott1: i would think twice before using something from mozilla again
16:53:21 <jascott1> oh sure but no one had mentioned it. got it.
16:53:21 <Jeffrey4l_> we'd better talk the alternative after the summit ;)
16:53:22 <wirehead_> heh
16:53:23 <berendt> logstash-forwarder was replaced by filebeat
16:53:24 <inc0> well, it's a 4th option to look at sdake
16:53:28 <berendt> logstash is not deprecated
16:53:40 <jascott1> logstash is 300M JVM iirc
16:53:49 <duonghq> sdake: is there some criterial for evaluate logging engine?
16:54:16 <wirehead_> Filebeat is a tiny little go app.
16:54:19 <inc0> it has to do what heka is doing today and not deprecated;)
16:54:21 <sdake> duonghq not that i know of - an action item
16:54:31 <duonghq> ya
16:54:32 <wirehead_> (Logstash-forwarder was also pretty cute and tiny)
16:54:44 <inc0> if it meets these 2 simple criteria - pick up best one
16:54:44 <sdake> wirehead_ minus the java jvm part...
16:54:59 <sdake> it should also be effecient - 3rd criteria
16:55:01 <wirehead_> You don’t need the Logstash engine actually running unless you are mediating queues and stuff, filebeat can go straight to elasticsearch
16:55:09 <sdake> 25% of our cpu consumption on controller nodes is the logging stack
16:55:25 <wirehead_> Naw, logstash-forwarder is also a go app
16:55:52 <sdake> guess its time to learn go :)
16:56:01 <duonghq> agreed with sdake
16:56:10 <sdake> i was making a joke
16:56:14 <sdake> but ok :)
16:56:14 <wirehead_> <best Marvin the paranoid android voice> It’s awful </best Marvin the paranoid android voice>
16:56:48 <wirehead_> I mean, Logstash / Filebeat / ElasticSearch is sufficently popular that there has to be a clear migration off of it if something changes.
16:56:52 <duonghq> about summit sessions, have we fixed topic about kolla-k8s?
16:57:08 <inc0> duonghq, 2 sessions for kolla-k8s
16:57:12 <sdake> duonghq the schedule is in #opnestack-kolla topic
16:57:13 <inc0> roadmap and arch
16:57:14 <rhallisey> duonghq, ya there are 2
16:57:36 <inc0> anwyay, we're running out of time
16:57:39 <duonghq> okay
16:57:44 <inc0> let's move to our normal channel please
16:57:56 <inc0> unless there are last comments to make?:)
16:57:57 <zhubingbing> bye；）
16:58:04 <duonghq> thank you all
16:58:12 <inc0> ok, thank you guys! see you in summit I hope!
16:58:12 <Jeffrey4l_> hey guys.
16:58:15 <inc0> #endmeeting kolla