15:59:54 <inc0> #startmeeting kolla
15:59:55 <openstack> Meeting started Wed May 31 15:59:54 2017 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:59:59 <openstack> The meeting name has been set to 'kolla'
16:00:04 <sdake> o/
16:00:06 <inc0> #topic w00t for Kolla!
16:00:09 <spsurya_> o/
16:00:15 <berendt> o/
16:00:17 <rwellum> w00t
16:00:18 <jascott1> o/
16:00:19 <spsurya_> w00t
16:00:28 <berendt> Woot
16:00:39 <inc0> we need to make video in one of summits where everyone in community will shout w00t;)
16:00:47 <inc0> (yes, with zeros)
16:00:52 <krtaylor> o/
16:00:53 <berendt> Great idea.
16:00:54 <duonghq> wo0t o/
16:01:55 <egonzalez> woot!
16:01:57 <inc0> lets give few more minutes
16:02:19 <Jeffrey4l> o/
16:03:04 <inc0> ok let's get to business
16:03:05 <TxGirlGeek> w00t!!
16:03:10 <inc0> #topic announcements
16:03:24 <inc0> 1. new resolution from TC was proposed that will directly affect us
16:03:52 <inc0> https://review.openstack.org/#/c/469265/ this resolution will provide guidelines for publishing images to dockerhub/quay.io
16:04:10 <inc0> I encourage everyone to read it carefully and provide feedback
16:04:26 * krtaylor looks
16:04:37 <berendt> #link https://review.openstack.org/#/c/469265
16:04:48 <inc0> thanks Christian
16:05:23 <inc0> so, on that note we'll kick off this whole project of publishing, so if anyone wants to contribute, I'd appreciate help
16:05:38 <inc0> there is a lot of cool code to be written;)
16:06:18 <inc0> 2. I haven't done this last week (since I was out), but I would like to congratulate Bertrand and welcome him to core team:)
16:06:38 <sdake> grats!
16:06:49 <inc0> any community announcements?
16:06:59 <sdake> question - is the ptg eventprint page up?
16:07:11 <inc0> I haven't heard anything yet
16:07:19 <sdake> thanks
16:07:34 <inc0> ok, let's move on
16:07:34 <sdake> need to book travel before the checkwriters change their minds :)
16:07:40 <sdake> can you ask around inc0?
16:08:00 <inc0> check with ttx, he was one who gave me link last time;)
16:08:14 <inc0> or I can do it
16:08:21 <inc0> anyway, let's move on
16:08:30 <inc0> #topic bp/ansible-specific-task-become (duonghq)
16:08:35 <inc0> duonghq: go ahead
16:09:44 <duonghq> inc0, thanks, but I don't remember that I register this topic for this meeting, but, I still need more suggestion for the bp
16:10:07 <duonghq> only 3ps
16:10:14 <duonghq> https://review.openstack.org/#/c/398682/
16:10:19 <duonghq> https://review.openstack.org/#/c/398684/
16:10:23 <duonghq> https://review.openstack.org/#/c/398685/
16:11:04 <duonghq> inc0, I saw you comment on 2nd ps, can you re-review that?
16:11:29 <inc0> sure
16:11:39 <duonghq> especially about gate in 2nd ps
16:11:43 <inc0> I'll wait for neutron to get fixed tho
16:11:56 <inc0> it's critical that this ps gets gate coverage
16:12:12 <duonghq> sorry, I don't get your point
16:12:29 <inc0> duonghq: neutron bug makes our gates red'
16:12:29 <duonghq> you mean the rolling update one?
16:12:45 <duonghq> got it
16:13:22 <inc0> ok, anything else Duong on this topic?
16:13:31 <duonghq> I'm done, thank you
16:13:38 <inc0> #topic nova cell instabilities (rwellum)
16:13:45 <inc0> rwellum: you have the floor
16:13:52 <rwellum> Yes thanks -  just a question: These micro-services were added to a service recently: nova-cell0-create-db-job and nova-api-create-simple-cell-job. For me nova seems unstable - takes a lot longer in kubernetes for the pods to stabilize. Curious if anyone else is seeing this, and if I should write a bug?
16:14:18 <inc0> rwellum: ocata nova requires cell0 setup
16:14:33 <inc0> and this is whole new database and such
16:15:04 <rwellum> Yes but I was adding them as micro-services - and it worked fine.
16:15:20 <rwellum> As long as the order was correct etc.
16:15:39 <inc0> kfox1111 around?
16:16:04 <rwellum> I think sdake mentioned he was suspicious about the change.
16:16:05 <sdake> rwellum i think that change is broken
16:16:09 <rwellum> ah
16:16:22 <inc0> I guess we need to un-break it then;)
16:16:30 <sdake> inc0 i dont know how
16:16:41 <sdake> inc0 its a circular dependency issue
16:16:46 <sdake> it works in the gate because the gate spams the operations
16:16:47 <inc0> mhm
16:17:03 <sdake> when done manually it will not work reliably
16:17:14 <inc0> I see
16:17:23 <sdake> my best recommendation would be to revert the change
16:17:28 <inc0> ok, publish a bug, mark it critical and let's work to fix it?
16:17:34 <rwellum> sdake: even for kolla-ansible?
16:17:42 <sdake> rwellum i dont follow for kolla-ansible
16:17:51 <sdake> kolla-ansible syncronizees in the correct order
16:17:55 <rwellum> got it
16:17:57 <sdake> kolla-kubernetes has no such synchronization
16:17:58 <inc0> rwellum: kolla ansible doesn't have this issue
16:18:17 <inc0> sdake: change dependency order in entrypoint?
16:18:18 <sdake> cell0 must be registered after the placement api
16:18:30 <sdake> inc0 unfortunately i think that wont work either
16:18:37 <inc0> why
16:18:38 <sdake> as cell0 is optinoal
16:18:40 <inc0> ?
16:18:54 <sdake> and entrypoint doesn't support optionals
16:19:00 <inc0> oh joy
16:19:08 <sdake> sorry conditonal
16:19:12 <sdake> (the stuff jascott1 added)
16:19:30 <sdake> the bestsolution i have is to make another service chart specifically for cell registration
16:19:45 <inc0> it's ugly
16:19:59 <sdake> this is why the dream of helm install kolla-kubernetes is just a dream atm :)
16:20:05 <inc0> ok, let's discuss it with Kevin and Serguey when they're back
16:20:10 <sdake> soundsgood
16:20:15 <sdake> i've brought it up prior
16:20:22 <sdake> "it works in the gate, nothign wrong"
16:20:27 <sdake> hard to argue against...
16:20:27 <inc0> maybe it's good time to ditch entrypoint and write it ourselves
16:20:35 <jascott1> :)
16:20:41 <sdake> +100
16:20:55 <rwellum> sdake: meantime a bug as inc0 suggested?
16:21:00 <inc0> yeah
16:21:07 <sdake> rwellum i guess - although kolla-kubernetes not really using bugs atm
16:21:16 <sdake> although sould be
16:21:22 <inc0> good time to start
16:21:26 <sdake> implementation is mature enough for it
16:21:47 <sdake> would be nice to have sbezverk and kofx here for that discusison
16:21:57 <inc0> entrypoint shouldn't be too hard to write (if we do it in python that is)
16:22:12 <sdake> agreed and we would have more control over the changes
16:22:18 <rwellum> I'll hunt them down and get them involved.
16:22:20 <sdake> it seems to me that the entyrpoint upstream is dead
16:22:32 <sdake> (i could be wrong, this is just my perception)
16:22:36 <inc0> it is
16:22:48 <sdake> we shouldn't base any deps on a dead upstream imo
16:23:23 <sdake> no security, no mainteannce, no enhancements, etc
16:23:32 <inc0> ok, let's get cracking on our new entrypoint...and for f* sake call it something else;)
16:23:41 <jascott1> haha
16:23:47 <inc0> k8s dep resolver or whatever
16:23:59 <sdake> need someone to write the code
16:24:03 <rwellum> covfefe?
16:24:11 <inc0> covfefe +1
16:24:12 <sdake> i am maxed out on capacity
16:24:22 <sdake> so i cna't do it :)
16:24:24 <inc0> jascott1? you and me?
16:24:33 <jascott1> sure
16:24:37 <sdake> personally I htink the approach we should use for the short term is to get a deploy workign with microcharts
16:24:40 <inc0> I could use some python programming for a change;)
16:24:53 <sdake> inc0 that sounds good to me :)
16:25:08 <sdake> however a microchart approach has a short runway
16:25:21 <inc0> we still need proper dep resolution
16:25:36 <sdake> we can do that in an ansible workflow wrapped in a container based on  microcharts
16:25:47 <inc0> so make it work ugly asap and we'll get crackin' on covfefe-resolver
16:25:48 <sdake> and then improve it to service charts next
16:26:03 <rwellum> inc0: +1 :)
16:26:14 <sdake> i am also not convinced about entrypoint's ordering
16:26:28 <sdake> as you know inc0 many openstack services require upgrade in a specific order
16:26:36 <sdake> entrypoint - who knows what happens ;)
16:26:56 <inc0> sdake: all ordering can be managed by proper dependency modelling
16:26:58 <sdake> whereas a microchart approach - ordering is forced
16:27:05 <inc0> I wanted to rewrite heat to use this algorithm
16:27:26 <inc0> microchart has no ordering whatsoever unless manually
16:27:33 <sdake> there would need to be two orderings -one for deploy and one for upgrade
16:27:41 <sdake> i think they are different - but not certain
16:27:46 <inc0> covfefe-resolver is microchart orchiestration tool
16:28:13 <inc0> jascott1 another idea, don't you think that would make awesome addition to tiller?
16:28:32 <jascott1> interesting idea
16:28:42 <inc0> since it already manages dependency, just doing this naively
16:28:56 <inc0> let's grab technosophos later and discuss that
16:28:59 <sdake> not familiar wtih the helm code - don't know how long that effort would take
16:29:13 <sdake> i'd like to get to the point poeple can deploy in the next 3-4 weeks :)
16:29:39 <rwellum> It deploys - those pods just flap around like crazy a few times :)
16:30:00 <sdake> rwellum by deploy - i mean error-free
16:30:16 <rwellum> Aha
16:31:16 <rwellum> Ok well that was it from me - glad that sdake sees it too.
16:31:31 <sdake> rwellum yup i knew when that went it it wouldn't work :)
16:31:44 <rwellum> Did you +1 then?
16:31:50 <sdake> gate was green.
16:31:55 <inc0> #action inc0 and jascott1 to figure it out;)
16:31:56 <sdake> shows the gate is imperfect
16:32:03 <rwellum> I missed the review unfortunately
16:32:11 <inc0> it's never perfect
16:32:28 <inc0> ok anything else?
16:32:29 <sdake> rwellum can you submit a revert patch for that change plz
16:32:55 <rwellum> sdake: sure
16:32:59 <sdake> rwellum thx :)
16:33:29 <inc0> ok moving on:)
16:33:33 <sdake> one last thing
16:33:37 <inc0> go ahead
16:33:43 <sdake> (oh its related to mariadb)
16:33:50 <sdake> dont' have agenda open - we can tackle it at end
16:34:01 <inc0> yeah, it's open discussion now
16:34:04 <sdake> ok
16:34:06 <inc0> #topic open discussion
16:34:16 <sdake> here is plan for mariadb
16:34:31 <sdake> 1. backport sean's pin of mariadb ot stable/ocata
16:34:38 <sdake> s we don't want  to upgrade mariadb in stable branches
16:35:04 <sdake> 2. use mariadb from rdo (which includes galera and xtrabackup-v2)
16:35:16 <sdake> this allows us to eject the percona and mariadb repos for centos
16:35:25 <inc0> just to be clear, we're only talking about centos right?
16:35:27 <sdake> and gets us on a well maintained version of mariadb
16:35:30 <sdake> yes just centos
16:35:39 <sdake> i don't know what ot do about ubuntu as i dont have it installed
16:35:54 <inc0> pinning versions will only work if we make security updates work
16:35:55 <sdake> 1 is already done
16:36:09 <sdake> ya everytime I see a pin I die a little inside
16:36:27 <inc0> you must be pretty dead inside sdake then
16:36:33 <egonzalez> btw, just checked and mariadb included missing packages for 10.0
16:36:34 <sdake> inc0 souless :(
16:36:51 <sdake> egonzalez for which distro?
16:36:54 <egonzalez> centos
16:37:02 <egonzalez> https://yum.mariadb.org/10.0.30/centos7-amd64/rpms/
16:37:04 <inc0> ah so they fixed their repo upstream?
16:37:16 <sdake> inc0 no, there is no security updates on 10.0.30 that i know of
16:37:18 <rwellum> sdake: what are they symptoms of the mariadb issue?
16:37:20 <sdake> inc0 although thatis pure speculation
16:37:27 <sdake> rwellum stable/ocata wasn't building
16:37:40 <rwellum> ah ok ty
16:37:46 <inc0> well, I'm saying is 10.0.* not 10.0.30
16:38:37 <sdake> egonzalez 10.0 doesn't work right?
16:38:54 <egonzalez> -w the backport pin change
16:38:55 <inc0> https://yum.mariadb.org/10.0/centos/7.3/x86_64/rpms/ we have packages for 10.0.31
16:39:00 <egonzalez> sdake, shoudl work now
16:39:10 <sdake> cool so we can unpin?
16:39:14 <egonzalez> yep
16:39:15 <inc0> also if it's fix, let's unpin master
16:39:19 <inc0> fixed
16:39:23 <sdake> inc0 agreed
16:39:31 <inc0> cool, I'm glad
16:39:40 <sdake> inc0 although stable/ocata wasn't building
16:39:43 <sdake> and it was unpinned
16:39:48 <inc0> sdake: let's try now
16:39:49 <sdake> and i rechecked 30 minutes ago
16:40:00 <inc0> right
16:40:04 <egonzalez> i rechecked a change now, lets see..
16:40:32 <sdake> egonzalez the 10.0.30 pin went into stable/ocata
16:41:25 <egonzalez> was not merged, -workflow  the change
16:41:27 <sdake> i think the course of action that make sense is to revert the pin that sean produced, and see if the  gate blows up on master
16:41:42 <sdake> and then replace it later with rdo mariadb
16:42:44 <sdake> sounds like new coure of action is
16:42:46 <sdake> 1) unpin master
16:42:52 <sdake> 2) backport unpin if it works
16:42:59 <sdake> 3) replace master mariadb with rdo version
16:44:03 <sdake> also could use some mariadb nerds to help wti hthe rdo transition
16:44:09 <sdake> the extend_start.sh script is busted in master
16:44:11 <sdake> it never wroked
16:45:04 <sdake> could be why ubuntu packaging doesn't work as well
16:45:26 <sdake> anyway thats it - need help with mariadb plz :)
16:45:53 <inc0> right
16:45:59 <inc0> can we wrap up?
16:46:24 <duonghq> o/
16:46:35 <duonghq> https://blueprints.launchpad.net/kolla-ansible/+spec/enable-osprofiler
16:46:35 <inc0> thank you all for coming!
16:46:42 <inc0> ah
16:46:43 <inc0> sorry
16:46:53 <inc0> I thought you were waving goodbye;)
16:46:59 <duonghq> :P, my fault
16:47:08 <duonghq> will we make the bp in P?
16:47:27 <duonghq> I think it's not a complicated bp, but interesting one
16:47:32 <inc0> do we want to make it default yes?
16:47:33 <duonghq> egonzalez's one
16:47:49 <inc0> is there anything bad that could happen from enabling it?
16:48:01 <duonghq> yeah, it's resource hungry
16:48:09 <duonghq> so it should be turned off by default
16:48:15 <inc0> ok
16:48:18 <duonghq> but easy to turn on
16:48:22 <egonzalez> i would say this change makes more sense for devs with pbourke_'s devstack change
16:48:58 <egonzalez> maybe for gates to have a better view of how resources work
16:49:30 <duonghq> tracing is quite hot topic nowaday, not only for dev :)
16:50:16 <duonghq> but osprofiler has some interesting features in its design, we should choose some for implement or try to implement all of this?
16:50:42 <duonghq> i.e. different hmac_key for different "trace path"
16:50:52 <inc0> btw if it's resource hungry, that's actually good candidate to be rewritten in golang
16:51:07 <inc0> golang would totally shine in this use case
16:51:34 <egonzalez> is resource hungry because need elasticsearch or all metering services to store traces
16:52:24 <duonghq> yep
16:52:54 <duonghq> but the main point I want to discuss is will we implement all of its nice option, or just some for beginning then improve latter
16:53:05 <duonghq> any ideas?
16:53:30 <inc0> I'd split it into multiple patchsets if that's what you're asking
16:53:34 <inc0> easier to merge/review
16:53:37 <egonzalez> what you mean of nice option?
16:53:46 <duonghq> i.e. HMAC_KEY
16:53:57 <duonghq> egonzalez, in your implementation, you use only one HMAC_KEY,
16:54:17 <duonghq> but osprofiler let we ultilize HMAC_KEY as kind of path chooser
16:54:44 <duonghq> I mean the path which osprofiler will trace in whole request processing sequence
16:55:27 <duonghq> also, we have some caveat for consider,
16:55:32 <duonghq> cannot recall right now
16:56:03 <duonghq> but, summerize: they have option to let user configure tracing quite flexible
16:56:36 <duonghq> anw, our time is running out?
16:56:41 <inc0> yeah
16:56:47 <inc0> I'll wrap it up:)
16:56:51 <inc0> thank you all for coming!
16:56:57 <inc0> #endmeeting kolla