21:01:02 <strigazi> #startmeeting containers
21:01:03 <openstack> Meeting started Tue Oct 16 21:01:02 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:07 <openstack> The meeting name has been set to 'containers'
21:01:30 <imdigitaljim> o/
21:01:33 <strigazi> #topic Roll Call
21:01:39 <cbrumm__> o/
21:01:42 <imdigitaljim> o/
21:01:43 <strigazi> o/
21:01:52 <eandersson> \o/
21:02:10 <flwang> \o/
21:02:38 <strigazi> #topic Announcements
21:03:21 <strigazi> thanks eandersson for fixing the magnum gate https://review.openstack.org/#/c/610740/
21:03:27 <eandersson> anytime
21:03:41 <strigazi> Do you know why it used work or why ot broke?
21:03:48 <strigazi> Do you know why it used work or why it broke?
21:03:59 <eandersson> I was looking at some recent changes with oslo.messaging that affects the defaults
21:04:23 <eandersson> Which I think caused it with 9.0.1
21:04:25 <flwang> that's bad
21:04:27 <strigazi> o.m version bump then, thanks
21:05:10 <eandersson> https://github.com/openstack/requirements/commit/29f8a80a71111ea59141c1aeab94bae9cf3f30a9#diff-0bdd949ed8a7fdd4f95240bd951779c8
21:05:16 <flwang> introducing changes in base components and breaking everybody
21:05:31 <strigazi> flwang: it is better than it used to be I think :)
21:05:44 <flwang> better for what?
21:05:54 <strigazi> less breaks/ time unit
21:06:03 <flwang> haha
21:06:11 <eandersson> Yea - it's usually 3rd party libraries now that break things
21:06:13 <eandersson> like evenlet
21:06:15 <canori02> O/
21:06:20 <flwang> it's true, but i think it's because its losing contributors :D
21:07:04 <strigazi> canori02: o/
21:07:18 <strigazi> #topic Stories/Tasks
21:08:39 <strigazi> I don't think I have anything from last week to discuss, flwang do you want to discuss the CT rename?
21:09:03 <imdigitaljim> flwang: i found a solution to making a single LB scenario
21:09:15 <imdigitaljim> huge cut in resource usage is nice :)
21:09:54 <strigazi> imdigitaljim: use a single LB for many services?
21:10:02 <imdigitaljim> for etcd/api
21:10:12 <strigazi> oh, right
21:10:27 <strigazi> imdigitaljim: is there a patch?
21:10:33 <imdigitaljim> i think our plan is to submit a v2 driver that is parity with the first one's features
21:10:44 <imdigitaljim> and huge improvements all around
21:10:57 <imdigitaljim> cluster speed/feature rich/compatibility
21:11:44 <flwang> imdigitaljim: cool, we can discuss the single lb later
21:11:50 <imdigitaljim> yeah definitely
21:12:03 <imdigitaljim> i can push up a WIP for you guys soon if you want to see where things are
21:12:03 <flwang> strigazi: yep, i'd like to know you guys opinions about CT rename
21:12:09 <flwang> imdigitaljim: go for it
21:12:41 <flwang> as you know, Catalyst Cloud just released magnum on our prod
21:12:50 <cbrumm__> Nice!
21:12:55 <cbrumm__> congrats
21:12:56 <flwang> and we're starting to get the pain of change/update public templates
21:13:04 <flwang> cbrumm__: hah, thanks
21:13:32 <colin-> that's great, how did the deployment go?
21:13:59 <flwang> colin-: we got some issues, but so far, most of the things are ok
21:14:27 <flwang> a big lesson learn is don't use docker_storage_size with overlay/overlay2
21:14:42 <imdigitaljim> we are considering a redesign (part of the v2)
21:14:46 <imdigitaljim> that decouples that entirely
21:14:50 <flwang> and magnum/heat-container-agent does have the multi region bug
21:15:01 <flwang> imdigitaljim: nice
21:15:04 <colin-> that sounds rough, glad it worked out though
21:15:12 <eandersson> what is the issue with overlay?
21:15:14 <imdigitaljim> so you dont have to worry about kube-tag or other quick changing data
21:15:22 <imdigitaljim> ^
21:15:28 <imdigitaljim> +1 eandersson
21:15:34 <imdigitaljim> what is the issues with overlay2?
21:15:35 <flwang> imdigitaljim: that's interesting
21:16:24 <flwang> with overlay2 and docker_volume_size(external volume for docker), we have seen some issue with docker, which cause pods can't be created
21:16:31 <flwang> it's not always repeatable
21:16:56 <flwang> we just realized it yesterday with strigazi's help, still need testing though
21:16:59 <flwang> but just FYI
21:17:17 <flwang> btw, our storage backend is ceph
21:17:18 <imdigitaljim> cgroup issue?
21:17:25 <eandersson> Interesting - feel free to share more when you have had time to dig deeper
21:17:32 <flwang> we're using cgroupfs already
21:17:35 <colin-> which cgroup driver? just curious
21:17:38 <colin-> ok
21:18:23 <flwang> I still don't know the root cause, may need help from storage expert or docker expert
21:18:33 <flwang> but i will keep you guys posted if there is any progress
21:18:43 <eandersson> Sounds good - thanks
21:18:48 <tobias-urdin> flwang: sorry for shiming in, regarding the mentioned multi-region bug in heat-container-agent is that addressed or is there a bug tracked somewhere with more info?
21:18:54 <tobias-urdin> we are planning to deploy magnum in another region soon
21:19:49 <tobias-urdin> eandersson: thanks for the review
21:19:56 <eandersson> anytime
21:20:37 <strigazi> tobias-urdin: it is addressed, one moment
21:21:27 <strigazi> tobias-urdin: https://review.openstack.org/#/c/584215/
21:21:56 <strigazi> tobias-urdin: included here: https://hub.docker.com/r/openstackmagnum/heat-container-agent/tags/ rocky-stable
21:21:57 <flwang> it has been fixed
21:22:20 <strigazi> tobias-urdin: this patch is pending: https://review.openstack.org/#/c/609579/
21:22:53 <flwang> tobias-urdin: you also need you update your existing deployment
21:23:00 <flwang> i mean Mangum
21:23:06 <flwang> for existing cluster, should be fine
21:23:09 <strigazi> flwang: and heat to queens, right?
21:23:23 <flwang> strigazi: yes, at least Queens
21:23:54 <flwang> I fixed it in Rocky, and I think i have backported all  necessary changes into queens
21:24:10 <tobias-urdin> ok thanks! so if I understand correctly when 609579 is merged and backported to stable/rocky we are all good?
21:24:21 <strigazi> tobias-urdin: yes
21:24:33 <tobias-urdin> got it, thanks
21:25:59 <flwang> all good for Magnum
21:26:14 <flwang> tobias-urdin: also need to upgrade Heat
21:27:24 <tobias-urdin> ack
21:28:18 <flwang> tobias-urdin: me and strigazi will have a topic at Berlin to share what we know ;)
21:31:40 <tobias-urdin> flwang: cool, i'll keep an close eye out for the video on that, unfortunately won't attend berlin :(
21:32:06 <flwang> no problem
21:32:20 <flwang> strigazi: i'm good now
21:32:33 <flwang> strigazi: imdigitaljim: anything else you want to discuss?
21:33:27 <imdigitaljim> just throwing it out, were working on inplace upgrades currently
21:33:35 <imdigitaljim> strigazi: has some work thats very close
21:33:39 <imdigitaljim> to what we started with
21:33:59 <imdigitaljim> we're adding new mechanisms but i should also be adding it to the v2 driver
21:34:21 <imdigitaljim> but id like to maybe discuss some basic approaches next time
21:34:22 <strigazi> imdigitaljim: will you push upstream something for v1?
21:34:44 <imdigitaljim> probably not for v1 since its too far back from the updates
21:34:54 <imdigitaljim> it perhaps can be backported
21:35:59 <imdigitaljim> we *are* keeping features of the v1 driver introduced on the v2
21:36:11 <imdigitaljim> so when were done with v2 it should be a minimum everything v1 has
21:36:46 <imdigitaljim> people can still choose v1 for their needs/legacy but because its a driver they can also choose v2
21:36:57 <imdigitaljim> :P
21:37:00 <strigazi> I think we should have a spec or another document for these changes
21:37:14 <strigazi> makes sense?
21:37:39 <imdigitaljim> im sure i can come up with something :)
21:38:18 <strigazi> maybe a spec to sync with nodegroups
21:38:30 <imdigitaljim> yeah i plan to watch your nodegroups changes and improve/port that
21:38:53 <imdigitaljim> (were not using it so i might need you to test it)
21:39:15 <strigazi> sure
21:39:23 <imdigitaljim> i will still be pushing a few for common
21:39:33 <imdigitaljim> that can help any/all drivers
21:40:11 <imdigitaljim> flwang: btw congrats on the prod deploy!
21:41:03 <flwang> imdigitaljim: thanks, it wouldn't happen without you guys help
21:41:12 <flwang> strigazi: +1 for spec for v2
21:41:32 <flwang> I assume it a big change, so better to have a spec
21:41:51 <flwang> or at least doing the code and spec in parallel, so that reviewer can understand the design correctly
21:41:52 <imdigitaljim> yeah totally understandable
21:42:11 <imdigitaljim> some of the goals of the v2 was improved readability as well
21:42:33 <flwang> imdigitaljim: quick question, does the v2 involve any API change?
21:42:39 <imdigitaljim> no
21:42:42 <flwang> nice
21:42:57 <imdigitaljim> core magnum is mostly untouched
21:43:33 <flwang> it's great
21:44:46 <imdigitaljim> there are a some constraints heat >= queens/rocky?, octavia (if used) >= rocky, kubernetes >= v1.11.X
21:45:11 <imdigitaljim> i think heat can be pike if not using octavia though
21:45:50 <imdigitaljim> octavia >= queens*
21:46:01 <strigazi> queens is a reasonable requirement
21:49:34 <strigazi> Anything else for the meeting?
21:49:51 <flwang> strigazi: i'm good
21:50:10 <strigazi> others?
21:50:40 <tobias-urdin> the issue we talked about earlier today is fixed in https://review.openstack.org/#/c/611097/
21:51:06 <tobias-urdin> nothing fancy :)
21:51:18 <strigazi> thanks +2
21:51:33 <tobias-urdin> should be safe for backports as well
21:52:49 <strigazi> thanks everyone for joining the meeting, see you
21:53:10 <strigazi> #endmeeting