15:58:43 <inc0> #startmeeting kolla
15:58:44 <openstack> Meeting started Wed Nov 15 15:58:43 2017 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:58:47 <openstack> The meeting name has been set to 'kolla'
15:58:59 <chason> o/
15:59:04 <inc0> #topic w00t
15:59:06 <pbourke> w00t
15:59:07 <inc0> woo!
15:59:14 <duonghq> wo0t
15:59:33 <Jeffrey4l> o/
16:01:30 <spsurya__> woot
16:01:38 <egonzalez> Woot!
16:02:13 <gema> o/
16:02:15 <coolsvap> o/
16:03:01 <inc0> #topic announcements
16:03:16 <inc0> 1. congrats to hrw! Our newest core
16:03:39 <egonzalez> Yay, congrats hrw
16:03:52 <Jeffrey4l> congrats hrw
16:04:01 <duonghq> congrats hrw
16:04:08 <inc0> he'll have to read log to see this;)
16:04:10 <gema> congratulations indeed :D
16:04:21 <spsurya__> grats hrw
16:04:27 <chason> congrats hrw
16:04:38 <hrw> yo
16:04:43 <inc0> hrw: man of the hour is here;)
16:04:47 <hrw> ;D
16:04:56 <hrw> I thought that meeting is in 1h from now
16:04:56 <inc0> we're just throwing congratulations - you're now a core:)
16:05:03 <duonghq> congrats hrw (again)
16:05:04 <hrw> omg.
16:05:09 <hrw> thanks ;)
16:05:17 <inc0> yeah time changes are problem
16:05:36 <inc0> so, let's move on
16:05:47 <inc0> any announcements from community?
16:06:07 <inc0> ok
16:06:19 <inc0> we don't have agenda, but there is at least one thing I wanted to talk about
16:06:29 <hrw> I am planning to give a talk about kolla on devconf.cz conference
16:06:29 <inc0> #topic gate state of union
16:06:37 <inc0> hrw: would be awesome
16:06:59 <hrw> inc0: indeed - would cut costs a lot for me ;d
16:07:37 <inc0> ok
16:07:53 <inc0> so as you know last month or so was little busy with gate work
16:08:28 <inc0> one big change is that (as for now, plans might change) we're moving to use dockerhub for our deploy gates
16:08:50 <inc0> #link https://hub.docker.com/u/kolla/
16:08:58 <inc0> images are published daily - for master
16:08:58 <Jeffrey4l> and remove the registry tarballs in tarballs.o.o site, right?
16:09:17 <hrw> inc0: so no internal-infra registry?
16:09:20 <inc0> #link https://hub.docker.com/r/kolla/ubuntu-source-keystone/tags/
16:09:36 <inc0> hrw: not right away, we might end up needing it too, but we need something now
16:09:45 <hrw> sure
16:09:46 <inc0> yeah, tarballs are going away when we're done
16:10:29 <inc0> logs from build jobs (periodic) can be found here
16:10:33 <inc0> #link http://logs.openstack.org/periodic/git.openstack.org/openstack/kolla/master/
16:12:15 <inc0> so one more thing
16:12:27 <inc0> I was thinking of rewriting quickstart to assumie dockerhub images
16:12:32 <inc0> that should make it a lot easier
16:12:51 <Jeffrey4l> great.
16:13:03 <hrw> inc0: instead of building those?
16:13:12 <inc0> well in quickstart, yes
16:13:17 <hrw> +2
16:13:27 <chason> +1
16:14:22 <inc0> I'll do it right after we backport image publish to pike
16:16:12 <inc0> ok thats it from me
16:16:20 <inc0> any comments
16:16:22 <inc0> ?
16:17:50 <hrw> inc0: good job
16:18:12 <gema> inc0: was going to ask for an update from anyone that was in Sydney last week
16:18:20 <gema> did anything noteworthy happen?
16:19:03 <hrw> inc0: we (Linaro) plan to publish queens images soon
16:19:37 <hrw> Pike ones got rebuilt last week: https://hub.docker.com/u/haerwu/ - aarch64 only
16:20:04 <inc0> cool
16:20:40 <hrw> testing generated some patches. most are merged into master already and backported ;)
16:21:02 <inc0> yeah when we'll end up having arm nodepool, (and learn how to put gates on it), we can build it daily too
16:21:21 <hrw> and now I have one more point for my coworkers to send patches directly instead of though me ;D
16:21:22 * inc0 looks at gema and hrw intently
16:21:25 <gema> inc0: still working on stabilizing the clouds
16:21:35 <gema> inc0: I don't forget , but it is taking time
16:21:44 <hrw> inc0: we have to get our clouds into order first
16:21:49 <gema> on the bright side our new devops person starts mid december
16:21:52 <gema> :)
16:22:39 <gema> we need to add capacity for this also
16:22:51 <gema> got the servers ready, just need the hands and a few days to get on with it
16:23:04 <hrw> more time for gema to work on other things from her far-too-long todolist :D
16:23:38 <inc0> right:)
16:24:06 <inc0> no worries, just saying that when we get it, we'll have means to make it really nice and streamlined
16:24:25 <gema> inc0: yep, I am hoping before queens to have some sort of gating in place
16:24:29 <gema> but it is going to be tight
16:24:33 <gema> hence the "hope"
16:24:45 <gema> I have the team ready to manually test everything though
16:24:48 <inc0> would be awesome
16:24:50 <gema> to get us through the release
16:24:54 <gema> that's plan B
16:25:04 <inc0> fwiw, once you have cloud up, it's just matter of creating tenant for infra
16:25:11 <gema> the cloud is up
16:25:18 <gema> it is just not very stable atm
16:25:24 <inc0> right
16:25:27 <inc0> that's priority
16:25:31 <gema> I know
16:25:41 <inc0> although you might consider using infra for monitoring/testing
16:25:45 <gema> as soon as I have a week of stability we'll get cracking
16:25:50 <gema> I will ask you about that this week
16:25:53 <inc0> it's not just charity, there is benefit of running nodepool
16:25:57 <gema> maybe we can start hooking it and see how it goes
16:26:04 <gema> before we schedule anything on it
16:26:14 <inc0> constant bashing of your cloud and good telemetry that comes with it is worthwhile
16:26:24 <gema> yep
16:26:41 <hrw> and termite
16:26:45 <gema> as soon as hrw doesn't swear when trying to use it, we are ready
16:27:00 <gema> that's my readiness sensor
16:27:09 <inc0> #link http://grafana.openstack.org/dashboard/db/nodepool-ovh
16:27:38 <inc0> this is what you get ootb if infra sets up nodepool there
16:27:56 <inc0> but yeah it doesn't make sense if cloud is just broken
16:28:22 <gema> it's not broken, just undergoing renovations x) almost there, will keep you guys posted
16:28:33 <inc0> cool:)
16:28:51 <inc0> #topic post-summit discussion
16:29:11 <inc0> soo I second gema's quesiton, anything interesting out there?:)
16:30:13 <gema> rwellum: ?
16:30:57 <inc0> I guess they're still healing up jetlag;)
16:31:14 <gema> very likely , oh well, we can always ask in the channel
16:31:22 <gema> people probably took some time off to visit
16:31:37 <inc0> right
16:31:51 <inc0> k
16:31:55 <inc0> #topic open discussion
16:32:04 <inc0> anything on your collective minds before we wrap up?
16:32:37 <duonghq> hmm, I just notice that my Keystone upgrade ps is in merge conflict, I'll fix it soon and hope that somebody can review it
16:32:59 <chason> I'd like to bring up the doc work.
16:33:16 <Jeffrey4l> inc0, about the tag name in dockerhub. how about use "master" for master branch?
16:33:33 <inc0> Jeffrey4l: we can have both for master branch
16:33:34 <duonghq> https://review.openstack.org/#/c/398685/ -> the nearly last ps of become series is ready for review, and I'll add after-the-last ps for new  services later
16:33:55 <inc0> you can tag same image multiple times
16:34:04 <Jeffrey4l> inc0, use master is easy when cut new stable branch ;p
16:34:07 <hrw> +1 for master so it is visible that it is development image
16:34:27 <hrw> or s/master/devel/
16:34:28 <inc0> ok, we can do just master
16:35:07 <Jeffrey4l> hrw, whatever. a fix tag name for master branch is better.
16:35:19 <inc0> there are 2 hard things in IT - naming things, cache invalidation and off-by-one errors
16:35:29 <duonghq> I see that our master tag didn't be used a while
16:35:40 <inc0> right
16:36:07 <gema> I'll throw race conditions there for good measure
16:36:21 <gema> ah, it was a joke xDDD
16:36:25 <inc0> on that note, default for opesntack_release in ansible is 5.0.0 or relevant release number, which is not ideal any more
16:36:32 <inc0> but changing defaults is...hard
16:37:01 <inc0> joke with some truth in it;
16:37:29 <Jeffrey4l> after release new rc, which i forgot in last time ;(
16:37:49 <Jeffrey4l> anyway, when we migrate to dockerhub, this is not a issue anymore.
16:38:16 <inc0> ok so let's focus on that:)
16:38:37 <egonzalez> I think keep tags is important too,  no need to change defaults
16:39:03 <inc0> yeah, I guess it's just one conf option
16:39:10 <Jeffrey4l> we should push tags images to dockerhub too.
16:39:54 <inc0> right, but built often as well
16:40:01 <inc0> maybe not daily but idk weekly?
16:40:18 <inc0> might be worth adding new pipeline for weekly builds
16:40:52 <egonzalez> Tags should be immutable and most if passed CI before pushing
16:41:08 <egonzalez> We ensure that works and no deps changed after release
16:41:14 <Jeffrey4l> inc0, maybe. idk too. only built once is also OK, i think.
16:41:16 <inc0> immutable - no
16:41:24 <inc0> code tags yes, but not images
16:41:32 <inc0> you freeze versions of deps
16:41:34 <Jeffrey4l> egonzalez, are you saying code or images?
16:41:40 <inc0> and deps might need upgrades
16:41:57 <hrw> finally added myself to https://wiki.openstack.org/wiki/Kolla#Active_Contributors table. Thx Gema for reminding
16:42:04 <egonzalez> Both
16:42:23 <Jeffrey4l> for images, as inc0 said it is not immutable.
16:42:38 <egonzalez> By tags mean 5.0.0, no daily pike builds
16:42:38 <Jeffrey4l> base on this. i think we should only build tag images once.
16:42:58 <inc0> we can't do this
16:43:08 <inc0> I mean we can build 5.0.0 every week or so
16:43:18 <inc0> but we need to refresh even 5.0.0
16:43:36 <hrw> 5.0.1 you mean?
16:43:43 <inc0> because image is not just code artifact, it's also sum of it's dependencies
16:43:51 <inc0> right, latest tag in 5.*
16:43:52 <Jeffrey4l> hrm, which question are we talking?
16:44:00 <hrw> stable/pike generates 5.0.1
16:44:38 <Jeffrey4l> i think, tag image only build once, branch image build daily.
16:44:40 <hrw> Pike still gets backports. Our team finds issues here and there adding me some work
16:45:03 <egonzalez> +1 Jeffrey4l
16:45:34 <Jeffrey4l> inc0, is this what you want?
16:45:46 <inc0> Jeffrey4l: no, even tag images have to be refreshed
16:45:55 <inc0> even if kolla code or openstack code inside won't change
16:46:03 <Jeffrey4l> inc0, do not agree at this point.
16:46:08 <inc0> let say openssl has a bug
16:46:15 <inc0> super critical CVE
16:46:29 <inc0> gets fixed upstream, new minor version of openssl gets released
16:46:44 <Jeffrey4l> if there are some bug in the tag code, it will become un-usable and we have no change to fix it.
16:46:49 <inc0> unless we rebuild all clouds which use our images are insecure
16:47:16 <Jeffrey4l> inc0, there is a possible, like a new release package crashed all our tag.
16:47:22 <Jeffrey4l> this is always happing.
16:47:27 <Jeffrey4l> happening.
16:47:32 <inc0> well if there is bug in tag code the same bug will be present in original image
16:47:45 <Jeffrey4l> inc0, not code bug.
16:47:46 <inc0> right, but that's why we need CI for images
16:47:56 <Jeffrey4l> inc0, like this https://review.openstack.org/519592
16:47:57 <inc0> and push only if CI passes
16:49:05 <inc0> right, I agree, but I'd rather have broken images with things we can fix than miss CVE tbh
16:49:19 <inc0> alternatively we can just release more often
16:49:24 <inc0> minor versions that is
16:49:42 <inc0> so if we run into critical bug in tag, we can release next tag right away
16:50:05 <Jeffrey4l> in my mind, docker image is just for newbie. if you really want to use kolla in prod env, build and manage your own images. :)
16:50:16 <inc0> it'll make sense because if someone would install kolla and build themselves, without it they'd be broken too
16:50:44 <inc0> right, which means if hit issue you described, these people will be screwed too
16:51:06 <Jeffrey4l> i still agree with egonzalez , tag ( 5.x.x ) images should be immutable.
16:51:06 <inc0> which means we should build often anyway:)
16:51:22 <inc0> but why?
16:51:36 <inc0> I mean, as you said, if we have bug like one you mentioned
16:51:43 <inc0> we need to fix and release fix anyway
16:51:50 <inc0> or people won't be able to build working images
16:52:10 <inc0> so what's the value of keeping stale images in dockerhub?
16:52:11 <Jeffrey4l> inc0, and old tag is death. build them is useless.
16:52:38 <inc0> no, build only latest tag of given release
16:52:50 <inc0> so we build latest 4.* adn 5.*
16:52:55 <Jeffrey4l> latest tag? hrm,
16:53:04 <inc0> no 4.0.0 and 4.0.1 and 4.0.2
16:53:08 <inc0> just 4.0.2
16:53:20 <inc0> we can delete 4.0.0 from dockerhub
16:53:27 <inc0> but we build 4.0.2 every week
16:53:31 <Jeffrey4l> got what you say.
16:53:36 <inc0> even if kolla code to do it won't change
16:53:46 <inc0> it's underlying deps I'm worried about
16:53:55 <inc0> all non-openstack ones
16:54:31 <egonzalez> But new built images may not work with ansible/k8s tagged code
16:54:51 <inc0> egonzalez: right, if that happens, we need to fix and tag asap anyway
16:54:57 <inc0> can't wait for next release window
16:55:07 <egonzalez> Users may need to change to code branchs rather than code tags
16:55:16 <egonzalez> Got it
16:55:19 <inc0> because main usecase as Jeffrey4l mentioned is still self-built images
16:56:38 <inc0> and they're going to be broken
16:56:47 <Jeffrey4l> ok. i think inc0's way is possible.
16:57:25 <Jeffrey4l> and on the other hand, we are far away to implement this :(
16:58:01 <inc0> right
16:58:09 <inc0> we need to fix our current gates first
16:58:15 <inc0> ok guys we're running out of time
16:58:31 <inc0> any last words? (see what I did there?)
16:58:54 <gema> comedian
16:58:54 <inc0> ok, thank you all for coming:)
16:59:04 <inc0> always
16:59:05 <gema> thank you!
16:59:08 <inc0> #endmeeting kolla