14:00:37 <EmilienM> #startmeeting tripleo
14:00:41 <openstack> Meeting started Tue Jun  6 14:00:37 2017 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:46 <EmilienM> #topic agenda
14:00:46 <openstack> The meeting name has been set to 'tripleo'
14:00:50 <jrist> o/
14:00:53 <EmilienM> * review past action items
14:00:55 <EmilienM> * one off agenda items
14:00:57 <mwhahaha> hola
14:00:57 <EmilienM> * bugs
14:00:59 <EmilienM> * Projects releases or stable backports
14:01:01 <EmilienM> * CI
14:01:03 <EmilienM> * Specs
14:01:05 <EmilienM> * open discussion
14:01:07 <EmilienM> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:01:09 <EmilienM> Hi everyone! who is around today?
14:01:18 <d0ugal> o/
14:01:30 <sshnaidm> o\
14:01:32 <cdearborn> \0
14:01:36 <jpich> o/
14:01:39 <akrivoka> hi o/
14:01:41 <marios> hello tripleowls
14:01:50 <panda> ii
14:01:55 <ccamacho> o/
14:02:10 <shardy> o/
14:02:15 <radeks> o/
14:02:20 <EmilienM> #topic review past action items
14:02:34 <EmilienM> (postponed) rasca follow up tripleo-quickstart-utils discussion on ML
14:02:41 <EmilienM> I see rasca sent out an email today
14:02:53 <rasca> o/
14:02:56 <adarazs> o/
14:03:04 <weshay> 0/
14:03:09 <lyarwood_> o/
14:03:14 <EmilienM> (I haven't looked at it yet, still catching up emails)
14:03:21 <janki> o/
14:03:37 <EmilienM> if folks interested by this topic could participate in the thread it would be great
14:03:39 <rasca> EmilienM, no prob, it's something we can discuss inside the thread
14:03:45 <EmilienM> rasca: yeah, thanks
14:03:55 <EmilienM> #topic one off agenda items
14:04:01 <rasca> EmilienM, and sorry again for being late in the answer
14:04:01 <fultonj> o/
14:04:01 <EmilienM> ccamacho: go ahead
14:04:07 <ccamacho> hey EmilienM, mm didn't know how to ask correctly, ill try. There are some folks interested in know better how the upstream CI is working /me among them (In my case the upgrades jobs) in order to help making it greener.
14:04:08 <EmilienM> rasca: no problem at all
14:04:25 <trown> o/
14:04:32 <jtomasek> o/
14:04:37 <EmilienM> ccamacho: you want a deep dive about tripleo CI?
14:04:47 <ccamacho> EmilienM that would be awesome
14:04:48 <florianf> o/
14:04:48 <bogdando> o/
14:04:51 <ccamacho> but at first
14:04:53 <jistr> o/
14:04:55 <EmilienM> I think we've already had some deep dives about things more or less related
14:04:59 <ccamacho> to at least know were to see
14:04:59 <EmilienM> we might want to be more specific maybe
14:05:27 <ccamacho> since it moved to quickstart for me its been a little more tricky to debug.
14:05:47 <adarazs> ccamacho: did you watch panda's quickstart deep dive?
14:05:55 <adarazs> that's sort of what you're asking here.
14:06:13 <dtrainor> hi
14:06:14 <adarazs> of course the CI system is a lot more, but that's pretty much the starting point for it.
14:06:38 <ccamacho> adarazs, this one you mean? http://www.anstack.com/blog/2017/05/05/tripleo-deep-dive-session-9.html
14:07:03 <EmilienM> ccamacho: yes this one
14:07:03 <adarazs> yep, that looks like it.
14:07:26 <trown> also upgrades jobs are still of the old type
14:07:27 <marios> ccamacho: so fwiw i don't think it can hurt to have a deepdive on ci/latest state of things, assuming you get a volunteer comfortable enough to run it. i know jistr is going to talk about containers and possibly ci related to that.
14:07:36 <sshnaidm> ccamacho, maybe it's better to focus on specific CI jobs, i.e. upgrades?
14:07:48 <marios> someone already mentioned panda but i know he'd hate me to say his name again so i wont ;)
14:07:48 <EmilienM> marios: i'm ok to run it, but we need more context
14:07:48 <shardy> yeah the baremetal upgrade job is still the old script, but the new upgrade-to-containers one uses quickstart IIRC
14:07:55 <jistr> yea in the containers deep dive, i want to have a part about where to look in the CI logs
14:08:01 <jistr> and how the container CI was done
14:08:04 <trown> nice jistr
14:08:04 <ccamacho> sshnaidm for me it would be awesome to focus on the upgrades jobs
14:08:11 <sshnaidm> I'm +1 to have deep dive about upgrades
14:08:12 <adarazs> jistr: \o/
14:08:12 <EmilienM> jistr: ++
14:08:14 <jistr> see https://etherpad.openstack.org/p/tripleo-deep-dive-containers
14:08:26 <jistr> feel free to add more suggestions at the bottom
14:08:30 <jistr> i've already addressed some
14:08:33 <ccamacho> thanks!
14:08:40 <jistr> if it grows too much we can split it into 2 deep dives
14:08:46 <EmilienM> jistr: excellent idea
14:09:16 <ccamacho> thanks jistr Ill look it :)
14:09:18 <jistr> we'd probably do this next week (15th) if that's ok
14:09:28 <EmilienM> ccamacho: please have a look at what jistr is preparing and if some things are missing, please dress a list and we'll see if we can schedule another deep dive
14:09:34 <sshnaidm> maybe after containers deep dives it will be great to have upgrades deep dive :)
14:09:35 <EmilienM> ccamacho: also, don't hesitate to be very specific
14:09:40 <matbu> o/
14:09:42 <weshay> I was thinking to eventually make the ocata -> pike containerized upgrade gating.. and make an experimental non-containerized -> non-containerized upgrade for ocata to pike in experimental
14:09:45 <ccamacho> sure thing EmilienM thanks!
14:10:19 <EmilienM> weshay: maybe a bit out of context now, isn't?
14:10:27 <weshay> sorry..
14:10:37 <EmilienM> ccamacho: I'll let you keep track on this topic
14:10:46 <EmilienM> ccamacho: and see if some topics haven't been covered by deep dive
14:10:55 <EmilienM> ccamacho: thanks for bringing this up, it's important to share knowledge
14:11:34 <EmilienM> bogdando: go ahead
14:11:38 <bogdando> hi folks
14:12:06 <bogdando> I'm seeking for COW solutions to de-duplicated the space consumed by docker images, in cache and in registry as well
14:12:43 <bogdando> nothing interesting googled out, asked for internal openshift ML as well. So, I'll need some help from experienced folks
14:13:16 <EmilienM> have you kicked off some discussion on openstack-dev ML?
14:13:21 <bogdando> and also wondering if that would be applicable for CI & nodepool images at all
14:13:28 <EmilienM> I'm pretty sure this is a cross-project topic
14:13:29 <bogdando> EmilienM: not yet, will do
14:13:50 <trown> seems applicable to CI since we are already running out of space on the undercloud
14:14:13 <bogdando> trown: yeah, I mean the btrfs partitions for VMs
14:14:25 <bogdando> if assumed they could help us
14:14:32 <shardy> when I discussed this with mandre last week, I wondered why we're populating the undercloud registry for CI jobs, vs just pulling from dockerhub and relying on infra caching?
14:14:47 <shardy> e.g if we did that we wouldn't use any additional undercloud space?
14:14:57 <shardy> does anyone know if that was looked at?
14:15:10 <bogdando> shardy: even if we don't deploy the registry, there is still room to de-duplicate kolla images fetched
14:15:36 <bogdando> if I do docker save | gzip -6, the resulting tarball is <3G!
14:15:40 <shardy> bogdando: agreed, I'm sure we can optimize, I was commenting re running out of space on the undercloud
14:15:41 <bogdando> < 3G
14:15:47 * beagles wanders in late
14:16:08 <bogdando> I mean for all images, like 30+
14:16:40 <bogdando> there is not much info wrt btrfs online de-duplication and COW and docker , sadly
14:17:02 <bogdando> so I wanted to research. That's it from my side
14:17:50 <EmilienM> I suggest to follow-up, maybe on the mailing-list if you don't mind
14:17:55 <bogdando> okay
14:18:14 <EmilienM> #topic bugs
14:18:24 <EmilienM> #link https://launchpad.net/tripleo/+milestone/pike-2
14:18:37 <EmilienM> #link https://launchpad.net/tripleo/+milestone/pike-3
14:18:44 <EmilienM> do we have outstanding bugs to discuss this week?
14:19:33 <beagles> I've one on my plate https://bugs.launchpad.net/tripleo/+bug/1695885
14:19:33 <openstack> Launchpad bug 1695885 in tripleo "nf_conntrack_proto_sctp is not compiled as a module in newer RHEL release" [Critical,Triaged] - Assigned to Brent Eagles (beagles)
14:19:55 <EmilienM> Critical?
14:20:09 <EmilienM> I've seen a tendency of use Critical for a bunch of things
14:20:26 <beagles> have we run into a situation like this before where different kernels have had different build options
14:20:29 <EmilienM> is it blocking CI?
14:20:45 <beagles> EmilienM: yeah, you're right I'll set to high
14:21:08 <beagles> EmilienM: it blocks upcoming distro changes, but doesn't affect the current state of t hings
14:21:13 <EmilienM> yeah, let's use Critical when it's really critical, like you can't deploy anything
14:21:17 <beagles> ack
14:21:31 <EmilienM> for example, promotion blockers are critical
14:21:39 <EmilienM> since we consider promotion important to keep track on trunk
14:21:43 <beagles> gotcha
14:21:52 <EmilienM> but classic bugs are High, unless they affect the whole deployment
14:22:05 <EmilienM> I'll move to the next topic
14:22:08 <EmilienM> #topic projects releases or stable backports
14:22:22 <EmilienM> this week is pike-2 milestone
14:22:51 <EmilienM> except tripleo-ui where I'm already in touch with jtomasek, do we have blockers in tripleo that we need to fix / merge before the m2 release?
14:23:18 <florianf> EmilienM: https://review.openstack.org/#/c/469554/ this is currently breaking validtions when there's no existing stack
14:23:42 <EmilienM> florianf: good to know, added on my review list
14:23:48 <florianf> EmilienM: not sure if this accounts for a blocker, but it should really be merged before m2
14:23:55 <florianf> EmilienM: excellent, thanks!
14:24:02 <EmilienM> florianf: it will merge before
14:24:31 <EmilienM> i'll propose the release tomorrow night or Thursday
14:24:37 <EmilienM> to make sure we can do it before Thursday night
14:25:03 <EmilienM> #topic CI
14:25:24 <EmilienM> right now, the main blocker to promote tripleo CI is https://bugs.launchpad.net/tripleo/+bug/1696094
14:25:25 <openstack> Launchpad bug 1696094 in tripleo "CI: ovb-ha promotion job fails with 504 gateway timeout" [Critical,Triaged]
14:25:38 <EmilienM> networking folks are working on it afik
14:25:50 <EmilienM> other than that, it seems like we're close to have CI promotion
14:26:18 <adarazs> yay, I hope that happens today. :S
14:26:23 <EmilienM> is there anything CI-related thing that we need discuss this week, beside what is being discussed in the tripleo ci squad meeting?
14:26:49 <EmilienM> adarazs: I hope too, 16 days without promotion is bad :(
14:27:01 <EmilienM> moving to the next topic
14:27:01 <sshnaidm> containers job have problems sometimes to pull docker images from dockerhub
14:27:04 <EmilienM> #topic specs
14:27:10 <EmilienM> #undo
14:27:11 <openstack> Removing item from minutes: #topic specs
14:27:28 <sshnaidm> tonight it happened about 10 times, it's always some other image
14:27:29 <EmilienM> sshnaidm: is it reported in launchpad?
14:27:37 <sshnaidm> EmilienM, no, will do
14:27:52 <EmilienM> sshnaidm: don't we use AFS for that yet?
14:28:00 <sshnaidm> just wanted to be sure - do we use AFS??
14:28:15 <EmilienM> I thought dprince (not online) did some work around this thing
14:28:24 <EmilienM> I haven't looked at it recently
14:28:37 <sshnaidm> I don't know too how it's finished..
14:29:00 <sshnaidm> ok, will open LP meanwhile
14:29:14 <shardy> sshnaidm: we should chat with pabelanger as IIRC he did some caching work which we were going to use to avoid this
14:29:26 <EmilienM> that was done: https://review.openstack.org/#/c/454893/
14:29:38 <EmilienM> we use in-rack HTTP docker registry proxy
14:30:11 <EmilienM> sshnaidm: let's follow-up with pabelanger and dprince
14:30:17 <sshnaidm> EmilienM, sure
14:30:29 <shardy> EmilienM: dprince did that for the OVB jobs IIRC but there was an infra solution planned for the multinode container jobs IIRC
14:30:43 <EmilienM> shardy: yeah, it makes sense
14:30:54 <EmilienM> anything else CI related?
14:31:12 <sshnaidm> nope
14:31:18 <EmilienM> #topic specs
14:31:21 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:31:28 <EmilienM> do we have any spec to discuss this week?
14:32:23 <shardy> https://blueprints.launchpad.net/tripleo/+spec/composable-networks
14:32:41 <shardy> I raised that BP to cover some patches I started aiming to j2 render network related parts of t-h-t
14:32:52 <shardy> it's been discussed for a while, trying to move it forward
14:33:03 <shardy> No spec, just a blueprint FYI
14:33:04 <EmilienM> shardy: do you need reviews on the patches?
14:33:40 <shardy> EmilienM: https://review.openstack.org/#/q/status:open+topic:bp/composable-networks
14:34:08 <shardy> I'll get those passing CI and chase for reviews during the week
14:34:13 <EmilienM> cool
14:34:18 <EmilienM> thanks!
14:34:25 <EmilienM> anything else for the specs?
14:35:05 <EmilienM> #topic open discussion
14:35:10 <EmilienM> quick note :
14:35:25 <EmilienM> you probably saw the discussion on openstack-dev about PTG
14:35:39 <EmilienM> we'll keep the same kind of schedule than last time
14:35:54 <EmilienM> where monday / tuesday are for horizontal topics and wednesday / thursday / friday are for vertical
14:35:57 <EmilienM> and tripleo is vertial
14:36:00 <EmilienM> vertical*
14:36:21 <EmilienM> I proposed to do the tripleo sessions between Wednesday and Friday morning, so folks can travel on the afternoon
14:36:47 <EmilienM> the sessions on Friday morning won't be as critical as the ones on Wednesday & Thursday, so folks can also travel on Friday morning or even Thursday night if needed
14:36:59 <EmilienM> how does it sound? Is 2 days enough? Do we need 3 days?
14:37:41 <shardy> I think 2 days with Friday for any follow-up topics is enough
14:37:51 <EmilienM> yeah
14:37:59 <EmilienM> I thought on Friday morning we could do more hands-on sessions
14:38:14 <EmilienM> if folks want to work together or do some hands on training, etc
14:38:20 <shardy> +1 sounds good to me
14:38:26 <EmilienM> cool
14:38:40 <EmilienM> anything feedback on ^ is welcome, on the thread or IRC (to me directly)
14:38:48 <EmilienM> thanks everyone, let's go back to work now ;-)
14:38:51 <EmilienM> #endmeeting