14:00:29 <mwhahaha> #startmeeting tripleo
14:00:29 <mwhahaha> #topic agenda
14:00:29 <mwhahaha> * review past action items
14:00:29 <mwhahaha> * one off agenda items
14:00:29 <mwhahaha> * bugs
14:00:29 <mwhahaha> * Projects releases or stable backports
14:00:29 <mwhahaha> * CI
14:00:30 <mwhahaha> * Specs
14:00:30 <openstack> Meeting started Tue Sep 19 14:00:29 2017 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:30 <mwhahaha> * open discussion
14:00:31 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:00:31 <mwhahaha> Hi everyone! who is around today?
14:00:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:34 <openstack> The meeting name has been set to 'tripleo'
14:00:35 <beagles> o/
14:00:43 <shardy> o/
14:00:43 <fultonj> o/
14:00:45 <jpich> o/
14:00:49 <slagle> hi
14:00:51 <atoth> o/
14:00:54 <marios> o/
14:01:03 <oidgar> o/
14:01:04 <jfrancoa> o/
14:01:37 <rasca> o/
14:01:54 <trown> o/
14:01:54 <cdearborn> o/
14:01:57 <gfidente> o/
14:02:17 <mwhahaha> #topic review past action items
14:02:18 <openstackgerrit> Janki Chhatbar proposed openstack/tripleo-common master: Add healthcheck for ODL container  https://review.openstack.org/504864
14:02:23 <mwhahaha> drop alert on Bug 1713832 and ping storage/zaqar people to investigate
14:02:25 <openstack> bug 1713832 in tripleo "Object PUT failed for zaqar_subscription" [Critical,In progress] https://launchpad.net/bugs/1713832 - Assigned to Marios Andreou (marios-b)
14:02:35 <mwhahaha> so we dropped the alert but I think it's still open
14:02:55 <mwhahaha> the last update points to https://bugs.launchpad.net/swift/+bug/1715177
14:02:56 <openstack> Launchpad bug 1715177 in OpenStack Object Storage (swift) "Container cache management is racy" [Medium,Confirmed]
14:02:58 <marios> mwhahaha: yeah i had a look and therve had a lead at some point but we still didn't find the root
14:03:10 <marios> mwhahaha: i can revisit though i didn't look there all of last week
14:03:16 <matbu> o/
14:03:32 <mwhahaha> marios: it looks like it's a swift problem so we should poke the storage folks
14:03:32 <marios> mwhahaha: is it still relatively infrequent? (1/day?) /me fetches elastic recheck query
14:03:45 <mwhahaha> yea i don't think it's happening as much as it used to
14:03:56 <chem> o/
14:03:57 <marios> http://status.openstack.org/elastic-recheck/#1713832 this one
14:04:41 <mwhahaha> looks like it's still a problem so we just need to track it in swift
14:04:45 <mwhahaha> moving on to the next item
14:04:51 <mwhahaha> shardy to look into version impacts in ci jobs related to Bug 1714361
14:04:52 <openstack> bug 1714361 in tripleo "Mistral on gates is old and does not have the required patches" [Critical,Fix released] https://launchpad.net/bugs/1714361 - Assigned to Alex Schultz (alex-schultz)
14:05:08 <mwhahaha> shardy: i think we fixed that with the priority on repos
14:05:12 <shardy> mwhahaha: Yeah I think we did that
14:05:37 <mwhahaha> mwhahaha to retarget rc2 bugs if not release critical - i believe this was done but I will double check today
14:05:43 <mwhahaha> shardy to look at how to reduce # of services deployed on ovb
14:05:51 <therve> mwhahaha, I found the issue, but it's stuck lacking tests
14:06:10 <mwhahaha> therve: ok let me know if there's anything we can do to help
14:06:14 <shardy> mwhahaha: I made a start on that but need to revisit
14:06:21 <mwhahaha> shardy: ok
14:06:31 <mwhahaha> #action shardy to look at how to reduce # of services deployed on ovb (continued)
14:06:31 <therve> mwhahaha, https://review.openstack.org/500978
14:06:46 <shardy> #link https://review.openstack.org/#/c/500942/ WIP ovb testing
14:07:11 <mwhahaha> that's it on the past action items
14:07:17 <therve> mwhahaha, Maybe they can relax the constraints so that we merge the fix
14:07:57 <mwhahaha> therve: since that's the upstream swift stuff they'd have to sign off on that unfortunately.
14:08:07 <therve> Yeah
14:08:15 <shardy> cschwede: ^^
14:08:30 <therve> mwhahaha, We could disable the cache middleware in swift, but it might makes things slow(er)
14:08:49 <mwhahaha> therve: since this is on the undercloud, how much perf impact?
14:09:04 <therve> mwhahaha, Hard to know really...
14:09:11 <therve> We use quite a bit between the templates and zaqar
14:09:14 <jtomasek> o/
14:09:38 <cschwede> shardy: will have another look later today, thx!
14:09:45 <adarazs> o/
14:10:07 <mwhahaha> ok moving on
14:10:09 <mwhahaha> #topic one off agenda items
14:10:09 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:10:14 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1716914
14:10:15 <openstack> Launchpad bug 1716914 in tripleo "CI: all periodic jobs fail to install undercloud: No resource and no name in property hash in nova_manage instance" [Critical,Triaged]
14:10:15 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1717274
14:10:16 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1717279
14:10:16 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1717545
14:10:17 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1717959
14:10:18 <openstack> Launchpad bug 1717274 in tripleo "ocata2pike: broken package dependency during upgrade" [Critical,In progress] - Assigned to wes hayutin (weshayutin)
14:10:20 <openstack> Launchpad bug 1717279 in tripleo "RDO registry denies images uploads" [Critical,Triaged]
14:10:21 <openstack> Launchpad bug 1717545 in tripleo "nova populates incomplete cell_mappings" [Critical,Triaged]
14:10:22 <openstack> Launchpad bug 1717959 in tripleo "RDO upstream-centos-7 nodepool image is out of date" [Critical,Triaged] - Assigned to Alan Pevec (apevec)
14:10:22 <openstackgerrit> Michele Baldessari proposed openstack/puppet-pacemaker master: Parse pcs auth output with all pcs versions  https://review.openstack.org/505284
14:10:30 <tdasilva> mwhahaha, shardy: regarding the swift issue, couldn't the client be modified to make sure to create the container first?
14:10:54 <marios> mwhahaha: for https://bugs.launchpad.net/tripleo/+bug/1717274 weshay posted https://review.openstack.org/#/c/505268/
14:10:59 <therve> tdasilva, It does create the container first :)
14:11:19 <mwhahaha> marios: ok i'll take a look after the meeting
14:11:25 <marios> mwhahaha: seems to just be missing base repo for the ethtool package which is a dependency (yum update fails)
14:11:34 <therve> tdasilva, I didn't get a nice workaround for it, but I'm opened to suggestions
14:11:50 <marios> mwhahaha: thanks (kudos jpena who called that last week already see comment #2 )
14:12:03 <tdasilva> therve: can you share the client side code?
14:12:32 <therve> tdasilva, https://github.com/openstack/zaqar/blob/master/zaqar/storage/swift/utils.py#L45
14:14:09 <mwhahaha> moving on
14:14:13 <mwhahaha> #topic bugs
14:14:13 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/queens-1
14:14:32 <mwhahaha> we're into queens so please take a look at the bugs since we're going for stability this cycle
14:15:03 <mwhahaha> any specific bugs people wish to talk about (besides that upgrade bug)
14:16:56 <mwhahaha> i'll take that as a no
14:17:03 <mwhahaha> #topic projects releases or stable backports
14:17:30 <mwhahaha> Anyoen have any backports they need reviews on?
14:18:22 <trown> is infra still hosed wrt releases?
14:18:30 <mwhahaha> i believe so
14:18:37 <mwhahaha> i saw mention of that this morning
14:18:47 <tdasilva> therve: ack, taking a look
14:19:18 <mwhahaha> speaking of things that are hosed
14:19:19 <mwhahaha> #topic CI
14:19:28 <trown> lol
14:19:45 <mwhahaha> the puppet-nova fix for the nova_manage command is in the gate
14:20:04 <mwhahaha> so that should merge soon (~30 mins)
14:20:13 <mwhahaha> so that should clear up some of the failures
14:20:21 <mwhahaha> any other outstanding issues that folks can talk about?
14:20:56 <trown> only other issue is in the periodic pipeline we are setting up on rdo infra
14:21:20 <mwhahaha> i saw there some bugs around images in the pipeline
14:21:23 <mwhahaha> has that been addressed?
14:21:38 <trown> needs a new infra image, but that is blocked by some new project that was created without a repo
14:21:50 <trown> https://git.openstack.org/cgit/openstack/networking-lagopus
14:21:59 <mwhahaha> nice
14:22:19 <trown> when that resolves we can rebuild the image in rdoproject and hopefully get the pipeline working
14:23:20 <mwhahaha> any other CI items?
14:24:17 <mwhahaha> sounds like no
14:24:23 <mwhahaha> #topic specs
14:24:23 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:24:31 <mwhahaha> please take some time to review any open specs
14:24:44 <mwhahaha> also retarget specs that might still be pointed to pike
14:25:18 <mwhahaha> additionally if there were any larger items from the PTG that probably should have a spec, now would be the time to propose it
14:25:38 <mwhahaha> anyone have a specific spec they would like to point out?
14:28:13 <mwhahaha> moving on
14:28:14 <mwhahaha> #topic open discussion
14:28:16 <mwhahaha> anything else?
14:28:20 <mwhahaha> if not i'll close in 2 mins
14:29:14 <mnaser> thoughts on moving logic of containerized deployments up to the puppet modules for the services specifically?
14:29:34 <mwhahaha> probably not
14:29:54 <mwhahaha> at the moment we only really using the providers for the config generation
14:30:11 <mnaser> so the actual docker container is created/deployed without puppet?
14:30:39 <mwhahaha> yea we're using kolla for the container build process but puppet for the config of the containers
14:31:11 <mwhahaha> and then we have a tool for running the containers themselves
14:31:14 <mnaser> mmkay.  i was hoping we can work together so to have service_name => docker type of thing
14:31:41 <mnaser> i wanna see what a poc looks like first but it was just an idea :>
14:31:58 <mwhahaha> that might simplify what we have
14:32:06 <mwhahaha> but we don't actually use the serivce items anymore
14:33:07 <mwhahaha> https://github.com/openstack/tripleo-heat-templates/blob/b5c18ded6a6122053fb3a6557c063a44a90e41b9/docker/docker-puppet.py
14:33:32 <shardy> mnaser: https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml shows the overview of our deploy steps now
14:33:58 <shardy> we use ansible to drive a tool called paunch, which consumes some json files and starts docker containers
14:34:20 <shardy> so it's kind of moving in the direction of only using puppet for config generation at this point
14:34:46 <shardy> (it is also used for some bootstrapping)
14:35:05 <shardy> mnaser: always good to discuss any ideas tho :)
14:35:41 <mnaser> i think containers are here to stay (kolla or whatever) but there is many ways of orchestrating them, given you have a ton of existing puppet infra, it might be easier than you having to maintain one more tool
14:36:33 <mnaser> i'm sure it'll present a few challenges but i plan to take sometime to do a poc of a simpler puppet service and then show it and see what it looks like (and gather feedback)
14:36:59 <mnaser> ideally i'd love to see it adopted by tripleo so catering to that use case and avoiding duplicating effort would be nice :>
14:37:06 <shardy> mnaser: yeah, understood - in this case paunch was derived from some code we already had in the heat docker-cmd hook
14:37:30 <shardy> mnaser: seeing the poc would be good, but it does kinda sound like a different direction than where we're headed at this point
14:37:59 <mnaser> gotcha, i'll take feedback anyways "this is where we shot ourself in the foot when we were taking this approach" is helpful :)
14:39:53 <mwhahaha> good questions
14:39:55 <mwhahaha> anything else?
14:41:15 <mwhahaha> ok thanks everyone
14:41:17 <mwhahaha> #endmeeting