14:00:22 <mwhahaha> #startmeeting tripleo
14:00:22 <openstack> Meeting started Tue Sep 26 14:00:22 2017 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:22 <mwhahaha> #topic agenda
14:00:23 <mwhahaha> * review past action items
14:00:23 <mwhahaha> * one off agenda items
14:00:23 <mwhahaha> * bugs & blueprints
14:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:24 <mwhahaha> * Projects releases or stable backports
14:00:24 <mwhahaha> * CI
14:00:24 <mwhahaha> * Specs
14:00:24 <mwhahaha> * open discussion
14:00:25 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:00:26 <openstack> The meeting name has been set to 'tripleo'
14:00:35 <mwhahaha> Hi everyone! who is around today?
14:00:38 <trown> o/
14:00:38 <slagle> Hi
14:00:47 <cdearborn_> \o
14:00:47 <jrist> o/
14:00:52 <jrist> two meetings at once!
14:00:59 <fultonj> o/
14:01:01 <thrash> o/
14:01:02 <ccamacho> hi!
14:01:09 <mwhahaha> multitask!
14:01:14 <mwhahaha> at least it's not 3
14:01:14 <owalsh> o/
14:01:15 <abishop> o/
14:01:15 <chem`> o/
14:01:35 <jpich> o/
14:01:40 <EmilienM> o/
14:01:42 <jfrancoa> o/
14:02:06 <oidgar> o/
14:02:29 <mwhahaha> ok lets go
14:02:29 <atoth> o/
14:02:33 <marios> o/
14:02:34 <mwhahaha> #topic review past action items
14:02:39 <mwhahaha> shardy to look at how to reduce # of services deployed on ovb (continued)
14:02:59 <mwhahaha> he's not here, so i guess we'll postpone that one again
14:03:03 <matbu> o/
14:03:10 <mwhahaha> #action shardy to look at how to reduce # of services deployed on ovb (continued)
14:03:22 <mwhahaha> That's it on the previous action items
14:03:29 <mwhahaha> #topic one off agenda items
14:03:35 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:03:52 <gchamoul> o/
14:04:07 <mwhahaha> Newton EOL & upstream support phases
14:04:17 <mwhahaha> So newton is going EOL next month
14:04:30 <EmilienM> ouch
14:04:32 <mwhahaha> so FYI if you're trying to propose anything to newton, it's unlikely that we should be merging anything
14:04:54 <mwhahaha> Additionally the release team is pushing back on some of the items we are backporting because they are not following the support phases
14:04:56 <mwhahaha> #link https://docs.openstack.org/project-team-guide/stable-branches.html#support-phases
14:04:58 <marios> mwhahaha: we have an upgrades related stable/newton one sec fetching link
14:05:12 <EmilienM> in my experience we're kind of free to EOL when we want, in a reasonable timeframe
14:05:20 <EmilienM> e.g. a few weeks after
14:05:27 <mwhahaha> please be aware that when we backporting stuff we need to make sure they are appropriately critical
14:05:43 <marios> mwhahaha: #link https://review.openstack.org/#/c/474967/ reviews please so we can land this in time for Newton EOL ?
14:05:46 * EmilienM hides
14:05:48 <mwhahaha> we follow stable-policy starting in Ocata so we need to be using their policy
14:05:54 <chem> mwhahaha: we have this one that should gets in https://review.openstack.org/#/c/507188/
14:06:04 <mwhahaha> marios: k i'll take a look
14:06:06 <chem> mwhahaha: currently testing that it's correct
14:06:26 <mwhahaha> chem i'll have to review that one
14:06:30 <EmilienM> we should get a gerrit query to review them all
14:06:48 <EmilienM> (abandon these who won't make it and highlight the ones we need)
14:06:54 <chem> mwhahaha: it fixes a "regression"
14:07:15 <mwhahaha> chem: yea but this late in the support cycle "regession" isn't necessarily what we should be fixing
14:08:10 <EmilienM> FWIW, my gertty says we have 14 patches in newton
14:08:26 <mwhahaha> ok let's take a look at them this week
14:08:36 <EmilienM> #action review newton backports in gerrit
14:08:52 <mwhahaha> sounds good
14:08:54 <EmilienM> mwhahaha: I'll help on that front
14:09:10 <mwhahaha> so any other open items folks would like to talk about?
14:09:45 <mwhahaha> k moving on
14:09:50 <mwhahaha> #topic bugs & blueprints
14:09:50 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/queens-1
14:09:50 <mwhahaha> We currently have 60 blueprints and about 470 open bugs. Please take some time to review your blueprint status and make sure it is properly up to date.
14:10:14 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1718387
14:10:16 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1719373
14:10:17 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1719525
14:10:17 <openstack> Launchpad bug 1718387 in tripleo "ping test is periodically failing for the gate-tripleo-ci-centos-7-nonha-multinode-oooq " [Critical,Triaged] - Assigned to Marios Andreou (marios-b)
14:10:18 <openstack> Launchpad bug 1719373 in tripleo "tripleo ocata ceilometer-db-upgrade fails, as gnocchi cant reach swift" [Critical,Triaged] - Assigned to Pradeep Kilambi (pkilambi)
14:10:19 <openstack> Launchpad bug 1719525 in tripleo "CI: scenario002 periodic job fails with Zaqar error "404 not found" in undercloud" [Critical,Triaged]
14:10:29 <mwhahaha> Anyone have any additional bugs they want to raise?
14:10:48 <EmilienM> do we need to discuss about the alerts?
14:10:56 <EmilienM> sounds like one is not assigned
14:11:01 <marios> for 1718387 i posted an elastic recheck query so we could see how frequent it is https://review.openstack.org/505574 but didn't get attention yet
14:11:28 <mwhahaha> isn't 1719525 a dupe of the swift one from 2 weeks ago
14:11:30 <sshnaidm> this one: https://bugs.launchpad.net/tripleo/+bug/1719525 I opened today, didn't look into it too much..
14:11:31 <openstack> Launchpad bug 1719525 in tripleo "CI: scenario002 periodic job fails with Zaqar error "404 not found" in undercloud" [Critical,Triaged]
14:11:39 <sshnaidm> mwhahaha, ok
14:11:45 <EmilienM> and the 2 others "Triaged" but not "In Progress"
14:11:53 <mwhahaha> that's the swift issue that therve fixed for master (i think the backport was pending)
14:11:59 <therve> mwhahaha, Yes I think so too
14:12:10 <sshnaidm> mwhahaha, therve so it's back?
14:12:19 <therve> sshnaidm, Not fixed in pikee
14:12:21 <therve> pike
14:12:21 <EmilienM> marios: how often 1718387 is hit?
14:12:26 <therve> https://review.openstack.org/#/c/505700/
14:12:41 <marios> i filed 1718387 last week as i wasn't sure how often it was happening (it passed on recheck for the change I saw it on originally) EmilienM exactly i don't know thats why i posted the query
14:12:53 <EmilienM> marios: you probably want to ping pabelanger about the query
14:12:54 <mwhahaha> sshnaidm: was that a pike or master periodic job?
14:13:20 <sshnaidm> mwhahaha, sorry, it was master gate job
14:13:26 <sshnaidm> just fixed the name
14:13:32 <marios> EmilienM: ack will do (the change i wanted to land, and where i first saw it and filed the bug, passed on the next run, so it isn't reliable/wasn't at least)
14:13:33 <sshnaidm> not periodic
14:13:40 <mwhahaha> therve: can you take a look again
14:13:42 <sshnaidm> therve, it's master
14:14:00 <therve> Urg
14:14:04 <therve> I guess
14:14:51 <mwhahaha> k any other bugs?
14:15:03 <mwhahaha> perhaps some not related to CI as we have a section for that
14:15:37 <mwhahaha> ok moving on
14:15:38 <mwhahaha> #topic projects releases or stable backports
14:15:54 <mwhahaha> EmilienM is working on getting stable releases out for newton/ocata/pike
14:16:06 <EmilienM> mwhahaha: I just remember that shardy is off today for OpenStackDaysUK, he's presenting TripleO !
14:16:20 <mwhahaha> anyone have any additional backports they want to raise (besides the previously mentioned newton one)
14:16:24 <EmilienM> mwhahaha: yes I'm working on it, and I made a mistake to approve https://review.openstack.org/#/c/492245/
14:16:37 <EmilienM> mwhahaha: it sounds like a feature and we merged it
14:16:49 <mwhahaha> EmilienM: so we weren't following stable policy in newton
14:16:51 <EmilienM> but our drivers keep pushing things in stable branches, it's hard to say no
14:17:00 <EmilienM> mwhahaha: right, I made a mistake
14:17:12 <mwhahaha> EmilienM: or is that something we can point out to the release team for newton
14:17:32 <EmilienM> mwhahaha: yeah I'm already with doug on #openstack-release, trying to figure that out
14:17:36 <mwhahaha> k
14:17:41 <EmilienM> I think it's fine this time
14:17:50 <EmilienM> but in the future I (we) need to be more careful
14:17:56 <mwhahaha> this is something we need to keep an eye on for >=ocata
14:18:17 <EmilienM> yes
14:19:29 <mwhahaha> any other release/backport stuff?
14:20:00 <atoth> one question on it
14:20:03 <EmilienM> (done for me)
14:20:26 <amoralej> we need https://review.openstack.org/#/c/507495/ to unblock ocata promotion
14:20:39 <atoth> thedid you do anything that created the backport for the lbaas bug fix or was that done automatically somehow?
14:20:49 <matbu> i'll need to backport in pike those two reviews (when will be merged) : https://review.openstack.org/#/q/topic:minor-update-workflow+(status:open+OR+status:merged)
14:21:02 <mwhahaha> amoralej: there's already a patch for that
14:21:22 <mwhahaha> amoralej: i thought that was already merged
14:21:23 <amoralej> mwhahaha, that's a cherry pick from patch in master
14:21:34 <mwhahaha> amoralej: i rolled it in to the backport let me see
14:22:04 <owalsh> amoralej: IIRC *something* in nova or libvirt uses machine-id
14:22:23 <slagle> matbu: nice progress on moving the logic to tripleo-common
14:22:26 <amoralej> mwhahaha, yes, it's in https://review.openstack.org/#/c/489618/
14:22:50 <mwhahaha> amoralej: i thought i fixed it in the backport but oh well +A
14:22:52 <EmilienM> slagle: is that something we should backport to pike?
14:23:18 <EmilienM> I'm afraid we don't have the choice for containerized deployments?
14:23:34 <amoralej> mwhahaha, i don't see it in https://review.openstack.org/#/c/493684
14:23:36 <matbu> slagle: thanks, i just need to fix the UTs and its sound like its working good according to folks from upgrade dfg
14:23:41 <slagle> EmilienM: i believe we have to? marios/ matbu can confirm
14:23:48 <mwhahaha> amoralej: https://review.openstack.org/#/c/493684/2/elements/remove-machine-id/post-install.d/70-remove-machine-id
14:23:50 <matbu> yep
14:23:53 <mwhahaha> amoralej: see the comment, i rolled it in
14:24:03 <social> slagle: yeah those patches are on right track
14:24:12 <mwhahaha> amoralej: we can pull in that patch (it won't hurt) but that's not the fix
14:24:26 <amoralej> mwhahaha, i don't follow you
14:24:32 <mwhahaha> amoralej: see my comment on that file
14:24:39 <mwhahaha> amoralej: see new file more 100755
14:24:47 <amoralej> the problem is not the mode
14:24:49 <slagle> matbu: fyi, i have soem patches based on cofing download, that i will rebase on top of your updates: https://review.openstack.org/#/q/status:open++branch:master+topic:config-download
14:24:57 <amoralej> but removing the file instead of empty it
14:25:02 <mwhahaha> amoralej: oh yea
14:25:08 <matbu> slagle: ack cool
14:25:10 <mwhahaha> amoralej: sorry thought it was the other patch
14:25:17 <trown> ya and if the file is missing journal cant start
14:25:24 <amoralej> exactly
14:25:35 <owalsh> amoralej: nova will not like an empty machine-id - https://github.com/openstack/nova/blob/328eb58a4a408706c75ba7493df24f6dcc472bb2/nova/virt/libvirt/driver.py#L3826
14:25:41 <amoralej> and it seems that if it doesn't exist systemd doesn't recreate it properly
14:25:55 <mwhahaha> amoralej: yea, pesky systemd
14:26:04 <amoralej> :)
14:26:12 <mwhahaha> owalsh: systemd pukes if the file is missing but if it's empty it'll populate it
14:26:26 <mwhahaha> owalsh: which is prior to the nova startup
14:26:31 <amoralej> yes
14:26:44 <owalsh> mwhahaha: ack, ok. maybe that's the problem I'm remembering :-)
14:27:03 <mwhahaha> yea it's a really weird interaction
14:27:21 <amoralej> mwhahaha, i'll propose the cherry-pick to newton also, as the element is being added there too
14:27:21 <mwhahaha> we'll need to also backport that to newton i think as well but that's on my plate
14:27:42 <mwhahaha> amoralej: ok please do
14:27:47 <trown> thanks amoralej for tracking that down!
14:28:12 <trown> was filed as a quickstart bug not collecting /var/log/messages and amoralej was very fast to find the actual cause
14:28:46 <EmilienM> amoralej: thanks!
14:28:47 <mwhahaha> ok any other backport things?
14:30:28 <mwhahaha> amoralej: https://review.openstack.org/#/c/505305/ is the previous patch for newton (is not merged yet). i guess we can just abandon that and i'll downstream it since newton will be EOL soon
14:30:35 <mwhahaha> #topic CI
14:30:49 <mwhahaha> how's CI today?
14:31:30 <amoralej> mwhahaha, ok, no problem from my sid
14:31:38 <amoralej> just remember to abandon https://review.openstack.org/#/c/505310 also
14:31:41 <weshay> <joke> A man walks into a ci system
14:31:50 <amoralej> what was the bug fixed by that?
14:31:53 <weshay> bnemec++
14:32:02 <weshay> fixed a lot of the timeouts
14:32:21 <mwhahaha> amoralej: it breaks a bunch of stuff because all the machine-ids are identical for all nodes
14:32:30 <mwhahaha> amoralej: it was an really old customer bug from like rhos7
14:33:05 <mwhahaha> amoralej: with some angry folks asking why it took 2+ years to clear a file
14:33:15 <amoralej> ok, then we could do the three backports, it shouldn't be hard
14:33:35 <mwhahaha> amoralej: i was only planning to backport to 10. i'll handle it
14:33:43 <openstackgerrit> Alfredo Moralejo proposed openstack/tripleo-puppet-elements stable/newton: Set remove-machine-id to executable  https://review.openstack.org/507561
14:33:51 <amoralej> mwhahaha, ^ the missing one
14:33:54 <mwhahaha> k
14:34:00 <mwhahaha> ok any other CI related problems?
14:34:53 <mwhahaha> ok moving on
14:34:54 <gchamoul> no big blocker today AFAIK
14:35:01 <mwhahaha> #topic specs
14:35:01 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:35:11 <mwhahaha> please take some time to review queens specs
14:35:18 <mwhahaha> also if you have any old specs, please retarget
14:35:32 <mwhahaha> anyone want to highlight a specific spec?
14:36:45 <mwhahaha> ok moving on
14:36:46 <mwhahaha> #topic open discussion
14:37:13 <mwhahaha> speak now or forever (until next week) hold your peace
14:37:19 <EmilienM> :-O
14:37:38 <EmilienM> (except on Fridays :-P)
14:38:28 <mwhahaha> sounds like that's it for the week
14:38:40 <mwhahaha> thanks everyone
14:38:43 <mwhahaha> #endmeeting