14:00:23 <EmilienM> #startmeeting tripleo 14:00:25 <openstack> Meeting started Tue Aug 1 14:00:23 2017 UTC and is due to finish in 60 minutes. The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:28 <openstack> The meeting name has been set to 'tripleo' 14:00:36 <EmilienM> #topic agenda 14:00:43 <EmilienM> * review past action items 14:00:45 <EmilienM> * one off agenda items 14:00:47 <EmilienM> * bugs 14:00:49 <EmilienM> * Projects releases or stable backports 14:00:51 <EmilienM> * CI 14:00:53 <EmilienM> * Specs 14:00:55 <EmilienM> * open discussion 14:00:57 <EmilienM> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:58 <jrist> o/ 14:00:59 <EmilienM> hello everyone 14:01:04 <rasca> o/ 14:01:04 <mwhahaha> hi2u 14:01:07 <ccamacho> hi folks o/ 14:01:10 <dprince> hi 14:01:15 <jkilpatr> morning 14:01:17 <marios> o/ hello 14:01:21 <openstackgerrit> Jiri Stransky proposed openstack/tripleo-common master: Allow tripleo-admin creation both for Nova-managed and split-stack https://review.openstack.org/489613 14:01:27 <chem> o/ 14:01:45 <numans> hi 14:01:49 <EmilienM> already, thanks for being here an August 1st :-) we're all survivors 14:01:54 <EmilienM> #topic review past action items 14:02:10 <EmilienM> abishop to force legacy key manager before end of pike even if barbican is not deployed 14:02:13 <EmilienM> I think it was done 14:02:18 <EmilienM> or in progress at least 14:02:22 <EmilienM> abishop to propose PTG topic about barbican key manager changes in tripleo/queens 14:02:26 <EmilienM> not sure about this one ^ 14:02:31 <EmilienM> jaosorior / abishop to kick-off some discussion about a tool to migrate barbican keys from a manager to another (maybe within barbican) 14:02:36 <EmilienM> same, not sure about this one 14:02:48 <EmilienM> team involved in containerization to review https://bugs.launchpad.net/bugs/1700912 and https://bugs.launchpad.net/tripleo/+bug/1706354 14:02:48 <openstack> Launchpad bug 1700912 in tripleo "Logrotate configs must be adjusted for containerized services logs" [High,Triaged] - Assigned to Bogdan Dobrelya (bogdando) 14:02:49 <openstack> Launchpad bug 1706354 in tripleo "Containerize Logrotate with cron" [High,Triaged] - Assigned to Bogdan Dobrelya (bogdando) 14:02:53 <EmilienM> bogdando: did you get the visibility you wanted? ^ 14:03:02 <EmilienM> team to review https://etherpad.openstack.org/p/tripleo-ceph-ansible (4 patches needed for m3) 14:03:03 <bogdando> EmilienM: not yet 14:03:22 <EmilienM> fultonj and gfidente are offline afik but I think everything or almost is merged 14:03:52 <jaosorior> EmilienM: me and abishop didn't really talk. Still waiting for that. 14:03:56 <EmilienM> the feature isn't working in CI yet: https://review.openstack.org/#/c/479288/ 14:04:03 <EmilienM> jaosorior: ok, please talk :) 14:04:20 <jaosorior> EmilienM: will do when I see him online. Let me write a post-it note. 14:04:35 <EmilienM> jaosorior: email on ML is better then 14:04:50 <jaosorior> ok 14:04:58 <jaosorior> (I have no idea who abishop is) 14:05:04 <EmilienM> tripleoclient folks to review https://review.openstack.org/#/c/471767/ + dependency asap: DONE 14:05:21 <EmilienM> jaosorior: Alan Bishop, he's working on Cinder and storage stuffs AFIK 14:05:35 <EmilienM> jaosorior: and he works at Red Hat :-) 14:05:47 <EmilienM> ci folks to review https://review.openstack.org/#/q/topic:3nodes-oooq: still in progress 14:05:54 <EmilienM> container team to review healthcheck spec asap https://review.openstack.org/#/c/465633/: done and merged 14:05:57 <jaosorior> EmilienM: thanks! 14:06:22 <cdearborn_> o/ 14:06:40 <EmilienM> #action (postponed) jaosorior and abishop to talk together about plans for queens relating to barbican backends (prepare ptg session if needed + discuss about migration tool) 14:06:56 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Fix keystone, cinder, heat-api cron containers https://review.openstack.org/489419 14:06:59 <EmilienM> #topic one off agenda items 14:07:05 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:07:17 <EmilienM> someone wrote about performances but no name 14:07:29 <EmilienM> rook maybe? ^ 14:07:36 <jkilpatr> oh that's my cue 14:07:43 <EmilienM> jkilpatr: ok please go ahead 14:07:50 <jrist> hehe 14:07:52 <jkilpatr> https://docs.google.com/presentation/d/1eQDH0UgI3UkbI7W1NNUVFnUA1v30FqJ86vHdjtOXh7s/edit#slide=id.p4 14:08:14 <jkilpatr> So for the past several months I've been working on a thirdparty job to run performance tests on promotion builds 14:08:51 <jkilpatr> after gathering several months worth of data we where able to build a machine learning based classifier that can determine if a build is performing much worse than it should and fail the build. 14:08:52 <EmilienM> jkilpatr: where the job is running exactly? 14:09:11 <jkilpatr> rdo passed phase2, mostly because that's the easiest place to get my dedicated hardware into the line. 14:09:19 <EmilienM> ok, gotcha 14:09:29 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement https://review.openstack.org/442716 14:09:29 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name https://review.openstack.org/442755 14:09:30 <openstackgerrit> Dan Prince proposed openstack/tripleo-heat-templates master: Add docker templates to configure Ironic inspector https://review.openstack.org/457822 14:09:32 <jkilpatr> it doesn't really work unless you can have exactly the same hardware every time, otherwise you have a hardware variable and a software variable and you can't draw any conclusions. 14:09:57 <jkilpatr> also this deploys a HA env with 2 comptues on baremetal blades. So it's good testing of 'real' deployment as well. 14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1674681 14:10:18 <openstack> Launchpad bug 1674681 in tripleo "buildlogs.centos.org CDN issues" [Critical,Triaged] 14:11:19 <EmilienM> jkilpatr: I don't think we have the resources to run this job in the gate but I think we should run it in a periodic manner, for ex in the promotion pipeline like you proposed 14:11:40 <jkilpatr> EmilienM, I don't think you'd want this in the gate exactly, you just want it to be passing regularly before you release. 14:11:55 <EmilienM> or even more, before you promote a new repo 14:12:16 <EmilienM> so we don't promote and can investigate which commit slowed us 14:12:27 <jtomasek> o/ 14:12:45 <jkilpatr> That's the idea, I hear there's some sort of thirdparty CI integration but I haven't found docs on how to get it working 14:12:54 <jkilpatr> jtomasek, is that hello or a question? 14:12:57 <EmilienM> jkilpatr: I think once we've moved the tripleo promotion workflow to RDO completely, we could add your job into this pipeline 14:13:07 <jtomasek> jkilpatr: just saying hi, sorry 14:13:15 <jkilpatr> don't worry about it. 14:13:26 <jkilpatr> EmilienM, that sounds great! 14:13:32 <EmilienM> jkilpatr: let's keep in touch 14:13:40 <EmilienM> jkilpatr: folks working on it are AFIK on PTO this week 14:13:49 <jkilpatr> The interesting thing is that it's easier to apply ML to logs + perf than just build logs, because a pass fail for most builds means different things, perf of a test acts as a 'source of truth' 14:14:06 <EmilienM> cool 14:14:10 <EmilienM> any question for jkilpatr before we move on? 14:14:41 <EmilienM> #action jkilpatr brings up perf testing discussion again when tripleo promotion workflow moved to RDO 14:15:03 <EmilienM> zshi_laptop: go ahead 14:15:22 <EmilienM> sounds like you want some review on a spec 14:15:23 <zshi_laptop> I'm requesting a review on spec : https://review.openstack.org/#/c/483227/ 14:15:34 <EmilienM> #action team to review PTP support spec https://review.openstack.org/#/c/483227/ 14:15:47 <zshi_laptop> it's ptp support in tripleo, ptp is precision time protocol, can be alternative of ntp 14:15:59 <zshi_laptop> this spec proposing add ptp configuration path in tripleo 14:16:12 <zshi_laptop> but leave ntp as default time synchronization method 14:16:38 <zshi_laptop> users who want to use ptp need explicitly configuration 14:16:44 <EmilienM> zshi_laptop: I think you can go ahead and start doing the code - I see this as an optional composable service in tripleo 14:17:06 <zshi_laptop> EmilienM, cool, I have started coding 14:17:06 <EmilienM> though we'll review the spec, as you took time to write it 14:17:09 <EmilienM> ok 14:17:15 <zshi_laptop> will submit the code next week 14:17:39 <zshi_laptop> sure, thanks! 14:17:57 <EmilienM> cool 14:18:02 <EmilienM> numans: o/ 14:18:05 <numans> hi 14:18:07 <numans> kolla build generates neutron-server-ovn docker image. 14:18:15 <numans> I wanted to check if it is possible to have this image uploaded into tripleoupstream docker hub ? 14:18:19 <EmilienM> dprince: ^ 14:18:23 <numans> This would be required for OVN container CI job 14:18:46 <numans> and also neutron-server-odl incase odl folks need it 14:19:14 <EmilienM> numans: please check with dprince - I think he manages the docker hub thing 14:19:23 <numans> ack. will check with him 14:19:25 <numans> thanks 14:19:46 <EmilienM> and I think this manual workflow is going to change 14:19:54 <EmilienM> once we have the promotion pipeline running in RDO 14:19:59 <EmilienM> dprince can confirm when he's back 14:20:07 <openstackgerrit> Honza Pokorny proposed openstack/tripleo-ui master: Implement Zaqar logger adapter https://review.openstack.org/459294 14:20:07 <numans> that's nice 14:20:08 <numans> thanks 14:20:21 <EmilienM> chem: o/ 14:20:24 <chem> he 14:20:49 <chem> so I wanted to know if we plan a switch from "latest" tag to some version for the docker images 14:21:05 <EmilienM> another question for dprince 14:21:07 <EmilienM> :) 14:21:08 <chem> it's important for update 14:21:14 <dprince> numans: if the neutron-server-ovn is listed in our tripleo-common 'container-images' directory it should get built and pushed to dockerhub 14:21:23 <chem> EmilienM: ... I was afraid you'd say this :) 14:21:46 <numans> dprince, i will check on it. thanks 14:21:54 <EmilienM> I think containers were updated last night 14:21:58 <dprince> chem: for a default, 'latest' makes the most sense 14:21:59 <chem> anyway we have the update review in the pipeline, any help welcome https://review.openstack.org/#/q/topic:add_update_tasks 14:22:01 <chem> :) 14:22:19 <dprince> chem: when pushing images however we probably could tag them with something more meaningful 14:22:36 <chem> dprince: hi, well, we're not sure how to detect that we need to upgrade a container 14:22:41 <numans> dprince, i submitted a patch to add it in overcloud-containers.yaml - https://review.openstack.org/#/c/484394/1/container-images/overcloud_containers.yaml, 14:22:41 <dprince> chem: so long as the pull 'latest' default still translates to pull that I'd be fine w/ it I think 14:22:45 <chem> dprince: we can have this discussion outside of the mtg 14:22:54 <dprince> chem: sure, sounds good 14:23:12 <numans> but CI failed since it couldn't find it in tripleoupstream docker hub. 14:23:16 <chem> EmilienM: that's all from me 14:23:21 <numans> chem, sorry. 14:23:28 <chem> numans: np 14:23:33 <openstackgerrit> Alex Schultz proposed openstack/tripleo-puppet-elements master: Set remove-machine-id to executable https://review.openstack.org/489618 14:23:50 <EmilienM> chem: ok thx 14:23:56 <EmilienM> anything else before we start regular agenda? 14:24:04 <EmilienM> #topic bugs 14:24:09 <EmilienM> #link https://launchpad.net/tripleo/+milestone/pike-rc1 14:24:41 <EmilienM> we have 107 Triaged and 56 In Progress 14:24:45 <EmilienM> all High or Critical 14:24:55 <EmilienM> (I moved Medium / low / wishlist to queens-1) 14:25:22 <EmilienM> right now the priority is to reduce the number of critical / high bugs targeted for pike-rc1 14:25:33 <EmilienM> at this stage - all upgrade bugs are Critical 14:25:39 <EmilienM> chem: ^ if you can help in triaging those 14:25:56 <EmilienM> is there any bug we need to discuss this week? 14:26:02 <chem> EmilienM: right, will check on them 14:26:13 <dprince> numans: I think I pushed your ovn images last night BTW https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=tripleoupstream+ovn 14:26:24 <openstackgerrit> Marios Andreou proposed openstack/tripleo-heat-templates master: Adds stop and disable for libvirtd on upgrade to containers https://review.openstack.org/489619 14:26:31 <dprince> numans: is that what you were after? 14:26:47 <numans> dprince, its neutron-server-ovn... let me check if it has now 14:27:04 <EmilienM> (moving on the agenda) 14:27:09 <EmilienM> #topic projects releases or stable backports 14:27:21 <EmilienM> I sent an email yesterday but we released pike-3 last week 14:27:37 <EmilienM> tripleoclient final was released last week - stable/pike was created for this one 14:27:45 <EmilienM> right now the focus is on rc1 14:28:05 <EmilienM> #link http://lists.openstack.org/pipermail/openstack-dev/2017-July/120357.html 14:28:20 <EmilienM> we're in Feature Freezze right now 14:28:38 <EmilienM> please do not merge code that is a feature and that doesn't have Feature Freeze Exception ( FFE ) 14:29:01 <EmilienM> any question about release management. 14:29:03 <EmilienM> ? 14:29:12 <atoth> 1 14:29:22 <atoth> still not exactly clear on FFE 14:29:46 <atoth> is there some tag that gets put onto things for it? 14:30:22 <EmilienM> atoth: so let me rephrase how it works 14:30:34 <EmilienM> during the cycle, some people work on blueprints, most of them are features 14:31:01 <EmilienM> at the end of milestone 3, we stop approving code that is a feature, because we want to stabilize and produce stable release candidates 14:31:07 <EmilienM> that's call Feature Freeze 14:31:19 <EmilienM> but some features are really needed and have limited risk to break something 14:31:37 <EmilienM> so features authors ask to the PTL and the team if we can grant FFE 14:31:51 <EmilienM> if yes, the code can be approved during RC 14:31:58 <EmilienM> if no, -2 and it will wait the next cycle 14:32:04 <EmilienM> it's all documented here: https://docs.openstack.org/project-team-guide/release-management.html#common-cycle-with-development-milestones 14:32:13 <atoth> perfect, thanks 14:32:20 <EmilienM> atoth: any question in specific? 14:32:49 <atoth> Just wondering about it for the lbaas stuff :-) 14:33:12 <atoth> Have I requested it clearly enough? 14:33:12 <EmilienM> atoth: what are the patches that you need for LBaaS support in tripleo? 14:33:30 <EmilienM> (patches not merged) 14:33:45 <atoth> most are already merged into master just have to figure out why it's crashing the multinode scenario 14:34:02 <atoth> so will be new patches 14:34:12 <atoth> once figured out 14:34:22 <EmilienM> that's why I wish we could develop features with tests in the same time 14:34:31 <atoth> agreed 14:34:32 <EmilienM> in your case, the code is merged but tests are now failing 14:34:43 <EmilienM> I don't think we need an FFE for that 14:34:48 <EmilienM> the feature is already there 14:34:51 <atoth> damn newbees ;-) 14:34:52 <EmilienM> but isn't working 14:35:17 <atoth> but FFE if fix is determined? 14:35:21 <EmilienM> so from now, we're probably considering LBaaS patches as bug fixes 14:35:27 <EmilienM> as the major feature has been implemented 14:35:39 <atoth> ok, bug fixes then, ack 14:35:47 <EmilienM> alright, moving on 14:35:49 <EmilienM> #topic CI 14:36:00 <EmilienM> a bunch (if not all) CI folks are on PTO this week 14:36:15 <EmilienM> our major issue now is https://bugs.launchpad.net/tripleo/+bug/1674681 14:36:15 <openstack> Launchpad bug 1674681 in tripleo "buildlogs.centos.org CDN issues" [Critical,Triaged] 14:36:28 <EmilienM> I'm investigating today, still a lot of red in http://tripleo.org/cistatus.html 14:36:28 <openstackgerrit> Rajini Karthik proposed openstack/tripleo-heat-templates master: Add support for Dell EMC VMAX ISCSI Backend https://review.openstack.org/489393 14:36:46 <EmilienM> please do not recheck blindly if CDN timeouts, it's still timeouting, no need to recheck 14:37:13 <EmilienM> before the meeting I was investigating why some jobs were failing, I found the CDN issue was still there but there might be something else 14:37:19 <EmilienM> any help on this is welcome 14:37:25 <EmilienM> (I'll spend my day on it probably) 14:37:38 <EmilienM> anything about CI this week? 14:38:07 <EmilienM> chem, jistr: at this stage of the cycle, we should have upgrade jobs working and if not we need to file critical bugs in launchpad 14:38:33 <chem> EmilienM: got it 14:38:42 <EmilienM> jistr: I think we should retire old upgrade jobs on master 14:38:47 <EmilienM> and keep the one you did 14:38:53 <EmilienM> from baremetal to container only 14:38:53 <chem> EmilienM: I'm relatively more concerned with update actually 14:39:12 <chem> EmilienM: and non-controller upgrade 14:39:17 <jistr> yea 14:39:18 <EmilienM> chem: I'm sad that gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 is not testing updates AFIK 14:39:33 <EmilienM> chem: gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 is the new ovb-updates job 14:39:36 <openstackgerrit> Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX Cinder Driver https://review.openstack.org/489624 14:39:48 <EmilienM> and AFIK it's not testing stack-update (a regression in quickstart move) 14:39:58 <EmilienM> so we don't cover overcloud updates anymore, just FYI 14:40:09 <chem> EmilienM: ack, good to know 14:40:09 <EmilienM> we might want to take care of this problem sometimes ^ 14:40:27 <EmilienM> jistr: yea = let's go for it? :) 14:40:37 <chem> EmilienM: and I don't think non-controller upgrade is covered somewhere neither 14:40:40 <EmilienM> jistr: if yes, I'll work on project-config patch 14:40:47 <jistr> EmilienM: yea :) but my previous yea was mainly to +1 chem 14:40:53 <EmilienM> chem: no, we deploying 1-node overclouds 14:40:54 <jistr> main issue is that we don't have updates 14:40:58 <jistr> and non-controller upgrades 14:41:01 <EmilienM> jistr: ok so yea and yea :D 14:41:02 <jistr> the job itself runs fine 14:41:03 <chem> EmilienM: ack, noted 14:41:06 <jistr> EmilienM: yea :D 14:41:09 <chem> yeah 14:41:15 <EmilienM> stop yea'ing us 14:41:17 <EmilienM> :) 14:41:19 <adarazs> EmilienM: just want to point out that most of the folks from the CI squad are on PTO or otherwise engaged, it's mostly me who's left, so if there's some issue with quickstart/etc, please let me know. 14:41:29 <marios> chem: jistr +1 wrt upgrade-non-controller, just filed this one (from a linked BZ https://bugs.launchpad.net/tripleo/+bug/1707926 thanks to mcornea 14:41:29 <openstack> Launchpad bug 1707926 in tripleo "Ocata to Pike upgrade: libvirtd service is running on host after upgrade and nova_libvirt container keeps restarting" [Critical,In progress] - Assigned to Marios Andreou (marios-b) 14:41:36 <EmilienM> adarazs: good to know, I thought I was alone 14:42:07 <EmilienM> #action EmilienM to switch master to run new upgrade jobs and not old ones anymore 14:42:08 <chem> mcornea: jistr let's tackle those 22 upstream bugs :) 14:42:41 <EmilienM> anything else about CI? 14:42:50 <jistr> chem: tackling https://review.openstack.org/#/c/489613/ presently fwiw :/ 14:43:29 <EmilienM> #topic specs 14:43:33 <EmilienM> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:43:38 <chem> jistr: oh boy 14:43:42 <EmilienM> do we have anything about specs to discuss this week? 14:43:51 <jistr> chem: that's blocker for ceph-ansible CI 14:43:58 <jistr> and k8s 14:44:00 <EmilienM> zshi_laptop had a spec to present before, so it's already covered 14:44:30 <EmilienM> #topic open discussion 14:44:37 <openstackgerrit> Rajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VMAX ISCSI Cinder Driver https://review.openstack.org/489624 14:44:48 <EmilienM> 1. Please propose topics for PTG: https://etherpad.openstack.org/p/tripleo-ptg-queens 14:45:25 <EmilienM> 2. I won't run PTL next cycle - it's nomination time by the way, this week. Don't miss the schedule and nominate yourself if you want to be PTL 14:45:57 <EmilienM> any feedback / comment / question before we close the meeting? 14:46:04 <panda> :'( 14:46:24 <EmilienM> I hate when pandas cry :-( 14:46:37 * ccamacho cries too :P 14:46:39 <EmilienM> already, we're done 14:46:44 <EmilienM> alright$ 14:46:45 <ccamacho> *** all the time 14:46:50 <EmilienM> thanks everyone 14:46:53 <EmilienM> #endmeeting