14:00:13 <EmilienM> #startmeeting tripleo
14:00:14 <openstack> Meeting started Tue Nov  1 14:00:13 2016 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:18 <openstack> The meeting name has been set to 'tripleo'
14:00:26 <EmilienM> #topic agenda
14:00:28 <EmilienM> * one off agenda items
14:00:30 <EmilienM> * bugs
14:00:32 <EmilienM> * Projects releases or stable backports
14:00:34 <EmilienM> * CI
14:00:36 <EmilienM> * Specs
14:00:38 <EmilienM> * open discussion
14:00:40 <EmilienM> hello folks!
14:00:44 <jtomasek> \m/
14:00:45 <mwhahaha> hola
14:00:46 <shadower> heyia
14:00:48 <akrivoka> hello! o/
14:00:50 <beagles> o/
14:00:51 <cdearborn> o/
14:00:52 <marios> o/
14:00:53 <adarazs> o/
14:00:59 <sshnaidm> o/
14:01:00 <dtantsur> o/
14:01:08 * marios always confused with the timechange... wasn't sure if now or 1 hour
14:01:08 <weshay> o/
14:01:12 <panda|pto|PH> o/ (kinda)
14:01:15 <florianf> o/
14:01:26 <EmilienM> I hope you had safe travel back to home!
14:01:40 <fultonj> here
14:02:25 <EmilienM> #topic off agenda items
14:02:28 <EmilienM> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:02:32 <EmilienM> sshnaidm: hi!
14:02:37 <sshnaidm> EmilienM, hi!
14:02:47 <dprince> hi
14:03:05 <sshnaidm> EmilienM, I have 2 questions in off items
14:03:10 <EmilienM> go ahead
14:03:53 <sshnaidm> I started POC patch when running quickstart in TripleO CI ovb environment and it seemed working
14:04:16 <sshnaidm> I'd like to do experimental job with quickstart running there to prov its correctness and stability
14:04:31 <sshnaidm> (and detect possible problems)
14:04:57 <sshnaidm> I'd like to ask if there is no objections to have such job in experimental pipeline
14:05:34 * adarazs looks forward to it.
14:05:34 <EmilienM> interesting, I'm wondering what path are we taking 1) using tripleo-ci to run quickstart or 2) implement new quickstart jobs in third party CI that deploy ovb
14:05:47 <hrybacki> o/
14:05:56 <EmilienM> 2) wouldn't use tripleo-ci
14:06:49 <adarazs> I'd prefer 1)
14:07:01 <EmilienM> 1) sounds complex to me
14:07:16 <EmilienM> as we're going to increase tripleo-ci logic complexity
14:07:40 <EmilienM> see https://review.openstack.org/#/c/381094/
14:07:55 <adarazs> if we want to have tripleo-ci be driven by quickstart in some way soon (which I hope we agreed that we do) then it would be a good first step.
14:08:20 <sshnaidm> EmilienM, 1) would help also to have CI externally consumable, btw
14:08:58 <slagle> sshnaidm: can you link to the patch you are referring to?
14:09:05 <sshnaidm> EmilienM, anyway, it's experimental job, everybody will look at it and if we hate it - it's always possible to kill it
14:09:09 <EmilienM> slagle: https://review.openstack.org/#/c/381094/
14:09:10 <d0ugal> o/
14:09:20 <slagle> oh right :)
14:09:27 <panda|pto|PH> 2) looks closer to what we're going to do in the end, I wouldn't add an intermediate step.
14:09:58 <EmilienM> 1) would be a transition to have 2) at the end, iiuc
14:10:12 <EmilienM> but anyway, it's an implemenation specific question, that we can talk outside this meeting.
14:10:36 <EmilienM> sshnaidm: +1 for experimental job
14:10:51 <sshnaidm> ok, thanks!
14:10:51 <EmilienM> sshnaidm: and wait 1 or 2 week(s) to make sure it's "stable"
14:11:14 <sshnaidm> sure
14:11:48 <sshnaidm> ok, then the second question is about moving tripleo-ci from "openstack-infra" to "openstack"
14:11:51 <EmilienM> sshnaidm: in the meantime, could we make progress on using oooq to deploy multinode jobs too?
14:12:19 <sshnaidm> EmilienM, yeah, I planned it after ovb is running
14:12:23 <EmilienM> excellent
14:12:31 <weshay> bkero, ^
14:12:41 <bkero> o/
14:12:58 <EmilienM> slagle: did you have thoughts about moving ooo-ci to openstack? I remember you were involved in this
14:13:58 <EmilienM> sshnaidm: +1 on my side
14:14:43 <EmilienM> sshnaidm: anything else?
14:14:45 <sshnaidm> ok, then maybe let's discuss it later in #tripleo
14:14:51 <sshnaidm> EmilienM, not, that's it, thanks
14:14:57 <EmilienM> dtantsur: your turn Sir
14:15:18 <dtantsur> yep, quick question: are there plans now to deprecate instack-virt-setup?
14:15:32 <dtantsur> it's not longer used in CI, and there seem to be a lot of efforts around quickstart
14:15:47 <dtantsur> finally, we're moving away from pxe_ssh drivers, so instack-virt-setup requires reworking
14:15:53 <shardy> dtantsur: tripleo-quickstart was always supposed to deprecate it, we've just not quite got to that yet in docs/CI
14:16:06 <slagle> EmilienM: i think it has more to do with moving to 3rd party
14:16:12 <shardy> dtantsur: have you looked at what rework will be required for quickstart?
14:16:24 <slagle> EmilienM: once that is complete, it can probably move
14:16:33 <EmilienM> slagle: right, ++
14:16:37 <dtantsur> shardy, not yet; but at least quickstart is covered by some CI, unlike tripleo-incubator
14:17:00 <shardy> dtantsur: ack, yeah and we ideally don't want to do this work twice
14:17:01 <slagle> EmilienM: but our multinode jobs won't be 3rd party, so maybe we'd split the repo, i dont entirely know if there is precedence for that
14:17:35 <shardy> dtantsur: what's the timeline for ironic moving away from pxe_ssh?
14:17:47 <EmilienM> slagle: mhh interesting, indeed; maybe wait a little more until we figure implementations
14:18:02 <dtantsur> shardy, no strict timeline, but it's already deprecated. we have full rights to remove it in Ocata after O-1
14:18:12 <EmilienM> slagle: fwiw, puppet openstack CI runs some scripts in openstack namespace...
14:18:13 <dtantsur> shardy, also we stop covering it with out CI pretty soon
14:18:56 <shardy> dtantsur: Ok, so it'd be very good if we could switch docs/CI to oooq before it's removed then ;)
14:19:06 <dtantsur> true :)
14:19:12 <shardy> o-1 seems like an ambitious target for that
14:19:25 <EmilienM> o-2 maybe?
14:20:01 <EmilienM> o-1 is in 2 weeks
14:20:30 <shardy> yeah, I think it'd be better if we could aim for o-2 and ask if Ironic would be willing to delay removing it until then
14:20:37 <shardy> so our docs etc don't break
14:20:44 <EmilienM> ++
14:20:58 <jrist> late o/
14:21:25 <shardy> dtantsur: would that be a reasonable compromise from the Ironic perspective, or are there pressures to remove it really soon?
14:22:19 <dtantsur> shardy, we can delay the removal, there is no too much pressue
14:22:33 <dtantsur> shardy, but the driver is deprecated, will have no CI soon and will only receive critical bug fixes
14:22:50 <shardy> dtantsur: Ok sounds good
14:23:11 <shardy> can anyone take an action to follow up on the oooq side to determine the changes needed?
14:23:47 <dtantsur> I can provide any help, but I can't do it right this second
14:23:57 <dtantsur> I know that packaging virtualbmc in RDO is step 0
14:24:26 <shardy> dtantsur: perhaps you could raise a bug against tripleo-quickstart with details of the work you're aware of, then we can figure out who has bandwidth to do it?
14:24:41 <dtantsur> shardy, I think I have a tripleo blueprint..
14:24:48 <dtantsur> I can raise a bug as well for sure
14:25:04 <dtantsur> #link https://blueprints.launchpad.net/tripleo/+spec/switch-to-virtualbmc
14:25:06 <EmilienM> dtantsur: your bp is enough imho
14:25:42 <EmilienM> dtantsur: maybe you can sync with weshay to see who from oooq experts could help there
14:25:47 <shardy> ack, yeah I didn't realize there was already a BP, thanks
14:25:51 <panda|pto|PH> weshay: should we add this to the transition etherpad ?
14:25:52 * weshay looks
14:25:56 <weshay> aye
14:26:17 <EmilienM> dtantsur: thanks for bringing this topic!
14:26:20 <weshay> oh this is cool
14:26:29 <weshay> dtantsur++
14:26:41 <dtantsur> no problem :)
14:26:41 <EmilienM> do we have anything else before we start regular agenda?
14:27:03 <weshay> panda|pto|PH, I think we can go after that outside of the transition
14:27:31 <EmilienM> #topic bugs
14:27:34 <panda|pto|PH> weshay: ok
14:28:05 <EmilienM> we are still fixing some bugs that we want backported in stable/newton, I consider them highest prio at this time
14:28:07 <EmilienM> https://bugs.launchpad.net/tripleo/+bugs?field.tag=newton-backport-potential
14:28:24 <marios> https://bugs.launchpad.net/tripleo/+bug/1638003 still an issue for upgrades and updates atm (overcloudrc change)... added a comment at https://review.openstack.org/#/c/391201/
14:28:25 <openstack> Launchpad bug 1638003 in tripleo "Stack update causes overcloudrc OS_PASSWORD update and fails overcloud keystone authentication" [High,In progress] - Assigned to Ryan Brady (rbrady)
14:28:54 <EmilienM> I suspect there are more, because I noticed not all team members were doing self triage :(
14:29:37 <EmilienM> marios: isn't it addressed by https://review.openstack.org/#/c/391201/ ?
14:29:47 <marios> EmilienM: not in my testing (sasha too)
14:30:02 <marios> EmilienM: so bringing up for visibility.. i needinfod rbrady on the BZ too
14:30:08 <EmilienM> oh, it's in bug description
14:30:17 <marios> EmilienM: (the BZ is linked from the launchpad bug)
14:30:40 <marios> EmilienM: it is in that list you gave anyway i mean the backport bugs
14:30:43 <EmilienM> his patch seems like failing on CI consistently
14:31:03 <marios> EmilienM: another one for uprades is here https://bugs.launchpad.net/tripleo/+bug/1634897 - still some discussino with pradk on the fix but concerned given the timeframe
14:31:05 <openstack> Launchpad bug 1634897 in tripleo " Upgrade Mitaka to Newton failed with gnocchi using swift backend: ClientException: Authorization Failure. Authorization Failed: Service Unavailable (HTTP 503)" [Critical,In progress] - Assigned to Pradeep Kilambi (pkilambi)
14:31:15 <marios> EmilienM: (also in the list as newton backport)
14:31:32 <EmilienM> don't forget to add milestones
14:32:08 <EmilienM> ok so https://review.openstack.org/#/c/388649/ will be backported when merged
14:32:27 <EmilienM> marios: thanks!
14:33:21 <EmilienM> do we have any other outstanding bug?
14:34:09 <EmilienM> #topic Projects releases or stable backports
14:34:18 <EmilienM> so 2 infos:
14:34:26 <EmilienM> ocata-1 will be released in 2 weeks!
14:34:53 <EmilienM> also, this week I plan to release a new version of newton, that will contain all our recent backports
14:35:09 <EmilienM> I might wait for critical bugs (eg: upgrades, etc) to be fixed first
14:35:22 <EmilienM> any thoughts on release management?
14:36:09 <shardy> should we perhaps aim to release stable branches around the same time as the ocata milestones?
14:36:21 <EmilienM> shardy: fyi, I'll be on PTO during ocata-1 release. I'll be online from time to time but not sure I can't send release patch request
14:36:28 <shardy> IIRC it was mentioned last week and would be a good way to ensure regular releases are made from the stable branches
14:36:40 <EmilienM> shardy: sounds good to me
14:37:07 <shardy> EmilienM: I will be around & can help with the ocata release
14:37:11 <EmilienM> thx
14:37:17 <EmilienM> ok so
14:37:21 <shardy> EmilienM: let me know before you go on PTO so we can sync up on the latest status etc
14:37:48 <EmilienM> #action EmilienM / shardy to release newton-2 and ocata-1 during Nov 14th week
14:37:55 <EmilienM> shardy: sure thing.
14:38:32 <EmilienM> marios: please let us know about upgrade things, and make sure they are all backported
14:38:52 <EmilienM> #topic CI
14:39:01 <marios> EmilienM: ack we are fighting every day :) looking forward to the rest of november
14:39:03 <EmilienM> this morning, I found out that ovb/ha job is really unstable
14:39:27 <EmilienM> I haven't filed a bug yet, because i'm still waiting for a CI job now
14:39:50 <EmilienM> http://logs.openstack.org/64/391064/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/2d15c11/console.html#_2016-10-31_10_19_54_304885
14:40:02 <EmilienM> that's the error, and it started on October 27th, during a promotion job
14:40:08 <EmilienM> I suspect a regression in OpenStack
14:40:22 <EmilienM> but this one has been hard to debug, I haven't seen useful logs in nova/cinder yet
14:40:39 <EmilienM> if you still see this error today and later, please let me know
14:41:17 <EmilienM> http://tripleo.org/cistatus.html - ovb/ha is red in most of recent runs :(
14:41:23 <sshnaidm> I see today a lot of errors like "ClientException: resources.volume1: Gateway Time-out (HTTP 504)" in pingtest stack creating
14:41:40 <EmilienM> it could be related
14:41:48 <sshnaidm> I'll look
14:41:50 <EmilienM> the error I saw was boot from volume not working
14:42:02 <EmilienM> sshnaidm: let's sync after meeting
14:42:16 <EmilienM> I have another info
14:42:16 <sshnaidm> ok
14:42:36 <EmilienM> slagle: pabelanger just told me we could submit a patch in nodepool to have a new multinode job with 3 nodes
14:42:37 <jaosorior> sshnaidm, EmilienM I have the feeling that it's a selinux issue, the lack of relevant logs is really confusing
14:42:56 <EmilienM> jaosorior: I haven't looked audit logs
14:43:01 <EmilienM> so maybe
14:43:14 <EmilienM> slagle: do you want to take action and submit it this week?
14:44:04 <slagle> EmilienM: that's something we've discussed, i know we can do it
14:44:16 <EmilienM> cool :)
14:44:22 <slagle> EmilienM: i dont honestly anticipate having time to work on it this week
14:44:46 <EmilienM> slagle: ok, no problem
14:44:54 <EmilienM> I'll see if I can submit it
14:45:18 <EmilienM> #action EmilienM or slagle to submit patch to nodepool to add 3-nodes jobs
14:45:31 <EmilienM> anything else about CI this week?
14:45:48 <slagle> i'd like to get this landed: https://review.openstack.org/381286
14:45:56 <slagle> so we can test undercloud upgrades
14:46:12 <slagle> and move forward with the composable undercloud work
14:46:31 <slagle> reviews appreciated :)
14:46:43 <EmilienM> slagle: the job is gate-tripleo-ci-centos-7-undercloud-upgrades-nv, right?
14:46:51 <slagle> yes
14:47:02 <EmilienM> cool!
14:47:15 <slagle> now that the ovb mitaka/liberty jobs are fixed, ovb should pass on that too
14:47:27 <EmilienM> #action team to review undercloud/upgrade patch https://review.openstack.org/381286
14:47:36 <marios> slagle: nice will have a look tomorrow
14:47:53 <slagle> thanks :)
14:48:39 <EmilienM> ok last topic
14:48:41 <EmilienM> #topic specs
14:48:48 <EmilienM> #topic https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:49:09 <marios> i put out a first pass from composable service upgrades discussion, thanks shardy for having a look already https://review.openstack.org/#/c/392116/
14:49:13 <EmilienM> if you have time, please review our specs that we discussed at Summit
14:49:39 <EmilienM> the tripleo/squads policy is ready for review and comments were all addressed https://review.openstack.org/#/c/385201/
14:49:49 <EmilienM> marios: I was reviewing yours before the meeting. I'll finish today
14:49:59 <marios> EmilienM: ack thanks appreciate
14:50:41 <EmilienM> does it sound reasonable to merge Ocata specs before Ocata-1?
14:50:48 <EmilienM> it gives us 2 weeks
14:51:00 <EmilienM> I would hate merging specs after doing the actual feature ;-)
14:51:50 <shardy> EmilienM: we can try, but historically TripleO spec reviews have been pretty slow
14:51:56 <marios> EmilienM: well that would be the *right* way to do it but ultimately we may not make it
14:51:58 <shardy> perhaps this is an opportunity to fix that
14:52:02 <EmilienM> right, and we might want to improve ourselves on this thing
14:52:15 <marios> EmilienM: so +1 nice thing to aim for
14:52:18 <EmilienM> marios: well, maybe we should be a bit more serious in our specs
14:52:19 <dtantsur> spec reviews are always slow, not only in tripleo :)
14:52:28 <dtantsur> but it's not a reason not to improve, I agree
14:52:31 <EmilienM> #action team to review Ocata specs for ocata-1 https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:52:40 <EmilienM> I'll send a note on ML about it
14:52:55 <EmilienM> any question / remark about specs?
14:53:31 <EmilienM> #topic open discussion
14:53:45 <EmilienM> any question or feedback is welcome now
14:54:15 <panda|pto|PH> I wanted to add that ovb/ha job has at least one big problem: since redis is unable to start, gnocchi-metricd is eating a lot of CPU, so things may be unstable also for that reason.
14:54:16 <EmilienM> just FYI I'll be on vacations from Friday night, and then during 2 weeks. I'll have emails & IRC sometimes.
14:54:43 <EmilienM> panda|pto|PH: we saw that before
14:54:49 <EmilienM> panda|pto|PH: there is a launchpad about it iirc
14:54:58 <EmilienM> panda|pto|PH: talk to pradk and carlos
14:55:16 <EmilienM> panda|pto|PH: why redis is unable to start?
14:55:21 <panda|pto|PH> EmilienM:yes, but it's a different problem, we ened a selinux command to let redis open its unix socket
14:55:54 <panda|pto|PH> EmilienM: I'm curre;lty discussing this with rhallisay in #rdo
14:56:15 <EmilienM> cool
14:56:26 <EmilienM> panda|pto|PH: please file a launchpad bug if needed, so we can track the work being done here
14:56:41 <panda|pto|PH> EmilienM: already there
14:56:42 <EmilienM> do we have anything else for this week?
14:57:14 <panda|pto|PH> EmilienM: https://bugs.launchpad.net/tripleo/+bug/1637961
14:57:15 <openstack> Launchpad bug 1637961 in tripleo "periodic HA master job pingtest times out" [High,Confirmed] - Assigned to Gabriele Cerami (gcerami)
14:57:45 <EmilienM> panda|pto|PH: thx
14:57:54 <EmilienM> well, sounds like we're done
14:58:09 <EmilienM> have a great week everyone
14:58:11 <EmilienM> #endmeeting