15:02:09 <mgoddard> #startmeeting kolla
15:02:09 <openstack> Meeting started Wed Jul  3 15:02:09 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:12 <openstack> The meeting name has been set to 'kolla'
15:02:22 <mgoddard> #topic rollcall
15:02:25 <mgoddard> \o
15:03:20 <yoctozepto> o/
15:03:38 <mgoddard> mnasiadka: around?
15:03:47 <mgoddard> egonzalez: ping
15:03:49 <mnasiadka> Yup
15:03:53 <mnasiadka> o/
15:04:16 <mgoddard> cloudnull: \o
15:04:26 <cloudnull> o/
15:04:41 <mgoddard> #topic agenda
15:04:47 <mgoddard> * Roll-call
15:04:48 <mgoddard> * Announcements
15:04:50 <mgoddard> ** Kolla Train priorities results are in https://etherpad.openstack.org/p/kolla-train-priorities
15:04:52 <mgoddard> ** Vote for kayobe becoming (or not) a kolla deliverable http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html
15:04:54 <mgoddard> ** Vote for yoctozepto becoming core: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007428.html
15:04:56 <mgoddard> ** Kevin Carter (cloudnull) is now Kolla liaison for TripleO
15:04:58 <mgoddard> * Review action items from last meeting
15:05:00 <mgoddard> * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:05:02 <mgoddard> * Stein release status
15:05:04 <mgoddard> * IPv6 community goal
15:05:06 <mgoddard> #topic announcements
15:05:09 <mgoddard> #info Kolla Train priorities results are in
15:05:15 <mgoddard> #link https://etherpad.openstack.org/p/kolla-train-priorities
15:05:26 <mgoddard> I updated the whiteboard with these work items
15:05:48 <mgoddard> #info Vote for kayobe becoming (or not) a kolla deliverable
15:05:55 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html
15:06:04 <mgoddard> #info Vote for yoctozepto becoming core
15:06:10 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007428.html
15:06:16 <mgoddard> #info Kevin Carter (cloudnull) is now Kolla liaison for TripleO
15:06:35 <cloudnull> o/
15:06:42 <mgoddard> Is that a reasonable way to describe your role cloudnull?
15:07:02 <cloudnull> correct
15:07:12 <yoctozepto> cloudnull: or would you prefer TripleO agent? ;D
15:07:26 <mgoddard> great, welcome
15:07:37 <mgoddard> any other announcements?
15:08:13 <mgoddard> No action items last time
15:08:17 <cloudnull> I'm still ramping up on the current state of Kolla, but I will be running point on Kolla espionage for TripleO :)
15:08:34 <yoctozepto> cloudnull: that's the spirit!
15:08:41 <mgoddard> cloudnull: we'll soon have you working as a double agent :p
15:08:49 <cloudnull> :P
15:08:59 <mgoddard> #topic Kolla whiteboard
15:09:07 <mgoddard> #link https://etherpad.openstack.org/p/KollaWhiteBoard
15:09:25 <yoctozepto> whiteboard pretty colorful nowadays
15:09:34 <mgoddard> checking CI
15:09:45 <yoctozepto> CI unblocked
15:09:51 <mgoddard> as long as the colour is GREEN I'm happy
15:09:54 <yoctozepto> infra issues handled as soon as they appear
15:10:03 <yoctozepto> sorry for my bg color
15:11:17 <mgoddard> CI mostly looking green
15:11:28 <yoctozepto> mgoddard: Bifrost FAILING on OVH cloud due to weird hostname issues, passing elsewhere.
15:11:31 <mgoddard> thanks for some great work on CI fixes yoctozepto
15:11:40 <yoctozepto> was to ask you about this point
15:11:52 <mgoddard> go ahead
15:11:52 <yoctozepto> mgoddard: you're welcome
15:12:21 <yoctozepto> mgoddard: I think I have some lag with IRC
15:12:29 <yoctozepto> sorry if I seem delayed
15:12:48 <yoctozepto> asking if there is an example of this failure
15:12:58 <yoctozepto> so I could take a look and take it off the whiteboard ;D
15:13:43 <mgoddard> yoctozepto: it's fixed in master because bifrost removed rabbitmq
15:13:48 <mgoddard> 'fixed'
15:14:02 <mgoddard> so might not be worth your time
15:14:11 <yoctozepto> mgoddard: k, thanks for explaining
15:14:18 <yoctozepto> could you put that on whiteboard?
15:14:24 <yoctozepto> this explanation
15:14:38 <mgoddard> done
15:15:04 <yoctozepto> thanks
15:15:24 <yoctozepto> maybe a quick peek at CI issues
15:15:58 <mgoddard> you mean https://etherpad.openstack.org/p/kolla-ci-errors?
15:16:01 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ci-errors
15:16:04 <yoctozepto> yup
15:16:13 <yoctozepto> there is this special category at the bottom
15:16:14 <mgoddard> sure, lead the way
15:16:23 <yoctozepto> for odd Zuul behavior
15:16:40 <yoctozepto> the thing is to notify infra and leave a link there
15:16:48 <yoctozepto> maybe they will find the root cause
15:16:51 <mgoddard> what is the symptom?
15:17:00 <mgoddard> could you add that to the page?
15:17:27 <yoctozepto> mgoddard: discrepancy between txt log/json log/ara/primary ara/primary log, oddities like files existing before being created
15:17:49 <mgoddard> messy
15:18:19 <yoctozepto> yup, Zuul must run the same instance twice or more
15:18:50 <yoctozepto> rare event I think but still
15:18:54 <yoctozepto> best if they track it down
15:18:59 <yoctozepto> onto the 'normal' issues
15:19:30 <yoctozepto> "    maybe we should consider retrying all such tasks like 3 times to ensure it is not a "once in a thousand" issue"
15:19:55 <yoctozepto> this is for trying to start a container once when docker registry has its break
15:20:16 <yoctozepto> worth considering I think
15:20:24 <mgoddard> +1
15:20:39 <yoctozepto> then the next issue
15:20:45 <mgoddard> we have seen a few issues recently where retries would have helped
15:20:48 <yoctozepto> 500 Internal Server Error creating services and endpoints
15:20:55 <yoctozepto> no idea what's the cause
15:21:00 <yoctozepto> help appreciated
15:21:22 <mgoddard> in that keystone bootstrap issue it looks like keystone just hadn't come up properly yet
15:22:05 <yoctozepto> hmm, yeah, leaving note to check keystone logs
15:22:09 <yoctozepto> let's not lag the meeting
15:22:32 <mgoddard> sure. Good info though, keep it up (and others too please help populate it)
15:22:44 <yoctozepto> others random
15:22:59 <yoctozepto> this I find bizarre: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_memtotal_mb'
15:23:18 <yoctozepto> this must be cosmic rays effects
15:23:25 <mgoddard> yeah, weird
15:23:28 <yoctozepto> let's go on with the meeting, please, mgoddard
15:23:37 <mgoddard> ok
15:23:45 <mgoddard> just to finish off the whiteboard
15:24:11 <mgoddard> if you are working on a feature or larger piece of work, please keep progress up to date on the whiteboard
15:24:20 <mgoddard> particularly for Train priorities
15:24:30 <mgoddard> #topic Stein release status
15:24:36 <mgoddard> I think we're getting close
15:25:00 <mgoddard> been a bit busy the last couple of days, but I think I'm converging on a solution for the galera issues
15:25:10 <yoctozepto> great!
15:25:34 <openstackgerrit> Mark Goddard proposed openstack/kolla-ansible stable/stein: DNM: try waiting for mariadb to become operational  https://review.opendev.org/668514
15:25:51 <mgoddard> ^ there it is... git review was patiently waiting for a yes/no
15:26:04 <yoctozepto> mgoddard: loyal servant
15:26:20 <mgoddard> if that works I'll squash it into the main bugfix patch
15:26:52 <mgoddard> the other main issue that concerns me is bifrost upgrades sometimes fail, fixed by https://review.opendev.org/667665
15:27:34 <mgoddard> although we merged a workaround in kayobe so we're at least unblocked there
15:27:46 * yoctozepto yoctozepto concerned about nova discovery too
15:27:53 <mgoddard> that's true
15:28:06 * yoctozepto asking whether you prefer him to finish the patch or leave it entirely to you
15:28:46 <mgoddard> I haven't looked at it today, do you think it needs much more work?
15:29:10 <yoctozepto> nope, not much, straigthen the naming
15:29:13 <yoctozepto> and fix ironic
15:29:23 <yoctozepto> probably only regex replace is wrong there
15:29:34 <yoctozepto> I linked a test for you
15:29:58 <yoctozepto> #link https://review.opendev.org/668742
15:30:18 <henriqueof> What's the problem iwth Ironic? Can someone point me the issue link?
15:30:35 <henriqueof> Oh, nevermind, thanks!
15:30:36 <openstackgerrit> gugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed  https://review.opendev.org/668773
15:30:45 <openstackgerrit> gugug proposed openstack/kolla-ansible master: Update the UPPER_CONSTRAINTS_FILE to releases.openstack.org  https://review.opendev.org/668777
15:31:08 <mgoddard> henriqueof: I think yoctozepto is referring to a problem with ironic in https://review.opendev.org/#/c/668623
15:31:29 <yoctozepto> mgoddard, henriqueof: exactly
15:31:30 <mgoddard> stein seems to go on and on
15:31:50 <henriqueof> mgoddard: Thanks!
15:31:51 <mgoddard> OTOH I think it will be our best tested release
15:31:53 <yoctozepto> mgoddard: better that than stagnation
15:32:07 <henriqueof> I am really looking forward to a Stein deployment in the nex weeks.
15:32:41 <mgoddard> henriqueof: great, do feed back any issues
15:32:47 <mgoddard> #topic IPv6 community goal
15:33:04 <mgoddard> I added this because yoctozepto offered to pick up the IPv6 work
15:33:08 <henriqueof> Well, it'll be a production so I need to know what problems I am expected to find,
15:33:13 <henriqueof> Before deploying.
15:33:17 <yoctozepto> mgoddard: I did
15:33:20 <mgoddard> #link https://blueprints.launchpad.net/kolla-ansible/+spec/ipv6-control-plane
15:33:45 <henriqueof> Also, I think the best way is using the source from head branch, right?
15:34:05 <yoctozepto> not yet tried coordinating with the other projects
15:34:06 <mgoddard> henriqueof: for the stein release use stable/stein
15:34:19 <yoctozepto> as for us we rely a lot on ipv4 ansible vars
15:34:22 <mgoddard> yoctozepto: I think this will be key
15:34:30 <yoctozepto> so that's the main issue to overcome
15:34:33 <mgoddard> also with Tripleo and OSA
15:34:36 <yoctozepto> there was a patch for fallback
15:34:47 <yoctozepto> cloudnull: TripleO ^
15:34:49 <mgoddard> cloudnull is well placed in both projects
15:35:09 * cloudnull reading back
15:35:23 <yoctozepto> #link https://review.opendev.org/575994
15:35:24 <mgoddard> cloudnull: IPv6 cross community
15:35:40 <henriqueof> I wanted to try TripleO but I don't have enough resources now to keep a undercloud running.
15:35:41 <yoctozepto> mgoddard: no OSA agents around?
15:36:02 <mgoddard> yoctozepto: cloudnull knows a thing or two about OSA
15:36:05 * cloudnull former agent of OSA
15:36:27 <yoctozepto> geez, you are a double agent already
15:36:35 <yoctozepto> we are making you triple now
15:36:38 <cloudnull> tripleo agent :P
15:36:39 <yoctozepto> ;D
15:36:44 <yoctozepto> hahahaha
15:36:46 <yoctozepto> good one
15:37:08 <mgoddard> I think a mailing list post will be a good idea, perhaps once we know what questions to ask
15:37:33 <yoctozepto> sure, I can handle the communication
15:37:40 <mgoddard> cloudnull: do you know what the current state of IPv6 is in Tripleo?
15:37:45 <mgoddard> or OSA for that matter
15:38:15 <cloudnull> I do not know what the current state is for TripleO
15:38:33 <henriqueof> yoctozepto: double agent? That was for who? lol
15:38:50 <yoctozepto> guys, I must be going now
15:39:03 <yoctozepto> henriqueof: to cloudnull
15:39:12 <cloudnull> OSA supports ipv6 in instances and has for a very long time, IPv6 infra works too, though only for physical host and lxc based deployments
15:39:21 <yoctozepto> sorry for leaving
15:39:26 <yoctozepto> will read backlog
15:39:29 <yoctozepto> see ya
15:39:44 <mgoddard> bye yoctozepto, thanks for attending
15:40:25 <cloudnull> the general rule in OSA was to do a dual stack network, I suspect that works for most deployments regardless of tech / deployment
15:40:28 <mgoddard> I've seen kolla-based deployments with IPv6 for instances, but currently we're tied to v4 for infra
15:41:18 <mgoddard> cloudnull: what would you put in the endpoint catalogue? (assuming no DNS)
15:41:42 <cloudnull> we'd assume DNS, especially when wanting to use ipv6
15:41:52 <mgoddard> ok
15:42:10 <cloudnull> otherwise [xx:yy:zz] should work ?
15:42:18 <cloudnull> though Ive never tried.
15:42:42 <mgoddard> just thinking at that point you'd need to choose between v4 or v6
15:43:00 <henriqueof> Okay, so no known critical problems for current stable/stein release, right? I can use for a production deployment, iright?
15:43:35 <mgoddard> given that yoctozepto has left let's move on from IPv6
15:43:41 <mgoddard> #topic open discussion
15:43:58 <mgoddard> let's start with henriqueof's last question
15:44:03 <mgoddard> > Okay, so no known critical problems for current stable/stein release, right? I can use for a production deployment, iright?
15:44:17 <mgoddard> henriqueof: strictly, kolla hasn't released stein yet
15:45:07 <mgoddard> henriqueof: it has a couple of issues we discussed earlier in the meeting
15:45:14 <mgoddard> the main one affecting upgrades
15:45:32 <mgoddard> anyone else have anything to discuss?
15:46:39 <mgoddard> thanks for attending al
15:46:43 <henriqueof> mgoddard: Thanks very much, so no official kolla release therefore, unmet depencency for kolla-ansible and that makes a no production ready version.
15:47:08 <mgoddard> henriqueof: I expect it to land in the next week, assuming nothing else goes wrong :)
15:47:29 <henriqueof> Oh great! I'll wait then.
15:47:39 <mgoddard> #emdmeeting
15:50:13 <ohwhyosa[m]> typo
15:50:30 <ohwhyosa[m]> mgoddard:
15:51:05 <ohwhyosa[m]> Didn't want to bother during the meeting, can we comment during them (related stuff, I mean)
15:58:17 <mgoddard> ohwhyosa[m]: if its related, please feel free to get involved
15:58:52 <mgoddard> ohwhyosa[m]: if it's unrelated please either ask in 'open discussion' at the end or wait until afterwards
15:59:35 <ohwhyosa[m]> Yes! That I meant
16:00:00 <ohwhyosa[m]> After reading you I believe some of the issues I've had with multinode might be solved with retries
16:00:22 <ohwhyosa[m]> The deployment attempts mainly failled at bootstraping something
16:01:02 <ohwhyosa[m]> oh mgoddard I don't know if it may be on purpose, but you wrote #emdmeeting instead of #endmeeting
16:01:45 <openstackgerrit> Merged openstack/kolla master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/668779
16:19:56 <priteau> #endmeeting