14:00:33 <mgoddard> #startmeeting kayobe
14:00:34 <openstack> Meeting started Mon Jun 24 14:00:33 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:37 <mgoddard> #topic rollcall
14:00:39 <openstack> The meeting name has been set to 'kayobe'
14:00:43 <mgoddard> \o
14:01:15 <priteau> o/
14:01:44 <mgoddard> around dougsz & jovial[m] ?
14:01:53 <priteau> jovial[m] might be busy
14:01:54 <dougsz> \o
14:01:58 <mgoddard> sure
14:02:08 <mgoddard> #topic agenda
14:02:18 <mgoddard> * Roll-call
14:02:20 <mgoddard> * Announcements
14:02:22 <mgoddard> ** Vote on kayobe becoming a kolla deliverable: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html
14:02:24 <mgoddard> * Review actions from last meeting
14:02:26 <mgoddard> * Kayobe whiteboard https://etherpad.openstack.org/p/kayobe-whiteboard
14:02:28 <mgoddard> * Discussion
14:02:30 <mgoddard> ** Stein release https://etherpad.openstack.org/p/kayobe-stein-release
14:02:32 <mgoddard> * Open Discussion
14:02:34 <mgoddard> #topic announcements
14:02:37 <mgoddard> #info Vote on kayobe becoming a kolla deliverable
14:02:42 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html
14:02:59 <mgoddard> open to kolla cores, but opinions from others in the community welcome
14:03:04 <mgoddard> any other announcements?
14:03:31 <dougsz> None from me
14:03:43 <priteau> Nope
14:04:11 <mgoddard> #topic Review actions from last meeting
14:04:32 <mgoddard> mgoddard to enquire about GitHub repository transfers
14:04:34 <mgoddard> mgoddard to schedule bug review and squash session
14:04:41 <mgoddard> bad mgoddard, didn't do either
14:05:11 <mgoddard> given that we are potentially close to becoming official through kolla, I don't think we need to do 1
14:05:24 <mgoddard> #action mgoddard to schedule bug review and squash session
14:05:26 <priteau> Agreed
14:05:29 <dougsz> +1
14:05:40 <mgoddard> #topic Kayobe whiteboard https://etherpad.openstack.org/p/kayobe-whiteboard
14:06:34 <mgoddard> Can't get to zuul builds results for some reason
14:07:03 <priteau> They can be very slow to load
14:07:09 <mgoddard> got there eventually
14:08:10 <mgoddard> main issue seems to be semi-regular failures in kayobe-seed-upgrade-centos
14:08:13 <mgoddard> http://zuul.openstack.org/builds?project=x%2Fkayobe&job_name=kayobe-seed-upgrade-centos
14:08:48 <mgoddard> I think it's a mariadb upgrade issue, probably need to be better at gracefully stopping the mariadb service
14:09:09 <mgoddard> http://zuul.openstack.org/builds?job_name=kayobe-seed-upgrade-centos
14:09:13 <mgoddard> that's a better link
14:09:36 <mgoddard> only seems to fail on master
14:09:58 <mgoddard> I'll try to investigate
14:10:11 <priteau> There were some failure on stable branches on 2019-06-17
14:10:12 <mgoddard> #action mgoddard to investigate errors in kayobe-seed-upgrade-centos job
14:10:19 <mgoddard> was that due to cliff?
14:10:24 <priteau> Oh, that was something else
14:10:26 <priteau> yeah, cliff
14:11:15 <mgoddard> thanks for updating the cumulus patch priteau
14:11:40 <priteau> It still needs more work but I haven't found the time to work on it lately
14:11:42 <mgoddard> hold up, actually you just commented :)
14:12:13 <mgoddard> we have quite a few patches requiring review
14:13:45 <mgoddard> http://tiny.cc/kayobe-review-dashboard
14:13:46 <priteau> dougsz just added a good comment on the custom blazar configuration
14:14:06 <dougsz> Just wondering about that in general
14:14:15 <dougsz> The kolla_extra_{service} thing
14:15:10 <dougsz> Should we only use it if Kayobe itself provides some custom configuration to Kolla Ansible?
14:15:16 <mgoddard> https://tiny.cc/kayobe-review-dash
14:15:37 <dougsz> Nice ^!
14:15:52 <mgoddard> dougsz: ideally, yes. But at the moment it's necessary for files like kolla/config/blazar.conf
14:16:55 <mgoddard> #topic discussion
14:17:05 <mgoddard> #topic Stein release https://etherpad.openstack.org/p/kayobe-stein-release
14:17:05 <priteau> It looks more complex that it should be for services that kayobe doesn't customise
14:17:10 <mgoddard> #undo
14:17:11 <openstack> Removing item from minutes: #topic Stein release https://etherpad.openstack.org/p/kayobe-stein-release
14:17:28 <mgoddard> it definitely is more complicated than it needs to be
14:17:51 <priteau> Just to make sure I understand correctly how it works: the operator writes custom configuration in etc/kayobe/kolla/<service>.conf, which gets moved to a variable, which gets templated into etc/kolla/config/<service.conf>
14:18:17 <mgoddard> I started this but didn't finish :(
14:18:19 <mgoddard> https://github.com/markgoddard/kayobe/commit/f069ea53fb4b07d5ed379ddb85b783f67da68135
14:18:58 <mgoddard> priteau: correct, except operator writes to etc/kayobe/kolla/config/<service>.conf
14:19:15 <priteau> Right, that's what I meant :)
14:19:44 <priteau> And it's a bit complicated because the same code is used for files that kayobe also customises, such as ironic.conf?
14:19:51 <mgoddard> yes
14:20:35 <priteau> Is your WIP commit just doing a plain copy or template for files that Kayobe doesn't customise?
14:20:44 <mgoddard> it's still a template
14:20:55 <mgoddard> I keep wondering if we just want a plain copy mode
14:21:03 <mgoddard> might be quicker
14:21:06 <dougsz> Ah, no templating
14:21:09 <mgoddard> yeah
14:21:10 <dougsz> That could be nice
14:21:26 <dougsz> Particularly for Prometheus config
14:21:32 <mgoddard> just another if to worry about though
14:21:41 <dougsz> which uses go templates, so there is some awkward escaping
14:21:51 <dougsz> Yeah, not a big issue
14:22:26 <mgoddard> simple fix is just wrap the whole file in {% raw %}
14:22:30 <priteau> Could we lose some capabilities if we stop templating on the kayobe side?
14:22:38 <mgoddard> yes
14:22:52 <mgoddard> sometimes it's useful to access kayobe variables
14:22:55 <dougsz> Yeah, but you can't use the {% raw %} thing twice
14:23:13 <priteau> I've once tried to template my ceph.conf file to dynamically gather IPs of the ceph mon servers, but didn't manage to make it work...
14:23:34 <mgoddard> we could make it easier to pass through variables to kolla-ansible, then access those
14:24:43 <dougsz> (example of work around for escaping twice: https://github.com/RSE-Cambridge/cumulus-kayobe-config/commit/307cd1129b57394fb0eb83d8afdfd2a8dcd9e9b3)
14:25:41 <mgoddard> ouch
14:25:53 <priteau> {% endraw %}{{ '{% endraw %}' }}
14:26:01 <mgoddard> so that's escaping for kayobe & kolla jinja?
14:26:17 <dougsz> Yeah
14:26:19 <priteau> inception templating levels
14:26:21 <mgoddard> nice
14:27:32 <dougsz> Maybe I could add some documentation on double escaping, escaping etc and that would be enough for now
14:27:35 <mgoddard> I suppose passing through variables is quite easy if you use globals.yml
14:27:50 <dougsz> that's true
14:28:16 <priteau> I propose that we leave it as it is for now and take it as discussion for a design session
14:28:22 <mgoddard> +1
14:28:35 <mgoddard> #topic Stein release https://etherpad.openstack.org/p/kayobe-stein-release
14:28:41 <mgoddard> This one again
14:29:02 <mgoddard> kolla release is still not quite there :(
14:29:27 <mgoddard> I think we are nearly on top of the last ceph issue
14:29:35 <mgoddard> now there is a galera upgrade bug
14:30:20 <mgoddard> but I will try to crack it soon as we do need to get this thing out
14:31:18 <mgoddard> #topic open discussion
14:31:25 <mgoddard> Anyone have anything for today?
14:32:19 <priteau> There are some small patches pending review
14:32:30 <priteau> Not important ones, but easy
14:32:38 <priteau> https://review.opendev.org/#/c/664870/
14:32:56 <priteau> https://review.opendev.org/#/c/662483/
14:33:07 <priteau> https://review.opendev.org/#/c/660973/
14:33:44 <mgoddard> +1
14:33:52 <mgoddard> or +2 even ;)
14:34:42 <mgoddard> One other thing, we should probably be a bit more systematic about backports
14:35:20 <mgoddard> In kolla I've started a ~monthly backport, and keep the last checked commit in the whiteboard
14:36:03 <priteau> What would be our backport policy? Important bug fixes obviously, but what about documentation fixes or small features?
14:36:10 <dougsz> Sounds like a good plan to track them, it's easy to forget
14:36:28 <priteau> Do you think anyone reads the stable docs?
14:37:03 <mgoddard> I try to use the correct version of docs for other openstack projects. I don't so much for kayobe
14:37:45 <mgoddard> +1 for docs fixes. Small features maybe, if there's a clear benefit and minimal risks
14:38:16 <mgoddard> if we become official we'll probably have to stick to the openstack stable backport policy
14:38:18 <priteau> That's what I do as well. Though looking at the latest docs for Kayobe could be unhelpful for things like which kolla configuration is copied
14:38:46 <mgoddard> it's always a risk - docs may improve over time, or go stale
14:38:58 <priteau> #link https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes
14:39:35 <priteau> #link https://docs.openstack.org/project-team-guide/stable-branches.html#review-guidelines
14:40:07 <mgoddard> thanks
14:40:20 <priteau> The best would be if we could support a few older OpenStack releases from the latest Kayobe version
14:40:56 <priteau> This way operators could upgrade to kayobe stein to get the new features, while keeping their OpenStack on Rocky or Queens. But it's a lot of extra testing...
14:41:29 <mgoddard> yeah, could be difficult
14:41:41 <mgoddard> still fairly coupled to the kolla version
14:42:26 <mgoddard> mainly through the inventory
14:42:41 <mgoddard> if we kept old inventories around it could work
14:44:10 <mgoddard> anything else for this week?
14:44:23 <priteau> Not from me
14:44:34 <mgoddard> ok, have 15 minutes back :)
14:44:40 <mgoddard> thanks priteau and dougsz
14:44:43 <mgoddard> #endmeeting