15:01:20 <mgoddard> #startmeeting kolla
15:01:21 <openstack> Meeting started Wed Oct 23 15:01:20 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:24 <mgoddard> #topic rollcall
15:01:24 <openstack> The meeting name has been set to 'kolla'
15:01:27 <mgoddard> \o
15:01:32 <yoctozepto> o/
15:01:42 <generalfuzz> o/
15:02:11 <yoctozepto> noxoid, mloza: contribute to docs :-)
15:02:53 <yoctozepto> mnasiadka: ?
15:03:16 <mloza> yoctozepto: ok once I have successful setup
15:03:26 <mnasiadka> o/
15:03:44 <mgoddard> #topic agenda
15:03:54 <mgoddard> * Roll-call
15:03:57 <mgoddard> * Announcements
15:03:59 <mgoddard> ** OpenStack Queens becomes extended maintenance (EM) on Friday
15:04:01 <mgoddard> * Review action items from last meeting
15:04:03 <mgoddard> * CI status
15:04:05 <mgoddard> * Train release planning
15:04:07 <mgoddard> * Review priorities
15:04:09 <mgoddard> * Idea: community meetings
15:04:11 <mgoddard> * Which images should we mark as maintained in the support matrix?
15:04:13 <mgoddard> * Reducing load of build & publishing jobs on CI & Dockerhub (continued) https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:04:15 <mgoddard> #topic announcements
15:04:31 <mgoddard> #info OpenStack Queens becomes extended maintenance (EM) on Friday
15:04:45 <mgoddard> We will then disable our Queens publishing jobs
15:04:54 <mgoddard> Any others?
15:05:38 <yoctozepto> na-ah
15:05:46 <mgoddard> #topic Review action items from last meeting
15:05:51 <openstackgerrit> Mark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/690598
15:06:04 <mgoddard> yoctozepto to fix tacker & NFV scenario job
15:06:07 <mgoddard> mnasiadka to look at not building or publishing deprecated images
15:06:20 <mgoddard> yoctozepto: done, right?
15:06:29 <mgoddard> mnasiadka: not done?
15:06:32 <yoctozepto> fixed, though ugly, I want to check if this glance store works like the glance api's store
15:06:46 <yoctozepto> then we could just apply analogous config
15:06:51 <mnasiadka> Haven’t looked, focused on fixing RDO and UCA ;)
15:06:54 <yoctozepto> and be done with ceph backend
15:07:15 <yoctozepto> i'll treat it as a bug fix if you don't mind :-)
15:07:23 <yoctozepto> probably be done before final release
15:07:29 <yoctozepto> so not that breaking
15:08:18 <mgoddard> ok
15:08:27 <mgoddard> #topic CI status
15:08:27 <yoctozepto> paraphrasing, yoctozepto looking at keeping ha for tacker-conductor
15:08:49 <yoctozepto> green, fixing ubuntu binary just today
15:09:00 <mgoddard> nice
15:09:22 <mgoddard> kayobe is currently broken due to the inventory change for cells
15:09:25 <mgoddard> patch in review
15:09:26 <yoctozepto> I noticed kayobe still has some struggle?
15:09:29 <yoctozepto> ah
15:10:45 <mgoddard> do we still have issues with ceph keys in CI?
15:11:33 <yoctozepto> mnasiadka: ^
15:11:37 <yoctozepto> I don't think so
15:11:47 <mnasiadka> Were there any?
15:12:02 <mnasiadka> Ah, ubuntu python3
15:12:06 <mgoddard> I kept seeing some a while ago
15:12:12 <mnasiadka> No, runs smoothly now
15:12:24 <yoctozepto> mnasiadka: https://review.opendev.org/688870 - while you around
15:12:29 <yoctozepto> (shameless plug)
15:12:34 <mgoddard> doesn't seem to be an issue
15:12:36 <yoctozepto> let's keep going
15:12:48 <mgoddard> ok cool
15:12:54 <mgoddard> #topic Train release planning
15:13:23 * yoctozepto approving mgoddard's patches
15:13:28 <mgoddard> I pushed up a bunch of patches on Monday
15:13:39 <yoctozepto> oh, he did
15:13:47 <mgoddard> still have https://review.opendev.org/688973
15:14:17 <mgoddard> still waiting on RDO
15:14:24 <yoctozepto> mnasiadka: ^^
15:14:59 <mgoddard> https://review.opendev.org/#/c/689706/
15:15:21 <mgoddard> well, I think we're waiting on a tripleoclient release
15:15:41 <mnasiadka> Yup
15:15:46 <yoctozepto> INFO:kolla.common.utils.tripleoclient:No package openstack-tripleo-validations available.
15:15:54 <mgoddard> after that we can branch
15:16:02 <mnasiadka> Today it will be there
15:16:05 <mgoddard> ok
15:16:07 <yoctozepto> great
15:16:47 <mgoddard> Let's try to work out which bug fixes we want in the release
15:16:52 <mgoddard> #link https://launchpad.net/kolla/+milestone/9.0.0
15:16:56 <mgoddard> starting with kolla
15:17:25 <mgoddard> top 3 in progress
15:17:34 <mgoddard> yoctozepto: is rally/trove complete?
15:18:17 <mnasiadka> yes it is, trove fixed in UCA, we disabled building rally because it's not in UCA
15:18:49 <mnasiadka> ah not yet
15:18:54 <mnasiadka> change is at the gates, rechecked :)
15:19:09 <mgoddard> cool
15:19:20 <mgoddard> I have patches for mariabackup
15:19:24 <mnasiadka> it has Closes-Bug, so will close it.
15:19:40 <mgoddard> I will revive that mariadb CI job for moar testing
15:19:45 <yoctozepto> complete
15:19:58 <yoctozepto> moar CI failures
15:20:05 <mgoddard> then we're down to 'apt_sources_list option does not work', which I think we can take or leave
15:20:14 <yoctozepto> drop that
15:20:19 <yoctozepto> that's 4 years lol
15:20:24 <yoctozepto> (almost)
15:20:27 <mgoddard> now onto our more difficult child
15:20:31 <mgoddard> #link https://launchpad.net/kolla-ansible/+milestone/9.0.0
15:20:46 <mnasiadka> mgoddard: I think hrw fixed the apt_sources thing?
15:20:56 <mgoddard> mnasiadka: there is a patch, I had a comment
15:21:10 <mgoddard> if it is updated & merged, it gets in
15:21:31 <mgoddard> starting with 'in progress' bugs
15:21:41 <mgoddard> mariabackup, ditto above
15:22:03 <mgoddard> quite a few of these have been open for a while
15:22:41 <mgoddard> #link https://bugs.launchpad.net/kolla-ansible/+bug/1830724
15:22:41 <openstack> Launchpad bug 1830724 in kolla-ansible train "fluentd not reconnecting to ES on failures" [Medium,In progress] - Assigned to Krzysztof Klimonda (kklimonda)
15:24:00 <mgoddard> Has not been updated in some time
15:24:12 <mgoddard> I'd guess it's not going to land
15:24:17 <mgoddard> I've commented but will move it out
15:24:39 <mnasiadka> I think there's a change a bit abandoned, and I don't know how it relates to current state after moving to td-agent
15:25:19 <mgoddard> it seems like a useful change though
15:25:35 <mgoddard> I certainly know some people who need it or something like it
15:25:47 <mgoddard> let's see if they reply
15:25:54 <mgoddard> #link https://bugs.launchpad.net/kolla-ansible/+bug/1833064
15:25:54 <openstack> Launchpad bug 1833064 in kolla-ansible train "nova-status upgrade check should be run after DB sync and online migrations" [Medium,In progress] - Assigned to Mark Goddard (mgoddard)
15:26:12 <mgoddard> this was a little controversial
15:26:34 <mgoddard> easy to change, but does make the check less useful
15:27:17 <mgoddard> #link https://bugs.launchpad.net/bugs/1845629
15:27:17 <openstack> Launchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac)
15:27:20 <mnasiadka> mgoddard: we can always run it before and after
15:27:28 <mnasiadka> (if it makes any sense)
15:27:33 <mgoddard> I'd like to get that one in, I marked it RP+1
15:28:20 <mnasiadka> yoctozepto: can you look at this change? :)
15:28:57 <mgoddard> we have a patch for https://bugs.launchpad.net/bugs/1846467
15:28:57 <openstack> Launchpad bug 1846467 in kolla-ansible "RabbitMQ container - high CPU load on multicore systems" [Medium,In progress] - Assigned to Jan Vondra (janvondra)
15:29:02 <mgoddard> do we want it?
15:29:04 <mgoddard> (in train)
15:29:20 <mgoddard> it's kind of a feature, but could be used to resolve that bug
15:29:47 <yoctozepto> mgoddard: I can handle rabbitmq
15:29:52 <yoctozepto> mnasiadka: look at which one
15:30:03 <mnasiadka> yoctozepto: fluentd mariadb
15:30:06 <mgoddard> yoctozepto: thanks
15:30:26 <mnasiadka> mgoddard: rabbitmq needs some release notes and maybe some docs - after that I think the code is fine.
15:30:32 <mnasiadka> ah, yoctozepto took this, good
15:30:59 <mnasiadka> yoctozepto: https://review.opendev.org/#/c/686428/
15:31:13 <mgoddard> dougsz was also going to look at https://bugs.launchpad.net/bugs/1845629 (https://review.opendev.org/686428)
15:31:13 <openstack> Launchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac)
15:31:31 <mgoddard> as he is fluent in fluentd
15:31:40 <mnasiadka> good
15:31:51 <mnasiadka> mgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/1673944 - approval to move milestone to 10.0 as this is a Wishlist?
15:31:51 <openstack> Launchpad bug 1673944 in kolla-ansible "external-ceph task for nova assumes nova/cinder ceph user" [Wishlist,Triaged] - Assigned to Michal Nasiadka (mnasiadka)
15:32:03 <mgoddard> mnasiadka: +1
15:33:04 <mgoddard> onto triaged bugs
15:33:16 <mgoddard> #link https://bugs.launchpad.net/kolla-ansible/+bug/1846820
15:33:16 <openstack> Launchpad bug 1846820 in kolla-ansible "nova-conductor may crash during deploy due to haproxy-keystone 504" [High,Triaged]
15:33:20 <yoctozepto> mnasiadka: ok, I see, where is your review?
15:33:48 <mnasiadka> yoctozepto: in progress :)
15:34:06 <mgoddard> we have marked it as high, nova as low :)
15:34:52 <mgoddard> do we really think it's high?
15:34:59 <mnasiadka> mgoddard: I had a couple of occurrences of keystone 504 when running ceph-ansible CI... I can propose a patch to improve the logging of mod_wsgi, but there won't be magic - it's hard to find the root cause
15:35:26 <mnasiadka> mgoddard: it's the worst type of bug, happens sometimes :)
15:35:44 <mgoddard> we have seen intermittent issues with keystone in CI since enabling vxlan
15:36:12 <mnasiadka> maybe it's a bit vxlan related, but as always it's hard to tell
15:36:25 <mgoddard> it might just have exposed an issue
15:36:41 <yoctozepto> mgoddard: since running haproxy
15:36:43 <mnasiadka> we could improve vxlan testing - like do ping test with near-mtu size
15:36:49 <yoctozepto> I tell you
15:36:56 <yoctozepto> this is timeouts on haproxy
15:36:59 <mnasiadka> and maybe improve haproxy logging to try to isolate the issue
15:37:02 <yoctozepto> we can raise them and see what happens
15:37:26 <yoctozepto> geez, I analyzed that, haproxy timeouts waiting
15:37:36 <yoctozepto> and closes the connection to the backend
15:37:49 <yoctozepto> I just did not get to raising timeout values
15:37:55 <yoctozepto> be my guest :D
15:38:01 <mgoddard> how high are the timeouts currently?
15:38:06 <mnasiadka> yoctozepto: so you raise timeouts, I'll improve logging here and there and let's see :)
15:39:07 <mgoddard> 10s for some, 1m for others
15:39:24 <mgoddard> sounds like mnasiadka and yoctozepto will investigate
15:39:44 <yoctozepto> mgoddard: where did you get that? ;p
15:40:08 <mnasiadka> yoctozepto: you wanted to raise haproxy timeouts, we saw that in your eyes :D
15:40:10 <mgoddard> you seem to have it in hand :)
15:40:19 <yoctozepto> I meant the timeouts, guys
15:40:37 <mnasiadka> yes, raise the bloody timeouts, I'll make sure if it happens again - we have proper logs.
15:40:43 <mnasiadka> agreed?
15:41:19 <mgoddard> I think the relevant timeout is 1m
15:41:30 <yoctozepto> ah, geez, I read 1h
15:41:38 <yoctozepto> and was like wtf
15:41:52 <yoctozepto> mnasiadka: more than agreed
15:41:54 <mgoddard> that sounds like it should be sufficient. maybe it is blocked in the backend
15:42:12 <yoctozepto> mgoddard: the relevant is 10s
15:42:30 <yoctozepto> at least that what it looked like
15:42:33 <yoctozepto> when I checked
15:42:33 <mnasiadka> connect and check timeout, yeah
15:42:36 <mgoddard> sure? 504 should be server timeout
15:42:55 <mgoddard> anyway, needs investigation
15:43:04 <yoctozepto> yeah, needs, not remember
15:43:11 <mgoddard> what about this one: https://bugs.launchpad.net/kolla-ansible/+bug/1845244
15:43:11 <openstack> Launchpad bug 1845244 in kolla-ansible "Nova scheduler is stopped after each reboot" [High,New]
15:43:18 <yoctozepto> but it's a timeout, no other relevant errors in there
15:43:23 <mgoddard> not targeted for train currently, but it is 'high'
15:43:30 <yoctozepto> mgoddard: yeah, it's sad
15:43:35 <yoctozepto> i'm stuck with it
15:43:44 <yoctozepto> it just drops out immediately
15:43:47 <mgoddard> I'll target to train to get it on our radar
15:43:53 <yoctozepto> sure, be my guest
15:44:01 <yoctozepto> but we need someone to figure this one out
15:44:15 <yoctozepto> I am clearly missing something "obvious"
15:45:15 <mnasiadka> it happens only in stein?
15:45:32 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: Zun: fix Cinder (volume) iSCSI support  https://review.opendev.org/690614
15:45:39 <yoctozepto> mnasiadka: looks like since stein
15:45:55 <yoctozepto> but not checked with current train
15:45:58 <mnasiadka> well, out of curiosity I can look into that
15:46:03 <yoctozepto> nor rocky for the record :D
15:46:06 <yoctozepto> ok
15:46:44 <mgoddard> I have a VM up, I'll give it a poke
15:47:21 <mnasiadka> yoctozepto: can you put in docker engine version in the bug?
15:47:58 <mgoddard> are there other k-a bugs we should be focussing on for train?
15:48:03 <yoctozepto> 19.03.2
15:49:04 <mgoddard> or any other train planning things we should discuss?
15:49:17 <generalfuzz> perhaps TLS?
15:49:28 <mgoddard> ah, hi generalfuzz
15:50:12 <mgoddard> I think you did not get the promised reviews
15:50:25 <mgoddard> strictly, we did say no more features after last friday
15:50:26 <mnasiadka> yeah, just got a heart attack because I forgot those
15:51:25 <generalfuzz> yikes - lower the priority then
15:51:45 <mnasiadka> we're talking about this change https://review.opendev.org/#/c/686024/ ?
15:51:49 <generalfuzz> yes
15:51:49 <mnasiadka> or there's another one?
15:51:57 <mnasiadka> it's in merge conflict
15:52:04 <mnasiadka> so what's there to review?
15:52:47 <mgoddard> sure, it needs an update. probably due to cells & ipv6
15:52:55 <generalfuzz> I'll update asap
15:52:57 <mnasiadka> I think it had the same problem end of last week
15:52:58 <yoctozepto> cells more likely
15:53:02 <mnasiadka> but maybe I'm wrong
15:53:02 <yoctozepto> ipv6 unlikely to conflict
15:53:05 <mgoddard> I think we'll have to push this one out to Ussuri generalfuzz, sorry
15:53:26 <generalfuzz> ok, so it goes
15:53:39 <mgoddard> we need to be putting review time into bugs right now
15:53:58 <generalfuzz> understand
15:54:40 <mgoddard> speak up once things have calmed down and we'll try to merge it early on in Ussuri
15:55:05 <mgoddard> 6 minutes left, let's move on
15:55:08 <mgoddard> #topic Idea: community meetings
15:55:31 <mgoddard> I was speaking to osmanlicilegi who was interested in attending the kolla PTG
15:55:41 <mgoddard> but also suggested having some community meetings
15:55:50 <mgoddard> I think it's a nice idea
15:56:13 <mgoddard> less development focussed, more about (human) networking
15:56:21 <yoctozepto> +1
15:56:28 <mgoddard> it could be a good way to onboard new contributors
15:56:33 <yoctozepto> +1
15:57:11 <mgoddard> I also thought it might be a good idea if people are interested to do a presentation of the onboarding slides I gave in Denver, but via video conference
15:57:14 <mgoddard> #link https://docs.google.com/presentation/d/11gGW93Xu7DQo_G1LiRDm6thfB5gNLm39SHuKcgSW8FQ/edit
15:57:23 <mgoddard> could be a separate thing
15:57:36 <mgoddard> but it has information on how to get more involved
15:57:48 <mgoddard> thoughts?
15:57:58 <mgoddard> how often should such a meeting be held?
15:58:06 <mgoddard> who would join?
15:58:49 <scottsol> a couple of us would :)
15:59:11 <mgoddard> great
15:59:29 <mgoddard> I assume osmanlicilegi would
15:59:36 <mnasiadka> mgoddard: maybe let's try to have it as part of the again virtual PTG?
15:59:39 <yoctozepto> nice presentation
15:59:44 <yoctozepto> kayobe needs updating :D
15:59:54 <mgoddard> yeah
15:59:57 <yoctozepto> mnasiadka: or semivirtual
16:00:01 <yoctozepto> if you come
16:00:12 <mnasiadka> yeah, we'll see - I need to talk to the boss
16:00:48 <mgoddard> I was thinking an hour, once a month
16:00:55 <mgoddard> too much? too little?
16:01:29 <mnasiadka> mgoddard: let's start with having an hour as part of kolla ptg, and then try to schedule another one - month later, and see what is the attendance?
16:01:31 <yoctozepto> mgoddard: we are always in hurry with an hour
16:01:50 <mnasiadka> or something like that
16:02:21 <yoctozepto> sounds good
16:02:43 <mgoddard> ok, time at the bar
16:02:45 <mgoddard> thanks all
16:02:48 <mgoddard> #endmeeting