15:01:20 #startmeeting kolla 15:01:21 Meeting started Wed Oct 23 15:01:20 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:24 #topic rollcall 15:01:24 The meeting name has been set to 'kolla' 15:01:27 \o 15:01:32 o/ 15:01:42 o/ 15:02:11 noxoid, mloza: contribute to docs :-) 15:02:53 mnasiadka: ? 15:03:16 yoctozepto: ok once I have successful setup 15:03:26 o/ 15:03:44 #topic agenda 15:03:54 * Roll-call 15:03:57 * Announcements 15:03:59 ** OpenStack Queens becomes extended maintenance (EM) on Friday 15:04:01 * Review action items from last meeting 15:04:03 * CI status 15:04:05 * Train release planning 15:04:07 * Review priorities 15:04:09 * Idea: community meetings 15:04:11 * Which images should we mark as maintained in the support matrix? 15:04:13 * Reducing load of build & publishing jobs on CI & Dockerhub (continued) https://etherpad.openstack.org/p/kolla-train-image-evaluation 15:04:15 #topic announcements 15:04:31 #info OpenStack Queens becomes extended maintenance (EM) on Friday 15:04:45 We will then disable our Queens publishing jobs 15:04:54 Any others? 15:05:38 na-ah 15:05:46 #topic Review action items from last meeting 15:05:51 Mark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups https://review.opendev.org/690598 15:06:04 yoctozepto to fix tacker & NFV scenario job 15:06:07 mnasiadka to look at not building or publishing deprecated images 15:06:20 yoctozepto: done, right? 15:06:29 mnasiadka: not done? 15:06:32 fixed, though ugly, I want to check if this glance store works like the glance api's store 15:06:46 then we could just apply analogous config 15:06:51 Haven’t looked, focused on fixing RDO and UCA ;) 15:06:54 and be done with ceph backend 15:07:15 i'll treat it as a bug fix if you don't mind :-) 15:07:23 probably be done before final release 15:07:29 so not that breaking 15:08:18 ok 15:08:27 #topic CI status 15:08:27 paraphrasing, yoctozepto looking at keeping ha for tacker-conductor 15:08:49 green, fixing ubuntu binary just today 15:09:00 nice 15:09:22 kayobe is currently broken due to the inventory change for cells 15:09:25 patch in review 15:09:26 I noticed kayobe still has some struggle? 15:09:29 ah 15:10:45 do we still have issues with ceph keys in CI? 15:11:33 mnasiadka: ^ 15:11:37 I don't think so 15:11:47 Were there any? 15:12:02 Ah, ubuntu python3 15:12:06 I kept seeing some a while ago 15:12:12 No, runs smoothly now 15:12:24 mnasiadka: https://review.opendev.org/688870 - while you around 15:12:29 (shameless plug) 15:12:34 doesn't seem to be an issue 15:12:36 let's keep going 15:12:48 ok cool 15:12:54 #topic Train release planning 15:13:23 * yoctozepto approving mgoddard's patches 15:13:28 I pushed up a bunch of patches on Monday 15:13:39 oh, he did 15:13:47 still have https://review.opendev.org/688973 15:14:17 still waiting on RDO 15:14:24 mnasiadka: ^^ 15:14:59 https://review.opendev.org/#/c/689706/ 15:15:21 well, I think we're waiting on a tripleoclient release 15:15:41 Yup 15:15:46 INFO:kolla.common.utils.tripleoclient:No package openstack-tripleo-validations available. 15:15:54 after that we can branch 15:16:02 Today it will be there 15:16:05 ok 15:16:07 great 15:16:47 Let's try to work out which bug fixes we want in the release 15:16:52 #link https://launchpad.net/kolla/+milestone/9.0.0 15:16:56 starting with kolla 15:17:25 top 3 in progress 15:17:34 yoctozepto: is rally/trove complete? 15:18:17 yes it is, trove fixed in UCA, we disabled building rally because it's not in UCA 15:18:49 ah not yet 15:18:54 change is at the gates, rechecked :) 15:19:09 cool 15:19:20 I have patches for mariabackup 15:19:24 it has Closes-Bug, so will close it. 15:19:40 I will revive that mariadb CI job for moar testing 15:19:45 complete 15:19:58 moar CI failures 15:20:05 then we're down to 'apt_sources_list option does not work', which I think we can take or leave 15:20:14 drop that 15:20:19 that's 4 years lol 15:20:24 (almost) 15:20:27 now onto our more difficult child 15:20:31 #link https://launchpad.net/kolla-ansible/+milestone/9.0.0 15:20:46 mgoddard: I think hrw fixed the apt_sources thing? 15:20:56 mnasiadka: there is a patch, I had a comment 15:21:10 if it is updated & merged, it gets in 15:21:31 starting with 'in progress' bugs 15:21:41 mariabackup, ditto above 15:22:03 quite a few of these have been open for a while 15:22:41 #link https://bugs.launchpad.net/kolla-ansible/+bug/1830724 15:22:41 Launchpad bug 1830724 in kolla-ansible train "fluentd not reconnecting to ES on failures" [Medium,In progress] - Assigned to Krzysztof Klimonda (kklimonda) 15:24:00 Has not been updated in some time 15:24:12 I'd guess it's not going to land 15:24:17 I've commented but will move it out 15:24:39 I think there's a change a bit abandoned, and I don't know how it relates to current state after moving to td-agent 15:25:19 it seems like a useful change though 15:25:35 I certainly know some people who need it or something like it 15:25:47 let's see if they reply 15:25:54 #link https://bugs.launchpad.net/kolla-ansible/+bug/1833064 15:25:54 Launchpad bug 1833064 in kolla-ansible train "nova-status upgrade check should be run after DB sync and online migrations" [Medium,In progress] - Assigned to Mark Goddard (mgoddard) 15:26:12 this was a little controversial 15:26:34 easy to change, but does make the check less useful 15:27:17 #link https://bugs.launchpad.net/bugs/1845629 15:27:17 Launchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac) 15:27:20 mgoddard: we can always run it before and after 15:27:28 (if it makes any sense) 15:27:33 I'd like to get that one in, I marked it RP+1 15:28:20 yoctozepto: can you look at this change? :) 15:28:57 we have a patch for https://bugs.launchpad.net/bugs/1846467 15:28:57 Launchpad bug 1846467 in kolla-ansible "RabbitMQ container - high CPU load on multicore systems" [Medium,In progress] - Assigned to Jan Vondra (janvondra) 15:29:02 do we want it? 15:29:04 (in train) 15:29:20 it's kind of a feature, but could be used to resolve that bug 15:29:47 mgoddard: I can handle rabbitmq 15:29:52 mnasiadka: look at which one 15:30:03 yoctozepto: fluentd mariadb 15:30:06 yoctozepto: thanks 15:30:26 mgoddard: rabbitmq needs some release notes and maybe some docs - after that I think the code is fine. 15:30:32 ah, yoctozepto took this, good 15:30:59 yoctozepto: https://review.opendev.org/#/c/686428/ 15:31:13 dougsz was also going to look at https://bugs.launchpad.net/bugs/1845629 (https://review.opendev.org/686428) 15:31:13 Launchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac) 15:31:31 as he is fluent in fluentd 15:31:40 good 15:31:51 mgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/1673944 - approval to move milestone to 10.0 as this is a Wishlist? 15:31:51 Launchpad bug 1673944 in kolla-ansible "external-ceph task for nova assumes nova/cinder ceph user" [Wishlist,Triaged] - Assigned to Michal Nasiadka (mnasiadka) 15:32:03 mnasiadka: +1 15:33:04 onto triaged bugs 15:33:16 #link https://bugs.launchpad.net/kolla-ansible/+bug/1846820 15:33:16 Launchpad bug 1846820 in kolla-ansible "nova-conductor may crash during deploy due to haproxy-keystone 504" [High,Triaged] 15:33:20 mnasiadka: ok, I see, where is your review? 15:33:48 yoctozepto: in progress :) 15:34:06 we have marked it as high, nova as low :) 15:34:52 do we really think it's high? 15:34:59 mgoddard: I had a couple of occurrences of keystone 504 when running ceph-ansible CI... I can propose a patch to improve the logging of mod_wsgi, but there won't be magic - it's hard to find the root cause 15:35:26 mgoddard: it's the worst type of bug, happens sometimes :) 15:35:44 we have seen intermittent issues with keystone in CI since enabling vxlan 15:36:12 maybe it's a bit vxlan related, but as always it's hard to tell 15:36:25 it might just have exposed an issue 15:36:41 mgoddard: since running haproxy 15:36:43 we could improve vxlan testing - like do ping test with near-mtu size 15:36:49 I tell you 15:36:56 this is timeouts on haproxy 15:36:59 and maybe improve haproxy logging to try to isolate the issue 15:37:02 we can raise them and see what happens 15:37:26 geez, I analyzed that, haproxy timeouts waiting 15:37:36 and closes the connection to the backend 15:37:49 I just did not get to raising timeout values 15:37:55 be my guest :D 15:38:01 how high are the timeouts currently? 15:38:06 yoctozepto: so you raise timeouts, I'll improve logging here and there and let's see :) 15:39:07 10s for some, 1m for others 15:39:24 sounds like mnasiadka and yoctozepto will investigate 15:39:44 mgoddard: where did you get that? ;p 15:40:08 yoctozepto: you wanted to raise haproxy timeouts, we saw that in your eyes :D 15:40:10 you seem to have it in hand :) 15:40:19 I meant the timeouts, guys 15:40:37 yes, raise the bloody timeouts, I'll make sure if it happens again - we have proper logs. 15:40:43 agreed? 15:41:19 I think the relevant timeout is 1m 15:41:30 ah, geez, I read 1h 15:41:38 and was like wtf 15:41:52 mnasiadka: more than agreed 15:41:54 that sounds like it should be sufficient. maybe it is blocked in the backend 15:42:12 mgoddard: the relevant is 10s 15:42:30 at least that what it looked like 15:42:33 when I checked 15:42:33 connect and check timeout, yeah 15:42:36 sure? 504 should be server timeout 15:42:55 anyway, needs investigation 15:43:04 yeah, needs, not remember 15:43:11 what about this one: https://bugs.launchpad.net/kolla-ansible/+bug/1845244 15:43:11 Launchpad bug 1845244 in kolla-ansible "Nova scheduler is stopped after each reboot" [High,New] 15:43:18 but it's a timeout, no other relevant errors in there 15:43:23 not targeted for train currently, but it is 'high' 15:43:30 mgoddard: yeah, it's sad 15:43:35 i'm stuck with it 15:43:44 it just drops out immediately 15:43:47 I'll target to train to get it on our radar 15:43:53 sure, be my guest 15:44:01 but we need someone to figure this one out 15:44:15 I am clearly missing something "obvious" 15:45:15 it happens only in stein? 15:45:32 Radosław Piliszek proposed openstack/kolla-ansible master: Zun: fix Cinder (volume) iSCSI support https://review.opendev.org/690614 15:45:39 mnasiadka: looks like since stein 15:45:55 but not checked with current train 15:45:58 well, out of curiosity I can look into that 15:46:03 nor rocky for the record :D 15:46:06 ok 15:46:44 I have a VM up, I'll give it a poke 15:47:21 yoctozepto: can you put in docker engine version in the bug? 15:47:58 are there other k-a bugs we should be focussing on for train? 15:48:03 19.03.2 15:49:04 or any other train planning things we should discuss? 15:49:17 perhaps TLS? 15:49:28 ah, hi generalfuzz 15:50:12 I think you did not get the promised reviews 15:50:25 strictly, we did say no more features after last friday 15:50:26 yeah, just got a heart attack because I forgot those 15:51:25 yikes - lower the priority then 15:51:45 we're talking about this change https://review.opendev.org/#/c/686024/ ? 15:51:49 yes 15:51:49 or there's another one? 15:51:57 it's in merge conflict 15:52:04 so what's there to review? 15:52:47 sure, it needs an update. probably due to cells & ipv6 15:52:55 I'll update asap 15:52:57 I think it had the same problem end of last week 15:52:58 cells more likely 15:53:02 but maybe I'm wrong 15:53:02 ipv6 unlikely to conflict 15:53:05 I think we'll have to push this one out to Ussuri generalfuzz, sorry 15:53:26 ok, so it goes 15:53:39 we need to be putting review time into bugs right now 15:53:58 understand 15:54:40 speak up once things have calmed down and we'll try to merge it early on in Ussuri 15:55:05 6 minutes left, let's move on 15:55:08 #topic Idea: community meetings 15:55:31 I was speaking to osmanlicilegi who was interested in attending the kolla PTG 15:55:41 but also suggested having some community meetings 15:55:50 I think it's a nice idea 15:56:13 less development focussed, more about (human) networking 15:56:21 +1 15:56:28 it could be a good way to onboard new contributors 15:56:33 +1 15:57:11 I also thought it might be a good idea if people are interested to do a presentation of the onboarding slides I gave in Denver, but via video conference 15:57:14 #link https://docs.google.com/presentation/d/11gGW93Xu7DQo_G1LiRDm6thfB5gNLm39SHuKcgSW8FQ/edit 15:57:23 could be a separate thing 15:57:36 but it has information on how to get more involved 15:57:48 thoughts? 15:57:58 how often should such a meeting be held? 15:58:06 who would join? 15:58:49 a couple of us would :) 15:59:11 great 15:59:29 I assume osmanlicilegi would 15:59:36 mgoddard: maybe let's try to have it as part of the again virtual PTG? 15:59:39 nice presentation 15:59:44 kayobe needs updating :D 15:59:54 yeah 15:59:57 mnasiadka: or semivirtual 16:00:01 if you come 16:00:12 yeah, we'll see - I need to talk to the boss 16:00:48 I was thinking an hour, once a month 16:00:55 too much? too little? 16:01:29 mgoddard: let's start with having an hour as part of kolla ptg, and then try to schedule another one - month later, and see what is the attendance? 16:01:31 mgoddard: we are always in hurry with an hour 16:01:50 or something like that 16:02:21 sounds good 16:02:43 ok, time at the bar 16:02:45 thanks all 16:02:48 #endmeeting