16:01:07 <noonedeadpunk> #startmeeting openstack_ansible_meeting 16:01:07 <openstack> Meeting started Tue Nov 19 16:01:07 2019 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:10 <openstack> The meeting name has been set to 'openstack_ansible_meeting' 16:01:17 <noonedeadpunk> #topic office hours 16:01:54 <noonedeadpunk> So, Queens has moved to extended maintenance status https://review.opendev.org/#/c/691318/ 16:02:21 <noonedeadpunk> I guess that evrardjp was going to discuss this here, but it's already merged, so... 16:03:29 <guilhermesp> o/ 16:04:41 <noonedeadpunk> Also I think we should wait for a while for the next rc until upgrade bugs will be figured out 16:06:56 <noonedeadpunk> And we have kinda problem with upgrade jobs... So first of all they're pretty close to timouts. For telemetry they fails with timeout for centos. 16:09:06 <noonedeadpunk> Another thing is that we can't use depends-on for lower release upgrade jobs. And as it's not in the independent template - we can't set them as non-voting or modify/exclude per role. 16:09:41 <noonedeadpunk> So for horizon train they're going to fail, as stein don't allow doing metal horizon deployments 16:10:18 <cjloader> o/ 16:11:45 <cjloader> noonedeadpunk: how much longer do I have to complete inspector to get it in the next rc for train? 16:12:16 <jrosser> do we use bootstrap-ansible.sh to get both releases in place for upgrade jobs? 16:12:26 <jrosser> i'm not sure depends-on will work for either branch 16:16:00 <noonedeadpunk> yes, we do. Actually run-upgrade.sh is supposed to. But, we're checking out back to the head... 16:16:07 <noonedeadpunk> But yes, not sure either... 16:16:35 <noonedeadpunk> And it looks like a problem, especially when it's inlcuded to all roles at once. 16:16:59 <noonedeadpunk> On the other hand - they already helped us several times and prevented from breaking upgrade 16:17:06 <noonedeadpunk> So they are pretty useful. 16:17:54 <noonedeadpunk> cjloader: so we have to release no later than 15 of december 16:18:10 <cjloader> okay 16:18:25 <cjloader> i'll keep you guys informed 16:20:43 <jrosser> noonedeadpunk: also have you seen a bunch of broken things with changing tempest to run smoke test? 16:21:49 <noonedeadpunk> Yep, I did:( 16:22:02 <jrosser> it is good and bad 16:22:16 <jrosser> lots of broken <- bad 16:22:24 <jrosser> lots more test coverage <- good 16:22:36 <noonedeadpunk> tbh I never was good enough in fixing stuff to apply full tempest test... 16:22:55 <jrosser> but biggest problem seems to be the ceph jobs which are broken now on master 16:23:29 <noonedeadpunk> oh, I haven't realized that 16:25:12 <jrosser> like this https://review.opendev.org/#/c/694253/ 16:25:29 <jrosser> i *think* this is related to the tempest test being more thorough 16:27:51 <noonedeadpunk> according to https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eac/694253/2/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/eace31a/logs/openstack/aio1_utility_container-1101bcb3/utility/stestr_results.html it fails swift... 16:28:03 <noonedeadpunk> but the thing is that swift is generally fails https://review.opendev.org/#/c/694783/ 16:28:18 <noonedeadpunk> so not sure that problem is in ceph itself 16:30:30 <noonedeadpunk> the patch don't fix stuff for ubuntu distro now, but it shows that actually nothing pass tests 16:31:02 <jrosser> for swift it is this https://zuul.opendev.org/t/openstack/build/4a20e01c392e4e2d93f228de2ff9e30e/log/logs/openstack/aio1-utility/tempest_run.log.txt.gz#175 16:31:57 <jrosser> i think horizon needs tempest telling to ignore the certificate check for public endpoint 16:32:14 <jrosser> and then congress is broken, looks like a bunch of missing config in the service 16:32:15 <noonedeadpunk> yep. And it looks like we don't deploy reseller user 16:33:31 <noonedeadpunk> this feels like tons of work to clean everythin out 16:34:10 <noonedeadpunk> but tbh I'm not sure we have an experts for each supported project 16:34:12 <jrosser> i think 4-6 roles are affected badly 16:35:07 <noonedeadpunk> I'd say swift is a critical one 16:35:36 <jrosser> imho improving test coverage is a good aim for this cycle, if we don't already have one 16:35:48 <jrosser> but right now when we try to release, not so good 16:36:01 <noonedeadpunk> ++ 16:36:35 <noonedeadpunk> ofc we can set some of stuff to non-voting untill release... But not sure it's the path to follow 16:37:17 <jrosser> or we can set this var back to how it was https://github.com/openstack/openstack-ansible-os_tempest/commit/6116c681a4119aed9b1fa7723de18ae0731ff500#diff-7eeda618087b49ae876084ab6c73fdbbL85-L99 16:37:23 <jrosser> in the integrated repo 16:37:46 <jrosser> and leave the smoke test patch in os_tempest as an indicator of where we want to end up 16:37:57 <noonedeadpunk> that's nice idea 16:38:16 <noonedeadpunk> and rollback this right after release 16:38:35 <jrosser> yes, or make a pile of experimental jobs on the things we know break 16:39:29 * noonedeadpunk always forget about experimental jobs 16:40:42 <jrosser> perhaps too we should add more scenraios on the os_tempest repo, becasue thats really whats caught is out here 16:48:19 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible master: Roll back use of tempest smoke test for the integrated repo https://review.opendev.org/695033 16:49:38 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible-os_designate master: Remove deprecated packages from centos installs https://review.opendev.org/694775 16:49:47 <jrosser> lets see if that works ^ 16:57:52 <admin0> anyone seen this on console and a jumbled display ? Unimplemented function 108(Inval All Palettes) [ further notices suppressed ] 16:57:53 <admin0> Unimplemented function 102(Display Mark) [ further notices suppressed ] 17:00:20 <noonedeadpunk> #endmeeting