16:01:07 <noonedeadpunk> #startmeeting openstack_ansible_meeting
16:01:07 <openstack> Meeting started Tue Nov 19 16:01:07 2019 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:10 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:01:17 <noonedeadpunk> #topic office hours
16:01:54 <noonedeadpunk> So, Queens has moved to extended maintenance status https://review.opendev.org/#/c/691318/
16:02:21 <noonedeadpunk> I guess that evrardjp was going to discuss this here, but it's already merged, so...
16:03:29 <guilhermesp> o/
16:04:41 <noonedeadpunk> Also I think we should wait for a while for the next rc until upgrade bugs will be figured out
16:06:56 <noonedeadpunk> And we have kinda problem with upgrade jobs... So first of all they're pretty close to timouts. For telemetry they fails with timeout for centos.
16:09:06 <noonedeadpunk> Another thing is that we can't use depends-on for lower release upgrade jobs. And as it's not in the independent template - we can't set them as non-voting or modify/exclude per role.
16:09:41 <noonedeadpunk> So for horizon train they're going to fail, as stein don't allow doing metal horizon deployments
16:10:18 <cjloader> o/
16:11:45 <cjloader> noonedeadpunk: how much longer do I have to complete inspector to get it in the next rc for train?
16:12:16 <jrosser> do we use bootstrap-ansible.sh to get both releases in place for upgrade jobs?
16:12:26 <jrosser> i'm not sure depends-on will work for either branch
16:16:00 <noonedeadpunk> yes, we do. Actually run-upgrade.sh is supposed to. But, we're checking out back to the head...
16:16:07 <noonedeadpunk> But yes, not sure either...
16:16:35 <noonedeadpunk> And it looks like a problem, especially when it's inlcuded to all roles at once.
16:16:59 <noonedeadpunk> On the other hand - they already helped us several times and prevented from breaking upgrade
16:17:06 <noonedeadpunk> So they are pretty useful.
16:17:54 <noonedeadpunk> cjloader: so we have to release no later than 15 of december
16:18:10 <cjloader> okay
16:18:25 <cjloader> i'll keep you guys informed
16:20:43 <jrosser> noonedeadpunk: also have you seen a bunch of broken things with changing tempest to run smoke test?
16:21:49 <noonedeadpunk> Yep, I did:(
16:22:02 <jrosser> it is good and bad
16:22:16 <jrosser> lots of broken <- bad
16:22:24 <jrosser> lots more test coverage <- good
16:22:36 <noonedeadpunk> tbh I never was good enough in fixing stuff to apply full tempest test...
16:22:55 <jrosser> but biggest problem seems to be the ceph jobs which are broken now on master
16:23:29 <noonedeadpunk> oh, I haven't realized that
16:25:12 <jrosser> like this https://review.opendev.org/#/c/694253/
16:25:29 <jrosser> i *think* this is related to the tempest test being more thorough
16:27:51 <noonedeadpunk> according to https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eac/694253/2/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/eace31a/logs/openstack/aio1_utility_container-1101bcb3/utility/stestr_results.html it fails swift...
16:28:03 <noonedeadpunk> but the thing is that swift is generally fails https://review.opendev.org/#/c/694783/
16:28:18 <noonedeadpunk> so not sure that problem is in ceph itself
16:30:30 <noonedeadpunk> the patch don't fix stuff for ubuntu distro now, but it shows that actually nothing pass tests
16:31:02 <jrosser> for swift it is this https://zuul.opendev.org/t/openstack/build/4a20e01c392e4e2d93f228de2ff9e30e/log/logs/openstack/aio1-utility/tempest_run.log.txt.gz#175
16:31:57 <jrosser> i think horizon needs tempest telling to ignore the certificate check for public endpoint
16:32:14 <jrosser> and then congress is broken, looks like a bunch of missing config in the service
16:32:15 <noonedeadpunk> yep. And it looks like we don't deploy reseller user
16:33:31 <noonedeadpunk> this feels like tons of work to clean everythin out
16:34:10 <noonedeadpunk> but tbh I'm not sure we have an experts for each supported project
16:34:12 <jrosser> i think 4-6 roles are affected badly
16:35:07 <noonedeadpunk> I'd say swift is a critical one
16:35:36 <jrosser> imho improving test coverage is a good aim for this cycle, if we don't already have one
16:35:48 <jrosser> but right now when we try to release, not so good
16:36:01 <noonedeadpunk> ++
16:36:35 <noonedeadpunk> ofc we can set some of stuff to non-voting untill release... But not sure it's the path to follow
16:37:17 <jrosser> or we can set this var back to how it was https://github.com/openstack/openstack-ansible-os_tempest/commit/6116c681a4119aed9b1fa7723de18ae0731ff500#diff-7eeda618087b49ae876084ab6c73fdbbL85-L99
16:37:23 <jrosser> in the integrated repo
16:37:46 <jrosser> and leave the smoke test patch in os_tempest as an indicator of where we want to end up
16:37:57 <noonedeadpunk> that's nice idea
16:38:16 <noonedeadpunk> and rollback this right after release
16:38:35 <jrosser> yes, or make a pile of experimental jobs on the things we know break
16:39:29 * noonedeadpunk always forget about experimental jobs
16:40:42 <jrosser> perhaps too we should add more scenraios on the os_tempest repo, becasue thats really whats caught is out here
16:48:19 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible master: Roll back use of tempest smoke test for the integrated repo  https://review.opendev.org/695033
16:49:38 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible-os_designate master: Remove deprecated packages from centos installs  https://review.opendev.org/694775
16:49:47 <jrosser> lets see if that works ^
16:57:52 <admin0> anyone seen this on console and a jumbled display ? Unimplemented function 108(Inval All Palettes) [ further notices suppressed ]
16:57:53 <admin0> Unimplemented function 102(Display Mark) [ further notices suppressed ]
17:00:20 <noonedeadpunk> #endmeeting