16:02:54 <noonedeadpunk> #startmeeting openstack_ansible_meeting
16:02:55 <openstack> Meeting started Tue Feb  9 16:02:54 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:56 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:58 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:05:31 <noonedeadpunk> trying to check if we have some bugs to discuss...
16:06:49 <noonedeadpunk> I think jrossercovered today most of them :)
16:07:04 <noonedeadpunk> #topic office hours
16:09:06 <jrosser> o/
16:09:08 <jrosser> hello
16:09:15 <noonedeadpunk> \o/
16:10:01 <noonedeadpunk> I don't have really much from my side to say, since I had pretty little time on my hands :(
16:11:17 <jrosser> feels like we need to get all this new-pip stuff merged
16:11:46 <noonedeadpunk> I'd say we almost did?
16:11:56 <noonedeadpunk> https://review.opendev.org/q/topic:%22osa-new-pip%22+(status:open)
16:11:59 <noonedeadpunk> it's super closr
16:12:02 <noonedeadpunk> *close
16:12:05 <jrosser> we don't yet land the patch to the integrated repo which turns it on
16:12:52 <jrosser> this is related for the tests repo https://review.opendev.org/c/openstack/openstack-ansible-tests/+/774651
16:13:04 <noonedeadpunk> we stuck on neutron pretty much
16:13:13 <noonedeadpunk> and tests repo does not make this easy for us
16:13:22 <jrosser> yeah lots of things there, the tests repo patch will help
16:13:36 <jrosser> then we need the bionic->focal patch for os_neutron
16:13:56 <noonedeadpunk> which just doesn't work actually...
16:14:55 <jrosser> indeed, the functional tests are all generally unhappy
16:15:27 <jrosser> https://review.opendev.org/773979 is failing horribly in CI just now
16:16:36 <jrosser> oh right
16:17:06 <jrosser> we can't and the change to the tests repo + bionic->focal without also the constraints->requirements changes for os_neutron
16:17:14 <jrosser> some of these patches are going to need to be squashed
16:17:20 <jrosser> *land
16:18:27 <noonedeadpunk> why constraints->requirements changes relate to bionic vs focal? I guess they will get same versions during play?
16:18:46 <noonedeadpunk> but see no issues in merging as well if it will be required
16:19:34 <noonedeadpunk> also I'm wondering what to do with octavia on centos
16:19:43 <noonedeadpunk> should we jsut mark it nv now?
16:20:19 <jrosser> i wonder if johnsom is around?
16:20:23 <johnsom> Hi
16:20:28 <jrosser> woah
16:20:30 <jrosser> :)
16:20:37 <johnsom> You rang?
16:20:40 <johnsom> What is up?
16:21:03 <jrosser> did you see this http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020218.html
16:21:13 <jrosser> we are a bit stuck on our centos-8 CI jobs
16:22:16 <johnsom> Hmm, reading through. The initial report is a nova bug it looks like. Let me read all the way down
16:23:45 <noonedeadpunk> the issue here is that nova and neutron tempest tests are passing for us..
16:23:53 <noonedeadpunk> maybe we're testing wrong things...
16:24:04 <jrosser> we should check they actually boot something :)
16:24:23 <johnsom> Well, Octavia tends to actually test more than other projects. We have true end-to-end tests, where some gloss over things
16:24:40 <johnsom> Is there a patch with logs I can dig in?
16:25:02 <noonedeadpunk> sure https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/769952
16:25:17 <johnsom> Thanks, give me a few minutes.
16:26:03 <noonedeadpunk> `tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops` should boot smth I guess
16:26:19 <noonedeadpunk> https://zuul.opendev.org/t/openstack/build/0a123e189be8445da96927be09220d7a/log/logs/openstack/aio1-utility/tempest_run.log.txt#135 (it's for nova role CI)
16:26:58 <johnsom> Hmm, those logs have expired. Another patch maybe?
16:26:58 <johnsom> Ah, nevermind, I had the wrong link
16:27:33 <noonedeadpunk> jrosser: yeah they do spawn isntance https://zuul.opendev.org/t/openstack/build/0a123e189be8445da96927be09220d7a/log/logs/host/nova-compute.service.journal-20-52-15.log.txt#5241
16:28:01 <jrosser> cool
16:29:10 <noonedeadpunk> johnsom: you an also check this if previous has expired https://zuul.opendev.org/t/openstack/build/df371a76c1ab4e76b97e4b6b974fe29a
16:30:18 <noonedeadpunk> btw for the last patch debian also failed in pretty much the same way I'd say...
16:36:46 <jrosser> noonedeadpunk: i did not know what to do about the 0.0.0 version here https://bugs.launchpad.net/openstack-ansible/+bug/1915128
16:36:48 <openstack> Launchpad bug 1915128 in openstack-ansible "OpenStack Swift-proxy-server do not start" [Undecided,New]
16:36:59 <jrosser> other than say we're not really supporting rocky.....
16:38:57 <noonedeadpunk> I'm wondering if it's because they checked-out to rocky-em tag
16:39:09 <noonedeadpunk> I could imagine that pbr might go crazy about that
16:39:13 <jrosser> oh interesting, could be
16:39:34 <jrosser> perhaps an assumption that a tag is a number
16:40:10 <jrosser> whilst we are in meeting time i guess we should also talk about CI resource use?
16:40:35 <noonedeadpunk> yeah
16:41:08 <noonedeadpunk> I think the best we can do, except reducing time, is also move bionic tests to experimental
16:41:16 <jrosser> i think that the conclusion on the ML is a good one, reducing job failures is the biggest win
16:41:18 <noonedeadpunk> not sure if we should actively carry on bionic
16:41:34 <jrosser> becasue that may be even 100% overhead right now, or more
16:41:37 <noonedeadpunk> and main issue with failures I guess is galera
16:42:21 <noonedeadpunk> yeah, there were another ones, like auditd bug...
16:42:27 <jrosser> i'm going to try and be a bit more disciplined with recheck to note on the etherpad (https://etherpad.opendev.org/p/osa-ci-failures) when there is some systematic error
16:42:32 <noonedeadpunk> and I guess looking into gnocchi is also useful
16:42:43 <jrosser> oh yes there is a whole lot of mess there
16:43:10 <jrosser> something very strange with the db access unless i'm reading the log badly
16:43:11 <noonedeadpunk> +1 to having that etherpad
16:43:55 <noonedeadpunk> I think I need to deploy it to see what's going on
16:45:12 <jrosser> what to do with mariadb? is this an irc sort of thing?
16:47:24 <noonedeadpunk> I actually no idea except asking, yeah.
16:47:31 * noonedeadpunk goes to #mariadb for this
16:47:50 <noonedeadpunk> * #maria
16:48:28 <admin0> hi all .. i am getting issue in setup-infra that i cannot understand .. this is the error:  https://gist.githubusercontent.com/a1git/bf7c55a1befd59e3682be485bc4b1e88/raw/785c1d0a32fc05ae23e5fa5dbd859d3934f6930a/gistfile1.txt -- does it mean i need to downgrade my pip ?
16:48:57 <admin0> i tried 22.0.0 .. but it fails on galera setup .. so going back on 21.2.2
16:52:30 <jrosser> admin0: have you used venv_rebuild=true ever on that deployment?
16:53:06 <noonedeadpunk> uh....
16:53:12 <admin0> i have not .. this is a new greenfield
16:53:59 <noonedeadpunk> we need to merge https://review.opendev.org/q/I6bbe66b699ce5ab245bb9779b61b5c4625eba927
16:54:16 <admin0> on one line in the log inside the python_venv_build  log, I find 021-02-09T22:13:01,803     error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
16:54:30 <spatel> noonedeadpunk ++++++1 for that patch
16:54:38 <admin0> aren't those installed by ansible inside the container ?
16:55:28 <noonedeadpunk> I guess it should be installed only on repo container where we usually delegate
16:55:43 <admin0> i will lxc-containers-destroy .. and retry once more
16:56:12 <spatel> admin0 it cook everything on repo and then just deploy on other container to reduce duplication work
16:56:12 <openstackgerrit> Merged openstack/openstack-ansible-tests master: Unpin virtualenv version  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/774651
16:56:58 <johnsom> jrosser noonedeadpunk I think we need to bring in a nova expert on this. I don't see why nova is going out to lunch, but there are a bunch of errors in the nova logs.  This seems to be related: https://zuul.opendev.org/t/openstack/build/df371a76c1ab4e76b97e4b6b974fe29a/log/logs/host/nova-api-os-compute.service.journal-12-56-44.log.txt#6893
16:57:03 <spatel> venv_rebuild can be evil without that patch :) I learnt that hard way
16:57:08 <jrosser> it should never be trying to build that wheel on the utility container lie spatel says
16:57:27 <jrosser> it means that for some reason it is not being taken from the repo server
16:57:51 <johnsom> This is the other key message: https://zuul.opendev.org/t/openstack/build/df371a76c1ab4e76b97e4b6b974fe29a/log/logs/host/nova-compute.service.journal-12-56-44.log.txt#5970
16:58:28 <johnsom> But that may be a side effect of the cleanup/error handling related to the above error
17:00:13 * jrosser sees eventlet......
17:03:13 <johnsom> Yeah
17:03:13 <noonedeadpunk> hm, that seems like libvirt issue indeed
17:03:38 <noonedeadpunk> wondering why we don't see it anywhere else...
17:03:42 <johnsom> Well, I really think it's related to the messaging queue problem. The libvirt very well may be a side effect
17:04:01 <johnsom> I'm just not sure what it is trying to message there.
17:04:39 <jrosser> rabbitmq log is totally unhelpful :(
17:06:24 <noonedeadpunk> eventually I saw this messages in my deployment with ceilometer
17:06:39 <noonedeadpunk> when it agent tries to poll libvirt
17:07:14 <noonedeadpunk> and the metric it's polling is not supported by libvirt
17:07:37 <noonedeadpunk> but here we don't have any pollster I guess (except nova)
17:08:49 <noonedeadpunk> well anyway, thanks for taking time and watching johnsom!
17:08:56 <noonedeadpunk> #endmeeting