16:00:16 <noonedeadpunk> #startmeeting openstack_ansible_meeting
16:00:17 <openstack> Meeting started Tue Feb 16 16:00:16 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:20 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:00:23 <noonedeadpunk> #topic bug triage
16:01:10 <noonedeadpunk> I'm wondering why this issue raised https://bugs.launchpad.net/openstack-ansible/+bug/1805630
16:01:12 <openstack> Launchpad bug 1805630 in openstack-ansible "Keystone install fail because it put node in maintenance and question it (503 unavailable)" [Undecided,New]
16:02:14 <noonedeadpunk> I was never facing this tbh...
16:02:38 <noonedeadpunk> I don't run IDP though...
16:03:10 <jrosser> i have never seen anything like that
16:03:56 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/victoria: Fix cert verification logic for cinder api  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/775771
16:03:57 <jrosser> however we've never deployed with federation in the config initially
16:04:15 <jrosser> i think it's always been something we layer on afterwards once the cloud is up
16:04:53 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/victoria: Fix cert verification logic for cinder api  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/775771
16:05:18 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/ussuri: Fix cert verification logic for cinder api  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/775772
16:06:07 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder stable/train: Fix cert verification logic for cinder api  https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/775773
16:06:32 <noonedeadpunk> ok
16:07:03 <noonedeadpunk> then re-raised this one https://bugs.launchpad.net/openstack-ansible/+bug/1848245
16:07:04 <openstack> Launchpad bug 1848245 in openstack-ansible "Deletion of routers with HA enabled fails" [Undecided,New] - Assigned to James Denton (james-denton)
16:07:44 <jrosser> if it's a real bug i guess that this may fix it https://github.com/openstack/openstack-ansible/commit/457447431fe0c46e67b91717897e89fb971b753a
16:08:07 <jrosser> as it moves all the federation setup to a second play, after keystone is active again in the LB
16:08:33 <noonedeadpunk> hm, I guess I've mentioned that patch in the bug
16:11:33 <jrosser> so back in rocky we carried a policy template inside the os_neutron role
16:11:35 <noonedeadpunk> no I haven't
16:11:51 <jrosser> perhaps this is somehow leftover policy from an old version
16:12:37 <jrosser> for the keystone thing it would also need https://github.com/openstack/openstack-ansible/commit/2bb60193028fc848e87cdc7f416019482b8cf2cb
16:12:47 * jrosser messed up first time :(
16:13:25 <noonedeadpunk> well that one I mentioned in bug)
16:15:04 <jrosser> ok
16:16:20 <noonedeadpunk> regarding rootwrap, I think we use smart_sources?:)
16:17:38 <noonedeadpunk> so we should just take this https://opendev.org/openstack/neutron/src/branch/master/etc/neutron/rootwrap.d/l3.filters
16:18:27 <noonedeadpunk> well, they have this https://opendev.org/openstack/neutron/src/branch/stable/stein/etc/neutron/rootwrap.d/l3.filters#L72-L77
16:18:31 <openstackgerrit> Merged openstack/ansible-role-python_venv_build stable/victoria: Remove preflight checks  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/775234
16:20:07 <noonedeadpunk> Ok, I guess that's it in terms of bugs?
16:20:21 <jrosser> yes
16:21:18 <noonedeadpunk> oh. one thing. I introduced bug with dropping default barbican kek and https://review.opendev.org/c/openstack/openstack-ansible/+/775856 to ccover this
16:21:47 <jrosser> could we go through this https://etherpad.opendev.org/p/osa-ci-failures
16:22:08 <noonedeadpunk> totally!
16:22:23 <jrosser> first one seems like progress on mariadb
16:22:38 <jrosser> hopefully 10.5.9 will fix the failure to startup
16:22:52 <jrosser> then next one "Fail to retrieve upper constraints"
16:23:00 <jrosser> i have some patches but kind of not sure on the approach
16:23:03 <noonedeadpunk> change of root -> admin fails on cluster bootstrap with missing permissions
16:23:37 <jrosser> do we need to split the bootstrap and user creation to be root vs. admin user?
16:23:46 <noonedeadpunk> I still don't like passing u-c as a content...
16:24:29 <noonedeadpunk> I don't really know why it takes admin user for bootstrap.. because of my.cnf?
16:24:39 <noonedeadpunk> needd to check this out
16:24:59 <jrosser> do you have a neater way for the u-c stuff
16:25:19 <jrosser> i was wanting to leave it overridable, to have several different SHA of u-c available on the repo server if needed
16:26:10 <noonedeadpunk> but what stopps us from jsut passing local path here? https://review.opendev.org/c/openstack/openstack-ansible/+/774518/4/playbooks/repo-install.yml
16:26:36 <noonedeadpunk> need of checkout?
16:26:54 <jrosser> outside of CI the path isnt local
16:27:14 <jrosser> actually does not exist at all
16:27:51 <MickyMan77> noonedeadpunk: when i check the /var/log/httpd/error_log, I can only see log entrys from the start of the httpd service.
16:28:05 <MickyMan77> the access_log is empty
16:28:06 <noonedeadpunk> but we can use get_url instead of uri?
16:29:09 <jrosser> except in CI when it's file:///
16:29:29 <noonedeadpunk> there's anyway `when: requirements_git_repo is search('http')`
16:29:48 <jrosser> yes so this is all about making a clean interface to the repo server role
16:29:54 <jrosser> which doesnt matter if CI or not
16:30:07 <MickyMan77> noonedeadpunk: the VirtualHost have this setting...
16:30:08 <MickyMan77> CustomLog "|/usr/bin/env logger -p daemon.info -t httpd" "%h %l %u \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""
16:30:12 <jrosser> but something still not really feeling right about my patches, not sure really why
16:30:33 <noonedeadpunk> yes, totally, but what I mean is - why we can't always provide it with local file path?
16:30:49 <jrosser> on the deploy host?
16:30:51 <noonedeadpunk> yep
16:31:05 <noonedeadpunk> and instead of content it will be just src
16:31:48 <jrosser> i had considered making /etc/openstack_deploy/u-c/
16:31:57 <jrosser> and anything in there just gets put to the repo server
16:31:57 <noonedeadpunk> and we won't need to retrieve u-c later as well, because we can use it all the time then
16:32:14 <noonedeadpunk> or that...
16:32:51 <noonedeadpunk> for realy deployments it's also a profit because in case of mirrors issues your deployment won't stuck
16:33:39 <noonedeadpunk> or I'm missing that on repo container it won't be retrived anymore?
16:34:00 <jrosser> see, this is more complicated than it seems on the surface :)
16:34:15 <noonedeadpunk> yeah...
16:34:35 <noonedeadpunk> I just really didn't have time to properly look this through...
16:34:48 <noonedeadpunk> and play around with code
16:35:02 <jrosser> ok, so related i left a comment here https://review.opendev.org/c/openstack/openstack-ansible/+/775095
16:35:44 <noonedeadpunk> ah damn it
16:37:02 <jrosser> ok cool
16:37:17 <jrosser> next one i wanted an opinion on was this https://review.opendev.org/c/openstack/openstack-ansible/+/775695
16:37:27 <noonedeadpunk> I need to spent time and configure gerrit email filters...
16:37:35 <jrosser> ceph_client role seems to define the vars kind of oddly
16:37:38 * noonedeadpunk has 6k emails from gerrit in folder
16:37:56 <jrosser> lots of things exist only in vars/blah.yml rather than defaults
16:38:02 <openstackgerrit> Merged openstack/openstack-ansible master: Collect contents of /etc/dnf from CI jobs  https://review.opendev.org/c/openstack/openstack-ansible/+/775677
16:38:57 <jrosser> either i have a mistake with the override i make, or it's not possible to override that role var
16:39:20 <noonedeadpunk> because it's included during runtime
16:39:46 <noonedeadpunk> yeah, I think we should move things to default...
16:39:54 <jrosser> right, so -e would only have precedence over vars/main.yml?
16:40:19 <noonedeadpunk> I'm not 100% sure but might be...
16:40:40 <jrosser> ok cool i will try to take a look at tidying up ceph_client vars a bit
16:41:03 <jrosser> thats the CI errors that i've looked into
16:41:10 <jrosser> the rest not so much yet
16:41:58 <noonedeadpunk> let me quikly test it out...
16:42:36 <jrosser> the "Keystone/memcached error" i think we need to talk to the oslo people
16:42:56 <jrosser> something funky there with the connection to memcached from keystone in a way i don't understand
16:45:04 <mgariepy> noonedeadpunk, added a comment on https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/768513
16:45:47 <noonedeadpunk> ok, cool, thanks!
16:46:30 <noonedeadpunk> jrosser: well, no, even when vars are included they can be overriden
16:46:44 <jrosser> interesting
16:47:06 <jrosser> must be something else i've got wrong there, just just wrong var name or somthing
16:47:13 <noonedeadpunk> http://paste.openstack.org/show/802699/
16:48:57 <noonedeadpunk> but, if you define in play vars it will be overriden
16:49:43 <noonedeadpunk> http://paste.openstack.org/show/802700/
16:50:14 <noonedeadpunk> so -e have prescedence over everything
16:50:21 <noonedeadpunk> but maybe it's not the case here
16:50:41 * jrosser facepalm
16:51:12 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible master: Use infra mirror for ceph_client role in CI  https://review.opendev.org/c/openstack/openstack-ansible/+/775695
16:51:21 <jrosser> 1 character wrong
16:51:34 <noonedeadpunk> ah:)
16:51:36 <noonedeadpunk> well)
16:52:25 <jrosser> the only other thing on the CI list which is pretty easy is adding erlang-solutions repo to the infra mirror
16:52:59 <jrosser> though i did spend some time wading around system-config repo trying to figure that all out
16:53:58 <noonedeadpunk> for focal we use native repo though, right?
16:54:17 <noonedeadpunk> eventually whatever...
16:55:01 <noonedeadpunk> we probably should ask fungi or clarkb ?
16:55:11 <noonedeadpunk> just to ssave up some time?
16:55:30 <jrosser> ah yes it was a buster job that broke
16:55:43 <jrosser> seems every time they release a new package the repo is bust until someone tweets them
16:56:09 <mgariepy> create a bot script to tweet them when it breaks !
16:56:10 <mgariepy> haha
16:56:40 <jrosser> i think that the infra reprepro stuff may shield us from that as it needs to repo to be good in order to mirror it
16:57:09 <noonedeadpunk> well yes, that would be probably nice to have
16:57:58 <jrosser> thats probably all on the CI fixes, but if anyone wants to dig at some of the more obtuse errors please do
16:58:27 <jrosser> imho this is the best way we can reduce the CI load for OSA
16:58:52 <noonedeadpunk> yeah But atm maria brings the most issues with ci
16:59:29 <jrosser> final thing from me would be centos-8 stream
16:59:39 <jrosser> i tried some stuff again in a VM today
17:00:11 <jrosser> i could install networkd from epel and lxc copr repo without a ton of install conflicts like i got before
17:00:36 <jrosser> and i got a chroot built with `sudo dnf --installroot=/home/centos/foobar install --setopt=install_weak_deps=False --nodocs rootfiles`
17:01:33 <noonedeadpunk> I think idea to have all of lxc images build from chroot instead all that nasty searches is really awesome
17:01:42 <jrosser> i will hack around in an AIO next to see what i can do
17:01:48 <noonedeadpunk> and it should be faster as well
17:01:56 <jrosser> agreed, was just looking at lxc_hosts and there is tons of complexity
17:02:08 <jrosser> could be really simplified
17:02:30 <noonedeadpunk> I was also thinking if we should add some centos forks support like AlmaLinux?
17:03:04 <noonedeadpunk> I used to rely on cloudlinux a lot previously...
17:03:21 <odyssey4me> jrosser not as far as I know - I can dig around for you if you like
17:03:21 <jrosser> well, it's still very much the case that we don't have an active contributor for centos stuff
17:03:46 <mgariepy> https://goo.gl/maps/mGtpF5rcW1T52rvU9 ?
17:04:05 <jrosser> odyssey4me: that would be great, there are some differences we need to handle but the regualar ansible vars don't seem very helpful
17:04:15 <odyssey4me> jrosser https://github.com/ansible/ansible/issues/73027
17:05:04 <noonedeadpunk> oh so it's patched https://github.com/relrod/ansible/commit/44f8b8b56929df1b81852b73f862f3254b3bde2e
17:05:40 <odyssey4me> yeah, that one may be worth proposing as a backport to the stable releases
17:06:09 <jrosser> yes that would be useful, as 2.10 kind of exists and centos changes underneath it
17:07:11 * noonedeadpunk should finally write up tests for systemctl ansible module....
17:07:34 <noonedeadpunk> #endmeeting