| noonedeadpunk | I'm not sure how things are passing in our CI, but we got fail right away on 27.3.0 on haproxy role due to that: https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/901993 | 09:04 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: use policy.yaml instead of policy.json due to latest change on gnocchi check the config Change-Id: If05936d25db3329b4204172f3b800ddf3417b9b7 https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/895542 | 09:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: use policy.yaml instead of policy.json due to latest change on gnocchi check the config https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/895542 | 09:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add variable to control queues redundancy https://review.opendev.org/c/openstack/openstack-ansible/+/901534 | 09:17 |
| opendevreview | Doug Goldstein proposed openstack/openstack-ansible master: abstract bootstrap host disk partition names https://review.opendev.org/c/openstack/openstack-ansible/+/901106 | 09:18 |
| jrosser | noonedeadpunk: column headings are a bit off here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e6d/901191/1/check/openstack-tox-docs/e6ddb17/docs/admin/upgrades/compatibility-matrix.html | 10:53 |
| noonedeadpunk | ah, true | 10:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add compatability matrix for 2023.2 (Bobcat) https://review.opendev.org/c/openstack/openstack-ansible/+/901191 | 10:54 |
| noonedeadpunk | I think it's about time to move things over to Legacy as well | 10:55 |
| noonedeadpunk | As C8 and 18.04 is about to go along with Yoga | 10:55 |
| noonedeadpunk | Actually... Looks like Ceph Reef is available through UCA nowadays? | 10:57 |
| noonedeadpunk | https://ubuntu.com/ceph/docs/supported-ceph-versions | 10:57 |
| noonedeadpunk | As well as from community? https://docs.ceph.com/en/reef/start/os-recommendations/ | 10:57 |
| jrosser | yes you need to get to Reef in order to do focal->jammy on your ceph cluster | 10:59 |
| noonedeadpunk | yeah https://download.ceph.com/debian-reef/dists/jammy/ | 10:59 |
| jrosser | and here we have not yet considered Reef mature enough to do that | 11:00 |
| jrosser | i think 18.2.1 will be the first time we try it | 11:00 |
| noonedeadpunk | So I was thinking what to do with note in docs regarding supported ceph versions | 11:01 |
| jrosser | our CI still does quincy? | 11:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add compatability matrix for 2023.2 (Bobcat) https://review.opendev.org/c/openstack/openstack-ansible/+/901191 | 11:02 |
| noonedeadpunk | yeah | 11:02 |
| noonedeadpunk | but given that UCA has reef as well... | 11:02 |
| noonedeadpunk | like comment seems to be not true | 11:02 |
| noonedeadpunk | I'm not really sure if ceph-ansible will do reef deployment just nice either... | 11:04 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Make glance_additional_stores work nicely with multi-store https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901034 | 11:20 |
| noonedeadpunk | jrosser: can you remind me if we're adding ops to a-c-r for bobcat? | 12:40 |
| noonedeadpunk | I think not? | 12:40 |
| noonedeadpunk | yeah, I'm pretty sure we're not... | 12:40 |
| jrosser | noonedeadpunk: we don't need to | 12:56 |
| jrosser | i figured out how to add one collection from a subdir in ops repo | 12:56 |
| jrosser | would be good if you could carefully look at this https://review.opendev.org/c/openstack/openstack-ansible/+/901471 | 12:57 |
| jrosser | becasue i'm not sure it was workig originally | 12:57 |
| jrosser | the string splitting in particular, why we ever did the split from the [-3....] end, rather than the start of the string | 12:58 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Fix cinder store configuration https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901046 | 13:01 |
| noonedeadpunk | I think we were taking from the end, because didn't know/though about subdirs. And otherwise it was same result | 13:13 |
| noonedeadpunk | like the only thing, if someone somehow is going to add `#` as part of url for $reasons | 13:14 |
| noonedeadpunk | like auth | 13:15 |
| noonedeadpunk | but we're talking here only about our CI basically... | 13:15 |
| noonedeadpunk | Oh... So indeed I can leverage this thing for sure.... | 13:22 |
| jrosser | if we’re going to use the ops repo for collections then we need the # in and out of CI | 13:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Add bugfixed version of ceph_crush_rule module https://review.opendev.org/c/openstack/openstack-ansible-ops/+/902058 | 13:36 |
| noonedeadpunk | yeah, but out of CI we don't need to split based on it? | 13:37 |
| noonedeadpunk | As I guess the split on `#` what potentially concerned me there... | 13:37 |
| noonedeadpunk | As likely ansible-galaxy does more proper urlparse then just split on # | 13:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use ops ceph collection for MDS deployment https://review.opendev.org/c/openstack/openstack-ansible/+/902059 | 13:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use ops ceph collection for MDS deployment https://review.opendev.org/c/openstack/openstack-ansible/+/902059 | 13:48 |
| jrosser | noonedeadpunk: required_projects for ops repo needed? | 13:55 |
| opendevreview | Doug Goldstein proposed openstack/openstack-ansible master: abstract bootstrap host disk partition names https://review.opendev.org/c/openstack/openstack-ansible/+/901106 | 14:08 |
| noonedeadpunk | jrosser: well. if we want depends on to work? | 14:11 |
| jrosser | sure - I guess we only know if it’s worked with a depends-on in Manila repo? | 14:13 |
| noonedeadpunk | yep | 14:14 |
| noonedeadpunk | though I wanna see if we don't just fail with retry limits... | 14:14 |
| noonedeadpunk | looks like it works.. at least bootstrap passes | 14:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/898914 | 14:17 |
| noonedeadpunk | yeah, I;'m not sure now that it works :( Or well, sample playbook seems like it does, but probably precedency work differently when module is in role tree... | 14:25 |
| noonedeadpunk | as while this works nicely https://paste.openstack.org/show/bAOCnLg3bQplQ87Wn4hn/ ceph-ansible seems to be using original version | 14:26 |
| noonedeadpunk | `However, any roles you call in your playbook define their own collections search order; they do not inherit the calling playbook’s settings.` | 14:31 |
| jrosser | yeah but ceph_crush_rule is a module? | 14:34 |
| jrosser | and not in a collection | 14:34 |
| jrosser | ^ the original one we want to fix | 14:34 |
| noonedeadpunk | yes, the original one is. I assumed that if I prioritize lookup through our collection, it will be used from collection fisst | 14:51 |
| noonedeadpunk | and that worked in sample playbook, but somehow not from ceph-ansible role itself, as likely role doesn't care about what you define in playbook :( | 14:52 |
| noonedeadpunk | so then in fact I don't really know how we can actually get unblocked :( | 14:54 |
| jrosser | fork :( | 14:57 |
| jrosser | or we lift/shift the role that calls the broken module into the ops repo | 14:57 |
| jrosser | and use the fqcn for the module | 14:57 |
| jrosser | and the role | 14:57 |
| jrosser | becasue we have our own playbooks i think? | 14:58 |
| noonedeadpunk | yes, but it's ceph_facts that's just included from other role.... | 15:01 |
| noonedeadpunk | so it's useless... | 15:01 |
| NeilHanlon | ope seems like i've missed a lot the past week or so.. | 15:01 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Nov 28 15:01:27 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| noonedeadpunk | #topic rollcall | 15:01 |
| noonedeadpunk | o/ | 15:02 |
| noonedeadpunk | #topic office hours | 15:10 |
| noonedeadpunk | So, given we're really on a deadline for 2023.2, I think we should leave openstack_resources for the next cycle | 15:10 |
| noonedeadpunk | And I will propose roles branching soonish to meet the deadline | 15:10 |
| noonedeadpunk | I've looked briefly through outstanding patches, and things look okeyish | 15:11 |
| noonedeadpunk | The most critical thing (by far) are for integrated repo | 15:12 |
| noonedeadpunk | basically this: https://review.opendev.org/c/openstack/openstack-ansible/+/901204 | 15:12 |
| NeilHanlon | I agree re: openstack_resources if we still need some time to iterate on it. would rather have it finalized more or less | 15:13 |
| noonedeadpunk | I also start seeing more rocky failures lately fwiw... | 15:13 |
| noonedeadpunk | With things like before: `"Failed to download metadata for repo 'baseos': Yum repo downloading error: Downloading error(s): repodata/0e0e0252-80d4-47bf-bb11-1f97d7aac5b0-PRIMARY.xml.gz - Cannot download, all mirrors were already tried without success` | 15:14 |
| noonedeadpunk | example: https://zuul.opendev.org/t/openstack/build/80791ccb907f47a2889e8b65c3abc367 | 15:14 |
| noonedeadpunk | NeilHanlon: had you any luck talking to infra ppl about mirrors? | 15:14 |
| NeilHanlon | yea, mnasiadka started the conversation and I need to follow back up with them. it seems like it's probably OK but they need some more info around the mirror source, how to verify they're up to date, etc | 15:17 |
| noonedeadpunk | ok, I see | 15:18 |
| noonedeadpunk | We had actually couple of bugs reported on launchpad. | 15:19 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2044629 | 15:19 |
| noonedeadpunk | this one I was not able to reproduce. | 15:20 |
| noonedeadpunk | Also I believe we should be having a job now that would catch invalid SHAs. | 15:20 |
| noonedeadpunk | ANd also, as Neil rightfully stated, Rocky 8 for 2023.1 is not a good combination anyway | 15:20 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2038818 | 15:21 |
| mnasiadka | Rocky 8? That's rather Yoga material ;-) | 15:22 |
| noonedeadpunk | I was looking through it shortly, but I'm actually not sure what we can/should do at this point | 15:22 |
| noonedeadpunk | mnasiadka: it exactly is! | 15:22 |
| noonedeadpunk | But I was considering it only because issue was related to `git clone` basically, which should not differ much | 15:23 |
| noonedeadpunk | andrewbonney: do you have anyy suggestions for 2038818? | 15:24 |
| andrewbonney | Not really. I couldn't see an easy option to work out which flags to switch on ahead of upgrade | 15:25 |
| andrewbonney | If we supported rolling upgrades it would be easier to avoid breaking things inadvertently, but that doesn't help with identifying the flags to switch on | 15:25 |
| noonedeadpunk | Well. We've enabled maintenance flag and used it couple of times. | 15:25 |
| noonedeadpunk | I can't say that helped.... | 15:26 |
| noonedeadpunk | As each time node was returning from maintenance consumers were still crashing and not always recovering | 15:26 |
| noonedeadpunk | So use maintenance and "rolling" upgrades still with caution ;) | 15:29 |
| noonedeadpunk | I had reported one bug as well btw | 15:29 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2044229 | 15:29 |
| noonedeadpunk | where basically, from some point in time, our connection plugin does not respect remote_user provided | 15:30 |
| noonedeadpunk | but ansible_user is being respected | 15:30 |
| noonedeadpunk | I didn't have to look into more details yet, but will do so | 15:30 |
| jrosser | i was sure there was some stuff in the tests repo to check user stuff with the connection plugin | 15:31 |
| jrosser | though perhaps that was more related to become: | 15:31 |
| noonedeadpunk | yes, become works as well | 15:31 |
| noonedeadpunk | what doesn't work basically is `openstack-ansible -u ubuntu` | 15:31 |
| noonedeadpunk | it's always end up as root | 15:32 |
| noonedeadpunk | but `openstack-ansible -e ansible_user=ubuntu` would work | 15:32 |
| noonedeadpunk | which was slightly /o\ | 15:32 |
| noonedeadpunk | As very confusing also to see correct user in ouput right before connection command, but not in the SSH itself | 15:34 |
| jrosser | NeilHanlon: would a rocky mirror actually help with `Failed to download metadata for repo 'baseos'` | 15:34 |
| NeilHanlon | i think it should, yes. the 'problem' so to speak is that the default configuration for our repos uses the mirrorlist URL, which can go to, more or less, anywhere (it's Geo-based, but, will fill in). so hard coding that to anything (even our canonical dl.rockylinux.org) should mitigate some percentage of the spurious failures | 15:36 |
| noonedeadpunk | So maybe we should do that then? | 15:37 |
| jrosser | is there an interim way we can do that in the zuul pre job? | 15:38 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Add no_log to setup_roles inlcude https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/901755 | 15:38 |
| noonedeadpunk | there;s some preped nodepool file that we consume iirc... | 15:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins stable/2023.1: Add no_log to setup_roles inlcude https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/902068 | 15:39 |
| jrosser | https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_aio_config.yml#L77-L92 | 15:41 |
| noonedeadpunk | `/etc/ci/mirror_info.sh` | 15:41 |
| noonedeadpunk | yeah that actually works as well | 15:42 |
| jrosser | here we grab it https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/gather_nodepool_vars.yml#L27 | 15:42 |
| noonedeadpunk | just found it as well | 15:43 |
| noonedeadpunk | So I guess we can add some logic for rocky specifically | 15:43 |
| NeilHanlon | the 'easy' thing is to just search/replace /etc/yum.repos.d/*.repo and replace `^mirrorlist` with `#mirrorlist` and `^#baseurl` with `baseurl` | 15:43 |
| NeilHanlon | https://git.resf.org/sig_cloud/images/src/branch/main/elements/rocky-minimal-yum/pre-install.d/03-yum-cleanup#L39 (a new thing I've been messing around with 👀 | 15:44 |
| noonedeadpunk | I'm actually confused about openstack_hosts_centos_mirror_url | 15:45 |
| noonedeadpunk | as well as centos_mirror_url | 15:45 |
| noonedeadpunk | as they're actually never used today? | 15:45 |
| noonedeadpunk | and eventually we don't touch default mirrors provided/configured for the image | 15:47 |
| NeilHanlon | 🤔 | 15:47 |
| NeilHanlon | yea i don't really see it used either | 15:47 |
| jrosser | that really is useless vars | 15:48 |
| noonedeadpunk | uca_apt_repo_url was jsut renamed to apt_repo_url | 15:49 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/releasenotes/notes/apt_repo-097cb34c12e60207.yaml | 15:49 |
| noonedeadpunk | probably openstack_hosts_centos_mirror_url was also for RDO specifically? | 15:49 |
| noonedeadpunk | but yes, that's need some love I believe... | 15:50 |
| jrosser | shame no-one tries "difficult" installed (like limited connectivity) with RH variants | 15:50 |
| NeilHanlon | i will poke at a zuul override for rocky using this, and stack it with a cleanup of that cruft | 15:50 |
| jrosser | becasue this surfaces all sorts of corner cases | 15:50 |
| NeilHanlon | agreed jrosser | 15:50 |
| NeilHanlon | we have also been considering releasing a package which provides an easy way to manipulate repo configs, as we often have our testing team need to perform this very operation (change mirrorlist->baseurl and modify contentdir) and it gets... boring | 15:51 |
| noonedeadpunk | yeah, totally understandable. but package sounds... slightly excessive?:)) | 15:57 |
| noonedeadpunk | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Tue Nov 28 15:58:35 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-28-15.01.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-28-15.01.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-11-28-15.01.log.html | 15:58 |
| NeilHanlon | heh, maybe | 16:11 |
| NeilHanlon | thanks for running the meeting noonedeadpunk :) | 16:11 |
| NeilHanlon | as always please ping me if reviews are needed. will try to keep an eye though | 16:11 |
| noonedeadpunk | NeilHanlon: sure thing, thank for stopping by:) | 16:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!