| noonedeadpunk | good morning | 07:19 |
|---|---|---|
| noonedeadpunk | yesterday we've got couple of new bug reports, so going to look through them | 07:19 |
| jrosser | o/ morning | 07:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Define default value for `neutron_default_availability_zones` https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930265 | 07:51 |
| jrosser | i wonder where we should ensure that /etc/apt/sources.list.d exists | 08:01 |
| jrosser | openstack_hosts doesnt really have an existing function for making directories | 08:01 |
| noonedeadpunk | that's a good question | 08:05 |
| jrosser | there is `openstack_hosts/tasks/openstack_hosts_configure_apt.yml` | 08:06 |
| jrosser | maybe thats the best place - it gets run on the host before the containers are created, so we should only need to add it once there | 08:06 |
| noonedeadpunk | right efore where repo is being added seems like not bad place | 08:07 |
| noonedeadpunk | yeah | 08:07 |
| noonedeadpunk | btw, talking about that - there's also a fresh bug report on replated topic: https://bugs.launchpad.net/openstack-ansible/+bug/2081775 | 08:07 |
| jrosser | i wonder why we dont see that fail | 08:09 |
| noonedeadpunk | do we test lxc on debian even? | 08:10 |
| noonedeadpunk | oh, yes, we do | 08:10 |
| noonedeadpunk | and they've submitted another one for arm: https://bugs.launchpad.net/openstack-ansible/+bug/2081764 | 08:11 |
| jrosser | doh ok | 08:12 |
| jrosser | i will need to make an experiment for that, its an ugly fixup already | 08:13 |
| noonedeadpunk | I'm thinking about just some mapping? | 08:13 |
| jrosser | well, it's either a list or a string | 08:13 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Ensure apt sources.list.d directory exists. https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930272 | 08:13 |
| jrosser | and it can be many architectures, some of which need fixing | 08:14 |
| noonedeadpunk | um, any reason to add many architectures to the same host? | 08:14 |
| noonedeadpunk | I checked and UCA has same thing - arm64, not aarch | 08:15 |
| jrosser | well - i just have made the code "pass through" then functionality of the ansible module | 08:15 |
| noonedeadpunk | I can recall having some arch mapping somewhere already... | 08:16 |
| jrosser | lxc i think? | 08:16 |
| jrosser | though we did get rid of a lot of complexity there | 08:16 |
| noonedeadpunk | yeah - _architecture_fixup | 08:16 |
| noonedeadpunk | * https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/defaults/main.yml#L29-L34 | 08:16 |
| noonedeadpunk | so pretty much - we can move such mapping to global level? | 08:17 |
| jrosser | that makes the roles not self contained then | 08:18 |
| noonedeadpunk | oh, yes, there's galera and rabbit as well | 08:18 |
| noonedeadpunk | as for lxc and openstack_hosts - they're probably never were self contained? | 08:18 |
| noonedeadpunk | then yeah - copy/paste I guess... | 08:20 |
| jrosser | aaah the osbpo key thing is bacuse we don't vendor the key into the repo | 08:20 |
| jrosser | we give an external url to the key instead | 08:20 |
| noonedeadpunk | ah | 08:20 |
| jrosser | i'll fix that | 08:20 |
| jrosser | as it's not helpful for offline things | 08:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930278 | 08:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Enable UCA repo for ubuntu noble https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/929631 | 08:37 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930278 | 08:38 |
| jrosser | noonedeadpunk: map('extract', openstack_architecture_mapping) would i think let you keep the original list / not list behaviour | 08:43 |
| jrosser | i think that will replace in turn each list member from the input without needing a loop | 08:44 |
| noonedeadpunk | I guess I don't get list/non-list issue? | 08:44 |
| noonedeadpunk | as module seems to accept both ? | 08:44 |
| noonedeadpunk | it's like package - where you can supply a list of packages or a single one, no? | 08:44 |
| jrosser | well its just that to expose the complete functionality of the deb822_repository module, the `architectures` parameter can be either list or string | 08:46 |
| jrosser | so i had a allowed that to be the case in the vars/debian.yml vars/ubuntu.yml repo definitions too | 08:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Vendor osbpo gpg key into the role https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930280 | 08:51 |
| noonedeadpunk | yeah, but why to control that? Whatrever user will provide in `_package_repos` for specific repo will be jsut passed to the module | 08:57 |
| noonedeadpunk | and then module on it's own doesn't care if it's a single element or a list | 08:58 |
| noonedeadpunk | one repo can be string, another list... whatever? | 08:58 |
| jrosser | well maybe you're right - i've put the complexity in the wrong place | 09:00 |
| jrosser | trying to fix up the data as it gets passed to the module, rather than have it be fixed up in the input data | 09:01 |
| noonedeadpunk | it wasn't bad, just if we need to have quite some arches supported - that become a bit tough... | 09:01 |
| noonedeadpunk | or well, dunno | 09:01 |
| jrosser | btw i am not really understanding whats happening with https://bugs.launchpad.net/openstack-ansible/+bug/2081775 | 09:02 |
| noonedeadpunk | if you wanna keep the _architecture_fixup - we can do that as well I guess | 09:02 |
| jrosser | becasue in our CI jobs the key is actually in /etc/apt/trusted.gpg | 09:02 |
| jrosser | well - _architecture_fixup is messy and it's likley better to simplify it | 09:03 |
| jrosser | so i think your solution is better/simpler actually | 09:03 |
| noonedeadpunk | regarding `/etc/apt/keyrings` - no idea actually | 09:05 |
| noonedeadpunk | I was thinking to spawn a debian sandbox | 09:05 |
| noonedeadpunk | I wonder if that has smth to do with deploy host being on Ubuntu | 09:05 |
| jrosser | yeah there is something odd there | 09:07 |
| jrosser | we are definatly missing copying the keyrings directory though | 09:07 |
| jrosser | but when i look on a zuul job log both host and container /etc/apt/keyrings directory is empty | 09:08 |
| jrosser | btw 930278 is not a direct fix for 2024.1 - only master | 09:10 |
| noonedeadpunk | yeah, as we don't use deb822 on 2024.1 yet | 09:15 |
| jrosser | and looking on stable/2024.1 i'm not sure i see architecture being used in the repo defnintion | 09:16 |
| noonedeadpunk | well it feels that reporter does not actualy use 2024.1 | 09:17 |
| noonedeadpunk | as they supply master links and reffer to it as well. | 09:17 |
| noonedeadpunk | but dunno | 09:17 |
| jrosser | yeah, perhaps we should ask about that | 09:17 |
| jrosser | i can try to make a debian/12 arm vm and try | 09:17 |
| noonedeadpunk | I wonder if we should just drop arch from repo definition instead :D | 09:19 |
| jrosser | fwiw we have an amd64 deploy host here and one aarch64 infra host in each deployment | 09:19 |
| jrosser | so this does work, at least for bobcat | 09:19 |
| noonedeadpunk | yeah, this could be really jsut master issue, as no changes in repos were made between bobcat and caracal | 09:20 |
| noonedeadpunk | though, I assume you haven't re-created containers at least since antelope | 09:21 |
| noonedeadpunk | so that could slip on bobcat | 09:21 |
| jrosser | probably not, and its a bit unusual as i think the aarch64 host only has utility and repo on it | 09:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/930282 | 09:22 |
| noonedeadpunk | oh, yeah. true as well | 09:23 |
| jrosser | we just sit that sort of off to the side as a 4th very minimal infra host for buiding wheels | 09:24 |
| noonedeadpunk | I guess there could be arm-only deployments... But yeah.... | 09:24 |
| jrosser | and the original 3 x86 ones are much more as you'd epect | 09:24 |
| jrosser | the bug report about osbpo keyring is also consistent with using master imho | 09:25 |
| noonedeadpunk | ++ | 09:25 |
| noonedeadpunk | so probably good as we likely don't need to backport anything | 09:26 |
| noonedeadpunk | and well - good it was raised timely :) | 09:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/930283 | 09:27 |
| noonedeadpunk | fwiw that seems to be rax who reported issues :) | 09:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Enable UCA repo for ubuntu noble https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/929631 | 09:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930278 | 09:35 |
| jrosser | oh well thats nice :) | 09:36 |
| jrosser | need to encourage them to join us here | 09:37 |
| noonedeadpunk | I think they're kind of one leg into k8s deployment already though | 09:37 |
| noonedeadpunk | just "old" deployments which left or so... | 09:37 |
| noonedeadpunk | but not sure | 09:38 |
| jrosser | yeah i've seen what cloudnull was doing | 09:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/930284 | 09:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Ensure that selected Apache MPM is enforced https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/929695 | 09:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Convert Skyline HAProxy httpcheck https://review.opendev.org/c/openstack/openstack-ansible/+/929892 | 09:49 |
| noonedeadpunk | jrosser: do you have same compute names in hypervisor list vs compute service list? Or fqdn vs hostname ? | 10:52 |
| jrosser | noonedeadpunk: they are the same, and they are fqdn | 10:54 |
| noonedeadpunk | ok, asking in context of https://opendev.org/openstack/openstack-ansible/blame/branch/master/scripts/upgrade-utilities/nova-restore-compute-id.yml#L33 | 10:54 |
| noonedeadpunk | as for us in most deployments `ansible_facts['fqdn']` will be correct option | 10:54 |
| jrosser | so for me ansible_nodename is also the fqdn | 10:57 |
| noonedeadpunk | ++ | 10:58 |
| jrosser | ansible_fqdn is "localhost"...... | 10:58 |
| * jrosser surprised | 10:58 | |
| noonedeadpunk | wait, what? | 11:00 |
| noonedeadpunk | smth sounds off with /etc/hosts I'd guess | 11:01 |
| jrosser | huh | 11:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use node FQDN for nova-restore-compute-id https://review.opendev.org/c/openstack/openstack-ansible/+/930292 | 11:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Change example to contain domain name instead of UUID https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/919563 | 11:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Change example to contain domain name instead of UUID https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/919563 | 11:21 |
| gokhan | hello noonedeadpunk, venv_wheels_rebuild is not wirking in skyline role. there is also a minor issue about masakari dashboard on skyline. In masakari role service type is ha https://github.com/openstack/openstack-ansible-os_masakari/blob/master/defaults/main.yml#L119 but in skyline role, it is defined as instance-ha https://opendev.org/openstack/openstack-ansible-os_skyline/src/branch/master/vars/main.yml#L46. | 11:26 |
| noonedeadpunk | gokhan: so you tried to replace instance-ha with ha and it worked? (or vice versa) | 11:31 |
| gokhan | noonedeadpunk, yes it worked | 11:43 |
| gokhan | without this change it can not add masakari proxy setting to skyline vhost config file | 11:44 |
| noonedeadpunk | um... maybe you can propose a change then ?:) | 11:46 |
| noonedeadpunk | preferably - to skyline role :) | 11:48 |
| jrosser | is it that we actually have it inconsistent in the masakari role (codesearch finds lots of instance-ha) | 11:50 |
| gokhan | jrosser, yes I think masakari role is inconsistent because in masakari installation guide service type is instance-ha | 11:51 |
| gokhan | https://docs.openstack.org/masakari/latest/install/install_and_configure_ubuntu.html | 11:51 |
| gokhan | noonedeadpunk, we need to also update zun role | 11:52 |
| jrosser | zun is likley very broken | 11:52 |
| noonedeadpunk | well. We can actually patch masakari role. but then there's a question about if (and how long) we wanna backport it | 11:52 |
| gokhan | https://review.opendev.org/c/openstack/kolla-ansible/+/904164 | 11:52 |
| gokhan | like kolla ansible did, we make changes on zun role it worked. we don't need to pin docker version | 11:53 |
| gokhan | we do not need to install etcd also | 11:53 |
| jrosser | gokhan: if you are using the zun role i would highly highly recommend taking on the maintaance of it | 11:53 |
| jrosser | if you've been able to make local patches that work, then you have done 99% of whats needed to contribute the necessary fixes | 11:54 |
| gokhan | jrosser, yes we are using it. we are very busy for internal works in these days, but next week we will plan to commit changes | 11:56 |
| noonedeadpunk | yeah, as the thing is, that we don't have Zun usage here, so really hard to catch-up with it | 11:57 |
| jrosser | same here - we do not use it and I have no spare time to learn it | 11:57 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/2024.1: Do not kill ipsec on L3 cleanup https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930188 | 12:02 |
| gokhan | ok harun will send the patches next week for zun, there are no new features on zun. Harun can fix the gates | 12:04 |
| noonedeadpunk | that would be actually great | 12:05 |
| noonedeadpunk | we will help where we can, but hard to justify time to fully dive into service that there're no plans to use :( | 12:07 |
| noonedeadpunk | I will check on masakari now | 12:07 |
| jrosser | when can we update this? https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/defaults/main.yml#L22 | 12:16 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia stable/2024.1: Provide better flexability for SSH keypair options https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/929649 | 12:24 |
| noonedeadpunk | jrosser: once rdo/ospb/uca release dalmatian packages | 13:00 |
| jrosser | i think that uca and debian are released | 13:00 |
| jrosser | oh maybe there is something for rdo too https://trunk.rdoproject.org/centos9-dalmatian/report.html | 13:02 |
| * jrosser not sure about that | 13:03 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Define default value for `neutron_default_availability_zones` https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930265 | 13:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Define default value for `neutron_default_availability_zones` https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930265 | 13:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Rename Masakari service type https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/930329 | 13:37 |
| noonedeadpunk | gokhan: ^ this should address masakari thing | 13:38 |
| noonedeadpunk | jrosser: sorry for stupid question, but does octavia dashboard work for you in Horizon as a user? | 14:21 |
| andrewbonney | noonedeadpunk: I've just taken a look and it certainly displays load balancers at least | 14:29 |
| jrosser | andrewbonney: which release did you look at? | 14:29 |
| andrewbonney | That's OSA 29.0.0 | 14:31 |
| noonedeadpunk | huh | 14:37 |
| noonedeadpunk | somehow for me it shows for admin, but not for any user on 29.0.0 | 14:38 |
| noonedeadpunk | so really o_O about why | 14:38 |
| noonedeadpunk | ok, will try to re-spawn things then, thanks! | 14:38 |
| * NeilHanlon waves g'morning | 15:00 | |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:01 |
| opendevmeet | Meeting started Tue Sep 24 15:01:08 2024 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:01 |
| noonedeadpunk | #topic rollcall | 15:01 |
| NeilHanlon | o/ | 15:01 |
| noonedeadpunk | o/ hey Neil! | 15:01 |
| damiandabrowski | hi! | 15:01 |
| NeilHanlon | hi damiandabrowski, noonedeadpunk :) | 15:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Link plugin settings extension separately https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/930346 | 15:05 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | so, we do have bunch of things for review | 15:05 |
| noonedeadpunk | and our gates on master are blocked by this: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930272 | 15:05 |
| NeilHanlon | +2 from me :) (and the W+1) | 15:06 |
| noonedeadpunk | nice, thanks a lot! | 15:07 |
| noonedeadpunk | we also got quite some bug reports this week already | 15:07 |
| noonedeadpunk | I was trying to go through them, but facing more and more nits along the way | 15:07 |
| noonedeadpunk | I think https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/929549 was one of latest backports to 2024.1 I wanted to have before making a release | 15:09 |
| noonedeadpunk | ah, and ofc https://review.opendev.org/q/topic:%22osa/apache_mpm_alignment%22 is quite a topic on itself | 15:09 |
| noonedeadpunk | but it kinda is also needs to be backported.. | 15:10 |
| NeilHanlon | fun... | 15:10 |
| noonedeadpunk | as on metal they do conflict, esp if skyline is in | 15:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Link plugin settings extension separately https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/930346 | 15:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Link plugin settings extension separately https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/930346 | 15:11 |
| noonedeadpunk | and with that new release is coming rapidly | 15:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Define default value for `neutron_default_availability_zones` https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930265 | 15:14 |
| NeilHanlon | yep yep | 15:15 |
| noonedeadpunk | there's also an update regarding mariadb 11.4 - the issue where we'd had to issue SSL cert for localhost was fixed upstream | 15:15 |
| noonedeadpunk | So it should be in the next release | 15:16 |
| noonedeadpunk | though, I don't know when it will be :( | 15:16 |
| noonedeadpunk | to be specific - in 11.4.4 | 15:17 |
| NeilHanlon | at least it's being fixed :) | 15:18 |
| NeilHanlon | on rocky mirrors: I met again w/ infra group last week and presented some findings--result of which is I will put in a Change to setup the sync for Rocky mirrors and then work with infra to get an afs share/quota configured | 15:19 |
| noonedeadpunk | ok, that is a nice update | 15:19 |
| noonedeadpunk | I haven't seen Rocky specific failures due to mirrors last week, fwiw | 15:20 |
| noonedeadpunk | so it seems they've improved a lot since ... winter? | 15:20 |
| NeilHanlon | if my theory is correct, there should be some failures in a day or so once we release a batch of updates RH put out last night | 15:20 |
| noonedeadpunk | I also had some fun with OVN and Neutron lately, found 1000x regression in performance, which also affected Nova listings | 15:23 |
| noonedeadpunk | fix for that landed yesterday for neutron and backports were proposed as well. | 15:23 |
| NeilHanlon | ouch! | 15:24 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/neutron/+/929941 | 15:24 |
| noonedeadpunk | And I do recall someone coming to IRC with question about performance regression after upgrade | 15:24 |
| noonedeadpunk | so here we are... | 15:24 |
| jrosser | o/ hello | 15:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Map all relevant architectures for deb822 repository setup https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/930283 | 15:26 |
| noonedeadpunk | jrosser: where we are with moving playbooks to collections? | 15:30 |
| noonedeadpunk | only services are left? | 15:30 |
| jrosser | ah yes, i have not yet managed to look at the remainder | 15:30 |
| jrosser | openstack services would indeed be the next thing to do | 15:30 |
| jrosser | which will leave some leftovers i think | 15:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix upgrade job on master to upgrade from 2024.1 to master https://review.opendev.org/c/openstack/openstack-ansible/+/928771 | 15:32 |
| noonedeadpunk | ok, I can look into that I guess | 15:32 |
| noonedeadpunk | once CI is unblocked I wanna take another look at SHA bump failres on master | 15:34 |
| noonedeadpunk | and potentially switch things to stable branches to track them | 15:34 |
| jrosser | what is left to make a release on Caracal? | 15:36 |
| jrosser | do you want to get the mpm stuff backported for that? | 15:36 |
| noonedeadpunk | I think so... | 15:36 |
| noonedeadpunk | as that is quite breaking change | 15:36 |
| noonedeadpunk | and very unpleasant issue to deal with | 15:37 |
| noonedeadpunk | and also https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/929549 | 15:37 |
| noonedeadpunk | I think that's all | 15:37 |
| noonedeadpunk | I don't know anything else which should be considered as critical for upgrade | 15:37 |
| noonedeadpunk | especially with recent neutron backports :) | 15:40 |
| noonedeadpunk | as our 29.1.0 is smth I'd consider minorly upgrading to, hehe | 15:41 |
| noonedeadpunk | (is going to be) | 15:41 |
| noonedeadpunk | damiandabrowski: it would be also nice if you could spend some time on reviews this week | 15:42 |
| damiandabrowski | okok, i will! | 15:42 |
| noonedeadpunk | ah - actually this patch is potentially a point for discussion: https://review.opendev.org/c/openstack/openstack-ansible/+/929775 | 15:44 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible/+/929775 | 15:45 |
| noonedeadpunk | while it kind of make sense - I think it's might be better to patch haproxy | 15:45 |
| noonedeadpunk | is there reason why we define SSL for `bind` if haproxy_balance_type is tcp? | 15:46 |
| noonedeadpunk | #link https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/templates/service.j2#L56 | 15:46 |
| jrosser | the commit message is not helping me much there | 15:47 |
| jrosser | were is this user certificate? | 15:47 |
| noonedeadpunk | if I'm not mistaken - it's about like any certificate | 15:47 |
| jrosser | can't you put sometimes tls on the console service itself | 15:48 |
| jrosser | and then this patch talks about haproxy, which is another place there could be a certficate | 15:48 |
| noonedeadpunk | I think what they are into is this | 15:49 |
| noonedeadpunk | #link https://zuul.opendev.org/t/openstack/build/a4616d160b9249289a16b5980bf3e58a/log/logs/etc/host/haproxy/conf.d/nova_novnc_console.txt#6 | 15:49 |
| noonedeadpunk | or well | 15:50 |
| noonedeadpunk | #link https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/nova_all/haproxy_service.yml#L19-L20 | 15:51 |
| jrosser | this is complicated | 15:51 |
| noonedeadpunk | so if they do define `nova_console_user_ssl_cert` - they don't want haproxy to be configured with TLS | 15:51 |
| jrosser | as there is compute hosts <> novnc proxy | 15:52 |
| jrosser | novnc proxy <> haproxy | 15:52 |
| jrosser | and haproxy <> user | 15:52 |
| jrosser | and afaik we have have tls in all of those places | 15:52 |
| noonedeadpunk | yes, true, but then we do have `haproxy_nova_console_http_mode | ternary('http', 'tcp')` | 15:52 |
| noonedeadpunk | so if there's a `tcp` - we should not be adding TLS to BIND | 15:53 |
| noonedeadpunk | I don't think it will matter or terminate TLS on haproxy at all | 15:53 |
| noonedeadpunk | but having `ssl` statement on bind is confusion | 15:53 |
| noonedeadpunk | I'm not 100% sure if I got it correctly | 15:54 |
| noonedeadpunk | as you said - it's really complicated | 15:54 |
| noonedeadpunk | and I personally can hardly asses what is intented behaviour here in fact | 15:54 |
| jrosser | no i am not able to make a clear understanding of the actual issue without much more thought | 15:56 |
| noonedeadpunk | but what I'm into - is that probably having ssl defined in haproxy when `mode tcp` doesn't make sense? | 15:57 |
| noonedeadpunk | as my guess was that it's the thing which is confusing | 15:58 |
| jrosser | i think that we might have really old code here for the user_cert stuff | 15:58 |
| jrosser | like here https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/consoles/nova_console_novnc_ssl.yml | 15:59 |
| noonedeadpunk | oh, wow, yes | 15:59 |
| noonedeadpunk | it might work though | 16:00 |
| noonedeadpunk | but doesn't make much sense indeed | 16:00 |
| jrosser | i don't know why this is not covered by the pki role | 16:01 |
| jrosser | as i do think that james worked specifically on tls for the consoles | 16:01 |
| jrosser | and those vars are not in defaults/main.yml for the nova role either | 16:02 |
| noonedeadpunk | yep, seems like leftover which needs to be covered as well | 16:02 |
| noonedeadpunk | so good patch from standpoint of raising attention at least | 16:03 |
| noonedeadpunk | #endmeeting | 16:04 |
| opendevmeet | Meeting ended Tue Sep 24 16:04:31 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:04 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-09-24-15.01.html | 16:04 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-09-24-15.01.txt | 16:04 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-09-24-15.01.log.html | 16:04 |
| jrosser | what actually is this even doing https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/templates/nova.conf.j2#L22-L27 | 16:06 |
| jrosser | it's not in a specific config section at all | 16:06 |
| noonedeadpunk | well, it's a valid config option... | 16:07 |
| noonedeadpunk | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.ssl_only | 16:07 |
| noonedeadpunk | I'm not sure it's related in any way though | 16:08 |
| jrosser | cert for what though :) | 16:08 |
| noonedeadpunk | and cert even does have console in related options... | 16:08 |
| noonedeadpunk | https://docs.openstack.org/nova/latest/configuration/config.html#console.ssl_ciphers | 16:09 |
| noonedeadpunk | so I assume, that's for nova-novncproxy service | 16:09 |
| noonedeadpunk | which is stupid enough and takes just defaults | 16:09 |
| noonedeadpunk | and that part could end up after merging all config files together for nova years ago | 16:11 |
| noonedeadpunk | but then I really don't know where this connection stands in global view | 16:11 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817222 | 16:12 |
| noonedeadpunk | oh, well | 16:13 |
| noonedeadpunk | but this is different | 16:13 |
| noonedeadpunk | I guess that's an encryption between compute and nova-novncproxy? | 16:13 |
| noonedeadpunk | but then proxy <-> haproxy is still unencrypted? | 16:14 |
| noonedeadpunk | as 6080 port - where you connect to novnc - is actually nova-novncproxy. and then proxy connects to vnc on computes, and it's where vencrypt makes difference? | 16:15 |
| jrosser | right yes ok | 16:15 |
| noonedeadpunk | yeah, I'm just realizing that we have this connection hardly covered right now | 16:16 |
| jrosser | so we do seem to have some mess for what happens with novnc proxy <> haproxy | 16:16 |
| noonedeadpunk | or well - in a very weird way | 16:16 |
| noonedeadpunk | yeah, exactly | 16:16 |
| jrosser | i don't really understand why we have a user certificate option for this | 16:16 |
| noonedeadpunk | as then we somehow enforce TCP connection? | 16:16 |
| jrosser | that sounds like something from the past (like we also had for horizon and keystone), but now the pki role makes that pretty redundant | 16:16 |
| noonedeadpunk | well, whatever, we provide user certs for many things. but how it's done does not make sense | 16:17 |
| jrosser | perhaps this got missed with the internal tls work | 16:17 |
| noonedeadpunk | yeah, likely | 16:17 |
| jrosser | or backend tls work, i mean | 16:17 |
| jrosser | i think james had done a whole bunch of stuff around nova, for tls live migration etc, and also covered the compute<>novncproxy part | 16:18 |
| noonedeadpunk | oh yes, that is covered for sure | 16:18 |
| noonedeadpunk | and then Damian was looking into internal TLS, which is this specific thing | 16:18 |
| jrosser | so kind of question is if really there is a sensible fix for master | 16:18 |
| jrosser | or if it is just wrong and needs to come into alignment with the rest of things | 16:19 |
| jrosser | i.e is it ever valid to apply tls on the novnc proxy and pass that through TCP at haproxy | 16:19 |
| noonedeadpunk | I don't think it makes sense to pass TCP at all in this case | 16:20 |
| jrosser | no, i agree | 16:20 |
| noonedeadpunk | so it should act as all others | 16:20 |
| noonedeadpunk | but then it feels that we might need to have a separate conf file for novnc proxy | 16:21 |
| noonedeadpunk | I kinda scared of [DEFAULT]/ssl_only | 16:21 |
| noonedeadpunk | but there's no other place to define that for novnc | 16:22 |
| noonedeadpunk | *proxy | 16:22 |
| jrosser | oh is it all related to this https://opendev.org/openstack/nova/src/branch/master/doc/source/cli/opts/websockify.rst | 16:23 |
| noonedeadpunk | how it's all related to ipv6 /o\ | 16:24 |
| jrosser | oh wtf :) | 16:24 |
| noonedeadpunk | I want to unsee this one | 16:25 |
| jrosser | oh its jsut formatting though isnt it? | 16:25 |
| noonedeadpunk | ah, lol, true | 16:26 |
| jrosser | https://opendev.org/openstack/nova/raw/branch/master/doc/source/cli/opts/websockify.rst | 16:26 |
| jrosser | but anyway those are truly terrible config option names, without a specific section | 16:26 |
| noonedeadpunk | but it's same as in config reference then | 16:26 |
| jrosser | hmm ok so we need to add this to the todo list :/ | 16:29 |
| noonedeadpunk | I added to https://etherpad.opendev.org/p/osa-epoxy-ptg | 16:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Define default value for `neutron_default_availability_zones` https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/930265 | 16:51 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Ensure apt sources.list.d directory exists. https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930272 | 16:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Increase MPM thread limits for Apache https://review.opendev.org/c/openstack/openstack-ansible/+/930362 | 17:05 |
| jrosser | ^ tiny typo in the commit message there "thereads" | 17:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Increase MPM thread limits for Apache https://review.opendev.org/c/openstack/openstack-ansible/+/930362 | 17:08 |
| noonedeadpunk | ah, damn, likely https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/929540 is also a plus for 2024.1 release, as otherwise centos will fail right away.... | 17:10 |
| noonedeadpunk | (or any alternative to it) | 17:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Ensure apparmor folder exists for ceph caching https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/929606 | 17:19 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use ceph mirror in CI jobs https://review.opendev.org/c/openstack/openstack-ansible/+/923777 | 17:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Switch codename to Dalmatian for 2024.2 https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/930368 | 17:24 |
| jrosser | i think we really need those mpm fixes in | 18:02 |
| noonedeadpunk | yeah, mpm as well | 18:04 |
| noonedeadpunk | damiandabrowski: can you check on https://review.opendev.org/q/topic:%22osa/apache_mpm_alignment%22 better sooner then later? | 18:04 |
| noonedeadpunk | gokhan: fwiw, Zun doesn't look too promising in service CI either: https://review.opendev.org/c/openstack/zun/+/928655 | 18:31 |
| mnaser | noonedeadpunk: have you made any progress on https://bugs.launchpad.net/magnum/+bug/2067345/comments/35 ? it's quite annyoing :\ | 19:32 |
| jrosser | super annoying :( | 19:35 |
| jrosser | we are also revert / pin oslo.db | 19:36 |
| mnaser | jrosser: i wonder if 14.1.0 makes sense -- https://github.com/openstack/oslo.db/compare/14.1.0...15.0.0 | 19:56 |
| mnaser | there's not much change ther, i wonder what is the oldest worknig | 19:56 |
| noonedeadpunk | mnaser: frankly - I spet close to zero time on that so far, except finding the bug and reverting back u-c | 19:57 |
| mnaser | i wonder if its a newer release that broke things, im looking at the diffs | 19:57 |
| noonedeadpunk | the oldest working would be the one having autocommit :D | 19:57 |
| mnaser | blargh i guess its easier to just do the workaround, i may make an ml post about this | 19:59 |
| noonedeadpunk | but no, I think your approach makes most sense | 19:59 |
| noonedeadpunk | It could be some single scenarion missed or covered with writes where only reads are needed or smth like that | 20:00 |
| noonedeadpunk | I was struggling to cover that for Vitrage for quite a while. And sqlite backend was a very-very good way for testing such things | 20:00 |
| mnaser | i mean it definitely could be, i know there's almost always the greenthread with listing all clusters where it is borked | 20:00 |
| noonedeadpunk | as connections there are locking, so tests would fail if connections remain open | 20:01 |
| noonedeadpunk | but I was told to leave the magnum alone for now, so... I did that... | 20:01 |
| mnaser | noonedeadpunk: how were you able to replicate those issues if you dont mind me asking, i can dig in deeper but i'm just at all, cause unit tests dont really trigger that issue | 20:02 |
| mnaser | https://paste.openstack.org/show/825346/ - its stuck on `clusters = objects.Cluster.list(ctx, filters=filters)` in this case | 20:03 |
| mnaser | which goes into `clusters = objects.Cluster.list(ctx, filters=filters)` | 20:03 |
| mnaser | sorry i mean `db_clusters = cls.dbapi.get_cluster_list(context, limit=limit,` | 20:03 |
| mnaser | when i compared to ironic, it's not too far out design wise - https://github.com/openstack/ironic/blob/master/ironic/objects/node.py#L362-L365 | 20:05 |
| noonedeadpunk | mnaser: to be fair - I did not replicated for magnum per say. it was taking us 5mins tops on production to hit this. I was more reffering to my "practise" on vitrage https://opendev.org/openstack/vitrage/commit/d6b5247dbb1acedd43b3dc8e7b1cac06519467c4 | 20:06 |
| noonedeadpunk | but only sqlite tests were catching this on unit | 20:07 |
| mnaser | noonedeadpunk: yeah exactly what i ran into as you ran into, without the context passing :( | 20:07 |
| noonedeadpunk | well, seems magnum uses sqlite by default anyway: https://opendev.org/openstack/magnum/src/branch/master/magnum/tests/conf_fixture.py#L30-L31 | 20:14 |
| noonedeadpunk | so yeah, then it's all irrelevant :( | 20:14 |
| cringdahl | hey, i'm the one that filed this business: | 20:20 |
| cringdahl | https://bugs.launchpad.net/openstack-ansible/+bug/2081831 | 20:20 |
| cringdahl | jrosser: you talked about a patch in your reponse? | 20:21 |
| cringdahl | in all fairness, I discovered the apply_security_hardening var and set it to false, which got me moving again | 20:22 |
| noonedeadpunk | oh yes, that would be one way of doing that. | 20:23 |
| noonedeadpunk | the role needs some love and update as well though | 20:24 |
| noonedeadpunk | it's kind of a pity that amount of donors with arm has dropped dramatically | 20:27 |
| noonedeadpunk | as basically arm64 path in openstack is left almost without testing overall | 20:27 |
| cringdahl | I'm happy to be contributing to the open source community, then, if only with smoke tests | 20:28 |
| jrosser | cringdahl: hi | 20:28 |
| jrosser | I will try to look at a patch for the security hardening role tomorrow to make it arm-aware, unless you want to submit something in the meantime? | 20:29 |
| * noonedeadpunk singing off for today | 20:29 | |
| cringdahl | thanks for that. I'm happy to wait until tomorrow, as I've bypassed with apply_security_hardening. | 20:30 |
| cringdahl | and also i'm still struggling to get the whole thing running. :) | 20:30 |
| cringdahl | anyone know why /etc/glance/glance-api-paste.ini wouldn't get written by role os_glance? | 20:31 |
| cringdahl | I'm digging around, this one is eluding me | 20:31 |
| jrosser | well - one thing you can do is to use the CI jobs as a reference | 20:40 |
| jrosser | in the irc topic you'll see a link to the code review dashboard at http://bit.ly/osa-review-board-v5 | 20:40 |
| jrosser | then i picked a job for the openstack-ansible repo that was passing on master branch https://review.opendev.org/c/openstack/openstack-ansible/+/929636 | 20:41 |
| jrosser | under the zuul summary tab you can see all the job results, so looking in one for debian targets with lxc containers https://zuul.opendev.org/t/openstack/build/bb4980fac89647bc8c5165536276c0c7 | 20:42 |
| jrosser | we can look in the job log and see where the glance-api-paste file is handled https://zuul.opendev.org/t/openstack/build/bb4980fac89647bc8c5165536276c0c7/log/job-output.txt#11001-11015 | 20:43 |
| jrosser | then we can also take a look at what it actually did for that file https://f4f8ba04bafc0163e558-e001c969608179e00bcb6eab56a8fbc9.ssl.cf5.rackcdn.com/929636/1/check/openstack-ansible-deploy-aio_lxc-debian-bookworm/bb4980f/logs/etc/openstack/aio1-glance-container-5f30c3f4/glance/ | 20:44 |
| cringdahl | jrosser: thanks for all that! 'Preserve original configuration file(s)' is where glance-api-paste.ini states failure to even exist, whereas rootwrap.conf comes back fine. | 20:49 |
| cringdahl | I could paste here, but I fear it'll be real spammy | 20:49 |
| jrosser | paste.opendev.org :) | 20:50 |
| cringdahl | sweet | 20:52 |
| cringdahl | https://paste.opendev.org/show/825584/ | 20:53 |
| mnaser | noonedeadpunk, jrosser: posted on the ML, maybe someone has a clue from oslo.db side | 20:53 |
| noonedeadpunk | ++ thanks! | 20:54 |
| jrosser | cringdahl: this is important https://zuul.opendev.org/t/openstack/build/bb4980fac89647bc8c5165536276c0c7/log/job-output.txt#10869 | 21:03 |
| jrosser | /etc/glance is supposed to by symlinked into the built venv for glance | 21:06 |
| cringdahl | jrosser: updated the paste with what I see instead of that output | 21:06 |
| jrosser | i think its made a new paste for you | 21:07 |
| cringdahl | https://paste.opendev.org/show/825586/ | 21:09 |
| cringdahl | There is no /usr/etc in this glance container | 21:09 |
| noonedeadpunk | could it be that wheels/venv build has failed on previous run? | 21:10 |
| noonedeadpunk | as then you might wanna try to re-run the role with `-e venv_rebuild=true` | 21:11 |
| cringdahl | where would that -e flag go? I'm running all this through the openstack-ansible script | 21:13 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Respect defined interface for external VIP with LE https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/924333 | 21:14 |
| jrosser | cringdahl: just to check - this is a source based install? | 21:15 |
| jrosser | the openstack-ansible script is a pretty thin wrapper around ansible-playbook | 21:16 |
| jrosser | so any of the options to ansible-playbook will be also ok as options for openstack-ansible | 21:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/2024.1: Respect defined interface for external VIP with LE https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/930384 | 21:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/2023.2: Respect defined interface for external VIP with LE https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/930385 | 21:20 |
| cringdahl | jrosser: distro based | 21:20 |
| jrosser | cringdahl: i was just going to say | 21:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/2023.1: Respect defined interface for external VIP with LE https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/930386 | 21:20 |
| jrosser | because in your paste https://github.com/openstack/openstack-ansible-os_glance/blob/master/tasks/glance_install.yml#L95 this is false | 21:20 |
| jrosser | so basically don't do a distro install | 21:20 |
| noonedeadpunk | as a matter of fact - distro-based is not really supported for Debian | 21:20 |
| cringdahl | ha | 21:20 |
| cringdahl | dang | 21:21 |
| jrosser | time to start again from fresh with a source install, i'm afraid | 21:21 |
| cringdahl | I got really far! | 21:21 |
| jrosser | you need a really good reason to choose a distro install | 21:21 |
| noonedeadpunk | I've started one day looking into it just to realize that packaging is completely different: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/794220 | 21:21 |
| cringdahl | my really good reason was thinking it would be faster overall | 21:21 |
| noonedeadpunk | yeah, we could implement some protection indeed | 21:22 |
| jrosser | that maybe true, but you will pay a large price in the long term | 21:22 |
| noonedeadpunk | ++ ^ | 21:22 |
| jrosser | and in the short term, our test coverage of distro installs is pretty minimal | 21:22 |
| noonedeadpunk | it's kinda positioned as more secuure, just because distro doe care about all package upgrades, but as consequnece, if you don't manage your own repo mirrors, you can't get reproducible environment | 21:23 |
| cringdahl | noonedeadpunk: installing glance-common in the container satisfied ansible | 21:25 |
| noonedeadpunk | but yeah, it's kinda slightly faster | 21:25 |
| cringdahl | this is absolutely a lab environment, so I'll continue forward with distro for now, and if it gives me a working openstack, i'll let you all know :D | 21:26 |
| noonedeadpunk | it can, but on ubuntu or rocky | 21:26 |
| noonedeadpunk | (or might be centos by luck) | 21:27 |
| noonedeadpunk | also, not all services are covered with distro method for now. So they might want to fallback | 21:27 |
| noonedeadpunk | on dabian I'd guess smth like that for glance would be needed for all services | 21:27 |
| noonedeadpunk | *debian | 21:28 |
| noonedeadpunk | but you;ve already reported really good bugs for master - things we didn't tested but wanted to make right | 21:29 |
| noonedeadpunk | so kudos for that | 21:29 |
| jrosser | cringdahl: the support for distro installs has been added by enthusiastic people close to those distros, we had this for suse in particular in the past | 21:29 |
| jrosser | but unfortunately that has never really translated into long term support or maintainance | 21:30 |
| jrosser | and in the meantime the source installs are super solid and repeatable | 21:30 |
| cringdahl | understood | 21:31 |
| jrosser | and i guess for debian in particular, no-one has yet taken on sorting out distro installs | 21:31 |
| cringdahl | heh | 21:31 |
| cringdahl | typical. i try to do the easiest thing and it winds up being the hardest. story of my career. | 21:32 |
| jrosser | so we have no test coverage at all for debian/distro | 21:32 |
| noonedeadpunk | offtopic - we're getting plenty "Connection failure: The read operation timed out" on fetching a u-c in CI | 21:32 |
| jrosser | ^ i looked at that a bit | 21:32 |
| jrosser | and got completely down the rabbit hole of why the upgrade jobs were on the wrong branch | 21:32 |
| noonedeadpunk | I can say - I have opendev activing weirdly locally as well | 21:32 |
| jrosser | i think it may be down to how we redefined the "base url" for the repos | 21:33 |
| noonedeadpunk | probably I should have reported that to infra though | 21:33 |
| noonedeadpunk | nah, I sometimes get timeouts in browser opening gitea | 21:33 |
| jrosser | oh yes i am getting that too | 21:33 |
| noonedeadpunk | and sometimes some issues with TLS negitiation | 21:33 |
| noonedeadpunk | so I think that what happens in ci as well | 21:34 |
| jrosser | and parts of the page rendering waaaaay behind others | 21:34 |
| noonedeadpunk | ++ | 21:34 |
| jrosser | but anyway i think we broke the detection of repo url starting file:/// | 21:35 |
| jrosser | and thats why we keep rraching out to u-c over https rather than off the disk | 21:35 |
| noonedeadpunk | but it usually happens for upgrade jobs? | 21:46 |
| noonedeadpunk | on N-1 or N-2? | 21:46 |
| noonedeadpunk | as there we never use file:// | 21:46 |
| noonedeadpunk | at least it's what I've spotted | 21:47 |
| jrosser | yes - so thats why i was trying to understand how the reposwere configured for upgrades | 21:51 |
| jrosser | and i wanted to know if we made the correct override for the file:/// repo locations - but there were no logs | 21:51 |
| jrosser | so we had broken logging for upgrades | 21:52 |
| jrosser | also upgrades on the wrong branches | 21:52 |
| jrosser | logging was missing because of parallel, which we patched | 21:52 |
| jrosser | i thought we have been using file:/// for really many releases now | 21:53 |
| noonedeadpunk | I think we do. But we use file:// only when we "rely" on zuul repos, right? | 21:56 |
| noonedeadpunk | and on N-1 we just don't use cached repos right now | 21:56 |
| jrosser | lets look tomorrow :) | 21:57 |
| noonedeadpunk | not to accidentally checkout them by bootstrap-ansible or smth, which would vanish pulled by depends-on patches | 21:57 |
| noonedeadpunk | ++ | 21:57 |
| noonedeadpunk | yeah, midnight here :D | 21:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!