| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Improve regex matching for infra_lxc_validate job https://review.opendev.org/c/openstack/openstack-ansible/+/924155 | 07:18 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use hosts setup playbooks from openstack-ansible-plugins repo https://review.opendev.org/c/openstack/openstack-ansible/+/924259 | 07:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 07:56 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on debian-12 https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923586 | 07:56 |
| jrosser | could do with this merged https://review.opendev.org/c/openstack/openstack-ansible/+/923368 | 07:56 |
| * noonedeadpunk checking | 08:03 | |
| noonedeadpunk | one thing to notice - is that tags: -always won't work without proper apply | 08:04 |
| noonedeadpunk | which, I think, is crucial in this case | 08:04 |
| noonedeadpunk | or pre_tasks are always executed? | 08:07 |
| noonedeadpunk | ok, let's fix that in a follow-up | 08:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update ssh service name for modern debian based systems. https://review.opendev.org/c/openstack/openstack-ansible/+/924306 | 08:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update ssh service name for modern debian based systems. https://review.opendev.org/c/openstack/openstack-ansible/+/924306 | 08:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add infrastructure playbooks to openstack-ansible-plugins collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/924171 | 08:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Properly apply `always` tag to haproxy_endpoint_manage https://review.opendev.org/c/openstack/openstack-ansible/+/924307 | 08:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Skip importing haproxy_service_config with no haproxy hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924308 | 08:51 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Fix incorrect copying of sources.list.d to container image https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924309 | 09:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add libpython entry for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924310 | 09:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Permit Ubuntu Noble for deploy host and targets in requirements checks https://review.opendev.org/c/openstack/openstack-ansible/+/924311 | 09:11 |
| jrosser | noonedeadpunk: i already did look a small amount at noble support as soon as it was released | 09:20 |
| jrosser | but i did not get very far with LXC on noble unfortunately | 09:21 |
| jrosser | i tried again just now and these are the patches that get to failure to start the lxc https://review.opendev.org/q/topic:%22osa/noble%22 | 09:21 |
| jrosser | like this https://paste.opendev.org/show/bspOKueFnRNtFOOMxcKF/ | 09:27 |
| jrosser | oh [ 4728.019644] audit: type=1400 audit(1721208370.799:1246): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=74697 comm="(sd-gens)" flags="rw, rslave" | 09:29 |
| noonedeadpunk | I can recall that they did updated smth for apparmor quite recently (like week or 2 ago) | 09:39 |
| noonedeadpunk | to be slightly less restrictive | 09:39 |
| jrosser | looks like this https://discuss.linuxcontainers.org/t/failed-to-fork-off-sandboxing-environment-for-executing-generators-protocol-error/19521/3 | 09:40 |
| noonedeadpunk | https://www.omgubuntu.co.uk/2024/06/apparmor-update-fix-coming-ubuntu | 09:40 |
| noonedeadpunk | yeah, ok | 09:41 |
| jrosser | and the apparmor file i have installed in my vm here is different from https://github.com/lxc/lxc/blob/lxc-5.0.3/config/apparmor/abstractions/start-container.in | 09:41 |
| jrosser | so somehow this is all messy | 09:42 |
| noonedeadpunk | and here's a blog: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces | 09:42 |
| noonedeadpunk | I wonder if `sysctl -w kernel.apparmor_restrict_unprivileged_userns=0` would help? | 09:42 |
| noonedeadpunk | but lxc shouldn't use unconfined profile on ubuntu iirc | 09:43 |
| noonedeadpunk | maybe it's different though | 09:43 |
| jrosser | huh `lxc.apparmor.profile = generated` | 12:52 |
| noonedeadpunk | huh.... | 12:54 |
| jrosser | so what i did was download and boot the upstream 24.04 image | 12:55 |
| jrosser | *lxc image | 12:55 |
| jrosser | and that line is in the config file, but missing from ours | 12:56 |
| noonedeadpunk | we have that: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all_containers.yml#L19 | 12:58 |
| noonedeadpunk | which I'd assume should end up in container config? | 12:58 |
| noonedeadpunk | and that's the profile: https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/templates/lxc-openstack.apparmor.j2 | 13:00 |
| noonedeadpunk | so apparently, smth is just missing there | 13:00 |
| jrosser | doesnt that get put in the config after the first start of the container? | 13:00 |
| jrosser | the issue i have is that the very first time they are started it all goes bad and they fail | 13:00 |
| noonedeadpunk | I think it should be put on creation | 13:01 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L17-L26 | 13:01 |
| jrosser | at the point of creation i have https://paste.opendev.org/show/boQ7IQNNU7qMZwktdhPk/ | 13:01 |
| noonedeadpunk | well, at least I'd expect that to happen before first startup | 13:08 |
| jrosser | https://paste.opendev.org/show/bIkSSS2ROb0vOZS17mOL/ | 13:08 |
| jrosser | ultimate lxc_container_create.yml is run before lxc_container_config.yml | 13:14 |
| jrosser | ultimately* | 13:14 |
| noonedeadpunk | yeah, true | 13:16 |
| noonedeadpunk | though it's kinda weird... | 13:16 |
| noonedeadpunk | even see it here https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/main.yml#L70-L90 | 13:17 |
| jrosser | i think that until you lxc-start it (or via the ansible module) the things in /var/lib/lxc/<container> dont exist | 13:18 |
| jrosser | so we do start -> configure -> restart | 13:18 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_create_dir.yml#L23 | 13:18 |
| noonedeadpunk | but we also supply a config there | 13:19 |
| noonedeadpunk | which is this by default: https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/templates/lxc-openstack.conf.j2 | 13:20 |
| noonedeadpunk | so adding some profile there might help.... | 13:21 |
| jrosser | yeah just trying | 13:21 |
| jrosser | thanks for the pointer - i was trying to figure out where the startup config came from | 13:22 |
| jrosser | \o/ thats working | 13:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Respect defined interface for external VIP with LE https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/924333 | 13:26 |
| jrosser | ok this now fails later on our lxc-openstack apparmour profile | 13:26 |
| jrosser | i wonder why we need that above the system one | 13:27 |
| noonedeadpunk | we templated there some custom paths... | 13:28 |
| noonedeadpunk | but not sure | 13:28 |
| noonedeadpunk | we could jsut do override of system profile if needed, I guess | 13:28 |
| noonedeadpunk | and also - that "just worked" and was never touched for years now... | 13:28 |
| noonedeadpunk | could be that system one was insufficient somewhere at 16.04.... | 13:29 |
| jrosser | yes, i'm just trying without that now and see what happens | 13:31 |
| jrosser | there was some historical hack for debian to run them unconfined which is not great | 13:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Skip importing haproxy_service_config with no haproxy hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924308 | 15:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Install systemd-resolved into container base image for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924339 | 15:21 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Use generated apparmor profile by default in lxc base config https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/924340 | 15:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Ensure haproxy_service_config targets right host group https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/924341 | 15:33 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 15:47 |
| noonedeadpunk | every time I try to deploy smth more then a sandbox - I hit quite /o\ bugs.... | 15:55 |
| jrosser | i know what you mean | 15:56 |
| jrosser | like every time we do a lab install after a release theres a slew of things to fix | 15:56 |
| noonedeadpunk | makes me think if it's our codebase that bad or it's just our awareness of capabilities keep pushing limits? | 15:56 |
| jrosser | complexity management maybe | 15:57 |
| noonedeadpunk | yeah, might be... | 15:57 |
| jrosser | like all these with / without ssl questions, i cant keep the model of all that in my head | 15:57 |
| jrosser | overall, simplifying would be good where we can | 15:58 |
| jrosser | has that haproxy env.d thing been there forever? | 15:59 |
| jrosser | oh right yes haproxy in containers | 15:59 |
| jrosser | i literally never did that | 15:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Permit Ubuntu Noble for deploy host and targets in requirements checks https://review.opendev.org/c/openstack/openstack-ansible/+/924311 | 16:09 |
| noonedeadpunk | like it feels code is getting better and more stable overall, but seeing so many things kinda sucks... so somehow got low on morale today :( | 16:11 |
| jrosser | i've found it good to keep a list of all the broken things | 16:11 |
| jrosser | then move them to a list of fixed things when i sort them out | 16:11 |
| jrosser | shows progress ratehr than just feeling like * is broken | 16:12 |
| jrosser | because omg the magnum stuff is like patch the whole world just to land small changes | 16:12 |
| jrosser | so i did get a 24.04 vm to get through the container setup | 16:14 |
| jrosser | there is some issue with the name of the netcat package in haproxy | 16:14 |
| jrosser | need to wipe it and start again as i don't trust the apparmor setup any more after hacking on it | 16:14 |
| jrosser | btw the thing i find most bad for morale is the situation with the unmaintained branches | 16:18 |
| jrosser | it does not seem possible to find time to fix the total breakage from the branch renaming, for something that was pretty much working beforehand | 16:19 |
| jrosser | its kind of some form of vandalism :( | 16:19 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Add vars for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/924344 | 16:35 |
| noonedeadpunk | oh, yes, unmainatained is really sucks... | 16:35 |
| noonedeadpunk | And I don't feel that motivation behind change really justifies the change itself | 16:36 |
| noonedeadpunk | but I felt kinda better when Yoga and Zed passed CI lately | 16:37 |
| noonedeadpunk | and frankly - it wasn't _that_ horrible on paper... | 16:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Combine debian and ubuntu vars, adding support for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/924345 | 16:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 16:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename haproxy_hosts to load_balancer_hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924348 | 16:45 |
| noonedeadpunk | dunno how much sense this makes... but see no other good outcome ^ | 16:45 |
| noonedeadpunk | as while https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/924341 fixes the immediate issue, it's really very easy to shoot yourself into the leg with this... | 16:46 |
| jrosser | i thought we had a similar thing with ironic | 16:49 |
| jrosser | where the groups were all mixed up | 16:49 |
| jrosser | or anyway, i'm sure weve had it that the containers/physical hosts were confused somewhere else, and there was an inventory fix for it | 16:50 |
| noonedeadpunk | yeah, and likely haproxy not the only place still | 16:52 |
| noonedeadpunk | and yes, I think it was Ironic indeed | 16:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename haproxy_hosts to load_balancer_hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924348 | 16:54 |
| jrosser | i was just looking for the patch | 16:54 |
| jrosser | i think i also made a subltle mistake in the k8s env.d file initially that had very similar outcome | 16:54 |
| jrosser | yeah so your patch 924348 does what i'd expect, have two names haproxy and loadbalancer, just like we have keystone and identity for example | 16:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename haproxy_hosts to load_balancer_hosts https://review.opendev.org/c/openstack/openstack-ansible/+/924348 | 17:03 |
| noonedeadpunk | yeah, but keeping ability to use haproxy_hosts still | 17:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-memcached_server master: Use the netcat-openbsd package on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/924350 | 17:11 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Combine debian and ubuntu vars, adding support for Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/924345 | 17:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 17:13 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 17:13 |
| jrosser | noonedeadpunk: actually if you make something working with containerised haproxy we could test that in the infra jobs | 17:36 |
| jrosser | becasue right now i think we never ever deploy keepalived in any tests | 17:36 |
| noonedeadpunk | yeah, I'm writing some doc around that right now | 17:37 |
| noonedeadpunk | it's a bit... messy I'd say... | 17:37 |
| noonedeadpunk | But totally doable | 17:38 |
| noonedeadpunk | and assumes at least https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/924341 being present | 17:38 |
| noonedeadpunk | and yeah, actually that's a good idea to have keepalived deployed | 17:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add documentation on spawning HAProxy inside LXC https://review.opendev.org/c/openstack/openstack-ansible/+/924353 | 18:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Use mirror.mariadb.org to install packages from https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924354 | 18:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Remove installation of libaio1 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924355 | 18:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add documentation on spawning HAProxy inside LXC https://review.opendev.org/c/openstack/openstack-ansible/+/924353 | 18:16 |
| noonedeadpunk | here it goes ^ | 18:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 18:17 |
| noonedeadpunk | about mariadb... I think infra is mirroring from downloads ? | 18:17 |
| noonedeadpunk | also - mirror.mariadb.org contains confusingly small amount of releases? | 18:18 |
| jrosser | yes and no | 18:19 |
| jrosser | they are all here https://mirror.mariadb.org/repo/10.11/ubuntu/pool/main/m/mariadb/ | 18:19 |
| noonedeadpunk | ah | 18:20 |
| jrosser | for some reason they also have a top level dir for whatever the latest release it, but that just looks like a bad idea as it will change | 18:20 |
| jrosser | you are right we need to update system-config for the mirror location | 18:21 |
| jrosser | i think it will break currently as we use the mirror but the path will be wrong | 18:21 |
| noonedeadpunk | wait. but we also used to add repo specifically for the minor release? | 18:23 |
| noonedeadpunk | especiually for rhel results will be very different? | 18:24 |
| noonedeadpunk | is "http://{{ galera_repo_host }}/MariaDB/mariadb-{{ galera_major_version }}.{{ galera_minor_version }}" giving specific major/minor, while http://{{ galera_repo_host }}/yum/{{ galera_major_version }}/{{ansible_facts['distribution'] | lower }} will give always latest from major? | 18:25 |
| jrosser | oh because pinning is not a thing there? | 18:25 |
| noonedeadpunk | yeah | 18:25 |
| jrosser | doh | 18:25 |
| noonedeadpunk | well... there might be some plugin for dnf implementing pinning... | 18:25 |
| noonedeadpunk | but I think that was the reason for selecting downloads.mariadb.com before | 18:25 |
| jrosser | could be - though it looks like totally different content now | 18:26 |
| jrosser | ooooh hold on | 18:27 |
| noonedeadpunk | but there's noble as well? https://downloads.mariadb.com/MariaDB/mariadb-11.4.2/repo/ubuntu/ | 18:27 |
| jrosser | here too https://downloads.mariadb.com/MariaDB/mariadb-10.11.8/repo/ubuntu/dists/ | 18:27 |
| jrosser | i'll redo this | 18:27 |
| noonedeadpunk | yeah | 18:27 |
| jrosser | what we actually need to do is update to 10.11.8 | 18:27 |
| noonedeadpunk | yeah, makes sense until 11.4 is fixed | 18:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Remove installation of libaio1 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924355 | 18:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Update to version 10.11.8 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/924357 | 18:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Test on Ubuntu Noble https://review.opendev.org/c/openstack/openstack-ansible/+/924342 | 18:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add documentation on spawning HAProxy inside LXC https://review.opendev.org/c/openstack/openstack-ansible/+/924353 | 18:31 |
| noonedeadpunk | jrosser: ubuntu 24.04 has libaio1t64 | 18:33 |
| noonedeadpunk | but not sure what's needed.... | 18:33 |
| noonedeadpunk | or well, why it's mneeded | 18:33 |
| jrosser | when you say "For instance, running these services on bare metal assumes that a default route for hosts should be set via Public API gateway" - this is actually meaning you need your bare metal hosts to have an address in a subnet with a default route? | 18:33 |
| jrosser | API gateway is kind of odd term that suggests some fancy enterprise product :) | 18:34 |
| noonedeadpunk | Yeah, wording is awful... | 18:34 |
| noonedeadpunk | But what I meant, is that default route should be in haproxy_keepalived_external_vip_cidr? | 18:35 |
| jrosser | yes with libaio1 it worked here on 24.04 vm without that, so i figure this is some legacy thing | 18:35 |
| noonedeadpunk | could be | 18:35 |
| noonedeadpunk | I'd need to draw that I guess.... | 18:35 |
| jrosser | even with haproxy/keepalived in lxc you still need a default route in the container though? | 18:36 |
| noonedeadpunk | yes, sure | 18:36 |
| noonedeadpunk | but it;s less of an issue? | 18:36 |
| noonedeadpunk | it doesn't mess up with way of accessing host, for instance | 18:36 |
| noonedeadpunk | there's no ssh inside container, so on | 18:37 |
| jrosser | oh yes i totally agree | 18:37 |
| jrosser | it makes it really very similar to having dedicated haproxy nodes but without extra hardware | 18:37 |
| noonedeadpunk | but I somehow have a bias, that on bare metal host, especially if it's like regular controller, having default route to public network is quite meh | 18:37 |
| noonedeadpunk | yeah | 18:37 |
| noonedeadpunk | (I even kinda wondered why it wasn't a default) | 18:38 |
| jrosser | me too, we don't have default route anywhere except network nodes and haproxy nodes | 18:38 |
| noonedeadpunk | you're not obliged to have it on net nodes either, do you? | 18:38 |
| jrosser | maybe not, will take another look at that | 18:39 |
| noonedeadpunk | feel free to re-phrase things there | 18:40 |
| jrosser | actually you are right, we don't have default route on the network nodes | 18:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add documentation on spawning HAProxy inside LXC https://review.opendev.org/c/openstack/openstack-ansible/+/924353 | 19:41 |
| jrosser | i am seeing quite a lot of jobs failing to bring up keystone db | 19:58 |
| jrosser | noonedeadpunk: this might be interesting for your haproxy thing - i had a way to provide static IP to containers here https://github.com/jrosser/openstack-ansible-ops/commit/ba8636bc51bb61ee784e2453820173e6ec6ea0ae#diff-56970b5c834ccc0b5640e6b6ab4e6e6aaa513d57ef4128281efffd8376daa403 | 20:09 |
| jrosser | tbh i would do it with host_vars to define the IP specifically for each host instead, but you can totally insert a custom static interface | 20:10 |
| jrosser | you might indeed want to know what the public IP are, as theres quite likley iptables or some hardware firewall involved as well that you need to put config into | 20:12 |
| spotz[m] | Hi my name is spotz and I love the new suggest edit feature in gerrit:) | 21:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!