| jrosser | good morning | 07:43 |
|---|---|---|
| jrosser | manuvakery1: i think that there would be a failed task before the ones that you have pasted | 07:44 |
| jrosser | most likley during building the python wheel for neutron | 07:45 |
| noonedeadpunk | mornings | 08:09 |
| noonedeadpunk | manuvakery1: try to re-run with `-e venv_rebuild=true` | 08:10 |
| noonedeadpunk | damiandabrowski: fwiw I've commented on https://review.opendev.org/c/openstack/openstack-ansible/+/881968 | 08:24 |
| noonedeadpunk | we have also zun to fix... wil try to check on that today | 08:24 |
| noonedeadpunk | I think we also want to land ceph cache before branching | 08:24 |
| jrosser | the ceph cache is pretty experimental | 08:24 |
| jrosser | i need to try to test deploying that from fresh today | 08:25 |
| jrosser | for $reasons it doesnt work in an AIO | 08:25 |
| noonedeadpunk | aha | 08:25 |
| jrosser | but we have a lab with it working :/ | 08:25 |
| noonedeadpunk | maybe because we use loop drives or smth like that | 08:25 |
| jrosser | could be, or glance->cinder snapshots are not right somehow | 08:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Allow ceph clients to be enabled or disabled https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882828 | 08:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add ceph client and configuration for immutable object cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882829 | 08:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add immutable object cache documentation https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882830 | 08:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add config and documentation for ceph perisistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882849 | 08:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add releasenote for ceph immutable object cache and persistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882850 | 08:28 |
| jrosser | i think also it requires RAW images not QCOW2 as theres a conversion which gets in the way of the cache | 08:29 |
| jrosser | either way when everything is aligned we saw a 3-4x improvement for both reads and writes when both caches were enabled | 08:30 |
| noonedeadpunk | well, it's always raw that ends up on ceph. if it's not raw - nova/cinder will convert it to raw regardless iirc | 08:42 |
| noonedeadpunk | but regarding cache - the biggest question is what happens during cache drive failure, as I'd assume it's same if writeback fails when battery on the raid controller is dead | 08:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch ubuntu upgrade jobs to Jammy https://review.opendev.org/c/openstack/openstack-ansible/+/879890 | 08:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Move repo_packages to group_vars https://review.opendev.org/c/openstack/openstack-ansible/+/881824 | 08:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add ceph client key for immutable object cache https://review.opendev.org/c/openstack/openstack-ansible/+/882831 | 08:48 |
| noonedeadpunk | I think what else I'd love to be reviewed is this https://review.opendev.org/c/openstack/openstack-ansible/+/870113 | 08:49 |
| noonedeadpunk | And I'm not sure I fully understood proposal for making things more clear | 08:49 |
| noonedeadpunk | Aha, so basically that `container_ip` is a bad property to name it | 08:51 |
| noonedeadpunk | well, technically, we can name it whatever, but yes, in dynamic_inventory it will map to container_address | 08:51 |
| jrosser | do we need a piece of work to migrate container_address to managment_address? | 08:53 |
| noonedeadpunk | that is very good question | 09:02 |
| noonedeadpunk | as it's kinda same address that's used as eth1 in containers and whatever for metal | 09:03 |
| noonedeadpunk | so somehow upgrade path should be handled I believe, as we're promoting to set `is_container_address: true` for br-mgmt in all docs | 09:04 |
| noonedeadpunk | then we should change that to is_management_address I believe | 09:04 |
| noonedeadpunk | but yes, that might make sense | 09:07 |
| noonedeadpunk | I wonder what to do with container_address after migration - keep that in inventory or drop... | 09:11 |
| noonedeadpunk | Will try to check how to do that now | 09:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename container_address to management_address https://review.opendev.org/c/openstack/openstack-ansible/+/883148 | 10:13 |
| noonedeadpunk | ^ hopefuly it works.... | 10:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename container_address to management_address https://review.opendev.org/c/openstack/openstack-ansible/+/883148 | 10:15 |
| opendevreview | Merged openstack/openstack-ansible stable/zed: Do not define cluster template labels https://review.opendev.org/c/openstack/openstack-ansible/+/883011 | 10:33 |
| jrosser | we should try to get this merged as well https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/881848 | 10:37 |
| noonedeadpunk | well... it looks like it needs some work | 10:40 |
| noonedeadpunk | aha, it's all rabbit thing | 10:41 |
| jrosser | yes also need to differentiate rocky/centos | 10:43 |
| noonedeadpunk | I still wonder if targets.__setitem__ really needed there | 10:44 |
| jrosser | i'll ask stuart to take another look | 10:46 |
| noonedeadpunk | Like we also have `{% set distro_arch = [distro, arch] | join('_') %}` below, so maybe we can make code slightly more alike | 10:47 |
| jrosser | yeah | 10:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Fix venv_build_targets when used with multi-arch https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/881848 | 10:52 |
| noonedeadpunk | I wonder if smth like that will work ^ | 10:52 |
| noonedeadpunk | I don't have any different arch to play with | 10:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Rename container_address to management_address https://review.opendev.org/c/openstack/openstack-ansible/+/883148 | 11:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add ceph client and configuration for immutable object cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882829 | 14:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add immutable object cache documentation https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882830 | 14:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add config and documentation for ceph perisistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882849 | 14:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add releasenote for ceph immutable object cache and persistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882850 | 14:26 |
| jrosser | so i just managed to run that against a fresh compute node, and the next vm i boot on it used the immutable object cache | 14:27 |
| jrosser | the permissions stuff is extremely difficult and not documented at all what the right thing to do is | 14:28 |
| noonedeadpunk | btw, I've realized that ubuntu Jammy upgrade fails for $reason from Zed | 14:29 |
| noonedeadpunk | And the reason is buggy ovs that get's upgraded and mess up with hostname before we apply "fix" | 14:30 |
| noonedeadpunk | also zun is messed up quite a lot to have that said... | 15:16 |
| jrosser | it is | 15:16 |
| noonedeadpunk | Like docker has dropped cluster-store in 23.0.0 | 15:17 |
| jrosser | just verified the scary rbd write cache on a fresh node | 15:17 |
| noonedeadpunk | and zun is relying on that. so it should be docker<23.0.0 | 15:17 |
| jrosser | thats much simpler, just create the cache mount, set owner/group and add config -> done | 15:17 |
| noonedeadpunk | Then, it seems it doesn't like etcdv3 | 15:17 |
| noonedeadpunk | It has zookeeper though, but can't make it work yet... | 15:17 |
| jrosser | zun seems like also in the category of magnum | 15:18 |
| noonedeadpunk | except that upstream project is unmaintained | 15:18 |
| jrosser | maintained by $few orgs meeting their use case | 15:18 |
| noonedeadpunk | I'm also not an expert in docker which makes things worse, in terms I'm not sure how it's interacting with kuryr | 15:20 |
| noonedeadpunk | And then kata on top.... | 15:20 |
| jrosser | andrew spent time on this - we did have it "working" as far as you could | 15:20 |
| noonedeadpunk | and then if attempt to use newer docker, some other swarm mode should be adopted rather then standalone one | 15:20 |
| noonedeadpunk | yeah, it's been a while though | 15:21 |
| jrosser | the big problem is that users can issue commands against the containers just randomly though the api | 15:21 |
| jrosser | and theres no record of any of it so if you lose a zun compute node everything is lost | 15:21 |
| noonedeadpunk | oh | 15:22 |
| noonedeadpunk | that's kinda.... suck a lot | 15:22 |
| jrosser | so depends how users use it, just upload images which can be remade easily is one thing, but kind of in place customisation was really not cool | 15:22 |
| noonedeadpunk | yeah, so there's no evacuate stuff or re-spawn the container elsewhere? | 15:22 |
| jrosser | nothing at all | 15:23 |
| noonedeadpunk | but yes, I think idea it must spawn from images plus some manila I assume | 15:23 |
| noonedeadpunk | or trove | 15:23 |
| noonedeadpunk | but I mainly try to fix gates right now :D | 15:24 |
| noonedeadpunk | no plans to get it adopted as of today | 15:24 |
| jrosser | it is probably better to end up with magnum tbh | 15:24 |
| jrosser | but yes gates are indeed broken | 15:25 |
| noonedeadpunk | well, magnum is different and still sucks kinda | 15:25 |
| noonedeadpunk | in terms of supported k8s versions, so you need to upgrade it constantly to get supported k8s | 15:25 |
| NeilHanlon | kubernetes ... sucks | 15:31 |
| NeilHanlon | o/ morning y'all (afternoon, i guess) | 15:31 |
| jrosser | o/ hello | 15:36 |
| noonedeadpunk | o/ | 15:38 |
| jrosser | when do we need to release? | 16:04 |
| noonedeadpunk | May 29 | 16:12 |
| noonedeadpunk | oh, huh, I wonder if it's ovn that makes things weird with zun | 16:18 |
| opendevreview | Merged openstack/openstack-ansible master: Update 'TLS for Haproxy Backends' section in docs https://review.opendev.org/c/openstack/openstack-ansible/+/883090 | 16:25 |
| noonedeadpunk | at least kuryr looks weird | 16:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace etcd usage with zookeeper for zun https://review.opendev.org/c/openstack/openstack-ansible/+/883194 | 16:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 16:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 16:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Use integrated linters job for ops repo https://review.opendev.org/c/openstack/openstack-ansible-ops/+/883197 | 17:08 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Add 'tls' scenario https://review.opendev.org/c/openstack/openstack-ansible/+/881968 | 17:41 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Add 'tls' scenario https://review.opendev.org/c/openstack/openstack-ansible/+/881968 | 21:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!