| jrosser | good morning | 09:50 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add immutable object cache documentation https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882830 | 09:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add config and documentation for ceph perisistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882849 | 09:53 |
| noonedeadpunk | mornings | 09:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add releasenote for ceph immutable object cache and persistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882850 | 09:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ceph_client master: Add releasenote for ceph immutable object cache and persistent write log cache https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882850 | 09:54 |
| NeilHanlon | morning all.. apologies for the rocky-related failures in gate/checks.. | 13:41 |
| NeilHanlon | https://zuul.opendev.org/t/openstack/build/94fc5806095c4db09d1cb585c364ede2 wondering if this is something truly transient or if something else is going on? | 13:41 |
| noonedeadpunk | not sure to be frank... that looks like not transient one and more related to `git` version being updated | 13:43 |
| noonedeadpunk | `git clone --filter=blob:none --quiet file:///openstack/src/opendev.org/openstack/placement /tmp/pip-wheel-0iz68v19/openstack-placement_110c0b7376414628a23c31da910a20fd did not run successfully.` | 13:43 |
| noonedeadpunk | though it's weird that only upgrade job is affected... | 13:45 |
| NeilHanlon | yeah that's what I was thinking too.. odd that it's just the upgrade | 13:48 |
| jrosser | thats odd as the git repo is local as well | 13:51 |
| derekokeeffe85 | Hi all, anyone any advice as to why I might be getting this in my neutron container? Thanks in advance https://paste.openstack.org/show/bwkxHWhfEoQfkpz6AHbl/ | 13:52 |
| jrosser | derekokeeffe85: it is saying in neutron.conf (or one of the other neutron config files) there is a `ovn` key defined in the [DEFAULT] section which should not be there | 13:55 |
| derekokeeffe85 | That's what I assumed jrosser but couldn't find it in any of the plugin files in /etc/neutron/plugins/ml2 | 13:56 |
| noonedeadpunk | well, I can recall issue with local git when they covered some vulnarability and made to add dir to trusted when it's owned by another user that trying to access it. | 13:58 |
| jrosser | yes i remember that | 13:59 |
| jrosser | https://github.com/openstack/openstack-ansible/blob/master/zuul.d/playbooks/pre-osa-aio.yml#L23-L29 | 14:00 |
| noonedeadpunk | btw errors look different between run or it's me? | 14:00 |
| noonedeadpunk | As for previous run I see this | 14:00 |
| noonedeadpunk | Running command git clone --filter=blob:none --quiet file:///openstack/src/opendev.org/openstack/placement /tmp/pip-wheel-yry9ytai/openstack-placement_6be38f41493243b8b1a4d129a400bc77\n fatal: '/openstack/src/opendev.org/openstack/placement' does not appear to be a git repository\ | 14:00 |
| noonedeadpunk | ah, no they're same | 14:01 |
| noonedeadpunk | BUT, I indeed don't see repo from being prepped by zuul | 14:03 |
| noonedeadpunk | And, um... We should not install placement for infra jobs.... | 14:04 |
| noonedeadpunk | like at the first place | 14:04 |
| jrosser | this is true | 14:04 |
| derekokeeffe85 | jrosser, I found this service_plugins = ovn-router in the neutron.conf on the container, commenting it out gets rid of the errors I was seeing but introduces this error: 2023-05-17 14:04:34.838 56 ERROR neutron.api.extensions [None req-d6141211-b023-40f4-919a-ab59fb9deb7a - - - - - -] Unable to process extensions (auto-allocated-topology) because the configured plugins do not satisfy their requirements. Some features will not work as | 14:05 |
| derekokeeffe85 | expected. | 14:05 |
| jrosser | which container? neutron-server? | 14:05 |
| noonedeadpunk | `export SCENARIO=aio_lxc` | 14:05 |
| derekokeeffe85 | Yeah in neutron.conf on the neutron container | 14:06 |
| jrosser | noonedeadpunk: `upgrade_yoga-infra_lxc` | 14:06 |
| jrosser | ^ wrong? | 14:06 |
| jrosser | upgrade_yoga_infra_lxc perhaps | 14:07 |
| noonedeadpunk | nah, it's correct... | 14:07 |
| noonedeadpunk | as upgradae_yoga is ACTION | 14:07 |
| jrosser | ahha | 14:07 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/7245140da1eb46719888e74a715d8f31/log/job-output.txt#3193-3198 | 14:08 |
| derekokeeffe85 | jrosser, seems to have done the trick. I can login now and have a working horizon | 14:08 |
| noonedeadpunk | but somehow it's not parsed correctly somewhere | 14:08 |
| jrosser | derekokeeffe85: i have `service_plugins = ovn-router` in my all-in-one neutron container and i think that is OK | 14:09 |
| noonedeadpunk | aha, I guess I see now where things go wrong | 14:13 |
| derekokeeffe85 | Dunno so jrosser, I'll be testing tomorrow so hopefully I won't run into any issues creating networks, images, flavours etc... Will pop back if I do. Thank you | 14:16 |
| jrosser | derekokeeffe85: you are on a slightly old minor version too - i have no idea if that will make any difference | 14:17 |
| derekokeeffe85 | Ah ok, might start from scratch again tomorrow, will see how I go. Thanks | 14:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Populate scenario properly in CI for N-2 upgrade job https://review.opendev.org/c/openstack/openstack-ansible/+/883377 | 14:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Populate scenario properly in CI for N-2 upgrade job https://review.opendev.org/c/openstack/openstack-ansible/+/883377 | 14:22 |
| noonedeadpunk | hopefully this helps ^ | 14:23 |
| jrosser | regex fun | 14:23 |
| noonedeadpunk | looks like it works, scenario is set now to aio_metal_integrated | 14:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Revert pinning rdo-deps for openvswitch 2.17 for rocky https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/883294 | 14:46 |
| noonedeadpunk | huh, looks like I've managed to fix zun.... | 14:52 |
| noonedeadpunk | except I had to use zookeeper instead of etcd.... | 14:58 |
| noonedeadpunk | as etcdv3 seems to be not supported | 14:58 |
| noonedeadpunk | (by docker itself) | 14:59 |
| noonedeadpunk | nah.... containers are not started.... | 15:36 |
| noonedeadpunk | they're created though.... | 15:37 |
| noonedeadpunk | but then zun-compute timeouting :( | 15:38 |
| noonedeadpunk | smth related to kuryr... | 15:39 |
| noonedeadpunk | aha, and that's related to permissions to run ovs-vsctl commands | 15:50 |
| noonedeadpunk | and looks like it should support ovn after all.... | 15:53 |
| noonedeadpunk | so it fails with ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied which is fair.... | 16:11 |
| noonedeadpunk | I'd assume it should call ovs with root... or smh | 16:11 |
| jrosser | this is like the ceph cache stuff all over again | 16:11 |
| jrosser | same sort of thing | 16:12 |
| jrosser | looks like everything there is root:root | 16:12 |
| jrosser | which seems maybe odd | 16:13 |
| noonedeadpunk | well, docks say `CapabilityBoundingSet = CAP_NET_ADMIN` | 16:13 |
| noonedeadpunk | but not sure that work with uwsgi.... | 16:13 |
| jrosser | isnt that more about system calls? | 16:13 |
| noonedeadpunk | yeah.... | 16:14 |
| noonedeadpunk | I'm jsut saying what doc says.. https://docs.openstack.org/kuryr-libnetwork/latest/install/compute-install-ubuntu.html | 16:15 |
| noonedeadpunk | that's probably why it worked with lxb.... | 16:15 |
| jrosser | i know really little about OVS but lots of this runs as root | 16:17 |
| noonedeadpunk | yup, it is. Though wonder how neutron workarounds that... | 16:18 |
| noonedeadpunk | ovn-controller runs as root though as well... | 16:19 |
| jrosser | neutron has rootwrap configfor ovn-ctl | 16:19 |
| noonedeadpunk | it doesn't for ovs-vsctl though? | 16:25 |
| noonedeadpunk | oh... https://bugs.launchpad.net/kuryr/+bug/1852105 | 16:37 |
| noonedeadpunk | that's totally solving only lxb usecase I believe | 16:38 |
| jrosser | not sure if this is helpful or not https://docs.openstack.org/kuryr-kubernetes/latest/installation/devstack/ovn_support.html | 16:45 |
| jrosser | i feel like that is somethnig different | 16:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 16:53 |
| noonedeadpunk | I think solution except of lxb is same - run as root.... | 16:53 |
| noonedeadpunk | but then uwsgi is unhappy | 16:54 |
| noonedeadpunk | also I really wonder what can be done to remain etcd as kv for docker, as replacing it with zookeeper is totally a breaking change | 17:00 |
| noonedeadpunk | though it would be soooo much easier for us to maintain only 1 storage... | 17:00 |
| jrosser | i wonder what kolla do | 17:02 |
| noonedeadpunk | run kuryr as root? | 17:05 |
| noonedeadpunk | dunno | 17:06 |
| NeilHanlon | cool if I push forward https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/883294 ? | 17:18 |
| jrosser | that cant merge until the depends-on does | 17:20 |
| NeilHanlon | oh, right | 17:21 |
| NeilHanlon | thanks. my brain's all over the place today | 17:22 |
| noonedeadpunk | I think would be cool to land https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/880484 | 17:26 |
| jrosser | looks like we dont use the infra mirror repo for debootstrap in lxc_hosts | 17:32 |
| NeilHanlon | 880484 looks good to me noonedeadpunk | 17:33 |
| noonedeadpunk | that's weird though, as we should be doing that regardless of the job.... | 17:36 |
| noonedeadpunk | though, I think we don't use infra mirrors for rocky | 17:37 |
| noonedeadpunk | as I'm not sure there's any existing | 17:37 |
| NeilHanlon | i'm gonna find out from infra core what needs to be done to facilitate that | 17:47 |
| NeilHanlon | seems like it'd be a good thing now | 17:47 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: Rename floating IP option for coe_cluster_template https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/880484 | 19:32 |
| BobZannapolis | sorry, another newbie question. Using AWS, we've gotten the aios/quick starts to run to completion and get to the dashboard;most of the times we can actually create vms but we haven't been able to ssh into them - what piece / doc are we missing that'll get us past that hurdle ? tia | 19:40 |
| BobZannapolis | Is it a user error in how we (1) created the vms ?, (2) created the OS networks, (3) a known limitation of running the aio in AWS, (4) other ? tia | 19:43 |
| jrosser | BobZannapolis: ssh from where? from the AIO host? | 19:46 |
| jrosser | if you’ve successfully run tempest then that should have created vm and logged in with ssh as part of the validation | 19:49 |
| noonedeadpunk | 1 more vote needed on https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/882826/ to proceed futher with the topic | 20:15 |
| noonedeadpunk | NeilHanlon: regarding mirrors there're 2 things I guess - first is trivial - reliable rsync mirror that can be used, and then what I'm not sure about is diskspace on the AFS | 20:26 |
| noonedeadpunk | fun... now just docker does crush completely once zun sends request to it | 20:33 |
| noonedeadpunk | dockerd[130187]: panic: runtime error: invalid memory address or nil pointer dereference | 20:34 |
| noonedeadpunk | /o\ | 20:34 |
| jrosser | what version of docker? | 20:35 |
| noonedeadpunk | 20.10.24 | 20:35 |
| noonedeadpunk | >=23.0.0 is broken by design with zun as standalone swarm mode is dropped there | 20:36 |
| noonedeadpunk | Well, stack trace boils down to libnetwork store, which I assume what zookeeper provides.... | 20:36 |
| NeilHanlon | noonedeadpunk: yeah, sounds like it. I can act as that Tier 2 mirror for them, so I'm not worried about that. Myself and some friends run a little mirror network :P | 20:37 |
| NeilHanlon | i'll put together a little proposal or so and send it to the list.. probably next week | 20:37 |
| NeilHanlon | though at least this current issue with ovs is actually not even a rocky mirror problem, it's the centos mirror | 20:38 |
| noonedeadpunk | I think I should try to make etcd working then... And I'd assume that would involve reverting back in versions where not only v3 was working... | 20:39 |
| noonedeadpunk | so.... etcd < v3.4 works.... | 21:25 |
| noonedeadpunk | pfffff | 21:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 21:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump down etcd version for zun https://review.opendev.org/c/openstack/openstack-ansible/+/883194 | 21:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump down etcd version for zun https://review.opendev.org/c/openstack/openstack-ansible/+/883194 | 21:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump down etcd version for zun https://review.opendev.org/c/openstack/openstack-ansible/+/883194 | 21:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Install kata containers from source https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/880288 | 21:40 |
| NeilHanlon | ugh https://review.opendev.org/c/openstack/openstack-ansible/+/883377 is failed again... https://zuul.opendev.org/t/openstack/status/change/883377,2 | 21:46 |
| noonedeadpunk | now on ceph, doh | 21:51 |
| noonedeadpunk | but I'd say there's smth off with infra... | 21:51 |
| noonedeadpunk | As I see jobs having multiple attempts, which shouldn't happen usually | 21:52 |
| NeilHanlon | yea... | 21:53 |
| NeilHanlon | i noticed that too | 21:53 |
| NeilHanlon | ah.. looks like infra restarted the executors, which restarted jobs | 21:53 |
| noonedeadpunk | damn, smth still off with zun. But now kuryr and docker behave properly according to their state - like docker can create networks in neutron through kuryr | 21:57 |
| noonedeadpunk | and likely vice versa, but not sure.... | 21:57 |
| noonedeadpunk | but zun reports getting 500 on attempt to attach network to the container.... | 21:57 |
| opendevreview | Merged openstack/openstack-ansible master: Remove redundant `repo_server_enable_sync_manager` variable https://review.opendev.org/c/openstack/openstack-ansible/+/882862 | 23:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!