| jrosser | o/ morning | 07:30 |
|---|---|---|
| noonedeadpunk | o/ | 07:41 |
| jrosser | that is a new error https://zuul.opendev.org/t/openstack/build/ff74a98242b34312a5ed091bf7eef90d/log/job-output.txt#11317 | 07:54 |
| frickler | jrosser: new tempest release was made on monday, likely related. will need some more digging though | 08:05 |
| jrosser | its interesting as it seems to be self infliced | 08:06 |
| jrosser | *inflicted | 08:06 |
| jrosser | its jut running `tempest run -l` i think | 08:06 |
| frickler | jrosser: there must be more to it, a quick test with tempest from master or 29.1.0 doesn't fail. might be either related to the set of plugins installed or some other environment condition. is this failing 100% so that holding a node for debugging could help? | 08:38 |
| jrosser | i've just rechecked it to see if it is repeatable | 08:38 |
| jrosser | of note it's on a noble job which we only just added, so there could easily be something unexpected happening there | 08:39 |
| frickler | ah, ok, let me add a hold to this then, just in case | 08:39 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Disable uWSGI usage by default https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/927881 | 12:49 |
| NeilHanlon | o/ morning folks. i started discussing rocky mirrors for opendev with the infra folks yesterday (again). I will be going through CI failures across zuul for rocky to see if I can come up with some numbers regarding how many mirror-related CI failures we see, on average | 13:08 |
| noonedeadpunk | oh, we saw quite some in recent 2 weeks I'd say | 13:09 |
| NeilHanlon | yeah, there's not really opposition to it, and there's space--just want to put some numbers behind it to justify the space request | 13:27 |
| jrosser | NeilHanlon: i think that most of the error that i have seen where "package blah-<version> cannot be installed because $reasons" | 13:29 |
| jrosser | rather than it really being a connectivity issue | 13:29 |
| NeilHanlon | yeah, i still think those are sort-of mirror issues, as that indicates a mirror being used is inconsistent or out of date | 13:30 |
| jrosser | so two things need to be considered differently, having a local mirror to avoid reaching out over the internet vs. having a mirror to help with consistency | 13:30 |
| jrosser | i think the purpose of the mirror is the first of those? | 13:30 |
| NeilHanlon | both | 13:30 |
| jrosser | i was not sure that the mirroring process guarantees that you will get a usable mirror at the end | 13:31 |
| NeilHanlon | the short version is that package deps can be spread across multiple repos, and dnf doesn't know to keep AppStream and BaseOS from the same mirror.. so you can end up in a situation where the AppStream mirror and BaseOS mirror are on different mirror systems | 13:31 |
| NeilHanlon | which may or may not have the same sync state | 13:32 |
| jrosser | ah right - so having a local mirror forces that all to be the same place? | 13:32 |
| NeilHanlon | yeah, a local mirror that our CI jobs can point to would mean all those are coming from the same place | 13:33 |
| NeilHanlon | https://drop1.neilhanlon.me/irc/uploads/549451f32823cf6a/image.png | 13:33 |
| NeilHanlon | this is a propagation chart for 9.4 BaseOS -- it's pretty consistent but there are delays sometimes | 13:33 |
| jrosser | btw did you see that i put a link to opensearch https://opensearch.logs.openstack.org/_dashboards which might help you dig out failures | 13:37 |
| NeilHanlon | jrosser: i did, thank you! | 13:48 |
| NeilHanlon | jrosser: any tips on login for that? | 13:51 |
| jrosser | oh openstack/openstack | 13:51 |
| NeilHanlon | ta | 13:52 |
| jrosser | i was unable to find this documented at all and had to dig though the mailing list | 13:52 |
| jrosser | noonedeadpunk: those job timeouts look worrying - as they occur exactly on the tasks added by the patch | 14:07 |
| jrosser | like here https://zuul.opendev.org/t/openstack/build/3286cc494ad5482cb112ad6de7c5b1c8/log/job-output.txt#10619 | 14:08 |
| jrosser | it could be that the next task (update apt cache) is not completing | 14:09 |
| noonedeadpunk | oh... I just spotted that they're a bit "random" | 14:09 |
| noonedeadpunk | meaning that some passed just nicely | 14:09 |
| jrosser | thats true | 14:10 |
| jrosser | everything was ok for check jobs | 14:10 |
| noonedeadpunk | and for rabbitmq we don't have mirrors, so I assumed it could be that | 14:10 |
| noonedeadpunk | why it's timeouting though.... | 14:11 |
| noonedeadpunk | but probably worth to recheck anyway to see | 14:11 |
| jrosser | it could easily be that the repo was broken for some moment | 14:18 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Return `amphora` provider back https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/928815 | 15:37 |
| spatel | Hello!! | 15:39 |
| noonedeadpunk | o/ | 15:46 |
| noonedeadpunk | long time no see :) | 15:46 |
| spatel | I was super busy :( | 16:05 |
| spatel | now I am back | 16:05 |
| noonedeadpunk | yeah, workload these days is just /o\ | 16:05 |
| spatel | lol | 16:11 |
| spatel | I was working on DC migration and it just finished | 16:11 |
| spatel | Next project is to deploy large cloud using OVN (little nervous) | 16:11 |
| noonedeadpunk | well, we've already found couple of nits in ovn in our POC | 16:21 |
| noonedeadpunk | as some things that used to work in OVS do not work for us in OVN | 16:21 |
| noonedeadpunk | but it still looks quite promising :) | 16:22 |
| jrosser | doh another timeout https://zuul.opendev.org/t/openstack/build/f2c7abed4efe482fb8bbb403dce6cdb8/log/job-output.txt#6190 | 16:26 |
| spatel | noonedeadpunk what do you means OVS do not work for us in OVN? | 17:55 |
| spatel | what are those nits? | 17:57 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/2023.2: Ensure python3-packaging is installed for distros https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/928119 | 18:02 |
| bjoernt | any plans to move the eom tags periodically or are the unmaintained branches to be used at that point ? | 20:24 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
