| elodilles | hi Oslo team, just a heads up that oslo.middleware release might have a valid cross job error: https://review.opendev.org/c/openstack/requirements/+/977145 | 11:23 |
|---|---|---|
| frickler | 2026-02-17 17:38:07.940492 | ubuntu-noble | 2026-02-17 17:38:07,125 WARNING [oslo_middleware.cors] Allowed origin [http://valid.example.com] already exists, skipping | 11:49 |
| frickler | this is generated right next to https://review.opendev.org/c/openstack/oslo.middleware/+/970631/1/oslo_middleware/cors.py which makes it very suspicous. see also the discussion between stephenfin and tkajinam there | 11:49 |
| frickler | hmm ... looks like this is actually nova's fault and it should return a list here? https://opendev.org/openstack/nova/src/branch/master/nova/tests/functional/test_middleware.py#L39 but that would also be an indication that adding the error would have been a good idea after all | 11:55 |
| opendevreview | Takashi Kajinami proposed openstack/oslo.rootwrap master: Refactor test_KillFilter https://review.opendev.org/c/openstack/oslo.rootwrap/+/977116 | 12:38 |
| opendevreview | Takashi Kajinami proposed openstack/oslo.rootwrap master: Find cat command externally https://review.opendev.org/c/openstack/oslo.rootwrap/+/977130 | 12:39 |
| tkajinam | elodilles, frickler checking | 12:59 |
| tkajinam | let me propose a fix but this is actually caused by internal hack to override the option. | 13:02 |
| elodilles | thanks tkajinam for working on this, too! o/ | 13:03 |
| tkajinam | frickler, maybe, though as long as the appropriate config load mechanism is used the value should be converted to a list | 13:11 |
| tkajinam | the hack there directly injects the final value there which has been outdated. I guess the warning has been shown for some time (though I don't know why I didn't notice that earlier when I did mass deprecation cleanup stuff | 13:12 |
| tkajinam | damani[m], do you have a minute to review https://review.opendev.org/c/openstack/oslo.utils/+/975115 ? | 13:15 |
| tkajinam | elodilles, https://review.opendev.org/c/openstack/nova/+/977217 will be merged soon and may unblock the bump. just fyi | 13:47 |
| elodilles | tkajinam: awesome \o/ thanks! | 13:52 |
| ralonsoh | gmaan, hberaud[m] haleyb: https://bugs.launchpad.net/neutron/+bug/2141304 is still broken | 14:34 |
| ralonsoh | and now we have a release with this issue | 14:34 |
| ralonsoh | https://bugs.launchpad.net/neutron/+bug/2142122 | 14:35 |
| gmaan | ralonsoh: I tested the neutron neutron-ovs-tempest-with-oslo-master and it passed on the fix | 15:33 |
| ralonsoh | gmaan, yes, its passing but one of the service is not | 15:35 |
| ralonsoh | At least one of the workers, I'm still investigating that | 15:35 |
| ralonsoh | in any case, we still have this nasty warning message | 15:36 |
| ralonsoh | https://paste.opendev.org/show/b63UDQh8c5dHcDI5OFxJ/ | 15:36 |
| ralonsoh | is that necessary? | 15:37 |
| gmaan | ralonsoh: do you have review link also where it is failing or job link to see config etc | 15:44 |
| ralonsoh | yes, one sec. It is happening in ovn agent and nova scheduler | 15:44 |
| ralonsoh | https://zuul.opendev.org/t/openstack/build/2f8feee00a3e4bf286e879ee03274d77/log/controller/logs/screen-q-ovn-agent.txt#305 | 15:45 |
| ralonsoh | https://zuul.opendev.org/t/openstack/build/2f8feee00a3e4bf286e879ee03274d77/log/controller/logs/screen-n-sch.txt#25 | 15:45 |
| gmaan | ralonsoh: ohk. different jobs | 15:48 |
| ralonsoh | gmaan, I think we need to revert the requirements patch to bump oslo.service for now | 15:53 |
| gmaan | ralonsoh: about warning, yes by default oslo.service try for spawn method and raise warning if service is not spawnable | 15:57 |
| gmaan | ralonsoh: let me check the bug and if I can fix | 15:58 |
| opendevreview | Merged openstack/tooz master: Deprecate etcd driver https://review.opendev.org/c/openstack/tooz/+/976838 | 16:00 |
| opendevreview | Merged openstack/tooz master: Deprecate Consul driver https://review.opendev.org/c/openstack/tooz/+/947733 | 16:00 |
| gmaan | ralonsoh: they are just warning and logging exception right? they are not stopping worker to start | 16:05 |
| ralonsoh | gmaan, yes, just logging | 16:07 |
| ralonsoh | sorry, I need to leave now, I'll connect later today | 16:07 |
| gmaan | ralonsoh: ohk, i got confused with 'is still broken' | 16:07 |
| ralonsoh | gmaan, well, the periodics job is | 16:08 |
| ralonsoh | it seems to be related to a specific periodic worker | 16:08 |
| gmaan | ralonsoh: k, can you link me the one failing | 16:08 |
| ralonsoh | I'm still investigating | 16:08 |
| ralonsoh | https://bugs.launchpad.net/oslo.service/+bug/2142122 | 16:08 |
| gmaan | thanks | 16:08 |
| gmaan | which job is this exactly, job name | 16:09 |
| gmaan | it seems it is telling to fallback to fork but cotyledon trying with spawn and fail | 16:13 |
| haleyb | i'll try and find the job he was looking at, but from what i can tell neutron-periodic-workers is only true in our grenade jobs | 16:16 |
| haleyb | gmaan: for example, if you look at this simple change https://review.opendev.org/c/openstack/neutron/+/976822 and go into the logs for the neutron-ovn-grenade-multinode job that log file is there with the pickle failures | 16:19 |
| gmaan | haleyb: ack, thanks | 16:21 |
| gmaan | haleyb: yeah, i think that is only log, i checked but I was asking the one failing | 16:22 |
| haleyb | i don't know what change that was from | 16:24 |
| gmaan | so if I understood. neutron gate is ok with oslo.service 4.5.0 but one periodic job failing | 16:25 |
| haleyb | it's a periodic worker in the job | 16:28 |
| haleyb | https://review.opendev.org/c/openstack/neutron/+/922110 | 16:29 |
| gmaan | yeah, so that works fine as it fall back to 'fork' and only log warnings with traceback. but does not fail right? or periodic worker does not cause job to fail and they does not start silently | 16:30 |
| haleyb | it's no longer eventlet | 16:30 |
| gmaan | I can disable the warning traceback as that is confusing in logs | 16:30 |
| haleyb | from the log i think it fails to run, so i think things like quotas could be impacted | 16:32 |
| gmaan | i see | 16:33 |
| haleyb | we'd have to find a previous log to see i guess | 16:35 |
| haleyb | from that same change, but on a previous PS with the older oslo.service this is the log, https://998fb44128151542469d-eb0bc8c584c4b7f1435696f4fd53c9a4.ssl.cf2.rackcdn.com/openstack/083c92e432844601ac012d64da34283e/controller/logs/screen-neutron-periodic-workers.txt | 16:37 |
| opendevreview | Ghanshyam Maan proposed openstack/oslo.service master: Improve warning and more logging https://review.opendev.org/c/openstack/oslo.service/+/977241 | 16:46 |
| gmaan | haleyb: ralonsoh : disabling that log traceback and adding more logs to see what's going on for neutron case , testing neutron with this https://review.opendev.org/c/openstack/neutron/+/977242 | 16:53 |
| haleyb | gmaan: ack, thanks! | 16:53 |
| opendevreview | Ghanshyam Maan proposed openstack/oslo.service master: Improve warning and more logging https://review.opendev.org/c/openstack/oslo.service/+/977241 | 17:20 |
| sean-k-mooney | has there been any work to remove eventlet form teh default deps in oslo service and oslo messaging and only install it via extras? | 18:19 |
| sean-k-mooney | im asking becasue it woudl be nice to be able to not pull them in for services that are not using them | 18:20 |
| gmaan | ralonsoh: if you have your DNM testing change for debugging, please use this so that it will give less warning and more logs about multiprocessing context. https://review.opendev.org/c/openstack/oslo.service/+/977241 | 18:31 |
| gmaan | I am going to debug the same but do not know where exactly things failed because neutron grenade multinode job did not fail. | 18:31 |
| sean-k-mooney | oslo log by defualt is callent eventlutils form oslo utiels https://github.com/openstack/oslo.log/blob/d0f0bf3ee5d5676b99b4ad7656089808c2aeb35b/oslo_log/log.py#L303C5-L341 that does a tryimport of eventlet https://github.com/openstack/oslo.utils/blob/master/oslo_utils/eventletutils.py#L42 which indirectly import greeenlet adn apprelntly that can cause segfault on python 3.13 | 18:42 |
| sean-k-mooney | in some cases at least in unit tests | 18:42 |
| sean-k-mooney | hum this might eb an issue with sqligt | 19:07 |
| sean-k-mooney | its unfortunate that oslo log is uncondtionally causes oslo.utils to try and import eventlet and greenlet even if its not monkey patching | 19:07 |
| gmaan | haleyb: ralonsoh: with my debugging change, I can see oslo.service selected the 'spawn' method because it did not find any pickle issue in service worker but later in multiprocessing it fail for the pickle issue - https://zuul.opendev.org/t/openstack/build/2b9a15586d6745dfb70b718b26b75e3f/log/controller/logs/screen-neutron-periodic-workers.txt#4027 | 21:12 |
| gmaan | haleyb: one question, I could not find this error in latest run of neutron-ovn-grenade-multinode in https://review.opendev.org/c/openstack/neutron/+/976822 where i can find the error in its previous run | 21:13 |
| gmaan | is it not 100% failing or it depends on the other runtime things | 21:14 |
| haleyb | gmaan: it might not be 100%, i don't see it in that change but others seem to have it | 21:52 |
| gmaan | haleyb: yeah, I think I am finding the issue here, maybe it is with the conf object pickle which is done by multiprocessing but not by oslo.service | 21:52 |
| gmaan | that is the difference I can see how multiprocessing check picklable and fail which oslo.serviec does not | 21:53 |
| gmaan | this one, it has the conf object into process_obj and it fail when it test the picklable https://github.com/python/cpython/blob/0bbdb4e897ae909af39cd93f7a25de89b724ab47/Lib/multiprocessing/popen_spawn_posix.py#L47 | 21:55 |
| gmaan | fail here https://github.com/python/cpython/blob/0bbdb4e897ae909af39cd93f7a25de89b724ab47/Lib/multiprocessing/reduction.py#L60 | 21:55 |
| gmaan | need to go for lunch, after that I will push the fix and recheck on neutron testing change | 21:56 |
| haleyb | sure, thanks, maybe rodolfo will be back by then | 21:57 |
| opendevreview | Ghanshyam Maan proposed openstack/oslo.service master: Threading mode: check conf object picklable before 'spawn' https://review.opendev.org/c/openstack/oslo.service/+/977241 | 23:00 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!