| opendevreview | Merged openstack/tempest master: Add response schemas for list_instance_actions 2.58 https://review.opendev.org/c/openstack/tempest/+/983368 | 05:20 |
|---|---|---|
| noonedeadpunk | Hey folks! I am seeing quite big output in devstack recently about eventlet, and it makes me think, this could be reason to some of tempest test failure, ie 'https://03aa7cceffa1a13f3cf8-a6bd106a0a856fe942ecaed6f7524924.ssl.cf5.rackcdn.com/openstack/6be3b59555d54bb6a4f418207214bcf2/testr_results.html | 08:38 |
| noonedeadpunk | as it feel like warning just get into the reference or smth | 08:39 |
| noonedeadpunk | but Freezer is not using eventlet from what I know | 08:39 |
| noonedeadpunk | https://codesearch.openstack.org/?q=eventlet&i=nope&literal=nope&files=&excludeFiles=&repos=openstack/freezer,openstack/freezer-api,openstack/freezer-specs,openstack/freezer-tempest-plugin,openstack/freezer-web-ui | 08:39 |
| noonedeadpunk | So I am not really sure where it's coming from | 08:39 |
| noonedeadpunk | but it for sure either just makes actual error unreadable, or interfere with expected/actual result by polluting stdout | 08:40 |
| noonedeadpunk | are you aware of smth like this? | 08:41 |
| frickler | noonedeadpunk: yes, I noticed the warning yesterday and stephenfin already produced a fix https://review.opendev.org/c/openstack/oslo.concurrency/+/983696 , we just need a fresh release with that. not sure if the test failure is actually related, though | 08:43 |
| noonedeadpunk | well, it goes `reference = 'Job EventletDeprecationWarning: created'` | 08:43 |
| noonedeadpunk | and in code it tries to take stdout from command | 08:44 |
| noonedeadpunk | https://opendev.org/openstack/freezer-tempest-plugin/src/branch/master/freezer_tempest_plugin/tests/freezer/scenario/test_backups.py#L80-L81 | 08:44 |
| noonedeadpunk | and seems that stdout is just polluted | 08:44 |
| noonedeadpunk | there's probably smth to be done with the tempest plugin itself to be more "robust" though... | 08:47 |
| frickler | that reference looks like you are expecting some error output which will no longer be there when eventlet is removed? | 08:48 |
| frickler | and yes, dealing with stdout from cli commands is always bound to be a bit fragile | 08:48 |
| noonedeadpunk | I think it should be stdout, not stderr... But yes, it feels like this eventlet message gets captured as part of the command output, which messes up expected vs actual | 08:53 |
| noonedeadpunk | or, it's completel;y unrelated, and just tempest logs are polluted to a point that it;s hard to understand what is going on | 08:53 |
| noonedeadpunk | but I can't reproduce the issue localy, so that's why I decided to ask around :) | 08:56 |
| frickler | the trigger was to have osprofiler installed in the venv, which pulls in oslo.concurrency then. devstack installs this into the shared venv | 09:03 |
| noonedeadpunk | Yeah, well, API logs shows that this warning is actually what messes things up... https://zuul.opendev.org/t/openstack/build/fe57f247ea98457f93ebb7c978874f6e/log/controller/logs/screen-freezer-api.txt#106 | 09:55 |
| noonedeadpunk | GET /backup/v2/e46079bc05aa4a57ac8c263f2530f34b/actions/EventletDeprecationWarning: | 09:55 |
| noonedeadpunk | but I think there's also an issue with client, as it writes to stderr for some reason | 09:58 |
| opendevreview | Masayoshi Mizuma proposed openstack/devstack master: Ensure wget is installed before use in _install_rdo https://review.opendev.org/c/openstack/devstack/+/983861 | 12:51 |
| dansmith | anyone else seeing 500 errors from opendev.org repos? | 14:02 |
| dansmith | clarkb: fungi ^ | 14:02 |
| fungi | dansmith: yes, everyone is | 14:02 |
| dansmith | I figured since it was a 500, but hadn't seen any discussion, soryr | 14:03 |
| dansmith | I mean, I thought the gitea problems were resolved earlier in the week | 14:03 |
| dansmith | so I figured this was new | 14:04 |
| clarkb | dansmith: its been a back and forth with the web crawlers. we mitigate they come back in new exciting ways. Yesterday we mitigated by increasing connection limits as we were able to push back properly but not quickly enough ude ot limits. Today they've gone around our filtes so the larger limits allow them to exhaust resources. And so on | 14:04 |
| clarkb | the current appraoch is to deploy anubis to attempt to make the filtering more nimble | 14:05 |
| dansmith | ack | 14:05 |
| dansmith | it makes me realize we can't even do things like `tox -epep8` locally without it :/ | 14:06 |
| dansmith | clarkb: so is the anubis config set to allow fetching individual files? or just not blocking non-browser agent strings? | 14:13 |
| clarkb | dansmith: currently its limited only to browser agent strings | 14:14 |
| clarkb | so git shoudl still work | 14:14 |
| clarkb | and curl | 14:14 |
| dansmith | ack, in the browser I got the challenge, curl worked, but pip is still getting a 500 (maybe that means it made it through but the load is still affecting things?) | 14:14 |
| clarkb | the reason is that the crawlers all spoof browsers for whatever reason. That may change in the future or they may start calculating the proof of work data but until then... | 14:14 |
| dansmith | I imagine the latter but yeah | 14:15 |
| clarkb | dansmith: ya I think a 500 like that may be due to landing on another backend due to load blancing issues | 14:15 |
| clarkb | I've only got one backend updated with anubis and am about to work through the others now that we know it is mostly working | 14:15 |
| dansmith | ack | 14:15 |
| dansmith | I just made it through with pip | 14:17 |
| * dansmith rolls the dice again | 14:17 | |
| clarkb | dansmith: all backends have been manually updated at this point. I would expect the behaviors to be more consistently good, but I guess let us know if touy notice any thing | 15:08 |
| opendevreview | Ivan Anfimov proposed openstack/grenade master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/grenade/+/983888 | 15:12 |
| opendevreview | Ivan Anfimov proposed openstack/grenade master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/grenade/+/983888 | 15:13 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 15:26 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 15:26 |
| -opendevstatus- NOTICE: Anubis is now deployed on our Gitea backends, and things are back to working normally though you may notice an Anubis screen flash briefly when starting to browse opendev.org; any jobs which failed prior to 15:00 UTC today can be safely rechecked | 15:35 | |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 15:57 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 16:12 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 16:36 |
| gmaan | it seems devstack is blocked 'The label "ubuntu-bionic" was not found.' https://review.opendev.org/c/openstack/devstack/+/983084 | 16:39 |
| clarkb | yes we announced that bionic would be removed after the openstack release which got released so we removed bionic | 16:40 |
| gmaan | or that is temporary failure? I | 16:40 |
| gmaan | but this is passing so conused https://review.opendev.org/c/openstack/devstack/+/983861 | 16:41 |
| gmaan | or is it triggered when only touching zuul.yaml ? | 16:41 |
| clarkb | the reasons are 1) bionic is EOL and we really shouldn't use it anymore 2) zuul only supports bionic if you opt into using an old ansible version that zuul will drop support for soon and 3) we need to make space on our mirrors for resolute which releases like next week or something | 16:41 |
| clarkb | yes, when you edit the zuul configuration zuul validates the configuration and will return errors | 16:42 |
| clarkb | if you don't edit the configuration zuul will use a cached config if available | 16:42 |
| gmaan | clarkb: k, I thought we cleaned up the devstack in advance of that frickler removed bionic things but maybe nodeset is left | 16:42 |
| gmaan | i see | 16:42 |
| gmaan | let me try moving nodeset unless projects using it then we are blocked | 16:43 |
| gmaan | we already have that change approved but it stuck in gate https://review.opendev.org/c/openstack/devstack/+/976894 | 16:44 |
| frickler | gmaan: oh, yes, another fallout from unmaintained branches not being cared for properly. I'd say let's force merge this and if by the end of the month when 2024.2 goes EOL we still have those config errors, it is a good indicator for me that unmaintained/2023.1 and older can be EOLed, too | 16:47 |
| opendevreview | Merged openstack/devstack master: Clean up old nodeset definitions for bionic https://review.opendev.org/c/openstack/devstack/+/976894 | 16:51 |
| gmaan | frickler: ++ agree. thanks | 16:52 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Update hacking to 7.0.0 https://review.opendev.org/c/openstack/os-performance-tools/+/983913 | 16:52 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Update hacking to 7.0.0 https://review.opendev.org/c/openstack/os-performance-tools/+/983913 | 16:54 |
| frickler | ok, 983084 is now running jobs | 16:54 |
| gmaan | cool | 16:54 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 17:06 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: Remove outdated comment in setup.py https://review.opendev.org/c/openstack/os-performance-tools/+/983900 | 17:06 |
| opendevreview | Ghanshyam Maan proposed openstack/os-performance-tools master: Updating testing as per 2026.1 testing runtime https://review.opendev.org/c/openstack/os-performance-tools/+/966108 | 17:06 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: tox: Drop basepython https://review.opendev.org/c/openstack/os-performance-tools/+/983915 | 17:08 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: tox: Drop basepython https://review.opendev.org/c/openstack/os-performance-tools/+/983915 | 17:08 |
| opendevreview | Ivan Anfimov proposed openstack/os-performance-tools master: tox: Drop basepython https://review.opendev.org/c/openstack/os-performance-tools/+/983915 | 17:09 |
| opendevreview | Merged openstack/tempest master: Add response schemas for show_instance_action 2.84 https://review.opendev.org/c/openstack/tempest/+/983374 | 20:30 |
| opendevreview | Merged openstack/devstack master: Fix running renamed ironic job https://review.opendev.org/c/openstack/devstack/+/983084 | 21:15 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!