| opendevreview | chandan kumar proposed openstack/watcher-tempest-plugin master: Add tests for host_maintenance strategy with backup node https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/947944 | 05:01 |
|---|---|---|
| morenod | Hello every one, friendly reminder, weekly meeting will start on this channel in one hour (12:00 UTC). Please add topics to https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L39 | 11:00 |
| morenod | #startmeeting | 12:01 |
| opendevmeet | morenod: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 12:01 |
| morenod | #startmeeting Watcher IRC Weekly Meeting - 8 May 2025 | 12:02 |
| opendevmeet | Meeting started Thu May 8 12:02:07 2025 UTC and is due to finish in 60 minutes. The chair is morenod. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:02 |
| opendevmeet | The meeting name has been set to 'watcher_irc_weekly_meeting___8_may_2025' | 12:02 |
| morenod | hi o/ - who's around today? | 12:02 |
| amoralej | o/ | 12:02 |
| chandankumar | o/ | 12:02 |
| jgilaber | o/ | 12:02 |
| dviroel | hi o/ | 12:02 |
| rlandy | o/ | 12:03 |
| morenod | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L36 agenda for today | 12:05 |
| morenod | #topic (dviroel) Eventlet removal | 12:06 |
| dviroel | so, i started this etherpad | 12:07 |
| morenod | #link etherpad: https://etherpad.opendev.org/p/watcher-eventlet-removal | 12:07 |
| dviroel | to follow up the progress of eventler removal on watcher | 12:07 |
| dviroel | for now, it has some references to eventlet usage in the code | 12:07 |
| dviroel | to help us identify how much work is pending, and how our progress is | 12:08 |
| dviroel | we can track there all related gerrit review too | 12:08 |
| dviroel | my expectation is that we will, from now, periodically revisit this topic and etherpad in our meetinds | 12:09 |
| dviroel | i am planning to work on the required changes, and should bring more updates soon | 12:10 |
| dviroel | it is also open for anyone that wants to join the effort :) | 12:11 |
| rlandy | dviroel: is there any task in particular you want help with? Is testing covered? | 12:11 |
| chandankumar | dviroel: thank you for adding the plan on etherpad, I will go over it and will happy to join the effort. | 12:11 |
| dviroel | after we raise changes that we need, we can probably split them | 12:13 |
| dviroel | we can probably work in different components, and test them locally and on CI | 12:14 |
| amoralej | wrt "WSGIService creates a eventlet wsgi server" i think deprecating it it's likely something we should do in this release | 12:14 |
| dviroel | amoralej: i think that sean-k-mooney already deprecated it | 12:14 |
| amoralej | good | 12:14 |
| amoralej | right https://review.opendev.org/c/openstack/watcher/+/948438/8/releasenotes/notes/add-wsgi-module-support-597f479e31979270.yaml | 12:15 |
| dviroel | yeah, need to double check | 12:15 |
| dviroel | right | 12:15 |
| dviroel | chandankumar: thanks for the help :) | 12:16 |
| dviroel | some interesting link wrt this topic: | 12:17 |
| dviroel | #link https://removal.eventlet.org/ | 12:17 |
| dviroel | ^ which also have other related links | 12:17 |
| dviroel | #link https://wiki.openstack.org/wiki/Eventlet-removal | 12:17 |
| dviroel | there are some important changes that are still being workd in oslo | 12:18 |
| dviroel | which we depend on | 12:18 |
| dviroel | like oslo.service, which seems to be close to merge | 12:19 |
| dviroel | I plan to add more details in the etherpad in the following days | 12:19 |
| dviroel | there is also oftc channel to track this effort: #openstack-eventlet-removal | 12:20 |
| dviroel | ok, that's what I want to bring today, any questions? | 12:21 |
| dviroel | we can continue discussing in the following weeks, and in the channel + etherpad | 12:22 |
| dviroel | morenod: i think that we can move on | 12:25 |
| morenod | #topic (jgilaber) Call for reviews | 12:25 |
| jgilaber | I added a few patcher that I think are ready for review | 12:25 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/948322 | 12:26 |
| jgilaber | this patch changes the default interface for keystone_client which was set as admin | 12:26 |
| sean-k-mooney | oh o/ | 12:26 |
| sean-k-mooney | sorry reading back please continue | 12:27 |
| jgilaber | second and third are related | 12:27 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/946182 | 12:27 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/945199 | 12:27 |
| jgilaber | one adds a test to reproduce the bug https://launchpad.net/bugs/2103458 | 12:27 |
| jgilaber | and the other fixes it | 12:28 |
| sean-k-mooney | dviroel: so for the eventletremoval we may want to consier having a spec for that or a seperate wathcer etherpad to capture the steps and the workitmes | 12:28 |
| sean-k-mooney | we might consider just haveign a specless blueprint but its not soemthign we shoudl just do without some kind fo design doc. | 12:28 |
| sean-k-mooney | and upstream tracker | 12:28 |
| sean-k-mooney | ok i think im caught up with jgilaber call for review | 12:29 |
| jgilaber | and final patch is a really small one that fixes a debug log call that had the wrong number of arguments | 12:29 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/949047 | 12:29 |
| jgilaber | no worries, unless there are questions I'm done with my topic, I just wanted to bring attention to the patches | 12:30 |
| sean-k-mooney | jgilaber: i have started reviwign that serise yesterday but i got pulled into other things ill try and get back to it today or monday | 12:30 |
| jgilaber | thanks sean-k-mooney! | 12:30 |
| sean-k-mooney | i htink you alrady adress my feedback from yesterday and ci is green so thats promising | 12:31 |
| dviroel | jgilaber: thanks, we can discuss more in the review, planning to go through them today | 12:31 |
| jgilaber | then I think we can move on to the next topic, thanks! | 12:31 |
| morenod | #topic (dviroel) Deprecated Noisy Neighbor strategy | 12:32 |
| dviroel | right, as discussed at flamingo ptg, we agreed on deprecating the noisy neighbor strategy | 12:32 |
| dviroel | so I start a patch for it | 12:33 |
| dviroel | and still going to send an email on ML later today | 12:33 |
| opendevreview | Joan Gilabert proposed openstack/watcher master: Set keystone_client default interface to internal https://review.opendev.org/c/openstack/watcher/+/948322 | 12:33 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/949065 | 12:33 |
| dviroel | so, just fyi, and calling for reviews | 12:34 |
| dviroel | that's all thanks | 12:36 |
| morenod | #topic Bug Triage | 12:36 |
| morenod | #link https://bugs.launchpad.net/watcher/+bug/2108855 (Watcher should include keystone session when creating PrometheusAPICLient) | 12:37 |
| jgilaber | should we close that as not a bug? | 12:38 |
| jgilaber | in the last comment Jaromir mentions that he'll create a spec for the request | 12:39 |
| sean-k-mooney | am we may want to wait until its created | 12:40 |
| sean-k-mooney | we also likely need to expalin that the current poc likely wont work for us | 12:40 |
| sean-k-mooney | so we likely will need them to change both the obslerablity client and ateos to supprot oru continuos audits | 12:40 |
| sean-k-mooney | i had hoped they coudl atten this or a future irc meeting to dicusss this realtime but i guess if they propose a spec we can explain | 12:41 |
| sean-k-mooney | the extra requirement we will have | 12:41 |
| morenod | #link https://bugs.launchpad.net/watcher/+bug/2108988 (Zone migration strategy fails if "dst_node" field is not passed) | 12:44 |
| sean-k-mooney | ill leave a comemnt on https://bugs.launchpad.net/watcher/+bug/2108855 and we can move on th the next one | 12:44 |
| jgilaber | I opened 2108988 while testing the zone_migration strategy | 12:45 |
| jgilaber | basically the strategy has the dst_node and dst_pool parameters as optional for instance and volume migrations | 12:45 |
| jgilaber | but if they are not set, when creating the migration action objects they are passed as None instead of ommitted | 12:46 |
| jgilaber | and that causes the error in the bug report | 12:46 |
| jgilaber | I have a patch to fix it, but omitting the parameter for instance migration, since nova can find out a destination node | 12:47 |
| jgilaber | but volume migration is trickier since there are three types of migrations, and only one of them requires to specify a dst_pool | 12:47 |
| jgilaber | so in my path I raise an exception if it's needed but not passed as input | 12:48 |
| jgilaber | s/path/patch | 12:48 |
| amoralej | that error message is seen by the user from cli ? | 12:49 |
| amoralej | i mean the one about dst_pool missing | 12:49 |
| jgilaber | I don't think so, it would be in the decision-engine logs | 12:50 |
| amoralej | but the user will have not access to logs (or may not) | 12:50 |
| jgilaber | I don't think we can easily check for it in the client | 12:50 |
| amoralej | yeah | 12:50 |
| jgilaber | as we need to check the storage details | 12:51 |
| jgilaber | whether volumes are attached or free | 12:51 |
| amoralej | i don't know how watcher manage failure in audits executions ... | 12:51 |
| jgilaber | the audit shows as failed | 12:51 |
| jgilaber | the bug is missing the importance field, should we set 'medium' or 'high'? | 12:54 |
| amoralej | this is probably the best we can do for now, but this is also related to the parameters validation we discussed recently related to audit creation | 12:54 |
| dviroel | seems high to me, the strategy doesn't seems to work as expected | 12:55 |
| amoralej | +1 | 12:55 |
| jgilaber | same, set as high, if there are no other questions we can move to the next one | 12:56 |
| dviroel | we can discuss more in the patch and in launchpad | 12:56 |
| dviroel | jgilaber: you can bring more details on lauchpad too, about your findings | 12:56 |
| jgilaber | yep, I'll do that | 12:57 |
| dviroel | going to take a look on it | 12:57 |
| dviroel | tks | 12:57 |
| morenod | #link https://bugs.launchpad.net/watcher/+bug/2109309 (migrate action fails in host maintenance strategy when we specify backup node) | 12:57 |
| sean-k-mooney | if we ever have a requrie parmater that is not passed | 12:57 |
| sean-k-mooney | we shoudl be passing a 400 excption back to the user at the pai | 12:58 |
| sean-k-mooney | however if the cli is not ensuring you passed all the required aguments | 12:58 |
| sean-k-mooney | that is a second bug as the api shoudl nto be the first palce we validate that | 12:58 |
| sean-k-mooney | https://bugs.launchpad.net/watcher/+bug/2109309 | 13:00 |
| * dviroel time check | 13:00 | |
| sean-k-mooney | is now fixed but we need to backport | 13:00 |
| jgilaber | in this case I think the both the api and cli are ok, they are validating agains the strategy schema ok | 13:00 |
| sean-k-mooney | lets add the target serise for each relase | 13:00 |
| chandankumar | Regarding bug 2109309, we merged the fix in master and backport is also up for review https://review.opendev.org/c/openstack/watcher/+/948773?usp=search | 13:01 |
| sean-k-mooney | ya i just added the 2024.1 abd 2025.1 tracker to the bug | 13:01 |
| chandankumar | for adding target series, Do we need to create epoxy or flamingo target series? | 13:01 |
| sean-k-mooney | bobcat is now EOL so we skip it | 13:02 |
| sean-k-mooney | actully 2024.2 should be there too | 13:02 |
| sean-k-mooney | we already have them | 13:03 |
| chandankumar | i can see now, thank you sean-k-mooney | 13:03 |
| sean-k-mooney | the offical names of the release are actuly 2025.1 for epoxy | 13:03 |
| dviroel | I don't remember how to add target series tbh | 13:03 |
| sean-k-mooney | epoxy is just a code name we are ment to avoid using it | 13:03 |
| sean-k-mooney | that a governace change that happen when we roled over | 13:03 |
| sean-k-mooney | i.e from antelope on | 13:04 |
| dviroel | ah, found "Target to series" | 13:04 |
| sean-k-mooney | dviroel: yep | 13:04 |
| sean-k-mooney | already did it | 13:04 |
| dviroel | yep | 13:04 |
| sean-k-mooney | so you can refresh | 13:04 |
| sean-k-mooney | ok so we can proceed with the backprot but i think we can wrap up the meeting | 13:04 |
| morenod | #link https://bugs.launchpad.net/watcher/+bug/2109945 (host maintenance strategy should not enable disabled hosts to do live migration) | 13:06 |
| morenod | that is the last one | 13:06 |
| chandankumar | here is the summary of the bug. We have 2 disabled compute nodes with disabled reason watcher_maintaining. | 13:06 |
| chandankumar | We tried to use host maintaince to migrate instances from maintaining node. the strategy recommended with the action plan. | 13:06 |
| chandankumar | It listed the migration of instances to disabled_node. It should not touch disabled node in theory. nodes might be disabled for a reason by user. | 13:06 |
| chandankumar | while going through the code, we found that, it enables the disabled node with disabled reason "watcher_disabled" and perform migration on that. | 13:07 |
| chandankumar | code in question: https://github.com/openstack/watcher/blob/master/watcher/decision_engine/strategy/strategies/host_maintenance.py#L236-L205 | 13:07 |
| chandankumar | we discussed on about it on irc. we thought about dropping the code. | 13:07 |
| chandankumar | what do we think about this bug? | 13:09 |
| chandankumar | I have a cr ready for the same https://review.opendev.org/c/openstack/watcher/+/948967 | 13:09 |
| dviroel | i'm planning to revisit this patch | 13:11 |
| chandankumar | thank you dviroel ! | 13:12 |
| amoralej | I think the proposed change make sense | 13:12 |
| jgilaber | +1 | 13:12 |
| morenod | #open topics | 13:15 |
| morenod | Anybody want to raise something? | 13:15 |
| chandankumar | is it ok to set importancer to medium for above bug and add target series to that? | 13:15 |
| dviroel | chandankumar: it ok | 13:16 |
| chandankumar | Done thank you! | 13:17 |
| morenod | #topic Volunteer to chair the next meeting, 15th May 2025 | 13:17 |
| morenod | dviroel has volunteered to chair the next meeting, is that right? | 13:18 |
| dviroel | yep | 13:18 |
| morenod | cool, then thanks everyone for joining today and see you next Thu! | 13:18 |
| jgilaber | thanks! | 13:19 |
| dviroel | o/ | 13:19 |
| morenod | #endmeeting | 13:19 |
| opendevmeet | Meeting ended Thu May 8 13:19:07 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:19 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___8_may_2025/2025/watcher_irc_weekly_meeting___8_may_2025.2025-05-08-12.02.html | 13:19 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___8_may_2025/2025/watcher_irc_weekly_meeting___8_may_2025.2025-05-08-12.02.txt | 13:19 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___8_may_2025/2025/watcher_irc_weekly_meeting___8_may_2025.2025-05-08-12.02.log.html | 13:19 |
| dviroel | sean-k-mooney: so, wrt eventlet removal | 13:23 |
| dviroel | we can have our discussions on etherpad, or should we really create a spec? | 13:24 |
| dviroel | i started this etherpad to follow up https://etherpad.opendev.org/p/watcher-eventlet-removal | 13:24 |
| sean-k-mooney | realaisically wheil i prefer to default to specs we chosee not to for nova because we know there are things we will discover along the way | 13:26 |
| sean-k-mooney | so im fine with having a specless blueprint and then linking to the etherpad to track the progres and things we need to do. and hten keeping most of the dicusssion in gerrit orn how to do it | 13:26 |
| sean-k-mooney | although haveign a spec might be nice fo historical reasosn. gibi agreed to treate it like a subtem for nova | 13:27 |
| sean-k-mooney | so that measn there is a slot in the weekly meeting for it where an update is given on teh progress and next steps | 13:28 |
| dviroel | yeah, even with some initial analysis, i am not sure if it is going to work as planned | 13:28 |
| sean-k-mooney | i kind of started on the removal work when i was tyring to fix 3.12 and then dropt that code | 13:28 |
| sean-k-mooney | i would start by swapping all usage of the GreenThreadPoolExecutor with the ThreadPoolExecutor form futureist | 13:29 |
| sean-k-mooney | well that or there is antoher step that cna be doen before that | 13:29 |
| sean-k-mooney | which is thigns that are monkey patched but dont need it can be moved to a diffent module | 13:30 |
| sean-k-mooney | and by things i mean the cli utils like the db sync commands | 13:30 |
| sean-k-mooney | for nova i proposed slitign the cmd module into freestanding/standalone (no eventlete) and patched/eventlet | 13:31 |
| sean-k-mooney | that what neuton do too | 13:32 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add check-uuid to pep8 tox tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/949178 | 13:32 |
| sean-k-mooney | dviroel: like this https://review.opendev.org/c/openstack/nova/+/904424 | 13:33 |
| sean-k-mooney | dviroel: https://github.com/openstack/watcher/commit/c5edad224659596a2534ee58be06a8a8190366c0 was partly a start on prepareign to remvoe eventlet too. | 13:34 |
| dviroel | right, that makes sense, easier to control what is being patched | 13:37 |
| dviroel | yeah, I remember this patch in watcher | 13:37 |
| dviroel | going to add a fixed topic for eventlet removal on weekly meetings | 13:38 |
| dviroel | going to take a look on these inital steps | 13:38 |
| dviroel | start proposing things | 13:38 |
| opendevreview | Joan Gilabert proposed openstack/watcher master: Fix incorrect logging format https://review.opendev.org/c/openstack/watcher/+/822559 | 13:52 |
| opendevreview | Joan Gilabert proposed openstack/watcher master: Check logs in some cinder and nova helper tests https://review.opendev.org/c/openstack/watcher/+/949187 | 13:52 |
| jgilaber | jfyi ^^, in the irc meeting I asked for reviews for https://review.opendev.org/c/openstack/watcher/+/949047 but I realized there was already an existing patch for the problem | 13:54 |
| jgilaber | I've sent a new one with tests that should reproduce the problem | 13:55 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add check-uuid to pep8 tox tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/949178 | 15:35 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add check-uuid to pep8 tox tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/949178 | 15:43 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add check-uuid to pep8 tox tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/949178 | 16:21 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: Set actionplan state to FAILED if any action has failed https://review.opendev.org/c/openstack/watcher/+/949225 | 16:22 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: Set actionplan state to FAILED if any action has failed https://review.opendev.org/c/openstack/watcher/+/949225 | 16:23 |
| opendevreview | Merged openstack/watcher master: Add test for EfficacyIndicator value in mysql https://review.opendev.org/c/openstack/watcher/+/946182 | 19:43 |
| opendevreview | Merged openstack/grian-ui master: bootstrap repo https://review.opendev.org/c/openstack/grian-ui/+/948661 | 22:18 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!