| opendevreview | chandan kumar proposed openstack/watcher-specs master: Add spec for improving watcher-dashboard testing https://review.opendev.org/c/openstack/watcher-specs/+/970220 | 07:26 |
|---|---|---|
| opendevreview | chandan kumar proposed openstack/watcher-specs master: Add spec for improving watcher-dashboard testing https://review.opendev.org/c/openstack/watcher-specs/+/970220 | 07:26 |
| opendevreview | chandan kumar proposed openstack/watcher-dashboard master: Add Playwright-based E2E testing framework https://review.opendev.org/c/openstack/watcher-dashboard/+/970353 | 07:40 |
| opendevreview | chandan kumar proposed openstack/watcher-dashboard master: Add Playwright integration test for skip action workflow https://review.opendev.org/c/openstack/watcher-dashboard/+/976594 | 07:41 |
| opendevreview | Takashi Kajinami proposed openstack/watcher master: Drop reference to legacy keystonemiddleware options https://review.opendev.org/c/openstack/watcher/+/988596 | 10:42 |
| *** jgilaber__ is now known as jgilaber | 10:58 | |
| jgilaber | Hi! Reminder that the IRC meeting will start in ~ 1 hour, feel free to add your topics to the agenda https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L31 | 10:58 |
| opendevreview | Takashi Kajinami proposed openstack/watcher master: Drop reference to legacy keystonemiddleware options https://review.opendev.org/c/openstack/watcher/+/988596 | 11:18 |
| jgilaber | #startmeeting Watcher | 12:01 |
| opendevmeet | Meeting started Thu May 14 12:01:04 2026 UTC and is due to finish in 60 minutes. The chair is jgilaber. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:01 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:01 |
| dviroel | o/ | 12:01 |
| sean-k-mooney | o/ | 12:01 |
| jgilaber | Hi everyone! Who is around today? | 12:01 |
| jgilaber | courtesy ping: amoralej chandankumar morenod rlandy | 12:01 |
| winiciusallan[m] | o/ | 12:01 |
| amoralej | o/ | 12:01 |
| rlandy | o/ | 12:01 |
| jgilaber | let's give a minute before starting | 12:01 |
| jgilaber | in the meantime feel free to add any last minute topic to the agenda https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L31 | 12:01 |
| chandankumar | o/ | 12:02 |
| jgilaber | ok let's start with the first topic | 12:02 |
| jgilaber | #topic Updates on watcher-dashboard UI testing playwright spec | 12:03 |
| jgilaber | chandankumar, go ahead | 12:03 |
| jgilaber | #link Updates on watcher-dashboard UI testing playwright spec | 12:03 |
| jgilaber | sorry | 12:03 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher-specs/+/970220 | 12:03 |
| chandankumar | jgilaber: sure | 12:03 |
| chandankumar | For dashboard ui testing spec, as a first step we have to get playwright requirement patch merge in requirement repo | 12:04 |
| chandankumar | https://review.opendev.org/c/openstack/requirements/+/983421/9#message-30b9c7cdbb579e99737a006aa39a18ef433cd2c6 we are getting pushback on this requirement change | 12:04 |
| chandankumar | as only single project watcher-dashboard is going to use and UI team is using something else | 12:04 |
| chandankumar | We got votes from horizon team also having no objection in merging it | 12:05 |
| chandankumar | In order to move forward with playwright work, We can add playwright to watcher-dashboard side | 12:05 |
| chandankumar | I was trying with test-requirements.txt file but it also requires us to add playwright in requirements project | 12:05 |
| sean-k-mooney | yes | 12:06 |
| sean-k-mooney | and that the only path forward to merging the playright code in my opion | 12:06 |
| chandankumar | So I am now left with one option to add playwright in tox.ini and go forward till someother project stats using it | 12:06 |
| sean-k-mooney | i dont want to workaorund the reuiqmente project even thoguh its trivial to do so | 12:06 |
| sean-k-mooney | no | 12:06 |
| sean-k-mooney | im -2 on that | 12:06 |
| sean-k-mooney | either we get this approve via the requireemtn proejct or we abandon the effort | 12:07 |
| dviroel | which other project could join us in the playwright adoption? if we have 2 projects using it, it would go from a -1 to a +2? | 12:07 |
| sean-k-mooney | well any ui project could | 12:07 |
| amoralej | yes, we should get it into the requirements project | 12:07 |
| chandankumar | dviroel: if we propose changes to grian-ui then? | 12:07 |
| sean-k-mooney | but i dont think that actuly the blocker | 12:08 |
| amoralej | i understand the complain is not that nobody else is using it but they perceived lack of consensus | 12:08 |
| dviroel | the closest seems to be grian-ui yeah | 12:08 |
| sean-k-mooney | right but the same woudl have been true for pytest | 12:09 |
| sean-k-mooney | it was added as an optional test runner and then horzon decied to actully use it for writing tests | 12:09 |
| sean-k-mooney | which was never inteneded to be allowed | 12:10 |
| amoralej | last comment from Jan may change the perception from Jens | 12:10 |
| sean-k-mooney | yep | 12:10 |
| sean-k-mooney | lets give them some tiem and we can follow up next week | 12:10 |
| chandankumar | sure, | 12:10 |
| amoralej | It may be good to reopen the conversation, yes | 12:10 |
| amoralej | comment is just from yesterday | 12:11 |
| chandankumar | I will follow up with Jens next week | 12:11 |
| jgilaber | thanks for the update chandankumar any other comments on this topic? | 12:11 |
| chandankumar | Do requirements team have weekly irc meeting? | 12:11 |
| chandankumar | May be I will bring playwright there to have more eyes on that | 12:12 |
| chandankumar | dviroel: no | 12:12 |
| dviroel | doesn't seems to have meetings anymore | 12:13 |
| dviroel | according with https://meetings.opendev.org/meetings/requirements/ | 12:13 |
| chandankumar | yes, correct. | 12:14 |
| chandankumar | I will follow up with them next week | 12:14 |
| chandankumar | thank you everyone on this topic! | 12:14 |
| jgilaber | ack, let's move on to reviews | 12:14 |
| jgilaber | #topic Reviews | 12:14 |
| jgilaber | amoralej, you have the first two | 12:14 |
| dviroel | tks chandankumar for pursuing this effort | 12:15 |
| amoralej | #info I did a first approach for next step in functional testing | 12:15 |
| amoralej | i got some time to retake the work for functional tests | 12:15 |
| dviroel | ++ | 12:15 |
| amoralej | #link https://review.opendev.org/q/topic:%22blueprint-functional-test-infrastructure%22+status:open | 12:15 |
| amoralej | in this first patch i create basic infra to run watcher services and run tests with gabbi | 12:16 |
| amoralej | we can test any api workflow with dummy goal and strategy | 12:16 |
| amoralej | not external services fixtures yet, that will be next | 12:17 |
| amoralej | i added some workflows in the patch to validate | 12:17 |
| sean-k-mooney | the gabi api test and the broader fucntional tests are actuly two related but seperate parts | 12:17 |
| sean-k-mooney | while the gabi test will be declaritvie | 12:17 |
| sean-k-mooney | the broader functional test are more imperitive in that they use the normnal way of write tests with unittest | 12:18 |
| amoralej | both ways are supported in the patch | 12:18 |
| sean-k-mooney | ack | 12:18 |
| amoralej | actually i left tests written in both ways | 12:18 |
| sean-k-mooney | just notign we proably wont end up testing say zone migration with gabi and emulating vm migrtion | 12:18 |
| amoralej | so that it can be examples of how to run in both ways | 12:18 |
| sean-k-mooney | alhtoughw e potitnally could | 12:19 |
| sean-k-mooney | ack | 12:19 |
| dviroel | i didn't review it yet, but +1 for the contributor doc addition :) | 12:19 |
| amoralej | because of the requirement to create declarative the model? | 12:19 |
| amoralej | ^ i mean why not to use gabbi for zone_migration i.e. | 12:19 |
| amoralej | i'm looking for a way to do it :) | 12:20 |
| sean-k-mooney | amoralej: its just not really intended fo rvery complex senairos | 12:20 |
| sean-k-mooney | i.e. doign the creation of server in nova then callign watcher | 12:20 |
| amoralej | i think as we create more complex tests we will see the limits | 12:20 |
| sean-k-mooney | you coudl do that but its really intnded for testign one service | 12:20 |
| sean-k-mooney | anyway we can see | 12:21 |
| amoralej | yep, we can see | 12:21 |
| amoralej | please take a look and let me know your thoughts | 12:21 |
| dviroel | ack | 12:21 |
| amoralej | let's move to next one | 12:22 |
| amoralej | #info rephrased docs in https://review.opendev.org/c/openstack/watcher/+/986777 | 12:22 |
| amoralej | as discussed last week, i rephrased the docs there | 12:23 |
| amoralej | no need to go into details now or we will get out of time again for other topics, just for awareness :) | 12:23 |
| dviroel | ok, thanks, it is on my list for today along with other patches | 12:24 |
| amoralej | unless you want to discuss in detail now, we can move on to next reviews | 12:24 |
| jgilaber | thanks amoralej I think we can move to the next one | 12:25 |
| jgilaber | #link https://review.opendev.org/q/topic:%22bug/2141951%22 | 12:25 |
| jgilaber | dviroel, I think this is yours? | 12:25 |
| dviroel | yeah | 12:25 |
| dviroel | there are a couple of patches propose | 12:25 |
| dviroel | related with workload_stabilization strategy improvements | 12:26 |
| dviroel | there are 3 changes propose in the end | 12:26 |
| dviroel | 1) is an improvement in the model locking that we discussed past week | 12:26 |
| dviroel | which amoralej already reviwed and pointed some issues in ci that can be related | 12:26 |
| dviroel | or may just rised now due to the improvement in the locking system | 12:27 |
| dviroel | but I still need to dig more into this issue | 12:27 |
| dviroel | the other 2 are possble improvements to the strategy itself | 12:27 |
| amoralej | I think the issue is related, actually we could see it as a pre-existing issue but was mitigated by the per-method shared lock | 12:28 |
| amoralej | but yeah, it deserves some careful review | 12:29 |
| dviroel | 2) destination nodes ordering and selection in simulate_strategy | 12:29 |
| dviroel | 3) source nodes ordering and selection in simulate_strategy | 12:29 |
| dviroel | the TL;DR; of those 23 are in the etherpad | 12:29 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-workload-stabilization-improvements | 12:29 |
| amoralej | I will check 2 and 3 soon i hope | 12:29 |
| dviroel | with some outputs from my local tests | 12:30 |
| dviroel | the lock improvement show 9.5x speedup there | 12:30 |
| dviroel | the other ones are open for discussion in the end | 12:31 |
| amoralej | thanks for the documentation dviroel++ looks great | 12:31 |
| dviroel | another thing | 12:31 |
| dviroel | there is a bug in the code | 12:31 |
| dviroel | #link https://bugs.launchpad.net/watcher/+bug/2152254 | 12:31 |
| dviroel | which we don't see it happen since all calls for map_instance are using instance objects, not uuid string | 12:32 |
| dviroel | but it is possible to reproduce using uuid strings | 12:32 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/988297 | 12:32 |
| dviroel | is a unit test that can reproduce the issue for instance | 12:33 |
| dviroel | next thing is to investigate the error in the | 12:34 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/987540 | 12:34 |
| dviroel | "Replace global lockutils locks with per-instance RLock" since is the most important fix/improvement there | 12:34 |
| dviroel | that's it | 12:35 |
| dviroel | i will follow up this ^ patch here and provide more updates soon | 12:35 |
| dviroel | if there is no other question about them, we can move on | 12:36 |
| jgilaber | thanks dviroel there is a lot of information in the ehterpad | 12:36 |
| dviroel | we can sync again next week in this topic too | 12:36 |
| jgilaber | we can move on to bug triage then | 12:36 |
| dviroel | ack | 12:36 |
| jgilaber | #topic Bugs | 12:37 |
| jgilaber | we have a couple from last week and some newer ones | 12:37 |
| jgilaber | #link https://bugs.launchpad.net/watcher/+bug/1837400 | 12:37 |
| jgilaber | this one is old but is still marked as "New" | 12:37 |
| jgilaber | we marked it as "needs-re-triage" last year | 12:37 |
| dviroel | needs to see if it reproduces in newer releases | 12:39 |
| dviroel | there is no info about which release was used in this test | 12:40 |
| dviroel | so it is incomplete, unless someone try to reproduce it | 12:41 |
| dviroel | no? | 12:41 |
| jgilaber | +1 to that, there is also no job in ci testing this datasource | 12:41 |
| jgilaber | any objection to marking it as incomplete? | 12:41 |
| amoralej | in recent job i see "CRITICAL watcher.decision_engine.datasources.grafana [None req-7fe25725-c0ee-41f8-8000-9b71131638b2 None None] GrafanaHelper authentication token not configured" | 12:42 |
| amoralej | which seems to be a different one ? | 12:42 |
| jgilaber | that seems different, which job is that? | 12:42 |
| jgilaber | I wasn't aware of any that would enable this datasource | 12:42 |
| dviroel | different message but could be the same source of issue | 12:43 |
| amoralej | https://zuul.opendev.org/t/openstack/build/ca7e211e7a75482ab1a78b6109924d75 | 12:43 |
| amoralej | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ca7/openstack/ca7e211e7a75482ab1a78b6109924d75/controller/logs/screen-watcher-decision-engine.txt | 12:43 |
| dviroel | "May 13 10:01:39.699638 np46f41325385c4 watcher-decision-engine[101325]: CRITICAL watcher.decision_engine.datasources.grafana [None req-b2e0fb46-8fab-4c63-b700-b92b856b367c None None] GrafanaHelper authentication token not configured" | 12:44 |
| amoralej | actually, it's not enabled | 12:44 |
| dviroel | it is not good to have a CRITICAL message logged | 12:44 |
| jgilaber | that job configures prometheus as the only datasource https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ca7/openstack/ca7e211e7a75482ab1a78b6109924d75/controller/logs/etc/watcher/watcher_conf.txt | 12:44 |
| amoralej | it's probably helper initialization image | 12:44 |
| amoralej | initialization error, i meant | 12:44 |
| dviroel | someone has cycles to take a look whats going on? | 12:45 |
| dviroel | but the LP Bug seems to be actually using grafana | 12:46 |
| dviroel | the issue with a CRITICAL log from grafana, which wasn't configured as datasource | 12:46 |
| dviroel | it something that shold be fixed | 12:46 |
| dviroel | I think that we can file another LP for that | 12:47 |
| jgilaber | yes, we should it's a different problem | 12:47 |
| dviroel | i would keep 1837400 as incomplete | 12:47 |
| dviroel | and file a new one to investigaste the Critical log in the job | 12:47 |
| jgilaber | agreed | 12:48 |
| dviroel | ack | 12:48 |
| amoralej | i can create it | 12:48 |
| dviroel | thanks amoralej | 12:48 |
| jgilaber | I'm also looking at the tags, and we don't have any for datasources | 12:48 |
| jgilaber | ok to create one? | 12:49 |
| dviroel | yeah | 12:49 |
| jgilaber | ack done | 12:49 |
| jgilaber | let's move to the next one | 12:49 |
| jgilaber | #link https://bugs.launchpad.net/watcher/+bug/2151776 | 12:49 |
| dviroel | david filed this one | 12:50 |
| dviroel | it is weird that strategy need to check in the model if a instance is excluded or not | 12:51 |
| amoralej | ftr https://bugs.launchpad.net/watcher/+bug/2152623 | 12:51 |
| dviroel | but it may have its own reason for that, which we may agree or not and plan a fix | 12:51 |
| * dviroel tks amoralej | 12:51 | |
| amoralej | I think there is a reason for that. We may change how the model works to change that behavior but until then, i think it's a valid bug | 12:53 |
| dviroel | I agree that is valid | 12:53 |
| amoralej | something we may do is to fix the existing issues with current model, and think on improvements on the model and scope behavior as separated issues | 12:54 |
| jgilaber | +1 it's valid and high imo since it could lead to users migrating instances by mistake | 12:54 |
| jgilaber | short term we could add the check for "watcher_exclude" to the strategies that are missing it | 12:54 |
| dviroel | this requires more investation on each strategy I think. Maybe some of them could have their doc updated to reflect that exclude instance would not work? | 12:54 |
| amoralej | yes, note that in some cases strategies may need to list all the instances even the excluded ones, and in other only the non-excluded | 12:55 |
| amoralej | i think that's why it was implemented in that way | 12:55 |
| jgilaber | that makes sense for host_maintenance but not so much for zone migration for example | 12:56 |
| dviroel | fixing with a check will require more testing on each too, so maybe each strategy needs an investigation | 12:56 |
| amoralej | or maybe not :) but needs careful investigation on each strategy as dviroel said | 12:56 |
| jgilaber | ack, we've only got 4 minutes, let's decide on the importance and revisit | 12:57 |
| jgilaber | or do we want to retriage next week with more time? | 12:57 |
| amoralej | what i'd like is to get more information from David too, to get the actual problematic behavior explained instead of the implementation detail | 12:57 |
| dviroel | i would say that is high, since in the end, the action plan could end up moving instance that user don't want to | 12:58 |
| amoralej | +1 | 12:58 |
| jgilaber | ack, let's bring it up next week again | 12:58 |
| jgilaber | dviroel, ok if we move your bug to next week as well? | 12:58 |
| dviroel | sure, np | 12:58 |
| jgilaber | thanks | 12:58 |
| jgilaber | last topic | 12:59 |
| jgilaber | #topic Stable releases | 12:59 |
| jgilaber | looks like we still have a few backports open for stable/2025.* branches | 12:59 |
| amoralej | i reviewed some backports in stable releases | 12:59 |
| jgilaber | thanks amoralej | 12:59 |
| amoralej | yep, it'd be good to get first the 2025.2 reviewed | 12:59 |
| amoralej | and merged, to make sure we don get stuff in 2025.1 which is not in .2 | 13:00 |
| dviroel | ++ | 13:00 |
| amoralej | also i sent one to 2026.1 btw, the one about the debug logs for workload_balance | 13:00 |
| jgilaber | agreed, please review when time allows | 13:00 |
| amoralej | https://review.opendev.org/q/project:openstack/watcher+branch:stable/2026.1+status:open | 13:01 |
| jgilaber | anything else on this topic? | 13:02 |
| jgilaber | last thing for today then | 13:02 |
| jgilaber | #topic Volunteers to chair next meeting | 13:02 |
| jgilaber | any volunteer? | 13:02 |
| amoralej | i can take it | 13:02 |
| jgilaber | thanks amoralej | 13:02 |
| dviroel | ++ | 13:02 |
| dviroel | thanks amoralej | 13:02 |
| jgilaber | and thanks everyone for the discussion! | 13:03 |
| jgilaber | #endmeeting | 13:03 |
| opendevmeet | Meeting ended Thu May 14 13:03:06 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-14-12.01.html | 13:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-14-12.01.txt | 13:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-14-12.01.log.html | 13:03 |
| amoralej | wrt https://review.opendev.org/c/openstack/watcher/+/981225 i understant we want that in 2026.1 as it's slurp ? | 13:03 |
| dviroel | thanks for chairing jgilaber++ | 13:03 |
| amoralej | thanks jgilaber++ ! | 13:03 |
| jgilaber | that is a good question, I was thinking the same thing | 13:03 |
| amoralej | i know little about grenade jobs tbh | 13:03 |
| amoralej | chandankumar, ^ | 13:04 |
| chandankumar | amoralej: skip-level-always runs from n-2 to n relase while normal grenade job runs from n-1 to n | 13:08 |
| chandankumar | 2026.1 is a slurp release | 13:09 |
| jgilaber | that seems to follow guidance from https://docs.openstack.org/project-team-guide/release-cadence-adjustment.html#details | 13:09 |
| jgilaber | +1 from me | 13:09 |
| amoralej | so, in slurp we need to run skip-level-always, right? and in non-slurp regular grenade ? | 13:10 |
| jgilaber | however the grenade jobs did not run in that patch, is that because of irrelevant-files? | 13:10 |
| chandankumar | https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L521 | 13:10 |
| sean-k-mooney | skip-level-alwasy | 13:10 |
| sean-k-mooney | is intended to be run on every release | 13:10 |
| chandankumar | It run on all branches | 13:10 |
| sean-k-mooney | but we offically only supprot .1 -> .1 | 13:10 |
| sean-k-mooney | and .2 -> .1 | 13:11 |
| sean-k-mooney | upgragdes | 13:11 |
| sean-k-mooney | the .2 -> .2 | 13:11 |
| sean-k-mooney | that it tests on non slup release | 13:11 |
| sean-k-mooney | is there to catch bugs early | 13:11 |
| sean-k-mooney | but not officaly supprotd | 13:11 |
| sean-k-mooney | it also useful when some downstream vendor use dto do 3 release of FFU... | 13:11 |
| amoralej | got it, thanks! | 13:12 |
| *** haleyb is now known as haleyb|away | 15:21 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!