| dviroel | hi all, watcher meering will start in 15m, here in this channel | 11:46 |
|---|---|---|
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: [WIP] Add end-to-end strategies execution tests https://review.opendev.org/c/openstack/watcher/+/962784 | 11:54 |
| dviroel | #startmeeting watcher | 12:01 |
| opendevmeet | Meeting started Thu Oct 2 12:01:22 2025 UTC and is due to finish in 60 minutes. The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:01 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:01 |
| dviroel | hi all o/ | 12:01 |
| dviroel | who is around today? | 12:01 |
| jgilaber | o/ | 12:02 |
| morenod | o/ | 12:02 |
| amoralej | o/ | 12:02 |
| dviroel | courtesy ping: sean-k-mooney chandankumar rlandy | 12:02 |
| dviroel | ok, let's start with today's meeting agenda | 12:02 |
| dviroel | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L28 (Meeting agenda) | 12:02 |
| dviroel | feel free to add your own topics to our agenda | 12:03 |
| sean-k-mooney | o/ | 12:03 |
| dviroel | #topic Announcements | 12:03 |
| dviroel | one quick announcement | 12:03 |
| dviroel | which you should already know | 12:04 |
| dviroel | OpenStack Flamingo is now released! | 12:04 |
| dviroel | https://lists.openstack.org/archives/list/openstack-announce@lists.openstack.org/thread/7ANVEGX7NMVJ7A6ROGCPGXGC7NWQ4UBT/ | 12:04 |
| dviroel | that reminds us that we can get our backports to stable/2025.2 | 12:05 |
| dviroel | and continue with backports to older releases | 12:05 |
| sean-k-mooney | yep that is very true | 12:05 |
| dviroel | i also added that comment in reviews topic | 12:05 |
| dviroel | any other announcement before we move on? | 12:06 |
| dviroel | ok | 12:06 |
| dviroel | #topic PTG schedule for Watcher | 12:06 |
| dviroel | we start this discussion last week | 12:07 |
| dviroel | we just need to define our schedule, so I can book our room | 12:07 |
| dviroel | there were 2 proposals from last week | 12:07 |
| dviroel | 1- Wed-Fri (13 UTC, 14 UTC, 15 UTC) | 12:07 |
| dviroel | 2 - Tue-Thu (13 UTC, 14 UTC, 15 UTC) | 12:07 |
| dviroel | both of them will conflict with other projects, like noa | 12:08 |
| dviroel | s/noa/nova | 12:08 |
| dviroel | I see that sean-k-mooney also proposes to go with Tue-Thu | 12:09 |
| sean-k-mooney | 2 woudl be my prefernce yes | 12:09 |
| sean-k-mooney | either are ok | 12:09 |
| amoralej | +1 to Tue-Thu | 12:09 |
| dviroel | this gives Fri as backup, or to join any other session | 12:09 |
| dviroel | I am also ok with Tue-Thu, and we still can block the time slots to join others team discussion | 12:10 |
| dviroel | so lets keep Tue-Thu (13 UTC, 14 UTC, 15 UTC) and if needed, we add Fri to the list | 12:10 |
| jgilaber | +1 to option 2 | 12:10 |
| dviroel | #agree watcher ptg schedule to be booked: Tuesday, Wednesday and Thurdsays, 13 UTC, 14 UTC, 15 UTC | 12:11 |
| dviroel | #undo | 12:12 |
| opendevmeet | Removing item from minutes: #agreed watcher ptg schedule to be booked: Tuesday, Wednesday and Thurdsays, 13 UTC, 14 UTC, 15 UTC | 12:12 |
| dviroel | #agreed watcher ptg schedule to be booked: Tuesday, Wednesday and Thurdsays, at 13 UTC, 14 UTC, 15 UTC | 12:12 |
| dviroel | ok, our planning etherpad is here: | 12:12 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-2026.1-ptg | 12:12 |
| dviroel | please add any other with your name, that you plan to cover | 12:13 |
| dviroel | i should work on setting the slots for each | 12:13 |
| dviroel | #action dviroel to book pts slot for watcher | 12:14 |
| dviroel | ok, anything else in this topic? | 12:14 |
| dviroel | #topic Unused code to handle Active/Active decisio-engine services | 12:14 |
| amoralej | that's mine | 12:14 |
| dviroel | all yours | 12:15 |
| amoralej | so, while digging into the A/A solution for decision-engine a found this https://github.com/openstack/watcher/blob/master/watcher/api/scheduling.py | 12:15 |
| amoralej | that implements a basic approach to move audits to alive decision-engines when one is detected as dead | 12:16 |
| sean-k-mooney | ya that more of a stopgap then a real solution | 12:16 |
| amoralej | yes | 12:16 |
| sean-k-mooney | that does nto mean we cant start with that but it shoudl nto be in the api | 12:16 |
| amoralej | so, i have two questions, cuould this be a short-term fix? | 12:16 |
| amoralej | if so, how/where should we run it? | 12:17 |
| sean-k-mooney | if we add a new watcher-schduler consolsript entry point that just runs this then yes | 12:17 |
| amoralej | yes, that's what i was thinking | 12:17 |
| sean-k-mooney | if we do that we can then build on that to buidl a real solution | 12:17 |
| amoralej | that'd be fine, but in that case, we should also what would be the long-term real solution (that may be the discussion in ptg) | 12:18 |
| sean-k-mooney | i mentioned this in the ptg topic but my approch would be to have the schduler dispatch work to the descion enginve via an rpc to a shared queue that the desions engines plural listen on | 12:18 |
| amoralej | another option would be t run it as a thread in the decision-engine, but i don't know if i like that | 12:19 |
| sean-k-mooney | so descion engins would not pool and would not do any schdulign of the continues audits at all | 12:19 |
| sean-k-mooney | amoralej: we coudl be if we did that | 12:19 |
| amoralej | decision-engine would just get a rpc at the time to execute it and would execute it, right? | 12:19 |
| sean-k-mooney | then we woudl need to also have a distibute locl/memofy view | 12:19 |
| sean-k-mooney | so that only one of the service did the reshuffel | 12:20 |
| sean-k-mooney | amoralej: yes that is my idea | 12:20 |
| amoralej | yes, that's another question, we should only run a instance of the scheduler | 12:20 |
| amoralej | i mean, in the short term solution | 12:20 |
| amoralej | with the simple approach | 12:20 |
| sean-k-mooney | we woudl remove the use of apschduler entirly form applier and descion engin | 12:20 |
| amoralej | or do some kind of locking | 12:20 |
| sean-k-mooney | and make them purly worker that execute action or audits in responce to an rpc message | 12:20 |
| sean-k-mooney | amoralej: we can also do a very simple form of leader election | 12:21 |
| sean-k-mooney | i did that in nova's schduler recently | 12:21 |
| amoralej | based on something in db? | 12:21 |
| sean-k-mooney | so all descison engins coudl run it in a new tread | 12:21 |
| sean-k-mooney | yes i can fidn the patch and link it to you after the meeting | 12:22 |
| sean-k-mooney | its a very simple apprhc based on a rendevour hash | 12:22 |
| amoralej | no, no, no problem, i was just thinking in if we would need some other infra service | 12:22 |
| amoralej | actually, i think something simple would be enough | 12:22 |
| sean-k-mooney | https://github.com/openstack/nova/commit/e98393c5c26743ec4c862af3e1a4beaa7f2d174b | 12:23 |
| amoralej | is it worthy to discuss and send patch for the 1st step intermediate solution before PTG or better to hold on and do the full discussion in PTG? | 12:23 |
| sean-k-mooney | i think having a poc to look at is good | 12:23 |
| sean-k-mooney | so fi you have time why not | 12:24 |
| sean-k-mooney | but if you look at https://github.com/openstack/nova/commit/e98393c5c26743ec4c862af3e1a4beaa7f2d174b#diff-ce71048fd132b0db262fad40554388acaa624bbb621e1141020c30840cdc1472R112 | 12:24 |
| amoralej | thanks | 12:24 |
| sean-k-mooney | i litrally just looked up the set of schdluers (in our case descion engions) filtered them by up and sorted them to make it stable | 12:24 |
| sean-k-mooney | and then the first one does the work and rest early return | 12:24 |
| sean-k-mooney | we coudl do the same thing with this reblance approch | 12:25 |
| amoralej | yes, i think that would be enough | 12:25 |
| dviroel | +1 | 12:25 |
| amoralej | so, i understand we are fine with going for this intermediate solution before implementing a rpc based one | 12:25 |
| sean-k-mooney | this bascilly rely on the fact that if it take a few seconds for the leader to change its fine because it will heal over time | 12:26 |
| amoralej | which will likely need more changes | 12:26 |
| sean-k-mooney | amoralej: i think yes. its better to make incremental progress | 12:26 |
| sean-k-mooney | as long as that does not block a more complete solution down the road | 12:26 |
| sean-k-mooney | i dont really see this as creating any tech debt | 12:26 |
| amoralej | i tested current approach by running a watcher-api standalone and worked fine, i mean audits were rebalanced and the new decision-engined picked them in next execution | 12:27 |
| amoralej | ack, thanks | 12:27 |
| sean-k-mooney | i think we shoudl proceed with that as an initial setp but in the ptg we shoud dicuss what the next step woudl be | 12:27 |
| amoralej | from reporting pov, may i report this as a bug? | 12:27 |
| sean-k-mooney | yes in that the functionalty is not supprted in the wsgi mode and we have deprecated the eventlet standalone verison of the api | 12:28 |
| amoralej | scheduling.APISchedulingService() is not executed when running as wsgi? | 12:28 |
| sean-k-mooney | so its a regression | 12:28 |
| amoralej | yep, agreed | 12:28 |
| amoralej | so, i think that's it about this topic | 12:28 |
| sean-k-mooney | and as such coudl be a bug. even if its a boarderline feature. | 12:28 |
| dviroel | amoralej: no it is not, afaik | 12:28 |
| amoralej | i meant the bug would be "scheduling.APISchedulingService() is not executed when running as wsgi" | 12:29 |
| amoralej | sorry i was not clear :) | 12:29 |
| amoralej | that can be understood as a bug although borderline feature, as Sean said | 12:30 |
| dviroel | i think that the functionality of migrating audits in the end | 12:30 |
| dviroel | but indeed, this will be a great topic for our PTG | 12:31 |
| sean-k-mooney | i think dviroel ment no its not called in wsgi mode | 12:31 |
| dviroel | yes ^ | 12:31 |
| amoralej | it's in the list | 12:32 |
| dviroel | ok, thanks amoralej | 12:32 |
| dviroel | ok, lets move to the next topic | 12:32 |
| dviroel | we can cover this with more details at the ptg | 12:32 |
| dviroel | #topic what about testing the end-to-end strategy execution as unit tests? | 12:32 |
| amoralej | #link https://review.opendev.org/c/openstack/watcher/+/962784 | 12:33 |
| amoralej | i sent that WIP patch to get feedback | 12:33 |
| dviroel | it is a patch that you just pushed, right amoralej ? | 12:33 |
| amoralej | yes | 12:33 |
| amoralej | i realized that at least in some of the strategies, our unit tests are focused on each method | 12:34 |
| amoralej | which is fine | 12:34 |
| amoralej | but, i was wondering if we should also run tests which execute the entire strategy for a predefined metrics and model | 12:34 |
| amoralej | and check the resulting solution | 12:34 |
| sean-k-mooney | amoralej: so technicaly that would not be a unit test | 12:35 |
| amoralej | i called this end-to-end strategy testing (may exist a better name) | 12:35 |
| amoralej | yes, that's my doubt | 12:35 |
| sean-k-mooney | but it woudl be a functional tests in nova parlance | 12:35 |
| sean-k-mooney | and i want to build a functional test suite | 12:35 |
| amoralej | it would allow us to test much more complex cases that we do in tempest | 12:35 |
| sean-k-mooney | so im ok with this type of testign but htere is more test setup we need to do to do it peroperly | 12:35 |
| amoralej | testing with hundred of computes and vms | 12:35 |
| sean-k-mooney | yep | 12:36 |
| amoralej | what would mean properly, from your pov? | 12:36 |
| amoralej | in my patch the coverage scope is restricted to the strategy itself | 12:36 |
| sean-k-mooney | ne of the tenant of doing correct functional testing is you minimize any mocking of the watcher project but use fixture for external services | 12:37 |
| amoralej | you mean, simulate prometheus, nova, etc... and run watcher in "real mode" with no mocks, right? | 12:37 |
| sean-k-mooney | so the way you do that is you start a watcher api descion engin and appler in the test using oslo messaging in memory message bus and sqlite | 12:37 |
| sean-k-mooney | then your test actully calls the api with a psot to triggert the audit | 12:38 |
| sean-k-mooney | but you use fixture to emulate the respocnes form nova ectra | 12:38 |
| sean-k-mooney | so yes | 12:38 |
| sean-k-mooney | i think you have a middel gorund | 12:38 |
| amoralej | got it, what i proposed is more that unit tests but less that funcional tests ... | 12:38 |
| amoralej | exactly | 12:39 |
| sean-k-mooney | so what i woudl suggst for now | 12:39 |
| sean-k-mooney | is we add watcher.test.unit.senario | 12:39 |
| sean-k-mooney | but eventulaly i would like to to have watcher.test.functional.* | 12:39 |
| sean-k-mooney | which will do even less mocking | 12:39 |
| amoralej | i like the idea of moving these intermediate to it's own folder and classes | 12:40 |
| amoralej | make sense | 12:40 |
| amoralej | i will | 12:40 |
| jgilaber | there are similar tests to what amoralej proposed in the zone_migration https://github.com/openstack/watcher/blob/03073a1b0d8dacfc49b2d220a1120be381d831d1/watcher/tests/decision_engine/strategy/strategies/test_zone_migration.py#L678 | 12:41 |
| jgilaber | I don't know if also for other strategies | 12:41 |
| amoralej | i check the workload_balancing and i think some other, but yeah, not all | 12:42 |
| sean-k-mooney | so as a general rule or guide line | 12:42 |
| amoralej | i think it may reserve a full check on all the strategies, at least the non-experimental ones | 12:43 |
| sean-k-mooney | unit test shoudl test one thing, they shoudl mock any calle that are to fucntion in a diffent module and any function in the current moduel that have sideefects | 12:43 |
| sean-k-mooney | that does nto mean they have to test exactly one fucntion but they shoudl be small and targeted | 12:43 |
| sean-k-mooney | have unit.senairo tessts in there onw folder | 12:44 |
| sean-k-mooney | makes it clear thaty they are not followign the normal pattern | 12:44 |
| sean-k-mooney | and make maintianing/reviewign the simpler | 12:44 |
| sean-k-mooney | so if we want to add more secenairo tests im fine with that | 12:44 |
| jgilaber | +1 to moving these existing tests to dedicated folder | 12:44 |
| amoralej | +1 | 12:44 |
| sean-k-mooney | but we should still writh the simple unit test to test the relevent fucniton on there own too | 12:45 |
| amoralej | i also created a way to define the metrics we want to get for each host and instance, instead of having them hardcoded as we have today | 12:45 |
| sean-k-mooney | dviroel: any input on ^ | 12:45 |
| amoralej | based on the uuid | 12:45 |
| amoralej | it's not very elegant, tbh | 12:46 |
| amoralej | but i'd like to get your feedback on that too | 12:46 |
| sean-k-mooney | amoralej: cool that could be come the basis fo t test fixture in the future | 12:46 |
| dviroel | i am not sure if .scenario or e2e are the best names in this situation, but I also don't have any other idea | 12:46 |
| amoralej | https://review.opendev.org/c/openstack/watcher/+/962784/1/watcher/tests/decision_engine/model/gnocchi_metrics.py | 12:46 |
| dviroel | but I am +1 on moving to a different directory | 12:47 |
| amoralej | "ComputeNode hostname="hostname_1" uuid="Node_1_CPU_5_RAM_46" :) | 12:47 |
| amoralej | i couldn't find a better way to add arbitrary metadata into the model | 12:47 |
| sean-k-mooney | oh i tought you ment it was done via a map lookup | 12:48 |
| sean-k-mooney | i was thinking more that in the tst you would do somtihng liek metrics_fixture.register_metics(uuid, {...}) | 12:48 |
| amoralej | currently, there are hardcoded values for specific compute names and instances in https://review.opendev.org/c/openstack/watcher/+/962784/1/watcher/tests/decision_engine/model/gnocchi_metrics.py | 12:48 |
| sean-k-mooney | and then when you used the metrics client it woudl return those metrics ectra | 12:49 |
| amoralej | i can look for a better way, i can think in something like that | 12:49 |
| sean-k-mooney | so we dont need to design this now | 12:49 |
| sean-k-mooney | before we move on | 12:49 |
| sean-k-mooney | one thing i wanted to do to make room for edxperiments and impovments | 12:50 |
| dviroel | we may want to have a topic to discuss more about our unit tests and functional tests at the PTG - in case someone wants to take it | 12:50 |
| amoralej | but i liked the idea of having both the metrics and model defined together in that xml | 12:50 |
| amoralej | i will add it dviroel | 12:50 |
| sean-k-mooney | is move all the tests form watcher/tests/ to watcher/tests/unit/ | 12:50 |
| amoralej | but anyone can take it | 12:50 |
| dviroel | amoralej: nice, thanks | 12:50 |
| sean-k-mooney | because i want to add watcher/tests/functional and watcher/tests/watcher_fixtures/ | 12:50 |
| sean-k-mooney | we can discuss that more in teh ptg too but does ^ sound ok to folks | 12:51 |
| dviroel | make sense | 12:51 |
| jgilaber | +1 | 12:51 |
| amoralej | sounds good | 12:51 |
| sean-k-mooney | cool anything more on this topic for today? | 12:52 |
| dviroel | ok, lets move on, and add more feedbacks in the patch | 12:52 |
| amoralej | not from my side | 12:52 |
| dviroel | ack | 12:52 |
| dviroel | tks amoralej | 12:52 |
| dviroel | #Reviews | 12:52 |
| dviroel | there is nothing new there | 12:53 |
| dviroel | I just added a reminder to our 2025.2 open backports | 12:53 |
| dviroel | #link https://review.opendev.org/q/project:openstack/watcher+branch:stable/2025.2+is:open | 12:53 |
| sean-k-mooney | i guess one minor update | 12:53 |
| dviroel | yes | 12:53 |
| sean-k-mooney | i approved the watcher-spec patch to create the 2026.1 folder yesterday | 12:54 |
| sean-k-mooney | so if we need to crate them that not possible | 12:54 |
| sean-k-mooney | *now | 12:54 |
| sean-k-mooney | thanks dviroel for propsoing that | 12:54 |
| dviroel | np, we also agree at some point, for the next release, we would split that change | 12:55 |
| dviroel | and create the 2026.2 earlier | 12:55 |
| dviroel | so we can decide to move specs to next release earlier too | 12:55 |
| sean-k-mooney | right we shoudl create teh new folder at m2 and do the approved -> implemntted mvoe at m3 | 12:55 |
| dviroel | +! | 12:56 |
| dviroel | +1 | 12:56 |
| dviroel | any other review that someone wants to bring to this meeting? | 12:56 |
| dviroel | there are some open changes in watcher-tempest-plugin | 12:56 |
| dviroel | but all under review I think | 12:56 |
| sean-k-mooney | jgilaber: i think you had 3? | 12:56 |
| dviroel | #link https://review.opendev.org/q/project:openstack/watcher-tempest-plugin+status:open | 12:57 |
| jgilaber | I have 2 currently in the tempest plugin | 12:57 |
| dviroel | I have been working on some refactoring too: | 12:57 |
| jgilaber | on was merged | 12:57 |
| dviroel | #link https://review.opendev.org/q/topic:%22organize_tests%22 | 12:57 |
| sean-k-mooney | and there is an nfs patch too? | 12:57 |
| jgilaber | not upstream | 12:58 |
| sean-k-mooney | ah. so going to dviroel series | 12:58 |
| dviroel | tks | 12:58 |
| sean-k-mooney | so the first is just reogainisng where tests are located to group them more logically | 12:59 |
| dviroel | we only have 1 minute | 12:59 |
| amoralej | i already commented in https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/960310, i think that refactor is good | 12:59 |
| amoralej | thanks for taking care of it | 12:59 |
| dviroel | yes, there are duplicated tests | 12:59 |
| dviroel | the ones in test_execute_strategies | 12:59 |
| sean-k-mooney | ya i see you are also changing the decorators.idempotent_id | 13:00 |
| sean-k-mooney | so normally we dont change that if we are renameing a test | 13:00 |
| dviroel | since there are duplicated tests, I was trying to keep one of them | 13:00 |
| dviroel | so it is possible that is moving the test and the id | 13:01 |
| sean-k-mooney | yep that is fine but we shoudl keep one of the two idempotent_id | 13:01 |
| dviroel | but requires some review on that too yes | 13:01 |
| dviroel | to make sure that is correct | 13:02 |
| sean-k-mooney | on the downstream side thsoe IDs are what is used to track thing in polarion | 13:02 |
| sean-k-mooney | so so that when we go form verion to version even if the test is renames we have continuity fo the test cases | 13:02 |
| morenod | try to keep the latest tests, they are which are being executed and tracked in polarion | 13:02 |
| dviroel | good point, i will double check that | 13:03 |
| dviroel | thanks for raising this | 13:03 |
| opendevreview | Joan Gilabert proposed openstack/watcher-tempest-plugin master: Test zone migration volume and compute migrations https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702 | 13:03 |
| dviroel | ok, we are over time | 13:03 |
| dviroel | please add your questions and comments in the patch, I will get to them asap | 13:03 |
| dviroel | anything else? | 13:04 |
| dviroel | ok, we will meet again next week | 13:04 |
| dviroel | thank you all for participating | 13:05 |
| dviroel | #endmeeting | 13:05 |
| opendevmeet | Meeting ended Thu Oct 2 13:05:06 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:05 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-02-12.01.html | 13:05 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-02-12.01.txt | 13:05 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-02-12.01.log.html | 13:05 |
| morenod | thanks dviroel | 13:05 |
| opendevreview | Takashi Kajinami proposed openstack/watcher master: Migrate bandit options to pyproject.toml https://review.opendev.org/c/openstack/watcher/+/962830 | 15:05 |
| opendevreview | Douglas Viroel proposed openstack/watcher-tempest-plugin master: Organize strategy tests and remove duplicated tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/961762 | 18:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher-tempest-plugin master: Refactor execute_strategy method into smaller helpers https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/960310 | 18:09 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!