| opendevreview | Merged openstack/watcher stable/2026.1: Add policy enforcement to webhook trigger endpoint https://review.opendev.org/c/openstack/watcher/+/1001507 | 00:36 |
|---|---|---|
| opendevreview | Merged openstack/watcher stable/2025.2: Add policy enforcement to webhook trigger endpoint https://review.opendev.org/c/openstack/watcher/+/1001508 | 07:16 |
| opendevreview | Merged openstack/watcher stable/2025.1: Add policy enforcement to webhook trigger endpoint https://review.opendev.org/c/openstack/watcher/+/1001509 | 10:36 |
| opendevreview | Merged openstack/watcher master: Add Nova/Placement emulators and host_maintenance functional tests https://review.opendev.org/c/openstack/watcher/+/993352 | 11:17 |
| opendevreview | Douglas Viroel proposed openstack/watcher stable/2026.1: Fix allowed_nodes corruption in ComputeScope https://review.opendev.org/c/openstack/watcher/+/1001644 | 11:20 |
| opendevreview | Douglas Viroel proposed openstack/watcher stable/2025.2: Fix allowed_nodes corruption in ComputeScope https://review.opendev.org/c/openstack/watcher/+/1001645 | 11:52 |
| opendevreview | Douglas Viroel proposed openstack/watcher stable/2025.1: Fix allowed_nodes corruption in ComputeScope https://review.opendev.org/c/openstack/watcher/+/1001648 | 12:05 |
| dviroel | #startmeeting watcher | 13:00 |
| opendevmeet | Meeting started Thu Aug 20 13:00:20 2026 UTC and is due to finish in 60 minutes. The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'watcher' | 13:00 |
| * dviroel forgot to send a reminder message :) | 13:00 | |
| dviroel | hi all o/ - who is around today? | 13:01 |
| morenod | o/ | 13:01 |
| winiciusallan[m] | o/ | 13:01 |
| dviroel | courtesy ping: amoralej sean-k-mooney chandankumar rlandy | 13:01 |
| amoralej | o/ | 13:02 |
| dviroel | alright, lets start | 13:03 |
| dviroel | starting with today's meeting agenda | 13:03 |
| rlandy_ | o/ | 13:03 |
| dviroel | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L34 (Meeting agenda) | 13:03 |
| *** rlandy_ is now known as rlandy | 13:03 | |
| chandank` | o/ | 13:03 |
| dviroel | feel free to add any topics that you want to cover/highlight | 13:03 |
| dviroel | we should be covering today the patches that need to merge before FF | 13:04 |
| dviroel | first | 13:04 |
| dviroel | #topic Announcements | 13:04 |
| sean-k-mooney | o/ | 13:04 |
| dviroel | there are a few | 13:04 |
| dviroel | 1) PTL/TC Elections | 13:04 |
| dviroel | so nominations ended, we have the candidates list here https://governance.openstack.org/election/#ptl-candidates | 13:05 |
| dviroel | as we can see, no ptl elections will happen | 13:05 |
| dviroel | and now | 13:06 |
| dviroel | the TC Election Campaigning Period starts | 13:06 |
| dviroel | you can find more info on | 13:06 |
| dviroel | #link https://governance.openstack.org/election/ | 13:06 |
| dviroel | or in the kickoff email sent | 13:07 |
| dviroel | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/LTBWW3CEIPYG6QPQAUUQHGJKPPV7GPSC/ | 13:07 |
| dviroel | and that cover the election announcements | 13:07 |
| dviroel | next one is | 13:08 |
| dviroel | 2) Feature Freeze is next week | 13:08 |
| dviroel | #link https://releases.openstack.org/hibiscus/schedule.html | 13:08 |
| dviroel | we should cover the pending reviews/blueprints in the following topics | 13:08 |
| dviroel | lets see how we are with them | 13:08 |
| dviroel | any other announcement that you folks want to cover? | 13:09 |
| dviroel | if not, we can move to our Review topic | 13:09 |
| dviroel | ack | 13:09 |
| dviroel | #topic Reviews | 13:10 |
| dviroel | there is a few already listed in the meeting etherpad | 13:10 |
| dviroel | lets start with the feature listed in the review etherpad | 13:10 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-2026.2-status | 13:10 |
| dviroel | 1) Openstacksdk migration | 13:10 |
| dviroel | we already discussed last week | 13:10 |
| dviroel | we are godd there | 13:11 |
| dviroel | 2) Audit Pipeline | 13:11 |
| dviroel | Audit template extension already merged, only tempest tests are still pending , which is ok | 13:11 |
| dviroel | we can merge them later too | 13:11 |
| dviroel | About the audit pipeline itself | 13:12 |
| dviroel | #link https://review.opendev.org/q/topic:%22bp/audit-pipeline%22+is:open | 13:12 |
| dviroel | there is already a lot of changes open | 13:12 |
| dviroel | we merged the first one about strategy metric cache | 13:13 |
| dviroel | there are some updates since last week | 13:13 |
| dviroel | I included an additional change just to cover the efficacy indicators piece | 13:13 |
| dviroel | it ended to be a little bit different from the spec | 13:14 |
| dviroel | I had to create a indicator grouping for similar indicators: e.g: zone_migrations provides both planned_live_migrate_instance_count and planned_cold_migrate_instance_count, while workload_stabilization provides instance_migrations_count | 13:14 |
| amoralej | what combination of strategies may be a good one to test? | 13:14 |
| dviroel | I have been using zone_migration/host_maintenance + workload_stabilization | 13:15 |
| amoralej | hardware_maintenance + workload_stabilization? | 13:15 |
| dviroel | depends how many compute nodes you have | 13:15 |
| sean-k-mooney | hum ok we may want to cosnier if we shoudl normalise those indicator in the future | 13:15 |
| sean-k-mooney | i think today the have mostly been adhoc rahter then standarised names across all the stragies for the same thing | 13:16 |
| dviroel | amoralej: since hardware_maintenance will disable one node, you will likely want 2 others to balance | 13:16 |
| amoralej | yes | 13:16 |
| amoralej | ah, right, good point | 13:16 |
| amoralej | actually, it's hard to test with two, i guess... | 13:16 |
| sean-k-mooney | ya | 13:16 |
| amoralej | zone_migration + workload_stabilization would send instances back to the origin one? | 13:17 |
| dviroel | but in CI i proposed zone_migration (which does not disable the source) to have it moveing an instance back (which also tests the audit pipeline collapsing migrations) | 13:17 |
| sean-k-mooney | you can use consolidation and then stabelisation | 13:17 |
| sean-k-mooney | but it may be hard to seee the final result | 13:17 |
| sean-k-mooney | realy you need 3 computes | 13:17 |
| amoralej | yep | 13:17 |
| sean-k-mooney | the funciotnal test would be good for testign this later | 13:18 |
| dviroel | we currently don't have a CI job running no check/gate with 3 nodes.. | 13:18 |
| sean-k-mooney | we can create more complex sensiors without the actull resouce need | 13:18 |
| dviroel | right | 13:18 |
| amoralej | yep, i was also thinking in the functional ... | 13:18 |
| sean-k-mooney | dviroel: that prety trival to add | 13:18 |
| sean-k-mooney | just need to defien a new nodeset | 13:18 |
| dviroel | sure, we can discuss if we want to move one job to 3 nodes | 13:19 |
| dviroel | we already have the nodeset | 13:19 |
| morenod | im going to need to add a job with 3 computes | 13:19 |
| sean-k-mooney | for now we can test with the actuator | 13:19 |
| sean-k-mooney | we can compose a number of actuator audit templeates toghether | 13:19 |
| morenod | for audit scope: https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/995799 | 13:19 |
| morenod | https://review.opendev.org/c/openstack/watcher/+/995797 | 13:19 |
| morenod | we can use that | 13:19 |
| sean-k-mooney | for example have one live migrate a vm and another cold migrate it | 13:20 |
| sean-k-mooney | put both in a pipelien | 13:20 |
| dviroel | good idea | 13:20 |
| sean-k-mooney | and see that all of the actions are applied | 13:20 |
| sean-k-mooney | in the event log | 13:20 |
| sean-k-mooney | it can be 2 live mgiration was well or any other to actions | 13:21 |
| sean-k-mooney | i.e. disabel in one and enable in the other | 13:21 |
| sean-k-mooney | as long as we can assert the sideffects | 13:21 |
| dviroel | so for now, there are one thing that is still missing in the changes | 13:22 |
| dviroel | that is the storage model mutation, current patches only works with compute model | 13:22 |
| dviroel | I plan to propose as an additional change | 13:22 |
| dviroel | to not have bigger changes | 13:22 |
| amoralej | +1 | 13:23 |
| dviroel | and see what we can have merged in this release | 13:23 |
| dviroel | I also have a spec update to made, which I just need to push | 13:23 |
| amoralej | honestly, we have just one supported strategy using the storage model, right? | 13:23 |
| dviroel | we also have StorageCapacityBalance | 13:24 |
| dviroel | but we don't even have a ci job running for it | 13:24 |
| dviroel | only for zone_migration | 13:24 |
| amoralej | yep | 13:24 |
| dviroel | so yeah | 13:24 |
| dviroel | depending on the amount of work to get all in, we may need to consider defering some specific funcitonalties to the next release | 13:25 |
| amoralej | i think it's fine to do storage mutation in separated review | 13:25 |
| dviroel | ack | 13:25 |
| sean-k-mooney | it would be ok for the storage support to come next cycle | 13:25 |
| sean-k-mooney | provide we docuemnt that adn call it out as a limiation | 13:25 |
| dviroel | sean-k-mooney: teim-ci has been very useful and got a lot of issues in the audit pipeline patches | 13:25 |
| sean-k-mooney | im glad :) | 13:25 |
| dviroel | i proposed updats based on its output | 13:26 |
| sean-k-mooney | just an fyi to everyone i made some changes to it on monday to improve its output after some regressiosn 3 weeks ago | 13:26 |
| dviroel | so for now, feel free to ping me for any audit pipeline related question | 13:26 |
| sean-k-mooney | so let me know how that is workign for folks | 13:27 |
| dviroel | ++ | 13:27 |
| dviroel | do you guys have any concern or question about this feature? otherwise we can move to the next one | 13:27 |
| sean-k-mooney | i think you have highlight the main risks | 13:27 |
| sean-k-mooney | so i am not sure we realsiticly will get everythign merged i.e. the storage supprot | 13:28 |
| sean-k-mooney | but we can try | 13:28 |
| dviroel | ack, lets see how we move forward during next weel | 13:28 |
| sean-k-mooney | and as i asid if we docuemnt that as a limitation i dont think that is a blocker for the basic supprot | 13:28 |
| dviroel | we can go with a proper documentation with limitations if needed | 13:28 |
| dviroel | ack | 13:28 |
| dviroel | ok, next one in the list | 13:29 |
| dviroel | Functional Tests Infrastructure | 13:29 |
| dviroel | i think that we are on track here, we recently merged some patches | 13:29 |
| dviroel | it will be very useful for us to expand our tests | 13:29 |
| amoralej | yes, thanks for merging | 13:29 |
| dviroel | I think that we can move forward in the list | 13:30 |
| amoralej | I'm working in adding the prometheus one asap | 13:30 |
| dviroel | +1 | 13:30 |
| amoralej | FYI, i'm creating some helpers functions for the topology to populate metrics based on mean or standard deviation | 13:30 |
| sean-k-mooney | sound useful | 13:31 |
| dviroel | that change will be very helpful for audit pipeline, mainly for strategies that we also simulate the metrics | 13:31 |
| amoralej | so, after adding the compute and instances using the builder pattern something like ..distribute_cpu(mean=X, sd=Y) | 13:31 |
| amoralej | I'm trying to get that working while keeping predictability | 13:31 |
| amoralej | which i think will be helpful when creating topologies | 13:32 |
| dviroel | yeah, for more specific test cases | 13:32 |
| amoralej | yep | 13:32 |
| dviroel | nice amoralej++ | 13:33 |
| amoralej | that's it wrt functional tests | 13:33 |
| dviroel | check if makes sense splitting changes, depending on the sizing | 13:33 |
| sean-k-mooney | the builder pattehr can have its limits | 13:33 |
| amoralej | yes it can :) | 13:33 |
| sean-k-mooney | its nic but we may also want to be abel to tweak spcrif comptue ectra | 13:34 |
| amoralej | we will never lose the option of defining specific compute, instance, etc... | 13:34 |
| amoralej | helper is just something optional on top | 13:34 |
| amoralej | if i can make it work :) | 13:34 |
| dviroel | ack | 13:35 |
| dviroel | ok, lets then move to the next feature in the list | 13:35 |
| amoralej | actually, it may also be follow up | 13:35 |
| dviroel | amoralej: sometimes make easier to merge | 13:35 |
| dviroel | next one is | 13:35 |
| dviroel | Preemtible Instances | 13:35 |
| dviroel | i think that winiciusallan[m] is with us | 13:35 |
| winiciusallan[m] | yes, I'm around | 13:36 |
| dviroel | let me update the etherpad | 13:36 |
| dviroel | since you merged the 2 actions in one patch | 13:36 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/1000168 | 13:36 |
| winiciusallan[m] | in general, the actions patch are ready to review | 13:36 |
| winiciusallan[m] | I need to push tempest tests for those actions, will try to work on it this week | 13:37 |
| amoralej | thanks for adding the docs winiciusallan[m]! | 13:38 |
| dviroel | ack, as I mention last week, let me know if you need help with that | 13:38 |
| winiciusallan[m] | I will probably need some help to setup an environment so I can test myself quickly | 13:38 |
| winiciusallan[m] | but yeah, I will let you guys know | 13:38 |
| dviroel | ack, you can ping me or someone else in the irc, no problem | 13:38 |
| winiciusallan[m] | do you have any concerns regarding the strategy patch? I saw that you left a few comments | 13:39 |
| amoralej | quick question, instance deletion in nova is synchronous or asynchronous ? | 13:39 |
| winiciusallan[m] | IIRC it's async | 13:39 |
| winiciusallan[m] | sean can confirm | 13:39 |
| amoralej | you are waiting for it in the helper | 13:41 |
| amoralej | so it should be finbe | 13:41 |
| dviroel | while we wait for sea n to confirm | 13:41 |
| amoralej | np | 13:41 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/1000170 | 13:41 |
| * dviroel got distracted | 13:43 | |
| dviroel | i have been reviewing the second patch | 13:43 |
| dviroel | that adds the new goal/strategy | 13:43 |
| sean-k-mooney | instance deleteion i belive returns a 204 | 13:43 |
| sean-k-mooney | so i think its async yes | 13:43 |
| sean-k-mooney | we poll for the instnace to not be in server show in tempetst normally | 13:44 |
| sean-k-mooney | actully idf definlty ascyn | 13:44 |
| dviroel | I saw an issue with sort_candidates method in the strategy | 13:44 |
| dviroel | since in the spec was mentioned that it would be using placement resource allocations | 13:44 |
| dviroel | but in the change it is using flavor resources already available in the model | 13:45 |
| sean-k-mooney | because we have a feature call "local delete" where if the compute agent is offlien when you ask to delete a vm it will be marked as deleted in teh db and then actully deleted when the compute agent is started again | 13:45 |
| sean-k-mooney | dviroel: ah yes | 13:45 |
| sean-k-mooney | i have not looked at the yet | 13:45 |
| dviroel | in order to keep in sync with the spec | 13:45 |
| sean-k-mooney | but we do not curently capture the allocation | 13:45 |
| dviroel | we would need to also extend the cdm | 13:45 |
| sean-k-mooney | yes | 13:45 |
| winiciusallan[m] | yes, it'd be needed a chnage in the CDM | 13:46 |
| sean-k-mooney | we shoudl do that | 13:46 |
| sean-k-mooney | but if we dont have time just remove the resouce match on the allcation for now | 13:46 |
| sean-k-mooney | and readded it later | 13:46 |
| sean-k-mooney | i think that will be simple to add | 13:46 |
| dviroel | we are 1 week from feature freeze, so we need to be careful on what we still plan to propose | 13:46 |
| dviroel | since we may not have time to review all | 13:47 |
| dviroel | yeah, we can move that for next release | 13:47 |
| sean-k-mooney | i would split the reouce matching part into its own comiit | 13:47 |
| sean-k-mooney | adn we can merge it if we have tiem and if not continue it into next cycle | 13:47 |
| winiciusallan[m] | ++ | 13:48 |
| winiciusallan[m] | but I'll do what's currently missing in the already opened patches, and then can owrk on this ^ | 13:48 |
| dviroel | winiciusallan[m]: so you can keep only the SORT_KEY_AGE in the sort_candidates | 13:49 |
| dviroel | and propose a new change on top of 1000170 adding the resource sorting piece + cdm | 13:50 |
| dviroel | as an extension of preemption_preference | 13:50 |
| dviroel | this additional change will have more risk of not making into hibiscus | 13:51 |
| dviroel | and focus first on solving all issues/comment in current strategy implementation + tempest test addition | 13:51 |
| winiciusallan[m] | so wouldn't be the CDM change an independent patch? | 13:51 |
| amoralej | sorry but, the cdm changes are to have the resource usage allocations in computes ? | 13:53 |
| sean-k-mooney | it woudl be a singel patch for resouce and cdm | 13:53 |
| sean-k-mooney | you coudl do it cdm and then reouce sort in that order | 13:53 |
| sean-k-mooney | as 2 patches | 13:53 |
| sean-k-mooney | but useing the flavor is not correct today | 13:53 |
| amoralej | we have logic in the model to calculate resources usage | 13:54 |
| sean-k-mooney | it devites form teh design in the spec | 13:54 |
| amoralej | based on flavor | 13:54 |
| amoralej | why is not correct sean-k-mooney ? | 13:54 |
| sean-k-mooney | the spec wasy this will be based on the placement allcoation so it works for all resocue classes | 13:54 |
| sean-k-mooney | i.e. cybrog device | 13:54 |
| sean-k-mooney | or anything else beyond cpu ram and disk | 13:55 |
| amoralej | ack | 13:55 |
| sean-k-mooney | we coudl have a limited form for cpu ram and disk | 13:55 |
| sean-k-mooney | provided that does not use resouce: | 13:55 |
| amoralej | but for cpu, ram and disk, we'd rely on existing logic to calculate used resources ? | 13:55 |
| sean-k-mooney | the `resouce:` syntx in the spec was specificly to be exlictvly based on placmente allcoation | 13:55 |
| amoralej | otherwise we need to check placement on instance notifications | 13:56 |
| sean-k-mooney | the placement allction only change in repsocne to api action like spawn or live migrate | 13:56 |
| sean-k-mooney | but yes they are not in the notifcaitons | 13:56 |
| amoralej | in my understanding, when discussing about cpu / mem / disk i was always thinking on the methods we already have | 13:56 |
| sean-k-mooney | so we woudl have to call placmenet on notificao | 13:56 |
| sean-k-mooney | amoralej: we had 2 diffent realted features | 13:57 |
| sean-k-mooney | one was supproting placeement allcaotion based resouces sorting | 13:57 |
| sean-k-mooney | `resouces:` constatins | 13:57 |
| sean-k-mooney | and the we had the concpe of a generic waying by usage | 13:57 |
| sean-k-mooney | evenully i think we will end up deprecating and removing the curren cpu/ram/disk methods | 13:58 |
| sean-k-mooney | in favor fo placmeent only beut that off topic | 13:58 |
| amoralej | yep | 13:59 |
| sean-k-mooney | https://opendev.org/openstack/watcher-specs/src/branch/master/specs/2026.2/approved/preemptible-instances.rst?display=source#L189-L201 | 13:59 |
| sean-k-mooney | that is what we are discssiong now and that is excpive based on placment allcoation | 13:59 |
| sean-k-mooney | not using any of the existign fields | 13:59 |
| amoralej | yep, that requires placement, right | 14:00 |
| sean-k-mooney | if we want to have a generic `size` that uses the exitng option we could btu that was not part of the spec dezie | 14:00 |
| amoralej | ack | 14:00 |
| sean-k-mooney | so size:cpu or usage:cpu | 14:01 |
| dviroel | it seems more feasible to get preemptible intances without including this, in this cycle | 14:01 |
| winiciusallan[m] | yeah. what you all suggest so we can make it mergeble until the end of the deadline? | 14:01 |
| dviroel | since it would require more discussion around the approach | 14:01 |
| * dviroel time check | 14:02 | |
| sean-k-mooney | winiciusallan[m]: move the resouce: related changes into its own patch at the end | 14:02 |
| sean-k-mooney | we will merge the feature with just age initally | 14:02 |
| sean-k-mooney | and then add resouce: with teh cdm and notifciaotn change slater | 14:02 |
| dviroel | winiciusallan[m]: let us know if you have any question while working on that | 14:02 |
| dviroel | you can ping us here in the irc | 14:03 |
| dviroel | this is the last feature in the list | 14:03 |
| dviroel | we are out of time now | 14:03 |
| winiciusallan[m] | ok. thanks!! | 14:03 |
| dviroel | note that morenod added some changes in the Review topic | 14:03 |
| morenod | just backports | 14:04 |
| morenod | nor urgents | 14:04 |
| dviroel | we can get into them afterwards | 14:04 |
| dviroel | morenod: ack, I was holding the backport yesterday due to another bugfix that was more critical | 14:04 |
| dviroel | but it got merged now | 14:04 |
| dviroel | and we can continue with our backports | 14:04 |
| dviroel | and also with your tempest changes | 14:04 |
| dviroel | but now we need to wrap up for today | 14:05 |
| dviroel | we will meet again next week | 14:05 |
| dviroel | thank you all for participating | 14:05 |
| dviroel | #endmeeting | 14:05 |
| opendevmeet | Meeting ended Thu Aug 20 14:05:34 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:05 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-08-20-13.00.html | 14:05 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-08-20-13.00.txt | 14:05 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-08-20-13.00.log.html | 14:05 |
| amoralej | thanks dviroel for chairing! | 14:05 |
| morenod | dviroel++ | 14:06 |
| opendevreview | Merged openstack/watcher-dashboard master: Add Playwright integration test for skip action workflow https://review.opendev.org/c/openstack/watcher-dashboard/+/976594 | 20:04 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!