| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add scenario tests for boot from volume instances https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/986616 | 08:00 |
|---|---|---|
| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add scenario tests for boot from volume instances https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/986616 | 08:29 |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add comprehensive tests for action precondition validation https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/966860 | 09:07 |
| opendevreview | Joan Gilabert proposed openstack/watcher master: Update configuration and integrations docs https://review.opendev.org/c/openstack/watcher/+/987129 | 09:54 |
| dviroel | hi all, watcher meeting will start soon (in 9 min) in this channel | 11:51 |
| dviroel | pls add your topics to our etherpad | 11:52 |
| dviroel | https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L31 | 11:52 |
| dviroel | winiciusallan[m]: around? we moved your topic to today's agenda, in case you want to cover something | 11:52 |
| winiciusallan[m] | dviroel: o/ yessir | 11:55 |
| winiciusallan[m] | dviroel: ack. will check | 11:55 |
| opendevreview | chandan kumar proposed openstack/watcher-tempest-plugin master: Replace ExtendPlacementClient with tempest lib https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/986622 | 11:59 |
| dviroel | #startmeeting watcher | 12:00 |
| opendevmeet | Meeting started Thu May 7 12:00:18 2026 UTC and is due to finish in 60 minutes. The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:00 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:00 |
| dviroel | hi all o/ | 12:00 |
| morenod | o/ | 12:00 |
| rlandy | o/ | 12:00 |
| winiciusallan[m] | o/ | 12:00 |
| chandank` | o/ | 12:00 |
| amoralej | o/ | 12:01 |
| dviroel | courtesy ping: jgilaber sean-k-mooney | 12:01 |
| dviroel | ok lets start, we have a full agenda today | 12:02 |
| dviroel | let's start with meeting agenda link | 12:02 |
| sean-k-mooney | o/ | 12:02 |
| dviroel | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L31 (Meeting agenda) | 12:02 |
| dviroel | ok, first one is from past meeting, that we didn't had time to cover | 12:03 |
| dviroel | #topic Preemptable Instances | 12:03 |
| dviroel | this is the feature that winiciusallan[m] is proposing for this cycle | 12:03 |
| jgilaber | o/ | 12:03 |
| dviroel | and we had some discussions at the ptg | 12:04 |
| dviroel | winiciusallan[m]: you want to cover something specific? it seems that you started some work there | 12:04 |
| dviroel | there is a initial spec proposal | 12:04 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher-specs/+/987171 | 12:04 |
| winiciusallan[m] | yeah. I started writing the specs and I'm currently working on the Proposed Change section | 12:05 |
| winiciusallan[m] | I have a question about the scope of this section. I mean, what is the level of specification? | 12:06 |
| winiciusallan[m] | I need to reiterate over this, but I was thinking about a new goal with a idea of "maximise the spare capacity usage" | 12:08 |
| winiciusallan[m] | I did not find any existing goal that met this | 12:08 |
| amoralej | the proposed change should describe how you plan to implement the goal of the spec | 12:08 |
| dviroel | winiciusallan[m]: you should provide in details how the solution will work, in this case yes, if no other goal fits, you should propose there the creation of a new goal, new strategy | 12:09 |
| amoralej | goal not as watcher goal, but as spec goal :) | 12:09 |
| winiciusallan[m] | got it | 12:09 |
| amoralej | yep, that, if you plan to create a new watcher goal, strategy, etc.. in that case explain what would those would look like in functional terms | 12:10 |
| amoralej | what would be the efficacy indicators for the goal, i.e., or what is expected to do for the new strategies | 12:10 |
| dviroel | yeah, there a plenty of details, but if more information is need on a specific point, we will probably ask you to include | 12:11 |
| winiciusallan[m] | ack. | 12:11 |
| dviroel | to make sure that we agree on the solution before you start implement it | 12:11 |
| winiciusallan[m] | about strategies, is here where we go for the different ways to preempt instances? oldest, newest, and so on... | 12:11 |
| amoralej | you can check other existing specs https://specs.openstack.org/openstack/watcher-specs/specs/pike/implemented/energy-saving-strategy.html i.e | 12:12 |
| dviroel | that would be part of the input parameters of your new strategy yes | 12:12 |
| winiciusallan[m] | from the docs, a strategy is the algorithm to find a solution to the goal | 12:12 |
| winiciusallan[m] | amoralej: ack. will take a look | 12:12 |
| sean-k-mooney | winiciusallan[m]: so in terms of the scope/detail fo the spec | 12:13 |
| dviroel | the proposed solution should be aligned with the use cases that you want to cover | 12:13 |
| sean-k-mooney | it shoudl have enough detail that some one familar with watcher could implemnet it if you could not | 12:13 |
| sean-k-mooney | but it does nto need to assume an absolute novice | 12:13 |
| sean-k-mooney | so it shoudl be written such that it capture all the key architeucal detail and highlihg the broad strocks fo the implemention | 12:14 |
| sean-k-mooney | but dont assume you need to expalin how the exsing watcher compoents work | 12:14 |
| dviroel | ++ | 12:14 |
| winiciusallan[m] | ok, that makes sense to me | 12:15 |
| winiciusallan[m] | dviroel: when you said about parameters of strategies, this is passed within the request or is it another resource? | 12:16 |
| dviroel | winiciusallan[m]: just in case you want to clarify something while writing it, you can just ping us here in the channel too | 12:16 |
| winiciusallan[m] | perfect. I won't take too much time since I'm still at the start of the writing | 12:16 |
| winiciusallan[m] | I would kindly ask for a revision in the initial sections when you guys have some time | 12:17 |
| dviroel | winiciusallan[m]: is part of the strategy spec, and are the parameters provided when creating the audit | 12:17 |
| dviroel | winiciusallan[m]: but feel free to ask question at any time | 12:18 |
| dviroel | and folks can start already looking at you spec when they have some time | 12:18 |
| dviroel | winiciusallan[m]: ok if we move on? or anything that you want to highlight? | 12:19 |
| winiciusallan[m] | no, that's all from my side. thank you all! | 12:19 |
| dviroel | thanks for the topic | 12:19 |
| dviroel | #topic Possible solutions for https://bugs.launchpad.net/watcher/+bug/2141951 | 12:19 |
| dviroel | this is a bug reported by amoralej | 12:19 |
| dviroel | "Audits with workload_stabilization strategy takes very long to execute" | 12:19 |
| dviroel | which i started to debug | 12:20 |
| dviroel | and have some outcomes to share | 12:20 |
| dviroel | I noticed when running some tests in my (simulated) environment that the locking process is causing a big bottleneck | 12:20 |
| dviroel | and it was really mentioned by amoralej in the LP | 12:20 |
| dviroel | and by only removing the locks during strategy execution, it reduce the execution time around 85% in my env (tests were running with 113 nodes and 15k instances) | 12:21 |
| dviroel | i was using the emulators that amoralej is proposing here: https://review.opendev.org/c/openstack/watcher/+/980257 btw | 12:21 |
| dviroel | again amoralej++ | 12:21 |
| dviroel | I proposed some DNM patches yesterday | 12:22 |
| dviroel | and there was a previous chat here in the channel yesterday, about the current locking mechanism used in models | 12:22 |
| sean-k-mooney | dviroel: we may want to look for that partern in other places | 12:22 |
| sean-k-mooney | i.e. where we ahve shared locks on copied obejcts | 12:22 |
| dviroel | sean-k-mooney: yes, it seems that we need to dig more | 12:23 |
| dviroel | so, the first patch just disables de locking when we get a copy of the CDM | 12:23 |
| dviroel | and after discussing about the usage of oslo.concurrency in these classes, we realized that it doesn't seems to be the better approach for CDM | 12:23 |
| dviroel | a second patch was proposed by replacing synchronized decorator with a instance Lock instead | 12:24 |
| dviroel | this was based on sean-k-mooney proposal form yesterday ^ | 12:24 |
| dviroel | and the result looks good too, close to the patch that it removing the lock (at least in my env) | 12:24 |
| dviroel | amoralej: do you have some results to share too? | 12:25 |
| amoralej | i tested running an audit with both unpatched and your two patches, i added my results in the etherpad. I will also run it with a different model | 12:25 |
| amoralej | in my tests, execution time dropped from ~ 18mins to ~ 10mins | 12:26 |
| dviroel | you had better results with threading lock that no locks:) | 12:26 |
| amoralej | yes, that's surprising | 12:26 |
| amoralej | i want to repeat the tests anyway to get more datapoint | 12:26 |
| dviroel | i see that you had ~50% of improvement | 12:26 |
| amoralej | yep | 12:27 |
| dviroel | of course that it depends where you compare the start and end time | 12:27 |
| dviroel | we can sync on that later I think | 12:27 |
| amoralej | I'll keep updating when i have more tests | 12:27 |
| dviroel | the point that I want to raise is | 12:27 |
| amoralej | sure, i take the time from the audit metadata "created at" and "Updated at" | 12:27 |
| amoralej | i was not runnina anything else in watcher, so the audit execution is triggered right after created | 12:28 |
| dviroel | i want to propose improvements/fixes to this strategy, and it should be a fix that we can backport too (at least part of it) | 12:28 |
| amoralej | definitively, improving locking is something that deserves time to dig into | 12:29 |
| dviroel | yeah | 12:29 |
| dviroel | so one thing is to improve the locking | 12:29 |
| dviroel | which will also benefit other strategies in the end | 12:30 |
| amoralej | yep | 12:30 |
| dviroel | and another propopsal is to include additional parameters that allows us to control the number of source nodes and instances evaluated | 12:30 |
| dviroel | today there is a parameter to control te amount of destination nodes evaluated | 12:31 |
| dviroel | which is by default 1 (random) | 12:31 |
| dviroel | by all nodes are considered as source nodes and all instances evaluated for migration | 12:31 |
| dviroel | controling these would also affect the execution time | 12:31 |
| dviroel | as also affect the resulted improvement proposed by the strategy | 12:32 |
| amoralej | actually, i think your idea of "sort the list of hosts based on load and select the most loaded ones" is good | 12:32 |
| dviroel | oh yeah | 12:32 |
| amoralej | but i don't think we need a parameter | 12:32 |
| dviroel | i forgot about that one | 12:32 |
| sean-k-mooney | dviroel: im actully not surpised by that | 12:32 |
| amoralej | i think we can improve the algorithm to be more effective | 12:32 |
| dviroel | today it evaluate all nodes, so they are not sorted by the most loaded | 12:33 |
| sean-k-mooney | so with the conditional locks you added a branch poirnt that the cpus branch predictor can get wrong | 12:33 |
| sean-k-mooney | we per object locks we alwasy aquire the lock and sicne the object is not shared we alwasy get it imedetaly | 12:33 |
| sean-k-mooney | so that shoudl be faster | 12:33 |
| dviroel | sean-k-mooney: makes sense, i didn't have the same result in my env in the end, but it was very similar | 12:34 |
| dviroel | amoralej: so we may want to discuss more, but there are 2 improvements here in the end, the lock system and inside the algorithm | 12:35 |
| amoralej | we may 1 "take a short (size tbd) list of the most loaded nodes as potential sources", 2. take a short list of the less loaded nodes as potential destinations. And simulate resulting load with migrations between them. If the resulting std deviation is below threashold, we are done. If none of the short list of destination is valid for the migrations take the following set of less loaded, etc... | 12:35 |
| amoralej | wrt the algorithm ^ | 12:35 |
| sean-k-mooney | so that almost feels like a diffent stragey | 12:36 |
| sean-k-mooney | but perhaps | 12:36 |
| dviroel | amoralej: iiuc your proposal, it would require more refactoring in the strategy | 12:36 |
| amoralej | not in the sense that it's based on standard deviation | 12:36 |
| sean-k-mooney | right but the stragey is kind fo ment to defein how | 12:37 |
| sean-k-mooney | anyway that is one approch | 12:37 |
| dviroel | ok, to close this topic, i will keep you folks updated in this channel | 12:37 |
| amoralej | I'd need to have a more clear idea about the required changes in the algorithm to have an opinion if another strategy is required tbh | 12:38 |
| dviroel | wrt to the locking issue | 12:38 |
| dviroel | and i will propose another patch to explain the second improvement wrt to limiting the amount of nodes and instance to be analyzed in the alg | 12:38 |
| dviroel | i think that is easier to get some feedback on that | 12:39 |
| dviroel | the question would be: proposing new input_parameter, would be a solution that we want to backport too? | 12:39 |
| dviroel | as part of the fix? | 12:39 |
| dviroel | i will let you answer async, so we move on to next topics | 12:40 |
| amoralej | i'd try to improve it withouth additional parameter, not because of backportability but because i feel we could get a net win | 12:40 |
| dviroel | amoralej: ack, can be multiple patches, some of them not suitable for backporting | 12:41 |
| dviroel | #topic https://review.opendev.org/c/openstack/watcher/+/986486 | 12:41 |
| dviroel | amoralej: ^ | 12:41 |
| dviroel | this is the BFV fix | 12:41 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/986486 (Fix disk accounting for BFV instances) | 12:41 |
| amoralej | just an update on that, I've fixed the tempest test to increase coverage as requested by sean-k-mooney | 12:42 |
| dviroel | ack, if tempest test are passing, i think that we are good to merge the fix | 12:42 |
| amoralej | https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/986616 has now all the tests, including end to end execution, and including ephemeral and swap both for bfv and for boot form image | 12:42 |
| dviroel | i will take a look after the meeting | 12:42 |
| dviroel | ++ | 12:43 |
| dviroel | sean-k-mooney: ptal when you have some time then, tks | 12:44 |
| sean-k-mooney | sure | 12:44 |
| sean-k-mooney | ill review it before i end my day | 12:44 |
| dviroel | anyone wants to raise any concern about amoralej fix/patches? | 12:44 |
| dviroel | if not, we can move on | 12:44 |
| sean-k-mooney | i really wish we had functional test for those type of changes but ya that out of scope for now | 12:44 |
| dviroel | #topic Reviews | 12:45 |
| dviroel | 1. First proposal for the datamodel api freeze | 12:45 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/986777 | 12:45 |
| dviroel | i started reviewing it | 12:45 |
| amoralej | thanks dviroel | 12:45 |
| dviroel | amoralej: are you also including some doc updates right? | 12:45 |
| amoralej | btw, it requires https://review.opendev.org/c/openstack/python-watcherclient/+/986830 to unblock the python-watcherclient-functional job | 12:46 |
| amoralej | yes, i included the docs updates we discussed in ptg | 12:46 |
| sean-k-mooney | s the api ref update is not exactyly what i would expect | 12:46 |
| dviroel | it is important for the team to look at this, since we are documenting the procedure on adding/removing fields to the CDM | 12:46 |
| sean-k-mooney | microveriosn are what guarentte the field of a repsonce never change (unless you use a new one) | 12:46 |
| dviroel | we need to agree on the proposal before merging ^ | 12:46 |
| amoralej | doc changes are mainly in https://review.opendev.org/c/openstack/watcher/+/986777/4/doc/source/contributor/plugin/cdmc-plugin.rst | 12:46 |
| sean-k-mooney | what we really need in https://review.opendev.org/c/openstack/watcher/+/986777/4/api-ref/source/watcher-api-v1-datamodel.inc | 12:47 |
| sean-k-mooney | is a note to say that this api will not be modifed in future microversion to incldue new fileds | 12:47 |
| dviroel | ack, so thanks for proposing it amoralej | 12:48 |
| amoralej | feel free to propose the wording that you think is more appropiate in the patch | 12:48 |
| sean-k-mooney | amoralej: i would also say that "dding a new field to a model element does not require a | 12:48 |
| sean-k-mooney | spec, a blueprint or a new API microversion." | 12:49 |
| sean-k-mooney | is not correct or soemthing we agreed too | 12:49 |
| amoralej | we did | 12:49 |
| sean-k-mooney | no we did not | 12:49 |
| amoralej | didn't we? | 12:49 |
| amoralej | in PTG? | 12:49 |
| sean-k-mooney | we said that we will not expose new filed in new microverions | 12:49 |
| sean-k-mooney | we did not say that you can add them without a spec or blueprint | 12:49 |
| amoralej | we did | 12:49 |
| amoralej | i was very explicit about that | 12:49 |
| sean-k-mooney | im -2 on that | 12:50 |
| amoralej | actually, i remember i even put examples | 12:50 |
| sean-k-mooney | we can dicuss if addign wone is approate in a bug | 12:50 |
| sean-k-mooney | but we need to have a discuss in a bug blueprint or spec for each addtion | 12:50 |
| jgilaber | IIRC we agreed that only adding the fields would not require a spec, but the specs would focus on the features they would be used on | 12:50 |
| amoralej | the change that requires the new fields may need a spec or blueprint, but not the new field itself | 12:50 |
| amoralej | exaclty | 12:50 |
| dviroel | we agreed that new fields could not require a spec, but the usage of them may require one. | 12:50 |
| dviroel | didn't remember anything about blueprint for instance | 12:51 |
| sean-k-mooney | right ahtat is very diffent then what that say | 12:51 |
| sean-k-mooney | the current text implies you coudl add a new feils as part of any patch | 12:51 |
| sean-k-mooney | without it being part of a large feature addtion that is tracked and apprved elsewhere | 12:52 |
| amoralej | i disagree | 12:52 |
| amoralej | every change in watcher requires a bug, spec or blueprint | 12:52 |
| sean-k-mooney | almost all yes | 12:52 |
| dviroel | almost all | 12:52 |
| amoralej | so anything that requires a new field will require ^ | 12:52 |
| sean-k-mooney | but bugfixes in general shoudl not make datamodel chnages | 12:52 |
| dviroel | it will require at least a launchpad bug | 12:52 |
| sean-k-mooney | right and the addtion of a new value in a bug fix may make that bugfix master only | 12:53 |
| dviroel | this is the case of the strategy that interacts with nova tpday | 12:53 |
| amoralej | it may in some cases, we don't want to close that case | 12:53 |
| amoralej | exactly | 12:53 |
| dviroel | i think that we agree | 12:53 |
| sean-k-mooney | i dont disagree with the last sentenc ein the paragraph A new field can therefore be introduced | 12:54 |
| amoralej | anyway, i'll be happy to make the text more clear, propose in the review please | 12:54 |
| sean-k-mooney | as part of the feature or bug fix that requires it. | 12:54 |
| dviroel | so rephrase the doc may be needed, i will also check | 12:54 |
| sean-k-mooney | but the into to the adding a new field goes beyond what we agreed | 12:54 |
| sean-k-mooney | i can propsoe a rephasing | 12:54 |
| dviroel | or even better to not mention anything XD | 12:54 |
| dviroel | ack, lets continue the discussion in the patch | 12:54 |
| dviroel | but again, thanks amoralej for proposing the patch | 12:55 |
| dviroel | we have time to cover one more review | 12:55 |
| dviroel | 2. Remove unused methods from KeystoneHelper | 12:55 |
| jgilaber | this one should be really quick | 12:55 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/987126 | 12:55 |
| jgilaber | I just wanted to mention that I proposed a series of patches to drop keystonclient in favour of the sdk | 12:56 |
| jgilaber | the linked patch is the first of the series | 12:56 |
| dviroel | ack, so we started with keystone, great | 12:56 |
| * dviroel started this cycle | 12:56 | |
| jgilaber | this is not urgent at all, but it's also a light review since the first patch removes most of the methods of the keystone helpler | 12:56 |
| jgilaber | since they were unused | 12:56 |
| amoralej | i started reviewing but need to double check all the config changes, which is likely the trickier | 12:57 |
| amoralej | all the deprecation and overriding | 12:57 |
| jgilaber | I think that is all that I wanted to mention | 12:57 |
| dviroel | it is important to get reviews earlier so you can move on with the following | 12:57 |
| dviroel | jgilaber: thanks | 12:57 |
| dviroel | any concerns wrt to jgilaber patch? | 12:57 |
| dviroel | winiciusallan[m]: btw, feel free to join us in the reviews XD - all opinions are very welcome | 12:58 |
| dviroel | you folks have any other review to cover? | 12:58 |
| dviroel | we have one last minute | 12:59 |
| dviroel | so i will move the bugs to the next meeting | 12:59 |
| dviroel | note that thre is one new bug from morenod | 12:59 |
| dviroel | feel free to take a look in advance | 13:00 |
| dviroel | #topic Volunteers to chair next meeting | 13:00 |
| dviroel | someone wants to? | 13:00 |
| jgilaber | I can do it | 13:00 |
| dviroel | 14th i will be around to chair if needed too | 13:00 |
| dviroel | ack, tks jgilaber | 13:00 |
| dviroel | let's wrap up for today | 13:00 |
| dviroel | we will meet again next week | 13:01 |
| dviroel | thank you all for participating | 13:01 |
| amoralej | thanks dviroel++ for chairing! | 13:01 |
| dviroel | #endmeeting | 13:01 |
| opendevmeet | Meeting ended Thu May 7 13:01:11 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-07-12.00.html | 13:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-07-12.00.txt | 13:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-05-07-12.00.log.html | 13:01 |
| * dviroel gets a coffee | 13:02 | |
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Add comprehensive tests for action precondition validation https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/966860 | 13:59 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!