| opendevreview | Takashi Kajinami proposed openstack/devstack-plugin-prometheus master: Use built-in utilities to manage services https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1011817 | 07:53 |
|---|---|---|
| opendevreview | David proposed openstack/watcher-tempest-plugin master: Use 'oc exec' instead of 'oc rsh' for podified commands https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/1011845 | 12:37 |
| dviroel | hi folks o/ - reminder: watcher meeting starts in 5 min | 12:55 |
| dviroel | #startmeeting watcher | 13:01 |
| opendevmeet | Meeting started Thu Oct 22 13:01:10 2026 UTC and is due to finish in 60 minutes. The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| opendevmeet | The meeting name has been set to 'watcher' | 13:01 |
| dviroel | hi there folks o/ | 13:01 |
| morenod | o/ | 13:01 |
| dviroel | who is around oday? | 13:01 |
| jgilaber | o/ | 13:01 |
| chandankumar | / | 13:02 |
| dviroel | courtesy ping: amoralej sean-k-mooney rlandy | 13:02 |
| dviroel | let's start with today's meeting agenda | 13:02 |
| amoralej | o/ | 13:02 |
| dviroel | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L28 (Meeting agenda) | 13:02 |
| dviroel | feel free to add your own topics to the agenda | 13:03 |
| dviroel | i added a couple of topics | 13:03 |
| dviroel | let us know if you want to discuss any other relevant topic | 13:03 |
| dviroel | lets start with | 13:03 |
| sean-k-mooney | o/ | 13:03 |
| dviroel | #topic PTG follow up | 13:03 |
| dviroel | so we had a full week of meetings | 13:04 |
| dviroel | across different projects | 13:04 |
| dviroel | watcher's sessions were on Wed and Thursday | 13:04 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-2027.1-ptg | 13:04 |
| dviroel | we covered all topics originally scheduled | 13:05 |
| dviroel | and we end up with a new list of Action Items | 13:05 |
| dviroel | some of them are carry over, but we made a good progress during the cycle I think | 13:05 |
| dviroel | so, i will ask you folks to review that list | 13:05 |
| dviroel | and be a volunteer in the ones that have no assignee :) | 13:05 |
| dviroel | or just put yout name in the ones that you already started working, but your name is missing there | 13:06 |
| dviroel | #link https://etherpad.opendev.org/p/watcher-2027.1-ptg#L76 | 13:06 |
| dviroel | ^ the current starting line | 13:06 |
| dviroel | as a follow up, we should continue working on the proposed features and enhancements | 13:07 |
| dviroel | updating our in-progress patches and specs | 13:07 |
| dviroel | when time permits :) | 13:07 |
| dviroel | i also sent a ptg summary to the ML recently | 13:08 |
| * dviroel search for the link | 13:08 | |
| dviroel | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/KSKWNOI2FXVECPQTLPDYJ5BEOIPI5BVE/ (Indri PTG Summary) | 13:09 |
| dviroel | anything else that you folks want to highlight about the vPTG? | 13:09 |
| dviroel | I added some topics to the meeting, more about the features that I have been working on | 13:10 |
| dviroel | I don't think that we need to cover each of the action item too... | 13:10 |
| dviroel | we can do this offline and raise any concern here afterwards | 13:11 |
| dviroel | any other comment before we move to the next topic? | 13:11 |
| dviroel | ack | 13:12 |
| dviroel | #topic AZ Drain Feature follow up | 13:12 |
| dviroel | I proposed some updates to the spec, after PTG discussions | 13:12 |
| dviroel | we discussed most of the details at the ptg | 13:13 |
| dviroel | I reflected some of the conclusions there, but maybe not all, not sure | 13:13 |
| dviroel | one thing that I still conflicted is | 13:13 |
| dviroel | the reuse of zone_migration VS a new zone draining strategy | 13:13 |
| dviroel | atm, the spec mention both ways as a possibilitu | 13:14 |
| dviroel | in case we go with a new strategy, i still think that we need to fix some design decisions in zone_migration | 13:15 |
| dviroel | which makes me think if we should have 2 | 13:15 |
| dviroel | the parallel_* paremeters for example | 13:15 |
| dviroel | which requires user to provide a value on every audit created, since default is a very low value set | 13:16 |
| dviroel | I also included that in the spec | 13:16 |
| jgilaber | overall I feel like working on a new strategy for the AZ use case will be simpler | 13:17 |
| jgilaber | we've had many discussion about zone migration because many of the parameters are not obvious | 13:18 |
| jgilaber | and while we've improved a lot with the doc changes we did, it's not simple to reason about | 13:18 |
| amoralej | but what would cover the new one? | 13:19 |
| amoralej | only the az case? | 13:19 |
| dviroel | a new strategy that would replace the zone_migration one? or just the AZ? | 13:19 |
| dviroel | maybe an user would like to migrate between 2 nodes too | 13:19 |
| dviroel | then they would need to use zone_migration | 13:19 |
| jgilaber | I'm thinking of a new one for the AZ case and keeping zone migration for the existing use cases of nodes | 13:20 |
| morenod | if the mechanism is the same, get a vm, select a destination, and migrate, for me, az migration is just a subset of a zone migration | 13:20 |
| dviroel | yeah, that's an option. We would still need to support zone_migration and provide some improvements.. | 13:20 |
| dviroel | in case of a new strategy ^ | 13:20 |
| dviroel | so the new one can just be the AZ case | 13:21 |
| dviroel | or: a copy of zone_migration (nodes and storage) + AZ | 13:21 |
| amoralej | i like having a single one for all | 13:22 |
| dviroel | and in the future we would deprecate the zone_migration | 13:22 |
| amoralej | so, modifying the existing one, or new one + deprecation | 13:22 |
| dviroel | there is the 3rd option of maintaining the 2, wher the new one just covers AZ + storage pools/retypes | 13:23 |
| dviroel | which I am not sure if makes sense to keep 2 in the end | 13:23 |
| dviroel | if we also consider that we plan to improve the upgrade process during the release, | 13:23 |
| dviroel | we could stay with zone_migration and provide deprecations to the parameters that we don't want to maintain too | 13:24 |
| dviroel | as also change some defaults | 13:24 |
| dviroel | or rename some parameters | 13:24 |
| amoralej | yes, that'd be my preference unless there is strong justification against | 13:25 |
| dviroel | I see an advantage on keep working on zone_migration | 13:25 |
| dviroel | which is that we can deliver enhancements is small pieces | 13:25 |
| amoralej | but, if others think doing a different one, it can also make sense, both options are doable and has their advantadges | 13:25 |
| dviroel | we can already propose small patches to improve zone_migration | 13:26 |
| jgilaber | changing the default will be easy I think from an upgrade pov, but renaming parameters will be harder I think | 13:26 |
| dviroel | renaming isn't also so critical, if we provide a good documentation | 13:27 |
| amoralej | we can mantain supporting both names | 13:27 |
| dviroel | yep | 13:27 |
| amoralej | i think we discussed that some time ago? | 13:27 |
| dviroel | we didd | 13:27 |
| jgilaber | yes I started a spec for that | 13:27 |
| jgilaber | but I could not finish in time and abandoned it | 13:28 |
| dviroel | we should still be able to get the gerrit patch if you pushed it | 13:28 |
| amoralej | no need to do it now, but my point was that we could do that when we want to | 13:28 |
| amoralej | as a separated task | 13:28 |
| dviroel | yep | 13:28 |
| jgilaber | it was this one https://review.opendev.org/c/openstack/watcher-specs/+/965943 | 13:29 |
| dviroel | thanks jgilaber - it is interesting to review what we discussed at that time yeah | 13:29 |
| dviroel | i will check again later | 13:29 |
| dviroel | so to not take more time in this topic | 13:29 |
| dviroel | would you folks take a look at the spec, and place your opinions? | 13:30 |
| dviroel | lets see if we agree on the working tasks | 13:30 |
| jgilaber | sure will do, probably next week | 13:30 |
| dviroel | and move forward | 13:31 |
| dviroel | thanks jgilaber | 13:31 |
| dviroel | any other comment around that topic? | 13:31 |
| dviroel | next one is | 13:31 |
| dviroel | #topic Decision for https://review.opendev.org/c/openstack/watcher/+/988348 | 13:32 |
| dviroel | the "Add least_loaded host_choice for workload_stabilization strategy" patch has a discussion around one of the new configuration parameters | 13:32 |
| dviroel | which is proposed as: least_loaded_percent | 13:33 |
| amoralej | so, the only open question is about least_loaded_percent, right? | 13:33 |
| dviroel | yes | 13:33 |
| dviroel | this would be a list of nodes that the strategy would pick as possible destination node | 13:33 |
| amoralej | the options are making it a configurable percentage of the computes or dynamically the list of nodes with usage below mean ? | 13:34 |
| amoralej | and get rid of the parameter? | 13:34 |
| dviroel | yeah, the second option (which amoralej proposed) would remove the parameter I think | 13:34 |
| amoralej | which one you prefer? | 13:34 |
| dviroel | i still prefer a configurable one | 13:35 |
| dviroel | but other reviwers must also agree :) since I need +2s | 13:35 |
| dviroel | lol | 13:35 |
| amoralej | I personally like having as less parameters as possible if we can maintain a good behavior | 13:35 |
| amoralej | but i won't block | 13:35 |
| jgilaber | I need to review again, it's been a while | 13:36 |
| dviroel | once we define one way, I don't think we can revert that too | 13:36 |
| jgilaber | but at first glance a percentage seems easier to understand than the mean | 13:36 |
| dviroel | if we go with below mean, is one thing, if we set the percentage of least loaded is another | 13:37 |
| amoralej | for me the advantadge of dynamic vs config value is that the list will always contain nodes that should always improve the sd | 13:37 |
| jgilaber | in the sense that for a user it will be easier to have an intuition of what the strategy will do | 13:37 |
| amoralej | well, the user wouldn't need to understand it, acutally :) | 13:37 |
| amoralej | i mean, he has no option to configure or decide | 13:38 |
| amoralej | just let the strategy do the work for him :) | 13:38 |
| dviroel | they would just accept that yeah | 13:38 |
| dviroel | not sure if they will really use the percentage in the end, or just always go with the default | 13:38 |
| amoralej | there may be cases where i.e. only 25% of the compute nodes are in below the mean, i.e. case where new nodes have been added | 13:38 |
| amoralej | that's what i foresee dviroel :) | 13:39 |
| amoralej | they will stick to 50% | 13:39 |
| dviroel | if they use a smaller scope | 13:39 |
| dviroel | they may want to switch that to 100% sometimes? to consider all | 13:40 |
| amoralej | but there is no much point in sending a vm from the top loade host to the 2nd top loaded host | 13:40 |
| amoralej | from a functional point of view | 13:40 |
| dviroel | i always think that for big environment, user would take the advantage of reducing the scope for a set of nodes | 13:40 |
| dviroel | yep | 13:41 |
| amoralej | the optimal behavior to balance is to move vms from the top loaded hosts to the least loaded ones | 13:41 |
| dviroel | event within an smaller scope | 13:41 |
| dviroel | s/event/even | 13:41 |
| amoralej | there may be corner cases | 13:41 |
| amoralej | but actually, as we are setting least_loaded_count to 3 | 13:42 |
| amoralej | the strategy should behave well with corner cases | 13:42 |
| dviroel | i have amoralej opinion already, let me know jgilaber and sean-k-mooney which you folks think is the best | 13:42 |
| amoralej | i.e. if the least loaded host is a small server and the vm to be migrated is big | 13:42 |
| amoralej | or very big | 13:42 |
| dviroel | right | 13:43 |
| amoralej | it may happen that moving it would not improve | 13:43 |
| dviroel | or not even possible | 13:43 |
| amoralej | but actually, iirc workload_stabilization simulates several scenarios | 13:43 |
| sean-k-mooney | sorry got distracted by a ping | 13:43 |
| dviroel | workload_stabilization simulates moving all instances from source_nodes and what we want is to reduce de destination nodes analyzed | 13:43 |
| dviroel | sean-k-mooney: np | 13:44 |
| sean-k-mooney | ill reply in gerrit on the value | 13:44 |
| dviroel | thanks sean-k-mooney | 13:44 |
| dviroel | i will ask the same for amoralej and jgilaber | 13:44 |
| dviroel | so we can move with the patch | 13:44 |
| dviroel | i already took most of the meeting :P | 13:45 |
| amoralej | sean-k-mooney, note it's not only about the value, but about the approach | 13:45 |
| sean-k-mooney | 3 is probaly fine but i need to load context again | 13:45 |
| jgilaber | sure, I need to do a full review | 13:45 |
| dviroel | sean-k-mooney: no, it is about the percentage vs below mean vs fixed value | 13:45 |
| sean-k-mooney | ya im not sure we shoudl use the percentage | 13:45 |
| jgilaber | so if you have a consensus before I reviewed don't hold the patch for me | 13:45 |
| dviroel | jgilaber: ack | 13:46 |
| sean-k-mooney | do we have an efficet way to caluate the mean without looign over all teh nodes | 13:46 |
| amoralej | I'd say the strategy calculates the mean at the beginning when checking if the cluster is above threshold? it calculates sd, but i'd say also mean? | 13:47 |
| sean-k-mooney | promethious proably could do it efficnly in one queyy i just dont htink we shoudl be leaking this stragey detail through the datastore abstraction | 13:47 |
| sean-k-mooney | well my point is to calucalte the mean ourselve we have to loop over al lthe values and sum them then device | 13:48 |
| sean-k-mooney | *devide | 13:48 |
| sean-k-mooney | but that alsomot as much work as checkign all the hosts | 13:48 |
| sean-k-mooney | that why i was leaning towrad only haveing a count | 13:49 |
| amoralej | https://github.com/openstack/watcher/blob/master/watcher/decision_engine/strategy/strategies/workload_stabilization.py#L376-L390 | 13:49 |
| sean-k-mooney | really waht we want is a min heap | 13:50 |
| amoralej | we can finish the discussion in the gerrit anyway :) | 13:50 |
| sean-k-mooney | where we can alwasy just get the least loaded host by takign the top node form the heap, update its value with the new workload adn let it do it | 13:50 |
| amoralej | let's cover other topics as dviroel said | 13:50 |
| sean-k-mooney | ack | 13:50 |
| dviroel | yeah | 13:50 |
| amoralej | but yeah, that'd also work | 13:50 |
| dviroel | #topic Reviews | 13:51 |
| dviroel | first one | 13:51 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/1011845 | 13:51 |
| dviroel | Use 'oc exec' instead of 'oc rsh' for podified commands | 13:51 |
| morenod | yep, thats me | 13:52 |
| dviroel | so morenod propose a change on how we run the commands inside prometheus containers | 13:52 |
| morenod | testing rhoso19, injecting metrics with promtool is failing because of that | 13:52 |
| dviroel | so we probably can get some feedback from RDO job on this | 13:52 |
| morenod | its seems that the answer is longer now, and using oc rsh trunks the output, and oc exec do not | 13:52 |
| dviroel | ack, thanks morenod i will take a look on it | 13:53 |
| morenod | thanks | 13:53 |
| dviroel | mostly the outputs | 13:53 |
| dviroel | thanks! | 13:53 |
| sean-k-mooney | hum | 13:53 |
| sean-k-mooney | i guess that is fine | 13:54 |
| dviroel | next one | 13:54 |
| dviroel | #link https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1010447 | 13:54 |
| sean-k-mooney | exec also does nto allcoate a interact tty | 13:55 |
| sean-k-mooney | so exec is more for running command | 13:55 |
| sean-k-mooney | where as rsh is for an actual shell | 13:55 |
| sean-k-mooney | so it proably better long term anyway | 13:55 |
| dviroel | chandankumar: this is an action item from ptg right, thanks for proposing | 13:55 |
| chandankumar | yes, we got vote from telemtry team | 13:55 |
| dviroel | shouldn't be added to the gate too? | 13:56 |
| chandankumar | We have not added any other project job in gate | 13:56 |
| sean-k-mooney | yes the general role is if votign in check it shoudl be in gate | 13:56 |
| sean-k-mooney | *ruel | 13:56 |
| jgilaber | +1, we should mirror the check jobs in gate | 13:57 |
| chandankumar | I will propose a follow up patch to add them in gate | 13:57 |
| sean-k-mooney | why not jsut update this? | 13:57 |
| dviroel | yeah | 13:57 |
| dviroel | should be quick | 13:57 |
| chandankumar | ok | 13:57 |
| chandankumar | let me do that | 13:57 |
| dviroel | tks chandankumar | 13:57 |
| sean-k-mooney | watcher should also be in gate | 13:57 |
| jgilaber | slightly off-topic, I just noticed I have +2 in that repo, is the watcher-core team carried there? | 13:57 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/993543 | 13:58 |
| sean-k-mooney | yes | 13:58 |
| chandankumar | jgilaber: yes | 13:58 |
| sean-k-mooney | we co maintiane it with telemery | 13:58 |
| dviroel | "Add Prometheus emulator and workload_stabilization functional tests" | 13:58 |
| jgilaber | ack, TIL :) thanks | 13:58 |
| * dviroel needs to review these emulators patches | 13:58 | |
| amoralej | yes, i sent a new PS with the change we discussed in ptg | 13:58 |
| dviroel | and the other one | 13:59 |
| dviroel | #link https://review.opendev.org/c/openstack/watcher/+/993634 | 13:59 |
| dviroel | amoralej: nice, ty | 13:59 |
| amoralej | metrics are now a dict in the compute and instance objects | 13:59 |
| sean-k-mooney | ack i see you moved the metrics in to there own dict | 13:59 |
| dviroel | I will add them to my review list | 13:59 |
| dviroel | ++ | 13:59 |
| amoralej | yep | 13:59 |
| sean-k-mooney | i think htat was the only real issue i had so shoudl be good to proceed with now | 13:59 |
| opendevreview | chandan kumar proposed openstack/devstack-plugin-prometheus master: Added aetos job to avoid breaking aetos https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1010447 | 13:59 |
| dviroel | ok, we are out of time already | 14:00 |
| dviroel | winiciusallan[m]: around? | 14:00 |
| dviroel | winiciusallan[m]: ok if we cover your topic next week? | 14:00 |
| sean-k-mooney | chandankumar: you didnt add telemetry-dsvm-integration-aetos to gate | 14:00 |
| dviroel | will you be around? | 14:00 |
| opendevreview | chandan kumar proposed openstack/devstack-plugin-prometheus master: Added aetos job to avoid breaking aetos https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1010447 | 14:00 |
| dviroel | tl;dr; is that winiciusallan[m] plans to update its spec for preemtible instances for the current cycle | 14:01 |
| dviroel | we may discuss the details next week then | 14:01 |
| sean-k-mooney | cool | 14:01 |
| chandankumar | sean-k-mooney: done | 14:01 |
| dviroel | thanks winiciusallan[m] o/ | 14:01 |
| dviroel | let's wrap up for today | 14:01 |
| dviroel | oh | 14:01 |
| sean-k-mooney | i think the only open we had thre was either adding the placmenet info or droping resouce: | 14:01 |
| dviroel | #topic Volunteers to chair next week | 14:01 |
| winiciusallan[m] | sorry, didn't receive the notification :( | 14:01 |
| winiciusallan[m] | yeah, I'm ok on moving this to next week | 14:01 |
| winiciusallan[m] | thank you guys | 14:01 |
| dviroel | someone wants to chair next week? | 14:02 |
| dviroel | the 29th? | 14:02 |
| dviroel | otherwise I am able to chair np | 14:02 |
| jgilaber | I can do it | 14:02 |
| dviroel | thanks jgilaber | 14:02 |
| dviroel | winiciusallan[m]: np thanks! | 14:02 |
| dviroel | we will meet again next week | 14:02 |
| dviroel | thank you all for participating | 14:02 |
| dviroel | #endmeeting | 14:02 |
| opendevmeet | Meeting ended Thu Oct 22 14:02:53 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-10-22-13.01.html | 14:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-10-22-13.01.txt | 14:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-10-22-13.01.log.html | 14:02 |
| opendevreview | Merged openstack/devstack-plugin-prometheus master: Added aetos job to avoid breaking aetos https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1010447 | 16:08 |
| opendevreview | Takashi Kajinami proposed openstack/devstack-plugin-prometheus master: Use built-in utilities to manage services https://review.opendev.org/c/openstack/devstack-plugin-prometheus/+/1011817 | 16:34 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Audit Pipeline Object and DBAPI https://review.opendev.org/c/openstack/watcher/+/996849 | 17:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Add AuditPipeline lifecycle notifications https://review.opendev.org/c/openstack/watcher/+/999891 | 17:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Add AuditPipeline execution handler and cascade execution mode https://review.opendev.org/c/openstack/watcher/+/1000031 | 17:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Add efficacy indicators for Audit Pipeline action plans https://review.opendev.org/c/openstack/watcher/+/1001009 | 17:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Add Audit Pipeline REST API endpoints https://review.opendev.org/c/openstack/watcher/+/997427 | 17:09 |
| opendevreview | Douglas Viroel proposed openstack/watcher master: Consolidate strategy metric-query params behind get_ methods https://review.opendev.org/c/openstack/watcher/+/1002694 | 17:09 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!