Thursday, 2026-11-12

opendevreviewchandan kumar proposed openstack/watcher-dashboard master: Fix HTTP 500 on the Audits page in non-English languages  https://review.opendev.org/c/openstack/watcher-dashboard/+/101287503:48
opendevreviewchandan kumar proposed openstack/watcher-dashboard master: Fix HTTP 500 on the Audits page in non-English languages  https://review.opendev.org/c/openstack/watcher-dashboard/+/101529203:57
opendevreviewchandan kumar proposed openstack/watcher master: [DNM] testing unified operator and S2I content provider job for watcher  https://review.opendev.org/c/openstack/watcher/+/101529404:06
opendevreviewchandan kumar proposed openstack/watcher master: [DNM] testing unified operator and S2I content provider job for watcher  https://review.opendev.org/c/openstack/watcher/+/101529404:07
opendevreviewchandan kumar proposed openstack/watcher-dashboard master: Fix Audits page HTTP 500 in non-English locales  https://review.opendev.org/c/openstack/watcher-dashboard/+/101287504:28
opendevreviewDouglas Viroel proposed openstack/watcher-specs master: Add spec for availability zone drain  https://review.opendev.org/c/openstack/watcher-specs/+/100785313:00
dviroel#startmeeting watcher13:00
opendevmeetMeeting started Thu Nov 12 13:00:39 2026 UTC and is due to finish in 60 minutes.  The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
opendevmeetThe meeting name has been set to 'watcher'13:00
dviroelhello folks o/13:01
jgilabero/13:01
dviroelwho is around?13:01
chandankumaro/13:01
winiciusallan[m]o/13:01
dviroelcourtesy ping: amoralej sean-k-mooney morenod13:02
morenodo/13:02
dviroelwhile folks join the meeting13:02
dviroellet me add the meeting agenda13:02
dviroel#link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L28 (Meeting agenda)13:02
dviroelthis is our agenda for today13:02
sean-k-mooneyo/13:02
dviroelfeel free to add topics to the agenda13:02
dviroellets see how can we cover13:03
dviroel#topic https://blueprints.launchpad.net/watcher/+spec/watcher-openstacksdk 13:03
dviroeljgilaber: spec13:03
dviroel"Watcher suppport in OpenstackSDK"13:03
jgilaberyes, we merged the spec this week13:03
jgilaberso I created the blueprint for it13:04
amoralejo/13:04
dviroelright13:04
dviroelcool, I think I can quickly review it and set the approval13:05
dviroelthanks jgilaber 13:05
jgilaberthanks dviroel 13:05
jgilaberthat's all for this topic13:05
dviroelah o13:05
dviroelwas about to ask :)13:05
dviroelsomeone wants to add something?13:05
dviroelnext topic then13:06
dviroel#topic Updates to Availability Zone Drain Spe13:06
dviroelthere was a round of reviews in this spec13:06
dviroel#link https://review.opendev.org/c/openstack/watcher-specs/+/100785313:06
dviroelty for reviewing it13:06
dviroelit is getting bigger13:07
dviroelsome updates and outcomes from it, after discussing here in the IRC and in the spec itself13:07
dviroelsean-k-mooney: had the idea of including "availability_zone_pinning_policy" which allow users to select how the plan should handle the pinning13:08
dviroelwhich I think that is a good way of proposing it, and to extend it in the future13:08
sean-k-mooneybasicly 3 option alwasy pin, never pin, and pin if it was pinned before the move13:08
dviroelother than a boolean13:08
amoralejfor instances previously pinned, right?13:08
dviroelright13:08
sean-k-mooneyamoralej: no for all instnaces13:08
sean-k-mooneywell that are moved13:08
sean-k-mooneyauto which woudl be the pin if it was already pinned case13:09
dviroelall instances in the action plan13:09
sean-k-mooneycoudl be the default13:09
sean-k-mooneyya that13:09
sean-k-mooneyi woudl be a high level plocy for the audit not a per instnace setting in the input parmaters13:10
amoralejwhat'd be the case for always? isn't that weird?13:10
dviroelthe new spec PS has 4 options now tbh13:10
amoraleji mean, pinning a vm which was unpinned before13:10
dviroelit is an option..13:11
sean-k-mooneyamoralej: if you are chaing the az in the first place its likely that you dont want that policy decsion to be lost in the furue13:11
dviroelthe same way that unpin_only is too13:11
sean-k-mooneyright so if you moved everythign because your draingin you mign not want the workload stablistatoin policy to move it back an hour later13:11
sean-k-mooneyif you do leave it un pinned13:12
sean-k-mooneyif you dont pin it13:12
amoralejok13:12
amoralejand wrt the default zone, the decission was to never pin to it, right?13:12
dviroelyeah, i forgot that in yesterday's PS, but I just added13:12
sean-k-mooneythat woudl be consitent with nova docs13:13
dviroelthe decision is to never pin to it13:13
sean-k-mooneyits still tbd if we are going to make that a hard error or not13:13
sean-k-mooneyon the nova side13:13
sean-k-mooneybut ya i woudl prefer to never pin to the default az 13:13
sean-k-mooneywe are trending in the direction 13:13
sean-k-mooneyof changing how the az is reproted for host in the default az13:14
sean-k-mooneyby reporting either `""` or `internal`13:14
sean-k-mooneyand then not inlcuding it in the AZ list13:14
dviroelsome other discussions in the spec:13:14
amoralejand for pinned instances to the default one, it's treated as any other zone, right?13:14
amoralejso, a vm pined to nova (or default zone) will not be migrated if include_az_pinned_instance is false13:15
sean-k-mooneyamoralej: righ tnow there is a bug in the schduler and yes effeictlvy but that may become a hard error13:15
sean-k-mooneyunless you have create a host aggrate with the az metadata with the same name13:15
amoralejin case src_az is nova, i mean13:15
sean-k-mooneyoh well for souce az we would still supprot moving form the default az13:16
sean-k-mooneybut as i noted the az reported on the compute node may change to the empty string13:16
dviroeli think that if skippin is enabled, so we would not include it13:16
dviroelif is pinned to the default already13:16
amoralejok13:16
dviroeli also renamed "skip_az_pinned_instances" to "include_az_pinned_instances".  - sean-k-mooney thinks that is better to not include them by default. Different from what we discussed previously13:17
amoralejas soon as it's documented, i think it will work13:17
sean-k-mooneythe main reason for the rename is to avoid the double negitive13:17
sean-k-mooneyi.e skip=false means include13:17
sean-k-mooneywhich is a double negitive13:17
dviroelack, makes sense13:17
amoralejyes, include_ seems more clear to me13:18
jgilaberthe new option has the advantage of circumventing the problem we discussed about stale AZ info in the CDM13:18
jgilaberfor the always and never pin option13:18
jgilaberthe default auto would still have the same issue13:19
jgilaberI haven't check the last PS, not sure if you're updated that dviroel 13:19
dviroelyes I updated13:19
dviroelthe auto may still move a pinned instances13:19
sean-k-mooneyyep, nova only allows you to pin to the az the vm is currely in, any other value will be rejected13:19
sean-k-mooneyif its not pinned and you ask for it to be unpinned i thin kthat wont raise an error13:20
sean-k-mooneyso worst case in teh auto case if it was stale and we tought it was pinned13:20
sean-k-mooneywe woudl move it an pin it13:20
sean-k-mooneyif we tought it was not pinned13:20
sean-k-mooneythen the move would fail13:20
sean-k-mooneybut i think thats fine13:20
sean-k-mooneyjust runing the auit again woudl work13:20
sean-k-mooneyby the way on the staleness i noted that there is an instance update event we get as a notificaion13:21
sean-k-mooneyso when we retive that we can just do a server show direct to the nova api to get the pinning info13:21
amoralejdviroel, in the spec, the policy relocate_pin is the auto mode?13:21
sean-k-mooneyif nova ever passes that in the notificaon we can optimise13:21
dviroelsean-k-mooney: which event?13:21
sean-k-mooneythere is an instance update notificaoin13:22
dviroelah yeah, but i don't think that it is sent13:22
sean-k-mooneypining and unpiing just use the server update api13:22
dviroelsince instance update_dict will be empty13:22
dviroelduring save()13:22
sean-k-mooneyit should be if not it woudl be a bug13:22
dviroelso it will return early13:22
dviroelthat's what I see from the code13:22
sean-k-mooneyupdate_dict shoudl nto be empty13:22
sean-k-mooneyit shoudl be seting the pinned_az filed to the empty sting13:23
sean-k-mooneyor to the current az name13:23
dviroeli think that is gets that from request_spec13:23
dviroelnot from the instance13:23
dviroelso there is no update to the instance obj13:23
sean-k-mooneyperhaps but it woudl still be a bug13:23
sean-k-mooneyif you did an update on the instance that notificaion should be sent13:24
dviroelit is a PUT to the instance yeah13:24
sean-k-mooneylets file that as a bug and dicsus it with the nova team13:24
dviroelack13:24
sean-k-mooneyif they accpate it we can add a followup to use it in watcher13:24
sean-k-mooneyto refhes the instace on instance updates13:24
dviroelamoralej: yes, relocate_pin was a suggestion, but not sure if is the best13:25
dviroelsean-k-mooney: sure, that would work yes13:25
dviroelyes can discuss more after the meeting, and look at code ,etc..13:25
dviroelwrt to the spec13:25
dviroelthere were some other small changes13:25
dviroelplease compare with the previous PS that you reviewed and let me know if you all agree with the changes..13:26
amoralejI will review, i'm not sure skip_conflicting should be a policy option13:27
dviroelteim review also help us to not have any contradictions around the spec 13:27
dviroelthat somethin that I just added, but would cover the issue with stable model13:27
dviroels/stable/stale13:27
dviroeland be the default for other strategies13:28
amoralejI will check after the meeting13:28
dviroeli think that we can discuss more in the spec, and in each option13:28
dviroelthanks amoralej 13:28
sean-k-mooneyoh by the way im usign tomorrow "day of learning" to "learn" how to run teim review on a non outdate k8s with workign logging :)13:28
dviroeloh nice13:29
sean-k-mooneyif there are other tweaks or refinements that folks want let me know13:29
dviroelack13:29
amoralejcool13:29
dviroelanything else that you folks want to chat about zone drain?13:29
dviroelotherwise we should move to the next topic13:29
sean-k-mooneyno i will try and review it again today ro monday13:30
dviroelthanks all!13:30
sean-k-mooneybut i think we are close to beign able to proceed wit the spec13:30
dviroelyeah, we are closing the gaps13:30
dviroel#topic Update goals and strategies in place feature13:30
dviroelamoralej o/13:30
dviroel#link https://review.opendev.org/c/openstack/watcher/+/101457113:31
amoralejIn the review a couple of questions were raised that i'd like to discuss here13:31
dviroeloh, this is not the spec13:31
amoralejoh, my bad with the link sorry13:31
dviroel:)13:31
jgilaberhttps://review.opendev.org/c/openstack/watcher-specs/+/101269613:31
amoralej#undo13:31
amoralej#link https://review.opendev.org/c/openstack/watcher-specs/+/101269613:32
amoralej1st topic is about audit scopes13:32
dviroelright13:32
amoralejas dviroel pointed, in future we may have problems with audit scopes if schemas for scopes change13:32
amoralejso it may be good to implement a similar approach of versioned schemas13:33
dviroeli asked about audit scopes, mainly because i would like to improve it in near future yeah13:33
amoralejcurrently, the auditscope is only checked when an audit is created13:33
amoralejon execution, it's just consumed and processed on the get_scoped_model13:34
amoralejhttps://github.com/openstack/watcher/blob/28da878b1ff1b4be820ef946af7c88563d43b9f3/watcher/decision_engine/scope/compute.py#L17613:34
amoralejso, it may happen if the scope schemas are modified in backwards incompatible ways13:34
dviroelthe scope is stored with the audit, right? and there is no check during the sync()?13:35
amoralejsaid that, imo, given that the syncer is not checking audit scopes, it's out of the scope for this spec13:35
amoralejyes, it's part of the audit13:35
amoralejno13:35
dviroelack13:35
amoralejscope definition is not in the database13:35
amoralejand sync doesn't care about it13:36
dviroel:(13:36
amoralejit may be good to check if an audit scope conforms with current schema at execution13:36
amoralejand fail the audit otherwise, as first step13:36
sean-k-mooneymaybe13:37
jgilaberit seems to be listed in the DB model for an audit https://github.com/openstack/watcher/blob/28e511b7f6a05ea6bfa66e3cc8e65c08b1a93fe0/watcher/db/sqlalchemy/models.py#L18213:37
amoralejand we could implement upgrade_scope in future13:37
sean-k-mooneybut we shoudl not change them in a backward incompatble way13:37
sean-k-mooneythat would break upgrades13:37
amoralejyes, i think that should be the default policy13:37
dviroelok to not include the scope in the current spec13:38
sean-k-mooneywell teh scope is already snaphost13:38
sean-k-mooneyits snapshot when the auit is create form teh template13:38
dviroelbut would be the next thing to fix I think13:38
sean-k-mooneyim not sure that it needs to be fixed13:38
amoralejthe scope snapshot of the audit is in database13:39
sean-k-mooneyyes13:39
amoralejwhat i meant before is that scope schema is not (as it's for the strategy spec)13:39
sean-k-mooneythe schema doe not need to be13:39
jgilaberah ok I misunderstood you amoralej 13:39
sean-k-mooneyand the fact that we have it for the stragies is actully kind of weird13:39
amoralejyep, i agree13:39
amoralejok, so we agree on keeping scope handling out of the scope in this spec?13:40
sean-k-mooneysure13:40
jgilaber+113:40
amoralej(scope is overloaded here :) 13:40
sean-k-mooneyto mean its already snapshot13:40
sean-k-mooney*to me13:40
sean-k-mooneyso we dont need to incldue it for now13:41
amoralejthe other topic is about how to hande non-versioned out-of-tree strategies.13:41
sean-k-mooneywell out of tree stratiges are not suproted by the watcher core team13:41
amoralejthere may be deployers using strategies not following the versioning method proposed13:41
sean-k-mooneyyep and its there responisbelty to maintian compatiablity themselves13:42
sean-k-mooneywe shoudl docuement our expecations but its not a watcher bug if they break13:42
amoralejWhat i was proposing is that in those cases CONTINUOUS audits will continue active. However, the decision-engine AuditHandler checks if the parameters content pass schema validation, so it will be fine while the changes are backwards compatible. When an audit fails to validate, it will be set to FAILED and leave a clear error in the status_message.13:42
sean-k-mooneyack that sound reasonable13:43
amoralejthat's what is currently in the spec, feel free to review the wording there13:44
dviroelack, sounds goods13:44
amoralejfinally, while working on it i found what i consider a pre-existing bug https://review.opendev.org/c/openstack/watcher/+/1014571 13:44
amoralejsorry, that link is wrong13:44
amoralejhttps://bugs.launchpad.net/watcher/+bug/217405113:45
dviroel" Terminated Audits and ActionPlans in SUCCEEDED and FAILED status are CANCELLED by Syncer Edit"13:46
amoralejso, if there is agreement that's a bug on current syncer implemention, i will propose fix before the new feature series13:46
* dviroel melting_face13:46
amoralejshort version, an Audit in SUCCEEDED or FAILED is moved to CANCELLED13:46
amoralejif the strategy is updated13:46
jgilaberyes looks valid, I can't imagine any reason to do that13:47
sean-k-mooneyyes its a bug and i would triage it as critial or more likely high impornatce13:47
dviroeldoes the state machine allow that?13:47
dviroelthey are final state13:47
amoralejnop13:47
* dviroel i think that it doesn't validate13:47
sean-k-mooneydviroel: the state diagrame does not13:47
amoralejthat's what is making grenade jobs to fail in https://review.opendev.org/c/openstack/watcher/+/1014569/ i.e.13:47
sean-k-mooneybut because we are not afctully using a state machine explcity13:47
sean-k-mooneyit is posisble due ot the direct upates of the db13:48
amoralejyes13:48
dviroelyeah13:48
amoraleji'm also preparing a patch for that13:48
sean-k-mooneythere is no enforcement in code on teh avlie transitions 13:48
sean-k-mooneywe coudl build that eventually13:48
sean-k-mooneybut fo now it shoudl just never update stauts that are already terminial13:48
amoraleji think we only validate transitions in the api13:48
dviroelamoralej: patch for what exactly?13:48
amoralejyes13:49
amoralejok, so i think that's it wrt this topic13:49
sean-k-mooneyamoralej: so if we dont already have a cosntat for this i woudl defeint one for the termal states and skip the actions plans and or actions that are already terminal in the sync13:49
amoraleji think we have agreements on all topic, thanks :) 13:49
amoralejsean-k-mooney, that's what i'm implementing13:50
sean-k-mooneycool13:50
dviroelack13:50
dviroelany ther question folks?13:50
jgilaberwe should triage the bug now13:50
jgilaberor list it for later13:50
amoralejbtw, i sent implementation proposals for the new behavior in https://review.opendev.org/q/topic:%22blueprint-update-goals-and-strategies-in-place%2213:50
dviroeloh yeah, lets do this now13:50
amoralejif you want to take a look13:50
jgilaberbut it should be quick now since we've already discussed13:50
dviroelit is already set to Hig13:51
dviroelwhich looks fine to me13:51
dviroeljgilaber: agree?13:51
sean-k-mooneydviroel: ya i just set it13:51
jgilaberah ok13:51
dviroelack13:51
jgilaberyeah High seems ok for me13:51
dviroel#topic Functional tests13:51
dviroelamoralej: you again13:51
amoralejyes, that's just ask for feedback13:51
dviroel#link https://review.opendev.org/c/openstack/watcher/+/99354313:51
amoraleji think the main open point is https://review.opendev.org/c/openstack/watcher/+/993543/comment/03229647_25ef63af/13:52
amoralejno need to discuss now, we are short on time13:52
amoralejbut it's just if/how we want to validate metrics consistency between host and instances metrics13:52
dviroelack, lets keep that thread open there and get more feedback13:53
dviroeli need to revisit it13:53
amoralejyep13:53
dviroelyeah13:53
sean-k-mooneyamoralej: long term i would probly add a helper to sync form instnace ot host13:53
sean-k-mooneyor visversa13:53
sean-k-mooneybut wihtout using it13:53
sean-k-mooneyim not sure which we will want to use more13:53
sean-k-mooneyi think the patern of updating the isntance metrics13:54
sean-k-mooneytehn callign a method on the host to refcaluate the host values form the instnace on the host13:54
sean-k-mooneywill be the simplest13:54
amoralejit's tricky because we may want to simulate cases for different host usage out of instances, etc... ceph, etc...13:54
sean-k-mooneybut ya i dont have a stong opion right now either way13:54
amoralejwe could parametrize13:54
amoralejbut anyway, let's discuss in the review13:54
dviroelthere are different use cases yeah...13:54
dviroelack13:54
sean-k-mooneywell i was more thinkin ghtat it would not be automatic13:54
sean-k-mooneywe would call the help in the test when we wanted it to sync them13:55
sean-k-mooneyif we dont wnat that in teh test just set the values driectly13:55
dviroelyeah13:55
sean-k-mooneyanyway we can likely move on13:55
dviroeli think that we will agree on an approach..13:55
dviroel#topic Reviews13:55
dviroel#link https://review.opendev.org/c/openstack/watcher/+/1014459 13:55
dviroel"fix for memory leak on openstacksdk connections"13:55
amoralejmorenod found bug https://bugs.launchpad.net/watcher/+bug/217307813:56
sean-k-mooney...13:56
amoralejwhich is a pretty big memory leak related to how connections are created13:56
sean-k-mooneyyou realise this should have been treaded as a security bug13:56
sean-k-mooneyprivately13:56
amoralejwe didn't triage that, bth, i'd set to high at least13:56
amoraleji didn't13:57
sean-k-mooneya memory leak is a denial of service vector13:57
amoralejok13:57
amoralejgood to know ...13:57
dviroelis this a openstacksdk only issue?13:57
amoraleji'd say so13:58
amoralejyes, not in the old clients implementation13:58
sean-k-mooneyso this looks like we are not release the client instnace13:58
sean-k-mooneythe copying of the model is expected in the desciosn engin case13:58
amoralejyes, we are not releasing and connections remain in memory13:59
sean-k-mooneyi..e when the collector or an audit runs we create a copy of the model13:59
amoralejyep13:59
sean-k-mooneynormally13:59
sean-k-mooneythis can oly happen if we are maintining a refence to the client13:59
amoralejactually, before this patch, memory usage grows linearly on periodic collections, i.e.14:00
sean-k-mooneyadn if we have created a cirular refnece14:00

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!