Thursday, 2026-06-18

dviroelhey folks, o/11:55
dviroelwatcher meeting will start in 5 mins 11:55
dviroel#startmeeting watcher12:00
opendevmeetMeeting started Thu Jun 18 12:00:12 2026 UTC and is due to finish in 60 minutes.  The chair is dviroel. Information about MeetBot at http://wiki.debian.org/MeetBot.12:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.12:00
opendevmeetThe meeting name has been set to 'watcher'12:00
dviroelhi all o/ 12:00
chandankumaro/12:00
jgilabero/12:01
amoralejo/12:01
dviroelcourtesy ping: sean-k-mooney morenod rlandy12:01
rlandyo/12:01
dviroellet's start with today's meeting agenda12:01
dviroel#link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L31 (Meeting agenda)12:02
dviroelfew topics today as expected since we are having a different meeting today12:02
dviroel#topic Announcements12:02
dviroelI believe that is important to remimder about our deadlines that are close12:03
dviroelSpec Freeze is July 2 (14 days)12:03
dviroelwe are 2 weeks from spec freeze12:03
* dviroel checking specs open12:03
dviroel#link https://review.opendev.org/q/project:openstack/watcher-specs+is:+open12:03
dviroelwe have one spec open yet12:04
dviroel#link https://review.opendev.org/c/openstack/watcher-specs/+/98717112:04
dviroelis winiciusallan[m] spec about preemtible instances12:04
amoralejoh, i forgot to sent my comments, i just did12:04
dviroelreminder for us to read and provide feedbacks12:04
dviroelas also a reminder for winiciusallan[m] to keep updating the spec as needed in the following weeks12:05
dviroelto we can get the spec in12:05
winiciusallan[m]o/12:05
dviroelsince we kind already agree that is a nice to have feature12:05
winiciusallan[m]sorry this week was really busy, but i'll take some time to update it this week12:05
dviroelso we just need to handle the details now12:05
dviroelwiniciusallan[m]: ack thanks, lets also try to review from our side here12:06
winiciusallan[m]thanks for pinging, doug12:06
dviroelso we don't let everything for the last week 12:06
dviroelnp winiciusallan[m] 12:06
dviroelanyone has other watcher spec to propose yet?12:06
amoraleji expect to send one12:07
amoralejrelated to optimization of actionplans12:07
dviroeli was originally thinking on proposing a spec related to filtering invalid destination migration12:07
amoralejif i have the time, hopefully12:07
dviroeli  may also propose this week if I have time, but unlikely to have time to implement it this release maybe12:08
dviroelbut it would be good to have an initial idea too12:08
amoraleji have similar plan12:08
dviroelack, so that's what I had in mind for announcements due to the deadline12:09
dviroelnext topic then12:09
dviroel#topic Stable branches review day12:09
dviroelso we agreed last week that we would use today's meeting to review stable patches12:10
dviroelthe idea is to use this remaining meeting time to go through stable patches and review them...12:10
dviroelnow, how do we want to drive this?12:10
dviroelideas?12:10
dviroelwe can go one by one in sync here, with links12:11
dviroelor each one goes through the patches and just comment here specific concerns?12:11
dviroelthoughts?12:11
jgilaberwe might have time to go through 2026.1 and 2025.212:11
sean-k-mooneyo/12:11
jgilaberbut I doubt we'll get through everything12:12
amoralejyes12:12
dviroel#link https://review.opendev.org/q/project:openstack/watcher+branch:stable/2026.1+status:open12:12
amoralejthat's easy :)12:12
dviroelso dowe go one by one then?12:12
dviroel981225: Add watcher-grenade-skip-level-always job | https://review.opendev.org/c/openstack/watcher/+/98122512:13
jgilaberI'd say yes, let's see how many we can go through12:13
amoralejbtw, last days unit jobs have been failing12:13
amoralejrandomly12:13
dviroel2026.1 is  a slurp12:13
dviroelack, once we approve it, we can handle zuul async later I think12:14
dviroelnot sure about the randim unit tests failures tbh12:14
dviroellogs already gone for this patch12:14
amoralejexample https://zuul.opendev.org/t/openstack/build/a9c20c282fff4291a6d3cf0cf16b6da112:15
dviroelI would just check the results, if is jsing the correct versions and approve it 12:15
amoralejbut yeah, let's discuss      later12:15
dviroelyep12:15
dviroel#link https://review.opendev.org/q/project:openstack/watcher+branch:stable/2025.2+status:open12:15
dviroelstarting 2025.212:15
dviroel980298: Handle missing fields building storage model XML or list | https://review.opendev.org/c/openstack/watcher/+/98029812:16
dviroeldone already12:16
dviroeli mean, lgtm 12:17
dviroelhandle failing when accessing the models, some try/except to avoid error in logs and so on...12:17
dviroelfeel free stop me in any of those...12:18
dviroel979129: Fix small issues in cinder_helper unit test | https://review.opendev.org/c/openstack/watcher/+/97912912:18
sean-k-mooneythe only thing ill note is while i think the backprot are valid12:19
dviroelactually this ^depends 979128: Fix check for volume type in cinder helper migrate | https://review.opendev.org/c/openstack/watcher/+/97912812:19
dviroelso this one ^12:19
sean-k-mooneyas we are doing this review we need to also check that there are no changes form the preir release12:19
sean-k-mooneyunless the change is called out in the commit message12:19
sean-k-mooneythe only reall change i expect going to 2025.2 woudl be auto pep8 vs ruff12:20
sean-k-mooneyor was that for master to 2026.1?12:21
jgilaberI think most of them will be from before we made the swith to ruff12:21
dviroeli don't recall 12:21
sean-k-mooneyanyway the frist oen was fine 12:22
sean-k-mooneyhttps://review.opendev.org/c/openstack/watcher/+/97912812:22
sean-k-mooneyyou emthion this has a depency?12:22
sean-k-mooneyoh you just mean https://review.opendev.org/c/openstack/watcher/+/979129/ need https://review.opendev.org/c/openstack/watcher/+/979128/12:22
dviroelcorrect 12:22
sean-k-mooneybut that is ok because they are stacked in that order12:22
dviroelbecause I started top to botton https://review.opendev.org/q/project:openstack/watcher+branch:stable/2025.2+status:open12:23
dviroelthere is now a list of zone_migration related fixes12:24
sean-k-mooneyack12:24
sean-k-mooneyso ya both of those look fine to me so +2w12:24
amoralejyep, zuul should know how to handle if we add +W to both, right?12:24
amoralejas are stacked12:24
dviroelyes12:24
dviroel979120: Add tests for get_dst_pool_and_type method | https://review.opendev.org/c/openstack/watcher/+/97912012:24
dviroelthis is the first one of zone_migration fixes12:25
amoralejthose are also properly stacked12:25
dviroelright, I am reviewing them in the order already12:26
dviroelmost of them amoralej already reviewed too, tks!12:26
sean-k-mooneysame12:27
amoraleji did a pass some time ago, yep12:27
sean-k-mooneyhttps://review.opendev.org/c/openstack/watcher/+/979952/1 is nextg12:28
sean-k-mooneyor rather https://review.opendev.org/c/openstack/watcher/+/979953/112:28
sean-k-mooneysince doug already did the other12:28
dviroeli already reviewed, but if you guys spot something, just W-112:29
dviroel:)12:29
dviroeli am on https://review.opendev.org/c/openstack/watcher/+/97995312:29
sean-k-mooneyits part of a diffent bug tecnially but ya it looks fine to me12:30
dviroelthis one mentions other patch12:30
dviroelwhich merged in 2025.2 already https://review.opendev.org/c/openstack/watcher/+/96881712:30
dviroeljgilaber: correct? ^12:30
jgilaberyes IIRC the patch linked introduced a regression12:30
dviroelack, so we need this in12:31
jgilaberwe merged this on in master just after feature freeze at the time I think12:31
sean-k-mooneywe have alrady backproted that https://review.opendev.org/c/openstack/watcher/+/96881712:31
sean-k-mooneyso the depency is in teh 2025.2 branch12:31
sean-k-mooneybut not yet in 2025.112:31
sean-k-mooneyso we will need to becareful when we get to that point 12:32
sean-k-mooneybut its fien on the current branch to continue12:32
jgilaberthey are also stacked in the 2025.1 branch12:32
dviroelright, wouldn't be the case of squashing them 12:32
jgilaberin a long chain12:32
dviroelack12:32
sean-k-mooneythe only time we squsion is if there is a gate bug that we cant resolve otherwise12:33
sean-k-mooney*squash12:33
dviroelbut yeah,  i would propose both in a single change to not miss the second one 12:33
sean-k-mooneyi would not12:34
dviroelnot because is really needed 12:34
sean-k-mooneyand ask youto back port it properly by stackign them12:34
sean-k-mooneyi get where your comming form but we really should try to preseve teh git history12:34
jgilaber^^ that was my thinking12:35
sean-k-mooneyin general you would get a merge conflict12:35
sean-k-mooneyif you put them in the wrong order anyway12:35
dviroelthe chain doesn't block the first one to get merged and miss the second one, that's my point12:36
sean-k-mooneyin anycase i think we are done with 2025.212:36
dviroelyep12:36
sean-k-mooneyright but for chains like this the better way to do it is to hold the +w on the bottom patch until we are happy with the chain12:36
dviroelwe are done with 2025.212:36
sean-k-mooneythen send them all at once12:36
dviroelack, requires more attention 12:37
sean-k-mooneyi mean there is no wrong answer here really other then not reviewing them and fixing it12:37
dviroelyeah, agree12:37
sean-k-mooneys/fixing/backporting/12:37
dviroelbtw, we are done with the proposed patches12:38
dviroelbut there are probably more fixes that could be backported12:38
sean-k-mooneyya so im not sure i fi have the mental capasity to rapid fire review 2025.112:38
jgilaberjfyi when all the patches land I'll propose a cherry pick for https://review.opendev.org/c/openstack/watcher/+/970381 which enables some zone migration tempest tests12:38
jgilaberit needed some of those fixes and I forgot to propose it before12:38
sean-k-mooneycan i suggest we do this again for 2025.1 next week12:38
amoraleji was checking at master to see if we want to propose anything else to 2026.1, maybe the fix for https://bugs.launchpad.net/watcher/+bug/2152645 ?12:39
jgilabersean-k-mooney, I think that's ok, we did great progress today12:39
sean-k-mooneyor async and then we can prepare release for the next milestone12:39
dviroelamoralej: good call, I think that worth backporting12:39
sean-k-mooneyya that seam valid to me as well12:39
amoralejother than that, i think no much to backport from master, the rlock, i wuldn't backport it at least until we run it in master for some time12:40
dviroelyou mean the fix for launchpad.net/bugs/215225412:41
amoralejyes12:41
sean-k-mooneyhttps://review.opendev.org/c/openstack/watcher/+/98939512:41
amoraleji'd start backporting only the ones for #215264512:41
sean-k-mooneyfor https://bugs.launchpad.net/watcher/+bug/215264512:42
amoralejhttps://review.opendev.org/c/openstack/watcher/+/989395/1?usp=related-change + https://review.opendev.org/c/openstack/watcher/+/989396/2?usp=related-change12:42
sean-k-mooneyya im not sure if we want to backport the full instance locking changes yet12:42
amoralejyep12:42
sean-k-mooneyim not against that but i think that large change could do with baking in ci a bit longer12:42
dviroelsean-k-mooney: i think that i mixed bugs in topics and i will update LP with the correct one12:42
sean-k-mooneythe notification locking is less invasive so we can start with that12:43
sean-k-mooneyand then perhaps near milestone 3 see if we want the per instance locking changes as well12:43
dviroelhttps://review.opendev.org/c/openstack/watcher/+/98939612:44
dviroelok, I can propose the backports for the sync12:44
amoralejack12:45
dviroelI'm not sure that I can do this in a chain using UI12:45
dviroeli will do that later from my laptop12:45
dviroelonce we get 2025.2 patches in, we can do a review in 2025.1 too, maybe async12:46
amoralejno rush, let me know and i will review async12:46
dviroelwhat else we have for today?12:47
dviroeloh, there is a bug 12:47
dviroel#topic Bugs12:47
dviroel#link https://bugs.launchpad.net/watcher/+bug/2156473 (vm_workload_consolidation strategy ignores allocation limits, overpacking instances onto nodes_12:48
dviroeli see the issue...12:48
amoralejyeah, i sent a patch already but didn't fill importance12:49
amoralejfor me is medium or high12:49
dviroel#link https://review.opendev.org/c/openstack/watcher/+/993088 (993088: Add allocation-based capacity checks to vm_workload_consolidation)12:49
amoraleji mentioned in last meeting but then i reproduced with a real environment12:50
dviroeljust wondering if more strategies do the same thing...12:50
amoralejalso, note that i stacked the fix for that on top of the one for https://bugs.launchpad.net/watcher/+bug/214193912:51
amoralejdviroel, we may easily test it with functional tests https://review.opendev.org/q/topic:%22blueprint-functional-test-infrastructure%22 :)12:51
dviroel+1 on functional tests infra, thanks a lot12:52
dviroelnee to take some time to look at it12:52
amoralejtbh, i want to dig more into validation12:52
dviroelfor the Bug reported, I think that is ery important to fix, and there is no way to workaround the issue probably12:53
amoraleji know that node_workload_consolidation does it well12:53
amoralejhardware_maintenance has a tricky behavior12:54
jgilaberthe bug seems valid to me and I could see it either as high or medium12:54
amoralejit checks if all the instances fit in backup_node if specified, if not all the instances in maintenance_host fits into the backup one, then it creates migrations without specifying destination and leaves nova to decide 12:55
dviroelbut if check only for usage, not allocation right12:56
amoralejonly allocation, i mean hardware_maintenance12:56
amoralejhardware_maintenance only checks allocations, not usage at all12:56
dviroelah ok12:56
amoralejlook for "test_backup_insufficient_capacity" in https://review.opendev.org/c/openstack/watcher/+/993352/3/watcher/tests/functional/test_host_maintenance.py#38912:57
amoralejthat's how i learn it12:57
amoralejthat seems to be by-design behavior12:57
amoralejbut yeah, that's another strategy, sorry i diverged12:58
dviroelyeah. as long is documented, it is a ok behavior12:58
dviroelok, wrt to the bug12:59
dviroeli would set to medium for now12:59
amoralejwfm12:59
dviroelwe could say that works by design :) 12:59
dviroelbut the design isn't good12:59
amoralejit could be better :) 13:00
dviroelsince there is already a fix to it, we can discuss more in gerrit13:00
dviroelthanks for reporting and proposing the fix amoralej 13:00
dviroel#topic Other discussions13:01
dviroelwe are out of time13:01
amoralejyep13:01
dviroelbut amoralej wants to propose that we change or shift our meeting time13:01
amoralejit'd be great if we could move this mtg to a different time13:01
amoralejit conflicts to other mtg that i'd like to atttend13:02
dviroel1h later works for me, but we need more feedback from the team13:02
sean-k-mooneyas long as its any time after about 11:00 utc i generaly have no issue with moving it13:02
amoralejwe can add it as first topic in next mtg13:02
dviroelack amoralej 13:02
dviroelI also added a place to vote there async13:02
jgilaberI'd be fine with 1h our earlier or before, but personally I'd prefer after13:02
sean-k-mooneyor keeping it at its current slot but ya lets revisit next week or do it async13:03
dviroelwe can cover that next meeting and take the decision13:03
amoralejsure13:03
dviroel#topic Volunteers to chair next meeting13:03
dviroeli will be around, but if someone want to chair13:03
jgilaberI can do it dviroel 13:03
dviroelthanks jgilaber 13:03
dviroellet's wrap up for today13:03
dviroelthank you all for participating13:04
dviroel#endmeeting13:04
opendevmeetMeeting ended Thu Jun 18 13:04:09 2026 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:04
opendevmeetMinutes:        https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-06-18-12.00.html13:04
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-06-18-12.00.txt13:04
opendevmeetLog:            https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-06-18-12.00.log.html13:04
amoralejthanks dviroel for chairing!13:04
dviroelty all for participating13:04
opendevreviewMerged openstack/watcher stable/2026.1: Add watcher-grenade-skip-level-always job  https://review.opendev.org/c/openstack/watcher/+/98122513:58
opendevreviewMerged openstack/watcher stable/2025.2: Handle missing fields building storage model XML or list  https://review.opendev.org/c/openstack/watcher/+/98029813:58
opendevreviewMerged openstack/watcher stable/2025.2: Add tests for get_dst_pool_and_type method  https://review.opendev.org/c/openstack/watcher/+/97912013:58
dviroelsean-k-mooney: hey, so I forgot to ask you about teim-ci14:07
dviroeldo you intend to continue maintaining it?14:07
amoralejyeah, i also had thought on that14:13
opendevreviewMerged openstack/watcher stable/2025.2: Fix zone migration dst pool and type selection  https://review.opendev.org/c/openstack/watcher/+/97912114:21
opendevreviewMerged openstack/watcher stable/2025.2: Add test for zone migration audit without dst_type  https://review.opendev.org/c/openstack/watcher/+/97995214:21
opendevreviewMerged openstack/watcher stable/2025.2: Ensure zone migration creates the correct action  https://review.opendev.org/c/openstack/watcher/+/97995314:44
opendevreviewMerged openstack/watcher stable/2025.2: Test zone migration audit without destination  https://review.opendev.org/c/openstack/watcher/+/97995414:44
sean-k-mooneydviroel: amoralej  so i can15:10
sean-k-mooneyits currntly broken 15:10
sean-k-mooneywhich is why its not reporting15:10
sean-k-mooneyteh vm it was running on was getting like 34kbps clones for some reason that i have not debugged15:10
sean-k-mooneyso i powered it off15:11
sean-k-mooneydviroel: amoralej  can i assume that ye found it useful and would like it to return?15:11
sean-k-mooneyfixign it is hopefully not hard i just need to sit down and do it, or have ai do it :)15:13
dviroelsean-k-mooney: I think that it is useful yes15:14
dviroelif you want you can consider a manual trigger too, so we trigger the job when we think that is better15:14
sean-k-mooneyit supprot both "temi-ci: manual" will triger it on watcher and cyborg15:15
dviroelmanual trigger is another demonstration of our interest in its report 15:15
sean-k-mooneyso im happy to go fix it and or make any tweaks folks want15:16
sean-k-mooneyi looked at it breifly a day or two ago  but it was like mindnight and when i saw the issue was networkign related i just went to bed :)15:17
dviroelit would be great, thanks sean-k-mooney - when you have some time of course15:18
dviroeli know that I can get similar outputs using your repo content15:18
dviroelI tested once15:18
sean-k-mooneyya i need to complete my refactor of that to make it more trivial to do15:18
dviroelso we can run it locally if needed until you get some time to debug the issue15:19
sean-k-mooneyi have mostly been focusing my side project efforts on the pci-sim work for cyborg and ARD for the last little bit15:19
sean-k-mooneyso i just didnt get back aorund to working on it15:19
dviroelno worries15:19
sean-k-mooneyi booted back up the vm https://zuul.teim.app/t/main/status and its triing to clone15:34
sean-k-mooney2026-06-18 15:33:14,795 DEBUG zuul.Repo: Cloning from https://******@review.opendev.org/a/openstack/watcher to /var/lib/zuul/executor-git/review.opendev.org/openstack/openstack%2Fwatcher15:34
sean-k-mooneyi might just need to reboot the host or my home switch/routere/ it workign fine form my laptop but not form the pod on my k8s vm15:35
sean-k-mooneyya its just timing out15:37
amoralejsean-k-mooney, yes, i think it's useful, actually i found it improved over time15:42
amoraleji'd vote for having it back :) 15:42
dviroelamoralej:  if you are still around, which job did you mention that was failing/16:55
amoralejlemme look for it16:55
dviroelbuild-openstack-releasenotes?16:55
dviroelthis one failed for me ^16:55
dviroelbut I remember a openstack-discuss thread around that16:55
amoralejsee https://review.opendev.org/c/openstack/watcher/+/96148616:55
amoralejhttps://zuul.opendev.org/t/openstack/build/22abb77ec9af48eaaa2ab95f4da57a6816:57
amoralejhttps://zuul.opendev.org/t/openstack/build/1303dcfee55840688299d48e3e63b59e16:57
amoralejactually, i saw a different message than those16:57
amoralejyeah and https://zuul.opendev.org/t/openstack/build/a9c20c282fff4291a6d3cf0cf16b6da116:57
amoralejlast one is "sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked" but previous ones are different16:58
amoralejall those are unit test jobs which are usually very reliable16:58
opendevreviewMerged openstack/watcher stable/2025.2: Fix check for volume type in cinder helper migrate  https://review.opendev.org/c/openstack/watcher/+/97912817:00
opendevreviewMerged openstack/watcher stable/2025.2: Fix small issues in cinder_helper unit test  https://review.opendev.org/c/openstack/watcher/+/97912917:00
dviroeli was not expecting database lock issues anymore in master branch, there is a lock to avoid that. maybe requires an investigation in this specific test17:14
amoralejyeah, that's what catched my attention17:14
amoraleji couldn't find any relevant merge that may affect17:15
amoralejat least in watcher17:16
opendevreviewJoan Gilabert proposed openstack/watcher stable/2025.2: Enable extra zone migration tests on master branch  https://review.opendev.org/c/openstack/watcher/+/99397617:29
opendevreviewMerged openstack/watcher-dashboard master: tox: Bump min_version to 4.28.0  https://review.opendev.org/c/openstack/watcher-dashboard/+/99264518:04
opendevreviewMerged openstack/watcher master: Remove unused PlacementHelper methods  https://review.opendev.org/c/openstack/watcher/+/99114620:26

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!