| opendevreview | chandan kumar proposed openstack/watcher master: Add watcher-grenade-skip-level-always job https://review.opendev.org/c/openstack/watcher/+/979137 | 04:02 |
|---|---|---|
| opendevreview | chandan kumar proposed openstack/watcher master: Add watcher-grenade-skip-level-always job https://review.opendev.org/c/openstack/watcher/+/979137 | 05:05 |
| jgilaber | Hi everyone! IRC meeting will start in ~30 minutes, feel free to add any topics to the agenda https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L24 | 11:29 |
| jgilaber | #startmeeting watcher | 12:00 |
| opendevmeet | Meeting started Thu Mar 12 12:00:15 2026 UTC and is due to finish in 60 minutes. The chair is jgilaber. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:00 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:00 |
| jgilaber | hi o/ - who's around for today's meeting? | 12:00 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: DNM - PoC nova and prometheus standalone emulators https://review.opendev.org/c/openstack/watcher/+/980257 | 12:01 |
| jgilaber | courtesy ping: dviroel amoralej sean-k-mooney chandankumar morenod rlandy | 12:01 |
| amoralej | o/ | 12:01 |
| chandankumar | o/ | 12:02 |
| dviroel | o/ | 12:02 |
| jgilaber | as usual, feel free to add your own topics to the agenda | 12:02 |
| jgilaber | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L24 | 12:02 |
| jgilaber | we'll give a couple more minutes to see if more people will join | 12:03 |
| jgilaber | otherwise we can start | 12:03 |
| rlandy | o/ | 12:04 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: DNM - PoC nova and prometheus standalone emulators https://review.opendev.org/c/openstack/watcher/+/980257 | 12:04 |
| jgilaber | ok, I think we can get started with the first topic | 12:04 |
| jgilaber | #topic Anouncements | 12:05 |
| jgilaber | we're in the RC1 target week | 12:05 |
| jgilaber | tomorrow RC1 should be cut | 12:05 |
| jgilaber | and it looks like we've merged all the pending bug fixes (except for one that is currently in gate) | 12:06 |
| jgilaber | great job everybody! and thanks for the reviews | 12:06 |
| dviroel | yeah, i saw this last one missing one vote | 12:06 |
| dviroel | I think that should be our last one | 12:06 |
| jgilaber | yes, I checked and I did not see any other bug fix proposed | 12:07 |
| jgilaber | that's all I wanted to announce, any comment on this topic? any other announcement? | 12:07 |
| sean-k-mooney | o/ | 12:08 |
| jgilaber | if not, we can move to the next topic | 12:08 |
| jgilaber | #topic Reviews | 12:09 |
| jgilaber | I added to the list a couple of patches I have for the tempest plugin | 12:09 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/962702 | 12:09 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/963559 | 12:09 |
| jgilaber | they add more tempest coverage for volume migrations in zone migration | 12:09 |
| jgilaber | not urgent but I think they are nice to have | 12:10 |
| jgilaber | any question/comment on these patches? | 12:10 |
| dviroel | dont think so, just need to take a look again | 12:11 |
| jgilaber | ack thanks dviroel | 12:11 |
| jgilaber | let's move to the next one from chandankumar then | 12:11 |
| dviroel | are they protected with config options | 12:11 |
| dviroel | right? because of the fixes that still need to merge in stable branches? | 12:12 |
| jgilaber | yes, they are under the 'run_zone_migration_storage_tests' flag | 12:12 |
| jgilaber | currently only set on master | 12:12 |
| sean-k-mooney | right we shoudl not have test that require backport to pass enabeld by default | 12:12 |
| sean-k-mooney | jgilaber: is that new | 12:12 |
| jgilaber | the flag? not really, we added it during this cycle | 12:13 |
| jgilaber | don't remember when exactly | 12:13 |
| sean-k-mooney | when we have feature gates like this we cant reuese it | 12:13 |
| sean-k-mooney | if it was this cycle its ok | 12:13 |
| sean-k-mooney | but we cant use that same feature flag for this next cycle | 12:13 |
| sean-k-mooney | i.e. if we have anohter storage fix | 12:13 |
| jgilaber | ack I did not know that | 12:14 |
| sean-k-mooney | tempest test are ment to work with alsl stable branches at all times | 12:14 |
| sean-k-mooney | if they need specififc featuer to work the entierh need to check the api microverison or have a flag | 12:14 |
| jgilaber | I double checked we added it on December | 12:14 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/969244 | 12:14 |
| dviroel | oh ywah, the extra flags | 12:15 |
| sean-k-mooney | ack we likely will want to have a set of bug flags at some point that we will eventuly deprecate and remove or have a better long term solution for this | 12:15 |
| sean-k-mooney | i guess we coudl ungate tests our of extra over time | 12:17 |
| sean-k-mooney | that woudl be an option | 12:17 |
| sean-k-mooney | i.e. after the final stable branch is patched move the option but when to do that is a bit tricky | 12:17 |
| jgilaber | yes, we have quite a few patches in the stable branches needed for that | 12:18 |
| jgilaber | ok, anything else on the topic? | 12:18 |
| sean-k-mooney | not for now we can liekly dicuss this on the review | 12:19 |
| jgilaber | thanks! let's move on | 12:19 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/979137 | 12:19 |
| jgilaber | chandankumar, do you want to highlight something about the patch? | 12:19 |
| chandankumar | yes | 12:19 |
| chandankumar | It adds watcher-grenade-skip-level-always job which will test upgrade from 2025.1 to 2026.1 (N-2 to N) release | 12:20 |
| chandankumar | It is an additional upgrade job apart from normal watcher-grenade job which tests N-1 to N release | 12:20 |
| chandankumar | It would be good to have it in our pipeline. | 12:21 |
| chandankumar | Please have a look, thank you! | 12:21 |
| dviroel | this is very importanto to have yes | 12:21 |
| jgilaber | thanks, that's great to have, I'll add it to my review list | 12:21 |
| chandankumar | we can see the logs here https://c50701f61b79b2f93421-0d0b23b7f62d479822a1b52b605369e3.ssl.cf2.rackcdn.com/openstack/5f7f8c97ef5e4b77be2f352bac1f81cd/controller/logs/grenade.sh_log.txt | 12:21 |
| chandankumar | that's it from my side on this review | 12:22 |
| jgilaber | thanks! any question? | 12:22 |
| jgilaber | if not we can move on to triage some bugs | 12:23 |
| sean-k-mooney | for what its worht im ok with the curernt patch | 12:23 |
| sean-k-mooney | i was +2 on it yesterday and i resproed it just now after the description update | 12:23 |
| sean-k-mooney | it would be good to include this in rc1 | 12:23 |
| sean-k-mooney | so that its there when the stabel branch is created | 12:24 |
| sean-k-mooney | althoguh we can cherry pick it if needed | 12:24 |
| chandankumar | thanks sean-k-mooney! | 12:24 |
| sean-k-mooney | ill also note tha we technially shoudl eb doing this on the watcher-dashboard too | 12:24 |
| jgilaber | +1, I'll try to review today | 12:24 |
| sean-k-mooney | but that is less imporant | 12:24 |
| jgilaber | ack, let's move to the next topic | 12:26 |
| jgilaber | #topic Bugs | 12:26 |
| dviroel | yeah, we lack testing in dashboard, we should solve that in the next cycle :) | 12:26 |
| jgilaber | I did not see any untriaged bugs in watcher, but there were two in the tempest plugin | 12:26 |
| jgilaber | #link Add more tempest coverage for volume migrations in zone migration | 12:26 |
| jgilaber | sorry about that | 12:26 |
| jgilaber | #undo | 12:27 |
| opendevmeet | Removing item from minutes: #link Add | 12:27 |
| jgilaber | #link https://bugs.launchpad.net/watcher-tempest-plugin/+bug/2139344 | 12:27 |
| jgilaber | that's the one :) | 12:27 |
| jgilaber | on first glance that looks more like a watcher feature than a tempest plugin bug | 12:27 |
| amoralej_ | i reported that as part of a previous bug discussion iirc but we ended up agreen that it's feature | 12:28 |
| amoralej_ | as it was documented in the spec | 12:28 |
| jgilaber | ok to move it to the watcher tracker then? or close it altogether and discuss the possible change in PTG? | 12:28 |
| amoralej_ | we can tag it as a feature ans set as triaged, yes | 12:29 |
| amoralej_ | and move it to watcher | 12:29 |
| amoralej_ | or close if we prefer not to track features as bugs, both wfm | 12:29 |
| sean-k-mooney | ya we found the orgianal spec and unfortuenly the non intuitive bevhior si what was desgined | 12:30 |
| sean-k-mooney | so we need to decieed how to adress that as a fetarue rather then a bug | 12:30 |
| jgilaber | so we keep the bug as wishlist for watcher? | 12:31 |
| sean-k-mooney | we can | 12:31 |
| amoralej_ | given that we are changing the state machine i think we should treat it as a spec feature ? | 12:31 |
| sean-k-mooney | but we will likely close it when we figure out how to impove it | 12:31 |
| sean-k-mooney | amoralej_: that my inclination but im ok with a wishlist bug to just track it in the backlog until it becomes a priority | 12:32 |
| amoralej_ | i think that's a good approach, keep it open as whishlist or rfe until to not forget it until we start the feature process | 12:32 |
| sean-k-mooney | amoralej_: unless you plan to start workign on it now | 12:32 |
| amoralej_ | nop | 12:32 |
| jgilaber | ack, sounds good I'll do that | 12:32 |
| sean-k-mooney | ya so let add an rfe tag too | 12:32 |
| sean-k-mooney | and ya we can dicuss at the ptg or a future meeting if it makes sense to expidite but it not super urgent to fix this as long as it clearly docuemented | 12:33 |
| sean-k-mooney | jgilaber: ithink you said there were two bugs? what is the ohter | 12:33 |
| jgilaber | #link https://bugs.launchpad.net/watcher-tempest-plugin/+bug/2134487 | 12:34 |
| jgilaber | this one comes from a ci failure | 12:34 |
| jgilaber | I'm not sure how often that happends | 12:34 |
| jgilaber | I don't remember seeing that test failing too much | 12:34 |
| jgilaber | dviroel, do you remember any details on this bug since you reported it? | 12:35 |
| dviroel | trying to remember but, i don't think I see that recently in ci too | 12:35 |
| chandankumar | may be we close it? If we see again, reopen it? | 12:37 |
| jgilaber | I would either do that or set it as low and revisit in the future, any thoughts? | 12:38 |
| dviroel | yeah, maybe set to incomplete and wait for more logs/failures | 12:39 |
| amoralej_ | yes, i think setting it as incomplete is good | 12:39 |
| jgilaber | ack, done | 12:40 |
| sean-k-mooney | is this realtede to the previous issue | 12:40 |
| sean-k-mooney | or is "CANCELLED instead of SUCCEEDED" a coincidnece | 12:41 |
| jgilaber | I don't think so, this is refering to the state of an audit | 12:41 |
| jgilaber | the other was about the action plan of a continous audit | 12:41 |
| amoralej_ | yes, it's audit, not action plan | 12:41 |
| sean-k-mooney | ok but is the audit a continuous one or a onehost | 12:42 |
| amoralej_ | i was also remembering another issue related with cancelled state and continuous but that was also for actionplans | 12:42 |
| sean-k-mooney | im wondierign if we cancel runing autis when the next interval triggers | 12:42 |
| sean-k-mooney | but superceed compelted ones | 12:43 |
| chandankumar | https://github.com/openstack/watcher-tempest-plugin/blob/49e68f7556c35248d3c960e67ecbb927183031bc/watcher_tempest_plugin/tests/api/admin/test_action.py#L74 | 12:43 |
| jgilaber | I think by default it creates a oneshot audit | 12:44 |
| jgilaber | in https://github.com/openstack/watcher-tempest-plugin/blob/49e68f7556c35248d3c960e67ecbb927183031bc/watcher_tempest_plugin/tests/api/admin/test_action.py#L37 | 12:44 |
| jgilaber | looking at https://github.com/openstack/watcher-tempest-plugin/blob/49e68f7556c35248d3c960e67ecbb927183031bc/watcher_tempest_plugin/tests/common/base.py#L154 | 12:44 |
| dviroel | yeah, when no set, it defaults to ONESHOT | 12:45 |
| chandankumar | yes it is oneshot audit | 12:45 |
| sean-k-mooney | ok thats defaulted at the api level? | 12:46 |
| sean-k-mooney | i was not seeign it really clearly | 12:46 |
| sean-k-mooney | in teh code | 12:46 |
| amoralej_ | not in the api | 12:46 |
| amoralej_ | it's jgilaber link | 12:47 |
| amoralej_ | setting ONESHOT | 12:47 |
| amoralej_ | in create_audit method | 12:47 |
| sean-k-mooney | there are 2 implemation of create_audit | 12:47 |
| sean-k-mooney | https://github.com/openstack/watcher-tempest-plugin/blob/master/watcher_tempest_plugin/services/infra_optim/v1/json/client.py#L140-L151 | 12:48 |
| sean-k-mooney | that one does nto default anything | 12:48 |
| sean-k-mooney | and that is the one we are using | 12:48 |
| amoralej_ | that's called by https://github.com/openstack/watcher-tempest-plugin/blob/49e68f7556c35248d3c960e67ecbb927183031bc/watcher_tempest_plugin/tests/common/base.py#L154 iiuc | 12:48 |
| dviroel | right | 12:49 |
| sean-k-mooney | ok | 12:50 |
| sean-k-mooney | its coming in via the leper mixing | 12:50 |
| amoralej_ | yes | 12:50 |
| sean-k-mooney | then ya | 12:50 |
| sean-k-mooney | i dont know of anything that woudl cause it to be canceld | 12:50 |
| sean-k-mooney | other then a direct api request for one shot | 12:50 |
| sean-k-mooney | beyond a restart | 12:50 |
| dviroel | yeah, the WatcherHelperMixin was added in the commit that I mentioned, but I remember that the job started failing after we merged that change | 12:51 |
| dviroel | but we need more failures and logs to debug | 12:51 |
| amoralej_ | yep | 12:51 |
| amoralej_ | restarting the decision engine i.e. can cancel audits, but i guess this is not the case | 12:51 |
| sean-k-mooney | of ya we need to have it fail then determin how it when to canceled | 12:51 |
| sean-k-mooney | amoralej_: well its possible there was a restart but we woudl see that in the logs | 12:52 |
| amoralej_ | yes, we need logs | 12:52 |
| sean-k-mooney | we print the config optiosn at start up so there coudl have been an OOM event | 12:52 |
| sean-k-mooney | or somethign like that | 12:52 |
| sean-k-mooney | i guess for now lets mark it incomplete | 12:53 |
| sean-k-mooney | since we dont have teh data or a theory of how it can happen outside fo the descion engine restarting | 12:54 |
| jgilaber | ack, done | 12:54 |
| jgilaber | we have one last topic | 12:54 |
| jgilaber | #topic PoC on nova and prometheus fixtures | 12:54 |
| jgilaber | #link https://review.opendev.org/c/openstack/watcher/+/980257 | 12:54 |
| amoralej_ | yes, i added last minute one :) | 12:55 |
| amoralej_ | when working in the scale bugs i ended up creating simple fake nova and prometheus services | 12:55 |
| amoralej_ | otherwise i was working blind | 12:55 |
| amoralej_ | i've thought those could be part also of the functional test | 12:56 |
| amoralej_ | so that we can use them both integrated in the functional tests or as standalone servers | 12:56 |
| sean-k-mooney | amoralej_: oh you made a start | 12:56 |
| amoralej_ | but before going beyong i wanted to check if i'm going in the right direction | 12:56 |
| sean-k-mooney | i can try and take a look befor i go on pto | 12:57 |
| amoralej_ | in that early state it has already be useful for me to replicate scalability issues locally | 12:57 |
| amoralej_ | i used flask to create them | 12:57 |
| sean-k-mooney | ill be way monday->wednesday next week | 12:57 |
| amoralej_ | it gives features as ssl, etc... | 12:57 |
| sean-k-mooney | amoralej_: im not sure we want to dothat | 12:57 |
| sean-k-mooney | i.e. create a webserver | 12:57 |
| sean-k-mooney | we dont wnat to test ssl ectrs in the fucntional tests | 12:57 |
| amoralej_ | yes, that's my question :) | 12:58 |
| amoralej_ | yes, but as said before, i think it'd be good to have a tool also useful beyond automated functional tests | 12:58 |
| amoralej_ | i.e. replacing actual services in a local deployment | 12:58 |
| sean-k-mooney | perhaps | 12:58 |
| amoralej_ | for devel, replicate issues, etc... | 12:58 |
| sean-k-mooney | but at that point it likely shoudl not be in the watcher repo | 12:58 |
| amoralej_ | other option would be that, to create two totally different things | 12:59 |
| amoralej_ | and not being part of watcher tests | 12:59 |
| sean-k-mooney | we dont currently use falks by the way | 12:59 |
| sean-k-mooney | *flask | 12:59 |
| amoralej_ | going to contrib folder or to even other repo | 12:59 |
| amoralej_ | yes, i know | 12:59 |
| amoralej_ | we don't have much time now | 13:00 |
| amoralej_ | but i wanted to open the discussion | 13:00 |
| sean-k-mooney | yep maybe a good topic for the ptg | 13:00 |
| sean-k-mooney | im not really afan of the idea of having giant xml files either by the way | 13:00 |
| sean-k-mooney | i dont like howe we are curertny using the them in our unit tests | 13:01 |
| sean-k-mooney | but it felt to risky to just rip them out | 13:01 |
| sean-k-mooney | but i was hoping we coudl entirly get away form that in the funtional tests | 13:01 |
| sean-k-mooney | and instead compute the moduels programaticly | 13:01 |
| amoralej_ | i need to join another mtg now | 13:02 |
| sean-k-mooney | cool lets loop back to this | 13:02 |
| jgilaber | thanks amoralej_ looks interesting but we're out of time, we can move discussion to the patch or defer to PTG | 13:02 |
| amoralej_ | but i think having the option to use different models or metrics as data files is good :) | 13:02 |
| dviroel | ack, we can also discuss more in the patch | 13:02 |
| jgilaber | any last minute topic for opens discussion? | 13:02 |
| jgilaber | I see dviroel++ has voluntereed for next meeting so we can wrap up here | 13:03 |
| jgilaber | thanks everyone! | 13:03 |
| jgilaber | #endmeeting | 13:03 |
| opendevmeet | Meeting ended Thu Mar 12 13:03:47 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-03-12-12.00.html | 13:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-03-12-12.00.txt | 13:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2026/watcher.2026-03-12-12.00.log.html | 13:03 |
| dviroel | jgilaber++ | 13:03 |
| chandank` | jgilaber++ | 13:03 |
| amoralej_ | thanks jgilaber++ | 13:04 |
| opendevreview | Merged openstack/watcher master: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/960265 | 13:18 |
| dviroel | jgilaber ^ so our last fix merged, we can update the release patch | 13:27 |
| dviroel | jgilaber: please check/review the 2 release patches: | 14:14 |
| dviroel | https://review.opendev.org/c/openstack/releases/+/979622 | 14:14 |
| dviroel | https://review.opendev.org/c/openstack/releases/+/979623 | 14:14 |
| chandankumar | dviroel: Do we want to include https://review.opendev.org/c/openstack/watcher/+/979137 grenade patch in rc1 patch or will backport later? | 14:19 |
| jgilaber | will do dviroel, thanks | 14:21 |
| dviroel | chandankumar: we can backport later if needed, but not a fix that we need to include in the release imho | 14:22 |
| jgilaber | +1, since it's a CI fix | 14:23 |
| jgilaber | I think we can merge today anyway, I'm +2 on the new grenade job | 14:23 |
| jgilaber | dviroel, did you plan to review it or should I +W as well? | 14:24 |
| dviroel | ack, i may review today too | 14:24 |
| jgilaber | ack | 14:24 |
| dviroel | reviewing now | 14:24 |
| chandankumar | ack | 14:25 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2025.2: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980298 | 14:42 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2025.1: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980299 | 14:43 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2024.2: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980300 | 14:43 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2024.2: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980300 | 16:11 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2025.2: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980298 | 16:12 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2025.1: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980299 | 16:13 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2025.1: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980299 | 17:54 |
| opendevreview | Joan Gilabert proposed openstack/watcher stable/2024.2: Handle missing fields building storage model XML or list https://review.opendev.org/c/openstack/watcher/+/980300 | 17:54 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!