12:01:19 <amoralej> #startmeeting Watcher meeting - 2025-01-09
12:01:19 <opendevmeet> Meeting started Thu Jan  9 12:01:19 2025 UTC and is due to finish in 60 minutes.  The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot.
12:01:19 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
12:01:19 <opendevmeet> The meeting name has been set to 'watcher_meeting___2025_01_09'
12:01:33 <jneo8> \o/
12:01:38 <jgilaber> o/
12:01:45 <hemanth> o/
12:01:47 <marios> hello o/
12:01:55 <chandankumar> o/
12:02:06 <dviroel> hi o/
12:02:12 <amoralej> #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting meeting agenda
12:02:27 <amoralej> please add your topics to the list if you have something else
12:02:30 <sean-k-mooney> o/
12:02:59 <amoralej> let's start with the agenda items for today
12:03:21 <amoralej> #topic     (marios) let's revisit the backports discussion prompted by https://review.opendev.org/c/openstack/watcher/+/937823/comment/eb0827ce_db971940/
12:03:47 <marios> o/ hello, so there has been some discussion on gerrit already in response to jneo8 patches but...
12:04:06 <rlandy> o/ (apologies for joining late)
12:04:25 <marios> tl;dr no objection to backports but they will be taken on a case by case basis. some of the proposed patches need to go on 2024.2 first and some of them need discussion because the proposed backports are still ongoing work in this cycle
12:04:54 <marios> the topic of backports has come up in the past and we werent' sure if folks were interested since the project had very low volume in the last few cycles
12:05:20 <marios> so it seems jneo8 is interested in deploying 2024.1 and will request a point release once the rquired backports are agreed and merged
12:05:39 <marios> k... thats the intro :) anyone have something to add to this topic comments concerns
12:06:37 <sean-k-mooney> one comment
12:06:56 <sean-k-mooney> the backprots seam to be motivated by supproting python 3.12
12:07:15 <jneo8> Yes
12:07:15 <sean-k-mooney> even if we back prot the rquired patches to make it work it wont add offical support to older brnahces
12:07:33 <sean-k-mooney> the first release to supprot python 3.12 upstream will be 2025.1
12:07:49 <sean-k-mooney> on older relases it will be conserded experimental
12:07:53 <jneo8> I see, but when will be the release time for 2025.1?
12:08:14 <sean-k-mooney> in about 6-8 weeks
12:08:16 <amoralej> https://releases.openstack.org/epoxy/schedule.html
12:08:28 <sean-k-mooney> for what it worth master does nto fully work on 3.12 yet
12:08:45 <sean-k-mooney> the inital paches i wrote to try and fix the eventlet issue sdidnt actully fix it properly
12:09:03 <sean-k-mooney> we still see blocking calls on the main loop
12:09:15 <sean-k-mooney> so we still have work to do to properly supprot it
12:10:07 <jneo8> I think making master branch working properly make sense.
12:11:24 <amoralej> so, if the goal is to have it running on python 3.12, should we wait at least until master is working before moving on with any backport ?
12:11:36 <amoralej> any backport related to 3.12, i mean
12:11:45 <marios> well i think maybe the backports will no longer be required
12:11:59 <marios> if the target is 2025.1
12:12:08 <amoralej> yep, likely by then the best option may be to move to 2025.1 ...
12:12:09 <jneo8> So look like the back port should be discuss after master branch support python3.12 properly, it maybe too early to do it now.
12:12:36 <sean-k-mooney> we can back port some of the fixes as they are unrelated to 3.12
12:12:57 <sean-k-mooney> but i think we shoudl wiat for the master supprot to be finalised before the 3.12 specic ones are considerd
12:13:31 <amoralej> I'd sugest to set the ones related to 3.12 as WIP or in some way that it is clear which ones are waiting for 3.12 support
12:13:33 <jneo8> I see, that make sense.
12:14:16 <sean-k-mooney> https://review.opendev.org/c/openstack/watcher/+/938429 specificaly is the one that shoudl wait
12:14:25 <amoralej> or even abandon them and restore later, whatever the owner prefers
12:14:50 <jneo8> I can abandon them first.
12:16:21 <amoralej> wrt proposing to 2024.2, i understand that's discussed in the reviews, no need to discuss here?
12:17:20 <marios> yeah  ithink so... some of the backports which are otherwise good to go (not related to 3.12 for example) are blocked because they need to be proposed to 2024.2 first
12:17:30 <sean-k-mooney> right, the summary is that stable poicly does nto allow skiping branches
12:17:41 <amoralej> yep, that's important
12:17:45 <amoralej> ack
12:17:54 <jneo8> ack
12:18:35 <sean-k-mooney> one final comment https://review.opendev.org/c/openstack/watcher/+/938435 cant be backported because it increase the requried version of oslo.utils
12:19:10 <sean-k-mooney> its not needed however as  datetime.utcnow() is only deprecated not removed in 3.12
12:19:20 <sean-k-mooney> again that captured in the review
12:19:41 <amoralej> yes, and oslo.utils-7.0.0 is caracal release
12:20:32 <amoralej> actually, caracal is 2024.1 so https://review.opendev.org/c/openstack/watcher/+/938435/1/requirements.txt may be backportable to 2024.1
12:20:47 <amoralej> if that value is correct
12:20:52 <sean-k-mooney> its not
12:21:00 <sean-k-mooney> with out this change we supprot 3.36
12:21:12 <amoralej> https://github.com/openstack/requirements/blob/stable/2024.1/upper-constraints.txt#L600
12:21:13 <sean-k-mooney> we are not allowed to raise min verison in a backprot
12:21:18 <sean-k-mooney> even if its released
12:21:25 <amoralej> ah, didn't know that!
12:21:46 <amoralej> i thought it was possible as soon as it was part of the release ...
12:21:47 <amoralej> ok
12:21:55 <amoralej> good to know
12:22:02 <sean-k-mooney> the expction is for security reaons
12:22:11 <sean-k-mooney> but that does nto apply here
12:22:16 <amoralej> yep
12:22:39 <amoralej> anything else about backports or we can move to the next topic ?
12:22:49 <sean-k-mooney> not from me :)
12:23:08 <jneo8> not from me. And thanks for the input!!
12:23:16 <amoralej> gook, let's move on then
12:23:27 <amoralej> #topic (amoralej) call for a triage session to triage existing bugs in launchpad
12:24:02 <amoralej> We have been discussing about how to proceed with the existing bugs reported in launchpad for watcher
12:24:13 <amoralej> #link https://bugs.launchpad.net/watcher
12:24:44 <amoralej> some of them have been there for some time and have not been triaged, some were triaged long time ago, and may be worthy to revisit
12:25:24 <marios> 43 is not an impossible number (but still some effort to try and understand and triage these)
12:25:33 <amoralej> so, the proposal is to schedule a triage session to work together and coordinate on irc so that we can see what can be closed and wat should be priorized
12:26:17 <amoralej> wdyt, any other proposal?
12:26:26 <amoralej> yep, i'd say 43 is doable
12:26:26 <sean-k-mooney> we might want to have a google meet as well
12:26:44 <rlandy> amoralej: there are also these: https://bugs.launchpad.net/watcher-dashboard
12:26:45 <marios> isnt there a foundation approved one jitsi?
12:26:53 <sean-k-mooney> we can use any tool
12:26:56 <amoralej> right
12:27:03 <amoralej> #link https://bugs.launchpad.net/watcher-dashboard
12:27:09 <amoralej> thanks for the reminder rlandy
12:27:23 <sean-k-mooney> my point was more we may need higher bandwith
12:27:33 <sean-k-mooney> we can start with irc
12:27:45 <sean-k-mooney> and perhaps have a second pass with a higher bandwith mediam after
12:27:46 <marios> ack yes i think call will be easier to coordinate we can get through more bugs
12:27:52 <rlandy> +1
12:27:57 <amoralej> from community pov, is fine to use gmeeting? i had no idea about how to create a meeting with opendev jitsi, but it'd be good
12:28:13 <marios> i guess as longas anyone interested can access the call ... ?
12:28:25 <sean-k-mooney> yes we use it form time to time for nova
12:28:35 <sean-k-mooney> the imporant things is to make it open to all
12:28:54 <amoralej> good, then I'd say google meet is the easier path ...
12:28:54 <sean-k-mooney> we have also used other toosl in the past but mostly we try to be irc first
12:29:15 <rlandy> I think one gmeet will be enough to set the process in motion
12:29:19 <rlandy> irc after that
12:30:03 <amoralej> gmeet in first call could be good for coordination
12:30:23 <amoralej> and I think it'd be good to schedule it asap, maybe next week ?
12:30:50 <marios> maybe this time slot on tuesday?
12:31:01 <amoralej> wfm
12:31:33 <rlandy> +1
12:31:35 <sean-k-mooney> sure we can make that work
12:32:31 <amoralej> so we can give that slot as agreed? anyone wants to propose a different one?
12:33:03 <jgilaber> +1 from me
12:33:21 <marios> we should announce the meeting on the mailing list
12:33:27 <amoralej> #agreed we will have a Watcher triage session on tuesday 14th at 12:00 UTC
12:33:35 <marios> and provide a link for anyone who wnats to join
12:33:40 <amoralej> yes, we will also announce in mailing list
12:33:41 <marios> (we can coordinate in this irc channel)
12:34:20 <amoralej> wrt irc only vs gmeet (+irc) vs other
12:35:52 <amoralej> i understood there is agreement on gmeet ?
12:36:10 <marios> if someone cannot join they should be able to reach us somewhere
12:36:15 <marios> is what i was thinking of
12:36:27 <amoralej> #agreed google meet + irc will be use for coordination
12:36:33 <marios> but yeah the meeting will be gmeet
12:37:11 <amoralej> so, i think we are done with this topic
12:38:09 <amoralej> next topic in the agenda is from jneo8 about support for 3.12, it was covered in previous one or there is something else you want to discuss?
12:39:05 <sean-k-mooney> perhaps a birfe comment on the current state of things
12:39:07 <jneo8> No, I think it's been covered.
12:39:25 <sean-k-mooney> on master we see eventlet related issues that only happen on 3.12
12:39:36 <amoralej> out of curiosity, other openstack projects work fine (experimentally) with 3.12 and 2024.1 branch?
12:39:41 <sean-k-mooney> sepcificly on ubuntu noble
12:39:50 <sean-k-mooney> yes
12:39:52 <sean-k-mooney> for the most part
12:40:06 <sean-k-mooney> the issue with watcher is it is mixing 3 concurancy modeles at once
12:40:28 <sean-k-mooney> its using native treads va APSchduler + eventlets + asyncio
12:40:34 <hemanth> I think magnum has issues as well with 3.12 and 2024.1 branch
12:40:51 <sean-k-mooney> ya not all project work with 3.12 in 2024.1
12:41:10 <amoralej> ack
12:41:16 <sean-k-mooney> it was only added to the testing runtim as experimetal in 2024.2 and requried this cycle for 2025.1
12:41:53 <sean-k-mooney> almost none of openstack uses APSchduler and its the interaction betwen that and eventlet and python 3.12 that is broken
12:42:09 <sean-k-mooney> which is why nova/neutron/glance are not affected in the same way
12:42:19 <amoralej> i expected that it would be in worse situation, tbh
12:42:55 <amoralej> so i think it's clear enough and we can move to next topic or we will get out of time
12:43:36 <amoralej> #topic (marios): update on prometheus datasource https://review.opendev.org/c/openstack/watcher/+/934423
12:44:12 <marios> thanks, so some summary of the current state. i'd still like to merge this 'real soon now'.
12:44:32 <marios> there have been some review requests in the last couple of days aroudn 2 themes that i am working on for v30 coming today or tomorrow
12:44:57 <marios> one theme is to add a retry when we can't resolve the prometheus exporter hostname in the internal fqdn_instance_map
12:45:10 <marios> retry means rebuild the instance maps and retry the query before giving up
12:45:34 <marios> the other theme is around the client config options naming and being consistent with other projecs
12:45:57 <marios> so removing the prometheus_ prefix and using the 'standard' name for the tls options
12:46:28 <marios> sean-k-mooney: my plan is to implement the naming changes but pusing the oslo config validation as future work agree?
12:46:42 <marios> not decided if max_min inversion will be included in v30 or also pushed
12:46:52 <marios> but agree on the client opts lets get those right before merge
12:46:58 <marios> (also spliting the host and port )
12:47:23 <marios> so any comments or questions on this or any other topic related to this patch ?
12:48:01 <sean-k-mooney> ok well moving the min_max around is purly internal
12:48:07 <marios> right
12:48:08 <sean-k-mooney> so that can be a follow up
12:48:48 <sean-k-mooney> i think we can proceed with that you plan to push up soon
12:49:08 <sean-k-mooney> so for now lets proceed with the split your propsoeding and we can take it form there
12:49:19 <marios> ack
12:50:20 <amoralej> we are done with the topic?
12:50:28 <marios> from my side yes
12:51:09 <amoralej> so, i think we can move to the last one
12:51:28 <amoralej> #topic (hemanth): noisy neighbour strategy not working as cpu_l3_cache metric is not collected
12:51:32 <amoralej> #link https://bugs.launchpad.net/ceilometer/+bug/2081128
12:51:59 <amoralej> hemanth, you want to introduce the topic ?
12:52:21 <hemanth> This is more of a question if anyone is using the noisy neighbour strategy, if so how?
12:52:44 <hemanth> The strategy relies on metric cpu_l3_cache which is not collected in ceilometer/gnocchi
12:52:50 <sean-k-mooney> we prbably shoudl deprecated it in the curent form
12:53:08 <sean-k-mooney> if i recall correctly we removed the fucntionatly related to thsi form livbirt too
12:53:14 <sean-k-mooney> or rather nova
12:53:42 <hemanth> yes
12:54:03 <sean-k-mooney> i guess it might be posible that the stats were collected soem other way to feed them in to gnooci
12:54:08 <amoralej> cpu_l3_cache is still inthe ceilometer documentation https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#openstack-compute i guess that should be considered a bug?
12:54:49 <amoralej> a mean, a documentation bug :)
12:54:52 <sean-k-mooney> likely yes
12:55:28 <opendevreview> Merged openstack/watcher stable/2024.1: Update .gitreview for stable/2024.1  https://review.opendev.org/c/openstack/watcher/+/913343
12:55:29 <opendevreview> Merged openstack/watcher stable/2024.1: Update TOX_CONSTRAINTS_FILE for stable/2024.1  https://review.opendev.org/c/openstack/watcher/+/913344
12:56:34 <sean-k-mooney> i think we shoudl likely mark the stagey as deprecated and reach out on the mailing lsit to see if anyoen is ueing it
12:56:39 <sean-k-mooney> we can also ask at the ptg
12:56:53 <hemanth> ok sure
12:56:57 <sean-k-mooney> if there is no feedback form user we can look to remove it or reimplement it next cycle
12:57:07 <hemanth> ack
12:57:56 <amoralej> and maybe report it as a bug to ceilometer too, to check if they can get the metric in some other way or remove it from doc
12:58:29 <sean-k-mooney> for what its worht there are other facotrs that can be used ot detect noisy neibghors  l3 cache usage is not really a good metric for this IMO
12:58:43 <sean-k-mooney> i think that off topci however
12:58:52 <amoralej> so, we will wait to PTG before marking it as deprecated, or somethign we can initiate it first ?
12:59:08 <amoralej> yep, we are almost out of time
12:59:17 <sean-k-mooney> no i think we shoudl mark it deprecated now
12:59:32 <sean-k-mooney> but we shoudl not look at remvoign it untl we have a wider dicussion
12:59:37 <amoralej> yep
12:59:53 <sean-k-mooney> per the SLURP policy
12:59:54 <hemanth> I can submit a PR to deprecate early next week
13:00:09 <amoralej> thanks hemanth, that'd be great
13:00:11 <sean-k-mooney> we are not allowed ot remvoe feature without adverstiing the deprecation in a SLURP release
13:00:43 <amoralej> so, unless there is some last minute topic, i'm closing the meeting
13:01:21 <amoralej> then thanks all for participating! see you on tuesday!
13:01:29 <sean-k-mooney> o/
13:01:31 <marios> thank you amoralej \o
13:01:32 <amoralej> #endmeeting