| marios | hello folks do we have the irc meet? anyone else around? | 12:01 |
|---|---|---|
| marios | or did i get the time wrong ;) | 12:01 |
| sean-k-mooney | o/ | 12:01 |
| rlandy | yes - amoralej is the chair | 12:01 |
| rlandy | waiting a few for amoralej | 12:01 |
| marios | k i will run it today looks like some miscommunication? | 12:01 |
| rlandy | otherwise we can start | 12:01 |
| marios | maybe amoralej something cam eup or whatever | 12:01 |
| amoralej | damn | 12:02 |
| marios | i'll start it and he can continue if he comes later ? | 12:02 |
| marios | ah k | 12:02 |
| marios | o/ | 12:02 |
| amoralej | sorry i toitally forgot it | 12:02 |
| rlandy | you ok to run it? | 12:02 |
| amoralej | sure | 12:02 |
| jgilaber | o/ | 12:02 |
| marios | you are 2 mins late i once forgot the whole thing ;) | 12:02 |
| amoralej | lemme create the mtg | 12:02 |
| amoralej | I whould have if you don't ping me :) | 12:02 |
| dviroel | :) | 12:02 |
| rlandy | that's my job | 12:02 |
| rlandy | the ping nagger | 12:03 |
| amoralej | #startmeeting Watcher meeting - 2025-01-16 | 12:03 |
| opendevmeet | Meeting started Thu Jan 16 12:03:21 2025 UTC and is due to finish in 60 minutes. The chair is amoralej. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:03 |
| opendevmeet | The meeting name has been set to 'watcher_meeting___2025_01_16' | 12:03 |
| amoralej | please, add your topics to https://etherpad.opendev.org/p/openstack-watcher-irc-meeting | 12:03 |
| marios | o/ | 12:04 |
| amoralej | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting meeting agenda | 12:04 |
| amoralej | let's start with the first topic | 12:04 |
| amoralej | #topic (rlandy): with Martin Kopec changing roles, we will need to new cores for watcher-tempest-plugin | 12:05 |
| amoralej | rlandy, you want to introduce the topic? | 12:05 |
| rlandy | martin has switched roles | 12:06 |
| rlandy | as such we will need to propose other cores for watcher-tempest-plugin | 12:06 |
| rlandy | in time | 12:06 |
| rlandy | this is just a team fyi | 12:06 |
| marios | i think we have a general problem with lack of cores (basically we now have just one active core sean-k-mooney ) | 12:07 |
| marios | in the tempest-plugin case, until now we had martin as well (it is the exception) so here we are also down to one core | 12:07 |
| marios | fetching the group for reference... | 12:07 |
| sean-k-mooney | well realisticlaly until late december we didnt have any active cores | 12:08 |
| sean-k-mooney | so martin has only been in the list for a few weeks and they were on pto for part of that | 12:08 |
| sean-k-mooney | i added them on the 28th of november | 12:09 |
| marios | #info https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3d4d14 watcher-tempest-core gerrit group | 12:09 |
| marios | #info https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3d4d14,members watcher-tempest-core members | 12:09 |
| sean-k-mooney | so i was going to propos that we do a review of core membership the first week of febuary | 12:09 |
| rlandy | +1 | 12:10 |
| dviroel | +1 | 12:10 |
| marios | sounds good i think we have discussed doing this around this timeframe before (in the context of things not being able to merge in watcher and dashboard repos) | 12:10 |
| sean-k-mooney | my plan was to review the review stats in https://openstack.biterg.io/app/dashboards and propsoe a set of potitla cores to each of the watcher group to the mailing list | 12:10 |
| sean-k-mooney | if we want i can try and prepare that email before the next meeting | 12:11 |
| sean-k-mooney | and then we can wait for feedback and dicusss in the next meeting | 12:11 |
| amoralej | #agreed we will do a review of core membership the first week of febuary | 12:12 |
| sean-k-mooney | if there are no objects by the meeting after that on jan 30th | 12:12 |
| sean-k-mooney | i can implemnt the changes | 12:12 |
| sean-k-mooney | *objections | 12:12 |
| amoralej | i guess that's a good plan | 12:12 |
| amoralej | so, we can move to next topic? | 12:13 |
| amoralej | i will open next one | 12:14 |
| amoralej | #topic (marios): update on prometheus datasource | 12:14 |
| marios | thanks amoralej | 12:14 |
| amoralej | #link https://review.opendev.org/c/openstack/watcher/+/934423 | 12:14 |
| marios | as discussed last week there were some requested changes around the auth options, making the fqdn_instance_map more like a cache with a rebuild & retry at least once | 12:15 |
| marios | those and some other smaller bits where implemented now (including removing the 'prometheus_' prefix on the config options) | 12:15 |
| marios | there have been no further comments or requests yet and we have a +2 from sean-k-mooney and +1 from various other folks who have requested changes | 12:16 |
| marios | thanks again to everyone for all your suggestions and improvements. | 12:16 |
| marios | since we have the core issue, I would propose that if there are no negative comments by end of next week we can merge it? | 12:17 |
| * marios checks date on the patch | 12:17 | |
| marios | (I discussed that bit about merge with sean-k-mooney privately already | 12:17 |
| sean-k-mooney | yep | 12:17 |
| marios | yeah so i updated that jan 10th | 12:17 |
| amoralej | +1 to merge it asasp | 12:18 |
| sean-k-mooney | so for singlel core approval i want ot 1 leave time for other to review ideally at least 2 weeks | 12:18 |
| marios | i'd say tomorrow is 1 week , so next friday sound good sean-k-mooney ? | 12:18 |
| sean-k-mooney | 2 see reviews form non cores with no objections | 12:18 |
| amoralej | btw, i started some work to integrate that in a deployment tool and I'm already relying in the config options set in latest PS :) | 12:18 |
| sean-k-mooney | and 3 adress it by buildign out the core team so it is not requried long term | 12:18 |
| sean-k-mooney | @marios yes the end of next week was what i had in mind so either after the next team meeting or fiday | 12:19 |
| marios | #info planning to merge https://review.opendev.org/c/openstack/watcher/+/934423 by next friday 24 unless there are objections | 12:20 |
| marios | so amoralej has already started iterating with the instance work on top | 12:21 |
| marios | i wouldn't want it slipping further into february to merge i mean | 12:21 |
| marios | thanks, that's all i had on this topic amoralej if there are no further comments we can move on? | 12:21 |
| amoralej | actually my instance work is next topic :) | 12:21 |
| amoralej | #topic (amoralej) add instance metrics into prometheus datasource | 12:22 |
| amoralej | #link https://review.opendev.org/c/openstack/watcher/+/938893/ | 12:22 |
| amoralej | this is mainly a call for review | 12:22 |
| amoralej | Given that the merge of previous one is approaching I'd like to also get this one reviewed when you have a chance | 12:22 |
| marios | i think its already looking good amoralej thanks for jumping on that | 12:23 |
| amoralej | it is much simpler that the one adding the datasource so i hope will be faster to review | 12:24 |
| sean-k-mooney | the main thing that i think is missing (and coudl be in a follow up patch) | 12:25 |
| sean-k-mooney | is i would like use to also extend the new tempest job to start testign with the new datasouce | 12:26 |
| sean-k-mooney | that does require work in the tempest plugin but we shoudl at elast enabel/configure the new data source sooner rather then later once its merged | 12:26 |
| marios | yes big +1 | 12:27 |
| marios | amoralej: has been testing on his env with the datasource but we should get into ci asap | 12:27 |
| amoralej | yes, +1 for me too | 12:27 |
| sean-k-mooney | im ok to defer the decion on if we start doing that in https://review.opendev.org/c/openstack/watcher/+/938893 or a follow up patch | 12:27 |
| amoralej | as you said, I'd propose to make that follow up patch | 12:27 |
| sean-k-mooney | but it would be nice to work on that before we merge it | 12:27 |
| marios | yeah i think it can/should be different patch | 12:28 |
| rlandy | chandankumar is close to being able to add prometheus metrics in the plugin | 12:28 |
| sean-k-mooney | ack | 12:28 |
| rlandy | so we should be able to extend the test shortly | 12:28 |
| amoralej | for me it is a matter of time, if we can add it soon, no problem in including it into the patch | 12:28 |
| sean-k-mooney | ok we can defer this to gerrit review and see how the various efforts come togheter | 12:28 |
| sean-k-mooney | i think we are generally in agreement on the direction | 12:28 |
| dviroel | yes | 12:29 |
| amoralej | ok | 12:30 |
| amoralej | so that's it for this one | 12:30 |
| amoralej | moving to next one | 12:30 |
| amoralej | #topic (rlandy) reminder bug triage continuation on Tuesday, January 21, 2025 12pm UTC (gmeet and IRC as sent on the ML) | 12:30 |
| amoralej | i guess this is just a reminder | 12:31 |
| amoralej | #info next bug triage session is on Tuesday, January 21, 2025 12pm UTC, details for anyone interested are in the mailing list | 12:31 |
| marios | rlandy: did a great job running that but... do you want to do it again or would you like us to rotate? | 12:31 |
| amoralej | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5KZUHLXUOGTBXMQ4HERO52XB7I5A3HXI/ | 12:32 |
| marios | rlandy: do you want us to rotate the chair on that? ^^ | 12:33 |
| rlandy | either way | 12:34 |
| rlandy | I can finish off this one - and then next time we do it, someone else can take it | 12:34 |
| rlandy | ie: I'll take this Tuesday's | 12:34 |
| marios | sounds good thank you rlandy | 12:34 |
| marios | we should be able to get through a fair chunk after next meeting | 12:34 |
| marios | we had some overhead on the first one ;) | 12:35 |
| amoralej | btw, i found a great exercise to keep learning about the status of watcher | 12:35 |
| amoralej | yep, we well go faster next time :) | 12:35 |
| amoralej | #topic next chair | 12:36 |
| amoralej | any volunteer to chair next meeting? i don't want to forget this time | 12:37 |
| amoralej | :) | 12:37 |
| marios | i didn't do this for a while, but if there is someone who wants to/didn't go yet i will not fight you for it | 12:37 |
| rlandy | looks like it's yours marios | 12:38 |
| marios | yup | 12:38 |
| amoralej | #action marios will chair next meeting | 12:38 |
| amoralej | #topic open floor | 12:38 |
| amoralej | out of the ones in the agenda, is there some other topic you'd like to discuss ? | 12:39 |
| sean-k-mooney | i have one minor update | 12:39 |
| sean-k-mooney | not in the adgeda | 12:39 |
| * dviroel proposing myself for chair the meeting on 30th | 12:39 | |
| amoralej | thanks dviroel | 12:39 |
| amoralej | ok, sean-k-mooney go ahead please | 12:39 |
| sean-k-mooney | #link https://bugs.launchpad.net/watcher/+bug/2086710 | 12:40 |
| sean-k-mooney | i have been looking into ^ on and off for a while | 12:40 |
| amoralej | that's an important one ... | 12:40 |
| sean-k-mooney | chattign to jayF yesterday i took a look at eventlet | 12:40 |
| sean-k-mooney | and found that the behavior was changed in | 12:40 |
| sean-k-mooney | #link https://github.com/eventlet/eventlet/pull/932 | 12:41 |
| marios | i think this randomly fails in the -strategies job right ? | 12:41 |
| sean-k-mooney | so i have filed | 12:41 |
| sean-k-mooney | #link https://github.com/eventlet/eventlet/issues/1014 | 12:41 |
| sean-k-mooney | marios: no this is not realted to -straegies | 12:41 |
| sean-k-mooney | that is a bug in the tempest plugin | 12:41 |
| sean-k-mooney | the -stragies job failure is | 12:42 |
| sean-k-mooney | #link https://bugs.launchpad.net/watcher-tempest-plugin/+bug/2090854 | 12:42 |
| marios | thank you looking at that | 12:42 |
| amoralej | sean-k-mooney, so we need to wait on the eventlet patch, righ? not fix from watcher side ? | 12:42 |
| sean-k-mooney | anyway my update is that the runtime error were previosusly asserts in eventlet prior to eventlet 0.36.0 | 12:42 |
| sean-k-mooney | at least one of the new expction seams to have been incorrect to raise | 12:43 |
| sean-k-mooney | the other exception may be valid and may be a sqlachemy 2.0 issue | 12:43 |
| sean-k-mooney | so right now its not clear if we will have to fix anything more in wathcer for this or if we will need to adress this in eventlet/sqlachmey | 12:44 |
| sean-k-mooney | ill continue to follow this and update folks btu that was what i wanted to highlight | 12:44 |
| amoralej | freeze for non-client libs is Feb 17 - Feb 21, i hope it arrives on timme | 12:45 |
| amoralej | we probably can manage it as some kind of exception otherwise | 12:45 |
| sean-k-mooney | so its also unclear why we are seeign diffent behavior on 3.9 vs 3.12 | 12:46 |
| sean-k-mooney | but yes we will have to see how we proceed if we do not have a reolution by then | 12:46 |
| sean-k-mooney | ill also not that i woudl have expected this to also impact other services like nova | 12:47 |
| sean-k-mooney | the fact its not impleis there is some watcher specific context to this | 12:47 |
| sean-k-mooney | which is why im gong to continue to look into this in paralle | 12:47 |
| *** dviroel is now known as dviroel_brb | 12:47 | |
| amoralej | as you said before, probably the specific thing is using APScheduler, right? | 12:48 |
| sean-k-mooney | yes | 12:48 |
| sean-k-mooney | i did a review of which project use it in openstack | 12:48 |
| sean-k-mooney | thereare 5 | 12:48 |
| sean-k-mooney | 3 are dead | 12:48 |
| sean-k-mooney | zuul does not use eventlet | 12:48 |
| sean-k-mooney | so watcher is the only "active" project with eventlet + apscheduler | 12:49 |
| sean-k-mooney | i also think it required the sqlachemy datastore to be used with apscheduler too | 12:49 |
| sean-k-mooney | im plannign to try and create a smaller repoducer with that combination today | 12:49 |
| amoralej | so, it'd make sense that the issue is in the combination... this is a hard one sean-k-mooney , thanks for investigating it | 12:49 |
| sean-k-mooney | a possibel PTG topic might be shoudl we remove/replace the usage of apschduler entirly | 12:50 |
| sean-k-mooney | that not somethign i think we can do this cycle however. removbing eventlet woruld also be an optin but again too large to consider for 2025.1 | 12:51 |
| sean-k-mooney | as a comunity it is somethign we shoudl evaluate however as long term apschduler is workign on a 4.0 relase that is not backwards comaptible | 12:51 |
| amoralej | we probably can follow what other active projects with similar usecase are doing | 12:52 |
| sean-k-mooney | so we woudl have to do a large migration in the next cycle ot two anyway form 3.x | 12:52 |
| amoralej | what are the alternatives to apscheduler ? | 12:52 |
| amoralej | which projects may have a similar usecase ? | 12:53 |
| sean-k-mooney | :) good question. today we useign it for two things | 12:53 |
| sean-k-mooney | 1 runing perodic tasks that done need to use it | 12:53 |
| sean-k-mooney | and the contiious audit | 12:53 |
| sean-k-mooney | the first usecase it easy to remvoe | 12:53 |
| sean-k-mooney | the second is why it was added in the first place | 12:53 |
| amoralej | yeah | 12:53 |
| sean-k-mooney | there is also a scalebality element | 12:54 |
| sean-k-mooney | currently its not clear if we can horizontally scale watcher | 12:54 |
| sean-k-mooney | i.e. to distibute the continous audits between deamons | 12:54 |
| sean-k-mooney | anyway i think we can take this out of the meeting | 12:55 |
| amoralej | right | 12:55 |
| amoralej | but it is an interesting conversation to have, out of the mtg :) | 12:55 |
| amoralej | also, we only have 5 more minutes | 12:55 |
| amoralej | anything else you want to add before closing ? | 12:55 |
| amoralej | I am closing then | 12:57 |
| amoralej | Thanks all for joining! | 12:58 |
| rlandy | thank you amoralej | 12:58 |
| jgilaber | thanks! | 12:58 |
| marios | thanks all and thanks for running amoralej o/ | 12:58 |
| amoralej | #endmeeting | 12:58 |
| opendevmeet | Meeting ended Thu Jan 16 12:58:23 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher_meeting___2025_01_16/2025/watcher_meeting___2025_01_16.2025-01-16-12.03.html | 12:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher_meeting___2025_01_16/2025/watcher_meeting___2025_01_16.2025-01-16-12.03.txt | 12:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher_meeting___2025_01_16/2025/watcher_meeting___2025_01_16.2025-01-16-12.03.log.html | 12:58 |
| *** dviroel_brb is now known as dviroel | 13:00 | |
| sean-k-mooney | just an fyi i closed https://bugs.launchpad.net/watcher/+bug/2091696 as fix released because sqlachmey 2.0 supprot was added in 2024.2 (dalmation) | 13:05 |
| rlandy | gmann: hello ... martin still has this open review: https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/936837. It has a number of +1 votes. Please could we get your review/vote on it. thank you | 14:46 |
| gmann | rlandy: +2, lgtm | 18:25 |
| sean-k-mooney | ill try an loop back to it before i finish for today | 18:32 |
| rlandy | gmann: thank you | 18:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!