| jgilaber | Hi all! Just a friendly reminder that in aprox 10 minutes our IRC meeting will start, feel free to add topics to the agenda https://etherpad.opendev.org/p/openstack-watcher-irc-meeting | 11:51 |
|---|---|---|
| jgilaber | #startmeeting watcher | 12:00 |
| opendevmeet | Meeting started Thu Oct 16 12:00:51 2025 UTC and is due to finish in 60 minutes. The chair is jgilaber. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:00 |
| opendevmeet | The meeting name has been set to 'watcher' | 12:00 |
| jgilaber | hi all, let's see who is around today | 12:01 |
| amoralej | o/ | 12:01 |
| chandankumar | o/ | 12:02 |
| rlandy | o/ | 12:02 |
| jgilaber | courtesy ping dviroel sean-k-mooney morenod | 12:02 |
| jgilaber | let's start with today's meeting agenda | 12:02 |
| jgilaber | #link https://etherpad.opendev.org/p/openstack-watcher-irc-meeting#L23 | 12:02 |
| jgilaber | feel free to add more topics to the agenda | 12:03 |
| jgilaber | let's start with the first topic | 12:03 |
| jgilaber | #topic watcher rally-jobs | 12:03 |
| chandankumar | I have added this topic. | 12:03 |
| jgilaber | chandankumar, feel free to cover it | 12:04 |
| chandankumar | I was going through rally-jobs directory in watcher repo and searching around openstack codesearch and here is what i found | 12:04 |
| jgilaber | #link https://github.com/openstack/watcher/blob/master/rally-jobs/watcher-watcher.yaml | 12:04 |
| chandankumar | In rally-openstack project, we have rally jobs defined for almost all project https://github.com/openstack/rally-openstack/tree/master/rally-jobs | 12:04 |
| jgilaber | #link https://github.com/openstack/rally-openstack/tree/master/rally-jobs | 12:05 |
| chandankumar | I compared the content of rally-openstack watcher rally content with https://github.com/openstack/watcher/blob/master/rally-jobs/watcher-watcher.yaml | 12:05 |
| chandankumar | it was kind of similar | 12:05 |
| chandankumar | the rally openstack watcher job is running against rally-openstack (receives few crs) in non-voting mode | 12:06 |
| chandankumar | job definition: https://github.com/openstack/rally-openstack/blob/master/.zuul.d/rally-task-watcher.yaml | 12:06 |
| chandankumar | and job results: https://zuul.openstack.org/builds?job_name=rally-task-watcher&skip=0 | 12:06 |
| jgilaber | #link https://zuul.openstack.org/builds?job_name=rally-task-watcher&skip=0 | 12:06 |
| chandankumar | logs of last run: https://7286c318f284276b918d-33403e99d7f0c0f7ca362b53c8ca1faf.ssl.cf2.rackcdn.com/openstack/bb656b811fba45e48d88edc1eee4659b/results/report.html | 12:06 |
| chandankumar | it only creates and deletes audit and audit templates to test crud operations. | 12:07 |
| chandankumar | I was also checking other projects. for neutron, they have two rally jobs, one for ovn and ovs using neutron repo rally directory | 12:07 |
| chandankumar | https://github.com/openstack/neutron/blob/master/zuul.d/rally.yaml and similar for cinder https://github.com/openstack/cinder/blob/master/.zuul.yaml#L164 | 12:07 |
| chandankumar | My question here is do we want to add a new rally job for watcher following cinder/neutron model and keep them running? | 12:08 |
| chandankumar | the new job will use rally files from watcher repo | 12:08 |
| jgilaber | I have no experience with rally, what would that job do/test? | 12:09 |
| chandankumar | Based on test results, I can see it is creating/deleting audit templates | 12:09 |
| amoralej | https://github.com/openstack/rally the idea is to run performance tests | 12:09 |
| amoralej | you can define an action and ask rally to run it a number of times concurrently | 12:10 |
| amoralej | or set of actions | 12:10 |
| amoralej | defining parallelism etc... | 12:10 |
| amoralej | in this case, we would be testing mostly api scalability, i.e. if it's able to create X audits, list them, etc... in how much time | 12:11 |
| jgilaber | this is using a real deployment with devstack? | 12:11 |
| chandankumar | different types of reports: https://7286c318f284276b918d-33403e99d7f0c0f7ca362b53c8ca1faf.ssl.cf2.rackcdn.com/openstack/bb656b811fba45e48d88edc1eee4659b/ | 12:11 |
| amoralej | yes, it's devstack | 12:12 |
| amoralej | https://7286c318f284276b918d-33403e99d7f0c0f7ca362b53c8ca1faf.ssl.cf2.rackcdn.com/openstack/bb656b811fba45e48d88edc1eee4659b/controller/logs/ | 12:12 |
| chandankumar | jgilaber: yes, it is using devstack with real deployment job definition https://github.com/openstack/rally-openstack/blob/master/.zuul.d/rally-task-watcher.yaml | 12:12 |
| jgilaber | ack thanks | 12:12 |
| amoralej | @chandankumar, you can define max expected time to run a test or something like that? | 12:13 |
| amoralej | what's considered a test failure? | 12:14 |
| rlandy | how likely are these tests to find a real problem with what they can actually run? | 12:14 |
| rlandy | perhaps the value add here is unclear | 12:14 |
| chandankumar | amoralej: those are good question, I donot have answer, I still need to explore it. | 12:16 |
| chandankumar | based on doc https://docs.openstack.org/rally/latest/quick_start/tutorial/step_4_adding_success_criteria_for_benchmarks.html | 12:16 |
| amoralej | i think it can be useful to find potential scalability issues, but i also see some limitations for the watcher case | 12:16 |
| jgilaber | #link https://docs.openstack.org/rally/latest/quick_start/tutorial/step_4_adding_success_criteria_for_benchmarks.html | 12:17 |
| morenod | I think that rally generates value on modules where every api call generates tasks which consume resources or create openstack objects. in our case, it will be useful if we can stress audits or actions | 12:17 |
| chandankumar | we can set SLA service on each task | 12:17 |
| amoralej | good, based on run max time and failure rate | 12:17 |
| jgilaber | the balancing strategies might be the most interesting to test I think | 12:17 |
| jgilaber | they are the most computationally expensive vs other like zone migration or host maintenance where nova/cinder do most of the work | 12:18 |
| morenod | but rally is not going to check if the strategy has been applied, it is going to validate that the api call of creating it will answer correctly, right? | 12:18 |
| chandankumar | morenod: that I need to explore. | 12:19 |
| amoralej | i think we could define what specific aspects we want to test | 12:19 |
| chandankumar | I was just checking things where things are defined and used currently. | 12:19 |
| jgilaber | hmm that's a good point morenod, the strategy would not make much of a difference then | 12:19 |
| amoralej | and see where rally can help, and where we need something else | 12:19 |
| amoralej | also, some tests will depend more on the environment that in the number of api call runs, etc... | 12:20 |
| chandankumar | May we I will try with one strategy end to end and see what happens via rally | 12:21 |
| chandankumar | that may give some data | 12:21 |
| amoralej | what would happen if we run a workload_stabilization on 100s of hosts and 1000s of vms? | 12:21 |
| chandankumar | s/we/be | 12:21 |
| amoralej | we don't need to run many api calls, but one in a big environment | 12:21 |
| amoralej | although rally may also help there to define max time | 12:22 |
| jgilaber | ack thanks chandankumar for looking into this, I think we can have a more detailed discussion in the future, once we now more | 12:22 |
| chandankumar | yup | 12:23 |
| jgilaber | if there is no further comments we can move on to the next topic, any last minute request for reviews? | 12:23 |
| chandankumar | for right now, I will look into workload_stablization case wuth rally case | 12:24 |
| chandankumar | *rally | 12:24 |
| chandankumar | jgilaber: sounds good | 12:24 |
| jgilaber | ok, no reviews this week I added a few bugs for triage | 12:25 |
| jgilaber | #topic Bug triage | 12:25 |
| jgilaber | first one is https://bugs.launchpad.net/watcher/+bug/2121807 | 12:25 |
| jgilaber | #link https://bugs.launchpad.net/watcher/+bug/2121807 | 12:25 |
| jgilaber | according to the report the bug is new in version 14.0 | 12:26 |
| jgilaber | which is based on epoxy https://releases.openstack.org/teams/watcher.html#team-epoxy-watcher | 12:27 |
| amoralej | we should recommend to use wsgi for the watcher api | 12:28 |
| amoralej | i think it's using eventlet standalone server | 12:28 |
| amoralej | right? | 12:28 |
| jgilaber | I think you're right | 12:28 |
| jgilaber | command: [ | 12:28 |
| jgilaber | "watchmedo", "auto-restart", "--directory=/app", "--pattern=*.py", "--recursive", "--", # Dev only | 12:28 |
| jgilaber | "bash", "-c", | 12:28 |
| jgilaber | "/usr/local/bin/watcher-api --config-file /etc/watcher/watcher.conf > /app/logs/app.log 2>&1" | 12:28 |
| jgilaber | ] | 12:28 |
| jgilaber | this is the command for the watcher-api container in the docker compose attached to the report | 12:29 |
| jgilaber | did we deprecate that usage? | 12:29 |
| amoralej | i'd say so | 12:29 |
| amoralej | as part of the eventlet changes | 12:30 |
| jgilaber | I'm trying to find something in the docs to that effect | 12:31 |
| jgilaber | found this release not but I'm not sure it's the same https://docs.openstack.org/releasenotes/watcher/2025.1.html#deprecation-notes | 12:32 |
| amoralej | that's different i think | 12:33 |
| jgilaber | the installation guide covers only installing using packages https://docs.openstack.org/watcher/latest/install/install-ubuntu.html | 12:35 |
| amoralej | i think there was a general recommendation about running api services as wsgi services | 12:37 |
| amoralej | i'm not sure if we document it properly, tbh | 12:37 |
| jgilaber | from the container logs it looks like it's running in python 3.13 which I don't think we have tested | 12:37 |
| jgilaber | 2025-09-01 14:11:02.093 8 DEBUG watcher.common.service [-] ******************************************************************************** log_opt_values /usr/local/lib/python3.13/site-packages/oslo_config/cfg.py:2804 | 12:37 |
| amoralej | yes, good point too | 12:38 |
| jgilaber | for now I think we can ask to run with python 3.12 and mark the bug as need info? | 12:39 |
| jgilaber | incomplete, actually | 12:39 |
| amoralej | i'd include the wsgi recommendation | 12:42 |
| amoralej | yes, and move it to incomplete | 12:42 |
| jgilaber | amoralej, can I ask you to add a comment with the wsgi recommendation? | 12:43 |
| amoralej | sure | 12:44 |
| amoralej | i will do it after the mtg | 12:44 |
| jgilaber | thanks! | 12:44 |
| jgilaber | we can move to the second bug https://bugs.launchpad.net/watcher/+bug/2127777 | 12:44 |
| jgilaber | # link https://bugs.launchpad.net/watcher/+bug/2127777 | 12:44 |
| jgilaber | amoralej, this was filed after last week's meeting discussion right? | 12:44 |
| amoralej | yes, actually i already sent patch for it | 12:45 |
| jgilaber | so I don't think we need to discuss much here, just set the importance to high or critical since it's already in progress | 12:45 |
| amoralej | i've just assigned to me and set as medium | 12:46 |
| amoralej | but i can raise | 12:46 |
| jgilaber | unless others object I'm ok with that | 12:46 |
| chandankumar | Can we also add target to series 2026.1? | 12:46 |
| amoralej | done | 12:47 |
| chandankumar | thanks! | 12:47 |
| jgilaber | ack, thanks, looks like we are done, on to the next one https://bugs.launchpad.net/watcher/+bug/2127485 | 12:48 |
| jgilaber | #link https://bugs.launchpad.net/watcher/+bug/2127485 | 12:48 |
| jgilaber | I opened this one because the cinder client method for migrate does an incomplete check when trying to determine if it can migrate a volume | 12:49 |
| jgilaber | it expects that the volume's type is configured with the same volume_backend_name as thedestination pool | 12:49 |
| jgilaber | which is not required | 12:49 |
| jgilaber | I already created a patch for it, it's missing importance, which I think would be medium | 12:50 |
| jgilaber | any comments/objections? | 12:51 |
| jgilaber | I'll take silence as a yes :) and set the importance | 12:52 |
| jgilaber | so with that we've reached the end of the agenda, any last minute topic? otherwise we just need a volunteer for next week's meeting | 12:53 |
| amoralej | i can take it | 12:54 |
| morenod | I will | 12:54 |
| amoralej | morenod wins :) | 12:54 |
| morenod | all yours :P | 12:54 |
| morenod | foto finish? | 12:54 |
| jgilaber | I'll let you fight for it ;) | 12:54 |
| chandankumar | one more thing, During ptg week, do we want to cancel weekly meeting? | 12:55 |
| rlandy | ack - was going to ask that | 12:55 |
| amoralej | morenod deserves it :) | 12:55 |
| amoralej | i'd cancel | 12:55 |
| rlandy | ie: the one after next week? | 12:55 |
| jgilaber | I think so, that is in two weeks time right? | 12:55 |
| chandankumar | yes | 12:55 |
| rlandy | there is one next week | 12:55 |
| rlandy | the one in two weeks would be ptg | 12:55 |
| rlandy | correct | 12:55 |
| rlandy | +1 to cancel | 12:56 |
| jgilaber | so we meet as usual next week, we cancel the next for ptg, and looks like there is consensus | 12:56 |
| amoralej | yep | 12:56 |
| jgilaber | ok so that's all for today, thanks for participating! | 12:57 |
| jgilaber | #endmeeting | 12:58 |
| opendevmeet | Meeting ended Thu Oct 16 12:58:03 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-16-12.00.html | 12:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-16-12.00.txt | 12:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher/2025/watcher.2025-10-16-12.00.log.html | 12:58 |
| rlandy | thanks jgilaber | 12:59 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!