| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add real-data scenario tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613 | 07:55 |
|---|---|---|
| *** sfinucan is now known as stephenfin | 10:23 | |
| amoralej | sean-k-mooney, wrt https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613/comment/884cc817_cd953567/ could i use directly tempest.common.compute.create_test_server from the watcher-tempest-plugin or you mean implement something like that in our base.py ? | 11:39 |
| amoralej | stuff under common is expected to be usable by any external plugin? | 11:40 |
| amoralej | oh, sorry, i just realized create_server is a wrapper on top of create_test_server, so i guess we can pass everything we need from there | 11:45 |
| sean-k-mooney | you can use it directly but yes its wrapper all teh way down | 11:45 |
| amoralej | thx | 11:45 |
| sean-k-mooney | waht im really suggesting is you do it like this https://github.com/openstack/tempest/blob/80c0477f78c71a2bd2e1a324c41cd2f50329b200/tempest/api/volume/test_volumes_backup.py#L124-L128 | 11:46 |
| sean-k-mooney | instead of creating a new keypair just create teh validation resouces and pass that in | 11:46 |
| sean-k-mooney | if you need to dynamicaly create a keypair for some reason that ok | 11:47 |
| sean-k-mooney | but i was hopign to remvoe https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613/3/watcher_tempest_plugin/tests/scenario/test_execute_strategies_real.py#67 | 11:47 |
| mtembo | #startmeeting Watcher IRC Weekly Meeting - 27 March 2025 | 12:00 |
| opendevmeet | Meeting started Thu Mar 27 12:00:07 2025 UTC and is due to finish in 60 minutes. The chair is mtembo. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:00 |
| opendevmeet | The meeting name has been set to 'watcher_irc_weekly_meeting___27_march_2025' | 12:00 |
| mtembo | o/ hello | 12:02 |
| mtembo | who's present today ? | 12:02 |
| rlandy | o/ | 12:03 |
| jgilaber | o/ | 12:03 |
| chandankumar | o/ | 12:04 |
| amoralej | o/ | 12:05 |
| mtembo | folks it looks like there is not much to discuss today. | 12:07 |
| mtembo | would anyone like to raise a topic ? | 12:07 |
| mtembo | alright, we shall move along ... | 12:10 |
| mtembo | #topic Bug Triage | 12:10 |
| mtembo | #link https://bugs.launchpad.net/watcher/+bug/2104220 | 12:10 |
| rlandy | we seems to have a few recent bugs on strategies | 12:13 |
| amoralej | that's architecture issue, | 12:14 |
| amoralej | the redesign choises the reported mentioned is our ptg topic (Watcher model collector improvement ideas) | 12:14 |
| amoralej | in the operator we are setting default to 15 mins instead of 1h, but the reported behavior can still happen | 12:15 |
| amoralej | optimal solution would be to make collection event based, but we need to make sure we don't overload nova api | 12:16 |
| amoralej | I can summarize this in a bug comment | 12:18 |
| rlandy | amoralej++ | 12:20 |
| mtembo | amoralej++ | 12:22 |
| mtembo | #topic: Volunteers for chair for 3rd April meeting | 12:22 |
| mtembo | I will take the next meeting: mtembo | 12:24 |
| rlandy | thanks mtembo - I can take the after PTG | 12:25 |
| mtembo | Thank you all. | 12:26 |
| mtembo | #endmeeting | 12:26 |
| opendevmeet | Meeting ended Thu Mar 27 12:26:56 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:26 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___27_march_2025/2025/watcher_irc_weekly_meeting___27_march_2025.2025-03-27-12.00.html | 12:26 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___27_march_2025/2025/watcher_irc_weekly_meeting___27_march_2025.2025-03-27-12.00.txt | 12:26 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/watcher_irc_weekly_meeting___27_march_2025/2025/watcher_irc_weekly_meeting___27_march_2025.2025-03-27-12.00.log.html | 12:26 |
| amoralej | sean-k-mooney, thanks, i will follow that approach | 12:28 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add real-data scenario tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613 | 13:18 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add real-data scenario tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613 | 13:20 |
| chandankumar | sean-k-mooney: amoralej Hello, please add it to your review queue https://review.opendev.org/q/topic:%22bug/2100741%22 when free, thank you! It will finish the functional tests move. | 13:36 |
| opendevreview | chandan kumar proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 13:53 |
| opendevreview | chandan kumar proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 13:54 |
| opendevreview | chandan kumar proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 14:01 |
| opendevreview | chandan kumar proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 14:24 |
| amoralej | sean-k-mooney++ thanks to your comments, my new test is much cleaner now | 14:30 |
| *** haleyb_ is now known as haleyb | 15:06 | |
| sean-k-mooney | amoralej: i agree 5 min time.sleep is a problem however | 15:46 |
| sean-k-mooney | unless we skip that test by default in upstream jobs | 15:46 |
| opendevreview | Francisco Seruca Salgado proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 15:52 |
| amoralej | sean-k-mooney, we are skipping it by default actually, https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613/5/watcher_tempest_plugin/tests/scenario/test_execute_strategies_real.py#35 | 15:53 |
| sean-k-mooney | that also a problem | 15:54 |
| amoralej | in our upstream jobs se are setting optimize.datasource to gnocchi or prometheus. Note that, that parameter is only used to set the backend where we are injecting metrics | 15:54 |
| sean-k-mooney | we need to be abel to run this with promethous enabled | 15:54 |
| amoralej | yes, the name of the parameter is missleading | 15:55 |
| amoralej | that's actually unrelated to the prometheus backend | 15:55 |
| sean-k-mooney | no i mean we should not have a seprate job to run this | 15:55 |
| sean-k-mooney | we shoudl be able to run it in any job | 15:55 |
| amoralej | my approach was to be mutually exclusive with inject-metrics jobs, is that fine? | 15:56 |
| sean-k-mooney | no i dont think thats ok | 15:56 |
| sean-k-mooney | well | 15:56 |
| sean-k-mooney | we are running this in perodic right | 15:57 |
| amoralej | yes | 15:57 |
| amoralej | and experimenatl | 15:57 |
| sean-k-mooney | i still think that too expensive honestly | 15:57 |
| opendevreview | Francisco Seruca Salgado proposed openstack/watcher master: [DNM] RDO Zuul check job https://review.opendev.org/c/openstack/watcher/+/945079 | 15:57 |
| amoralej | see https://review.opendev.org/c/openstack/watcher/+/945627 | 15:57 |
| sean-k-mooney | maybe its ok but a full multi node job for one test is a bit much | 15:57 |
| sean-k-mooney | i know we can run other tests that done need metrics | 15:58 |
| sean-k-mooney | but even still that expensive | 15:58 |
| amoralej | ok, i see your point | 15:58 |
| sean-k-mooney | i would invert that if | 15:58 |
| sean-k-mooney | have it skip if there is no data souce | 15:58 |
| sean-k-mooney | CONF.optimize.datasource is ment to be the data souce that is deployed | 15:59 |
| sean-k-mooney | its not intened to contole if we inject or not | 15:59 |
| amoralej | actually, i know we have some other tests which do not require inject-data, we may include them in the same job | 15:59 |
| sean-k-mooney | most of the api tests | 15:59 |
| sean-k-mooney | and one or two of the stagies tests doen need any metrics | 15:59 |
| amoralej | yes, all the api ones + some scenario | 16:00 |
| sean-k-mooney | i need to rebase this too https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/940171 | 16:00 |
| amoralej | i.e. the consolidation one does work perfectly fine with no fake data iirc | 16:00 |
| sean-k-mooney | so https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/940171 removes the injection form test that done need it | 16:00 |
| sean-k-mooney | but i started that ebfore we changed how they run so its in merge conflict | 16:01 |
| amoralej | yes, that's good | 16:01 |
| amoralej | and i think we should implement a mechanism to skip the tests which require injection when injection is not configured | 16:02 |
| amoralej | and the oposite, tests we want to skip when injection is setup | 16:02 |
| amoralej | we should do a new parameter? wdyt ? | 16:03 |
| amoralej | I'm afraid of mixing real and injected data jobs | 16:03 |
| amoralej | i.e. for fake data jobs in prometheus we disable the cpu metrics from the node_exporter | 16:03 |
| amoralej | mixing real and injected will lead us to issues imo | 16:04 |
| sean-k-mooney | i dont think we shoudl test them seperaly | 16:04 |
| sean-k-mooney | what type of issues do you see | 16:05 |
| amoralej | i.e. see this https://github.com/openstack/watcher/commit/539be503f066c50f83d86c4d2476da5231c823cf | 16:06 |
| sean-k-mooney | right so we didnt need to do that | 16:06 |
| sean-k-mooney | it was not causing any actual instablity | 16:06 |
| sean-k-mooney | but any one test would not have both injection and not | 16:06 |
| sean-k-mooney | currently the suite is set up to only run 1 test at a time | 16:07 |
| amoralej | won't you think it will be much harder to be deterministic in the metrics we expect if we mix two sets of metrics ? | 16:07 |
| sean-k-mooney | if we are using real load it wont be determinsitc | 16:07 |
| sean-k-mooney | which is why i didnt want to add tests that do that at all | 16:08 |
| amoralej | one saying, instance_cpu_usage is ~ 0 (real one) and fake one is 70% or whatever | 16:08 |
| amoralej | well, it's less deterministic that fake data, that's right but it can be "deterministic enough" for some tests | 16:08 |
| sean-k-mooney | i guess to move forward we can do the following | 16:09 |
| sean-k-mooney | lets add a annotation to tests that use real data. | 16:09 |
| sean-k-mooney | and instead of usign config optiones to decied if they run | 16:09 |
| sean-k-mooney | we can use teh regex | 16:09 |
| amoralej | you mean skip annotation ? | 16:10 |
| sean-k-mooney | then we can split things up in the job config rather then the plugin code | 16:10 |
| sean-k-mooney | tempest has teh idea of annoating tests https://github.com/openstack/tempest/blob/80c0477f78c71a2bd2e1a324c41cd2f50329b200/tempest/scenario/test_network_v6.py#L240 | 16:11 |
| amoralej | it implies that an anmotated tests is disabled by default ? | 16:11 |
| amoralej | https://docs.openstack.org/tempest/latest/HACKING.html#test-attributes | 16:13 |
| sean-k-mooney | exluded by the test regex | 16:13 |
| amoralej | i see | 16:13 |
| amoralej | we'll need to add regex | 16:13 |
| amoralej | lgtm | 16:13 |
| sean-k-mooney | well so im trying to avoid having multiple way to deceide if a test shoudl run | 16:14 |
| sean-k-mooney | config option are fine but we shoudl generally use them for providing info that cna change between deployments or chagne the behavior | 16:14 |
| sean-k-mooney | rather then choosing if a test runs or not | 16:15 |
| sean-k-mooney | im a litel uncomfortable with test skiping when we change the datasouce we set in the config | 16:15 |
| sean-k-mooney | its not terrible just harder to debug | 16:16 |
| amoralej | ok, that works for me | 16:16 |
| sean-k-mooney | so im wondering if we shoudl do somethign like this @decorators.attr(type=['slow','real_load']) | 16:17 |
| amoralej | I'm still thinking in one case, where we want to run real + tests which does not require injection | 16:17 |
| amoralej | managing that via regex will be harder | 16:17 |
| sean-k-mooney | ok for now lets just leave it as you have it now | 16:18 |
| amoralej | that's where i think it'd may be good to a have a parameter to enable/disable injection, maybe a different one | 16:18 |
| sean-k-mooney | we can revist this in the future | 16:18 |
| amoralej | but i fully agree using CONF.optimize.datasource is missleading | 16:19 |
| sean-k-mooney | but the implciation fo this is we wont be able to run thi in check in github or downstream in the componet pipeline or downstream in gitlab | 16:19 |
| sean-k-mooney | we will need to put it into a sepreate perodic downstream too | 16:19 |
| sean-k-mooney | that really what i wanted to avoid | 16:19 |
| amoralej | but if you check the actual usage of the parameter you will see the only real use is for data injection | 16:20 |
| amoralej | but yes, it's probably inadequate | 16:20 |
| sean-k-mooney | but that not what it for | 16:20 |
| sean-k-mooney | its to tell you what datasouce is deploy | 16:20 |
| sean-k-mooney | so that you can make any decision based on that | 16:20 |
| sean-k-mooney | injection is one thing that needs that | 16:21 |
| sean-k-mooney | we may have other in the future | 16:21 |
| amoralej | so we should probably rework that, remove the empty option and add a new one | 16:21 |
| sean-k-mooney | well i dont want ti to contol inejection | 16:21 |
| amoralej | ok, maybe we will be able to make it work with real + injected in the same job | 16:22 |
| sean-k-mooney | for now let get it wrokign in a seperate job like you started with but we can revisit it | 16:22 |
| amoralej | ok | 16:22 |
| sean-k-mooney | i dont want ot change the requirement on you mid way true | 16:23 |
| sean-k-mooney | we can get it working in a useabel way and hten evolve it later | 16:23 |
| amoralej | yes, i think there are more improvements to do, like https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/940171 | 16:24 |
| amoralej | also, https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945325 | 16:24 |
| amoralej | currently, test_execute_workload_stabilization_strategy is always creating empty action plan | 16:25 |
| amoralej | but i need to refine that | 16:25 |
| amoralej | if you are too busy i probably can take over https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/940171 | 16:26 |
| amoralej | but yeah, step by step | 16:26 |
| amoralej | wrt runtime for the test, 30-60secs is probably enough for upstream jobs where we have some flexibility to setup polling interval in both prometheus and ceilometer, but in operators env, that's hardcoded to 120secs https://github.com/openstack-k8s-operators/telemetry-operator/blob/7cc05b83658fab71f30fff180b23f37983db9809/templates/ceilometercompute/config/polling.yaml#L4 | 16:30 |
| amoralej | there will be a way to modify it soon, but not implemented yet | 16:30 |
| amoralej | that's where the 300 comes from, we need at least two points | 16:30 |
| amoralej | I may do that time a parameter in [optimize] section | 16:31 |
| sean-k-mooney | amoralej: i put it on the back burner becuase we had a plan to actully get injection workign for promethus | 16:42 |
| sean-k-mooney | if you want to take it over feel free there was one test that failed because it was in a clas sthat had teest that need injection and others that dont | 16:42 |
| sean-k-mooney | so i need to go back and split that into a seperat class ro rethink how i group them | 16:43 |
| sean-k-mooney | currently i was disabling it per class but you could do it per test | 16:43 |
| amoralej | yeah, that's what i was thinking | 16:43 |
| amoralej | disabling by test | 16:44 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add real-data scenario tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613 | 16:51 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher master: Add real-data based tests to experimental pipeline https://review.opendev.org/c/openstack/watcher/+/945627 | 16:53 |
| opendevreview | Alfredo Moralejo proposed openstack/watcher-tempest-plugin master: Add real-data scenario tests https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/945613 | 18:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!