| sean-k-mooney | hi folks, not urgent but one topic that came up in the watcher seesion today was the intorduction of pytest-based selenium test and fixture to horizon in the context of addign similar watcher-dashbaord testing in the future. manila-ui has also just added it https://review.opendev.org/c/openstack/manila-ui/+/961503 the problem is that using pytest as anythign more then a test | 19:09 |
|---|---|---|
| sean-k-mooney | runner is explcity agaisnt the pti for python projects https://github.com/openstack/governance/blob/master/reference/pti/python.rst#python-test-running and changing this in the past has previously been rejected on the ground that we did nto want to split the comunity by using two diffent python testing framewrosk that work in a fundemaly diffent way. i.e. every thign shoudl be | 19:09 |
| sean-k-mooney | based on the standard lib unittests module and its helpers like fixutre and test tools | 19:09 |
| fungi | it's worth approaching pragmatically... is there a specific reason why it's using pytest? is it doing something that ((o)s)testr can't? | 19:10 |
| sean-k-mooney | one fo the thigns we are evaulateing in wathcer is how to test the ui and we were going to explore usign playwrite instead of selenium as well but we now have a problem in that we could build on the work the horizon team is doing | 19:11 |
| sean-k-mooney | but i really dont want ot have to mix pytest with the rest of our test or have to ways of writing test for watcher | 19:11 |
| sean-k-mooney | fungi: not that im aware of. they used it because there wre django fixtures for pytest | 19:11 |
| fungi | for some reason i thought testr could use the same fixtures as pytest | 19:12 |
| sean-k-mooney | but django uses the standard unit test modules and you can build the exact same fixture effectivly without pytest | 19:12 |
| gouthamr | yes, that's the answer i got when i asked a similar queston.. i actually don't know if there are equivalents in unittest itself | 19:12 |
| sean-k-mooney | there is | 19:12 |
| gouthamr | is that code we'd need to create/maintain in horizon though? | 19:12 |
| fungi | if it's just a matter of the people who contributed the tests didn't know how to port them to unittest, then that seems worthwhile to help them fix | 19:13 |
| fungi | is the code not in the horizon repo? if not, how is it being consumed in jobs? | 19:13 |
| gouthamr | +1 | 19:13 |
| sean-k-mooney | so this was in teh manial-ui repo bug there is code in horizong too | 19:14 |
| sean-k-mooney | https://github.com/openstack/horizon/blob/4b81bd78fc22b846c82b18fc70bec5b49163f514/openstack_dashboard/test/selenium/ui/conftest.py | 19:15 |
| sean-k-mooney | https://github.com/openstack/horizon/blob/4b81bd78fc22b846c82b18fc70bec5b49163f514/openstack_dashboard/test/selenium/integration/test_credentials.py | 19:16 |
| sean-k-mooney | the seleinima based integration test were rewrite to this approch recently | 19:16 |
| sean-k-mooney | well not super recently | 19:17 |
| sean-k-mooney | it seams to have started in 2023 https://github.com/openstack/horizon/commit/36536272ff6e1f191cf150892af814a6d41ba0ae | 19:17 |
| sean-k-mooney | so as of bobcat horizon has depened on ptytest in the defintion of the selenium tests not just as an optional test runner | 19:18 |
| sean-k-mooney | djangos native test suite uses unitests by the way https://docs.djangoproject.com/en/5.2/topics/testing/overview/#module-django.test and there example code use it too https://docs.djangoproject.com/en/5.2/topics/testing/tools/#liveservertestcase | 19:20 |
| sean-k-mooney | pytest is also commonly used with django but it seam like the team was not ware of the pti requirements when this work began. | 19:21 |
| fungi | got it, so really it's just something that should have been fixed/cleared up during development but got missed | 19:23 |
| sean-k-mooney | right, and now there is 2 years of familarity with how it works and its starte to be adopted by other teams | 19:24 |
| sean-k-mooney | so beign pargmatic we coudl update the pti to allow it with all the implciation that has. we coudl accpat the fact that horizon has alwasy been a littel special and bless it in that one case (selenium testing) | 19:25 |
| sean-k-mooney | or we need to have a wider dicussion | 19:25 |
| gouthamr | you got thoughts out of my brain ^ | 19:25 |
| gouthamr | when running tests with manila-ui (or horizon, or any ui plugin), i see differences | 19:25 |
| sean-k-mooney | to be totally transparent i would prefer not to have to use selenium either i woudl perfer to explroe plywrite btu if we have a common integration based on selenium and it now stable that not beyond reaons | 19:26 |
| * gouthamr #TIL about playwright | 19:28 | |
| sean-k-mooney | i did a bit of reasuch for the grian-ui plugin in to how to do django based applciation testing in 2025 and what was/is the current trends | 19:29 |
| sean-k-mooney | mainly i didnt want to end up wiht technial debt | 19:29 |
| fungi | it sounds like we already ended up with technical debt in this case, but have an opportunity to fill in the hole rather than digging it even deeper | 19:30 |
| clarkb | fungi: some parts of pytest are standard conforming and work broadly and other parts are not. This means that you can use pytest to run the tests if you want as long as we stick to the standard but if you are not careful and add pytest bits that are not standards based then you may need to use pytest to run the tests | 19:30 |
| clarkb | this is the upside to using standards based runners in CI, it ensures everyone else can use whichever runner they like | 19:31 |
| fungi | aha, so maybe we could just run this with testr regardless of whethet it's pytest or unittest based | 19:31 |
| gouthamr | don't think it'll work :P | 19:31 |
| gouthamr | https://opendev.org/openstack/horizon/src/commit/83117a21845e8b94939ea89b33df9287bd9fc8cf/tox.ini#L104 this is the invocation | 19:31 |
| sean-k-mooney | pytest can run the stdlibs unittest | 19:32 |
| sean-k-mooney | but i dont think the revers works | 19:32 |
| sean-k-mooney | pytest is doign dependcy injection | 19:32 |
| sean-k-mooney | so to enabel a fixture you defein a funciton that takes an argumetn with the same name as the fixture | 19:33 |
| sean-k-mooney | a standard test runner wont do that as far as im aware | 19:33 |
| clarkb | separately I think pytest has effectively won so the idea of a standard for unittesting is less valuable these days. THat said I'm not sure what pytests parallel test running support is like these days (it was poor once upon a time but I know it has improved) | 19:34 |
| clarkb | it probably wouldn't be the end of the world to just use pytest given its popularity assuming the CI runtimes can be kept down | 19:35 |
| clarkb | but ya the theory behind using a standard based test runner was in part that developer A could choose pytest and developer B could choose whatever their IDE supports and developer C could just run what CI runs and everything would interoperate | 19:35 |
| sean-k-mooney | are of the issues in the past was lack fo subunit output and integration our ci tooling as a result | 19:35 |
| sean-k-mooney | but ya its very common outside the openstack comunity | 19:35 |
| sean-k-mooney | but it entirly new to us | 19:36 |
| clarkb | as a non IDE user I'm not sure what that landscape looks like today | 19:36 |
| clarkb | but that is probably the main reason to continue to commit to standards: if it makes IDE usage easier | 19:36 |
| sean-k-mooney | its supproted in both | 19:36 |
| sean-k-mooney | and i fiend it tends to run our test beter sometimes | 19:36 |
| sean-k-mooney | eventlet makes thigns hard | 19:36 |
| fungi | also worth pointing out a second time, there is a standard in python that the openstack community adheres to, and pytest is not entirely standards-compliant in that regard | 19:37 |
| sean-k-mooney | that and we break tings by havign top leve moduels called fixtures | 19:37 |
| clarkb | it == pytest or the standards based system? | 19:37 |
| sean-k-mooney | which alis the fixtures package | 19:37 |
| sean-k-mooney | i.e. this https://github.com/openstack/nova/tree/master/nova/tests/fixtures breaks the ide integration because you ingetrate by addign the test path to your python path | 19:38 |
| sean-k-mooney | pytest sometime work better for me in vs code but not alwasy again nova having a fixtures module causes test loading problems but eventlet cause more | 19:38 |
| sean-k-mooney | we shoudl really rename that modlue local_fixture to fix that but time | 19:39 |
| clarkb | thats a funny interaction since it really shouldn't be so magical that it breaks like that | 19:39 |
| sean-k-mooney | so pytest is messing with the python path to supprot things like the src layout for packages | 19:39 |
| sean-k-mooney | and because the ide integration put your test path first it cause issues | 19:40 |
| sean-k-mooney | the workaround is to go one level depter say to https://github.com/openstack/nova/tree/master/nova/tests/functional/regressions | 19:40 |
| sean-k-mooney | baiscly you update the test search path to avoid the aliasing issue. but that nova specific | 19:41 |
| gouthamr | besides horizon and plugins, i see pytest being used in testing swift, skyline, python-observabilityclient, rally, etc | 19:41 |
| clarkb | re subunit I don't know how critical the use of that is as long as there are other serialization formats that can be used to debug cross test interactions | 19:43 |
| clarkb | with subunit and ostestr you can grab a subunit file from CI or someone else, load it into your local db then have the test tool try to auto discover cross test interactions that produced an observed failure | 19:43 |
| fungi | i think swift never adopted unittest fully (they were the last holdouts for nosetest for ages) | 19:44 |
| clarkb | I have no idea if pytest can do that. And maybe no one debugs test cases like that anyway so it doesn't matter (I used it a lot back when we made the test cases run in parallel but once the initial set of problems were sorted out then it became a lot simpler to debug one offs) | 19:44 |
| fungi | skyline and rally are good examples of projects developed outside the openstack community and injected later after they'd made their own choices independently of community standards | 19:44 |
| fungi | (rally much longer ago than skyline but similar situations) | 19:45 |
| fungi | i'm unsure where python-observabilityclient came from, but again seems likely to have been developed outside the usual circle | 19:46 |
| clarkb | anyway long story short its possible that through its popularity pytest is now supported everywhere it matters and that parallelized testing works well enough for us to not worry too much about being standards compliant to ensure IDEs etc work | 19:47 |
| gouthamr | tons of plugins to pytest, but don't know if they're all maintained well, pytest-subunit promises to provide subunit compatibility, but i don't see a reference to it in code.o.o .. we're probably just getting an HTML o/p with pytest-html | 19:47 |
| clarkb | 8 years ago or whenever pytest first showed up this wasn't the case but its been a long time | 19:48 |
| gouthamr | ++ | 19:49 |
| sean-k-mooney | so on ide supprot in watcher i get https://paste.opendev.org/show/bvIAOgzbBFnv81fpQOld/ with unitests selected. i.e. eventlet makes it explode | 19:50 |
| sean-k-mooney | but if i installed pytest in the tox env | 19:50 |
| sean-k-mooney | then it works and dicusovers all the tests | 19:50 |
| fungi | will it be an issue once eventlet is gone? | 19:51 |
| sean-k-mooney | honestly it worked in the past | 19:51 |
| sean-k-mooney | and i used to be able to work around this i just have not set the env var | 19:51 |
| sean-k-mooney | i sent am mailing list post about this a while ago i think on how to make it work i just normlaly use the cli | 19:52 |
| clarkb | I'm the weirdo that just uses the cli for java development so I'm really the wrong person to dig into that | 19:52 |
| clarkb | but I know that was one of the goals with being standard compliant (the other was parallel testing that worked in order to speed up round trip times in ci) | 19:52 |
| sean-k-mooney | ya so paralle testing is athing that pytest does supprot | 19:53 |
| clarkb | yup and it actually supported it when it first showed up. It just didn't work well | 19:53 |
| sean-k-mooney | but since almost all my ptyhon expericne is form openstack i cant really comment on how well | 19:53 |
| clarkb | but I think it has improved a lot since | 19:53 |
| fungi | i still do all my development in vim | 19:54 |
| fungi | with ~no plugins | 19:54 |
| sean-k-mooney | fungi: python-observaiblity client was orginally develop by redhat for a seprate downstream product call STF (service telemetry framework) | 19:55 |
| fungi | got it, so basically the same situation as skyline and rally: developed outside openstack and then added | 19:55 |
| sean-k-mooney | effectivly yes | 19:56 |
| sean-k-mooney | it does 2 thigns provide an osc plugin and python binding for talkign to what used to be ceilometner then gnocchi and now promethous | 19:56 |
| fungi | or at least enough initial development was done in each case that their test frameworks had already been chosen by that time | 19:56 |
| sean-k-mooney | apprenlty thsi was a surpsie to a lot of folks. 3 or 4 peopel have told me they ere not aware we were not ment ot use pytest in test code | 19:58 |
| sean-k-mooney | i have been around to remember the debates about this in the pti and the other converstaion that happend a few years ago | 19:59 |
| sean-k-mooney | *few bing like 8+ at this point | 20:00 |
| sean-k-mooney | https://github.com/openstack/requirements/commit/41bc11e2807275755018e8f2ead4047788b897c6 | 20:01 |
| sean-k-mooney | so that when we added it to the requiremetn repo | 20:02 |
| fungi | i mean, lots of things apparently came as a surprise to the skyline devs too, such that it took them years to redo stuff to make it more consistent with existing projects. but i feel like that's going to happen no matter what larger project you try to add a program to | 20:02 |
| clarkb | ya and I think the friction of a different test runner is low if it works with your IDE or you run tox and it runs | 20:03 |
| sean-k-mooney | ya i guess im one of the few peole that regually read the pti and runtimes docs each cycle | 20:03 |
| clarkb | that is different than say "can't translate the repo at all bceause it doesn't use gettext" or "can't release the project because its packaging is bespoke" | 20:03 |
| fungi | we could opt as a community to just let people add whatever they like with whatever frameworks and tools they want, but that could lead to an explosion of things we need to support and even more fragmentation between projects which people already complain is at an unsustainable level | 20:03 |
| sean-k-mooney | clarkb: ya my conserin is not with having tox -e py3 call pytest | 20:03 |
| sean-k-mooney | its with the fact the test framework works fundementally diffently | 20:04 |
| sean-k-mooney | i.e. there are now self.assertEqual() pattherns | 20:04 |
| sean-k-mooney | you do assert X == Y | 20:04 |
| fungi | yeah, a lot of our linting assumes unittest paradigms, for example | 20:04 |
| clarkb | sean-k-mooney: I think both should work with pytest, but yes the language is a bit different | 20:05 |
| clarkb | (the important thing is you raise an exception when you want to trigger failures) | 20:05 |
| sean-k-mooney | yes and it dose som stack trace paring magic to give pretty good output | 20:05 |
| sean-k-mooney | but its also magic | 20:06 |
| clarkb | ya I am a fan of the "use the explicit assertion that explicitly knows how to handle x and y and report failures nicely if it goes wrong" over do bare assertions and hope for the best | 20:06 |
| clarkb | its a bit more upfront work but it helps ensure you're asserting the right thing imo | 20:06 |
| clarkb | side note I helped create the original horizon selenium testing and nothing about it required a special test runner. Just needed to have something bootstrap selenium | 20:07 |
| sean-k-mooney | clarkb: ya so apprently thy have been rewriten ot use test fixture to stablise them | 20:08 |
| sean-k-mooney | and speed them up | 20:08 |
| sean-k-mooney | so now they are usign fixture to do thing like create a nova instance and clean it up after the test | 20:08 |
| sean-k-mooney | but they coudl have done that with https://github.com/testing-cabal/fixtures | 20:09 |
| sean-k-mooney | or the fixture supprot that is build into the django test testcases | 20:09 |
| sean-k-mooney | https://docs.djangoproject.com/en/5.2/topics/db/fixtures/#fixtures-explanation | 20:10 |
| sean-k-mooney | or you know just use self.addCleanup() | 20:11 |
| sean-k-mooney | anyway we dont need to decied or propose anything now. | 20:11 |
| sean-k-mooney | but it just gave me pause when the suggetion was made we adopt this test setup for watcher-dashboard was made | 20:11 |
| sean-k-mooney | on teh one hadn id dont want watcher-dashboard to be special for horizon plugins and use unittest + playwright on the other hand im not conviced we shoudl use selenitum or pytest either | 20:13 |
| fungi | it wouldn't strictly be "special" if it's conforming to standards that the rest of openstack is following | 20:14 |
| sean-k-mooney | well sepical is more by using playwright instead of selenium. we had not actully made a desion to do that but we wanted to explore our options | 20:16 |
| clarkb | the main issues I think we've run into with selenium are sorting out how to make it play nice with firefox snaps | 20:17 |
| clarkb | to the point where I think we switched jobs at one point to chromium because that was a normal package on ubuntu | 20:17 |
| clarkb | opendev uses selenium for our deployment testing and one nice feature is that it can be manipulated remotely | 20:17 |
| sean-k-mooney | https://github.com/microsoft/playwright-python works with a number of backend including jsut webkit | 20:18 |
| sean-k-mooney | so that helps but ya i dont have enough expirnce with either to be honest | 20:18 |
| sean-k-mooney | i think we are still going to try and do a poc of both aprpoches | 20:19 |
| fungi | but yeah the pti doesn't mandate selenium | 20:20 |
| sean-k-mooney | strictly speaking watcher already has selenium tests https://github.com/openstack/watcher-dashboard/blob/master/test-requirements.txt#L5 | 20:20 |
| sean-k-mooney | but like 5 of them | 20:20 |
| sean-k-mooney | and those tests https://github.com/openstack/watcher-dashboard/blob/master/watcher_dashboard/test/integration_tests/pages/admin/optimization/auditspage.py | 20:20 |
| sean-k-mooney | inherit form the olos onse so they probly broke | 20:20 |
| sean-k-mooney | when the rewrite happend | 20:21 |
| sean-k-mooney | fungi: ya i also check the javascript pti but its not specifed there either | 20:22 |
| fungi | basically, not enough of openstack is a webui to warrant standardizing things for that specific niche | 20:26 |
| fungi | we have a lot of rest api services, python libraries, et cetera that trying to standardize patterns across those makes sense | 20:26 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!