| opendevreview | Merged openstack/tempest master: Skip test_list_all_networks if no known network configured https://review.opendev.org/c/openstack/tempest/+/883892 | 00:39 |
|---|---|---|
| gmann | melwitt: we can try that, I also do not think we need to keep historical timing data | 04:34 |
| gmann | melwitt: also I am still waiting for this concurrency increase change which should decrease job time to noticeable amount https://review.opendev.org/c/openstack/tempest/+/887220/7 | 04:34 |
| gmann | most of the time, workers stay ideal when scenario test run as they need to run serially I need to check if we make a few of the scenario test in parallel | 04:35 |
| gmann | slow scenario tests already has been separated out to tempest-slow job, I hopeful concurrency increase should solve the job timeout issues | 04:49 |
| gmann | let's see | 04:49 |
| opendevreview | Maxim Sava proposed openstack/tempest master: Add image task client and image tests task APIs. https://review.opendev.org/c/openstack/tempest/+/888755 | 11:06 |
| opendevreview | Merged openstack/tempest master: Run test_port_security_macspoofing_port slow tests serially https://review.opendev.org/c/openstack/tempest/+/889129 | 12:11 |
| opendevreview | Merged openstack/devstack master: Support RHEL 9 https://review.opendev.org/c/openstack/devstack/+/873361 | 12:35 |
| dansmith | melwitt: they're grouped by class always because we force it that way | 13:06 |
| dansmith | melwitt: I did look at alphabetical distribution of the classes because I thought we have too many "Test..." and "Server..." class names, but it doesn't seem like it's actually distributed that way when I look at some samples | 13:07 |
| dansmith | meaning one worker gets some Attach.. and Volume... tests | 13:07 |
| opendevreview | Maxim Sava proposed openstack/tempest master: Add image task client and image tests task APIs. https://review.opendev.org/c/openstack/tempest/+/888755 | 13:19 |
| dansmith | gmann: do we need to run the scenario tests alone? because that definitely increases the linear time | 14:09 |
| dansmith | melwitt: so I'm not sure adding random will help, but it's worth a shot to confirm | 14:15 |
| opendevreview | Merged openstack/devstack master: Switch TLS tests to TLSv1.2+ only https://review.opendev.org/c/openstack/devstack/+/670739 | 16:37 |
| gmann | dansmith: in past as well as in current attempt of slow scenario test to run parallel cause some race condition and fail. | 16:40 |
| dansmith | gmann: ack | 16:41 |
| dansmith | it's astounding how hard it is to get a +1 on a patch right now :/ | 16:41 |
| gmann | yeah, let me think on those. at least slow job is almost half of them time now | 16:44 |
| dansmith | not on the other projects, that I've seen | 16:45 |
| dansmith | but on tempest yeah | 16:45 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Increase the default concurrency for tempest run https://review.opendev.org/c/openstack/tempest/+/887220 | 16:58 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Increase the default concurrency for tempest run https://review.opendev.org/c/openstack/tempest/+/887220 | 16:59 |
| gmann | dansmith: humm server actions test class is too large than I expected it :) reviewing your change | 17:06 |
| dansmith | gmann: yeah it's huge | 17:07 |
| dansmith | gmann: so, I know this is a can of worms, but have you ever considered letting tests use a little bit of threading to parallelize some setup tasks? | 17:13 |
| dansmith | there is a neutron test that takes 84s on my system, but it creates two servers and a port, all serialized, before it starts doing what it really wants to do | 17:14 |
| dansmith | and that goes 20s faster on my system if I delegate those activities to threads | 17:14 |
| dansmith | there are some other tests like volume multi-attach which are probably similar | 17:14 |
| dansmith | places where we create a volume and server serially and then attach could likely see some speedup there | 17:15 |
| gmann | I am wondering if we see more race condition due to that, we experience that in scenario tests | 17:15 |
| dansmith | well, you'd have to be careful of course, and only really allow it for things that are really separate, like creating two unrelated servers or a volume and a server | 17:16 |
| dansmith | it's just that we have some tests that run in <1s and some that take five minutes to run | 17:16 |
| dansmith | testr considers those equivalent because it doesn't know better | 17:17 |
| gmann | but how many tests we have who create multiple servers in setup ? it should be fewer | 17:18 |
| dansmith | would you consider stashing a common testr run file in the job so that we don't always run with no time history? we could update that file periodically, but starting from any known-good run would likely help it make better decisions | 17:18 |
| dansmith | gmann: it's not just multiple servers, server+volume would benefit too, but if you look at stackviz, there are some reeeealy chunky single tests, and many of them are servers and volumes (or two servers and a volume in the case of multiattach) | 17:19 |
| dansmith | anyway, I know it's not a silver bullet, just looking for whatever gains we can get :) | 17:19 |
| gmann | but those are async call at service side right at least creation | 17:20 |
| gmann | but yes we can parallel the waiter things for resource to particular status like SSHABLE, Available etc | 17:20 |
| dansmith | well, not if we wait for active or sshable before we get on to the next thing | 17:20 |
| dansmith | and also, create server creates lots of other stuff like validation resources | 17:20 |
| dansmith | this is the one I was playing with: https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/servers/test_attach_interfaces.py#L295 | 17:21 |
| dansmith | create port before we do anything, | 17:21 |
| dansmith | then create one server and validation resources, wait for active, then create another, wait for active, then check sshable on each | 17:22 |
| dansmith | the port and server creations can all run in parallel | 17:22 |
| dansmith | anyway, I know it's a can of worms | 17:23 |
| dansmith | I'm just looking for where we can shrink some of the way-too-big single tests | 17:23 |
| gmann | I feel waiter is the key thing which we can untie from resource creation and run in parallel. like create server1 (wait_until=None), create server2 (wait_until=None), wait for server1 status, wait for server2 status | 17:26 |
| gmann | I remember in some tests we did that wya | 17:27 |
| gmann | but I agree that if we can do parallel creation of different service resources it will be helpful | 17:27 |
| dansmith | yeah, you can only get so far with just the waiters, but definitely that's where most of the gain is | 17:27 |
| gmann | we made create_server common method so heavy that it takes lot of time | 17:28 |
| dansmith | yeah | 17:29 |
| gmann | it is difficult to do all test refactoring for parallelism but we can observe and make note of those slow running tests and make improvement in those gradually | 17:30 |
| opendevreview | Dan Smith proposed openstack/tempest master: Reorder server and port creation for overlap https://review.opendev.org/c/openstack/tempest/+/889196 | 17:34 |
| dansmith | gmann: here's one ^ | 17:34 |
| dansmith | 10s gain vs 20s with threads.. better than nothing | 17:34 |
| gmann | dansmith: lgtm, 1 comment there about wait | 17:39 |
| gmann | cool | 17:39 |
| dansmith | gmann: ack thanks | 17:40 |
| opendevreview | Dan Smith proposed openstack/tempest master: Reorder server and port creation for overlap https://review.opendev.org/c/openstack/tempest/+/889196 | 17:46 |
| opendevreview | Dan Smith proposed openstack/tempest master: Reorder device rescue with volume for overlap https://review.opendev.org/c/openstack/tempest/+/889198 | 17:46 |
| dansmith | gmann: fixed the first one and here's another that saves almost 20s locally ^ | 17:46 |
| gmann | nice, will check them. | 17:47 |
| opendevreview | Dan Smith proposed openstack/tempest master: Overlap volume and server creation in backup https://review.opendev.org/c/openstack/tempest/+/889200 | 18:00 |
| *** melwitt is now known as jgwentworth | 18:26 | |
| opendevreview | Dan Smith proposed openstack/tempest master: Avoid duplicate server creation in rebuild test https://review.opendev.org/c/openstack/tempest/+/889202 | 18:43 |
| dansmith | gmann: this is a juicy one ^ | 18:43 |
| dansmith | kopecmartin: around by change to get this headed into gate? https://review.opendev.org/c/openstack/tempest/+/889109 | 18:44 |
| dansmith | change? chance. | 18:45 |
| gmann | dansmith: nice, i am doing for scenario tests and there are lot of such cases there | 18:45 |
| dansmith | cool | 18:47 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Optimize the resource creation and status waiting in scenario tests https://review.opendev.org/c/openstack/tempest/+/889207 | 19:14 |
| gmann | dansmith: ^^ I audited the scenario tests (there are a few network tests which need refactoring too) and modified the one that needs parallel resource creation/status waiting | 19:17 |
| dansmith | gmann: ack | 19:21 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 19:53 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Optimize the resource creation and status waiting in scenario tests https://review.opendev.org/c/openstack/tempest/+/889207 | 20:39 |
| dansmith | no timeouts on any of these patches so far is pretty awesome | 20:45 |
| *** haleyb is now known as haleyb_out | 21:54 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!