| *** tosky has quit IRC | 00:36 | |
| openstackgerrit | Artom Lifshitz proposed openstack/whitebox-tempest-plugin master: [WIP] Different approach to TripleO job https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/762866 | 00:45 |
|---|---|---|
| openstackgerrit | melanie witt proposed opendev/elastic-recheck master: Add query for bug 1911574 https://review.opendev.org/c/opendev/elastic-recheck/+/770688 | 00:58 |
| openstack | bug 1911574 in OpenStack-Gate "SSH to guest sometimes fails publickey authentication: AuthenticationException: Authentication failed." [Undecided,New] https://launchpad.net/bugs/1911574 | 00:59 |
| openstackgerrit | Artom Lifshitz proposed openstack/whitebox-tempest-plugin master: [WIP] Different approach to TripleO job https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/762866 | 02:31 |
| openstackgerrit | melanie witt proposed openstack/devstack-plugin-ceph master: DNM debug logging https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/770697 | 02:48 |
| *** rcernin has quit IRC | 03:04 | |
| *** rcernin has joined #openstack-qa | 03:25 | |
| *** rcernin has quit IRC | 03:27 | |
| *** brinzhang0 has joined #openstack-qa | 03:27 | |
| *** brinzhang0 has quit IRC | 03:29 | |
| *** brinzhang0 has joined #openstack-qa | 03:29 | |
| *** brinzhang_ has quit IRC | 03:30 | |
| *** brinzhang0 has quit IRC | 03:30 | |
| *** brinzhang0 has joined #openstack-qa | 03:31 | |
| *** rcernin has joined #openstack-qa | 03:31 | |
| *** rcernin has quit IRC | 03:36 | |
| *** rcernin has joined #openstack-qa | 03:37 | |
| *** rcernin has quit IRC | 03:38 | |
| *** rcernin has joined #openstack-qa | 03:38 | |
| *** rcernin has quit IRC | 03:39 | |
| *** rcernin has joined #openstack-qa | 03:39 | |
| *** ricolin_ has joined #openstack-qa | 04:13 | |
| *** whoami-rajat__ has joined #openstack-qa | 04:22 | |
| *** ricolin_ has quit IRC | 04:24 | |
| *** artom has quit IRC | 04:24 | |
| *** ramishra has quit IRC | 04:25 | |
| *** ramishra has joined #openstack-qa | 04:26 | |
| *** amotoki has quit IRC | 04:44 | |
| *** amotoki has joined #openstack-qa | 04:44 | |
| openstackgerrit | melanie witt proposed openstack/devstack-plugin-ceph master: DNM debug logging https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/770697 | 05:12 |
| *** dave-mccowan has quit IRC | 05:43 | |
| *** ricolin has quit IRC | 06:17 | |
| *** brinzhang_ has joined #openstack-qa | 06:25 | |
| *** gcheresh has joined #openstack-qa | 06:25 | |
| *** brinzhang0 has quit IRC | 06:28 | |
| *** abdysn has joined #openstack-qa | 06:28 | |
| openstackgerrit | melanie witt proposed openstack/devstack-plugin-ceph master: DNM debug logging https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/770697 | 06:55 |
| *** ricolin has joined #openstack-qa | 07:39 | |
| *** ccamposr has joined #openstack-qa | 07:39 | |
| *** rcernin has quit IRC | 07:40 | |
| *** ccamposr__ has quit IRC | 07:41 | |
| *** ramishra has quit IRC | 07:44 | |
| *** akahat|rover is now known as akahat|lunch | 07:45 | |
| *** ramishra has joined #openstack-qa | 07:45 | |
| *** rpittau|afk is now known as rpittau | 07:47 | |
| *** ralonsoh has joined #openstack-qa | 07:51 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** ralonsoh_ has joined #openstack-qa | 07:56 | |
| *** rcernin has joined #openstack-qa | 07:56 | |
| *** ralonsoh has quit IRC | 07:59 | |
| *** rcernin has quit IRC | 08:01 | |
| *** slaweq has joined #openstack-qa | 08:03 | |
| *** gfidente|afk is now known as gfidente | 08:11 | |
| *** rcernin has joined #openstack-qa | 08:26 | |
| *** ralonsoh has joined #openstack-qa | 08:28 | |
| *** ralonsoh_ has quit IRC | 08:28 | |
| *** brinzhang0 has joined #openstack-qa | 08:45 | |
| *** brinzhang_ has quit IRC | 08:48 | |
| *** tosky has joined #openstack-qa | 08:49 | |
| *** slaweq has quit IRC | 08:55 | |
| *** slaweq has joined #openstack-qa | 09:00 | |
| *** lucasagomes has joined #openstack-qa | 09:08 | |
| *** yamamoto has quit IRC | 09:21 | |
| *** yamamoto has joined #openstack-qa | 09:53 | |
| *** abdysn has quit IRC | 09:57 | |
| *** abdysn has joined #openstack-qa | 09:58 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** rcernin has quit IRC | 10:14 | |
| *** rcernin has joined #openstack-qa | 10:23 | |
| *** rcernin has quit IRC | 10:40 | |
| *** dtantsur|afk is now known as dtantsur | 10:56 | |
| *** brinzhang has joined #openstack-qa | 10:57 | |
| *** brinzhang has quit IRC | 10:58 | |
| *** brinzhang has joined #openstack-qa | 10:58 | |
| *** brinzhang0 has quit IRC | 10:59 | |
| *** brinzhang has quit IRC | 11:25 | |
| *** brinzhang has joined #openstack-qa | 11:25 | |
| *** brinzhang_ has joined #openstack-qa | 11:26 | |
| *** brinzhang_ has quit IRC | 11:28 | |
| *** brinzhang_ has joined #openstack-qa | 11:29 | |
| *** brinzhang has quit IRC | 11:30 | |
| *** rcernin has joined #openstack-qa | 11:55 | |
| *** rcernin has quit IRC | 11:58 | |
| *** ramishra has quit IRC | 12:02 | |
| *** brinzhang0 has joined #openstack-qa | 12:03 | |
| *** ramishra has joined #openstack-qa | 12:05 | |
| *** ramishra has quit IRC | 12:05 | |
| *** ramishra has joined #openstack-qa | 12:05 | |
| *** brinzhang_ has quit IRC | 12:05 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** akahat|lunch is now known as akahat|rover | 12:36 | |
| *** rfolco has left #openstack-qa | 12:36 | |
| soniya29 | gmann, thanks for the links, I will go through them and then get in touch with you | 12:55 |
| *** rh-jelabarre has joined #openstack-qa | 13:02 | |
| *** brinzhang0 has quit IRC | 13:13 | |
| *** brinzhang0 has joined #openstack-qa | 13:14 | |
| *** sboyron has joined #openstack-qa | 13:26 | |
| openstackgerrit | Lee Yarwood proposed openstack/devstack-plugin-ceph master: WIP zuul: Introduce a multinode ceph job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/711625 | 13:27 |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** artom has joined #openstack-qa | 13:30 | |
| *** nweinber has joined #openstack-qa | 13:54 | |
| *** yamamoto has joined #openstack-qa | 14:03 | |
| *** yamamoto has quit IRC | 14:08 | |
| *** dave-mccowan has joined #openstack-qa | 14:20 | |
| *** abdysn has quit IRC | 14:55 | |
| *** ramishra has quit IRC | 14:57 | |
| *** andrebeltrami_ has joined #openstack-qa | 15:00 | |
| *** ramishra has joined #openstack-qa | 15:01 | |
| *** sboyron has quit IRC | 15:31 | |
| gmann | kopecmartin: can you backport this to stable/stein too https://review.opendev.org/q/I71102095f3603915f0bc7d21f2e18c4eac4e95ec | 16:23 |
| gmann | soniya29: hanks, anytime | 16:23 |
| *** dosaboy has quit IRC | 16:24 | |
| *** vhari has quit IRC | 16:24 | |
| *** dosaboy has joined #openstack-qa | 16:25 | |
| *** gcheresh has quit IRC | 16:26 | |
| *** vhari has joined #openstack-qa | 16:33 | |
| openstackgerrit | Martin Kopec proposed openstack/devstack stable/stein: Remove tempest deprecated img_dir option https://review.opendev.org/c/openstack/devstack/+/770839 | 16:45 |
| kopecmartin | gmann: sure ^^ | 16:45 |
| openstackgerrit | Lee Yarwood proposed openstack/devstack-plugin-ceph master: nova: Make configure_ceph_nova multinode compatible https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/756323 | 16:48 |
| openstackgerrit | Lee Yarwood proposed openstack/devstack-plugin-ceph master: zuul: Introduce a multinode ceph job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/711625 | 16:48 |
| gmann | kopecmartin: thanks | 17:01 |
| *** lucasagomes has quit IRC | 17:02 | |
| *** jpena is now known as jpena|off | 17:07 | |
| *** raildo has quit IRC | 17:42 | |
| *** rpittau is now known as rpittau|afk | 17:42 | |
| *** gfidente is now known as gfidente|afk | 17:53 | |
| dansmith | gmann: so one of the glance multistore tempest tests is failing periodically | 17:56 |
| dansmith | it's reported by subunit as "inprogress" and doesn't actually show as a fail in the test run output, | 17:57 |
| dansmith | but does in the testr-results, but with no useful error | 17:57 |
| dansmith | .MultiStoresImportImagesTest.test_glance_direct_import_image_to_all_stores [] ... inprogress | 17:57 |
| dansmith | I've been running tempest in a loop locally trying to repro, but I can't | 17:57 |
| dansmith | does inprogress mean test timeout maybe? | 17:57 |
| dansmith | total runtime is 1887s which would be just a little over 1800s == 30m | 18:04 |
| *** andrebeltrami_ has quit IRC | 18:05 | |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| gmann | dansmith: humm, it should return timeout in that case | 18:17 |
| gmann | is it happening frequently? | 18:17 |
| dansmith | yeah, ish | 18:18 |
| dansmith | enough that I've seen it a few times, and people have asked me to fix it :) | 18:18 |
| gmann | ohk | 18:18 |
| dansmith | gmann: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tempest.api.image.v2.test_images.MultiStoresImportImagesTest.test_glance_direct_import_image_to_all_stores.*inprogress%5C%22 | 18:21 |
| dansmith | so, maybe just once a day, so not so much, but... still noticeable | 18:21 |
| *** dosaboy has quit IRC | 18:21 | |
| gmann | dansmith: checking.. | 18:22 |
| gmann | it passing scenario, it took only 31 sec - https://zuul.opendev.org/t/openstack/build/ad4328066cdc4526aa1c9526022cbada/log/job-output.txt#27634 | 18:24 |
| dansmith | yeah | 18:24 |
| dansmith | I noticed that in the one I'm looking at, there's an OOM in dmesg | 18:24 |
| dansmith | I think it was just haproxy, but could have stalled something maybe | 18:25 |
| dansmith | oh, nm, | 18:25 |
| dansmith | haproxy triggered, but it killed a python | 18:25 |
| dansmith | that must be it.. if one of the workers just gets kill -9'd it reports as "inprogress...never finished" maybe? | 18:25 |
| dansmith | because the test report also says workerN never returned a value | 18:26 |
| *** raildo has joined #openstack-qa | 18:26 | |
| dansmith | that python doesn't show up as using a lot of memory in the dstat | 18:27 |
| dansmith | but I wonder if we're pushing the limits with ceph | 18:28 |
| gmann | but its failing in non-ceph job too https://zuul.opendev.org/t/openstack/build/28c47bb373084072ae3e8d1ca7f01359/logs | 18:29 |
| dansmith | ah, okay I haven't seen it other than on ceph multistore jobs, but... okay | 18:30 |
| dansmith | Killed process 103758 (python) total-vm:1532428kB, anon-rss:1295772kB, file-rss:5272kB, shmem-rss:0kB, UID:1002 pgtables:2660kB oom_score_adj:0 | 18:36 |
| dansmith | I'm guessing that's a test runner that got to 1.5g? | 18:36 |
| *** slaweq has quit IRC | 18:38 | |
| *** ralonsoh has quit IRC | 18:41 | |
| dansmith | even with just running v2.images tests, my subunit runner master process swells to 1.5G over the course of a 36s run | 18:48 |
| dansmith | I wonder if we're generating a ton of debug logs somewhere or something | 18:49 |
| gmann | humm, may be | 18:50 |
| gmann | showing running with 0 duration https://9bc86082a26342dc4413-58cf4a3a4fc59e9f2b77716e9b3e3ff8.ssl.cf1.rackcdn.com/743695/25/check/glance-multistore-cinder-import/28c47bb/controller/logs/stackviz/index.html#/stdin/test-details/tempest.api.image.v2.test_images.MultiStoresImportImagesTest.test_glance_direct_import_image_to_all_stores | 18:59 |
| dansmith | gmann: https://pastebin.com/6waTd7Ab | 19:00 |
| dansmith | multistore tests make it use 1.5g, 65m otherwise | 19:00 |
| gmann | humm | 19:01 |
| dansmith | gmann: help me with math.. this is 1m not 1g, right? https://github.com/openstack/tempest/blob/79650295718af6eeb4d4438a40fed531e19a253d/tempest/api/image/v2/test_images.py#L161 | 19:02 |
| gmann | 10 m | 19:03 |
| dansmith | I don't think so | 19:03 |
| dansmith | okay, maybe it is, but it takes a looong time to run in a REPL shell | 19:04 |
| dansmith | maybe just because of random(), I dunno | 19:05 |
| gmann | its 1e7. | 19:05 |
| gmann | is it adding more | 19:05 |
| dansmith | thought maybe that was going haywire but it returns 10m of stuff, just takes a long time | 19:05 |
| gmann | in other non multi store test, we just do randombyte - https://github.com/openstack/tempest/blob/2262cced388fea86afc6e645232e406d6ba36bae/tempest/api/image/v2/admin/test_images.py#L101 | 19:07 |
| gmann | which is default 1024 | 19:08 |
| dansmith | yeah | 19:08 |
| *** gcheresh has joined #openstack-qa | 19:08 | |
| gmann | with multistore it is 10M for loop | 19:09 |
| dansmith | randint() for 10M bytes, still generates 10M | 19:10 |
| gmann | we can add log in random_bytes which i think will be helpful for many other cases too | 19:10 |
| dansmith | takes a while, but doesn't seem like that would inflate it | 19:11 |
| dansmith | I ran it locally myself and it generated 10M of stuff | 19:11 |
| *** dosaboy has joined #openstack-qa | 19:11 | |
| dansmith | we do duplicate that in stage_image, so that's 20m | 19:12 |
| dansmith | lemme knock that down and see if it runs smaller | 19:12 |
| gmann | may be with 1024 | 19:13 |
| dansmith | whoa, yeah | 19:14 |
| dansmith | that made a difference :) | 19:14 |
| dansmith | with 1485760 | 19:14 |
| dansmith | Maximum resident set size (kbytes): 262924 | 19:14 |
| dansmith | 256m instead of 1.5g | 19:14 |
| gmann | wow | 19:14 |
| dansmith | so I dunno why that's happening, but I also don't think we need to use 10m there, do you/ | 19:15 |
| gmann | yeah, we can go with default only | 19:15 |
| gmann | checking review of that change if any rational behind that | 19:16 |
| dansmith | setting to default makes it run at 68m, basically the same as when I exclude that test | 19:16 |
| dansmith | so... yeah | 19:16 |
| gmann | nothing specific comment of discussion on this, https://review.opendev.org/c/openstack/tempest/+/745712/1 | 19:18 |
| gmann | yeah we can go with the default. | 19:18 |
| dansmith | patching | 19:18 |
| gmann | dansmith: what log you think we can addfor future help may be before return in data_utils.random_bytes ? | 19:19 |
| *** whoami-rajat__ has quit IRC | 19:20 | |
| dansmith | gmann: I was thinking maybe random_bytes should raise if you try to generate more than 1M or so? | 19:20 |
| dansmith | we don't know what the actual problem is, so I dunno if it's with random_bytes, or our use of it later | 19:20 |
| dansmith | but I'm not sure when we'd need that much data anyway | 19:20 |
| gmann | yeah we should limit it with some value may be 10M if 1M is less | 19:21 |
| gmann | but yeah i am not sure who need that much data | 19:21 |
| openstackgerrit | Dan Smith proposed openstack/tempest master: Fix memory explosion in multi-store image tests https://review.opendev.org/c/openstack/tempest/+/770850 | 19:23 |
| gmann | anyways let's do data_utils.random_bytes limit separately as it is lib function and need reno and all | 19:24 |
| dansmith | ack | 19:24 |
| gmann | gate is taking too long as you know I will check result tomorrow if available :) | 19:25 |
| * dansmith nods | 19:25 | |
| dansmith | I'm hoping for results by at least next week :) | 19:26 |
| gmann | your devstack constraint one still in queue- data_utils.random_bytes | 19:26 |
| gmann | https://review.opendev.org/c/openstack/devstack/+/770662 | 19:26 |
| gmann | heh | 19:26 |
| dansmith | yeah | 19:26 |
| *** raildo has quit IRC | 19:37 | |
| *** raildo_ has joined #openstack-qa | 19:37 | |
| dansmith | gmann: what reno section would this go under? deprecations == more than 1M is deprecated? :) | 19:37 |
| gmann | dansmith: i think we can add in upgrade as we are going to reject more than that now onwards | 19:38 |
| gmann | or we can just log warning if it is more than 1M ? | 19:39 |
| dansmith | ack, didn't know if a test suite had "upgrade" problems.. :P | 19:39 |
| dansmith | we'll never see it I think | 19:39 |
| gmann | ok | 19:39 |
| dansmith | and when we're OOM killed, our subunit buffer dies | 19:39 |
| gmann | yeah | 19:39 |
| dansmith | I can still try to figure out why it's exploding, but I don't think tests really need to use large data blobs in almost all cases, despite the human need to think it makes it more realistic ;) | 19:42 |
| gmann | true. | 19:43 |
| openstackgerrit | Dan Smith proposed openstack/tempest master: Make random_bytes() enforce sane size limits https://review.opendev.org/c/openstack/tempest/+/770852 | 19:47 |
| *** hyang has joined #openstack-qa | 19:47 | |
| *** ccamposr__ has joined #openstack-qa | 19:54 | |
| *** ccamposr has quit IRC | 19:56 | |
| openstackgerrit | Cyril Roelandt proposed openstack/tempest master: Make _create_and_stage_image always return a list of stores https://review.opendev.org/c/openstack/tempest/+/755686 | 19:58 |
| *** slaweq has joined #openstack-qa | 20:19 | |
| *** gcheresh has quit IRC | 20:22 | |
| *** slaweq has quit IRC | 20:27 | |
| *** dosaboy has quit IRC | 21:00 | |
| *** dosaboy has joined #openstack-qa | 21:02 | |
| *** raildo_ has quit IRC | 21:15 | |
| openstackgerrit | Ghanshyam proposed openstack/tempest master: Create default network for scenario tests https://review.opendev.org/c/openstack/tempest/+/770169 | 21:15 |
| *** hamalq has joined #openstack-qa | 21:22 | |
| openstackgerrit | Lingxian Kong proposed openstack/patrole master: Remove the tests for unsupported Nova APIs https://review.opendev.org/c/openstack/patrole/+/770867 | 21:22 |
| *** nweinber has quit IRC | 21:42 | |
| *** vhari has quit IRC | 21:50 | |
| *** rcernin has joined #openstack-qa | 22:05 | |
| *** yamamoto has joined #openstack-qa | 22:18 | |
| *** hyang has quit IRC | 22:31 | |
| openstackgerrit | Merged openstack/devstack stable/ussuri: Remove tempest deprecated img_dir option https://review.opendev.org/c/openstack/devstack/+/770552 | 22:42 |
| *** yamamoto has quit IRC | 22:52 | |
| *** yamamoto has joined #openstack-qa | 22:52 | |
| openstackgerrit | Merged openstack/devstack stable/train: Remove tempest deprecated img_dir option https://review.opendev.org/c/openstack/devstack/+/770555 | 23:07 |
| *** nweinber has joined #openstack-qa | 23:08 | |
| *** nweinber has quit IRC | 23:11 | |
| openstackgerrit | Merged openstack/tempest master: Delete wrong argument from creating HTTP connection https://review.opendev.org/c/openstack/tempest/+/770659 | 23:51 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!