| opendevreview | Takashi Kajinami proposed openstack/puppet-manila master: Replace legacy facts and use fact hash https://review.opendev.org/c/openstack/puppet-manila/+/876007 | 02:16 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/puppet-manila master: Replace legacy facts and use fact hash https://review.opendev.org/c/openstack/puppet-manila/+/876007 | 02:45 |
| opendevreview | Felipe Rodrigues proposed openstack/manila master: NetApp ONTAP: Fix SnapMirror snapshots not being cleaned up https://review.opendev.org/c/openstack/manila/+/849849 | 11:33 |
| opendevreview | Carlos Eduardo proposed openstack/manila-tempest-plugin master: WIP Fix share group tests https://review.opendev.org/c/openstack/manila-tempest-plugin/+/875981 | 11:35 |
| opendevreview | Carlos Eduardo proposed openstack/manila-tempest-plugin master: WIP Fix share group tests https://review.opendev.org/c/openstack/manila-tempest-plugin/+/875981 | 12:10 |
| opendevreview | Felipe Rodrigues proposed openstack/manila master: NetApp ONTAP: Fix SnapMirror snapshots not being cleaned up https://review.opendev.org/c/openstack/manila/+/849849 | 12:24 |
| opendevreview | Felipe Rodrigues proposed openstack/manila-tempest-plugin master: Change test manage share server with invalid id https://review.opendev.org/c/openstack/manila-tempest-plugin/+/857961 | 12:37 |
| opendevreview | Felipe Rodrigues proposed openstack/manila-tempest-plugin master: Change test manage share server with invalid id https://review.opendev.org/c/openstack/manila-tempest-plugin/+/857961 | 12:38 |
| opendevreview | Merged openstack/manila master: [NetApp] Fix DHSS mode fails to create share on a new pool https://review.opendev.org/c/openstack/manila/+/834503 | 12:44 |
| opendevreview | Carlos Eduardo proposed openstack/manila-tempest-plugin master: Correct share group tests cleanup order https://review.opendev.org/c/openstack/manila-tempest-plugin/+/875981 | 13:12 |
| opendevreview | Merged openstack/manila-tempest-plugin master: Fix incorrect url request for share_group_reset_state https://review.opendev.org/c/openstack/manila-tempest-plugin/+/872076 | 13:47 |
| opendevreview | Hassan Asghar proposed openstack/manila-ui master: Create Share Network panel workflows https://review.opendev.org/c/openstack/manila-ui/+/840326 | 14:21 |
| opendevreview | Merged openstack/puppet-manila master: Replace legacy facts and use fact hash https://review.opendev.org/c/openstack/puppet-manila/+/876007 | 20:08 |
| clarkb | hello, I've noticed that your nonvoting manila-tempest-plugin-cephfs-nfs job in opendev's zuul seems to be filling its disk or something similar resulting in three attempts to run the job each taking about 3 hours. THis causes zuul to not report results for all of your other jobs for many hours. I would suggest that you either debug and correct this or move the job to the | 21:39 |
| clarkb | experimetal queue so that it doesn't hold up your changes | 21:39 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d26a8dd0b2554f729acfb15e2fe45732/console is an example failure that resulted in a retry | 21:40 |
| gouthamr | hi clarkb; we did see the job becoming flaky - thanks for the root cause | 22:07 |
| gouthamr | i can think of several reasons why we'd consume a lot of disk space - for one, the test VMs aren't lightweight; and we're also writing random non-compressible data into the disk with our tests | 22:14 |
| gouthamr | instead of moving the job, i'll try to stop running some data intensive tests and continue investigation | 22:14 |
| clarkb | sounds good. The reason I believe it is filling the disk is that ansible is reporting that it cannot find a suiatble /tmp or similar to use for its connection startup activity. Ansible treats this issue as a network connection error and that causes zuul to retry the job | 22:15 |
| gouthamr | clarkb: ++ that's very useful info to start digging into this | 22:17 |
| gouthamr | https://zuul.opendev.org/t/openstack/build/d26a8dd0b2554f729acfb15e2fe45732/log/job-output.txt#24180-24194 | 22:21 |
| gouthamr | https://bugs.launchpad.net/manila/+bug/2009083 | 22:26 |
| opendevreview | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: [CI] Ceph/NFS: skip data-intensive tests with ipv6 https://review.opendev.org/c/openstack/manila-tempest-plugin/+/856540 | 23:33 |
| clarkb | you might want to dd 500MB to disk as a reserve and then if you job timesout you can rm it and then get logs? I dunno maybe that isn't necessary | 23:40 |
| opendevreview | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: [CI] Ceph/NFS: skip data-intensive tests with ipv6 https://review.opendev.org/c/openstack/manila-tempest-plugin/+/856540 | 23:43 |
| gouthamr | clarkb: i'm proposing skipping a couple of these sorta-duplicate tests to see if that would help... i looked at logs for the past 10 or so jobs that ended in RETRY and in each of the failures, one of these tests has failed because we fail to spawn a VM... | 23:48 |
| gouthamr | we have cleanup logic to delete shares as soon as the test is done, but cephfs reclaims space asynchronously.. | 23:49 |
| clarkb | ya my idea was mostly in reserving space that can be deleted immediately allowing ansible to connect and properly collect logs | 23:50 |
| clarkb | since i Think one the issues here is that logging isn't great due to he connection issues | 23:50 |
| gouthamr | ah | 23:50 |
| gouthamr | +1 i'll try that trick | 23:52 |
| gouthamr | i'll monitor this test-skip patch too. it might be a temporary bandaid.. | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!