| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: Pass multipath config while creating connector object https://review.opendev.org/c/openstack/glance_store/+/786789 | 02:56 |
|---|---|---|
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: Pass multipath config while creating connector object https://review.opendev.org/c/openstack/glance_store/+/786789 | 05:59 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: Pass multipath config while creating connector object https://review.opendev.org/c/openstack/glance_store/+/786789 | 05:59 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/wallaby: DNM: check lower constraints job removal https://review.opendev.org/c/openstack/glance_store/+/804822 | 06:21 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: DNM: test lower constraint job removal https://review.opendev.org/c/openstack/glance_store/+/804829 | 07:30 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: Pass multipath config while creating connector object https://review.opendev.org/c/openstack/glance_store/+/786789 | 07:38 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: Pass multipath config while creating connector object https://review.opendev.org/c/openstack/glance_store/+/786789 | 07:42 |
| opendevreview | yatin proposed openstack/glance_store stable/victoria: [DNM] Test lower constraints trigger https://review.opendev.org/c/openstack/glance_store/+/804833 | 08:27 |
| opendevreview | Rajat Dhasmana proposed openstack/glance_store stable/victoria: DNM: test lower constraint job removal https://review.opendev.org/c/openstack/glance_store/+/804829 | 08:42 |
| opendevreview | yatin proposed openstack/glance_store stable/victoria: [DNM] Test lower constraints trigger https://review.opendev.org/c/openstack/glance_store/+/804833 | 08:43 |
| opendevreview | yatin proposed openstack/glance_store stable/victoria: [DNM] Test lower constraints trigger https://review.opendev.org/c/openstack/glance_store/+/804833 | 08:47 |
| whoami-rajat | croelandt, hey, can you take a look at my glance store patch, i think the release deadline is this week, https://review.opendev.org/c/openstack/glance_store/+/786410 | 09:16 |
| whoami-rajat | dansmith, ^^ | 09:17 |
| croelandt | whoami-rajat: this would also be a good candidate for a 16.x backport, right? | 13:37 |
| abhishekk | dansmith, if you have time, functional-py36 timing out on this patch consistently https://review.opendev.org/804463 (it passes locally every time though) | 13:57 |
| dansmith | only py36? that seems suspicious | 13:58 |
| abhishekk | yeah | 14:00 |
| croelandt | Let's mark 36 as "too old" and remove the job, problem solved | 14:00 |
| abhishekk | :P | 14:01 |
| croelandt | whoami-rajat: apparently this big volume attachment patch conflicts with https://review.opendev.org/c/openstack/glance_store/+/718361 (also by you) | 14:44 |
| croelandt | which one should be reviewed and merged first? | 14:44 |
| whoami-rajat | croelandt, i think we can leave the refactor patch, the multiattach one is fixing a bigger problem | 14:45 |
| croelandt | ok | 14:45 |
| croelandt | whoami-rajat: wouldn't it make sense to have functional tests for this? | 15:05 |
| whoami-rajat | croelandt, hmm, i don't think a functional test would work since we need to create a multiattach volume type (cinder operation) then try to create multiple bootable volumes/instances from that image | 15:10 |
| whoami-rajat | a tempest test would make more sense since this usecase involves nova, cinder and glance and we already have one test in cinder-tempest-plugin | 15:11 |
| whoami-rajat | croelandt, but if you have any ideas regarding functional test, do let me know and will add one | 15:11 |
| croelandt | ok, thanks | 15:12 |
| croelandt | will you be pushing this to wallaby as well? | 15:12 |
| whoami-rajat | thanks for the review croelandt | 15:13 |
| whoami-rajat | yes, i think we've decided to backport this right? https://review.opendev.org/c/openstack/glance_store/+/782200 | 15:14 |
| whoami-rajat | because the multiattach patch depends on the above ^ | 15:14 |
| croelandt | I hope so, will make my life easier | 15:15 |
| whoami-rajat | ack, currently the cherry pick is giving merge conflict, i will compare the wallaby and master code to check for differences and propose backports | 15:16 |
| * abhishekk going for dinner break | 15:25 | |
| dansmith | abhishekk: I spent a little time looking at the timeout thing, | 16:08 |
| dansmith | and it looks like one entire worker hung from the beginning or crashed immediately | 16:08 |
| dansmith | never reporting running a single test | 16:08 |
| abhishekk | oh | 16:08 |
| dansmith | I dunno what would cause that, but it's really only py36 that ever fails, I would probably be looking/thinking about some library-related thing | 16:09 |
| abhishekk | ack, should we find out that any other project facing same issue ? | 16:10 |
| dansmith | yeah, might be good, but it's quite possible it's just something we're doing that others aren't that is triggering it, so I wouldn't rule anything out based on the answer from other projects | 16:11 |
| abhishekk | ack, will look for other projects result | 16:12 |
| dansmith | is it 100% fail on py36 now or just occasional? and I assume it's happening on other patches, not just our refactor ones right? | 16:12 |
| abhishekk | dansmith, out of 6 times it failed 5 times with timeout on this patch | 16:15 |
| abhishekk | I haven't uploaded any other patch since then | 16:16 |
| dansmith | abhishekk: okay it would be good to know if it's just this patch or if other glance master patches are failing similarly | 16:16 |
| abhishekk | May be we should have one DNM patch to run only functional-py38 job only | 16:16 |
| abhishekk | *py36 | 16:16 |
| dansmith | sure | 16:17 |
| abhishekk | I will push that patch as soon as I finish with deactivate/reactivate work | 16:17 |
| abhishekk | croelandt, are you talking about lbragstad tenant-id to project-id change > | 16:32 |
| opendevreview | Merged openstack/glance_store master: Add volume multiattach handling https://review.opendev.org/c/openstack/glance_store/+/786410 | 16:32 |
| croelandt | that and "moFify" | 16:35 |
| croelandt | if this were the first patch of the series I'd +2, but it depends on 2 patches that need to go through the gate (and I think they hit the py36 timeout) so we might as well add another PS | 16:36 |
| abhishekk | croelandt, ack, only thing is I need to change in parent patches as well because they also using tenant-id | 16:37 |
| croelandt | oh I missed that | 16:42 |
| croelandt | I think figuring out the timeout might matter more | 16:42 |
| croelandt | does it only fail on this patchset? | 16:42 |
| abhishekk | yrs | 16:43 |
| abhishekk | yes | 16:43 |
| abhishekk | this one, https://review.opendev.org/c/openstack/glance/+/804463 | 16:44 |
| croelandt | do you think we can get help from the upstream CI maintainers? | 16:48 |
| dansmith | so it's only the one patch after all, nothing else? | 16:52 |
| abhishekk | not sure, haven't added any other patch since then, | 16:52 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Check deactivate, reactivate policy in the API https://review.opendev.org/c/openstack/glance/+/798266 | 16:56 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 17:02 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 17:05 |
| abhishekk | dansmith, it passing on deactivate patch | 17:10 |
| dansmith | okay I thought the discussion above was more sure | 17:12 |
| abhishekk | its stuck again for same patch only (upload patch) | 17:34 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Check upload_image policy in the API https://review.opendev.org/c/openstack/glance/+/804463 | 17:38 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Check download_image policy in the API https://review.opendev.org/c/openstack/glance/+/804547 | 17:38 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Check policies for staging operation in API https://review.opendev.org/c/openstack/glance/+/804558 | 17:38 |
| dansmith | abhishekk: so here's what I'd do: put a patch on top that removes all the test changes and see if that passes or not | 17:40 |
| dansmith | abhishekk: if it does, then maybe some test change you have is getting stuck and affecting other things, and if not, then it's probably the actual policy check | 17:41 |
| abhishekk | dansmith, ack | 17:41 |
| abhishekk | I have also rebased the patch with master now | 17:41 |
| dansmith | I see | 17:41 |
| abhishekk | so lets wait 5-10 minutes to see what happens now | 17:42 |
| dansmith | sure, but also, I meant a DNM patch on top that reverts the test changes, to leave the original intact | 17:44 |
| abhishekk | ack | 17:44 |
| abhishekk | dansmith, passed now | 17:50 |
| dansmith | but needs a couple rechecks to be sure right? | 17:50 |
| abhishekk | yes | 17:51 |
| abhishekk | may be instead of recheck approving will run it one more time in gate , or just add one more recheck before approving ? | 17:52 |
| abhishekk | croelandt, made changes in staging patch | 17:53 |
| dansmith | gate resets cost everyone a lot | 17:58 |
| dansmith | I think the responsible thing to do here is recheck several times locally | 17:58 |
| abhishekk | ack, it is not stuck for staging patch | 18:00 |
| abhishekk | I think I can rebase my DNM patch on top of it which will run only py36 job | 18:01 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 18:04 |
| abhishekk | functional-py36 execution failed on this patch as well ^^^ | 18:16 |
| abhishekk | s/failed/stuck | 18:18 |
| abhishekk | also I can see TIMED_OUT failures for glanc projects, https://zuul.opendev.org/t/openstack/builds?result=TIMED_OUT | 18:27 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 18:42 |
| abhishekk | removed functional test added in upload patch ^^^^ | 18:43 |
| abhishekk | stuck even after removing the test | 18:58 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 19:22 |
| abhishekk | stuck even after removing policy check from image_data.py ^^^^ | 19:33 |
| dansmith | bizarre | 19:34 |
| dansmith | abhishekk: there are some python-version-specific things in upper-contraints.txt | 19:37 |
| dansmith | like networx, which recently changed | 19:37 |
| dansmith | wonder if any of those could be related | 19:37 |
| dansmith | but did you ever get several clean rechecks on a patch not based on your upload one? | 19:38 |
| abhishekk | I have added one on deactivate again and it did passed | 19:38 |
| abhishekk | and DNM before rebasing with upload I ran twice and it passed both the times | 19:39 |
| dansmith | but you've reverted all the bits related to the functional test run from the DNM on top and it still fails right? I mean.. there has to be some reason | 19:40 |
| dansmith | like maybe you got super lucky with the couple of passes | 19:40 |
| abhishekk | might be | 19:40 |
| abhishekk | I have renamed the test to start with do_not_run_test | 19:41 |
| abhishekk | and removed policy check from upload function as well | 19:41 |
| dansmith | I wonder if the easiest thing at this point is to try to get a py36 machine to repro it on | 19:46 |
| dansmith | or, see if we can repro it there and nail it down to a python version and deps | 19:47 |
| dansmith | also, in your stripped-down testing it's probably good to keep both py versions in there to help verify that it's really only py36 | 19:47 |
| abhishekk | I have installed py3.6 now and it is working locally | 19:47 |
| abhishekk | ack | 19:48 |
| dansmith | hrm | 19:48 |
| dansmith | how many times? | 19:48 |
| abhishekk | ran 3 times in a row | 19:48 |
| abhishekk | now running it again | 19:49 |
| dansmith | I would run it 30 times before I was convinced.. previous timing-related test fails often took way more cycles to repro than in CI, because it's a fast bare metal machine, no "noisy neighbors" etc | 19:49 |
| abhishekk | ack | 19:50 |
| abhishekk | will add py38 to my patch and then start running it locally continuously | 19:50 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 19:54 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 20:11 |
| abhishekk | 25+ times consecutive run in local environment on upload patch for functional-py36 - NO failure | 21:06 |
| abhishekk | dansmith ^^ | 21:06 |
| abhishekk | grep -i "functional-py36: commands succeeded" /opt/stack/glance/functional.txt | wc -l >>> output 30 | 21:23 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 21:28 |
| abhishekk | After successful run 4 times without upload changes, now added those back | 21:29 |
| lbragstad | this one looks good to +W, no? https://review.opendev.org/c/openstack/glance/+/804463 | 21:40 |
| abhishekk | lbragstad, we are facing time out issue for functional-py36 on this patch :( | 21:42 |
| abhishekk | https://review.opendev.org/c/openstack/glance/+/804898/8 | 21:42 |
| abhishekk | as soon as I added upload changes back this job started to time out again | 21:42 |
| * lbragstad nods | 21:43 | |
| opendevreview | Abhishek Kekane proposed openstack/glance master: DNM [Master] functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804921 | 21:45 |
| abhishekk | this one is on top of master ^^^^^^ | 21:46 |
| abhishekk | DNM functional-py36 timeout test https://review.opendev.org/c/openstack/glance/+/804898 | 21:46 |
| abhishekk | This one is on top of upload policy change | 21:46 |
| lbragstad | cool- thanks, i'll keep an eye on it | 21:51 |
| abhishekk | thank you | 21:52 |
| opendevreview | Merged openstack/glance master: Load options from the castellan library https://review.opendev.org/c/openstack/glance/+/804754 | 21:53 |
| * abhishekk signing out for the day | 21:58 | |
| abhishekk | something went wrong with zuul | 22:00 |
| abhishekk | this one timed out in manila | 22:20 |
| abhishekk | https://zuul.opendev.org/t/openstack/build/ff958756709748e187aa5cacf049ceb2 | 22:20 |
| * abhishekk out of clues now :/ | 22:35 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!