Tuesday, 2019-10-15

*** brault has quit IRC00:16
*** brault has joined #oooq00:18
*** hamzy has joined #oooq01:01
*** apetrich has quit IRC02:09
*** dtrainor has quit IRC02:12
*** dtrainor has joined #oooq02:15
*** ykarel has joined #oooq02:22
*** ykarel has quit IRC03:39
*** ykarel has joined #oooq03:57
*** dsneddon has quit IRC03:57
*** epoojad1 has joined #oooq04:08
*** holser has joined #oooq04:13
*** epoojad1 has quit IRC04:15
*** epoojad1 has joined #oooq04:15
*** brault has quit IRC04:19
*** dsneddon has joined #oooq04:26
*** ykarel has quit IRC04:31
*** epoojad1 has quit IRC04:34
*** soniya29 has joined #oooq04:41
*** raukadah is now known as chandankumar04:47
*** surpatil has joined #oooq04:49
*** ratailor has joined #oooq05:05
*** ykarel has joined #oooq05:06
*** epoojad1 has joined #oooq05:14
*** surpatil has quit IRC05:14
*** surpatil has joined #oooq05:14
*** dsneddon has quit IRC05:15
*** dsneddon has joined #oooq05:20
*** dsneddon has quit IRC05:24
*** holser has quit IRC05:27
*** marios has joined #oooq05:27
*** udesale has joined #oooq05:33
*** ykarel_ has joined #oooq05:34
*** marios is now known as marios|rover05:35
*** ykarel has quit IRC05:36
*** udesale has quit IRC05:38
*** udesale has joined #oooq05:39
*** jfrancoa has joined #oooq05:42
*** dsneddon has joined #oooq05:54
*** ccamacho has joined #oooq05:59
*** sshnaidm_ has joined #oooq05:59
*** sshnaidm|pto has quit IRC06:00
*** sshnaidm has joined #oooq06:03
*** ldumont has quit IRC06:04
*** sshnaidm_ has quit IRC06:04
*** ldumont has joined #oooq06:05
*** sshnaidm_ has joined #oooq06:07
*** sshnaidm has quit IRC06:10
*** ldumont has quit IRC06:10
*** ldumont has joined #oooq06:10
*** ykarel__ has joined #oooq06:14
*** holser has joined #oooq06:15
*** ykarel_ has quit IRC06:17
*** yolanda__ has quit IRC06:20
*** yolanda has joined #oooq06:20
*** ykarel__ is now known as ykarel06:28
*** ratailor_ has joined #oooq06:38
*** ratailor has quit IRC06:40
*** saneax has joined #oooq06:42
*** dsneddon has quit IRC06:45
*** matbu has joined #oooq06:57
*** jpena|off is now known as jpena07:00
*** holser has quit IRC07:04
*** tesseract has joined #oooq07:12
*** udesale has quit IRC07:13
*** udesale has joined #oooq07:13
*** tosky has joined #oooq07:19
*** dsneddon has joined #oooq07:19
chandankumarsshnaidm_: Welcome back :-)07:22
*** apetrich has joined #oooq07:26
*** ykarel is now known as ykarel|lunch07:35
*** kopecmartin|off is now known as kopecmartin07:38
*** dtantsur|afk is now known as dtantsur07:53
*** holser has joined #oooq07:55
*** holser has quit IRC08:10
*** holser has joined #oooq08:10
*** dsneddon has quit IRC08:12
*** dsneddon has joined #oooq08:13
*** dsneddon has quit IRC08:17
*** ykarel|lunch is now known as ykarel08:29
*** amoralej is now known as amoralej|mtg08:32
*** akahat has joined #oooq08:48
*** dsneddon has joined #oooq08:49
*** chem|eod is now known as chem08:50
marios|roverykarel: so i think we might have to scale back the tempest tests if nothing else, at least temprarily unless we have some better way before EOD today (EU time at least) cos we are red on master09:16
marios|roverykarel: thanks very much ... fs20 yeah rfolco|ruck filed a new bug for that last night was switching to that next09:16
*** amoralej|mtg is now known as amoralej09:26
ykarelmarios|rover, yes agree need to have a proper plan09:26
ykareland good to have all info in one bug if they are related09:26
*** sshnaidm_ is now known as sshnaidm|pto09:27
chandankumarmarios|rover: ykarel my reproducer failed at other place09:27
sshnaidm|ptochandankumar, nope :)09:27
chandankumarsshnaidm|pto: let me know when you are back!09:27
ykarelchandankumar, what issue?09:27
ykarelstandalone full tempest?09:27
*** dsneddon has quit IRC09:29
chandankumarykarel: yes,09:29
ykarelchandankumar, and what's the issue?09:31
ykarelat what step it failed09:31
chandankumarykarel: one min09:31
chandankumarkopecmartin: fixed https://bugzilla.redhat.com/show_bug.cgi?id=1743569, please test it09:32
openstackbugzilla.redhat.com bug 1743569 in python-stestr "missing stestr only stestr-3 available" [Medium,Modified] - Assigned to chkumar09:32
ykarelchandankumar, ack09:33
kopecmartinchandankumar: ok09:35
chandankumarykarel: the reproducer got failed at run-test : shell09:38
ykarelchandankumar, login to node and see quickstart_install.log at /home/zuul/workspace09:39
ykarelor check logs, reproducer pushes logs at :800009:39
*** derekh has joined #oooq09:43
*** dsneddon has joined #oooq09:59
*** ratailor_ has quit IRC10:12
chandankumarykarel: marios|rover http://logs.rdoproject.org/71/20171/60/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/dea9b9a/logs/undercloud/var/log/extra/errors.txt.txt.gz10:12
chandankumarykarel: marios|rover ERROR neutron.plugins.ml2.managers [req-65806fe3-9e5b-4b5c-827b-a2488b38d1ae efd0b97a36ab41f68de5f66bf0080492 2f954310ba2e4a3d89313d7499ae5fe5 - default default] Failed to bind port f1a0b5cc-6b44-432f-b7c7-7ac96870169f on host badhost2 for vnic_type normal using segments [{'network_id': 'bb97b07a-5b31-4b59-93e6-a3b83a0cd142', 'segmentation_id': 45, 'physical_network': None, 'id':10:13
chandankumar'a566b2cf-b507-4cba-a9da-3574a89db0d4', 'network_type': u'geneve'}]10:13
chandankumarykarel: just grep for ids you can find that10:13
chandankumarhttp://logs.rdoproject.org/71/20171/60/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/dea9b9a/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz10:13
*** ratailor has joined #oooq10:13
ykarelchandankumar, sorry context? u mean ^^ is causing timeout issue?10:15
chandankumarykarel: I think so10:15
ykarelchandankumar, ack good to see other runs too if there is same error and also look timing10:15
ykarelit doesn't look that error is eating up much time though10:15
ykarelall those errors around 07:28:5310:16
ykareland from that error it looks it can be a negative test10:16
ykarelbut need to check code to confirm, saying as per:- badhost210:17
chandankumarykarel: sorry it is found in passing job10:17
chandankumarwait10:17
ykarelneed to check passing one which taking less time10:18
ykarel< 2 hours10:18
marios|roverchandankumar: ykarel yeah it does looks like bona fide timeout... might be just few mins... e.g. see fs20 http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/3fbae7e/logs/undercloud/home/zuul/tempest.log.txt.gz10:23
marios|roverfirst line - last line 2019-10-14 04:09:22 --> 2019-10-14 06:51:1210:23
marios|roverykarel: and if you see the last line its actually in the process of reporting the results... compare to good file at http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/47fc7f9/logs/undercloud/home/zuul/tempest.log.txt.gz10:23
marios|roverbut still... even the 'good' job takes like 2019-10-07 16:25:39 --> 2019-10-07 18:52:57  2.5 + hours10:24
marios|roverthat seems insane10:24
marios|roverchandankumar: ^^10:24
marios|roverrfolco|ruck: chandankumar: ykarel i updated the bug to include fs20 https://bugs.launchpad.net/tripleo/+bug/184758510:25
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,New]10:25
ykarelchandankumar, yes and now it's not finishing in 2.5 hours also, it still need to run few hundreds test10:29
ykarelsorry marios|rover ^^10:29
marios|roverykarel: ack ... not sure if we want to bump timeout or do something about the tests. maybe both ... timeout first cos red and then revisit what we're running there in the meantime and lower timeout again asap10:31
ykarelmarios|rover, yes ^^ can be a backup plan if nothing constructive found soon10:31
ykarelbut for fs020 already timeout for job is 5 hours10:31
ykarelincreasing more seems not that good10:32
marios|roverouch10:32
marios|roverack ykarel10:32
*** ratailor_ has joined #oooq10:38
ykarelmarios|rover, chandankumar btw i am looking at https://review.opendev.org/#/q/topic:token-cache+(status:open+OR+status:merged), at first glance it looks it can affect the timing10:39
ykarelbut let's see if can found more info, before trying revert10:39
marios|roverykarel: ack thanks10:39
ykarellet me see if i have environment, i can try disabling that locally10:40
marios|roverykarel: great10:40
*** ratailor has quit IRC10:40
*** dsneddon has quit IRC10:46
*** soniya29 has quit IRC10:49
ykarelmarios|rover, chandankumar see http://paste.openstack.org/show/783897/10:52
ykareli can observe good difference with cache enabled=false vs enable=true10:52
chandankumarykarel: can you revert and try that10:53
ykarelchandankumar, ok can try, seems both needs revert10:54
ykarelor we can just try enabling cache, that should also work, wdys?10:55
chandankumarykarel: just try with enable=true and see10:56
*** dsneddon has joined #oooq10:56
ykarelchandankumar, ack /me prepares patch10:56
marios|rovernice ykarel lets try it10:57
marios|roverykarel: chandankumar but still though we need to talk about those tests... i think more than 2 hours is already too much10:58
chandankumarmarios|rover: Yes, I am figuring out a way to stdout the test name time when that executes10:59
ykarelmarios|rover, yes that's a different story though, i remember when we fixed a performance regression last year, it was around 1.5 hour for full tempest10:59
chandankumarso that we can compare with runs which tests has taken to much time and then find the id and dig it10:59
ykarelbut now it's more than 2 hours10:59
*** recheck has quit IRC11:00
*** recheck has joined #oooq11:00
*** dsneddon has quit IRC11:00
*** soniya29 has joined #oooq11:02
*** recheck has quit IRC11:03
*** dsneddon has joined #oooq11:03
*** recheck has joined #oooq11:04
*** recheck has quit IRC11:05
*** recheck has joined #oooq11:05
*** recheck has quit IRC11:06
*** recheck has joined #oooq11:06
*** recheck has quit IRC11:06
*** recheck has joined #oooq11:07
*** recheck has quit IRC11:08
*** recheck has joined #oooq11:08
*** dsneddon has quit IRC11:08
*** recheck has quit IRC11:11
*** recheck has joined #oooq11:11
panda|offchandankumar: question11:12
chandankumarpanda|off: anwser11:14
chandankumar*answer11:14
chandankumarpanda|off: yes please ask11:14
panda|offchandankumar: surprise statement. acknowledgements. grettings.11:14
panda|offchandankumar: w/r/t podman gating. Did we ever start a discussion with people in pdoman github org to push the app and .zuul.yaml files ?11:14
marios|roverchandankumar: so fs20/fs21 are running full tempest?11:14
chandankumarpanda|off: fs020 -> full tempest on overcloud11:15
chandankumarfs021 -> skip list on overcloud11:15
marios|roverchandankumar: ie. like standalone-full-tempest-master?11:15
marios|roverchandankumar: ah ok but f20 like ^^ ? full?11:15
marios|rover14:15 < chandankumar> panda|off: fs020 -> full tempest on overcloud11:15
marios|roverchandankumar: thanks ^11:16
*** udesale has quit IRC11:16
marios|roverchandankumar: gonna propose the timeout bump +1 hour as fall back but that can only be temporary measure... not even sure we want to do it its crazy 5 hours already11:16
chandankumarpanda|off: EmilienM told when we start working, he will point us and start the conversation to the right person in podman team11:17
chandankumarneed to wait for EmilienM to come online11:17
*** dsneddon has joined #oooq11:18
chandankumarmarios|rover: yes fs020 and standalone full tempest has almost same test list11:19
arxcruzchandankumar: weshay when you guys have time, can you explain a little bit better what we want to achieve at https://tree.taiga.io/project/tripleo-ci-board/us/1318 ?11:20
ykarelmarios|rover,  for standalone timeout bump was already proposed, but wrong place, see comment https://review.rdoproject.org/r/#/c/23042/11:21
ykarelso possible if we get good result with that cache enable, timeout increase would not be needed11:21
ykareland if we get sure about the fix, we can just promote ignore full tempest11:22
ykarelas fix will be in TripleO itself in this case11:22
*** dsneddon has quit IRC11:22
ykarelbut let's wait for the result11:22
ykarelshould be there in less than 3 hours11:22
marios|roverykarel: ok11:23
marios|roverykarel: yeah i don't want to bump it11:23
marios|roverbut if its worth doing we need to do it asap11:23
marios|roverykarel: to catch the next runs11:23
*** saneax has quit IRC11:24
*** dsneddon has joined #oooq11:32
chandankumarykarel: please put depends-on in above test patch https://review.opendev.org/#/c/688684/11:33
ykarelchandankumar, the data from ^^ would be available in the jobs running in that patch11:35
chandankumarykarel: it is dump the tests run timing in a file11:36
chandankumarif the task got killed, we still have the data11:36
ykarelchandankumar, i have increased the timeout in test patch so will have data anyway, just wanted to save 30 minutes of run11:37
ykarelif depends-on is not that necessary11:37
*** dsneddon has quit IRC11:37
weshaymarios|rover, https://review.opendev.org/#/c/688433/11:38
weshayykarel, thanks for catching that11:38
ykareloh u are here, so i got reminded one more thing11:39
* ykarel checks11:39
ykarelwrt. contentdir patch11:39
ykarelweshay, re. https://review.opendev.org/#/c/688534/ i have seen issue outside of reproducer, so actual issue is somewhere else11:40
*** amoralej is now known as amoralej|lunch11:40
ykarelsome theory: related to centos-release rpm, and availability of uname binary11:40
ykarelbut need to dig more, if have more data where it seen11:41
arxcruzykarel: ping11:41
weshayykarel, ya.. I believe you.. I think altarch is written by something else as well11:41
weshayykarel, is it blocking other ci?11:41
ykarelweshay, nope, i didn't see that in any CI job11:42
ykareljust noticed it in a vm on rdo cloud11:42
*** jpena is now known as jpena|lunch11:42
marios|roverweshay: ack checking11:45
marios|roverweshay: nit i think the default also needs to flip ? rfolco|ruck  https://review.opendev.org/#/c/688433/1/roles/build-containers/tasks/main.yaml11:46
*** ccamacho has quit IRC11:46
marios|roverrfolco|ruck: i added the fs20 issue to the full-tempest bug... they are both suffering from timeout thanks ykarel i updated the title and description and added some comments https://bugs.launchpad.net/tripleo/+bug/184758511:47
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,New]11:47
rfolco|ruckmarios|rover, cool11:48
marios|roverrfolco|ruck: ykarel is trying to improve it with https://review.opendev.org/688677 but even then... a 'good' run is almost 2.5 hours far too long for tempest ... see comment #2 and we can't even bump the timeout as temp measure see comment 411:48
marios|roverrfolco|ruck: i proposed the timeout bump but it fails cos 5 hours is max already11:48
marios|roverrfolco|ruck: like we don't want to but it was to get us passed the promotion cos red.11:49
marios|roverrfolco|ruck: chandankumar we still need to discuss those tempest tests do we need to cut some of them... e.g. for fs20 at least if the standalone can run them in better time for example11:49
rfolco|ruckmarios|rover, concurrency=3 ? can we tweak it?11:50
marios|roverrfolco|ruck: don't know ...11:50
marios|roverrfolco|ruck: where is it on tempest/zuul confg?11:51
chandankumarrfolco|ruck: it is not related to concurreny there is something else going on I think11:51
rfolco|ruck:-(11:54
chandankumarrfolco|ruck: I am trying to get the tempest tests timing for each tests11:55
chandankumarthen we have a clear picture what is going on11:55
*** epoojad1 has quit IRC11:56
weshayrfolco|ruck, take the bug triage time to sync w/ marios and I'll join perhaps a few minutes late11:58
marios|roverrfolco|ruck: send me meeting info?11:59
marios|roverdon't see it on calendar11:59
rfolco|ruckok I have an errand now should be back in 30 min marios|rover chandankumar weshay11:59
rfolco|ruckmarios|rover, ovt12:00
rfolco|ruckpvt12:00
*** dsneddon has joined #oooq12:00
marios|roverrfolco|ruck: ack tx12:00
*** epoojad1 has joined #oooq12:00
EmilienMchandankumar, panda|off : /join #podman and we can chat there12:02
chandankumarEmilienM: ack12:02
chandankumararxcruz: for above user story, we need to try all the 4 options and see which one is better and viable12:04
chandankumarbrb12:04
*** chem is now known as chem|lunch12:06
*** epoojad1 has quit IRC12:08
*** saneax has joined #oooq12:09
ykarelmarios|rover, https://bugs.launchpad.net/tripleo/+bug/1847585 is still not in CIX12:10
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,New]12:10
ykarelneed to change status and milestone12:10
ykarelpromotion-blocker tag is since 5 days but not in CIX12:10
marios|roverykarel: ack12:13
ykarelack Thanks12:14
*** ratailor_ has quit IRC12:21
chandankumarrfolco|ruck: we are cancelling bug traige meeting?12:23
rfolco|ruckchandankumar, we need to talk about promotion blockers, and I invite you to be there12:24
rfolco|ruckchandankumar, and maybe arx, since its tempest related12:24
chandankumarrfolco|ruck: ok12:24
rfolco|ruckarxcruz, https://meet.google.com/unv-qweu-tdp12:25
rfolco|ruckmarios|rover, chandankumar ^12:30
*** ykarel is now known as ykarel|afk12:32
*** hamzy has quit IRC12:34
*** chem|lunch is now known as chem12:36
*** hamzy has joined #oooq12:37
*** saneax has quit IRC12:38
arxcruzrfolco|ruck: marios|rover https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/47fc7f9/logs/stackviz/#/testrepository.subunit/timeline12:39
*** jpena|lunch is now known as jpena12:41
chandankumarrfolco|ruck: joining back12:50
arxcruzmarios|rover: rfolco|ruck https://review.rdoproject.org/r/#/c/23015/12:51
chandankumararxcruz: marios|rover rfolco|ruck https://review.opendev.org/#/c/688684/12:53
marios|roverthnks arxcruz chandankumar12:53
chandankumarmarios|rover: arxcruz rfolco|ruck rfolco|ruck http://logs.rdoproject.org/84/688684/1/openstack-check/tripleo-ci-rhel-8-standalone-rdo/7b84d49/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz12:56
arxcruzchandankumar: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/3fbae7e/logs/undercloud/home/zuul/tempest.log.txt.gz12:56
weshayrfolco|ruck, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L2613:01
*** panda|off is now known as panda13:01
weshayrfolco|ruck, you'll also need to handle the case where /opt/git does not exist13:01
weshaymarios|rover, fyi ^ we missed that13:01
*** dsneddon has quit IRC13:01
rfolco|ruckweshay, find won't find it13:01
rfolco|ruckweshay, did it failed ?13:01
weshayyes13:01
rfolco|ruckfail*13:02
rfolco|ruckwhat if we ignore13:02
weshayand I think it's the wrong file13:02
rfolco|ruck|| true13:02
weshayI think || true is fine13:02
rfolco|ruck:13:02
rfolco|ruckko13:02
rfolco|ruckok13:02
*** ykarel|afk is now known as ykarel|away13:04
weshay+(/opt/stack/tripleo-ci/toci_gate_test.sh:37): sudo find /opt/git -delete13:04
weshayfind: ‘/opt/git’: No such file or directory13:04
weshayrfolco|ruck, ^13:04
rfolco|ruckweshay, fixing13:05
*** ykarel|away has quit IRC13:10
marios|roverweshay: ack rfolco|ruck you looking at that? (the opt git fix?)13:14
rfolco|ruckmarios|rover, yes fixing13:14
marios|roverrfolco|ruck: k thankx13:14
rfolco|ruckmarios|rover,    https://review.opendev.org/688707 Ignore error when /opt/git does not exist13:15
weshayrfolco|ruck, https://review.opendev.org/#/c/688707/113:19
rfolco|ruckaaah the template13:22
*** amoralej|lunch is now known as amoralej13:26
weshaytripleo community call https://meet.google.com/bqx-xwht-wky13:30
*** epoojad1 has joined #oooq13:31
*** Goneri has joined #oooq13:44
rfolco|ruckagenda for community call: https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw13:44
*** ykarel has joined #oooq13:46
* chandankumar headed home13:50
*** soniya29 has quit IRC13:55
*** surpatil has quit IRC13:55
rfolco|ruckmarios|rover, do you know how to report once for train/master so we save some time and resources for promotions on both releases?13:55
zbrweshay: are you sure https://review.opendev.org/#/c/686192/2/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2 is still needed?13:57
zbri am trying to fix it and already spent a huge number of hours with bash tricks. editing these files with sed is a recipe for failure13:57
zbrthere are huge number of cases: file existing now, value missing or present, present without quoates, with simple or double quotes....13:59
marios|roverrfolco|ruck: not really sure what it means 'report once for train/master'14:00
marios|roverrfolco|ruck: in the promoter? or something else14:01
marios|rovertripleo weekly irc starting #tripleo14:01
zbrdoes anyone knows the official fileformat used by sysconfig files? is real shell or ini like? official docs fail to mention that.14:02
rfolco|ruckmarios|rover, train is master now, so we don't need to run jobs twice to promote both releases14:02
rfolco|ruckmarios|rover, what weshay said in the mtg14:02
marios|roverrfolco|ruck: yeah so it will involve updating the zuul layout then?14:02
rfolco|ruckmarios|rover, criteria?14:02
marios|roverrfolco|ruck: oh fine ok14:03
marios|roverrfolco|ruck: but14:03
marios|roverrfolco|ruck: that will have to be on the promoter itself14:03
rfolco|rucknot sure if that is a option14:03
marios|rovernot the upstream critera14:03
marios|roverrfolco|ruck: cos it is frozen right14:03
marios|roverrfolco|ruck: sure will  post that in a sec14:03
marios|roverrfolco|ruck: well14:03
marios|roverrfolco|ruck: also post it ?14:03
marios|roverrfolco|ruck: or just on the promoter14:03
rfolco|ruckmarios|rover, yes hack n smile14:03
marios|roverrfolco|ruck: no i mean should we update the main criteria upstream too with a review14:04
marios|roverrfolco|ruck: cos this is meant to be temporary anyway right14:04
marios|roverrfolco|ruck: will post worst case we abandon it14:04
rfolco|ruckmarios|rover, lets chat again now ?14:04
marios|roverrfolco|ruck: yeah sure14:04
rfolco|ruckwe need to set a plan in case timeout is gone14:04
rfolco|ruckand in case its not gone14:04
rfolco|ruckand for train/master14:04
marios|roverlooks like ykarel found the root cause anyway the cache thing helped a lot apparently14:04
rfolco|rucknice14:05
rfolco|ruckykarel++14:05
rfolco|ruckykarel += 114:05
rfolco|ruckmeh nm14:05
marios|roverykarel: ack lets use oooq ... but we could just use enabled cache for now?14:06
ykarelmarios|rover, that patch doing multiple things related to memcache14:08
ykareland i am not sure how far memcache is there by default14:08
marios|roverykarel: i see so you mean enabling it impacts lots of things not just our tempest tests14:08
ykarelmarios|rover, yes14:08
ykarelso those patches needs rework14:09
rfolco|ruckmarios|rover, ykarel: did the test finish?14:09
ykarelrfolco|ruck, yes it finished in 1 hr 41 minutes14:09
rfolco|ruckwow14:09
marios|roverright?14:09
marios|roveri mean even the 'good' run from 10 days ago was like 2.5 hours14:10
ykarelmarios|rover, standalone-tempest14:10
rfolco|ruckykarel, but this is tempest only ?14:10
marios|roverykarel: ah ok so a bit shorter than fs20? ack14:10
ykarelrfolco|ruck, yes tempest only14:10
rfolco|ruckok good14:10
ykarelrfolco|ruck, https://logs.rdoproject.org/71/20171/61/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/dc02a8b/job-output.txt14:10
rfolco|ruckmarios|rover, wanna chat bro14:11
marios|roverrfolco|ruck: lets talk after tripleo14:13
marios|roverrfolco|ruck: cos weshay is on and he is referring to us14:13
rfolco|ruckk14:13
*** recheck has quit IRC14:20
*** ykarel is now known as ykarel|away14:20
*** recheck has joined #oooq14:20
*** recheck has quit IRC14:20
*** recheck has joined #oooq14:21
ykarel|awaymarios|rover, rfolco|ruck /me need to go out now, tomorrow also would be available for only half day14:21
*** recheck has quit IRC14:21
ykarel|awayyou can consider revert or fix, /me will check later14:21
*** recheck has joined #oooq14:21
ykarel|awaybut remember in case of revert, both tht and puppet-tripleo patch need to be reverted14:22
ykarel|awaypuppet-tripleo depend on tht one14:22
marios|roverthanks ykarel|away14:23
marios|roverykarel|away: comment #7 https://bugs.launchpad.net/tripleo/+bug/184758514:25
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,Triaged]14:25
marios|roverrfolco|ruck: o/14:27
*** recheck has quit IRC14:27
*** recheck has joined #oooq14:28
marios|roverpanda: if i update the criteria on the promoter do i need to restart the service?14:31
marios|roverto pick up the criteria i mean ? we will update train to be same as master14:31
marios|roverweshay: ^^ this is what you meant on the call earlier  ? ^14:31
marios|roverpanda: i think it will pick up just sanity checking14:33
pandamarios|rover: no need to restart14:33
weshaymarios|rover, no14:33
weshaymarios|rover, rfolco|ruck but only change the criteria w/ my coordination please :)14:34
marios|roverweshay: me and folco discussing now14:34
marios|roverweshay: do you have time to join us?14:34
weshaysure14:34
marios|roverweshay: otherwise, was that14:34
marios|roverk14:34
marios|roversec14:34
*** dtantsur is now known as dtantsur|brb14:40
*** dsneddon has joined #oooq14:42
chandankumarweshay: ykarel|away with cache enabled the deployment reproducer is running tempest14:45
chandankumarweshay: ykarel|away http://38.145.34.6:9000/t/tripleo-ci-reproducer/stream/72f8575a71a44f99818fbf0b359187ba?logfile=console.log14:45
*** dsneddon has quit IRC14:46
marios|roverweshay: https://bugs.launchpad.net/tripleo/+bug/184758514:47
openstackLaunchpad bug 1847585 in tripleo "periodics centos-7-standalone-full-tempest-master + centos-7-ovb-1ctlr_2comp-featureset020-master timeout tempest tests" [Critical,Triaged]14:47
zbrweshay: for which release(s) did we need the iptables hack?14:47
chandankumarmarios|rover: reproducer is also up npw14:48
weshayhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released14:49
weshaymarios|rover, rfolco|ruck baseurl=http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/centos7-train/11/c9/11c9839df8d9b5abb46f2c7bee2c8db975f77867_3f633a8714:49
weshayrfolco|ruck, marios|rover https://trunk.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=11c9839df8d9b5abb46f2c7bee2c8db975f77867&distro_hash=3f633a877cc44c0fa774e06dbf99c7db29419b2f14:51
marios|roverweshay: ack14:58
weshayhttps://trunk.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=11c9839df8d9b5abb46f2c7bee2c8db975f77867&distro_hash=3f633a877cc44c0fa774e06dbf99c7db29419b2f14:58
weshayhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-train/be6b4f8/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz14:58
*** holser has quit IRC15:01
*** skramaja has quit IRC15:09
chandankumarmarios|rover: now we have results http://logs.rdoproject.org/37/23137/1/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/23110aa/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz15:13
chandankumarin timedout job15:13
chandankumarmarios|rover: I will put this result in job compare to have a full idea15:14
marios|roverchandankumar: ack can you add comment on the bug ?15:15
*** matbu has quit IRC15:21
*** holser has joined #oooq15:23
*** holser has quit IRC15:27
marios|roverweshay: rfolco|ruck like that? https://review.rdoproject.org/r/#/c/23141/1/.zuul.yaml15:35
weshaydlrn_hash_tag:15:36
marios|roverweshay: ah k sec sorry15:36
marios|roverweshay: done15:37
*** jfrancoa has quit IRC15:37
weshaymarios|rover, my example.. https://review.rdoproject.org/r/#/c/22878/3/.zuul.yaml didn't need a hash for mine... different case15:38
marios|roverweshay: you think i need the registry disabled too then ?15:38
marios|rover+ iptables... adding15:38
chandankumarweshay: registry disabled patch merged in reproducer15:39
weshaychandankumar, thanks15:39
weshaymarios|rover, chandankumar anyone know how to disable mirrors in a job?15:39
marios|roverweshay: sorry i do not15:39
chandankumarweshay: we discovered for periodic job force_periodic and rdo registry var should set in15:39
chandankumarweshay: jpena is the correct person15:40
weshaychandankumar, have a libvirt repro working... using the new repro.. but fails on the mirror15:40
weshaychandankumar, good.. thanks15:40
marios|roverweshay: updated https://review.rdoproject.org/r/#/c/22878/3/.zuul.yaml15:40
marios|roverweshay: sorry thats your one15:40
marios|roverweshay: here https://review.rdoproject.org/r/#/c/23141/3/.zuul.yaml15:40
weshayaye.. ok.. you should see it kick in rdo zuul15:40
weshaymarios|rover, I see it15:41
*** dsneddon has joined #oooq15:42
chandankumarmarios|rover: you can use rdo jobs instead of test project15:42
marios|roverweshay: ack15:43
marios|roverchandankumar: ok thanks15:43
chandankumarjust modify projects.yaml and get the job done15:43
*** matbu has joined #oooq15:44
*** jpena is now known as jpena|brb15:45
*** dsneddon has quit IRC15:46
*** jfrancoa has joined #oooq15:52
*** dsneddon has joined #oooq15:59
recheck[gh-pytest-molecule] 1.2.2 → https://github.com/pycontribs/pytest-molecule/releases/tag/1.2.215:59
rfolco|ruckmarios|rover, working on the rdo jobs patch ? let me know if you wanna some help or chat about16:03
rfolco|ruckmarios|rover, need to talk 5 min before you leave16:03
marios|roverrfolco|ruck: https://review.rdoproject.org/r/#/c/23141/3/.zuul.yaml there16:03
marios|roverrfolco|ruck: am almost out now16:03
rfolco|ruckplease don't go16:03
rfolco|ruckdon't gooooo16:04
rfolco|ruckmarios|rover, this is testproject16:04
marios|roverrfolco|ruck: 18:35 < marios|rover> weshay: rfolco|ruck like that? https://review.rdoproject.org/r/#/c/23141/1/.zuul.yaml16:04
marios|roverrfolco|ruck: pinged you a while ago man...16:04
marios|roverrfolco|ruck: yeah should it be rdo jobs? chandankumar mentioned that but why not testproject16:05
rfolco|ruckmarios|rover, <chandankumar> marios|rover: you can use rdo jobs instead of test project16:05
marios|roveris there an advantage?16:05
marios|roverrfolco|ruck: ^ if there is sure post it there instead?16:05
chandankumarmarios|rover: yes, it is easier16:05
rfolco|ruckmarios|rover, ok whatever :)16:05
marios|roverchandankumar: easier how16:05
chandankumarmarios|rover: you can do the stuff from termianl16:05
chandankumarlocally16:05
rfolco|ruckmarios|rover, just do what you find easier, both works16:05
marios|roverchandankumar: what stuff you mean to write the review?16:05
weshay2019-10-15 15:54:14 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleotrain/centos-binary-glance-api:9fc9eb4112461e3f5db82e8c2dfeb3fb661011b0_062875d616:05
marios|roverchandankumar: not questioning it tryiung to understand if there is some advantage to rdo-jobs16:06
*** akahat has quit IRC16:06
chandankumarmarios|rover: since most of the jobs defined there, we can manipulate there and run there16:06
chandankumarand then once testing finish, convert it to proper review16:07
chandankumarin a single repo itself16:07
*** dsneddon has quit IRC16:07
chandankumarthis is the advantage16:07
weshayhrm.. maybe I'm wrong about the var for dlrn_hash16:08
weshaybaseurl=http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-train/9f/c9/9fc9eb4112461e3f5db82e8c2dfeb3fb661011b0_062875d616:08
weshayhttp://logs.rdoproject.org/41/23141/3/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/c4592f2/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz16:08
chandankumarweshay: replace it with this https://trunk.rdoproject.org/centos7-train/9f/c9/9fc9eb4112461e3f5db82e8c2dfeb3fb661011b0_062875d6/16:09
chandankumarto find the repo16:09
chandankumaror use skopeo inspect whether image exists or not16:09
weshaythe override in zuul16:09
weshayis dlrn_hash:16:09
weshayor dlrn_hash_tag:16:09
marios|roverweshay: is it not working with dlrn_hash_tag?16:09
weshayhttps://review.rdoproject.org/r/#/c/23141/3..4/.zuul.yaml16:10
weshaymarios|rover, nope.. but updated https://review.rdoproject.org/r/#/c/23141/3..4/.zuul.yaml16:10
marios|roverweshay: thank you will catch it again tomorrow then16:10
weshaymarios|rover, jobs retarted :)16:10
weshaycool.. c ya16:10
marios|roverweshay:ack16:10
chandankumarmarios|rover: see ya!16:10
*** matbu has quit IRC16:11
*** kopecmartin is now known as kopecmartin|off16:11
marios|rovero/ chandankumar16:12
*** dtantsur|brb is now known as dtantsur16:13
chandankumarmarios|rover: weshay dlrn_hash_tag would be tripleo-ci-testting or current tripleo16:13
chandankumardlrn_hash would be just hash16:13
weshayaye.. hard to keep it straight16:16
weshaythanks16:16
weshaychandankumar, in the repoducer.. we give a hash.. for hash tag16:16
weshayI need to go through the logic again16:17
chandankumarweshay: yes, we need to rework that part, /me waits for rlandy or sshnaidm|pto to come back16:17
*** jpena|brb is now known as jpena16:18
*** dsneddon has joined #oooq16:24
*** marios|rover has quit IRC16:24
*** dsneddon has quit IRC16:29
*** jfrancoa has quit IRC16:32
weshayit's dlrn_hash_tag16:33
weshayforce_periodic is overriding it as designed16:33
*** dsneddon has joined #oooq16:35
*** dtantsur is now known as dtantsur|afk16:37
*** dsneddon has quit IRC16:40
*** derekh has quit IRC16:47
*** holser has joined #oooq17:00
*** jpena is now known as jpena|off17:01
*** dsneddon has joined #oooq17:03
*** amoralej is now known as amoralej|off17:09
*** dsneddon has quit IRC17:13
*** ykarel|away is now known as ykarel17:15
*** dsneddon has joined #oooq17:17
*** dsneddon has quit IRC17:22
*** holser has quit IRC17:22
*** epoojad1 has quit IRC17:37
*** epoojad1 has joined #oooq17:38
rfolco|ruckweshay, arxcruz do we know why jobs started timing out? it seems to be happening since last sprint17:51
*** dsneddon has joined #oooq17:55
*** dsneddon has quit IRC18:06
*** holser has joined #oooq18:09
*** epoojad1 has quit IRC18:22
*** chandankumar is now known as raukadah18:24
*** dsneddon has joined #oooq18:46
*** ykarel is now known as ykarel|away18:47
weshayrfolco|ruck, you there?18:58
rfolco|ruckweshay, yes, whats going on https://review.rdoproject.org/r/#/c/23141/18:58
rfolco|ruckweshay, why only standalone18:58
rfolco|ruckweshay, ??19:01
weshayrfolco|ruck, well.. I was having trouble forcing the hash.. and didn't want to kill rdo19:01
weshayrfolco|ruck, but found an interesting problem19:01
rfolco|ruckps5 failed on19:02
rfolco|ruckHTTPError: 503 Server Error: Service Unavailable for url: http://mirror.regionone.rdo-cloud.rdoproject.org:8082/v2/19:02
weshayaye19:02
weshayrfolco|ruck, let's chat so I can explain19:02
weshayin person19:02
rfolco|ruckok19:02
weshayalso noting it in etherpad19:02
weshayrfolco|ruck, https://meet.google.com/zyi-unuk-sfq19:03
*** ykarel|away has quit IRC19:05
*** holser has quit IRC19:14
weshayrfolco|ruck, http://codesearch.openstack.org/?q=hash_info.sh&i=nope&files=&repos=19:14
rfolco|ruckweshay, https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/pre.yaml#L1119:28
rfolco|ruckand https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/get-hash.sh#L3419:28
rfolco|ruckso perhaps we can hack promote_name there19:29
weshayrfolco|ruck, ya.. $PROMOTE_NAME needs to equal the actual hash19:36
weshaynot an arbitary name19:37
rfolco|ruckactually more than that19:37
rfolco|ruck11/c9/11c9839df8d9b5abb46f2c7bee2c8db975f77867_2932899d19:37
rfolco|ruckhttps://trunk.rdoproject.org/centos7-master/11/c9/11c9839df8d9b5abb46f2c7bee2c8db975f77867_2932899d/commit.yaml19:37
weshayyes.. the hash formatted correctly19:37
rfolco|ruck4 first chars split in sub dirs19:37
weshaywe have a little tool for that..19:37
weshayya..19:37
weshayexactly19:37
rfolco|ruckso this is hardcoded19:38
weshayya19:38
weshayrfolco|ruck, when is that executed?19:38
rfolco|ruckI can propose a patch to get this from job definition var and include a if there in https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/pre.yaml#L1119:38
weshaydo you know if that is executed after the repo's are setup?19:39
rfolco|rucklet me check that pre19:39
weshayperhaps we try to rerun it in post.. if /etc/yum.repos.d/delorean.repo exists19:40
weshaybecause there may be times when jobs don't get as far as creating the repo19:40
weshaybut really who cares if it reports at in that case.. but better to run it twice19:40
weshayrfolco|ruck, you know what I mean?19:41
rfolco|ruckI am trying to parse your comments19:42
weshayrfolco|ruck, let me explain.. back to hangout for just a sec19:42
weshayit's important19:42
rfolco|ruckk19:42
rfolco|ruckhttp://logs.rdoproject.org/41/23141/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/b385d76/job-output.txt19:42
rfolco|rucksearch for Populate hash19:43
rfolco|ruckyou'll see when it happens... I am trying to find when the repo is setup19:43
weshayrfolco|ruck, ya.. we need to run it twice19:43
weshayrfolco|ruck, it's much later19:43
*** tesseract has quit IRC19:49
*** holser has joined #oooq19:50
*** holser has quit IRC19:53
rfolco|ruckweshay, can I read delorean.repo, get the hash and append a line to the end in the hash_info.sh here ? https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/get-hash.sh#L4920:18
rfolco|ruckso no matter what you export at the top, the last one is what counts20:19
* weshay looks20:19
rfolco|ruckhmm I need to validate that as well if the path exists for commit.yaml in dlrn trunk20:20
weshaythe delorean.repo does not exist at that time20:20
rfolco|ruckpost task not pre20:20
rfolco|ruckhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/post.yaml#L1320:21
rfolco|ruckwhen that shell runs it should exist20:21
weshayrfolco|ruck, ya.. we can override it in the same file for sure..20:22
weshayI would echo a comment into it20:22
rfolco|ruckk20:22
rfolco|rucklets see20:22
weshayso it just doesn't look like it's there twice for no reason20:22
rfolco|ruckit populates hash in pre20:23
rfolco|ruckit loads it in post20:23
rfolco|ruckhash_info I mean20:23
weshayya20:23
weshayrfolco|ruck, we should log this file too.. http://logs.rdoproject.org/41/23141/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/b385d76/logs/quickstart_files/20:24
weshayoh we do20:24
weshayhttp://logs.rdoproject.org/41/23141/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/b385d76/logs/get_hash_log.log20:24
rfolco|ruckk20:25
weshayrfolco|ruck, so ya.. so good comments will make that clear to a reader here: http://logs.rdoproject.org/41/23141/7/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/b385d76/logs/get_hash_log.log20:25
rfolco|ruckhash_info.sh you mean?20:25
weshay+ curl -sLo /home/zuul/workspace/commit.yaml https://trunk.rdoproject.org/centos7-train/tripleo-ci-testing/commit.yaml20:25
rfolco|ruckthe generated file we do not save20:25
weshayrfolco|ruck, aye.. the execution is enough20:26
weshayrfolco|ruck, even though it's sourced.. you can still http://paste.openstack.org/show/784006/20:27
zbrweshay: fyi: centos-8 nodes can now be used, you will see a msg soon.20:29
rfolco|ruckzbr, ooohooo20:29
zbris too late now to add jobs, but i will do tomorrow morning. starting with build containers? new release files?20:30
weshayrfolco|ruck, don't get too excited, we still don't have centos8 packages for rdo20:30
weshay:)20:30
weshayzbr, very cool.. what reviews?20:30
rfolco|ruck:(20:30
weshayzbr, don't add anything20:30
weshayzbr, we were told to hold off..  it's good to have the node..20:31
zbrweshay: is true that we have centos-8 on openstack-zuul only, because is built from scratch, on rdo we do not have them.20:31
zbrbecause centos-8 did not release images.20:31
weshayzbr, too late to chat?20:32
zbr5min should be ok.20:32
weshayok.. ya will be quick20:32
weshayhttps://meet.google.com/icb-foqe-cvi20:32
weshay{7} tripleoclient.tests.v1.test_container_image.TestContainerImagePush.test_take_action_local_path [0.021741s] ... FAILED21:05
weshaymostly caused by mageUploaderException: No entry for centos-7-rax-dfw-0012323069.ctlplane in /etc/hosts21:06
*** Goneri has quit IRC21:10
weshayFYI https://bugs.launchpad.net/tripleo/+bug/184827521:12
openstackLaunchpad bug 1848275 in tripleo "rpm DLRN builds failing due to ctlplane entry missing from /etc/hosts " [Critical,Triaged]21:12
*** tosky has quit IRC23:38

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!