Thursday, 2018-05-17

hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (2 more messages)00:13
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch,  (2 more messages)02:13
*** agopi has quit IRC02:40
*** agopi has joined #oooq02:42
*** agopi has quit IRC02:57
*** agopi has joined #oooq02:59
*** agopi has quit IRC03:22
*** jbadiapa has quit IRC03:35
*** jbadiapa has joined #oooq03:35
*** ykarel|away has joined #oooq03:36
*** udesale has joined #oooq03:49
*** ykarel|away has quit IRC03:58
*** udesale has quit IRC04:10
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)04:13
*** udesale has joined #oooq04:13
*** links has joined #oooq04:24
*** holser__ has joined #oooq04:33
*** udesale has quit IRC04:36
*** udesale has joined #oooq04:37
*** ykarel|away has joined #oooq04:40
*** agopi has joined #oooq04:44
*** holser__ has quit IRC04:48
*** ratailor has joined #oooq05:01
*** marios has joined #oooq05:05
*** saneax-_-|AFK is now known as saneax05:09
*** udesale has quit IRC05:20
*** udesale has joined #oooq05:21
*** udesale has quit IRC05:26
*** jtomasek has quit IRC05:36
*** udesale has joined #oooq05:43
*** quiquell|off is now known as quiquell05:49
*** agopi has quit IRC05:54
*** ykarel|away is now known as ykarel05:56
*** rlandy|bbl has quit IRC05:56
*** jfrancoa has joined #oooq06:04
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)06:13
*** jaganathan has joined #oooq06:15
*** marios has quit IRC06:16
*** marios has joined #oooq06:17
*** jaganathan has quit IRC06:28
*** jaganathan has joined #oooq06:31
*** holser__ has joined #oooq06:36
*** tesseract has joined #oooq07:01
*** ccamacho has joined #oooq07:03
*** kopecmartin has joined #oooq07:06
*** bogdando has joined #oooq07:09
*** florianf has joined #oooq07:14
*** zoli is now known as zoli|wfh07:14
*** zoli|wfh is now known as zoli07:15
*** sshnaidm|off is now known as sshnaidm|rover07:15
*** amoralej|off is now known as amoralej07:54
*** tosky has joined #oooq07:55
*** ykarel is now known as ykarel|lunch08:05
*** skramaja has joined #oooq08:05
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages)08:13
*** jtomasek has joined #oooq08:16
*** dtantsur|afk is now known as dtantsur08:26
*** links has quit IRC08:32
*** gkadam has joined #oooq08:35
*** ykarel|lunch is now known as ykarel08:37
*** jtomasek has quit IRC08:37
*** jtomasek has joined #oooq08:39
*** links has joined #oooq08:49
*** d0ugal__ has quit IRC08:51
*** d0ugal has joined #oooq08:51
*** zoli is now known as zoli|lunch09:28
*** sshnaidm|rover has quit IRC09:52
*** sshnaidm|rover has joined #oooq09:55
*** panda|off is now known as panda10:07
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate- (2 more messages)10:13
*** kopecmartin is now known as mkopec|lunch10:22
*** jaosorior has quit IRC11:01
*** jaosorior has joined #oooq11:05
quiquellpanda: It's worth to look at upgrade jobs logs at https://review.openstack.org/#/c/566565 ?11:16
*** links has quit IRC11:16
*** zoli|lunch is now known as zoli|wfh11:17
*** zoli|wfh is now known as zoli11:17
pandaquiquell: https://review.openstack.org/#/c/567060 has the full changes11:19
*** mkopec|lunch is now known as mkopec11:21
*** mkopec has quit IRC11:21
*** kopecmartin has joined #oooq11:22
*** links has joined #oooq11:30
*** amoralej is now known as amoralej|lunch11:47
*** udesale has quit IRC11:53
*** trown|outtypewww is now known as trown12:03
*** tcw has quit IRC12:03
quiquellpanda: Do we need to add a dummy change to tripleo-upgrades to test it ?12:04
quiquellAnd exercise all the featuresets we support ?12:04
quiquellOr run check-rdo-experimental12:05
pandaquiquell: we are currently testing fs 10 11 47  5012:06
pandaquiquell: the jobs are directly tested on the change, despite having a ifferent name12:07
panda47 is undercloud upgrade, 50 is undercloud containerized upgrade, 10 is containers upgrade and 11 is  ... uhm12:07
quiquellso we are missing, 36, 37 and 4012:07
pandaquiquell: only 47 and 50 are in scope for the sprint, we are adding 10 and 11 as test ofr the backward compatibility of the script12:08
quiquellpanda: Damn you are right...12:08
pandaas always!12:08
*** tcw has joined #oooq12:09
*** quiquell is now known as quique|lunch12:09
*** jaosorior has quit IRC12:12
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)12:13
*** atoth has joined #oooq12:19
*** jaosorior has joined #oooq12:19
weshaypanda, sorry.. my cal did not update on phone12:30
weshayon my phone12:30
weshaypanda, can you chat now?12:30
pandaweshay: sure, n12:31
pandanp12:31
*** tosky has quit IRC12:32
*** rlandy has joined #oooq12:33
*** tosky has joined #oooq12:36
*** Goneri has joined #oooq12:46
weshayarxcruz, ready12:46
weshayarxcruz, sorry12:46
*** quique|lunch is now known as quiquell12:48
quiquelltrown: Sorry about the rebase, was playing with the top commit12:54
trownquiquell: ya no worries, easy fix, was just a bit confused looking at it this morning12:56
quiquelltrown: hufff ok :-/12:56
quiquellWill be more careful12:56
trownworking on wiring in the get_dlrn_hash this morning... we have no current-tripleo symlink on newton :(12:59
trownpinche newton12:59
quiquell"pinche" This is the release before newton ?13:00
quiquellpinche -> newton13:00
quiquell:-)13:00
trownhehe13:00
trownthey probably dont use that as much in Spain... I learned Spanish from Mexicans :)13:00
quiquelltrown: Using the dlrn python client maybe it covers that13:01
quiquelltrown: Not much, no, maybe we have to13:01
trownquiquell: no you still pass a link name... I want to save dlrn python client for later implementation... first run was just to mirror what we do in bash with python13:02
trowntoo late in the sprint to change that13:02
quiquellok13:02
quiquellWill take a look too at least for learning13:02
*** amoralej|lunch is now known as amoralej13:05
*** links has quit IRC13:09
quiquelltrown: Maybe we can use this ?13:11
quiquellhttps://trunk.rdoproject.org/centos7-newton/promote-current-tripleo.log13:11
quiquellHumm 2016 :-/13:13
quiquellGoing to ask in the #rdo channel13:13
trownquiquell: I think we just use current... will need to special case newton13:15
quiquelltrown: Current is the same a current-tripleo ?13:16
trownquiquell: hmm looks like we use https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/newton.yml#L1713:18
trowncurrent-passed-ci13:18
quiquellThe problem will be with previous, but we are far from using it I think13:20
trownnah it wont be a big deal, we dont need to do updates on newton13:21
trownwe have no previous there13:21
quiquellAhh ok cool13:23
weshaysshnaidm|rover, you have a sec to help me understand https://review.openstack.org/#/c/568838/13:24
quiquellDamn mockin in python is much more easier than c++13:25
sshnaidm|roverweshay, sure13:26
*** ratailor has quit IRC13:26
sshnaidm|roverweshay, here?13:28
trownquiquell: we will have to add that case to unsupported... currently it tries to do newton undercloud upgrade13:29
trownquiquell: really? mock is super tricky to me... I have to relearn how to do it right every time13:29
weshaysshnaidm|rover, think I understand the outcome, just not the fix well enough13:29
weshaysshnaidm|rover, comments in patch.. voted +213:29
sshnaidm|roverweshay, yeah, you can see bug in commit message13:31
quiquelltrown: Mocking is like this https://www.mnn.com/earth-matters/animals/photos/amazing-animal-abilities/starfish-expel-their-stomachs-eat13:31
trownquiquell: it does feel like a super power13:32
trownespecially mocking language builtins like open13:32
weshaysshnaidm|rover, ya.. when we have some downtime, this is an opportunity for me to learn something so I may ping you at another time13:33
weshaybut I'd like to understand this more completely13:33
weshay%gatestatus13:36
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)13:36
chandankumar!gatestatus13:37
openstackchandankumar: Error: "gatestatus" is not a valid command.13:37
weshaychandankumar, hey brotha13:37
chandankumarweshay: Hello :-) The bot only understands anything starting with '%'13:38
*** zoli is now known as zoli|afk13:39
*** ratailor has joined #oooq13:40
*** ratailor has quit IRC13:41
weshaychandankumar, I don't see tempest getting logged to /var/log in https://review.openstack.org/#/c/566526/13:42
weshaychandankumar, can you bonk me on the head where I would see that13:42
chandankumarhttp://logs.openstack.org/26/566526/12/check/tripleo-ci-centos-7-undercloud-containers/208fec9/logs/undercloud/var/log/containers/tempest/tempest.log.txt.gz13:43
rlandymyoung|ruck: sorry didn't get a chance yesterday - will look at this bug today https://bugs.launchpad.net/tripleo/+bug/177156313:44
openstackLaunchpad bug 1771563 in tripleo "ovb reproducer fails with "AnsibleUndefinedVariable: 'dict object' has no attribute 'ansible_host'"" [High,Triaged]13:44
weshaychandankumar, heh.. ok I did see that.. why did my brain not register that as the tempest log.. maybe the formatting..13:44
weshaylolz13:44
weshaychandankumar, thanks13:45
chandankumarweshay: https://review.openstack.org/#/q/topic:tempest_log+(status:open+OR+status:merged)13:45
chandankumar3 patches in queue13:45
weshayya.. looking at those13:45
rlandyweshay: rechecking the zuul change on update - I think we were missing a var definition ... https://review.openstack.org/#/c/568946/2/playbooks/multinode-undercloud-upgrade.yml13:45
weshayapetrich, you around?13:45
rlandyupdated patch13:45
apetrichweshay, aye sir13:45
weshayrlandy, k cool.. I'm running it locally too13:45
weshayapetrich, need to chat?13:45
rlandyk - you'll need the update13:46
weshayrlandy, thanks13:46
myoung|ruckrlandy: no worries...we've had hair on fire and have not been idle anyway :)13:49
myoung|ruckci-squad: standup in 11, updated cards are fast cards :)13:50
*** skramaja has quit IRC13:52
*** kopecmartin_ has joined #oooq13:57
myoung|ruckweshay, rlandy, trown, quiquell, panda: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting, https://bluejeans.com/705085945514:00
*** kopecmartin has quit IRC14:00
*** ccamacho has quit IRC14:01
*** ccamacho has joined #oooq14:02
myoung|ruckweshay: joining standup?14:03
weshaymyoung|ruck, have a conflict14:04
*** zoli|afk is now known as zoli|wfh14:06
*** zoli|wfh is now known as zli14:06
*** zli is now known as zoli14:06
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)14:13
*** kopecmartin_ has quit IRC14:20
*** hamzy_ has joined #oooq14:28
*** hamzy has quit IRC14:29
*** kopecmartin_ has joined #oooq14:33
myoung|ruckweshay: ack no prob14:34
*** ykarel is now known as ykarel|away14:43
*** hamzy_ is now known as hamzy14:43
*** strattao has quit IRC14:44
myoung|ruckweshay:, chandankumar, kopecmartin_, arxcruz, tempest squad in ~10m: https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting, https://bluejeans.com/705085945514:49
*** ykarel|away has quit IRC14:49
*** strattao has joined #oooq14:51
rlandyweshay: pls ping me when you have a minute14:51
rlandyre: http://logs.openstack.org/31/568931/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/9cae57f/job-output.txt.gz14:52
rlandypanda: hi14:52
rlandyI'd like to fix my dry-run patch to address comments14:53
rlandybut I's like to rebase it on your fixed patch14:53
rlandyplease ping me when your patch is good to go and I'll fix and rebase mine14:53
pandarlandy: sure, give me a minute14:53
rlandypanda; no rush :)14:54
weshayrlandy, k14:55
weshayrlandy, what's up a have a couple14:55
weshayrlandy, still skipping? http://logs.openstack.org/31/568931/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/9cae57f/job-output.txt.gz#_2018-05-17_14_19_37_49007214:55
rlandyweshay: hey - ok - http://logs.openstack.org/31/568931/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/9cae57f/logs/undercloud/home/zuul/ shows the results with the renamed dir14:56
rlandyactually that happens in all14:56
rlandyhttp://logs.openstack.org/31/568931/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/9cae57f/logs/undercloud/home/zuul/install_built_repo.log.txt.gz14:56
rlandyif you compare logs14:56
weshayoh ok14:56
* weshay scrolls down then14:56
weshay2018-05-17 14:24:33.736149 | primary | TASK [build-test-packages : Run createrepo on generated rpms] ******************14:57
weshayok.. this is good14:57
rlandythe jobs still fails14:57
rlandybut it looks like on this ... https://bugs.launchpad.net/tripleo/+bug/176531314:57
openstackLaunchpad bug 1765313 in tripleo "tripleo-upgrade fails with inventory's output file awk: fatal: cannot open file `/tmp/fake-oc-hosts.yaml'" [Critical,Fix released] - Assigned to Yurii Prokulevych (yprokule)14:57
weshayon gen inventory?14:57
weshayrlandy, ok.. but your part I think worked14:57
* weshay looks at logs14:57
rlandyweshay: I think so - I want to add similiar fix for overcloud upgrades and we can review the results14:58
rlandyreally I am interested to see if the releases script is doing its job here14:58
rlandyI am reusing your review14:58
weshayrlandy++14:59
hubbotweshay: rlandy's karma is now 114:59
rlandyhard to tell if it worked entirely because the upgrade never actually ran14:59
weshayrlandy, so http://logs.openstack.org/31/568931/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/9cae57f/logs/delorean_logs/14:59
weshaylooks good to me.. a big step forward thank you14:59
weshayrlandy, so I think we just need get your work merged and let the inventory issue be handled in a diff bug14:59
rlandyweshay: ok- will keep iterating here - will ping you with next result to review14:59
rlandyfine14:59
weshayrlandy, thank you so much15:00
rlandywe have some rebasing to do before rerun and integration15:00
myoung|ruckweshay:, chandankumar, kopecmartin_, arxcruz, tempest squad in 0m: https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting, https://bluejeans.com/705085945515:01
quiquelltrown, rlandy, panda: going to push the --output-file more unit test can be added later on.15:09
*** ykarel|away has joined #oooq15:09
rlandyquiquell: fine by me15:09
quiquellDonde did a -R to prevent weird rebases15:10
trownquiquell: cool, I will do big rebase to get pep8 tests at the bottom, and then fix all the patches above15:16
pandaoh no, another rebase.15:16
pandarlandy: https://review.openstack.org/566565  anything else missing ?15:16
rlandypanda: looking15:17
quiquelltrown: Sound like a plan, already fixed pep issues now.15:17
rlandythen I'll have the rebase fun15:17
trownpanda: should be last major one... only WIP patch will be all they way at the top15:17
quiquellSee you tomorrow15:17
*** quiquell is now known as quiquell|off15:17
* panda ducks15:17
trownquiquell|off: have a good night15:17
pandaas much as a panda can duck.15:17
*** saneax is now known as saneax-_-|AFK15:21
rlandypanda: I think https://review.openstack.org/#/c/566565/ looks good. I ma going to rebase the dry run patch on those changes and check the full run15:23
*** saneax-_-|AFK is now known as saneax15:23
pandarlandy: ok, expect seom conflict, I didn't exaclty backport15:24
*** saneax is now known as saneax-_-|AFK15:27
rlandyyep - fun rebase times15:28
rlandyalso I need to pick the top releases file patch to depend on15:29
rlandy^^ trown: what is the top patch now days?15:30
trownrlandy: give me a sec, and I will have the rebase, just fixing some pep8 stuff along the way15:32
rlandytrown: sure - no rush - I am having a rebase party of my own right now15:32
trownrlandy: hopefully it doesnt involve the pep8 and unit test patches15:32
rlandytrown: no - I am not touching those patches15:33
rlandypanda's work and mine in toci15:33
trownrlandy: cool, I will have the other patches rebased with passing pep8 shortly15:33
rlandygreat15:34
*** sshnaidm|rover is now known as sshnaidm|bbl15:41
*** marios has quit IRC15:45
*** sshnaidm|bbl has quit IRC15:46
*** lucasagomes is now known as lucas-afk15:48
*** agopi has joined #oooq15:58
trownrlandy: ok... that was harder than expected :P, but https://review.openstack.org/569120 is the head of the chain for now, only excludes my WIP patch to wire in the dlrn hash which is giving me an issue16:01
rlandytrown: perfect - thanks16:02
rlandyI just need something to test with16:02
*** trown is now known as trown|lunch16:02
weshaymyoung|ruck, please start preparing for master/queens rdo216:02
weshaymyoung|ruck, I'm asking for the packstack blocking jobs in rdo1 be waived16:03
weshaymyoung|ruck, which will immediately send a red block to rdo phase 216:03
myoung|ruckack, btw looks like container build job failed again with 504's, this time tagging...updating bug16:04
myoung|ruckweshay: re: rdo2, ack, things will fire when rdo1 promotes16:04
weshaymyoung|ruck, are the branches updated?16:08
weshayhave you kicked it off recently to see if we have any infra issues16:09
weshayetc16:09
weshayetc16:09
* myoung|ruck will do those things and adds it to the list16:10
myoung|rucklist.incoming_rate > this.capacity()16:11
myoung|ruckwill have update on rdo2 status shortly16:11
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate- (2 more messages)16:13
*** jaganathan has quit IRC16:23
*** jaganathan has joined #oooq16:23
*** jrist has quit IRC16:24
rlandypanda: what do you think of sagi's comment regarding capturing env vars? https://review.openstack.org/#/c/567060/22/toci_quickstart.sh16:25
rlandyI am not sure what to do here16:25
rlandy"Let's save only things we need to run oooq"16:26
rlandythat I don't know16:26
rlandyit would require some update16:26
*** kopecmartin_ has quit IRC16:26
rlandyif we change what vars we use16:27
pandarlandy: line 121 ?16:27
rlandypanda: yes16:27
rlandyreally we are capturing the release args in a diff file16:28
rlandyand the playbook commands in a separate file16:28
rlandythis was supposed to be a catch all for other vars16:28
rlandyfor sure it should still go to the logs dir16:29
pandarlandy: we ma not need to log the whole env, but then again, anone can put a patch with echo $ENV and see everything even if it's not merged16:29
pandam keboard is eating all the16:30
rlandypanda: or I could do what the comment suggest16:30
pandayyyyyyyyy16:30
rlandyand add the dumpvar function and call it16:30
rlandythe point is avoid people having to comb through the console output16:31
rlandypanda: ^^? thoughts?16:32
rlandykeeping that list up to date is what concerns me16:32
pandathis was present in tripleo.sh. I think it's a valid point for the bash dictionaries, and to log only what we effectively use. For the security, everyone can put a patch with a echo $env and see everything. If we do it first, we may catch something  we don't want to expose earlier, and the solution is to not export it at all in bash16:33
pandaYes the other concern is maintaining this list16:33
myoung|ruckweshay: regarding RDO2: queens, https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 @L6016:34
* myoung|ruck will biaf, lunch/food time16:34
pandaand I would not do it like this, using cat << EOF > file for me is an antipattern16:34
pandaI will comment on the patch also16:35
rlandypanda: for now, I'll accept the comment - add the function as he has the list and call it in toci-quickstart16:35
rlandywe can iterate on that going forward16:35
*** sshnaidm|bbl has joined #oooq16:35
rlandyit's the safer if not the most complete solution16:35
pandasafer meaning ?16:36
rlandyno risk of leaking16:36
rlandybut may be incomplete very soon16:36
pandaeverybody can leak16:37
pandaI put a patch now in tripleoci with echo $ENV and I'll leak16:37
pandawe should not put eanything taht can appear in env in the first place16:38
rlandyright but that will be your fault not mine16:38
*** tesseract has quit IRC16:38
rlandycorrect there because it will be in the console16:38
pandait's one of the reason why te-broker exists and it's hidden to the user16:38
pandanot my or oyur fault16:39
pandaany persone with access to gerrit can put a echo $ENV as patch in tripleo-ci16:40
rlandypanda: ok - the question is what do we want in this patch??16:40
*** holser__ has quit IRC16:41
rlandypanda: I am submitting the patch with the changes (other than env vars - which is undecided at the moment) so we can see the tests run. We can update that piece later16:48
*** bogdando has quit IRC16:50
pandarlandy: have time to chat ?16:53
rlandyok16:53
*** agopi has quit IRC16:53
rlandyyour bj?16:54
pandarlandy: ok16:54
*** amoralej is now known as amoralej|off16:58
*** ccamacho has quit IRC17:02
*** trown|lunch is now known as trown|mtg17:02
*** florianf has quit IRC17:02
*** dtantsur is now known as dtantsur|afk17:04
*** ykarel|away has quit IRC17:05
*** jtomasek has quit IRC17:07
weshaymyoung|ruck, hands on deck for rdo217:08
weshaymyoung|ruck, we're going to send master/queens through17:08
*** ykarel has joined #oooq17:09
*** panda is now known as panda|off17:14
*** jfrancoa has quit IRC17:24
*** ykarel_ has joined #oooq17:25
*** jfrancoa has joined #oooq17:27
*** ykarel has quit IRC17:27
*** ykarel_ has quit IRC17:30
*** udesale has joined #oooq17:32
*** udesale has quit IRC17:33
*** ykarel has joined #oooq17:33
*** jfrancoa has quit IRC17:52
*** ykarel has quit IRC17:59
*** links has joined #oooq17:59
*** zoli is now known as zoli|gone18:04
*** zoli|gone is now known as zoli18:04
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, gate- (2 more messages)18:13
*** trown|mtg is now known as trown18:22
myoung|ruckweshay: fyi etherpad has a borderline obscene level of detail on this particular promotion. ci.centos jobs just finished, next promoter shoudl promote.  as soon as current jobs finish I'll immediatly kick it.18:23
weshaymyoung|ruck, master has been promoting for the last 30min18:23
myoung|ruckweshay: been going thru the jobs18:23
myoung|rucktalkin queens18:23
myoung|rucktracking master too18:24
myoung|ruckweshay: talked with reldel folk btw re: 13, they are not doing imports so queens is a meh for them...at least not a super priority as it's just cherrypicks at this point.  that said jobs in rdo2 will run anyway and I'll watch it thru18:25
myoung|ruckdetails in etherpad etc.  prepping rdo2 for master now18:25
weshaymyoung|ruck, ya.. so I would start w/ master18:25
weshayhowever we'd also rather not go into the program call w/ a RED rhos-13/queens build18:25
myoung|ruckindeed...hence the level of detail @ https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 L64.  We were at 6d coming into this18:27
myoung|ruckFF stable/master branches now as well18:27
weshaymyoung|ruck, thanks man.. let's get er done18:30
weshaymyoung|ruck, fs20, 1 out 3 bm, and maybe the ha job18:30
weshaythat's all we need18:30
weshayfs20 should pass https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/18:31
weshaymyoung|ruck, we may need to update to cent75 for the vm as well18:32
weshayshouldn't matter but just sayin18:32
weshayyum update will suck there too18:32
weshaymyoung|ruck, k?18:32
myoung|ruckyeah...promote's gonna be another hour...tempted to kick rdo2:master for the data point...last run was 11 may and 100% fail18:33
*** links has quit IRC18:35
myoung|ruckweshay: ack on the centos 75 vm18:41
myoung|ruckweshay: will see what happens..might just be a long yum update18:41
weshaymyoung|ruck, well.. it's all overcloud as undercloud right?18:42
weshayso they should be at 7.518:42
weshayto start18:42
myoung|ruckyes18:42
weshaynow that I'm thinking through where those jobs are18:42
myoung|rucki thought you meant the slave VM's18:42
weshaythe slaves are fedora18:42
myoung|ruckthose are fedora tho and wires crossed18:42
weshaynot related to the 7.4 -> 7.5 centos update18:43
myoung|ruckcorrect18:43
myoung|ruckweshay: sry was talking about a diff issue. https://review.rdoproject.org/etherpad/p/ruckrover-sprint13 L 9718:44
weshaythe bm jobs run on weshay-private18:45
weshaymyoung|ruck, do you see why that failed18:47
weshayit's certainly not what I hit a few weeks ago18:48
*** rlandy is now known as rlandy|brb19:01
*** matbu has quit IRC19:19
*** matbu has joined #oooq19:25
*** rlandy|brb is now known as rlandy19:27
*** dtantsur|afk has quit IRC19:29
*** moguimar has quit IRC19:30
*** moguimar has joined #oooq19:32
*** dtantsur has joined #oooq19:36
*** jrist has joined #oooq19:36
*** moguimar has quit IRC19:37
*** moguimar has joined #oooq19:39
*** dtantsur has quit IRC19:40
*** jrist has quit IRC19:48
*** dtantsur has joined #oooq19:51
*** jrist has joined #oooq19:51
*** gkadam has quit IRC20:08
hubbotFAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @  (2 more messages)20:13
myoung|ruckweshay https://code.engineering.redhat.com/gerrit/139133 ospphase0: latest --> passed_phase_1 puddle symlinks20:22
myoung|ruckweshay: re: queens rdo2, (on previous hash) fs020 is into tempest, and we have the rasca bmu job passing on previous hash.  one of the BM (envE) is failing on OC validate.  when queens promotion (in flight) finishes those jobs will kick.20:34
myoung|ruckweshay: since https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-queens-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans is the most reliable BM job for queens atm, methinks we should adjust queens.ini20:34
myoung|ruckrasca: fyi ^^20:34
weshaymyoung|ruck, I think we should try to fix what is killing the 2 jobs in 5 min20:37
weshayhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-queens-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/20:38
weshayis actually pretty consistent20:38
myoung|ruckaye up till this month20:38
* myoung|ruck watches that one with a hairy eyeball w/ incoming hash20:38
myoung|ruckthat one just failed (on previous hash) a few mins ago during OC validate, unclear why20:39
rlandyweshay: hello - I cleaned up the test job ... https://review.openstack.org/#/c/569229/ - depending on https://review.openstack.org/#/c/568946/ which has both the undercloud and overcloud upgrades playbooks updated20:39
rlandylet me know if we need anything else at this point20:40
*** ccamacho has joined #oooq20:40
weshayrlandy, https://review.openstack.org/#/c/567060/24/README.rst is deprecated20:42
weshayI think I saw a patch to remove it20:43
weshayThe doc you added is great but should be in the log that is presented in the log files20:43
rlandyha - was there when I edited it20:43
weshaydid someone suggest otherwise?20:44
weshayeven if it's linked to https://github.com/openstack-infra/tripleo-ci/blob/master/docs/tripleo-quickstart-logs.html20:44
weshaythat is fine20:44
weshayrlandy, thank you for http://logs.openstack.org/60/567060/22/check/tripleo-ci-centos-7-undercloud-containers/b9b4a5d/logs/playbook_executions.txt.gz20:44
weshayso my eyes don't have to bleed anymore20:45
rlandyweshay: np  - I can move the doc there20:45
rlandylol20:45
weshayrlandy, while your there can you please put a direct link to undercloud/var/log and subnode-2/var/log20:45
rlandywith any partocular text?20:45
weshayrlandy, under Links to common log files20:46
rlandyfine20:46
weshayrlandy, maybe we want a used_variables section20:47
rlandywe have that ...20:47
weshaythat links to toci_env_args_output.log and http://logs.openstack.org/60/567060/22/check/tripleo-ci-centos-7-undercloud-containers/b9b4a5d/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz20:47
* weshay looks20:47
rlandy$LOGS_DIR/toci_env_args_output.log20:47
rlandyoh20:47
weshayrlandy, right.. but to add to the html20:47
rlandysomething on the doc20:48
weshaysuch there is a section just for vars20:48
weshaywhere you point to $LOGS_DIR/toci_env_args_output.log and dump_variables_vars.json.txt.gz20:48
myoung|ruckweshay: (observation) we're still building RDO on RHEL undercloud images for master (not queens tho...we never decided to turn it on).  is rfolco still looking at replicating this on internal SF (rdo on rhel)?  should I just rip this out and/or disable, or is it still worthwhile to have a canary to see if we can still build UC's with RDO atop RHEL bits?  We're not testing them as we don't have RDO on RHEL containers20:48
weshayrlandy, no good deed goes unpunished when asking for suggestions20:48
* myoung|ruck see's an hour of jobtime that can evaporate if it's no longer a goal20:48
weshaymyoung|ruck, we should not be testing or building rdo on rhel in master or queens20:49
rlandyweshay: actually I asked for suggestions about the zuul changes20:49
weshayor pike20:49
rlandybut happy for feedback wherever :)20:49
weshay:)20:49
weshayk20:49
* myoung|ruck proceeds to kill his child20:49
weshayrlandy, any hoo.. coming add it w/ the perspective of someone who has never used the logs before20:49
weshaymyoung|ruck, ok abraham :)20:50
myoung|rucklolz20:50
myoung|ruckhaven't had dirty hands in here for a while20:50
myoung|ruckwant me to rip it out with extreme predjudice?  or just disable it?20:50
weshaydoesn't make any sense to do it w/o the containers20:50
weshayI don't see it coming back20:50
myoung|ruckwhat i meant was...is rdo on rhel (containers) still a goal.20:50
myoung|ruckin the future20:50
weshayto consume them here, yes20:51
weshaynot to build them here20:51
myoung|ruckif it is...then the kung fu to deal with dependenciy issues we sorted on those UC jobs is still worth preserving20:51
myoung|ruckok20:51
myoung|ruckfor now i'll just switch it off20:51
rlandylol - I just saw the abraham joke above20:51
myoung|ruckwhoever / wherever builds rdo-on-rhel containers can rip off the repo setup stuff20:52
myoung|ruckrlandy: i like the clean logs too.  we all end up compiling text without line breaks and it's silly :)20:52
*** dtantsur has quit IRC21:02
*** dtantsur has joined #oooq21:03
*** moguimar has quit IRC21:04
*** moguimar has joined #oooq21:04
weshaymyoung|ruck, afaict.. the haa job does not validate21:08
weshayso using it for bm is not an option21:08
*** Goneri has quit IRC21:08
*** trown is now known as trown|outtypewww21:09
weshayrlandy, before you leave for the holiday21:13
weshayrlandy, maybe we can pause the doc work, as I can probably back fill on that21:13
rlandyweshay: I am here tomorrow21:13
rlandyI am only out on monday21:14
weshayrlandy, right.. so how close are we to merging your unit test job21:14
rlandythey can merge afaik21:14
rlandythere is an optimization I am working on21:14
rlandyto get rid of the multiple files lines but the original patches are good to go21:14
weshayrlandy, panda|off trown|outtypewww quiquell|off I think we want to merge rlandy's job prior to merging anything else right?21:15
rlandyweshay: it's part of the stack that trown|outtypewww rebased today21:15
rlandyI figure we can start merging that whole stack tomorrow21:15
rlandyzuul permitting21:15
* weshay looking at https://review.openstack.org/#/c/567320/21:15
weshayfor your job21:15
rlandyhttps://review.openstack.org/#/c/567649 ... and21:16
rlandyhttps://review.openstack.org/#/c/56828721:16
weshayya.. looking at those21:17
myoung|ruckweshay: https://code.engineering.redhat.com/gerrit/13913821:18
rlandyyou will see the duplication I want to get rid of21:18
myoung|ruckweshay: ^^ Remove RDO on RHEL images and tests (master,queens,pik21:18
rlandybut they can go as is right now21:18
weshaythanks myoung|ruck21:19
weshayrlandy, ok.. so the most important thing imho is getting your tests fixed to your liking.. you can pass doc work to me21:20
weshayif needed21:20
rlandyit's fine - I can do both21:20
rlandyand the tests can go as they are21:21
rlandyit's just an optimization21:21
rlandythey are functional21:21
weshayrlandy, hrm.. is this turned on for any job? https://review.openstack.org/#/c/566565/21:25
weshaynot finding the emit log for 3721:26
rlandyif you run the fs files specified21:26
rlandyif [[ $FEATURESET_FILE =~ '047' || $FEATURESET_FILE =~ '050' || $FEATURESET_FILE =~ '010' || $FEATURESET_FILE =~ '011' ]];then21:26
rlandy    RELEASES_SCRIPT_ENABLED_FEATURESET=true21:26
rlandyfi21:26
weshay47?21:26
rlandylook at fs05021:26
weshaywhat is 47?21:26
* weshay looks21:26
weshaywe wanted 3721:26
rlandymaybe that should be 37?21:26
weshaylolz :)21:26
rlandyclose enough???21:26
weshaylolz21:26
rlandyoff by 10 error?21:27
rlandyI'll fix it21:27
rlandyfew minutes just to finish doc21:27
weshayk21:27
rlandyweshay: wrt direct link to subnode2 - that is not always there21:30
rlandyundercloud is always there21:30
weshayya.. tru21:30
rlandyI can make a comment that there is a similar dir on other nodes21:31
weshayI wish html didn't suck21:31
weshayrlandy, just would save me 20 seconds everyday21:31
weshayI've started to use those21:31
rlandyyou can log that one after wish for world peace21:31
weshaysure21:31
weshayit has to be static too21:32
rlandyif we were using templated json, we could do something better21:32
weshayrlandy, we can't though..21:32
weshayit's a infra thing..21:33
weshayI have it bookmarked somewhere, we could comment it in the html.. if exists <link>21:33
weshaybut meh21:33
weshaymyoung|ruck, +221:38
*** matbu has quit IRC21:53
myoung|ruckweshay: fyi rhos-12 and rhos-13 gate ovb gate jobs are flying overnight with stable branches --> master:head for tq/tqe21:54
weshaymyoung|ruck, they need to be at head21:55
weshayhead + the change21:55
weshaymyoung|ruck, if they have been set up to use stable branches then we were doing it wrong21:56
myoung|ruckweshay: https://review.openstack.org/#/c/566911 (from notes) was the "next great hope"21:56
myoung|ruckoh yes i rebased ({12,13} * {tq, tqe})21:57
rlandyweshay: better? https://review.openstack.org/#/c/567060/21:57
weshaymyoung|ruck, right. .but the jobs for the ovb gate osp13 or 1221:57
* rlandy fixes fs 047 to 037 now21:57
weshayshould not use stable branches, myoung|ruck please reveiw that21:57
weshayrlandy, ya.. but we still  need to note.. the emit log and other new artifacts21:59
weshayrlandy, referring to https://review.openstack.org/#/c/567060/25..26/docs/tripleo-quickstart-logs.html21:59
weshayI think21:59
weshayrlandy, paste the diff.. I can't find the 47 -> 37 now22:00
weshay:((22:00
weshayrlandy, OHH https://review.openstack.org/#/c/567060/24..26/docs/tripleo-quickstart-logs.html22:01
weshayrlandy++22:01
hubbotweshay: rlandy's karma is now 222:01
rlandyweshay: the emit log is releases.sh22:01
rlandyand 47-37 change is happening on another patch22:01
rlandycoming up in a sec22:01
weshayha.. /me read * rlandy fixes fs 047 to 037 now22:01
weshaytoo quickly22:01
weshaystupid wes22:01
* rlandy is slow22:01
rlandyand half thinking about cheesecake recipes :)22:02
weshayhow could you not22:02
myoung|ruckweshay: ack you're correct...TQ gate uses master of TQE and gerrit branch for TQ, converse for TQE gate.  in a perfect world IMHO we would be using stable for fixed, and gerrit branch for change in test...so always going from known --> known.  presently these gates have both varying at same time22:03
weshayboth check jobs need to head22:03
myoung|ruckanyway...kicking rhos-12 and 13 (that DO use stable branches for TQ/TQE overnight as well, hitting phase1 puddles22:03
weshaymyoung|ruck, k.. for periodic jobs.. we can use stable22:04
rlandyweshay: ok - https://review.openstack.org/#/c/566565/22:04
weshaythat is fine22:04
weshayrlandy, rock.. thanks!22:04
myoung|ruckyeah...i feel mixed about it...what we have will not run into the issue of dependencies being chicken/egg for tq/tqe22:04
weshaymyoung|ruck, it's not an opinion I think22:05
myoung|ruckby using head/master for TQ (in the TQE gate) for example22:05
weshaymyoung|ruck, do our other check jobs run off stable branches22:05
weshaynope22:05
weshaymyoung|ruck, not sure why that happened22:05
weshaymyoung|ruck, it's not even valid imho to test one w/ a change and have the other on some older branch22:05
weshaynot sure why anyone would think it was22:06
weshaynot sure why I'm having this conversation22:06
myoung|ruckwhat i mean is...when we're gating a change to TQE using for example rhos-12, the version of TQ is head.  so a delta exists between TQ:head, and TQ:{last-known-good-commit-working-with-rhos12},22:06
* myoung|ruck lets it go for a different day22:07
rlandyweshay: rechecking https://review.openstack.org/#/c/569229 to see zuul updates with latest changes22:09
hubbotFAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages)22:13
*** myoung|ruck is now known as myoung|ruck|off22:31
*** rlandy is now known as rlandy|biab22:38
*** tcw has quit IRC22:56
*** tcw has joined #oooq22:57
*** tosky has quit IRC23:36
*** rlandy|biab is now known as rlandy23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!