Friday, 2018-09-21

*** agopi|brb has joined #oooq00:05
*** rlandy|bbl is now known as rlandy01:46
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @  (2 more messages)01:47
rlandyagopi|brb: you around?01:53
rlandyhttps://logs.rdoproject.org/10/16310/4/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/adaffd2/job-output.txt.gz#_2018-09-20_23_36_50_61899501:54
agopi|brbyes rlandy02:23
agopi|brbi'll look into it tomorrow about to hit the hay02:24
agopi|brbthanks rlandy02:24
*** apetrich has quit IRC02:48
*** ykarel has joined #oooq03:00
*** rodrigods has quit IRC03:02
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @  (2 more messages)03:47
*** skramaja has joined #oooq04:20
*** abishop has quit IRC04:28
*** udesale has joined #oooq04:29
*** sanjayu_ has joined #oooq04:30
*** kopecmartin|away has quit IRC04:31
*** sshnaidm|pto has joined #oooq04:34
*** sanjayu_ has quit IRC04:37
*** abishop has joined #oooq04:37
*** kopecmartin has joined #oooq04:37
*** rfolco has quit IRC04:41
*** abishop has quit IRC04:46
*** abishop has joined #oooq04:46
*** ssbarnea|bkp has joined #oooq05:08
*** saneax has joined #oooq05:09
*** ssbarnea|bkp2 has quit IRC05:10
*** rlandy has quit IRC05:10
*** ratailor has joined #oooq05:11
*** ykarel has quit IRC05:21
*** quique|rover|off is now known as quiquell|rover05:43
hubbot1FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo- (2 more messages)05:47
*** ssbarnea|bkp has quit IRC05:49
*** ykarel has joined #oooq05:51
*** sshnaidm|pto has quit IRC06:06
*** jfrancoa has joined #oooq06:10
*** gkadam has joined #oooq06:13
*** gkadam has quit IRC06:13
*** jtomasek has joined #oooq06:22
*** chkumar|off is now known as chkumar|ruck06:23
*** quiquell|rover is now known as quique|rover|brb06:25
*** jaganathan has joined #oooq06:32
*** marios has joined #oooq06:38
*** marios has quit IRC06:38
*** marios has joined #oooq06:40
*** ssbarnea|bkp2 has joined #oooq06:58
*** saneax has quit IRC07:00
*** saneax has joined #oooq07:00
*** saneax has quit IRC07:12
*** saneax has joined #oooq07:14
*** quique|rover|brb is now known as quiquell|rover07:14
*** saneax has quit IRC07:15
*** saneax has joined #oooq07:17
*** saneax has quit IRC07:17
*** gkadam has joined #oooq07:19
*** holser_ has joined #oooq07:34
*** bogdando has joined #oooq07:37
*** jfrancoa has quit IRC07:39
*** tosky has joined #oooq07:40
hubbot1FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo- (2 more messages)07:47
*** ykarel is now known as ykarel|lunch07:48
*** dtantsur has joined #oooq07:54
quiquell|rover%config plugins.GateStatus.changeIDs08:06
hubbot1quiquell|rover: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab Ia5bcd5568086a6afbcebe4cbdc86c26502039f23 I67ba1d4f80d8cc34fd3735d647b62e3179aaaa1408:06
quiquell|rovershit we have the noop changes screwed up08:08
*** jfrancoa has joined #oooq08:09
quiquell|rovermarios: Do you know why wes did the master noop to tht ?08:10
quiquell|rovermarios: hubbot cannot recheck on gerrit changeid with multiple commits08:10
mariosquiquell|rover: no idea08:10
marioshttps://review.openstack.org/#/c/560445 this you mean?08:11
mariosquiquell|rover: you must mean something else08:11
quiquell|rovermarios: You were asking about rocky noop not executed ?08:12
quiquell|roverThis is the issue https://review.openstack.org/#/q/change:+I67ba1d4f80d8cc34fd3735d647b62e3179aaaa1408:13
quiquell|roverchkumar|ruck: ^ noop rocky is not running08:13
quiquell|roverchkumar|ruck: Going to reconstruct rocky/master changes08:13
chkumar|ruckquiquell|rover: which is what i was taking twodays back like rocky one is not getting updated08:13
chkumar|ruckquiquell|rover: does it get auto triggered?08:14
quiquell|roverchkumar|ruck: Do you know why wes added the master noop to tht ?08:14
quiquell|roverchkumar|ruck: yep by hubbot08:14
quiquell|roverchkumar|ruck: Going to fix it, but I want to know if know master have to do the change at tht08:14
chkumar|ruckquiquell|rover: no idea, those were added to check the gate health chekc status08:14
mariosquiquell|rover: OH, HOW THE HELL DID THAT HAPPEN ?08:14
quiquell|roverchkumar|ruck: we can add master to tqe and tht though08:14
mariosquiquell|rover: sorry for shouting08:14
mariosaccidental08:14
marios:)08:14
*** hubbot1 has quit IRC08:42
chkumar|ruckbrb08:43
*** hubbot1 has joined #oooq08:44
quiquell|rover%config plugins.GateStatus.changeIDs08:44
hubbot1quiquell|rover: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab Ia5bcd5568086a6afbcebe4cbdc86c26502039f23 If78f672a730bd5d512ee1129edc0b6e2f8abcf5d Icfa6098b294009b4c04a6d9db22162763ea153fd08:44
quiquell|rovermarios, chkumar|ruck: ^ fixed08:47
arxcruzykarel|lunch: chkumar|ruck what do i need to have this merged? https://review.rdoproject.org/r/#/c/16216/08:53
*** saneax has joined #oooq08:58
*** jtomasek has quit IRC09:05
chkumar|ruckarxcruz: I need to run fedora-review, I will get it done EOD09:08
*** jaosorior is now known as jaosorior_lunch09:09
*** ykarel|lunch is now known as ykarel09:11
*** jtomasek has joined #oooq09:21
ssbarneaquiquell|rover: please revive https://review.openstack.org/#/c/588475/ - the queue is normal now.09:29
quiquell|roverssbarnea: we cannot workflow09:29
quiquell|roverssbarnea: sone needed reviews for promotion are still not merged,09:29
ssbarneaquiquell|rover: ohh,... can I help with something?09:30
quiquell|roverssbarnea: Maybe you can do an offer to some minor god09:30
quiquell|roverssbarnea: or the like09:30
ssbarneaquiquell|rover: yeah, i wante to say something similar but I didn't want to offend anyone, or any-god.09:31
quiquell|roverssbarnea: you are smart09:32
quiquell|roverjfrancoa: You have to change new workflow here https://review.openstack.org/#/c/60412109:32
quiquell|roverjfrancoa: Or you will not see any effect upstream09:33
jfrancoaquiquell|rover: ack, thanks! I was wondering why it didn't work for upstream jobs09:34
quiquell|roverjfrancoa: toci_gate_test.sh and similars are only used at RDO jobs and reproducer09:37
quiquell|roverjfrancoa: you have to change it in both09:37
jfrancoaquiquell|rover: I didn't know they were duplicated..I guess you'll get rid of one once legacy jobs are also migrated to the new workflow, right?09:37
quiquell|roverjfrancoa: yep09:37
quiquell|roverjfrancoa: fix it upstream, then translate it to legacy, maybe you are lucky and legacy does not exists when you go to it09:38
quiquell|roverjfrancoa: Problem is upgrades are running at RDO, most of them09:38
jfrancoaquiquell|rover: yes, if we manage to fix this, we should do it for both workflows so far09:39
quiquell|roverjfrancoa: the playbook list of the new workflow is at base jobs09:41
quiquell|roverjfrancoa: we have new base jobs for multinode, singlenode, standalone09:41
jfrancoaquiquell|rover: yes, I found it before. I added the new playbook in there too09:41
ssbarnea|bkp2btw, regarding legacy jobs, how hard is to convert them? if is not too complex and if someone can show me an example, I would maybe do some of them. infra guys are keep asking me about that.09:42
quiquell|roverssbarnea|bkp2: very, you have to talk with rest of the team, mainly rlandy09:46
panda|offssbarnea|bkp2: we're almost there10:17
*** quiquell|rover is now known as quique|rover|afk10:26
*** quique|rover|afk is now known as quiquell|rover10:29
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata- (2 more messages)10:44
chkumar|ruckquiquell|rover: do we have the bug for this issue https://logs.rdoproject.org/24/567224/109/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/b3b9f81/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-09-21_09_15_2710:52
chkumar|ruckfs01 and fs025 queens third party jobs are failing in check jobs10:52
chkumar|ruck*fs02510:52
chkumar|ruck*fs03510:53
quiquell|roverThat's the from yesterday, looks like related to the stacks not deleted10:57
quiquell|roverchkumar|ruck: goint to verify it11:00
chkumar|ruckquiquell|rover: ack11:00
*** udesale has quit IRC11:05
quiquell|roverchkumar|ruck: is the rootwrap thing11:05
quiquell|roverI think we have a fix for it, but we need a promotion11:06
quiquell|roverchkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/179307311:07
openstackLaunchpad bug 1793073 in tripleo "[queens] fs01 noop job failed with Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)11:07
*** abishop has quit IRC11:09
*** udesale has joined #oooq11:13
chkumar|ruckquiquell|rover: ykarel I might logout not felling well after CIX call11:31
ykarelchkumar|ruck, ack have rest11:31
*** udesale has quit IRC11:34
quiquell|roverchkumar|ruck: yep rest, it's a hard week11:34
*** udesale has joined #oooq11:34
*** chkumar|ruck is now known as chkumar|off11:46
*** apetrich has joined #oooq11:48
weshayruck/rover killed chandan :(12:02
arxcruzssbarnea|bkp2: hey, i'm working on another stuff, not on the https://review.openstack.org/#/c/580361 plus, this role will be decoupled from tqe near future ;)12:06
ssbarnea|bkp2arxcruz but until that moment, this CR should go it, right? that was my impression.12:08
arxcruzssbarnea|bkp2: well, there are other stuff broken right now on stackviz, so, i put this change on hold, once we have the rpm package for stackviz, we can get rid of all this pip and virtualenv stuff12:08
ssbarneaarxcruz: in this case feel free to abandon it, and include a link to the other work.12:09
ssbarneait can easily be restored if needed.12:10
*** jaosorior_lunch is now known as jaosorior12:11
*** trown|outtypewww is now known as trown12:14
*** rfolco has joined #oooq12:14
*** udesale has quit IRC12:26
*** toure|gone is now known as toure12:28
*** rlandy has joined #oooq12:32
*** quiquell|rover is now known as quique|rover|lch12:32
*** gkadam has quit IRC12:34
*** ratailor has quit IRC12:34
rlandypanda|off is away today :(12:38
*** skramaja has quit IRC12:42
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch @ https://review.openstack.org/560445, stable/queens:  (1 more message)12:44
rlandyweshay: hello - not sure if you were following the progress yesterday but we got the main of the job passing with an error in the browbeat piece - I am asking about that on #sf-dfg12:44
*** agopi|brb is now known as agopi12:45
rlandyok if we move forward and add a fs001 job to try?12:45
rlandyalso we will need to try multinode jobs - doing that now12:45
rlandyrfolco: maybe you know the answer ...12:45
rlandyhttps://github.com/openstack/browbeat/blob/master/ansible/oooq/roles/pre-install-setup/tasks/main.yml#L1812:45
rlandythe browbeat team wants to copy their repo12:46
rlandy in the v3 setup, where is that repo put down?12:46
rfolcorlandy, once you add as required-project, it will under /home/zuul/src/git.openstack.org/openstack/browbeat12:47
rlandyrfolco: ok - changing that line12:48
agopirlandy, that will mean the zuul legacy job will fail right?12:48
rfolcorlandy, there is a variable for that path, searching12:49
rlandyagopi: not necessarily12:49
agopiyeah that'd be great rfolco12:49
rlandyyeah12:49
agopiokay rlandy12:49
rlandysee line 1912:49
rlandythat is exactly what we did for workspace12:50
rlandywe need another line like that12:50
agopiyeah maybe zuul_version or something like that ?12:50
agopihttps://logs.rdoproject.org/10/16310/4/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/adaffd2/job-output.txt.gz#_2018-09-20_21_34_40_22446812:51
agopilooks like these are the only env vars available? maybe i'm wrong12:51
rfolcorlandy, found it.. "{{ ansible_user_dir }}/{{ zuul.projects['git.openstack.org/openstack/browbeat'].src_dir }}"12:53
*** panda|off is now known as panda12:53
rfolcolooking if there is a shorter version of this ^12:53
*** ssbarnea|bkp2 has quit IRC12:54
pandarlandy: I'm here12:54
rlandypanda: hi12:54
rlandypanda: fixing the browbeat issue ^^12:54
*** quique|rover|lch is now known as quiquell|rover12:55
rlandypanda: today's fun will be a test multinode job :)12:55
pandarlandy: yay12:55
quiquell|roverweshay, rlandy, panda: forcing periodics https://review.rdoproject.org/r/1636012:55
rfolcorlandy, that should be the right var... can't think on anything better than that12:56
rlandyquiquell|rover: usual - pls add a revert but +212:57
pandaquiquell|rover: that will force only the periodic for stable jobs12:57
agopirfolco, just curious, does this var work with legacy job as well?12:57
pandaquiquell|rover: is it what you're trying to do ?12:58
rfolcoagopi, no, zuulv3 only12:58
agopicool, thanks rfolco12:58
pandaquiquell|rover: also, not sure what's the time zone of those, 13:30 may have already passed12:58
rfolcoah 'legacy' rdo jobs are zuulv3 already, and they use required-projects... so as long as the job already uses required-project, it should work12:59
rfolcoagopi, ^12:59
quiquell|roverpanda: it's UTC, lat's time it worked12:59
quiquell|roverpanda: the periodic jobs are not running at this pipeline ?12:59
quiquell|roverpanda: ack, openstack-periodic13:00
pandaquiquell|rover: that one is every 24h, only for stable jobs13:01
quiquell|roverpanda: I see only queens/pike there13:01
agopioh nicee, that's awesome. thanks rfolco.13:02
pandaquiquell|rover: yes oldstable jobs13:02
agopiwant me to make the change rlandy ?13:02
pandaquiquell|rover: anyway, too late13:02
pandaquiquell|rover: already merged.13:02
rlandyagopi: yeah - of you and rfolco can out in a change that I can test v3 with, that would be awesome13:02
agopiokay i'll go ahead and make changes to browbeat repo then13:03
quiquell|roverpanda: wee ned queens too13:03
rlandypanda: weshay: can you vote on https://review.openstack.org/#/c/604150/?13:03
rlandythe rdoproject stuff we can merge easily13:04
*** quiquell|rover is now known as quique|rover|afk13:13
rlandyagopi: rfolco: pls ping me when the change is ready and I'll test with it13:13
rfolcoagopi, let me know any questions that I can help13:14
agopirlandy, it's ready https://review.openstack.org/#/c/604355/13:17
agopirfolco, i'm unable to git review -d 1631013:17
agopihttps://review.rdoproject.org/r/#/c/16310/13:17
agopifor this project so that i can add depends on13:17
rlandyagopi - I'll do that part13:17
agopiokay cool rlandy thanks :)13:18
agopilmk if anything else needs to be done13:18
rlandyrechecking - let's see13:19
rlandythank you13:19
agopiack rlandy13:20
rfolco3 depends-on and counting....13:23
rfolcorlandy, :)13:23
rlandyrfolco: 3 depends and we're out13:23
rlandythis needs to work13:24
rlandyfor my sanity13:24
rlandydoing the multinode equivalent now13:24
rlandypanda: let's talk multinode ...13:28
rlandynodeset ...13:28
rlandywould we want two tripleo-ovb-centos-7 nodes?13:28
rlandyonly on one tenant13:29
rlandycomparing with ... https://review.rdoproject.org/r/#/c/16337/8/zuul.d/nodesets.yaml13:29
pandarlandy: in a meeting13:31
quique|rover|afkrfolco: remove timeout logic just mis a +1w https://review.openstack.org/#/c/589068/13:34
*** quique|rover|afk is now known as quiquell|rover13:41
rfolcoquiquell|rover, \o/13:47
quiquell|roverrlandy, panda: Let's workflow the timeout logic removal https://review.openstack.org/#/c/589068/13:50
pandarlandy: currently multinode jobs in rdo are suing both rdo-cloud and rdo-cloud-tripleo tenant13:50
rlandypanda: I am using the upstream-centos-7-2-node nodeset to test13:52
pandarlandy: ok13:53
rlandytwo-centos-7-nodes13:54
rlandyactually that one13:55
quiquell|roverpanda: About tenants, they are going to keep tthem ?13:55
pandaquiquell|rover: as far as I know, they are not going anywhere13:56
quiquell|roverpanda: Did we have some code to adapt to that or is all gone now ?13:57
quiquell|roverpanda: code selecting rdocloud as cloud name in tripleo-ci13:58
quiquell|roverIt's still needed ?13:58
pandarlandy: meeting ended, how's the job going ?13:59
rlandysec - submitting13:59
pandaquiquell|rover: it's based on the nodeset13:59
pandaquiquell|rover: which tenant is decided based on which node label you're using13:59
pandaquiquell|rover: in the timeout, I still can't find for example where are we saving the unbound cache14:00
pandarfolco: pasted me some link but either gertty is broken or it's not there14:00
quiquell|roverpanda: https://review.openstack.org/#/c/589068/27/playbooks/tripleo-ci/templates/oooq_common_functions.sh.j214:03
quiquell|roverwe have move the "pre post" stuff there14:03
pandaquiquell|rover: oooh it' buried into the collect logs script14:04
pandanow I see14:04
quiquell|roverpanda: I suppose that later on this will be splitted on ansible stuff14:05
quiquell|roverpanda: it's more collect_foo.sh14:05
pandaquiquell|rover: yes, absolutely14:05
pandawe have a lot to rearrange14:05
quiquell|roverpanda: how are you doing guys with the reproducer, I have see some reviews about it14:06
pandaquiquell|rover: I have only a WIP crappy patch because I'm tired of seeing people running it manually. Noting serious yet14:06
pandaI don't know if there's a more serious attempt somewhere else14:06
quiquell|roverand about using the new workflow on it ?14:07
rfolcoweshay, rlandy sent an invite for TC role talk, please yell if it does not work14:07
rlandyyou did?14:08
rfolcoI did rlandy14:09
rlandydon't see it14:09
rfolcosorry, english is ambiguous on hidden subject14:09
pandaquiquell|rover: last question, you removed also the hash_info part14:10
rlandypanda: pls check https://review.rdoproject.org/r/#/c/16363/1/zuul.d/zuul-v3-jobs.yaml14:10
pandaquiquell|rover: I think it's used moslty during promotion14:10
rlandyadding a test-project job14:10
quiquell|roverpanda: After talking with Sagi, it's not used anymore14:11
quiquell|roverpanda: JOB_EXIT_VALUE is not used14:11
pandammhh, how do the job report to dlrn api ....14:12
pandarefresh refresh14:12
quiquell|roverpanda: ansible I think, we found it14:12
quiquell|roverpanda: Don't remember where14:13
pandarlandy: looks good, not sure if the irrelevant files part is needed.14:14
quiquell|roverdrop now14:14
*** quiquell|rover is now known as quique|rover|off14:14
rlandypanda: ok - our multinode test job ... https://review.rdoproject.org/r/#/c/16364/14:41
pandaI'll look a it like a dog looks at the washing machine.14:43
pandagazing14:43
rlandystupid me14:43
*** gkadam has joined #oooq14:43
rlandyno job definition14:43
rlandyadding14:43
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch @ https://review.openstack.org/560445, stable/queens:  (1 more message)14:45
rlandyoh it is there14:45
rlandynvm14:45
rlandywoohoo ... We have enabled your tenants on Upshift Cloud14:47
rlandypioneers!!14:47
pandawat ?14:50
pandaI'm always the last to know these stuff14:51
arxcruzpanda: you and me14:55
arxcruzdidn't get any mail about it14:55
arxcruz:D14:55
*** sshnaidm|pto has joined #oooq14:56
pandaarxcruz: I think you should actually be involved and have requested one. rlandy was not sleeping like us and got her email.14:58
pandatoday is the date of long check queues14:59
rlandyarxcruz: panda: internal stuff15:00
rlandyI'll share with you both - no worries15:00
*** sshnaidm|pto has quit IRC15:02
rlandyweshay: meeting?15:02
weshayrlandy, need a sec, coming out of another mtg15:03
arxcruzweshay: 1-1 now ?15:03
rlandyarxcruz: mine as well :)15:04
*** ykarel is now known as ykarel|away15:04
rlandyoh - mine git moved to 11:3015:04
weshayrlandy, I moved ur 1-115:04
weshayarxcruz, joining15:04
rlandysaw that - it's juts called 11:0015:04
pandadamn I forgot we shouldn't +w anything at the moment :(15:06
*** saneax has quit IRC15:07
*** gkadam has quit IRC15:10
*** ykarel|away has quit IRC15:10
rlandyugh - why are we setting up the bridge here??15:24
pandarlandy: it's multinode15:31
pandarlandy: shouldn't we ?15:31
rlandywe setting it up twice15:31
pandammmhh15:31
rlandywe should not set it up on the undercloud15:32
rlandyI thought we passed those vxlan vars15:32
*** ykarel has joined #oooq15:34
pandarlandy: mmhh I think we're not including the common.yaml file15:34
pandathat's part of tripleo-ci repo, and we are not including it in the required projects15:34
pandaso the file is physically not there15:35
pandaeither we include tripleo-ci in the projects15:35
pandaor we copy the common.yaml file in rdo-jobs15:35
rlandyugh - supposed to be in meeting15:35
rlandywill check in a bit15:35
rlandypanda: so we are running the bridge set in the the pre- which is good15:38
*** jfrancoa has quit IRC15:38
* rlandy checks common15:38
rlandyparent: tripleo-ci-base-multinode15:38
pandarlandy: the problem is the file common.yaml, that's where we store the values that deactivate our vxlan bridge15:39
rlandy common_featureset: featureset-multinode-common.yml15:39
rlandy^^ in the parent15:40
weshayrlandy, I'm in15:41
pandarlandy: here's the problem https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run-v3.yaml#L715:41
rlandysec - in meeting now15:42
pandathis file is present only in tripleo-ci repo15:42
pandarlandy: sure, I'll continue to analyze and explain meanwhile15:42
pandarlandy: this file is only here https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/vars/common.yaml15:46
pandarlandy: we don't have it on rdo-jobs repo15:46
pandarlandy: here is where we stop our vxlan https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/vars/common.yaml#L515:47
pandalet me check if we don't really include it15:48
*** marios is now known as marios|home15:49
*** marios|home is now known as marios15:49
pandammmhh15:49
pandaplaybook run says ok when trying to include common vars15:50
pandarlandy: yeah, nevermind, I see the option passed as extra vars, so we're good there15:51
pandaso I don't understand15:51
pandarlandy: where do you see it we are running it twice ?15:52
pandarlandy: not in our role15:52
rlandyin meeting15:52
pandayeah I know15:53
pandajust thinking loud15:53
rlandypanda: ok - looking now16:18
rlandyout of meeting16:18
*** holser_ has quit IRC16:18
rlandypanda: looking at the console output ... ok - no it actually skipped those tasks - we're fine16:21
rlandybreathe16:21
rlandyadding singlenode and standalone16:22
*** agopi is now known as agopi|long_lunch16:22
pandarlandy: :))16:23
*** vkmc is now known as vkmc|afk16:37
*** dtantsur is now known as dtantsur|afk16:41
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch @ https://review.openstack.org/560445, stable/queens:  (1 more message)16:45
*** gkadam has joined #oooq17:03
*** bogdando has quit IRC17:06
rlandypanda: https://review.rdoproject.org/r/#/c/16364/- we got a pass17:10
rlandychecking logs to be sure17:10
rlandyadding singlenode and standalone17:10
pandarlandy: how's ovb ?17:11
rlandy(./toci_quickstart.sh:151): [[ 143 != 0 ]]17:11
rlandy:(17:11
rlandyprobably not our fault17:11
rlandyon overcloud_deploy17:11
pandammmhh17:12
rlandyI'll rekick when results are in17:13
pandaa step back, but we have a build that works end to end, we know the workflow isok17:13
rlandypanda: ask17:13
rlandyack17:13
rlandy+(/opt/stack/new/tripleo-ci/scripts/oooq_common_functions.sh:89): run_with_timeout(): TIME_FOR_COMMAND=14317:14
rlandylegacy also hit the timeout17:14
pandahhm17:15
panda143 should be the remaining time17:15
rlandylegacy actually failed deploy17:16
pandarlandy: fs037 looks ok, I don't see tempest result, do we run it on updates  ?17:17
rlandyhmmm - checking legacy17:17
rlandyhttps://logs.rdoproject.org/50/604150/4/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/6e11796/job-output.txt.gz#_2018-09-21_14_48_54_32623917:18
rlandyskipped17:18
rlandyhttps://logs.rdoproject.org/50/604150/4/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/6e11796/logs/17:19
pandayep probably not even considered for the upgrades at all17:19
rlandypanda: should I remove  irrelevant-files: for fs037?17:20
rlandyI am editing that file to add another parent dor standalone17:20
rlandyfor17:20
rlandyand singlenode17:20
rlandyhttps://review.rdoproject.org/r/#/c/16363/1/zuul.d/zuul-v3-jobs.yaml17:21
pandarlandy: the ancestors are tripleo-ci-base-multinode-rdo tripleo-ci-base-multinode tripleo-ci-base. This last one already has the irrelevant files configuration17:22
rfolcorlandy, weshay can we quick chat now?17:32
rlandywhenever17:34
rfolcoweshay, ^17:34
pandashakira intensifies.17:42
rlandypanda: https://review.rdoproject.org/r/#/c/16363/2/zuul.d/zuul-v3-jobs.yaml17:44
rlandywant to test a branch (not master and standalone)17:45
rlandywe can reverse those if they don;t work17:45
*** trown is now known as trown|lunch17:46
rlandyhttps://review.rdoproject.org/r/#/c/16367/17:48
pandarlandy: \o/17:48
rlandylet's juts see what this does17:48
*** ssbarnea|bkp has joined #oooq17:49
pandarlandy: you're testing standalone in a different patch ?17:49
rlandyyes17:49
pandaok17:49
rfolcoweshay, rlandy: good time now?17:50
rfolcochance = last17:50
weshaylolz17:50
rfolcotimeout = 317:50
rfolco:)17:50
rfolcorlandy, joined https://bluejeans.com/u/whayutin/17:51
*** panda has quit IRC18:08
*** ykarel is now known as ykarel|away18:09
*** ssbarnea|bkp has quit IRC18:10
*** panda has joined #oooq18:10
rlandytripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master stuck collecting logs :(18:13
rlandyI could just recheck that18:13
rlandyweshay: we have passing multinode - singlenode is being checked now18:14
pandahow long stuck ?18:15
rlandygone now18:17
rlandyte-broker timed out18:17
rlandyrechecked18:17
ykarel|awaypromotion jobs failing: stack_status_reason   | Resource CREATE failed: NeutronClientException: resourc                                                                                       |18:17
ykarel|away2018-09-21 15:43:42.340264 | tripleo-ovb-centos-7 | |                       | es.baremetal_env.resources.openstack_baremetal_servers.                                                                                       |18:17
ykarel|away2018-09-21 15:43:42.340485 | tripleo-ovb-centos-7 | |                       | resources[3].resources.baremetal_ports.resources.public                                                                                       |18:17
ykarel|away2018-09-21 15:43:42.340704 | tripleo-ovb-centos-7 | |                       | _port: <html><body><h1>504 Gateway Time-out</h1> The                                                                                          |18:17
ykarel|away2018-09-21 15:43:42.340914 | tripleo-ovb-centos-7 | |                       | server didn't respond in time. </body></html>18:17
rlandymaybe we have a problem on the tenant18:18
ykarel|awayseems there is issue in RDO cloud, is this being alrready addressed18:18
rlandyno report so far that I can see18:18
rlandyie: nothing on #rhos-ops18:18
rlandymy job also just timed out18:19
panda:(18:19
ykarel|awaypanda, looks like same metadata thing hitting now18:19
ykarel|awaycurl http://169.254.169.254/openstack/2015-10-15/meta_data.json18:19
ykarel|away2018-09-21 16:40:02.349232 | tripleo-ovb-centos-7 | ++(/opt/stack/new/tripleo-ci/toci_gate_test.sh:128): python -c 'import json, sys; print json.load(sys.stdin)["uuid"]'18:19
ykarel|awaythis failed18:19
* rlandy checks tenant18:19
rlandypanda: hate to do this to you but I am out monday and tuesday18:19
rlandypanda: spoke with weshay about next steps ...18:20
rlandyI am adding a job to testproject with some check jobs and a few periodics18:20
rlandyif those pass, we can merge those and add a trigger for a few check jobs non-voting18:21
rlandyafter that add a few jobs to the periodic pipeline - also not in criteria18:21
rlandyand see if that works18:21
pandayou can't do this to me !18:22
rlandypanda: I'll be back on wednesday .. two days ... you'll be fine18:23
* rlandy relies on panda more than panda relies on rlandy18:23
pandaas long as my nemesis the introspection doesn't show up18:24
* rlandy respectfully asks introspection to behave while I am out18:25
rlandyI'll update the card18:25
pandaok18:26
weshaypanda, rlandy before you guys go18:28
weshaywe have no rocky 3rd party jobs https://review.rdoproject.org/r/#/c/16305/18:28
rlandyweshay: where is the rocky-branch playbook?18:31
weshaywe need a rocky branch playbook??18:31
weshaywha?18:31
rlandy run: playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/run.yaml18:31
ykarel|awayweshay, rlandy https://review.rdoproject.org/r/#/c/16146/18:32
ykarel|awaysagi already had a patch, not sure why that's not picked up18:32
weshayykarel|away, ya.. thanks for pointing that out.. I just missed it18:32
* weshay looks18:32
*** jtomasek has quit IRC18:32
rlandyykarel|away: rocky vs rocky-branch?18:33
weshayykarel|away, only issue is https://review.rdoproject.org/r/#/c/16146/3/playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-rocky/run.yaml ?18:33
weshayre: default18:33
weshayvs the branch18:33
ykarel|awayweshay, yes afaik18:34
ykarel|awayrlandy, in branchless project like tq/tqe etc, those -branch jobs run18:34
weshayykarel|away, rlandy k.. updated18:35
rlandyweshay: https://review.rdoproject.org/r/#/c/16305/ is no longer?18:36
weshayrlandy, same thing afaict18:36
ykarel|awayweshay, apart from this looks like some infra issue is lying up, i pinged on #rhos-ops seems no one available currently, if u can check those18:36
rlandywe're looking at https://review.rdoproject.org/r/#/c/16146/ now?18:36
weshaywhich ever works18:36
rlandyykarel|away: also watching those rdocloud issues18:36
weshayI just want rocky 3rd party18:36
rlandygot reparenting jobs that need to go through18:37
rlandypick one and I'll review it18:37
rlandyyou want 21 in?18:40
rlandyrdocloud should be ok again - recheck done18:43
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens- (2 more messages)18:46
rlandytenant looks ok18:53
*** ykarel|away has quit IRC18:54
rlandyagopi|long_lunch: I'm rechecking https://review.openstack.org/#/c/604355/ so see legacy run19:07
rlandysinglenode is running :)19:08
*** agopi|long_lunch is now known as agopi19:17
agopiokay rlandy i hit recheck because overcloud deploy failed19:18
rlandyweshay: panda: question about organization of rdo-jobs zuul.yaml ... working on a commit to add some check and periodic job definitions  to run with testproject ... ok if we use the same naming as in tripleo-ci?19:24
rlandymultinode-jobs.yaml19:25
rlandysinglenode-jobs.yaml19:25
rlandybase.yaml19:25
rlandyand19:25
rlandyovb.yaml19:25
weshayrlandy, ya19:25
rlandyand get rid of the zuulv3 part19:25
weshay+119:25
rlandyok19:25
rlandyso we won't merge what we have - leave those reviews just for a testing19:28
rlandyputting in new reviews for the organization19:28
pandarandy: agreed on consistency19:31
*** gkadam has quit IRC19:35
*** rlandy is now known as rlandy|brb19:47
*** rlandy|brb is now known as rlandy20:16
ssbarneaanyone still around here? i wonder if this would be a good pick for a homebrew openstack testing bed: https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.ebay.co.uk%2Fulk%2Fitm%2F113253504265 - mem could be upgraded later, i am more concerned about cpu.20:27
agopissbarnea, if you want to know if a particular machine would be good enough i suggest you to talk to scalelab folks20:30
agopithey're more used to these machines20:30
agopiwe got super micros and dells in the lab20:30
rlandypanda: weshay: we're green on singlenode https://review.rdoproject.org/r/#/c/16367/20:31
agopihttp://scalelab.redhat.com/20:31
rlandyhttps://logs.rdoproject.org/67/16367/2/check/periodic-tripleo-ci-centos-7-singlenode-featureset027-master/2d6df50/logs/tempest.html.gz20:31
rlandytempest ran20:31
weshayship it20:31
weshay:)20:32
weshaythanks rlandy that is great news20:32
rlandyalmost - one more ovb browbeat job to clear20:32
rlandybit we are ready to start testing check jobs20:32
ssbarneaagopi: which irc channel to to use?20:41
agopiinternal scalelabrdu20:42
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci- (2 more messages)20:46
agopissbarnea, ^20:46
pandarlandy:  \o/20:47
* rlandy checks the output20:48
rlandytries not to just trust the green20:48
weshayrlandy, does the libvirt repro run w/ the jinja template version of toci_quickstart?20:56
weshaynot seeing https://review.openstack.org/#/c/595374/17/playbooks/tripleo-ci/templates/toci_quickstart.sh.j220:56
weshayansible_distribution get picked up here20:56
rlandyum - so ...20:58
rlandyreading reproducer ...20:58
rlandy "screen -LS ci -h 25000 -dm bash -c 'set -m; source ~/env_vars_to_src.sh; ANSIBLE_FORCE_COLOR=true /opt/stack/tripleo-ci/toci_gate_test.sh 2>&1 | tee ~/console.log; exec bash' && screen -RD"20:59
rlandy^^ uses old toci_gate_test.sh20:59
rlandyweshay: ^^20:59
weshayrlandy, ah THANK YOU!!21:00
rlandy604150,4 - still waiting fir this to merge21:05
rlandyweshay: panda: https://review.rdoproject.org/r/#/c/16370/ - new layout21:41
rlandyfor your review21:41
rlandy^^ going to use that to add periodic and check jobs21:42
rlandyagopi: rfolco: no dice ...21:43
rlandyfatal: [undercloud]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'zuul' is undefined\n\nThe error appears to have been in '/home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/browbeat/pre-install-setup/tasks/main.yml': line 17, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Copy browbeat21:44
rlandyto the undercloud - zuul user\n  ^ here\n"}21:44
rlandy2018-09-21 21:39:19.025677 | primary |21:44
pandaI don't see a layout in your layout :)21:44
agopirlandy, so i posted on review21:44
agopiwill push a new patchset which just starts with projects, maybe it'll work?21:44
rlandyok21:44
rlandypanda: you are correct - didn;t add a layout yet :(21:45
rlandypanda: review updated - added a blank layout.yaml file21:46
*** ssbarnea|bkp2 has joined #oooq21:47
rlandypanda: I'll be doing some test work on sunday21:48
rlandyI'll update the card with the latest then21:48
rlandyweshay: ^^21:48
pandaok21:48
rlandybrowbeat still needs  modification21:49
rlandythe rest of the ovb job looks fine21:49
agopibrowbeat repo you mean rlandy ?21:49
* weshay reads21:49
agopilmk what needs to be done and i'll do it21:49
weshayagopi, check out that error21:50
weshayWOOOT21:50
weshaylibvirt repro running on Fedora2821:50
agopithat looks more zuul related weshay, i posted on review already will try using projects.{..} instead21:51
rlandyagopi: yep - the error we just discussed21:51
rlandyweshay: nice!!21:51
agopiack rlandy21:51
rlandyagopi: if you put in a change, I'll check it on sunday21:51
*** ssbarnea|bkp2 has quit IRC21:52
pandaso the plan is to add job to the layout one at a time and remove the corresponding  legacy when it passes ?21:52
rlandypanda: yep - test out on testproject21:53
rlandyand then add to layout21:53
rlandywe can add and merge the definitions21:54
rlandyjust the triggering21:54
rlandyas soon as testproject tests pass, we can merge definitions21:54
rlandygood new is that rdo-jobs review merge easily21:54
* rlandy needs to log off ...21:55
pandaok thanks21:55
rlandyI'll leave you notes21:55
pandaleave us alone in all this despair21:55
pandaand have a nice weekend21:56
agopiack rlandy pushed review21:56
rlandythanks21:56
panda:)21:56
*** rlandy has quit IRC21:56
agopi| RUN END RESULT_ABORTED: [untrusted : review.rdoproject.org/rdo-jobs/playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/run.yaml@master]21:57
agopiat the end of the log21:57
agopiis it normal rlandy weshay rfolco ?21:57
pandaagopi when you put a new patchset22:00
pandathe running build aborts22:01
agopiooh okay gotcha panda. thanks :)22:02
*** zul has quit IRC22:04
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode- (2 more messages)22:46
*** tosky has quit IRC22:58
*** tosky has joined #oooq22:59
*** jrist has quit IRC23:12
*** tosky has quit IRC23:13
*** dsneddon has quit IRC23:58
*** dsneddon has joined #oooq23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!