Thursday, 2018-10-04

*** agopi__ is now known as agopi00:12
*** dtrainor has joined #oooq00:22
*** rlandy|afk has quit IRC00:35
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq,  (2 more messages)00:51
*** jtomasek has joined #oooq01:03
*** jtomasek has quit IRC01:07
*** dtrainor has quit IRC01:18
hubbot1FAILING CHECK JOBS on master: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages)02:51
*** udesale has joined #oooq03:14
*** ykarel|away has joined #oooq04:02
*** saneax has joined #oooq04:35
hubbot1FAILING CHECK JOBS on master: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (2 more messages)04:51
*** udesale has quit IRC05:09
*** udesale has joined #oooq05:15
*** ykarel|away is now known as ykarel05:23
*** jfrancoa has joined #oooq05:49
quiquell|offssbarnea|bkp2: Don't know man, marios|rover can you look at the bug ? https://bugs.launchpad.net/tripleo/+bug/1786106 ?05:49
openstackLaunchpad bug 1786106 in tripleo "containers-prep fails w/ update_containers is undefined due to lack of default" [High,Fix released] - Assigned to Quique Llorente (quiquell)05:49
*** quiquell|off is now known as quiquell05:49
marioso/ quiquell05:52
*** udesale has quit IRC05:53
*** udesale has joined #oooq05:58
*** jtomasek has joined #oooq06:05
*** ratailor has joined #oooq06:06
*** holser_ has joined #oooq06:21
hubbot1FAILING CHECK JOBS on master: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (2 more messages)06:51
*** bogdando has joined #oooq07:11
*** skramaja has joined #oooq07:23
*** ykarel_ has joined #oooq07:24
*** ykarel has quit IRC07:26
*** ykarel__ has joined #oooq07:28
*** ykarel_ has quit IRC07:30
*** udesale has quit IRC07:30
*** ykarel_ has joined #oooq07:41
*** ykarel__ has quit IRC07:44
*** tosky has joined #oooq07:45
*** ykarel__ has joined #oooq07:46
*** ykarel__ is now known as ykarel07:46
*** ykarel has quit IRC07:47
*** ykarel_ has quit IRC07:48
bogdandohi folks. I can't recall some recent patch which allows config files opened in the browser, instead of downloading mode07:50
bogdandoany help with that? wanted to add logrotated files there as well07:50
bogdandoso folks could surf the job logs better :)07:51
*** udesale has joined #oooq07:53
*** ykarel has joined #oooq07:54
*** amoralej|off is now known as amoralej07:55
*** chem has joined #oooq08:02
*** ykarel_ has joined #oooq08:12
*** ykarel has quit IRC08:15
*** ykarel__ has joined #oooq08:20
*** ykarel_ has quit IRC08:23
*** ykarel_ has joined #oooq08:23
jfrancoagood morning! can anybody solve me a doubt about the deployed_server variable in the featuresets?08:25
*** ykarel__ has quit IRC08:26
jfrancoawhy do we have it set to true in some featuresets https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset040.yml#L24 and conditionaly to true or false depending on the release in other featuresets https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset018.yml#L87 ?08:28
*** gkadam has joined #oooq08:30
*** gkadam has quit IRC08:31
kopecmartin|ruckmarios|rover, have you seen something similar like described in these bugs? https://bugs.launchpad.net/tripleo/+bug/1796040 https://bugs.launchpad.net/tripleo/+bug/179366508:34
openstackLaunchpad bug 1796040 in tripleo "Define stestr facts conditaion check fails in validate-tempest role" [Undecided,New]08:34
openstackLaunchpad bug 1793665 in tripleo "Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]08:34
kopecmartin|ruckI'm wondering if i can put promotion-blocker there or not08:34
marios|roverkopecmartin|ruck: heh i was just looking at this http://logs.openstack.org/98/604298/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/037bd06/job-output.txt.gz#_2018-10-04_04_39_49_23461208:35
marios|roveri think is the same?08:35
marios|rover2018-10-04 04:39:49.234612 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "tail -10 tempest.log; exit 1", "delta": "0:00:00.012430", "end":08:35
marios|roverrimary | TASK [validate-tempest : Exit with tempest result code if config08:35
kopecmartin|ruckyes, i tihnk it's the same as in the two bugs I shared .. when you scroll up, you can see, that tempest execution failed08:36
kopecmartin|ruckalso stestr init failed, which made generation of HTML report failed too08:37
kopecmartin|rucki think there are two bugs, in the job .. one is the failure of tempest execution and the other failure of stestr init08:37
marios|roverkopecmartin|ruck: added comment to 179604008:37
marios|roverkopecmartin|ruck: oh perhaps i chose the wrong one i thought they were duplicates /me reads more carefully08:38
marios|roverkopecmartin|ruck: no i think that testr you point to is invalid since it has ignore on it?08:38
marios|roverkopecmartin|ruck: here https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/6760f43/job-output.txt.gz#_2018-10-01_21_03_01_76218408:39
marios|roverkopecmartin|ruck: yes it fails but then you see it has 2018-10-01 21:03:01.814479 | tripleo-ovb-centos-7 | ...ignoring08:39
marios|roverkopecmartin|ruck: and if you scroll down https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/6760f43/job-output.txt.gz#_2018-10-01_21_03_22_32649608:39
marios|roveryou see the same error which is the actual issue08:40
marios|roveradding comment08:40
kopecmartin|ruckoh, yeah, that one is probably invalid as you said, ok, i think we can mark it as invalid08:40
kopecmartin|ruckthen the comment you added to  1796040 should go to 1793665 :)08:41
marios|roverkopecmartin|ruck: done08:42
marios|roverkopecmartin|ruck: so that took long for the gates to break08:42
marios|roverlike 12 hours !08:42
ssbarnea|bkp2is anyone aware of a specific pip bug where it downgrades editable packages? i found some weird behavior yesterday where without clear reason pip decided to downgrade tqe from ~dev1000 to something like ~dev850 which obviously broke that build.08:43
marios|roverarxcruz: any ideas on this one please ? because 'tempest' https://bugs.launchpad.net/tripleo/+bug/179366508:43
openstackLaunchpad bug 1793665 in tripleo "Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]08:43
*** panda|off is now known as panda08:44
*** ykarel__ has joined #oooq08:45
arxcruzmarios|rover: checking08:45
arxcruzmarios|rover: oh, i've seen this error, and i'm already debugging it :)08:46
marios|roverarxcruz: excellent :)08:46
arxcruzi saw it on my patch to fix stackviz08:46
*** udesale has quit IRC08:46
*** ykarel_ has quit IRC08:48
bogdandoPTAL https://review.openstack.org/#/c/593103/08:49
*** ykarel_ has joined #oooq08:49
*** udesale has joined #oooq08:49
bogdandoI think all of the concerns had been settled and it's good to go08:49
bogdandoand then there is https://review.openstack.org/#/c/576746/ that really needs that one08:50
*** ykarel__ has quit IRC08:51
hubbot1FAILING CHECK JOBS on master: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (2 more messages)08:51
arxcruzmarios|rover: https://review.openstack.org/605419 should fix it08:52
pandaquiquell: how's going ?08:53
quiquellpanda: Testing build-containers here https://review.rdoproject.org/r/#/c/16768/08:55
quiquellpanda: fs001 here https://review.rdoproject.org/r/#/c/16390/ gives error08:55
quiquellpanda: ^ cannot find the issue08:55
pandaquiquell: it has been pretty random08:58
quiquellpanda: the "ERROR" ?08:58
quiquellpanda: About build-containers, Don't know the implications of running at post, also I don't know if we can override required-projects08:59
quiquellpanda: overriding it will override the parent ?08:59
quiquellpanda: humm found the issue09:00
quiquellERROR Unable to find playbook /tmp/tmp0xy6_j82/279a04d46dd147ba97c407d30dcdbbd0/trusted/project_0/review.rdoproject.org/config/playbooks/tripleo-rdo-base/pre.yaml09:01
quiquelldamn.. it's yml not yaml09:03
quiquell:-/09:03
quiquellpanda: https://review.rdoproject.org/r/#/c/16771/09:04
quiquellsshnaidm: Do you have super powers for https://review.rdoproject.org/r/#/c/16771/ ?09:05
quiquellpanda: ^ ?09:05
sshnaidmquiquell, yeah09:05
pandaquiquell: I think the standard there is to use yaml09:05
quiquellpanda: Ok will move the file, sshnaidm wait09:06
*** ykarel_ has quit IRC09:07
*** ykarel_ has joined #oooq09:07
quiquellsshnaidm, panda: Done, please merge https://review.rdoproject.org/r/#/c/16771/09:07
quiquellSo we can continue09:07
*** ykarel__ has joined #oooq09:10
quiquellsshnaidm, panda: Please merge this https://review.rdoproject.org/r/#/c/16771/ we cannot continue without it09:12
*** ykarel_ has quit IRC09:13
pandaquiquell: commented 1676809:13
pandaquiquell: I don't have +2 there09:13
quiquellsshnaidm: pleaseee09:14
sshnaidmquiquell, I see it's merging09:15
quiquellsshnaidm: thanks sir09:15
quiquellsshnaidm: We will need your super powers until rlandy is around, so be prepare09:15
sshnaidmalways ready!09:16
quiquellpanda: fixed09:17
*** ykarel has joined #oooq09:18
*** ykarel__ has quit IRC09:19
sshnaidmquiquell, how are you gonna hide secrets in rdo jobs now?09:22
quiquellsshnaidm: It's a prototype to run without merge, it will fail at secrets, but we will check everything else09:23
* sshnaidm remembers a lot of arguing with pabelanger about it09:23
sshnaidmquiquell, but in general?09:23
quiquellsshnaidm: base jobs at 'config' project it's a trusted one09:23
sshnaidmquiquell, right09:23
sshnaidmquiquell, so all periodic jobs will be there?09:24
quiquellsshnaidm: yep09:24
sshnaidmcool09:24
quiquellsshnaidm: Do we have alternatives with vault or the like to store secrets at untrusted ?09:24
sshnaidmquiquell, not much alternatives.. with ssh I did something like that: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/legacy-tripleo-ci-periodic-base-upload/post.yaml#L6-L2109:26
sshnaidmquiquell, but not sure if it's possible to do with other passwords09:27
sshnaidmquiquell, although the problem will be that we will not be able run periodic jobs in patch09:28
sshnaidmquiquell, and couldn't test changes with periodic jobs..09:28
sshnaidmit was pretty nice feature to use09:29
quiquellsshnaidm: where is this uploader_ssh_key.ssh_private_key  ?09:29
sshnaidmquiquell, in config repo09:29
quiquellsshnaidm: Ahh so you still depends con cofig repo but you can define jobs at rdo-jobs, that's it ?09:30
sshnaidmhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml09:30
sshnaidmquiquell, yep09:30
quiquellsshnaidm: how zuul knows how to decrypt the priv key ?09:31
quiquellsshnaidm: where is the password for decypt ?09:31
sshnaidmquiquell, I encrypt it with zuuls key09:31
quiquellsshnaidm: Ahh that's associated with trusted projects, isn't it ?09:31
sshnaidmquiquell, and only zuul can decrypt it with its private key in job09:32
*** ykarel has quit IRC09:32
quiquellsshnaidm: so zuul internally decript it for you before use it09:32
*** ykarel has joined #oooq09:32
quiquellsshnaidm: so we can do the same for normal DLRN passwords and the like09:32
sshnaidmquiquell, if we use passwords in post/pre that are located in trusted, we are good09:32
sshnaidmquiquell, but you shouldn't be able to reveal them during the job09:32
quiquellsshnaidm: then why didn't we use this instead of move jobs to config ?09:32
quiquellsshnaidm: Yep you have to be careful09:33
sshnaidmquiquell, because it's not easy to hid password in job, that's why I used ssh-agent09:33
quiquellsshnaidm: I see09:33
sshnaidmquiquell, if it's just variable, you can create a job in rdo-jobs and post there "echo $MY_SECRET"09:33
sshnaidmquiquell, tbh not sure if we can hide it good enough.. if dlrn_api could provide us a time-based token, it would be easier..09:35
*** ykarel_ has joined #oooq09:35
*** ykarel has quit IRC09:38
pandaquiquell: : "The task includes an option with an undefined variable. The error was: 'toci_jobtype' is undefined"09:42
*** ykarel_ is now known as ykarel09:43
pandaquiquell: why are you still running the legacy playbook ?09:44
quiquellpanda: where ?09:49
pandaquiquell: dnm-test-job09:50
pandaquiquell: in your testing tripleo-ci-base-singlenode-containers-build09:50
pandain the container build, we install the undercloud in the host for a series of hitorical reason, the same job was originally scheduled to build the images too, but the idea was changed. The part that is important is the repo setup09:51
pandacloud images09:52
quiquellpanda: give me a sec, I am preparing the proto for promote09:52
quiquellpanda: so we don't need to run it ?09:53
quiquellpanda: Didn't know why we were running it09:54
quiquellpanda: Just try to mimic the present stuff09:54
quiquellpanda: only repo setup is needed ?09:54
pandaquiquell: yes, but don't change the process now, just run the undercloud install09:55
quiquellpanda: ack, we can narrow down it later on09:55
*** dtantsur|afk is now known as dtantsur09:55
quiquellpanda: btw has try to do an "include: ../zuul-v3.yaml" so we run container build at run not at post, but it's not working09:56
quiquellpanda: promote proto https://review.rdoproject.org/r/#/c/16773/09:57
quiquellpanda: so we have to run build-test-package forcing it to build ?09:59
*** udesale has quit IRC10:04
pandaquiquell: the periodic base alrady runs the undercloud install10:06
pandaquiquell: so you don't have to specify a run10:06
pandaquiquell: then we can do the rest in post10:06
pandathe problem may be with timeout10:06
quiquellpanda: So it's not a problem to build the images at post ?10:07
pandawe take a lot to build and upload the images10:07
quiquellpanda: It times out we will stil try it10:07
quiquellpanda: we can configure post timeouts10:07
pandaquiquell: yep, and we better.10:07
pandaquiquell: also, if install fails at run, we're still going to try to build images10:08
pandawhih may be not ideal10:08
quiquellpanda: we can check that with zuul_success variable10:08
quiquellpanda: as we do with DLRN reporting at post10:08
quiquellpanda: goint to do the timeouts and zuul_success10:09
quiquellpanda: Do you know if the variables we have at common.yaml can be used at post ?10:09
pandaquiquell: wait10:12
pandawhy are we using post  ?10:12
quiquellpanda: because we cannot have more than one playbook at run10:12
quiquellpanda: if you set run: it will overwrite parent10:13
pandaquiquell: yes we can10:13
quiquellpanda: include ?10:13
pandaquiquell: well eyah, not directly with zuul10:13
pandaquiquell: yeah ?10:13
*** udesale has joined #oooq10:13
pandais it not working ?10:13
pandaoh yu tird10:13
quiquellpanda: I have trie to move post.yaml to run.yaml and include: ... run-v3.yaml10:14
pandado you have the error ?10:14
quiquellyep, maybe the path is wrong10:14
quiquellpanda:  is at one of the ps10:14
pandaalso, use include_playbook10:14
pandanah, doesn't exist10:14
quiquellpanda: https://logs.rdoproject.org/68/16768/16/check/periodic-tripleo-centos-7-master-containers-build/66f644b/job-output.txt.gz10:14
quiquellpanda: humm wait...10:15
quiquellpanda: I know how10:15
quiquellpanda: zuul.projects['...tripleo-ci']/playbooks/tripleo-ci/run-v3.yaml10:15
quiquellpanda: {{ zuul.projects['...tripleo-ci'].src_dir }}/playbooks/tripleo-ci/run-v3.yaml10:15
quiquellpanda: How important is to run build-containers at run and not at post ?10:16
pandaquiquell: too many unknowns10:17
quiquellpanda: ack, let's do it at run10:17
pandayou change the process, you ask for trouble10:17
quiquellpanda: ok, changed, let's see if it works with absolute path10:21
pandaquiquell: absolutely10:23
quiquellpanda: check job failing https://review.rdoproject.org/r/#/c/16390/10:25
quiquellpanda: te-broker client failing10:27
quiquellpanda: failing here https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/templates/toci_gate_test.sh.j2#L16610:30
pandaI see it10:31
pandamarios|rover: any problem with ovb jobs recently ?10:32
marios|roverpanda: yeah seeing lots of overcloud-deploy poking at the moment10:32
pandaquiquell: I think it may be just high load , te-broker is working correclty10:33
quiquellpanda: ack10:34
quiquellpanda: Can i change config/playbooks/legacy-tripleo-ci-periodic-base/post.yaml  ?10:34
quiquellpanda: Or do I copy paste and change ?10:34
pandaquiquell: copy on write10:34
pandaquiquell: we may need to revert10:34
marios|roverpanda: this what you mean http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/9cda9bb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz10:34
marios|roverstatus ERROR due to "Message: No valid host was found. , Code: 500"10:35
pandait may be easier if we don't change everything10:35
quiquellpanda: ok will create playbooks/tripleo-ci-periodic-base for the new stuff10:35
*** skramaja_ has joined #oooq10:35
*** skramaja has quit IRC10:36
*** kopecmartin|ruck is now known as kopecmartin|scho10:38
*** udesale has quit IRC10:39
*** udesale has joined #oooq10:40
quiquellpanda, sshnaidm: https://review.rdoproject.org/r/1677710:41
sshnaidmquiquell, commented10:44
pandaquiquell: some fixes needed10:49
quiquellpanda: yep, on the way for container-builds10:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009 (2 more messages)10:51
quiquellsshnaidm, panda: https://review.rdoproject.org/r/16777 changed10:56
marios|roverkopecmartin|scho: OVB jobs are failing for "No valid host was found. , Code: 500" during the overcloud deployment Edit10:57
marios|roverkopecmartin|scho: https://bugs.launchpad.net/tripleo/+bug/179606910:57
openstackLaunchpad bug 1796069 in tripleo "OVB jobs are failing for "No valid host was found. , Code: 500" during the overcloud deployment" [High,Triaged] - Assigned to Marios Andreou (marios-b)10:57
marios|roverpanda: ^^^ is that what you meant or are you seeing something different?10:57
pandamarios|rover: no, testenv client is failing on us, probably heavy load, don't worry for this, and thanks.10:59
marios|roverpanda: k11:01
pandaquiquell: still indentation11:01
pandaquiquell:  on the first line11:01
pandaquiquell: you didn't fix that11:01
quiquellpanda: it's not needed11:01
quiquellpanda: identation is ok11:02
ssbarnea|bkp2panda: what is the meaning of "execute quickstart.sh --clean to ensure the dependencies are installed "? I found this failure at https://ci.centos.org/job/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/834/console and I find the msg bit confusing.11:03
pandaquiquell: o_O11:03
quiquellpanda:  ERROR! 'import_paybook' is not a valid attribute for a Play11:03
*** amoralej is now known as amoralej|lunch11:03
quiquellhttps://logs.rdoproject.org/68/16768/19/check/periodic-tripleo-centos-7-master-containers-build/c4d3eda/job-output.txt.gz#_2018-10-04_10_55_17_38947811:03
pandaquiquell: yes, import_playbook doesnt' exist11:04
quiquellpanda: https://docs.ansible.com/ansible/2.5/modules/import_playbook_module.html11:04
quiquellpanda: it exists11:04
Tenguquiquell: well, without the typo, it does exist, yep11:04
Tenguquiquell: an "l" is missing.11:04
quiquellTengu: typo ?11:04
pandaquiquell: oh, I see the indentation ... ok11:04
quiquellargggg !!!11:04
Tengu:)11:04
pandaihihih11:05
quiquellTengu: book to pay :-)11:05
Tengucall me eagle eye ;)11:05
quiquellTengu++11:05
hubbot1quiquell: Tengu's karma is now 211:05
quiquellTengu++11:05
hubbot1quiquell: Tengu's karma is now 311:05
quiquellTengu+++11:05
quiquellArggg triple karma11:06
Tengu:D11:06
ssbarnea|bkp2http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/quickstart.sh#n538 - not clear to me why it exits with this message if the file does not exist.11:07
kopecmartin|schomarios|rover, checking, thanks11:12
quiquellsshnaidm: what do you mean here ? https://review.rdoproject.org/r/#/c/16777/1/playbooks/tripleo-ci-periodic-base/post.yaml@811:17
sshnaidmquiquell, is $workspace defined anywhere?11:18
quiquellsshnaidm: fixed11:22
quiquellpanda: Why it didn't do anything at run here ? https://logs.rdoproject.org/68/16768/22/check/periodic-tripleo-centos-7-master-containers-build/28ae198/job-output.txt.gz11:24
quiquellpanda: It's not running the imported plabook or the next tasks ?11:26
pandawhere is that playbook defined ?11:26
quiquellpanda: tripleo-ci/playbooks/tripleo-ci/run-v3.yaml11:27
pandareview.rdoproject.org/rdo-jobs/playbooks/legacy/tripleo-ci-centos-7-containers-build/run.yam411:29
quiquellpanda: this is where the import_playbook is11:29
quiquellsshnaidm: Let's merge this https://review.rdoproject.org/r/#/c/16777/11:30
quiquellsshnaidm: Only affects the new jobs it's a separte post.yaml11:30
sshnaidmquiquell, done11:31
quiquellsshnaidm: I don't know if we are going to have 'featureset' and the other there we will see11:31
sshnaidmquiquell, do you have 10 min for ovb sync?11:44
quiquellsshnaidm: sure11:44
sshnaidmquiquell, https://bluejeans.com/u/sshnaidm/11:44
quiquellsshnaidm: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ovb-jobs.yaml11:50
quiquellsshnaidm: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ovb-jobs.yaml#L1611:52
arxcruzpanda: hi, how can i block a task with a dependence of another card ?11:56
pandaarxcruz: there is not effective mechanism, you mark the card blocked and add the url of that task, but nothing prevents updates on the card11:57
arxcruzpanda: ack11:58
weshaymarios|rover, kopecmartin|scho good morning...12:02
weshayI see the gate is once again foooked12:02
marios|roverweshay: yup12:03
weshayI saw a small trend last where qemu jobs timeout and kvm worked fine12:03
weshaymarios|rover, http://logs.openstack.org/41/601241/16/gate/tripleo-ci-centos-7-containers-multinode/5a05846/job-output.txt.gz#_2018-10-03_21_37_01_70964612:04
weshayhttp://logs.openstack.org/90/605090/10/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6edb958/job-output.txt.gz#_2018-10-03_18_20_58_50087212:04
marios|roverweshay: https://storyboard.openstack.org/#!/story/200396612:07
sshnaidmquiquell, https://github.com/rdo-infra/rdo-jobs/commit/ff0b7d5fb98f85c6585e101765531ef0ea43f67e12:08
marios|roverweshay: this is killing us https://bugs.launchpad.net/tripleo/+bug/1793665 we should just disable tempest12:09
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]12:09
marios|rover:)12:09
*** trown|outtypewww is now known as trown12:13
*** rfolco has joined #oooq12:14
weshayugh.. this one 2018-10-04 06:45:04 | {0} tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern [24.936596s] ... FAILED12:14
*** rlandy has joined #oooq12:18
quiquellrlandy: o/12:20
rlandyquiquell: hi - how goes it?12:20
quiquellrlandy: we are doing some prototypes for build-containres and promote-current-to-tripleo-ci-testing12:21
rlandyquiquell: k - good  - we need those12:21
quiquellrlandy: promote https://review.rdoproject.org/r/#/c/16773/12:21
* rlandy is looking through email for reviews12:21
quiquellrlandy: build-containers https://review.rdoproject.org/r/#/c/16768/12:22
quiquellrlandy: forget about 'config' repo one, let's make the repo work first, then we move them to 'config'12:22
quiquellrlandy: I think the promote prototype near to work now12:23
*** ssbarnea_ has joined #oooq12:23
rlandyquiquell: nice - looking12:23
*** ssbarnea|bkp2 has quit IRC12:25
*** amoralej|lunch is now known as amoralej12:26
quiquellrlandy: We are also testing the check job here https://review.rdoproject.org/r/#/c/16390/12:27
quiquellrlandy: without the Depends-On to see if c'config' stuff is working12:27
rlandyquiquell: cool - same failure on both jobs?12:28
rlandyquiquell: before you log off today, let's juts touch base on where I should pick up12:29
quiquellrlandy: don't know, taking care of the promotion protos12:29
rlandyyeah12:29
quiquellrlandy: sure, btw, I have to bypass the DLRN reporting post for promote job but it's not working12:29
rlandyquiquell: panda around this morning?12:30
quiquellrlandy: yep12:30
quiquellOk going for lunch now12:32
rlandyquiquell: moving https://tree.taiga.io/project/tripleo-ci-board/task/108?kanban-status=1447274 to progress12:32
rlandysince we are working  on it12:32
quiquellrlandy: ack, put the test review there12:33
*** quiquell is now known as quiquell|lunch12:33
*** tosky__ has joined #oooq12:37
*** tosky is now known as Guest839912:37
*** tosky__ is now known as tosky12:37
*** Guest8399 has quit IRC12:39
rlandyquiquell|lunch: hmmm  - we have a diff failure in the translated job :(12:39
quiquell|lunchI am around12:40
quiquell|lunchrlandy: do you have pointer ?12:40
rlandyyep - have lunch - will investigate12:40
quiquell|lunchrlandy: ack12:40
*** agopi has quit IRC12:41
*** agopi has joined #oooq12:41
rlandy2018-10-04 10:00:22.235040 | primary | +(/home/zuul/src/git.openstack.org/openstack/tripleo-ci/toci_gate_test.sh:159): sudo kill -9 472512:43
*** agopi has quit IRC12:45
*** agopi has joined #oooq12:45
*** kopecmartin|scho is now known as kopecmartin|ruck12:46
*** agopi has quit IRC12:47
*** agopi has joined #oooq12:47
*** agopi has quit IRC12:48
*** agopi has joined #oooq12:48
rlandyquiquell|lunch: think that was just a hitch with te-broker  - rerun is going better so far12:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario004-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode- (4 more messages)12:51
weshayrfolco, please lead the scrum w/o me12:57
*** quiquell|lunch is now known as quiquell13:05
quiquellrlandy: ok, let's see13:05
rlandyquiquell; I am adding a review for the actual trigger13:05
ssbarnea_kopecmartin|ruck : i will need some help with https://bugs.launchpad.net/tripleo/+bug/1796048 as it seems to affect some builds.13:07
openstackLaunchpad bug 1796048 in tripleo "quiskstart.sh installs downgraded version of tripleo-quickstart-extras" [High,Triaged]13:07
kopecmartin|ruckssbarnea_, checking13:08
kopecmartin|ruckssbarnea, a dummy question, are you sure the failure is not related to the changes?13:13
kopecmartin|ruckor maybe it got fixed already, because the job worked here https://review.openstack.org/#/c/601850/13:13
*** vinaykns has joined #oooq13:17
*** ratailor has quit IRC13:21
ssbarnea_kopecmartin|ruck weshay helped me yesterday and he said something about a pip caching bug, but I am not sure about that. first, apparently the weird behaviour is while using setup install method and not inside pip.13:27
ssbarnea_i didn't use eggs too much as I went to wheels as soon it was possible, so i am not sure exact how they behave.13:29
pandassbarnea_: ping13:33
*** holser_ has quit IRC13:34
ssbarnea_panda yep.13:34
pandassbarnea_: can you please rejoin thee meeting ?13:35
ssbarnea_sure13:36
*** ykarel_ has joined #oooq13:36
*** holser_ has joined #oooq13:36
kopecmartin|ruckssbarnea, interesting, the bug was seen three months ago https://bugs.launchpad.net/tripleo/+bug/1777261, then it was closed and a new bug was created a month after that https://bugs.launchpad.net/tripleo/+bug/1786106 which looks very similarly13:38
openstackLaunchpad bug 1777261 in tripleo "convert-overcloud-undercloud.yml fails on missing update_containers variable" [Critical,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)13:38
openstackLaunchpad bug 1786106 in tripleo "containers-prep fails w/ update_containers is undefined due to lack of default" [High,Fix released] - Assigned to Quique Llorente (quiquell)13:38
kopecmartin|ruckweshay, are you sure, the bug https://bugs.launchpad.net/tripleo/+bug/1786106 was fixed, because ssbarnea experience the same now https://bugs.launchpad.net/tripleo/+bug/179604813:38
openstackLaunchpad bug 1796048 in tripleo "quiskstart.sh installs downgraded version of tripleo-quickstart-extras" [High,Triaged]13:38
*** ykarel has quit IRC13:38
weshaykopecmartin|ruck, ssbarnea is not using the latest code13:41
weshayhe's hitting a pip cache issue in ci13:41
weshaykopecmartin|ruck, because jenkins .. weee13:41
weshaywould need to see it recreated locally13:42
rascasoftweshay, we should not using the pip cache, that problem hits me every time13:42
rascasoftweshay, would it be possible?13:42
weshayrascasoft, I have not found a way to prevent jenkins from doing it13:42
weshayrascasoft, and this issue is in ci.centos where we are not admins13:42
weshayrascasoft, kopecmartin|ruck I was thinking about NOT using pip to install tq tqe on jenkins ci13:43
rascasoftweshay, what is curious is that I've hit the same problem without using Jenkins13:43
weshayand just git pull and setup the ansible role path13:43
rascasoftweshay+++++++++++++++++++++++++++++13:43
rascasoftthat will also speed up things13:43
rascasoftif we don't need it for specific reasons13:44
weshayrascasoft, ya.. there is no reason why quickstart.sh can't setup the config and roles properly13:44
*** agopi has quit IRC13:44
weshayrascasoft, and more of the deps could be rpm based I suppose13:44
weshayit's worth revisiting13:44
rascasoftweshay, count me in13:44
* rascasoft is all for simplifying things13:44
quiquellrlandy, panda: promote job is 'working' now at the proto https://review.rdoproject.org/r/#/c/16773/13:46
rlandycool13:46
quiquellrlandy, panda: Failing as expected at Auth of DLRN, you can review them13:46
quiquellrlandy: has force the release directly in the job13:46
rlandysec - just adding check job review13:48
quiquellrlandy: ack13:48
quiquellrlandy: going to move the proto into config review13:49
ykarel_weshay, ssbarnea i think need to adjust that tripleo-quickstart-gate-quick job: https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-quickstart-templates.yml#L129, add repo-tripleo-quickstart-extras to scm, i don't know how that works, if u know about it, good to try that13:49
ykarel_looks like tripleo-quickstart-extras is cached somewhere13:50
*** ykarel_ is now known as ykarel13:50
*** gkadam has joined #oooq14:00
weshaymarios|rover, kopecmartin|ruck https://review.openstack.org/60790414:01
weshayoops14:01
weshaysorry14:01
weshayopenstack-periodic-24hr14:02
*** agopi has joined #oooq14:02
marios|roverack cool weshay thanks14:02
weshaythat has been running for 130 hrs14:03
weshaylet's ask #sf-dfg to kill it14:03
quiquellrlandy, panda: promote proto added to 'config' repo review14:06
quiquellrlandy, panda: About build-containers I don't know why it's not working https://logs.rdoproject.org/68/16768/23/check/periodic-tripleo-centos-7-master-containers-build/64a906b/job-output.txt.gz#_2018-10-04_11_46_28_83352914:07
quiquellrlandy, panda: Also maybe we can just call build-test-packages instead of parenting14:08
quiquellDo you have need some catch up ? I will drop soon14:08
rlandyin 5 mins14:08
quiquellok14:08
rlandysorry - just getting this review in14:08
quiquellrlandy: Very important too14:08
*** hamzy has quit IRC14:10
rlandyquiquell: k - let's meet14:14
rlandypanda: ^^14:14
pandaquiquell: have you tried putting the import_playbook taskunder a host: declaration ?14:15
pandarlandy: where ?14:15
rlandytripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-masterqueued (non-voting)14:15
rlandytripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branchqueued (non-voting)14:15
rlandyha - we have queued check jobs14:15
quiquellpanda: nope14:15
rlandywe can use my bj14:15
pandarlandy: id ?14:16
pandasshnaidm: do you want to join ?14:16
*** jaosorior has quit IRC14:16
sshnaidmpanda, join what?14:16
quiquellrlandy, panda: connected14:17
pandasshnaidm: the rebellion!14:17
pandarlandy: id ?14:17
sshnaidmpanda, where?14:17
rlandyno idea14:17
pandano IDea14:18
pandaok14:18
rlandyblue jeans has gone on strike14:18
quiquelldman14:18
sshnaidmin https://bluejeans.com/u/rlandy/ ?14:19
quiquellI am there now14:19
pandasshnaidm: that's hte idea, but app works only with ids so I'll be late until I find the id14:19
rlandyhttps://bluejeans.com/116727655614:22
weshayarxcruz, thanks for your help today14:26
weshayarxcruz, one last thing if you are still with me.. easy one14:26
*** tosky__ has joined #oooq14:27
*** tosky has quit IRC14:28
*** tosky__ is now known as tosky14:28
*** bogdando has quit IRC14:29
*** saneax has quit IRC14:30
*** ykarel is now known as ykarel|afk14:32
*** bogdando has joined #oooq14:32
*** Vorrtex has joined #oooq14:34
*** skramaja_ has quit IRC14:36
*** hamzy has joined #oooq14:45
*** kopecmartin|ruck is now known as kopecmartin|scho14:47
arxcruzweshay: just my job :)14:48
arxcruzweshay: what do you need boss?14:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario004-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode- (4 more messages)14:51
weshayarxcruz, scenario tests added to fs2714:55
weshaysorry for the delay14:55
arxcruzweshay: so... need to check the list of skipped because tempest is running in undercloud only14:56
arxcruzweshay: if panda authorize me to work on that, i will :P14:56
weshayarxcruz, I'm putting up a patch now.. just asking for review14:57
arxcruzahhh14:57
arxcruzi was just kidding :)14:57
*** toure is now known as toure|biab14:59
marios|roverdtantsur: ++ thanks for the patch https://review.openstack.org/#/c/607904/14:59
dtantsurnp15:00
*** holser_ has quit IRC15:11
*** holser_ has joined #oooq15:11
*** quiquell is now known as quiquell|off15:18
*** toure|biab is now known as toure15:19
weshayarxcruz, https://review.openstack.org/60798715:19
arxcruzweshay: k, let's hope it will pass the vm getting booted15:20
*** sshnaidm is now known as sshnaidm|afk15:20
weshaypanda, rlandy how we doing on making the change to legacy --> zuulv315:27
rlandypanda: sshnaidm|afk: https://review.rdoproject.org/r/#/c/16761 is updated ...  https://review.rdoproject.org/r/16783 contains the trugger15:29
rlandytrigger15:29
rlandyweshay: you just missed the whole discussion15:29
weshaystill in mtgs15:31
weshaysorry15:31
*** agopi is now known as agopi|long_lunch15:31
weshayjust looking for essentially, green,yellow, red..15:31
weshayhappening today, tomorrow.. never15:31
pandarlandy: sshnaidm|afk quiquell|off There is not way to make the import_playbook work. Commented in https://tree.taiga.io/project/tripleo-ci-board/task/119?kanban-status=144727515:33
pandaweshay: hopefully today, by the end of your day15:34
weshayvery nice15:34
weshaynice work everyone15:34
*** sshnaidm|afk is now known as sshnaidm15:36
rlandypanda: let's go for post15:36
*** udesale has quit IRC15:36
rlandypanda: if you see my original view, I had the build in post15:36
rlandyweshay: first check jobs testing on https://review.openstack.org/#/c/60796315:39
*** hamzy_ has joined #oooq15:42
*** bogdando has quit IRC15:43
*** hamzy has quit IRC15:44
*** ykarel|afk is now known as ykarel15:52
*** jfrancoa has quit IRC15:55
*** jaganathan has joined #oooq15:58
*** trown is now known as trown|lunch16:04
*** jaganathan has quit IRC16:05
ssbarnea_is this url correct https://images.rdoproject.org/master/rdo_trunk/current-passed-ci/overcloud-full.tar.md5 ?16:08
ssbarnea_because there is nothing there16:08
rlandysshnaidm: regarding https://review.rdoproject.org/r/#/c/16777/3/playbooks/tripleo-ci-periodic-base/post.yaml - your comment refers to line 13 - "periodic-" is also required on line 15, right?16:09
sshnaidmrlandy, yep16:09
pandassbarnea_: where did you see that ?16:10
ssbarnea_panda https://ci.centos.org/job/tripleo-quickstart-gate-master-full-images/96/console16:10
sshnaidmwe don't use current-passed-ci for last 3 years16:10
*** dtantsur is now known as dtantsur|afk16:10
rlandysshnaidm: https://review.rdoproject.org/r/#/c/16786/ - pls review - thanks16:16
*** ykarel_ has joined #oooq16:18
pandacontainers build job is installing the undercloud right now16:19
*** ykarel_ is now known as ykarel|away16:20
*** ykarel has quit IRC16:20
weshayrfolco, ssbarnea_ you guys get the email about the infra mtg next week?16:22
rfolcoweshay, I did, not sure it will conflict with meetup16:22
weshayrfolco, ya.. it conflict, if you can try to attend the infra mtg.. stepping away from the tc/ua that would be very helpful16:23
rfolcoweshay, yes, accepted16:23
ssbarnea_weshay: yep, accepted right away.16:26
ssbarnea_rfolco weshay : https://review.openstack.org/#/c/607624/16:28
weshayssbarnea_, thinking you can do better than that :)16:31
rlandypanda: anything I can help with?16:33
weshayssbarnea_, have a minute? I'll show you what is more likely the issue16:39
ssbarnea_weshay: yep. while I suspect the cause is somewhere else I am confused by the fact that extras-common/tasks/main.yml:24 does not exist, the file is empty.16:40
ssbarnea_weshay i seen the same task in another role....16:40
weshayssbarnea_, ya.. this is a shitty problem, probably a few things going on16:41
weshaylet's try to solve on blue?16:41
ssbarnea_weshay i am on https://bluejeans.com/2655417928 if you have time16:41
pandarlandy: just iterating on the containers build, fixed workspace variable, just rerunning16:43
pandarlandy: undercloud install passes, of course, testing build in post16:43
*** holser_ has quit IRC16:45
weshayssbarnea_, https://github.com/openstack/tripleo-quickstart-extras/commit/c2a3aac33f175f71575d426115791171e7ad60d3#diff-23af0a8485e71d2131516d07ef64d3d8R2416:49
ssbarnea_weshay : https://bugs.launchpad.net/tripleo/+bug/162393316:50
openstackLaunchpad bug 1623933 in tripleo "Quickstart doesn't allow to use different pip mirrors or cache" [High,Triaged]16:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario004-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode- (4 more messages)16:51
weshayssbarnea_, https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-gate-master-delorean-quick-basic/ws/usr/local/share/ansible/roles/extras-common/tasks/main.yml/*view*/16:53
ssbarnea_weshay : the one i created today https://bugs.launchpad.net/tripleo/+bug/179604816:53
openstackLaunchpad bug 1796048 in tripleo "quiskstart.sh installs downgraded version of tripleo-quickstart-extras" [High,Triaged]16:53
rlandypanda:  - anyone worked on commenting out the promoter?16:54
rlandyotherwise will pick that up16:54
*** ykarel|away has quit IRC16:58
*** trown|lunch is now known as trown17:04
*** gkadam has quit IRC17:08
pandarlandy: you want to comment to prevent false positives ?17:10
rlandypanda: ack17:23
*** amoralej is now known as amoralej|off17:37
*** vinaykns has quit IRC17:38
rlandypanda: https://review.rdoproject.org/r/16793 - for when we are ready to test promotions17:49
weshayrlandy, so that will not attempt to promote because ci-testing -> current-tripleo is remarked out17:51
weshayright?17:51
rlandyweshay: that is the hope17:51
weshayrlandy, was thinking we could add a fake criteria17:51
weshaylike featureset09917:51
weshaybut either is fine by me17:52
rlandyhttps://review.rdoproject.org/r/#/c/16783 - switches the jobs17:52
weshayunless we're not confident17:52
weshay\0/17:52
rlandyI think we can start uncommenting some lines when we see one successful run17:52
weshayrlandy++17:52
hubbot1weshay: rlandy's karma is now 2417:52
weshayrlandy, is your confidence 90%+ on https://review.rdoproject.org/r/#/c/16793/17:53
weshayor should we merge say.. pike.ini17:53
rlandyweshay: panda is working on the promote/container build stuff - we need to wait on that job17:53
weshayand watch the promoter17:53
rlandyif panda is confident on that piece, I am semi confident on the rest17:54
rlandyweshay: the check jobs could move forward though17:55
rlandyhttps://review.rdoproject.org/r/#/c/16781/17:56
weshayrlandy, that's just an addition right17:57
weshayso we'll see legacy + the new job17:57
weshaytogether atm17:57
rlandyweshay: yep for the moment17:57
weshayk cool17:57
weshayI only have +117:57
weshaythere17:57
rlandyif the current run passes, we can switch the legacy out - your choice17:57
rlandystill in test rim17:58
rlandyrun17:58
weshayif they work sure.. move forward.org17:58
weshay :)17:58
rlandylol - quite the movement for the moment17:58
*** ykarel|away has joined #oooq17:59
*** ykarel_ has joined #oooq18:04
*** holser_ has joined #oooq18:05
*** holser_ has quit IRC18:05
*** holser_ has joined #oooq18:06
*** ykarel|away has quit IRC18:07
weshayrfolco, any thoughts on how to resolve https://bugs.launchpad.net/tripleo/+bug/179571818:12
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]18:12
pandaspank the server18:14
rfolcohahaha18:14
pandabad boy, bad.18:14
rfolcotake videogame from him18:15
weshayrfolco, it's in our zuul config18:16
weshaypanda, you added it :)18:16
rfolcoweshay, did we test lower ttl ?18:16
weshayhttps://github.com/openstack-infra/tripleo-ci/blame/master/zuul.d/base.yaml#L1518:18
* rfolco looks to panda 18:18
weshayhttp://codesearch.openstack.org/?q=unbound_cache_min_ttl&i=nope&files=&repos=18:18
weshaymaybe remove it?18:18
weshaypanda, did paul or david mention anything about that setting?18:19
weshayI do like the jokes btw :)18:19
weshayI laughed18:19
rfolcoweshay, I can test it while we don't find an explanation18:20
rfolcoweshay, like overriding ttl (removing it from the job)18:20
weshayrfolco, k.. thanks.. just curious how one would test that?18:20
weshayjust in ci?18:20
rfolcoweshay, yes, patch to tripleo-ci, remove that line, and use default (0)18:20
rfolcoweshay, also can hide other jobs to speed up tetsing18:21
weshayrfolco, fine thanks.. panda please let me know if it was added based on some advice or something18:21
rfolcowip18:21
rfolcoweshay, https://review.openstack.org/#/c/608051 meanwhile can look at the bug but I don't feel very familiar with whats going on there18:28
pandarfolco: the build starts but we cannot access the rdo registry, so it's not going too far18:29
pandarlandy: ^18:29
pandaweshay: rfolco I added that variable only because I copied from somewhere else.18:29
weshaypanda, where?18:29
pandamindless ccopying18:29
weshayI don't see it in the rest of openstack18:29
weshayack k18:29
pandalegacy job of something similar18:30
pandaor18:30
weshaylegacy had zuul_vars?18:30
weshaypanda, are you pto tomorrow?18:31
weshayssbarnea_, what is openstack coats?18:35
weshayhttps://wiki.openstack.org/wiki/Jacket18:35
weshay?18:35
weshaydeltacloud much ^ rlandy lolz18:35
rlandyoh my gosh ... flashback18:36
rlandywell new jobs fail the same way the old jobs did18:40
rlandynot sure how happy we should be about that18:40
ssbarnea_weshay: coats = some (greasemonkey) scripts to improve the browsing experience, wait few days to get mpeterson code merged into it and I will send an email.18:48
weshayah neat18:48
weshaytotally diff than jackets :)18:48
weshaylolz18:48
ssbarnea_weshay: has nothing to do with jacket, didn't know about jacket when I picked that name.18:49
ssbarnea_initially i was considering opentack-monkeys but infra guys told me that some people may find the term controversial18:50
ssbarnea_anyway, is not something we will release, just a place to keep internal goodies that makes our day to day life easier.18:50
weshayheh18:50
pandaweshay: no pto tomorrow, leaving a bit early to fly18:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007 (3 more messages)18:51
pandarlandy:  ?18:51
rlandypanda: ?18:52
pandarlandy: the new jobs failing like the old ?18:53
rlandyfs001 master v3 and legacy both fail overcloud deploy with 'no valid host' error18:53
pandathat's great ?18:54
rlandypanda: well in a weird way, yes18:54
pandaI agree18:54
pandarlandy: ok , so the container build runs, fails on the credentials18:54
rlandyI just reset the test job with another change18:54
*** agopi|long_lunch is now known as agopi18:54
pandaI think we can migrate at this point. We need the real deal18:54
rlandypanda: legit fails on creds?18:54
pandarlandy: yes, it doesn't have credentials, so can't access the rdo registry to check the images18:55
pandarlandy: https://logs.rdoproject.org/68/16768/29/check/periodic-tripleo-centos-7-master-containers-build/20c9b6d/job-output.txt.gz#_2018-10-04_18_18_12_68439218:55
rlandypanda: ok - if you merge your job, I can merge https://review.rdoproject.org/r/#/c/16793/ (no promote) and https://review.rdoproject.org/r/#/c/16783/18:56
rlandypanda: weshay: can we kill the current promotion18:57
rlandyfs016 still queued18:57
weshayrlandy, which?18:57
weshayrlandy, oh.. I already asked18:57
rlandy134 hours18:57
weshayin sf-dfg18:57
rlandythat will kill our testing18:57
weshayactually have asked a few poeple18:57
weshayjebus18:58
weshayrlandy, btw.. rocky bm works :) https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/all%20top%20level%20multijobs/job/rdo-promote-rocky-rdo_trunk/13/18:59
rlandy135hr - wow18:59
rlandyweshay: woohoo - how nice18:59
* rlandy hasn't looked at downstream in ages19:00
weshayrlandy, no worries19:00
weshayignore it ;)19:00
weshayI'm not worried about it at all.. just wanted to let you know it was working19:00
rlandyworry is part of genetic make up19:00
pandarlandy: modyfying https://review.rdoproject.org/api/16761  to include the containers build part, we can merge that, then we can start the switch19:01
rlandypanda++19:01
hubbot1rlandy: panda's karma is now 319:01
rlandypanda: we should change the periodic trigger to start the test run as soon as we are ready ...19:07
pandarlandy: yes19:08
rlandypanda: ok- adding a separate review so we can revert it easily19:09
pandarlandy: I guess we'll change it a lot more in the next few days19:09
rlandyif they don;t kill the current promote, that will harm us19:10
*** ykarel__ has joined #oooq19:13
*** ykarel__ has quit IRC19:14
*** ykarel__ has joined #oooq19:14
rlandywaiting on when we are ready to switch before modifying time19:15
*** ykarel_ has quit IRC19:15
*** ykarel_ has joined #oooq19:17
*** ykarel__ has quit IRC19:20
*** ykarel__ has joined #oooq19:21
*** ykarel_ has quit IRC19:23
pandarlandy: ok, first shot https://review.rdoproject.org/api/1676119:24
* rlandy checks19:24
*** ykarel__ has quit IRC19:26
rlandypanda: ok - here we go19:27
rlandylet's merge19:27
rlandycan't do damage in its own19:27
pandaka-boom19:28
rlandypanda: pls review https://review.rdoproject.org/r/#/c/16793/ so we don;t damage anything yet19:28
rlandydone and done19:31
pandarlandy: commented19:32
rlandyyeah - good point but idk what that would do to phase1 promotions19:33
rlandypanda: ok - merging https://review.rdoproject.org/r/#/c/16793/19:34
pandathere's no phase1 if there's not tripleo promotion <add black man touching his head with the finger meme>19:34
rlandywell, I didn;t check if phase 1 promoted yet off the last tripleo promotion19:35
rlandyhmmm - zuul ci? gate19:36
rlandyah running now19:36
pandarlandy: is this done ? https://tree.taiga.io/project/tripleo-ci-board/task/108?kanban-status=144727519:37
rlandypanda: we have not merged the review yet ... https://review.rdoproject.org/r/#/c/16781/4/zuul.d/tripleo.yaml - so no19:38
rlandybut this will be soon - can reassign to me and unblock ...19:38
rlandyoh it is unblocked19:39
rlandyhttps://tree.taiga.io/project/tripleo-ci-board/task/105?kanban-status=144727519:39
pandaI unblocked it19:41
rlandypanda: ^^ what's the right process- do I assign that card to me or leave it with quique?19:41
pandaIt was blocked only by the notice19:41
pandarlandy: at this point we're collaborating so closely it really doesn't matter19:41
rlandyok19:42
rlandynow for the real damage ...19:42
rlandyhttps://review.rdoproject.org/r/#/c/16783/1/zuul.d/tripleo.yaml19:42
rlandyplease vote19:42
rlandyso we have some legitimacy19:42
pandacan I vote -2 just because I'm a coward ?19:42
rlandyyou could but we would be here forever19:43
pandaugh, I can't vote -219:43
rlandyat some point I would go and retire in the south of spain and it would be all yours19:43
* panda wonder where quiquell|off will run then19:44
pandahe's already in the south of spain (more or less)19:44
rlandylol19:44
rlandyhe can come to ny19:44
rlandywe'll switch places19:44
rlandyhow do you get any work done if you live in such a nice, sunny place???19:45
rlandypanda: it's not going to get any easier :(19:46
pandano spain no gain.19:46
rlandylol19:46
pandarlandy: did I mention I'll be in Brno all next week ?19:46
rlandythat was actually very funny19:46
rlandypanda: I deserve that - after I left for two days every week19:47
rlandyugh - I had to rebase it anyways19:47
rlandywhatever19:48
rlandypanda; ok - now the question is which pipeline do you want to trigger and at what time?19:49
rlandyheaven help us ... here we go19:50
pandarlandy: frankly, I'll be in and out for the next two hours then I'll disappear, better that you decide your faith19:51
rlandypanda: are you in at all tomorrow?19:51
pandarlandy: yes, but I'll leave earlier to catch my flight for Brno19:52
pandarlandy: at around 15 UTC19:52
pandaI'l fight all morning with quique19:52
pandaand sagi19:52
rlandyalrighty then19:53
pandaoh such bravery!19:53
rlandywe've done it now19:53
* panda hears a strange sound of gears grinding19:54
rlandy      timer:19:55
rlandy        - time: '45 4,9,14,22 * * *'19:55
rlandytime now is ...19:55
rlandy19:5519:55
* rlandy wonders if it's worth changing the time19:55
rlandychanging 22 to 2019:56
rlandypanda: before you disappear19:58
rlandyhttps://review.rdoproject.org/r/1679719:58
rlandycorrect that that will kick in 10 mins?19:59
rlandyhmmm ... making that 20 - we may need merge time19:59
rlandyweshay; ^^ ok?20:04
rlandywill revert as soon as job kicks20:04
pandarlandy: should be ok20:05
pandarlandy: why do you want to revert ? I think We'll need to change the trigger time a lot20:05
rlandyok20:06
pandaI would almost suggest to run it only manually at this point20:06
rlandyhow do we kick that manually?20:06
pandawell, manually I mean, put only a single time to trigger there20:06
rlandythat is a much better suggestion20:06
pandathen move it wehn needed to trigger20:07
pandaagain20:07
rlandypanda: ack - and we should comment out the second pipeline20:07
pandaagree, let's deal with master and rocky first20:08
rlandypanda: ok - putting in another review20:08
rlandywill merge after first job kicks20:09
rlandyhmmmm - how do you disable a pipeline?20:12
rlandyOMG - we have triggering20:21
rlandypanda: still there?20:24
rlandywell that was quick20:25
pandaa quick failure ?20:26
pandausually the failure are mapped with retries in a fibonacci way : first retry fails in 1 minute, second in 1 minute, third in 3 minutes, and so on20:27
rlandypanda: yep - could not find file20:27
rlandybut I ma not sure where I get those logs20:27
rlandyI know where the promoter logs are but not the container buildlogs20:27
rlandypromote to tripleo-ci testing seemed to work20:28
rlandyalso ... https://review.rdoproject.org/r/16798 Temp disable 24hr pipeline and trigger periodic pipeline once20:28
pandarlandy: hash_info.sh ?20:32
rlandypanda: ack20:32
rlandyhow did you know?20:33
rlandylogs?20:33
weshayrlandy, ok.. the job was killed20:34
rlandyweshay: ack - container build failed20:34
weshayah.. hrm..20:34
weshaywonder if we could point it at a known good hash20:34
rlandyweshay: stupid question - but where are the logs?20:35
weshayfor which hash was good?20:35
weshayor jobs?20:35
rlandycontainer build jobs20:35
rlandyjob run, failed, no trace20:35
pandaweshay: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/20:35
weshayrlandy, https://review.rdoproject.org/zuul/builds.html20:35
pandaoh, uhm20:36
weshaypipeline = openstack-periodic20:36
weshaypost-failure20:36
rlandythank you20:36
weshayhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/98a6239/20:36
weshaymary |   "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'change'\n\nThe error appears to have been in '/tmp/tmpaklh4emf/98a623969dd44c69ae739bd06ccebd59/untrusted/project_3/git.openstack.org/openstack-infra/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml': line 92, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to20:37
weshaybe:\n\n\n    - shell:\n      ^ here\n"20:37
weshay2018-10-04 20:24:50.539569 | primary | }20:37
rlandywell let's start here ... https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/49cf39a/job-output.txt.gz - really a success?20:37
weshayhttps://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build/98a6239/job-output.txt.gz#_2018-10-04_20_24_50_53951720:37
*** holser_ has quit IRC20:39
*** jtomasek has quit IRC20:39
pandammhhh20:41
pandawe'll need to change run-v320:42
rlandyon the upside ...20:42
rlandytripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch success (non-voting)20:42
weshaypanda, rlandy anything I can do to help?20:42
*** tosky has quit IRC20:42
*** tosky has joined #oooq20:43
pandayeee20:43
pandaweshay: from now it's iteration and quick fixes20:43
pandaso the first problem is that to form LOG_PATH we need zuul.chanes20:43
pandawhich is not defined in periodic pipeline20:43
weshaypanda, ok.. the gate quueue is short20:44
pandaI can't even find where we are using this LOG_PATH20:44
weshayanything need review?20:44
*** hamzy_ has quit IRC20:44
weshaythe LOG_PATH var is defined from zuul.changes wtf?20:44
* weshay looks20:44
pandait's used in tempest stuff, tempestmail to post results20:44
weshaythat's what caused the post failure?20:45
weshaypanda, we can turn that off in the periodic jobs20:45
weshayno need for tempest mail notifications from that20:45
pandaok20:45
pandalet's add a conditional then20:45
pandawe set LOG_PATH only if zuul.change is defined20:46
weshayhttp://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/playbooks/tripleo-ci/run-v3.yaml#n10820:47
pandaweshay: yes20:48
weshayhttp://codesearch.openstack.org/?q=LOG_PATH&i=nope&files=&repos=tripleo-ci,tripleo-quickstart,tripleo-quickstart-extras20:48
rlandyside point ... https://review.rdoproject.org/r/#/c/16781/ - I think this can merge20:48
weshayodd20:48
rlandypicking up the change20:49
rlandyfailures were the same as legacy20:49
rlandyin master20:49
rlandyqueens passes20:49
pandahttps://review.openstack.org/60808020:50
pandaI love those change id20:50
pandathat20:50
rlandycool20:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007 (4 more messages)20:51
rlandypanda: we'll have the same problem in ... https://github.com/openstack-infra/zuul-jobs/blob/master/roles/set-zuul-log-path-fact/tasks/main.yaml#L420:53
rlandyin playbooks/tripleo-ci-periodic-base/containers-build.yaml20:54
*** trown is now known as trown|outtypewww20:54
pandarlandy: something tells me that role is prepared for problem like this20:55
rlandywe20:55
rlandywill find out20:55
pandarlandy: yep it's handled already20:56
rlandynice20:56
pandarlandy: wait there's more20:56
rlandyoh what now?20:56
rlandyhttps://review.openstack.org/#/c/607963/20:57
rlandysee resultes20:57
rlandyresults20:57
pandarlandy: it's actually handled a lot better than we do20:57
rlandycopy, copy20:57
weshay?20:57
rlandy when: zuul.change is defined20:57
rlandyweshay, https://github.com/openstack-infra/zuul-jobs/blob/master/roles/set-zuul-log-path-fact/tasks/main.yaml#L420:58
rlandy^^ handle the problem20:58
pandayes20:58
weshayah20:58
pandaI'll add the role to run-v3 playbook20:58
weshayso that can change to {{ zuul_log_path }}20:58
weshaynice find rlandy20:58
weshayrlandy, so my last comment on https://review.openstack.org/#/c/608080/120:59
weshayis that correct?20:59
weshayhttps://review.openstack.org/#/c/60448721:01
rlandywe would have to call the role to have zuul_log_path defined21:01
pandauploaded patch set 221:01
weshayright21:01
weshaypatch needs a depends on21:02
weshayor patch set 221:02
weshaycool21:02
rlandylooks good to me ...21:02
rlandyso ... to trigger again21:02
rlandyhttps://review.rdoproject.org/r/#/c/16798/1/zuul.d/upstream.yaml21:03
rlandyto retrigger21:03
rlandychange the time21:03
rlandyweshay: panda: ok - since we are living dangerously, - can I merge https://review.rdoproject.org/r/#/c/16781/21:05
weshay+1 from me21:05
rlandyhere we go21:05
weshayrlandy, when you have a minute https://review.openstack.org/#/c/604487/21:06
rlandywill look in a bit21:07
rlandycould we add a depends-on to this review https://review.rdoproject.org/r/#/c/16798/ and trigger?21:08
rlandythat wouldn't work21:09
pandanope21:09
rlandyyeah21:10
pandato make it work we would need all the jobs to clone the change before running21:10
rlandypanda: unless we made a temp run-v3 playbook21:12
*** agopi has quit IRC21:12
rlandyand merged that21:12
rlandyin rdo-jobs21:12
pandarlandy: but you have to change all the base jobs to run it21:18
rlandyat least we can merge there21:19
rlandydepends how long it will take us to get this change to merge21:19
rlandygetting through gates etc.21:19
pandait makes sense in the short run21:20
pandait may not be the only change we have to make to it ...21:20
rlandyreally only ovb base, multinode base, singlenode base21:20
rlandycorrect21:20
rlandypanda could even be a depends on change to the trigger21:20
rlandyforget that21:21
* rlandy keeps trying to limit the damage21:21
rlandybut we're all in now21:21
*** toure is now known as toure|gone21:22
rlandypanda: are you making that change or should I?21:28
pandarlandy: I'm about to call it a day21:32
rlandypanda: sure - it's late for you21:32
rlandyI'll look into the change21:32
rlandyeven just to have it ready for you21:33
rlandymaybe your review will go through21:33
rlandyI'll watch it21:33
pandarlandy: thanks. See you tomorrow.21:36
*** panda is now known as panda|off21:36
*** Vorrtex has quit IRC21:49
*** weshay is now known as weshay_pto21:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario004-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode- (4 more messages)22:51
*** tosky has quit IRC23:17

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!