Thursday, 2018-10-18

*** dsneddon_away is now known as dsneddon00:21
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (3 more messages)00:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024,  (3 more messages)02:23
*** saneax has joined #oooq04:06
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (3 more messages)04:23
*** ykarel has joined #oooq04:33
*** udesale has joined #oooq04:45
*** ccamacho has quit IRC05:03
*** jfrancoa has joined #oooq05:24
*** rlandy|bbl is now known as rlandy05:39
*** quiquell has joined #oooq05:45
*** ratailor has joined #oooq06:01
*** holser_ has joined #oooq06:02
*** jtomasek has joined #oooq06:18
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (3 more messages)06:23
*** ykarel has quit IRC07:13
*** ykarel has joined #oooq07:26
*** tosky has joined #oooq07:31
*** apetrich has quit IRC07:39
*** bogdando has joined #oooq07:48
*** dtantsur|afk is now known as dtantsur07:56
*** skramaja has joined #oooq07:57
*** apetrich has joined #oooq08:00
*** kopecmartin|off is now known as kopecmartin08:02
*** holser_ has quit IRC08:22
ssbarneamorning! does anyone knows if the upgrade jobs were fixed? i guess that hubbot1 statement is still true...08:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (3 more messages)08:23
*** ykarel_ has joined #oooq08:27
*** ykarel has quit IRC08:30
quiquellssbarnea: yep, a review has pass the gates an break it08:33
quiquellssbarnea: marios has a fix for it already, let's merge it08:33
mariosssbarnea: 'fix' == revert :)08:33
mariosquiquell: i tried pinging you about the commit message but i just updated it seeing my one was merging ./logfile_17Oct2018.log:139:2018-10-17 01:14:04 | Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 108:33
mariosdamint08:33
marioshttps://review.openstack.org/#/c/611526/ here quiquell08:33
marios(i pinged in tripleo)08:34
*** ykarel_ is now known as ykarel08:35
ssbarneamarios quiquell : i am a bit confused because weshay told me yesterday about emillien change that was supposed to fix the upgrade issue by removing it from the gate.08:36
ssbarneaso i got the impression that this was the right thing to do, now I see a revert.08:37
quiquellssbarnea: emilien stuff depends on this https://review.openstack.org/#/c/608282/08:39
quiquellssbarnea: And that is already merged, so we can put back upgrade jobs at gates08:39
quiquellssbarnea: The broken stuff at check is because the upgrade job was not in the gates :-(08:39
*** ratailor has quit IRC08:43
*** ratailor has joined #oooq08:43
ssbarneaquiquell: well my job was failing on upgrades long time before emillien patch, and failed again even after it was merged.08:45
ssbarneabut i wonder what kind of error is that: http://logs.openstack.org/18/610018/9/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/0b01cb9/logs/undercloud//home/zuul/overcloud_deploy.log.txt.gz08:45
mariosssbarnea: no error there ^08:45
ssbarneai see only Timed out waiting for port 22 from 192.168.24.608:45
quiquellssbarnea: Emilien pach just make able to broken stuff to pass the gates08:45
*** skramaja_ has joined #oooq08:45
mariosssbarnea: sorry was talking on tripleo08:45
mariosssbarnea: there is also an overcloud deploy fail if that is what you are asking (i don't see fail in 11:45 < ssbarnea> but i wonder what kind of error is that:08:46
marioshttp://logs.openstack.org/18/610018/9/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/0b01cb9/logs/undercloud//home/zuul/overcloud_deploy.log.txt.gz08:46
*** skramaja has quit IRC08:46
mariosoh timeout08:46
mariossee it08:46
mariosbut it is complete?08:46
mariosssbarnea: k that is something different to the undercloud thing though?08:46
*** jaganathan has quit IRC08:47
mariosssbarnea: there is an overcloud deploy error here https://bugs.launchpad.net/tripleo/+bug/1798538 which may be related to the undercloud bug08:47
openstackLaunchpad bug 1798538 in tripleo "tripleo-ci-centos-7-containers-multinode and scenario-multinode failing overcloud deploy ": Incorrect arguments: Items to concat must be lists"" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)08:47
ssbarneayeah, clearly different. i was asking because i seen only a timeout after 5mins and I wonder if 5mins is a long enough waiting value or not.08:48
*** marios|rover has joined #oooq08:49
*** marios|rover is now known as marios|rover|bac08:49
*** marios|rover|bac is now known as marios|rover|sti08:49
*** marios|rover|sti is now known as marios|rover08:50
mariosmarios|rover: you again!08:52
* marios|rover hids08:52
quiquellssbarnea: Was thinking about adding Depends-On to https://review.openstack.org/#/c/610018/08:53
quiquellssbarnea: But it will removed votes and all08:53
*** sshnaidm_ has joined #oooq08:59
pandammhh09:10
*** holser_ has joined #oooq09:17
*** sshnaidm_ has quit IRC09:18
*** gkadam has joined #oooq09:21
*** gkadam has quit IRC09:21
*** chem is now known as chem_brb09:39
*** holser_ has quit IRC09:56
*** holser_ has joined #oooq09:57
*** jbadiapa has quit IRC09:58
quiquellpanda, marios, ssbarnea: To have patchset filter at compare-reviews.py https://review.openstack.org/61155809:58
mariosquiquell: ack added for next reviews10:02
*** sshnaidm has joined #oooq10:06
quiquellsshnaidm: https://review.openstack.org/#/c/610018/10:07
ykarelsshnaidm, panda can u check my comment in merged rdo-jobs patch: https://review.rdoproject.org/r/#/c/16876/10:11
ykareland get a fix10:12
*** sshnaidm has quit IRC10:12
panda.... rdoinfo10:13
* panda screams10:14
quiquellssbarnea: Has check RPMs and PIP from the py3 changes they look ok, only different is 'pluggy' at tripleo-quickstart review but I think it's not related10:14
quiquellpanda: do you have a bj minute ?10:16
pandait should be noted that a bj minute corressponds to a real world hour10:16
pandaquiquell: give me 5 minutes, I have to fix rdoinfo jobs10:17
quiquellpanda: True, all we do is very unreal10:17
quiquellpanda: ack, just ping me when you are ready10:17
*** chem_brb is now known as chem10:21
pandaykarel: https://review.rdoproject.org/r/1701410:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (3 more messages)10:23
pandaquiquell: ready10:24
ykarelpanda, Thanks10:24
quiquellpanda: connected to your room10:25
ssbarneaquestion regaltedto freeipa: on f28 which version is expected to be installed the py2 or py3 one?10:29
*** ykarel is now known as ykarelunch10:40
ssbarneapanda: we are doing something really wrong on use of taige: a story should not change its ID from sprint to sprint. The entire idea of "leftovers from" does not make any sense from the SCRUM perspective.10:49
ssbarneathe story was not implemented, it needs to me MOVED to the next sprint, not destroyed and recreated10:50
pandassbarnea: let's have a chat after I finish with quiquell10:50
ssbarneaall the CRs that point to the original story have wrong links now... and I don't expect us to update reviews for that reason.10:51
ssbarneasure, that one is not urgent. i just discovered it.10:51
ssbarneachallenges of using a new tool10:51
pandassbarnea: I don't remember when the practice of adding the back reference to taiga US was introduced.10:52
pandassbarnea: but it seems to me that the task should be referenced, not the US10:52
pandaand the tasks will not be destroyed10:53
pandassbarnea: let me finish here, we'll have a chat about this.10:53
*** sshnaidm has joined #oooq10:58
*** udesale has quit IRC11:04
*** jjoyce has quit IRC11:05
pandaquiquell:  https://review.openstack.org/60423211:05
*** jjoyce has joined #oooq11:06
*** jbadiapa has joined #oooq11:14
*** jjoyce has quit IRC11:15
*** jjoyce has joined #oooq11:16
*** jjoyce has quit IRC11:18
*** jjoyce has joined #oooq11:19
mariospanda: quiquell folks what is tqmerge branch at https://github.com/openstack/tripleo-quickstart/tree/tqmerge please anyone know?11:29
pandamarios: in preparation for the merge11:30
mariospanda: maybe few more words :)11:31
mariosplease11:31
marioslike should i submit pull request to it?11:31
mariospanda: here it is causing problems cos we want to remove a job https://review.openstack.org/#/c/609708/11:32
*** ratailor has quit IRC11:32
mariospanda: or do we need to rebase it onto master?11:32
panda/o\11:33
mariospanda: oh ok great thanks!11:34
pandamarios: I'll talk to infra11:35
quiquellmarios: No idea, that's super weird11:37
pandait's not weird11:37
pandait's the temporary branch used for the merge of quickstart and -extras11:37
quiquellpanda: What temporal branch ?11:37
quiquellpanda: we need a branch for that ?11:38
pandaI didn't know zuul was considering it in the global configuration11:38
pandaquiquell: yes, because we are trying to merge two project with completely unrelated histories11:38
quiquellpanda: Maybe you need to add it to sprint issues11:38
quiquellpanda: We are not going to follow tripleo-quickstart history ?11:38
pandaquiquell: it's not an issue, it's part of an epic, and it's a HUGE change11:38
quiquellpanda: ok we have an epic11:39
pandaquiquell: we are merging two big projects,  histories need to be merged, and they don't have a single common ancestor11:39
pandait's going to be a shitshow.11:39
*** panda is now known as panda|lunch11:39
*** ykarelunch is now known as ykarel11:42
* quiquell takes the shit umbrella11:42
mariosquiquell: /me grabs a tent11:43
quiquellmarios: undercloud upgrade working https://review.openstack.org/#/c/611518/11:43
mariosquiquell: yah was watching it in zuul its waiting on your patch then it will go to gate i hope11:44
quiquellmarios: Why it's waitting on my patch ?11:44
mariosquiquell: other way around, your patch was wating for it11:45
mariosquiquell: anyway it is still in check queue11:45
mariosafaics11:45
mariosquiquell: oh no it is in the gate11:45
mariosat the bottom11:45
marios4th one up http://zuul.openstack.org/status when you filter for 61151811:46
quiquellmarios: puff... let's hope up staff does not fail11:46
marioseasy review please folks https://review.openstack.org/#/c/610009/11:47
mariosthanks you11:47
*** sshnaidm_ has joined #oooq11:50
quiquellmarios: Do we need at all featureset we run at RDO ?11:52
*** sshnaidm has quit IRC11:53
*** quiquell is now known as quiquell|lunch11:53
mariosquiquell|lunch: yeah weshay asked me to check it out because we will start ot use it11:54
mariosquiquell|lunch: is why i posted that11:54
mariosand was looking at fs2111:54
quiquell|lunchmarios: So what's the criteria to need it ?11:56
quiquell|lunchmarios: Maybe it make more sense to add it a base jobs definitions '11:56
quiquell|lunchmarios: If it's generally needed11:56
weshaymarios, what do you make of this? https://logs.rdoproject.org/12/603612/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/3c14cd6/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-10-18_10_34_1711:58
weshaywe have a return code of 011:58
weshayafter 2/4 nodes are introspected11:58
weshayseems like a bug to me11:59
weshaylooking for a second opinion11:59
mariosweshay: joining seals call and looking11:59
mariosweshay: saw this one yesterday pretty sure lemme see if i have it noted12:00
mariosweshay: did you see master gate issue btw https://bugs.launchpad.net/tripleo/+bug/179852512:03
openstackLaunchpad bug 1798525 in tripleo "The undercloud-upgrades job is failing during the upgrade with "error was: 'ironic_api_short_bootstrap_node_name' is undefined" " [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)12:03
*** ykarel has quit IRC12:07
*** ykarel has joined #oooq12:07
*** panda|lunch is now known as panda12:13
mariossshnaidm_: panda: ** folks if you have time vote here please https://review.openstack.org/#/c/610653/112:18
*** apetrich has quit IRC12:21
pandamarios: does it mean that we are not setting containerized_undercloud in rocky ?12:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (4 more messages)12:23
pandassbarnea: ping12:27
pandassbarnea: pingpingping12:27
*** ykarel_ has joined #oooq12:27
marios|roverweshay: wait is there a cix call now is that what jan meant or something else?12:28
* marios|rover cant see it in calendar12:28
weshaymarios|rover, naw12:28
weshaymarios|rover, I'm probably reading this wrong12:29
weshayhttps://logs.rdoproject.org/12/603612/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/3c14cd6/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-10-18_10_34_1712:29
weshay2/4 intially failed12:29
weshaythen it goes into a retry12:29
pandassbarnea: straping12:29
weshayand 3/4/ and 4/4/ pass?12:29
marios|roverweshay: yeah it announces the pass damnit it couldn't find the trace from yesterday but pretty sure i saw exactly this12:30
*** ykarel has quit IRC12:30
marios|roverweshay: hmm not _exactly_ the same but almost http://logs.rdoproject.org/48/602248/9/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/98220fc/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz12:30
*** trown|outtypewww is now known as trown12:30
weshayanyone seen this one? https://logs.rdoproject.org/58/609858/4/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/1ed7439/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-10-18_11_40_2712:31
weshayI need to stop looking at check12:31
weshaynevermind12:31
* weshay looks at no-ops12:31
weshaymarios|rover, ya.. ok..12:32
marios|roverweshay: yes this is the same as the undercloud one i mean 2018-10-18 11:40:27 | an undefined variable. The error was: 'haproxy_public_tls_inject_short_bootstrap_node_name' is undefined\n\nThe error appears to have been in '/var/lib/mistral/overcloud/Controller/host_prep_tasks.yaml': line 195, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n      path: '{{cert_path}}'12:32
*** holser_ has quit IRC12:32
marios|roverweshay: the trace in that overcloud deploy is same root cause as     https://bugs.launchpad.net/tripleo/+bug/179852512:32
openstackLaunchpad bug 1798525 in tripleo "The undercloud-upgrades job is failing during the upgrade with "error was: 'ironic_api_short_bootstrap_node_name' is undefined" " [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)12:32
marios|roverweshay: and is being reverted12:32
*** holser_ has joined #oooq12:33
*** apetrich has joined #oooq12:33
marios|roverweshay: btw https://review.rdoproject.org/etherpad/p/ruckrover-sprint20.112:34
*** gkadam has joined #oooq12:34
*** rlandy has joined #oooq12:36
ssbarneaweshay : i am trying to inderstand what happened with https://bugs.launchpad.net/tripleo/+bug/1798198 as I see it marked as fixed release but without any link to the what was fixed. I only see alembic failing to install.12:41
openstackLaunchpad bug 1798198 in tripleo "pip installs failing in rdo-cloud based jobs" [Critical,Fix released]12:41
weshayssbarnea, in an rdo cloud job?12:41
weshayssbarnea, link?12:41
ssbarneaweshay see https://bugs.launchpad.net/tripleo/+bug/1798198 - your comment with link to error.12:42
openstackLaunchpad bug 1798198 in tripleo "pip installs failing in rdo-cloud based jobs" [Critical,Fix released]12:42
ssbarnea15min ago.12:42
weshayssbarnea, it's an import error12:43
weshaynot a down load12:43
weshayissue12:43
weshayI scanned 5 ovb jobs currently running.. didn't see the issue12:43
ssbarneai seen the bug marked as fix released and I was wondering what fixed it, i suspect that only alembic!=1.0.1 would have fixed it.12:45
*** quiquell|lunch is now known as quiquell12:45
rlandyweshay: hello - we have introspection failures on master withe real bm related to clean wait ... https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans-6/undercloud/home/stack/overcloud_prep_images.log.txt.gz#_2018-10-17_23_49_2112:49
rlandyjuts like we do in:12:49
rlandyhttp://logs.rdoproject.org/02/611302/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/41e5dc2/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz12:49
rlandyweshay: in rhos-14, we disabled cleaning12:50
rlandyI would guess that is causing some issues12:50
mariosrlandy: 15:29 < weshay> marios|rover, I'm probably reading this wrong12:50
marios15:29 < weshay> https://logs.rdoproject.org/12/603612/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/3c14cd6/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-10-18_10_34_1712:50
marios15:29 < weshay> 2/4 intially failed12:50
marios15:29 < weshay> then it goes into a retry12:51
marios15:29 < panda> ssbarnea: straping12:51
marios15:29 < weshay> and 3/4/ and 4/4/ pass?12:51
marios15:30 < marios|rover> weshay: yeah it announces the pass damnit it couldn't find the trace from yesterday but pretty sure i saw exactly this12:51
marios15:30 -!- ykarel [~ykarel@106.78.198.12] has quit [Ping timeout: 246 seconds]12:51
marios15:30 < marios|rover> weshay: hmm not _exactly_ the same but almost12:51
marioshttp://logs.rdoproject.org/48/602248/9/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/98220fc/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz12:51
mariosrlandy: i saw it once yesterday and weshay saw it this morning12:51
rlandymarios: so you are saying that is not the real issue, it's another problem?12:52
rlandymarios; thing is  - I am pretty sure I disabled cleaning on bm12:53
mariosrlandy: not at all we don't know what the issue is just pointing out it was discussed _just_ before you joined12:53
rlandychecking12:53
rlandymarios; I have been chasing down node cleaning for a while12:53
rlandythis was always a problem on bm12:53
weshayrlandy, marios I think it needs to be looked at w/ wireshark12:56
weshayrascasoft, updated my patches12:56
weshaythink we need to land those12:56
rlandyweshay: you can disable cleaning to check12:56
rlandywe can test that with a few patches12:56
weshayrlandy, ya.. worth trying as well12:56
weshayI put up a patch to turn that off perm. and got negative feedback12:57
rlandyweshay: I chased this around a lot when we started fs001 on bm12:57
weshayk k12:57
* rlandy picks this up after meeting12:57
weshayeveryone is going to start having some fun w/ wireshark12:57
* weshay thinks12:57
weshayrlandy++12:57
hubbot1weshay: rlandy's karma is now 3112:57
weshayssbarnea, quiquell fyi https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=112:59
weshayno data12:59
mariosquiquell: can you check https://review.openstack.org/#/c/604768/5/playbooks/tripleo-ci/templates/toci_quickstart.sh.j2 when you get a a chance thanks for your comments i updated13:00
rfolcoping sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, rasca, chandankumar, ssbarnea13:01
rfolcocll13:01
rfolcocall13:01
weshayrfolco, which blue?13:02
rfolcohttps://etherpad.openstack.org/p/tripleo-ci-squad-meeting13:02
rfolcohttps://bluejeans.com/587845809713:02
weshaykopecmartin, ^13:02
rfolcokopecmartin, add your nick to pinglist at the top of etherpad13:03
rfolco:)13:03
kopecmartinrfolco, ok, thanks .. also maybe we should update the blue link in the calendar event13:03
rfolcoI thought it was updated13:04
*** quiquell has quit IRC13:05
*** quiquell has joined #oooq13:05
quiquellrfolco: Is the calendar link ok ?13:05
quiquellweshay, panda, rfolco: Do you have the link of the meeting ?13:07
rfolcohttps://etherpad.openstack.org/p/tripleo-ci-squad-meeting13:07
rfolcohttps://bluejeans.com/587845809713:07
*** jrist has quit IRC13:11
*** weshay is now known as weshay_meeting13:12
*** jrist has joined #oooq13:13
*** dalvarez has joined #oooq13:22
rlandymarios: btw: testing no undercloud cleaning https://review.openstack.org/#/c/61160213:25
rlandyykarel_: ^^13:25
ykarel_ack13:36
*** gkadam has quit IRC13:59
*** gkadam has joined #oooq13:59
mariosright marios|rover go away now!14:06
marios|rovermarios: :(14:06
matbumarios: :D14:07
matbumarios: you are going crazy man ... upgrade did that to you ..14:07
pandamatbu: get help14:10
*** apetrich has quit IRC14:11
*** hamzy has quit IRC14:15
matbupanda: im going to the same situation man ...... :D14:18
*** apetrich has joined #oooq14:20
*** saneax has quit IRC14:20
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (4 more messages)14:23
*** ccamacho has joined #oooq14:30
*** gkadam_ has joined #oooq14:33
*** gkadam has quit IRC14:35
*** holser_ has quit IRC14:40
*** holser_ has joined #oooq14:41
*** hamzy has joined #oooq14:42
mariosrfolco: list of services for standalone role https://github.com/openstack/tripleo-heat-templates/blob/master/roles/Standalone.yaml14:46
*** ykarel_ is now known as ykarel|away14:49
rlandymarios: prep-images works just fine w/o cleaning15:05
weshay_meetinghttps://review.openstack.org/#/c/606046/15:13
*** skramaja_ has quit IRC15:28
*** ccamacho has quit IRC15:36
*** agopi has quit IRC15:38
*** gkadam_ has quit IRC15:38
*** ccamacho has joined #oooq15:38
*** jfrancoa has quit IRC15:40
*** hamzy has quit IRC15:42
*** hamzy has joined #oooq15:43
*** ccamacho has quit IRC15:47
*** dtantsur is now known as dtantsur|afk16:00
*** ykarel|away has quit IRC16:02
*** bogdando has quit IRC16:10
*** sshnaidm_ is now known as sshnaidm16:14
*** holser_ has quit IRC16:16
mariosmarios|rover: srsly man, begone16:18
*** marios|rover has quit IRC16:18
rlandymarios: arxcruz: want to meet tomorrow morning re: reproducer work?16:18
rlandymarios: arxcruz: we have had suggestions to rework the reproducer and I would like others' opinions16:19
rlandymarios: ^^ unrelated ... https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans-8/undercloud/home/stack/overcloud_deploy.log.txt.gz16:19
arxcruzrlandy: yes maam16:19
rlandy^^ I can get as far as overcloud deploy w/o cleaning on bm16:19
rlandychecking if that;s a legit failure16:19
rlandyweshay_meeting: ^^ fyi16:20
mariosrlandy: yeah lets talk tomorrow /me getting outa here16:21
rlandymarios: ack - have  a good night16:22
mariosthanks have a good day then night bai16:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (4 more messages)16:23
*** agopi has joined #oooq16:26
quiquellrlandy: The ideas at the repo car right now is running directly the run-v3.yaml playbook filtering with ansible tag16:26
quiquellrlandy: And generating zuul vars and inventory in the repro bash script16:26
rlandyquiquell: "repo car"?16:27
quiquellrlandy: Don't know how can we filter the jobs vars correctly without something that filters it16:27
quiquellrlandy: the reproduer epic I mean16:27
rlandyquiquell: right - so we need to run as closely as possible to CI16:27
rlandyso that would mean running run-v316:27
rlandyor at least everything that kicks16:28
quiquellrlandy: The proto reachs to run the individual playbooks more or less16:28
rlandyso we need to look at per and post playbooks16:28
quiquellrlandy: we can even entertain the idea or runnin pre and post to have collect logs16:28
rlandyand the order in which they are kicked16:28
rlandysure16:28
quiquellrlandy: Filtering by "reproduce" ansible tag16:28
rlandyquiquell: we have had some suggestions to do away with bash running16:28
rlandythere are lots of ideas on the table16:29
rlandyand I kind of wanted someone else to taje the lead on design here16:29
rlandyso we got some fresh ideas16:29
rlandyquiquell: I will review your current patches16:29
quiquellrlandy: Sure don't know if refacting and removing old workflow are the same task16:29
rlandyfeel free join our design session tomorrow16:30
quiquellrlandy: I was putting stuff to really remove old workflow it's getting a pain16:30
rlandyquiquell: I think everything is on the table right now16:30
quiquellrlandy: Yep16:30
rlandyquiquell: I'll set up a meeting for tomorrow and invite you16:31
rlandy^^ sorry sshnaidm about the day16:31
rlandywould have appreciated your take16:31
quiquellrlandy: ok I will try to hand over worries and findings16:31
rlandypls do16:31
quiquellrlandy: thanks, dropping now, see ya'16:31
rlandybye16:31
*** quiquell is now known as quiquell|off16:32
rfolcosshnaidm, can you be ruck first ?16:43
rfolcomarios, still on? can you please just update the promotion status on https://review.rdoproject.org/etherpad/p/ruckrover-sprint20.1 ~ L18 ?16:53
rfolcomarios, ignore. Updated16:59
*** dsneddon has quit IRC17:29
*** panda is now known as panda|off17:30
*** dsneddon has joined #oooq17:30
*** rfolco is now known as rfolco|rucker17:31
ssbarnearlandy rfolco|rucker weshay_meeting: https://review.openstack.org/#/c/611636/ adds f28 job and zuul run it as expected, so we should merge it? https://review.openstack.org/#/c/611636/17:38
rfolco|ruckerssbarnea, where is the zuul +1 ?17:39
ssbarneadid finish yet, but the new job did run and failed as expected.17:40
ssbarneain ~50min we should also get the official reply17:40
rfolco|ruckerssbarnea, imo it should meet the dod criteria to merge it.. and get some reviews on it17:41
rfolco|ruckerbut yes I understand the point to merge it even if not perfect, the job is in place17:42
*** hamzy has quit IRC17:44
*** hamzy has joined #oooq17:45
weshay_meetingssbarnea, /me looks17:52
weshay_meetingssbarnea, ya.. that is good progress.. sounds like rfolco|rucker thinks the upstream nodeset will work fine.. so I would adjust that and merge as long as it's nv17:55
rfolco|rucker!gatestatus18:09
openstackrfolco|rucker: Error: "gatestatus" is not a valid command.18:09
*** apetrich has quit IRC18:10
ssbarneaweshay_meeting or rfolco|rucker : now we have the chance to +W this one https://review.openstack.org/#/c/610018/1018:13
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024,  (4 more messages)18:23
*** apetrich has joined #oooq18:24
rlandyrfolco|rucker:18:30
rlandy%gatestatus18:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024,  (4 more messages)18:30
rlandyrfolco|rucker: ^^ see command18:31
rfolco|ruckerrlandy, thanks18:31
*** weshay_meeting is now known as weshay18:33
* weshay looks18:33
weshayssbarnea, done18:33
weshayrlandy, ssbarnea https://review.openstack.org/#/c/610072/18:34
zulwhats the recommended hardware specs for trippleo-quickstart?18:44
rlandyhttps://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans-8/undercloud/home/stack/overcloud_deploy.log.txt.gz#_2018-10-18_15_55_4818:47
rlandyweshay: seen that one before?18:47
rlandynvm - I see it here ...18:48
rlandyhttp://logs.rdoproject.org/65/610365/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/547fa9a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz18:48
rlandyweshay: ok - so bm gets us as far as overcloud_deploy18:49
rlandyI think that error is legit18:49
*** panda|off has quit IRC18:59
*** panda has joined #oooq19:00
*** hamzy has quit IRC19:07
*** hamzy has joined #oooq19:08
weshayrlandy, /me looks19:19
weshayrlandy, master?19:19
weshayrlandy, https://bugs.launchpad.net/tripleo/+bug/179548819:21
openstackLaunchpad bug 1795488 in tripleo "Ansible "dry-run" fails with TASK [set is_bootstrap_node fact] - FAILED! => {"msg": "The task includes an option with an undefined variable." [Medium,In progress] - Assigned to James Slagle (james-slagle)19:21
weshayrlandy, hrm.. can you check to see if you have this patch https://review.openstack.org/#/c/605430/19:21
rlandylooking19:23
rlandymarios reverted it19:23
weshayrlandy++ ok your environments are sane19:24
hubbot1weshay: rlandy's karma is now 3219:24
weshaythe product is insane19:24
weshayrlandy, so is that ovb job upshift?19:25
rlandyweshay: no - that is bm but making progress on upshift19:25
rlandyI can stack create manually19:25
weshaycool19:25
*** dsneddon has quit IRC19:25
rlandyfiguring out why my job doesn't19:25
*** dsneddon has joined #oooq19:26
*** hamzy has quit IRC19:41
weshayrlandy, in case we need this just fyi https://review.openstack.org/#/c/607655/19:42
weshaysshnaidm, ^19:42
*** apetrich has quit IRC19:44
rlandylooking19:47
weshayrfolco|rucker, fyi https://bugs.launchpad.net/tripleo/+bug/179866419:47
openstackLaunchpad bug 1798664 in tripleo "ara pip install failing across various branches" [Critical,Triaged]19:47
weshayrfolco|rucker, so this is partially a sprint task.. and you'll need to weigh in on the qs dep install card19:47
weshayara when left w/o pinning deps makes the overcloud install fail19:48
weshayara w/ deps makes some of the branches fail19:48
weshayafaict it looks like we need to specify per branch requirements for ara so that ara reqs the correct version of ansible19:48
rlandyweshay: well julia has a -1 on that19:48
weshayrlandy, yes.. we'll see what happens there19:49
rlandypersonally, I added https://review.openstack.org/#/c/611602/19:49
weshayjust wanted to bring it to your attention19:49
rlandyweshay: I'll comment with that linked19:49
rlandybut the results are nothing to see19:49
rlandybecause of the ara failure19:49
rlandywe get nowhere19:49
rlandyI can prove it gets by introspection in bm19:49
rlandyif that helps19:49
weshayrlandy, that helps to prove it's rdo-cloud19:50
weshayI think.. so ya if you can put that in the bug..19:50
rlandywell, no exactly but ok19:50
weshay?19:50
weshaywhat don't I understand19:50
rlandyweshay; pls dig me out of this j2 hole ...19:50
weshay?19:51
rlandyhttps://review.openstack.org/#/c/611674/3/roles/ovb-manage-stack/templates/clouds.yaml.j219:51
weshaysends down a yaml ladder?19:51
rlandysomehow it's joining all the values together19:51
rlandywith no line breaks19:51
rlandythe if statements19:51
weshayk /me looking19:52
rlandypls see pvt19:52
rlandythis is a stupid formatting mistake19:52
rlandybut I don;t know what19:52
rfolco|ruckerweshay, ok, looking at the issue now to understand what versions apply to each branch19:54
weshayrfolco|rucker, so others who have context.. alex ( mwhahahhaha ) and yatin19:55
weshayrfolco|rucker, and sagi19:55
weshaybut he is ooo tomorrow19:55
rfolco|ruckeryep, thats why I am rucker19:56
rfolco|ruckerrfocker19:56
weshayrfolco|rucker, heh.. so it's probably worth just turning off ara pip installs every where in the code atm19:56
weshayLOLZ19:56
weshayrfolco|rucker, until we get the deps right and you raise this issue to the sprint card that installs deps19:56
rfolco|ruckerweshay, if we just turn it off, we don't have ara ?19:57
weshayara may be causing more problems than it's resolving19:57
rfolco|ruckeroh ok19:57
weshayrfolco|rucker, ya.. that is fine for now19:57
weshayrather have a working job19:57
rfolco|ruckerthats what I wanted to hear19:57
rfolco|ruckersign here please ______________19:57
weshaywell.. I'm just one guy.. but ya.. I think it makes sense19:57
weshayara was going to be very handy in diagnosing overcloud deployment problems19:58
rfolco|ruckerk thanks for the background info19:58
weshaywe already had to turn that19:58
weshayoff19:58
weshayand now it's blowing up the undercloud where it doesn19:58
weshay't even have as much value19:58
weshayso.. my advice is to turn it off, put in patches where it installs across all our branches and we can specify which ara version by the branch19:59
weshayvia the releae file in config/releases/19:59
weshaythat's my 2 cents19:59
weshayrfolco|rucker, you can cut and paste this conversation in the bug if you like20:00
rfolco|ruckerweshay, sounds good20:00
weshaythanks rfolco|rucker20:00
*** ssbarnea_ has joined #oooq20:03
*** vinaykns has joined #oooq20:06
*** hamzy has joined #oooq20:10
rfolco|ruckerweshay, sounds a dup of https://bugs.launchpad.net/tripleo/+bug/179262220:15
openstackLaunchpad bug 1792622 in tripleo "Ara fails to install in some of tripleo jobs with ImportError: 'module' object has no attribute 'check_specifier'" [Medium,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)20:15
weshayrfolco|rucker, ah.. hopefully20:16
rfolco|ruckerI'm proposing a patch to ignore errors on ara for now20:16
weshaywas looking20:16
weshayrfolco|rucker, hrm..20:16
weshaythat's an old bug20:16
weshayI don't care, use either20:16
rfolco|rucker  ignore_errors: True20:17
rfolco|ruckeron deps20:17
rfolco|ruckerbut nothing on install ara20:17
rfolco|ruckerthats why it fails20:17
rfolco|ruckeror run install ara when deps rc = 020:18
weshayrfolco|rucker, so which line of code20:19
weshaypaste it20:19
rfolco|ruckersure sec20:19
rfolco|ruckerhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-setup/tasks/ansible_cfg.yml#L420:20
rfolco|ruckerweshay, ^20:20
rfolco|ruckerif deps play fails, it will ignore and try to install ara... this is wrong imo, we should ignore both or don't even try to install ara if deps failed20:21
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024,  (4 more messages)20:23
*** agopi has quit IRC20:24
*** agopi has joined #oooq20:24
weshayrfolco|rucker, so I'm suggesting we add a switch on the install of ara20:27
weshayundercloud_ara|bool20:27
rfolco|ruckerok but ara install should not block / fail our job, agree ?20:28
rfolco|ruckerI can combine both: switch on/off plus error handling20:29
rfolco|ruckerlet me propose something for review20:29
weshayrlandy, /me getting20:31
rlandyI got all except one working20:31
weshayjoining blue rlandy20:31
weshayrfolco|rucker, looks like something is wrong w/ tripleo-ci-centos-7-containerized-undercloud-upgrades20:45
weshayI think these are known issues20:46
rfolco|ruckerhmm I'll have to check when I come back later tonight, gotta uber kids to soccer20:47
weshayrfolco|rucker, np.. thanks20:51
*** weshay is now known as weshay_pto20:52
*** panda has quit IRC21:08
*** panda has joined #oooq21:12
*** trown is now known as trown|outtypewww21:13
*** agopi_ has joined #oooq21:26
* rlandy goes to the dentist21:27
rlandyback later21:27
*** rlandy is now known as rlandy|bbl21:27
*** agopi has quit IRC21:28
*** agopi_ is now known as agopi21:47
*** ssbarnea_ has quit IRC21:53
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci- (4 more messages)22:23
*** vinaykns has quit IRC22:37
*** tosky has quit IRC23:02
*** agopi is now known as agopi|brb23:08
*** agopi|brb has quit IRC23:13
*** rlandy|bbl is now known as rlandy23:19
*** agopi|brb has joined #oooq23:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!