Friday, 2019-02-15

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario002-standalone, tripleo-ci-centos-7-ovb- (5 more messages)00:56
*** zul has quit IRC01:37
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario002-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (5 more messages)02:57
*** apetrich has quit IRC03:15
*** skramaja has joined #oooq03:36
*** skramaja_ has joined #oooq03:51
*** skramaja has quit IRC03:51
*** ykarel has joined #oooq04:04
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario002-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (4 more messages)04:57
*** ratailor has joined #oooq05:05
*** vinaykns has quit IRC05:08
*** rnoriega has quit IRC05:15
*** rnoriega has joined #oooq05:16
*** ykarel has quit IRC05:30
*** ykarel has joined #oooq05:44
*** rlandy|bbl has quit IRC06:04
*** saneax has joined #oooq06:51
*** quiquell|off is now known as quiquell|rover06:54
*** saneax has quit IRC06:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci- (4 more messages)06:57
*** saneax has joined #oooq06:59
*** jtomasek has joined #oooq07:00
*** jfrancoa has joined #oooq07:12
*** kopecmartin|off is now known as kopecmartin07:15
*** chkumar|pto is now known as chkumar|ruck07:29
chkumar|ruckquiquell|rover: \o/07:29
*** apetrich has joined #oooq07:30
quiquell|roverchkumar|ruck: You are alive !!!07:30
chkumar|ruckquiquell|rover: yes, going through the email so dpdk  break all ci07:30
quiquell|roverchkumar|ruck: rocky still broken07:30
quiquell|roveras todays news07:30
quiquell|roverso we are using a OVS without DPDK now07:31
quiquell|roverat least at master but rocky is not getting the latest OVS07:31
quiquell|rovera prart from thara gnocci github still not merged07:31
chkumar|ruckquiquell|rover: Ok, I ma pinging silhet to get it merged07:32
chkumar|ruckand fixing py3 review07:32
quiquell|roverchkumar|ruck: ack, also more people is being affected by build-test-package not able to build github projects07:32
quiquell|roverchkumar|ruck: Can I take that task ?07:32
chkumar|ruckquiquell|rover: sure07:32
quiquell|roverchkumar|ruck: I am preparing some molecule test for it with another issue of build-test-packages ?07:33
chkumar|ruckquiquell|rover: we need find the host of the package then I can think we can able to build it07:33
chkumar|ruckquiquell|rover: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/library/zuul_deps.py#L5607:33
quiquell|roverchkumar|ruck: will look at it today07:34
chkumar|ruckquiquell|rover: it is always overrides with git.openstack.org07:34
quiquell|roverchkumar|ruck: I am preparing stuff like this https://review.openstack.org/#/c/636588/07:34
chkumar|ruckwe need to somehow fetch project canonical name from zuul inteventoy07:34
quiquell|roverchkumar|ruck: so we can put one scenario for all this test cases07:34
quiquell|roverLet me try that at you rpatch07:34
quiquell|roverso we can easily exercise that07:35
quiquell|roverand make the fix07:35
chkumar|ruckquiquell|rover: cool, :-)07:35
quiquell|roverchkumar|ruck: have to drop for a few07:36
*** quiquell|rover is now known as quique|rover|brb07:36
chkumar|ruckok07:55
*** ykarel is now known as ykarel|lunch07:59
*** fmount has quit IRC08:09
*** fmount has joined #oooq08:10
*** quique|rover|brb is now known as quiquell|rover08:17
*** amoralej|off is now known as amoralej08:19
chkumar|ruckzbr: Hello, around?08:36
zbrchkumar|ruck: hi! yes.08:36
chkumar|ruckzbr: I need some help here http://logs.openstack.org/04/634904/17/check/openstack-tox-linters/7b1a8d4/job-output.txt.gz#_2019-02-15_08_32_27_81988808:37
chkumar|ruckzbr: does it mean at each place where I used | should be replaced with set -o pipefail?08:37
*** tosky has joined #oooq08:38
chkumar|ruckzbr: related review https://review.openstack.org/63490408:38
zbrchkumar|ruck: yes, new rule in ansible-lint 4.1.0 -- smart one.08:39
zbrchkumar|ruck: just add it at the top of the shell command.08:39
zbrchkumar|ruck: btw, do we still have the ci-scripts/ansible-lint.sh file? i had the impression that got rid of it long time ago.08:40
zbrchkumar|ruck: if you do not have a fix, i can raise one in few minutes and show it to you, very easy,08:40
chkumar|ruckzbr: show me,08:41
chkumar|ruckzbr: will i remove this file https://github.com/openstack/ansible-role-tripleo-modify-image/blob/master/ci-scripts/ansible-lint.sh ?08:42
chkumar|ruckzbr: so this one https://review.openstack.org/#/c/634904/17/files/dev_install.sh@1 is ok?08:43
zbrahh, we didnt introduce the pre-commit to that repo yet. this may take an hour probably.08:43
chkumar|ruckzbr: let me do it for thsi role08:43
chkumar|ruckzbr: /me need to learn08:43
zbrchkumar|ruck: good idea: start by copying adding .pre-commi* file from one of other repos and the linters env.08:45
chkumar|ruckzbr: on it08:46
*** ykarel|lunch is now known as ykarel08:48
zbrchkumar|ruck: i would recommend tripleo-quickstart to be used as reference08:48
chkumar|ruckzbr: copying fromt hat one08:49
*** rascasoft has joined #oooq08:50
zbri was planning to write an article as there are several files involved but don't want for me to do that, just ask.08:50
zbralso need to check: all files form repo root for linting related config.08:51
mariospanda quiquell|rover when you have time please re-add your votes merge conflict https://review.openstack.org/#/c/635222/08:54
quiquell|rovermarios: you got it08:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-quickstart-extras-gate- (3 more messages)08:57
*** jpena|off is now known as jpena08:57
*** holser_ has joined #oooq08:58
*** bogdando has joined #oooq09:02
mariosquiquell|rover: thanks :)09:05
quiquell|rovernp09:09
*** ccamacho has joined #oooq09:10
pandamarios: when gertty decides to collaborate, you'll have mine too.09:26
*** derekh has joined #oooq09:28
mariospanda: thanks09:29
*** chem has joined #oooq09:38
zbrpanda: can please get a wf on https://review.openstack.org/#/c/636944/  ?09:40
*** holser_ has quit IRC09:41
pandazbr: the best I can do is a wtf09:41
pandazbr: where else is this repo used ? because we have a limited set of repos in which we are allowed to +209:44
pandazbr: marios has more superpowers09:44
*** holser_ has joined #oooq09:45
*** ccamacho has quit IRC09:48
*** ccamacho has joined #oooq09:48
*** saneax has quit IRC09:59
*** saneax has joined #oooq09:59
*** saneax has quit IRC10:00
*** saneax has joined #oooq10:01
* marios hids10:02
marioszbr: panda ?10:02
marios(looking)10:02
pandamarios: you already +2 there10:02
marioslets live dangerously10:03
* marios runs for the hills 10:03
*** saneax has quit IRC10:03
*** saneax has joined #oooq10:04
mariospanda: i just read the above. i am sure you'd be fine to +2 there :/10:04
mariosi mean +A panda anyway10:04
*** saneax has quit IRC10:04
pandamarios: if you say so, I'll start +2 on pythonclient, tripleo-common, dib,  neutron and even a bit on kernel modules. :)10:05
*** saneax has joined #oooq10:05
*** saneax has quit IRC10:06
mariosi need to asign global shortcut for string "/logs/undercloud/home/zuul/"10:06
mariospanda: yah totally fine just tell them marios sent you10:06
* marios runs faster for the hills10:06
*** dtantsur|afk is now known as dtantsur10:09
quiquell|rovermarios: do you have hills there ?10:10
quiquell|rovermarios, panda: something to merge from sprint ?10:10
pandaquiquell|rover:  https://review.openstack.org/636355  https://review.openstack.org/634723 https://review.openstack.org/63472510:11
mariosquiquell|rover: yeah i can see one outside my window10:12
mariosquiquell|rover: where do you think i live man its not the sahara here10:13
zbrpanda: to answer your question, tripleo-repos repo is used by jobs/roles defined in tripleo-ci repo.10:19
quiquell|roverpanda: you are going to kill me10:20
quiquell|roverpanda: We don't want to override "ironic_overcloud: true" ?10:20
quiquell|roverpanda: and add it to fs052 with false ?10:20
quiquell|roverpanda: so we don't repeat all this ?10:21
quiquell|roverpanda: if you have already discussed it at team I will vote with +210:21
zbrquiquell|rover: one of the benefits of working remotely, panda would have to go an extra mile to reachl you. ;)10:21
pandaquiquell|rover: you mean adding "ironic_overcloud:" to featureset overrides ?10:22
quiquell|roverpanda: yep10:22
quiquell|roverpanda: and set it to false at fs05210:22
quiquell|roverpanda: I know you are not fan of feturset override10:23
quiquell|roverpanda: but we are designing standalone with it in mind10:23
pandaso when I said that was clear that overriding featureset was going to become the preferred way to pass variable, I wasn't wrong :)10:23
quiquell|roverpanda: and looks like we just need a flag10:23
quiquell|roverweshay|ruck: at least we can add it as false to the fs052 so the variable is there you know the knobs10:24
quiquell|roverpanda: but if using ironic woul dinvolve more stuff we keep it like this10:24
pandaquiquell|rover: we still don't have a clear decision on this, to me overriding something taht can be solved by adding a new featureset is abuse of the overriding possibility. I don't like it, and I would prefer to avoid it. So unless you're strongly for -2 and would require a discussion, I would leave it like this. I'm producing a doc on how we pass configuration variable to jobs, so we have a common10:28
pandaunderstanding on the process. And when that is merged, I already have a proposal to change it.10:28
quiquell|roverpanda: ack, will +2 let me just compare that we are not missing anything from fs05210:31
pandaquiquell|rover: the draf of the doc is here  https://review.openstack.org/636600, I already have to change it following feedback from ronelle and wes, but if you have time and want to contribute, I need everybody to confirm this is how we handle configuration variables to the jobs.10:34
*** skramaja_ has quit IRC10:35
quiquell|roverpanda: will do10:41
quiquell|roverpanda: fix layout here https://review.openstack.org/#/c/63472310:43
pandaquiquell|rover: lol10:45
pandaquiquell|rover: I completely forgot that10:45
pandaquiquell|rover: thanks10:45
pandaI have to fix layout on both10:46
quiquell|roverpanda: ack10:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-containerized- (2 more messages)10:57
*** zul has joined #oooq11:27
*** rfolco|off is now known as rfolco11:33
*** saneax has joined #oooq11:50
*** tosky has quit IRC12:30
chkumar|ruckzbr: now twine check is failing http://logs.openstack.org/50/637150/4/check/test-release-openstack/9e2b6bd/job-output.txt.gz#_2019-02-15_09_42_57_92644212:31
chkumar|ruckfixing it12:32
*** jpena is now known as jpena|lunch12:32
*** tosky has joined #oooq12:36
*** ratailor has quit IRC12:43
mariosanother round of votes please https://review.rdoproject.org/r/#/c/18729/12:51
mariosits the new periodic which inherits from the upstream build containers. it shouldn't break anything, is a new job. not wired into any dependencies yet. we just want to see it running12:52
mariospanda: rfolco weshay|ruck sshnaidm|off ** thanks folks ^^^12:52
mariosquiquell|rover: is that tripleo-ci-centos-7-containers-multinode-pike TIMED_OUT in 3h 22m 47s   a known thing? https://review.openstack.org/#/c/635222/2212:53
*** ccamacho has quit IRC12:55
*** ccamacho has joined #oooq12:55
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-containerized- (2 more messages)12:57
quiquell|rovermarios: let me check didn't take a look13:02
*** trown|outtypewww is now known as trown13:07
quiquell|rovermarios: noop review at master is wroking fine at -pike13:11
mariosquiquell|rover: ack is a timeout anyway not fail just wanted to know if it is known one of one off rechecked there lets se13:13
chkumar|ruckquiquell|rover: Do we want to add multinode scenario job against mistral project?13:13
chkumar|ruckquiquell|rover: or we wait till it finishes on rdo testproject side?13:13
weshay|ruckpanda, enough w/ the so when I said that was clear that overriding featureset was going to become the preferred way to pass variable, I wasn't wrong :)13:13
quiquell|roverchkumar|ruck: we wait, they are not going to want a multinode there13:14
quiquell|roverchkumar|ruck: it's undercloud or standalone what is easy to sell13:15
chkumar|ruckquiquell|rover: scenario3 standalone is already there13:15
quiquell|roverchkumar|ruck: this does not exercise the issue13:15
quiquell|roverchkumar|ruck: standalone just dump a playbook and execute it13:16
weshay|ruckchkumar|ruck quiquell|rover can you guys show me some examples of dlrn failing13:16
quiquell|roverchkumar|ruck: there is no mistral involve13:16
* weshay|ruck looks for bugs13:16
quiquell|roverweshay|ruck: what dlrn failing ?13:16
quiquell|roverweshay|ruck: like the api ? or the build of RPMs ?13:16
weshay|ruck<quiquell|rover>  chkumar|ruck: ack, also more people is being affected by build-test-package not able to build github projects13:17
weshay|ruck00:32 <quiquell|rover>  chkumar|ruck: Can I take that task ?13:17
weshay|ruck00:32 <chkumar|ruck>  quiquell|rover: sure13:17
weshay|ruck00:33 <quiquell|rover>  chkumar|ruck: I am preparing some molecule test for it with another issue of build-test-packages ?13:17
chkumar|ruckweshay|ruck: it is related to testing gnocchi13:17
weshay|ruckchkumar|ruck the mistral project should be gated by a tripleo job13:18
* weshay|ruck looks13:18
quiquell|roverweshay|ruck: https://review.openstack.org/#/c/636511/13:18
quiquell|roverweshay|ruck: In this case tht patch is not applied but also the job does not fail13:19
quiquell|roverweshay|ruck: panda  and sshnaidm|off agree with make jobs fail if some of the reviews cannot be applied13:19
weshay|ruckquiquell|rover which job?13:19
chkumar|ruckweshay|ruck: testing here https://review.rdoproject.org/r/#/c/13943/7913:20
weshay|ruckchkumar|ruck quiquell|rover which job is failing to apply the tht patch.. all of them I suppose? https://review.openstack.org/#/c/636511/13:20
quiquell|roverweshay|ruck: yep all of them13:20
quiquell|roverweshay|ruck: so we need this https://review.openstack.org/#/c/636114/13:21
weshay|ruckhrm.. rpm builds http://logs.openstack.org/11/636511/3/check/tripleo-ci-centos-7-standalone/02536a1/logs/delorean_logs/6e/24/6e249ccb6c24c85cb42544480f1fc36a9f6a8c6a_dev/build.log.txt.gz13:21
quiquell|roverweshay|ruck: to make it apply github and tht13:21
quiquell|roverweshay|ruck: but also this to make job fail if not https://review.openstack.org/#/c/636588/13:22
quiquell|roverweshay|ruck: the rpm is build13:22
quiquell|roverweshay|ruck: but repo_build.skipped is true13:22
quiquell|roverweshay|ruck: it get overwritten by the skiped one13:22
quiquell|roverweshay|ruck: so it does not do the install repo13:22
quiquell|roverThey don't get injected13:22
weshay|rucksorry?13:23
weshay|ruckI see it installed?13:23
weshay|ruckFeb 14 20:13:18 Installed: openstack-tripleo-heat-templates-10.3.1-0.20190214200355.fc9efc1.el7.noarch13:23
weshay|ruckquiquell|rover chkumar|ruck can you guys jump on the line w/ me13:23
quiquell|roversure let's verify13:23
weshay|ruckhelp me understand what you are seeing but I'm not on my first pass13:23
weshay|ruckin my blue13:24
quiquell|roverweshay|ruck: he has remove Depends-On at github13:24
quiquell|roverweshay|ruck: that's why now is working (we say it to Herald)13:24
*** tosky has quit IRC13:25
*** tosky has joined #oooq13:26
chkumar|ruckweshay|ruck: quiquell|rover I was testing this http://logs.rdoproject.org/43/13943/78/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/8f64dc8/logs/ara_oooq/result/07d70675-b82b-4e6c-97e5-bf5cc87785df/13:27
chkumar|ruckweshay|ruck: quiquell|rover just need one change to grab gnocchi url at the place of host13:27
zbr@oooq: how do I talk from zuul job to gerrit? it seems to be asking about my user (zuul implicit user didn't had the right key)13:27
chkumar|ruckweshay|ruck: the changes of gnocchi is already clonned by zuul13:27
zbri only need it for cherry picking some changes.13:28
chkumar|ruckweshay|ruck: the change is https://review.rdoproject.org/r/#/c/13943/7813:28
chkumar|ruckif we get the correct host, we can test gnocchi pr in the rdo ci13:28
chkumar|ruckhost == github.com (need to try out zuul.projects.canonical name) for the same to grab it13:29
weshay|ruckDepends-On: https://github.com/voxpupuli/puppet-rabbitmq/pull/77713:29
weshay|ruckNOT SUPPORTED atm13:29
weshay|ruckchkumar|ruck get on my line now please13:30
chkumar|ruckweshay|ruck: bj link13:31
weshay|ruckhttps://bluejeans.com/u/whayutin/13:31
pandaI'm seeing double13:33
chkumar|ruckweshay|ruck: https://review.rdoproject.org/r/#/c/13943/7813:35
chkumar|ruckweshay|ruck: https://review.rdoproject.org/r/#/c/18791/13:36
*** holser_ has quit IRC13:37
*** holser_ has joined #oooq13:37
chkumar|ruckweshay|ruck: for openstack https://review.openstack.org/#/c/636093/13:38
chkumar|ruckweshay|ruck: https://review.openstack.org/#/c/636114/13:41
*** jpena|lunch is now known as jpena13:41
*** rlandy has joined #oooq13:42
rlandymarios: hey - standalone update13:48
mariosrlandy: o/13:49
*** dtrainor has quit IRC13:49
rlandymarios: so - wrt emilien's email, I guess pacemaker stays13:50
*** dtrainor has joined #oooq13:50
rlandymarios: I've been doings some testing on my local zuul13:50
mariosrlandy: cool anything good13:50
rlandymarios: _ put back all the pacemaker stuff I could recognize ...13:51
*** dtrainor has quit IRC13:51
*** dtrainor has joined #oooq13:51
rlandyhttps://review.openstack.org/#/c/635208/12/ci/environments/scenario009-standalone.yaml13:51
rlandymarios: 2019-02-14 23:48:16 |     "Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]",13:52
rlandy2019-02-14 23:48:16 |     "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]",13:52
mariosrlandy: o_O13:52
rlandyafaict, the real complaint is here ...13:52
rlandy2019-02-14 23:48:15 |         "stderr: Error: unable to find resource 'redis-bundle'",13:52
rlandy2019-02-14 23:48:15 |         "stderr: Error: unable to find resource 'redis-bundle'",13:53
rlandy is there though13:53
mariosrlandy: yeah i was digging today about redis for example we have this https://github.com/openstack/tripleo-heat-templates/blob/1c46d1850a8de89daeecd96f2f5288336e3778f8/environments/standalone/standalone-tripleo.yaml#L61-L6213:53
mariosrlandy: in my case i don't have pacemaker so thought i needed to override that tried with scen1 but fails like http://logs.openstack.org/63/636563/4/check/tripleo-ci-centos-7-scenario001-standalone/e38186e/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-02-15_13_06_4413:54
rlandymarios: my other thought was to get rid of all the pacemaker stuff and try run with podman13:54
weshay|ruckhttps://github.com/voxpupuli/puppet-rabbitmq/pull/77713:55
rlandymarios: hmmm ... so I understand your error  because of https://github.com/openstack/tripleo-heat-templates/blob/1c46d1850a8de89daeecd96f2f5288336e3778f8/environments/standalone/standalone-tripleo.yaml#L61-L6213:55
mariosrlandy: yeah i'm still trying to get the non pacemaker working ( https://review.openstack.org/#/c/636563/ ) and as discussed there we can also make it podman on the job definitions https://github.com/openstack-infra/tripleo-ci/blob/1d890565feeeea6ce637cf0384da822926480f07/zuul.d/standalone-jobs.yaml#L45113:55
weshay|ruckhttps://review.openstack.org/#/c/636511/13:55
rlandykimd of confused about mine though ...13:55
mariosweshay|ruck: we talking in 5 or you wana push?13:55
rlandymarios: since I should have environments/standalone/standalone-tripleo.yaml, and the two lines you linked, honestly, I am not sure what other redis bundle element I a mmissing13:57
rlandyso I guess my next run will be no pacemaker at all and podman13:57
mariosrlandy: i think so, at least that hasn't been tried yet13:57
mariosrlandy: i'm also trying to fix redis still and that https://github.com/openstack-infra/tripleo-ci/blob/1d890565feeeea6ce637cf0384da822926480f07/zuul.d/standalone-jobs.yaml#L45113:57
mariosdidnt work ^13:57
mariosrlandy: ^13:58
chkumar|ruckweshay|ruck: https://trunk.rdoproject.org/centos7-master/report.html13:58
mariosrlandy: oops wrong paste13:58
mariosrlandy: i mean this didnt work https://review.openstack.org/#/c/636563/4/ci/environments/scenario001-standalone.yaml@2613:58
*** agopi_ is now known as agopi13:58
rlandymarios: ^^ I have been following that to see what to put back in13:58
rlandybut really the maultinode scenario never had it13:59
rlandyhttps://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario009-multinode.yaml13:59
chkumar|ruckweshay|ruck: https://github.com/voxpupuli/puppet-rabbitmq/commit/1ccddad57e4d53bb3861754a1712b78f3a0fb13f14:00
rlandymarios: I think though you may have hit on the problem ... https://github.com/openstack/tripleo-heat-templates/blob/1c46d1850a8de89daeecd96f2f5288336e3778f8/environments/standalone/standalone-tripleo.yaml#L61-L6214:01
rlandy^^ maybe I have to go through this and purge it of anything that could be pacemaker14:01
mariosrlandy: yeah there is no neutron in the standalone for the ip address management14:01
mariosrlandy: which is why they use that deployed neutron port14:02
rlandyOS::TripleO::Services::Redis: OS::Heat::None14:02
rfolcowhat's the equivalent to http://mirror.centos.org for fedora ?14:02
rfolcoDEFAULT_CENTOS_MIRROR I mean14:03
rlandy OS::TripleO::Services::HAproxy: OS::Heat::None14:03
rfolcopanda, you know ^?14:03
mariosrlandy: right14:04
pandarfolco: https://mirrors.fedoraproject.org/ I think14:04
mariosrlandy: and we are enabling these. with pacemaker at least it is managed OK. without they have problems (like haproxy/redis and mysql had issues on my review)14:04
rfolcopanda, thx14:04
rlandymarios; what does OS::Heat::None mean/do?14:05
mariosrlandy: it disables that resource14:05
mariosrlandy: none14:05
rlandymarios: ok - so then the standalone file does the right thing  - maybe I need  OS::TripleO::Services::Pacemaker: OS::Heat::None14:06
mariosrlandy: http://logs.openstack.org/98/604298/231/check/tripleo-ci-centos-7-standalone/6ae7be6/logs/undercloud/var/log/containers/ see default standalone14:06
mariosrlandy: well or just remove it from your environment. it defaults to none in the resource registry14:06
chkumar|ruckweshay|ruck:14:06
chkumar|ruckhttps://github.com/redhat-openstack/rdoinfo/blob/c5a7d8e61e675b2d7b26686056f84e0f117fbc2e/tags/stein.yml#L75014:06
mariosrlandy: https://github.com/openstack/tripleo-heat-templates/blob/1c46d1850a8de89daeecd96f2f5288336e3778f8/overcloud-resource-registry-puppet.j2.yaml#L17214:07
*** vinaykns has joined #oooq14:07
rlandymarios: so maybe I need that explicitly   OS::TripleO::Services::Pacemaker: OS::Heat::None14:08
rlandyOS::TripleO::Services::PacemakerRemote: OS::Heat::None14:08
mariosrlandy: its already set via the resource registry14:08
weshay|ruckhttps://trunk.rdoproject.org/centos7-master/consistent/puppet-rabbitmq-8.2.3-0.20190214223744.1ccddad.el7.noarch.rpm14:08
rlandyanyways we know pacemaker is staying for standalone14:09
quiquell|roverhttps://trunk.rdoproject.org/centos7-master-head/consistent/14:09
rlandyso I need to figure out what is still missing from 'redis-bundle'14:10
weshay|ruckquiquell|rover https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/master.yml#L3714:11
weshay|ruckmarios sorry I'm late14:11
mariosweshay|ruck: np14:12
mariosweshay|ruck: let me know when you want to ttalk14:12
rlandymarios: ok - I'll try a no pacemaker + podman run - for debug at least14:13
mariosrlandy: ack sounds good14:14
chkumar|ruckweshay|ruck: https://github.com/voxpupuli/puppet-rabbitmq/commit/1ccdda14:16
quiquell|roverThis is the distgit14:17
quiquell|roverhttps://review.rdoproject.org/r/gitweb?p=puppet/puppet-rabbitmq-distgit.git;a=blob;f=puppet-rabbitmq.spec;h=457f3a65a4e1322fc9581f72156434561eb7b4cc;hb=HEAD14:17
chkumar|ruckquiquell|rover: weshay|ruck https://github.com/redhat-openstack/rdoinfo/tree/master/tags14:17
rlandymarios: one question on your patch ... https://review.openstack.org/#/c/636563/1..4/ci/environments/scenario001-standalone.yaml14:18
rlandywhy did you remove mysql in patch #4?14:18
mariosrlandy: well cos it is defaulted to that version in the resource reg14:18
mariosrlandy: shouldn't make difference if we have it there /not as long as we aren't overriding it anywhere else (and isn't overriden in the standalone/standalone-tripleo.yaml14:19
chkumar|ruckweshay|ruck: quiquell|rover https://github.com/redhat-openstack/rdoinfo/blob/master/tags/stein-uc.yml#L86314:19
*** amoralej is now known as amoralej|lunch14:26
*** ykarel is now known as ykarel|away14:29
*** ska has left #oooq14:31
*** ykarel|away has quit IRC14:33
weshay|ruckquiquell|rover https://mojo.redhat.com/groups/open-decision-framework14:37
weshay|ruckmarios rfolco zbr have time to chat about f28 in 5min?14:39
zbri think so.14:40
zbrweshay|ruck: btw, wes, can you help me with http://logs.openstack.org/44/636144/8/check/tripleo-build-containers-fedora-28/0c2820c/job-output.txt.gz#_2019-02-15_12_41_23_325582 ?14:40
mariosweshay|ruck: sure14:40
zbri am trying to figure what is the correct way to talk via ssh with gerrit14:41
quiquell|roverweshay|ruck: updated https://review.openstack.org/#/c/636588/ you can remove the -214:43
weshay|ruckquiquell|rover how so?14:43
quiquell|roverweshay|ruck: removed the molecule part but keeped the fix14:43
weshay|ruckquiquell|rover keep working w/ the patch, but we have to hold14:43
quiquell|roverweshay|ruck: to make build-test-packages fail14:43
weshay|ruckright k14:43
quiquell|roverok14:43
* weshay|ruck looks again14:43
weshay|ruckI thought it was just a molecule patch14:44
quiquell|roverweshay|ruck: nope14:44
quiquell|roverweshay|ruck: a fix and molecule testing it14:44
weshay|ruckk.. removed14:44
weshay|ruckthanks!14:44
weshay|ruckzbr marios rfolco ok..  I'm ready in my blue14:46
weshay|ruckhttps://bluejeans.com/u/whayutin/14:46
mariosweshay|ruck: need 3 mins pls brb14:46
weshay|ruckzbr rfolco?14:49
rfolcoweshay|ruck, will join but having lunch, so will be on mute for a while14:50
*** quiquell|rover is now known as quiquell|off14:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-containerized- (2 more messages)14:57
weshay|ruckrlandy  please review this asap https://review.rdoproject.org/r/#/c/18729/15:03
rlandyk15:03
*** ykarel|away has joined #oooq15:04
rlandyweshay|ruck: marios: want to merge this ... minor nits in comments15:12
*** ratailor has joined #oooq15:18
*** amoralej|lunch is now known as amoralej15:23
chkumar|ruckquiquell|off: on reverting it worked but blocked at tempest http://logs.rdoproject.org/43/13943/82/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/4b53345/logs/undercloud/home/zuul/tempest.log.txt.gz15:30
*** ratailor has quit IRC15:30
*** chkumar|ruck is now known as chandankumar15:38
chandankumarquiquell|off: and without revert it is also working, it might be a race condition then15:41
pandazbr: import_role is not the problem. It's effectively running twice15:42
mariosrlandy: thanks looking15:55
rlandymarios: nothing major - mergeable - of you want15:56
rfolcomarios, panda zbr weshay|ruck: this is the mirror patch, adding to the story15:56
rfolcohttps://review.openstack.org/63721215:56
mariosrlandy: done panda please revote rfolco https://review.rdoproject.org/r/#/c/18729/15:56
*** jfrancoa has quit IRC15:57
mariosrfolco: ack15:57
zbrrfolco: thanks, I will user depends on with it.16:00
rlandyare we ready for merge here?16:19
rlandyrfolco: marios: ^^?16:19
chandankumarweshay|ruck: please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/181602616:20
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)16:20
chandankumarweshay|ruck: the issue is very weired, I tried to re-run the job against testproject it is passing now16:20
* weshay|ruck looks16:21
mariosrlandy: yes i think so16:21
mariosrlandy: should not impact anything else https://review.rdoproject.org/r/#/c/18729/ isn't wired into anything16:22
rlandymarios: k - w'ed16:22
mariosrlandy: thanks !16:22
marios]/win 1216:22
rfolcoI voted already16:23
weshay|ruckchandankumar odd... I wonder if we need to add fs016 to the mistral jobs16:26
weshay|ruckchandankumar ok.. so in your test w/ the revert it's passing the deployment.. failing tempest16:26
chandankumarweshay|ruck: it is a docker issue16:27
weshay|ruckapetrich ping re: mistra issue16:27
weshay|ruckapetrich for monday, not tonight https://bugs.launchpad.net/tripleo/+bug/181602616:27
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)16:27
weshay|ruckchandankumar quiquell|off you guys need to add the target milestone to the bugs16:28
weshay|ruckor they don't get escalated16:28
weshay|ruckoh.. fs37, 30, 19, 18, 17, 16 in all of them got timeout.16:28
weshay|ruckchandankumar I would let these run over the weekend.. rdo-cloud may still be fubar'd16:30
*** saneax has quit IRC16:30
weshay|ruckchandankumar let's revisit this issue next week16:31
rlandymarios: http://10.10.122.58:8000/02/1002/1/check/tripleo-ci-centos-7-scenario009-standalone/50c0773/logs/undercloud/var/log/containers/ - this is the list of containers we have with scenario00916:34
rlandycompare with16:34
rlandystandard list16:34
rlandywhat provides httpd16:34
rlandyseems an obvious missing piece16:34
chandankumarweshay|ruck: sure16:36
rlandyhttp://10.10.122.58:8000/02/1002/1/check/tripleo-ci-centos-7-scenario009-standalone/50c0773/logs/undercloud/var/log/extra/docker/containers/mysql/stdout.log.txt.gz16:36
rlandystarts16:37
rlandy2019-02-15 16:26:39 0 [Note] Server socket created on IP: '192.168.24.1'.16:38
rlandy2019-02-15 16:26:39 0 [ERROR] Can't start server: listen() on TCP/IP port: Address already in use16:38
rlandy2019-02-15 16:26:39 0 [ERROR] listen() on TCP/IP failed with error 9816:38
rlandy2019-02-15 16:26:39 0 [ERROR] Aborting16:38
rlandy^^ error16:38
zbrdamn, cannot attend kubecon europe overlaps with vacation...16:40
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-containerized- (2 more messages)16:57
*** dtantsur is now known as dtantsur|afk17:08
marios[docs] please checkout https://review.openstack.org/#/c/635881/ http://logs.openstack.org/81/635881/2/check/openstack-tox-docs/597378b/html/ci/standalone_scenario_jobs.html17:15
marioshave a good weekend all17:15
* marios almost out17:16
*** bogdando has quit IRC17:18
*** amoralej is now known as amoralej|off17:38
*** ccamacho has quit IRC17:42
*** ccamacho has joined #oooq18:00
*** jpena is now known as jpena|off18:01
*** derekh has quit IRC18:04
rfolcozbr, lets make a quick call on monday to sync on f2818:12
rfolcomarios, panda fyi ^18:12
zbrrfolco: sure. we may had discovered another serious issue that could prevent us from merging it.18:30
rlandyweshay|ruck: can we merge this: https://review.openstack.org/#/c/636784/?18:32
*** kopecmartin is now known as kopecmartin|off18:46
*** chem has quit IRC18:48
rlandypanda: hi - ever seen this error on standalone: http://logs.openstack.org/64/635464/5/check/tripleo-ci-centos-7-scenario009-standalone/411424a/logs/undercloud/var/log/containers/mysql/mariadb.log.txt.gz#_2019-02-12_15_10_2718:53
rlandyDo we need something like:18:54
rlandy  LoggingServers:18:54
rlandy    - host: 127.0.0.118:54
rlandy      port: 2422418:54
rlandy  MonitoringRabbitHost: 127.0.0.118:54
rlandy  MonitoringRabbitPort: 567618:54
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-containerized- (2 more messages)18:57
*** ykarel|away has quit IRC18:59
*** holser_ has quit IRC19:16
*** holser_ has joined #oooq19:23
*** holser_ has quit IRC19:30
-openstackstatus- NOTICE: The StoryBoard service on storyboard.openstack.org is offline momentarily for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002666.html20:02
*** holser_ has joined #oooq20:07
rlandypanda: weshay|ruck: rfolco: anyone standalone savvy around?20:20
rlandyhttp://10.10.122.58:8000/02/1002/1/check/tripleo-ci-centos-7-scenario009-standalone/94b5446/logs/undercloud/var/log/extra/netstat.txt.gz20:20
rlandy^^ can anyone tell me why haproxy is started on very port where other services should start?20:20
*** agopi is now known as agopi|bbiab20:32
*** ccamacho has quit IRC20:36
*** agopi|bbiab has quit IRC20:37
weshay|ruckrlandy hrm..20:41
rlandyyay - somebody is home20:41
weshay|rucksorry.. my dad wanted to go to lunch20:41
rlandyweshay|ruck: np20:42
weshay|ruckwill poke around..  /me sees if there is any old successful run of the multinode scen00920:42
rlandyweshay|ruck: I think this is more a standalone thing20:43
weshay|ruckhrm..20:43
*** agopi|bbiab has joined #oooq20:43
rlandycompare that with20:43
rlandyhttp://logs.openstack.org/56/636656/3/check/tripleo-ci-centos-7-scenario001-standalone/45e5358/logs/undercloud/var/log/extra/netstat.txt.gz20:43
rlandythe services are all on the ports we expect20:43
weshay|ruckit shouldn't be the case.. that the overcloud version of a deployment should not have same services and config as the standalone20:43
weshay|ruckrlandy did you guy try rocky?20:44
rlandythat is a secnario001 standalone  compared with20:44
weshay|ruckrocky is the only known good version of scen009 I think20:44
rlandyI've been trying to avoid that - but I will go there20:45
rlandyI keep thinking I am missing something basic20:45
rlandybut the error is consistent20:45
weshay|ruckhaproxy.. don't see it here http://logs.openstack.org/73/636873/2/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/99ec013/logs/undercloud/var/log/extra/netstat.txt.gz20:45
rlandycant start mysql20:45
rlandythe port is always taken20:45
rlandythat is because haproxy has run all over everyone20:46
rlandyand should not even be present20:46
rlandyit shouldn't be there20:46
rlandyand the haproxy issue is there if I keep the pacemaker stuff or not20:46
rlandyso at least now we got the source of the error20:47
* weshay|ruck still poking20:49
weshay|ruckrlandy it's in inherting at all from https://github.com/openstack/tripleo-heat-templates/blob/master/environments/standalone/standalone-tripleo.yaml#L88 ?20:50
rlandyyes20:50
rlandyhttp://logs.openstack.org/17/634317/48/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/fef19b3/logs/undercloud/var/log/extra/netstat.txt.gz20:50
weshay|ruckrlandy  -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \20:50
rlandyweshay|ruck: ^^ that is normal20:51
rlandymultinode scenario00920:51
rlandyit fails but not over that20:51
weshay|ruckrlandy looking at http://logs.openstack.org/73/636873/2/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/99ec013/logs/undercloud/home/zuul/overcloud-deploy.sh.txt.gz20:52
weshay|rucknoting that the openshift yaml comes before scenario0920:52
weshay|ruckmaybe that plays a role in it20:53
rlandywho knows?20:53
rlandycan switch it around and see what happens20:53
rlandybut something is very wring here20:53
weshay|ruckhttps://github.com/openstack/tripleo-heat-templates/blob/master/environments/openshift.yaml20:53
rlandyhaproxy should not get setting up house everywhere20:53
rlandyOS::TripleO::Services::HAproxy: sould override that to none20:54
rlandythis is the problem20:54
weshay|ruckrlandy def don't see   OS::TripleO::Services::HAproxy: ../deployment/haproxy/haproxy-container-puppet.yaml20:55
rlandysorry - don;t see it where?20:55
weshay|ruckin your deployment config20:56
weshay|ruckhttps://review.openstack.org/#/c/635208/15/ci/environments/scenario009-standalone.yaml20:56
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-quickstart-extras-gate-master-delorean-quick-basic @ https://review.openstack.org/560445, stable/queens:  (1 more message)20:57
rlandywell - I havn't overwritten it20:57
weshay|ruckrlandy so.. I'm walking through what I see in scenario09 merged vs.. what is defined in the review20:57
weshay|ruckdef. .missing20:57
weshay|ruckit's here https://github.com/openstack/tripleo-heat-templates/blob/master/environments/openshift.yaml20:58
weshay|rucknot https://review.openstack.org/#/c/635208/15/ci/environments/scenario009-standalone.yaml20:58
rlandyI'm following this:20:59
rlandyhttps://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario009-multinode.yaml20:59
rlandyweshay|ruck: pls comment where you would expect environments/openshift.yaml to be added21:01
rlandyit's not in fs03321:01
rlandyoh ... here21:01
rlandy -e {{ overcloud_templates_path }}/environments/openshift.yaml21:01
rlandyhttps://review.openstack.org/#/c/635461/9/config/general_config/featureset059.yml21:02
rlandyhas the same21:02
rlandy  -e {{ working_dir }}/containers-prepare-parameter.yaml21:03
rlandy  {% endif %}21:03
rlandy  -e {{ overcloud_templates_path }}/environments/openshift.yaml21:03
rlandyand here21:03
rlandyhttps://review.openstack.org/#/c/635464/6/zuul.d/standalone-jobs.yaml21:03
weshay|ruckhrm21:04
rlandyhttp://10.10.122.58:8000/02/1002/1/check/tripleo-ci-centos-7-scenario009-standalone/94b5446/logs/undercloud/home/zuul/standalone.sh.txt.gz21:05
weshay|ruckrlandy order matters though21:09
weshay|rucktoo.. although now I see it.. thanks21:10
weshay|ruckhttp://logs.openstack.org/73/636873/2/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/99ec013/logs/undercloud/home/zuul/overcloud-deploy.sh.txt.gz21:11
* weshay|ruck comparing to that ^21:11
rlandyhttp://10.10.122.58:8000/02/1002/1/check/tripleo-ci-centos-7-scenario009-standalone/94b5446/logs/undercloud/var/log/config-data/haproxy/etc/haproxy/haproxy.cfg.txt.gz21:14
rlandycheck the ports assigned here21:14
rlandylisten mysql21:15
rlandy  bind 192.168.24.1:3306 transparent21:15
rlandy^^ correct21:15
weshay|ruckwhich variable is 3306 assigned to?21:19
weshay|ruckis it overwritten in https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-container-puppet.yaml#L170 ?21:19
rlandyvar/lib/config-data/haproxy21:20
*** holser_ has quit IRC21:30
rlandyweshay|ruck: no worries - I'll pick this up again on sunday - do a few more runs with different templates and ordering21:30
weshay|ruckrlandy ya.. sorry still juggling some things21:31
rlandyI think I need to bring back all  the ha stuff and ovveride it21:31
weshay|ruckrlandy I think I still would suggest going for rocky21:31
rlandyweshay|ruck: sure21:31
*** fultonj has quit IRC21:39
*** fmount has quit IRC21:39
*** fmount has joined #oooq21:39
*** fmount has quit IRC21:40
*** fmount has joined #oooq21:40
*** trown is now known as trown|outtypewww21:56
*** rlandy has quit IRC21:58
*** agopi|bbiab has quit IRC22:22
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo- (2 more messages)22:57
*** rascasoft has quit IRC23:06
*** vinaykns has quit IRC23:10
pandawoot!! http://logs.openstack.org/25/634725/7/check/tripleo-ci-centos-7-scenario012-standalone/2df8c63/job-output.txt.gz23:24
*** panda is now known as panda|off23:28
weshay|ruckpanda|off nice23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!