Friday, 2019-03-01

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages)00:10
*** tosky has quit IRC00:18
*** rascasoft has joined #oooq00:32
*** trown is now known as trown|outtypewww00:39
*** rascasoft has quit IRC00:41
*** hamzy has joined #oooq01:11
*** rascasoft has joined #oooq01:28
*** rascasoft has quit IRC01:40
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages)02:10
*** rascasoft has joined #oooq02:16
*** rascasoft has quit IRC02:26
*** dsneddon has quit IRC02:44
*** rlandy has quit IRC02:49
*** dsneddon has joined #oooq03:09
*** apetrich has quit IRC03:15
*** dsneddon has quit IRC03:15
*** dsneddon has joined #oooq03:43
*** skramaja has joined #oooq03:48
*** skramaja has quit IRC03:53
*** dsneddon has quit IRC03:58
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (3 more messages)04:10
*** ykarel|away has joined #oooq04:47
*** ykarel|away is now known as ykarel04:47
*** dsneddon has joined #oooq04:51
*** dsneddon has quit IRC05:05
*** dsneddon has joined #oooq05:06
*** raukadah is now known as chandankumar05:09
*** ratailor has joined #oooq05:44
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (3 more messages)06:10
*** dsneddon has quit IRC06:29
*** saneax has joined #oooq06:42
*** quiquell|off is now known as quiquell07:01
*** dsneddon has joined #oooq07:01
*** dsneddon has quit IRC07:16
*** dsneddon has joined #oooq07:20
*** jfrancoa has joined #oooq07:21
*** kopecmartin|off is now known as kopecmartin07:23
*** saneax has quit IRC07:23
*** saneax has joined #oooq07:24
*** dsneddon has quit IRC07:25
*** rascasoft has joined #oooq07:30
*** jtomasek has joined #oooq07:37
*** gkadam has joined #oooq07:37
*** dsneddon has joined #oooq07:41
*** quiquell is now known as quiquell|brb07:54
*** dsneddon has quit IRC07:57
kopecmartinchandankumar, arxcruz hi, when you have a moment, please, have a look https://review.openstack.org/#/c/625191/08:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (3 more messages)08:10
*** apetrich has joined #oooq08:12
chandankumarkopecmartin: done!08:14
kopecmartinchandankumar, thanks08:14
*** amoralej|off is now known as amoralej08:14
chandankumarkopecmartin: since now os_tempest playbook is merged now, I will take a look at standlone insallation and user stuff08:15
kopecmartinchandankumar, wait, what do you mean, I'm working on usage of os_tempest via ansible-playbook command08:16
chandankumarkopecmartin: ok you are looking into then good, I was about to start!08:17
chandankumarkopecmartin: I have to deal with few os_tempest deps and tempest users patches08:17
kopecmartinchandankumar, ok08:18
chandankumarkopecmartin: let me know if you face any issue08:18
kopecmartinsure08:18
chandankumarkopecmartin: one more thing if you need to install any plugin just do tempest_service_available_[service] = true08:18
chandankumarit will install the plugin for that08:18
chandankumarif source then it will clone from git08:19
kopecmartinyeah, i get it08:19
chandankumarif distro then from packages08:19
*** ccamacho has joined #oooq08:19
chandankumarcool!08:19
*** quiquell|brb is now known as quiquell08:26
*** jfrancoa has quit IRC08:29
*** bogdando has joined #oooq08:42
*** tosky has joined #oooq08:42
*** jpena|off is now known as jpena08:48
*** dsneddon has joined #oooq08:55
*** holser_ has joined #oooq08:55
*** dsneddon has quit IRC09:00
*** panda|ruck|off is now known as panda|ruck09:05
panda|ruckTra-la-laaa09:06
* panda|ruck captain underpands09:06
*** saneax has quit IRC09:08
*** saneax has joined #oooq09:09
panda|ruckweeee all ovb jobs are failin09:09
*** saneax has quit IRC09:10
*** saneax has joined #oooq09:10
*** ykarel is now known as ykarel|lunch09:27
*** dsneddon has joined #oooq09:32
panda|ruckykarel|lunch: was the python-hardware bug fixed ?09:32
*** dtantsur|afk is now known as dtantsur09:37
*** dsneddon has quit IRC09:40
*** derekh has joined #oooq09:42
*** sshnaidm is now known as sshnaidm|off10:00
ykarel|lunchpanda|ruck, yes that is fixed10:03
*** ykarel|lunch is now known as ykarel10:03
panda|ruckykarel: ok thanks10:05
panda|ruckweshay: rfolco|rover ykarel shutting down promoter-server for maintenance10:05
ykarelpanda|ruck, ack10:06
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (3 more messages)10:10
*** saneax has quit IRC10:25
*** saneax has joined #oooq10:26
*** dsneddon has joined #oooq10:27
*** saneax has quit IRC10:30
*** dsneddon has quit IRC10:32
quiquellpanda|ruck: do you have desing time for me ?10:42
panda|ruckquiquell: I have always design time for everyone. except when I'm battling promotion issues10:43
quiquellpanda|ruck: are you battling promotions ?10:43
panda|ruckquiquell: just finished. Updating the bug, then we can talk10:43
quiquellack10:43
panda|ruckquiquell: ready10:49
panda|ruckquiquell: on your room10:49
quiquellgoing there10:50
marios https://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build)10:51
quiquellmarios: you can join10:52
quiquellmarios: design BM trigger10:52
mariosquiquell: sure gimme 2 mins joining10:52
mariosquiquell: number/?10:52
quiquellwait cannot join my blue10:53
quiquell789106523210:53
sshnaidm|offquiquell, panda|ruck marios please help merge logs size reducing: https://review.openstack.org/#/q/topic:newara11:04
mariossshnaidm|off: ack in a bit (in a call with panda|ruck now)11:06
marioshttps://review.openstack.org/#/c/639359/611:12
mariosquiquell: ^11:13
marios                  http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003330.html11:13
mariosquiquell: ^11:13
zbrmarios or panda|ruck: on bindep: https://review.openstack.org/#/c/639951/ - you are not core but votes may help getting the wf from those that can do it.11:16
zbronce we get this, we could avoid installing packages with ansible and use bindep11:17
zbr(in some places)11:17
*** jfrancoa has joined #oooq11:17
chandankumarzbr: need some help here https://review.openstack.org/#/c/640089/2/roles/validate-tempest/templates/run-tempest.sh.j2@88 I am not sure why jinja is not working11:27
chandankumarplease have a look when free!11:28
*** dsneddon has joined #oooq11:28
quiquellpanda|ruck: do you know why we use dlrn user "review_rdoproject_org" at reporting and "ciuser" at promotion ?11:31
quiquellhumm nope same for promote :-/11:32
zbrchandankumar: lgtm, what is the failure? do you have the generated file?11:34
*** dsneddon has quit IRC11:34
panda|ruckquiquell: we use ciuser at promotion11:34
panda|ruckquiquell: because review_rdoproject_org is a secret in zuul11:35
chandankumarzbr: 2019-03-01 06:10:14 | /home/zuul/tempest-setup.sh: line 159: KRB5_CLIENT_KTNAME: unbound variable11:35
panda|ruckquiquell: and we didn't probably want to use the same credentials for an external server11:35
chandankumarzbr: I am thinking adding two stuff there, first var is defined and then check the bool11:35
quiquellpanda|ruck: where do we that password ?11:35
chandankumarzbr: what do you say?11:36
quiquellpanda|ruck: I think the secret we have put at internal sf is for ciuser passwoerd :-/11:36
zbrchandankumar: is not jinja issue, the env var is not defined11:36
zbrif it can be null do use.  "${FOO:-}" to default its expension to empty string11:36
zbrinstead of "${FOO}"11:36
chandankumarzbr: sure let me update11:37
zbrchandankumar: you jinja2 is good. btw read http://redsymbol.net/articles/unofficial-bash-strict-mode/ when you have time.11:38
zbris one of my fav articles, very useful11:38
chandankumarzbr: sure, adding to the list11:38
panda|ruckzbr: I think ou'll need to add tests for https://review.openstack.org/63995111:38
panda|ruckzbr: like  in  def test_detects_rhel(self):11:39
panda|ruckzbr: and you'll have to create fixtures11:39
zbrpanda|ruck: already did, kinaof. Added f28 job which was missing in the past and updated the tests/bindep.txt file. not a full test. I hope no core asks for more on that.11:40
panda|ruckzbr: I wouln't count on it.11:40
zbrcurrent bindep tests do not have something like this, even for other cases. i would rather prefer to alter the integration testing to check if right platforms are reported.11:41
panda|ruckzbr: you're adding a new platform: category and a new conditional, they'll probably want tests. at least check that verify a Contains("platform:dnf") when the version is right11:41
quiquellpanda|ruck: I am going to create new user for internal sf it's also a good idea11:42
marioshttps://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build)11:43
zbradding tests for it would require a lot of mocking...  the fact that is passing integration testing is good enough for me. anyway, lets see..11:43
panda|ruckzbr: I don't think the integration testing is checking if bindep honours the new platform:yum  atom at all11:46
zbrpanda|ruck: true, i does only check that it does not choke. your point is valid, is just that I am trying to minimize the effort. me kinda lazy, just want to do minimal effort to get it done ;)11:49
panda|ruckzbr: is this review required ?11:50
panda|ruckzbr: for the US ?11:50
zbrarguable / non-blocking, it done we could get rid of stuff like https://review.openstack.org/#/c/636160/53/roles/tripleo-repos/defaults/main.yml11:52
zbrwe do lots of hacks now to install alternative rpms, also affects new reproducer. so is in our interest to make bindep better. but is not directly blocking us form using (ugly) workarounds.11:53
panda|ruckzbr: ok, analyzing the amount of effort needed at this point, if they're going to ask you to add more tests, I would probably suggest to put this aside for this sprint, or work on it on the 20%11:53
zbrthis is why i am trying to fix it without making any other work depending on it11:54
panda|ruckzbr: yes, but adding the test would probably double the workload on this patch11:54
zbrpanda|ruck: totally agree.11:54
panda|ruckzbr: I reached the conclusion some time ago. For how easy a patch seems to take to write, it's the design and the testing that you should consider more time consuming11:56
zbrpanda|ruck: my estimates are 5-10x more time than doing the fix locally. i count this most of the time, with one side note: every time i need something fix upstream i raise a bug (or pod fix), just to start the process, even if i never endup finishing it, at least someone else can continue it.12:00
panda|ruckquiquell: correction, until the end of the sprint, WE have 7 days, YOU have 512:00
mariosquiquell: you can hear him smiling as he types that srsly man12:01
panda|ruckIt's astonishly easy to forget thos details on time.12:01
quiquellmarios: PTG is payback :-)12:01
panda|ruckastonishingly ?12:01
panda|ruckmarios: a smile with 52 teeth12:02
panda|ruckmarios: no wait, 43 and half until the end of the sprint12:02
quiquellpanda|ruck, marios: To make DLRN API user configurable https://review.rdoproject.org/r/1906412:03
chandankumarsshnaidm|off: I have updated the tls temepst taiga us feel free to take a look https://tree.taiga.io/project/tripleo-ci-board/us/670 when free, thanks!12:04
quiquellpanda|ruck: linters are broken ?12:08
mariosquiquell: ack added for later12:08
panda|ruckquiquell: where ?12:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages)12:10
panda|ruckmmhh node failure12:10
quiquellyep12:11
panda|ruckquiquell: where's the graph in the cockpit that says how many staks are broken ?12:12
panda|ruckquiquell: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=175&fullscreen12:12
panda|ruckthis one12:12
quiquellnope12:12
panda|ruckno ?12:12
panda|ruckI see 600 servers and 30 in error12:12
quiquellhttp://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=1551269576074&to=1551442376074&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=rocky&var-releases=queens&var-releases=pike&panelId=231&fullscreen12:13
quiquellthe problematic one12:13
quiquellshorter12:13
quiquell200~http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=231&fullscreen12:13
panda|ruckI don't see any particular explosion12:14
quiquellfailed again12:15
quiquellwell I think tox-liners doe not run at heat stacks12:15
quiquellalso I think they have change at run at containers12:15
panda|ruckquiquell: better ask in rdo then ... wee need zuul logs ..12:16
panda|ruckweeeee12:16
*** panda|ruck is now known as panda|ruck|lunch12:16
*** ykarel is now known as ykarel|afk12:18
*** dsneddon has joined #oooq12:19
*** dsneddon has quit IRC12:25
mariospanda|ruck|lunch: how do you automatically appear as reviewer in rdo config reviews. i just posted it but you're already on it?! https://review.rdoproject.org/r/#/c/19066 are you filed under 'default' somewhere?12:46
mariospanda|ruck|lunch: quiquell ** we need this asap so we can test the actual push https://review.rdoproject.org/r/#/c/19065/ https://tree.taiga.io/project/tripleo-ci-board/task/817 (and then https://review.rdoproject.org/r/#/c/19066/ will be the last piece but we need this first)12:50
mariostox-linters tox-linters : NODE_FAILURE12:50
quiquellmarios: Yep I am having same failure there12:51
*** jpena is now known as jpena|lunch12:51
quiquellmarios: I think is related to move to runc12:51
quiquellof tox jobs12:51
mariosweshay: panda|ruck|lunch: rfolco|rover fyi 2tasks under the centos container story https://tree.taiga.io/project/tripleo-ci-board/task/817 for testing the push (and fixing anything) then finally https://tree.taiga.io/project/tripleo-ci-board/task/818 switch to new job if we are brave ;)12:52
mariosquiquell: thx12:52
weshaychandankumar who's bluejeans?12:58
chandankumarweshay: arxcruz kopecmartin https://redhat.bluejeans.com/1571313919/6145/?src=meet_now12:59
chandankumarweshay: mine!13:00
*** dsneddon has joined #oooq13:01
weshaychandankumar link?13:01
chandankumarweshay: https://redhat.bluejeans.com/1571313919/6145/?src=meet_now13:01
chandankumararxcruz: we are waiting for you!13:02
*** panda|ruck|lunch is now known as panda|ruck13:04
*** ratailor has quit IRC13:07
*** dsneddon has quit IRC13:10
weshayarxcruz ping13:12
arxcruzweshay: sorry13:12
*** trown|outtypewww is now known as trown13:16
chandankumarweshay: arxcruz kopecmartin https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/tempest.yml13:16
chandankumarweshay: arxcruz kopecmartin https://github.com/openstack/openstack-ansible-tests/blob/master/test-vars.yml13:17
zbrmarios: quiquell weshay : please revote on https://review.openstack.org/#/c/636160/ - f28 fixed last remark and passed zuul.13:18
chandankumarweshay: arxcruz kopecmartin https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tests/os_tempest-overrides.yml13:18
*** holser_ has quit IRC13:19
quiquellzbr: I don't see the third party there13:20
zbrquiquell: go there and check manually, fedora passted.13:21
zbris not my fault that rdo is very slow today13:21
quiquellyou mean zuul ?13:22
quiquellyep checked :-)13:22
quiquellcool13:22
quiquellmarios: going to workflow it13:23
marioszbr: ah you mean it didn't report yet but it passed? anyway +2 added note about waiting for rdo13:23
quiquellhttp://logs.rdoproject.org/60/636160/53/openstack-check/tripleo-build-containers-fedora-28/6c1718b/13:23
quiquell^ passed13:23
*** holser_ has joined #oooq13:23
*** holser_ has quit IRC13:23
mariosquiquell: thanks13:23
quiquellzbr: +w13:23
*** holser_ has joined #oooq13:24
zbrthanks!13:24
chandankumarweshay: arxcruz kopecmartin http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-os-tempest13:24
marioshttps://review.openstack.org/#/c/638651/ needs more votes please when folks have time. part of https://tree.taiga.io/project/tripleo-ci-board/task/773 (tripleo-ci-testing for repos in containers build)13:24
quiquellmarios: can we workflow tripleo-repos ?13:28
quiquellpanda|ruck: ^ Â?13:29
panda|ruckquiquell: yes, logic is working, deps is implicit, marios added also unit tests and they are working, so +2 from me13:32
quiquellpanda|ruck: I mean tripleo-repos is like tht that we can review vote but not workflow ?13:33
rfolco|roverpanda|ruck, you get 5 min?13:33
quiquellmarios: +w13:33
mariosquiquell: yeah13:34
mariosquiquell: i think so13:34
panda|ruckquiquell: well, tripleo-repos now affects directly our jobs, I'd say we are forced to take part of the ownership, even only if we want to merge it to repo-setup and use only one method13:34
mariosquiquell: i mean i think its done we tested it13:34
panda|ruckrfolco|rover: yes, I'm gonna get 5 minutes13:34
*** rlandy has joined #oooq13:34
rfolco|roverpanda|ruck, can you bj my room pls?13:34
mariosquiquell: it got the tripleo-ci-testing repo in the test job review.rdoproject.org/r/#/c/1900013:34
mariosthanks quiquell panda|ruck13:34
quiquellzbr, marios: what else can we merge ?13:35
zbrthis stupid one https://review.openstack.org/#/c/640320/13:36
mariosquiquell: we also need this one https://review.rdoproject.org/r/19065 so we can test the push13:37
mariosquiquell: ah thanks you already voted there sorry13:38
quiquellzbr: I don't see the tox ci there13:44
quiquellweshay, rlandy: for the containers-build-push cofing project -> https://review.rdoproject.org/r/#/c/19065/13:45
zbrquiquell: that's because someone decided that tox.ini is on exclude pattern. you need to run it manually to see that it works.13:45
quiquellzbr: remove from exclude13:45
quiquellzbr: it really affects the tox jobs13:45
quiquellAll of them13:46
zbrquiquell: it may upset others, i could try but not if others do not want it.13:47
rlandyquiquell: zbr: can +2 that but zuul would have to clear its -113:48
quiquellrlandy: damn... the tox NODE_FAILURE13:48
quiquellrlandy: btw, dlrn report not ready for QE at baremetal13:48
quiquellrlandy: password was failing13:48
quiquellrlandy: at test job13:48
rlandyquiquell: ok - I'll I can recreate it and check13:49
quiquellrlandy: is not that13:49
quiquellrlandy: the password is not the one fron ciuser13:49
zbrpanda|ruck: ^ i think we have aproblem on rdo, with nodes, node failure with linters13:49
quiquellrlandy: is the user is review_rdoproject_org <- cannot find it13:50
rlandyquiquell; I took the one from  centos13:50
rlandyquiquell: from the promoter server13:50
rlandywent to user centos13:50
quiquellrlandy: nope this is not the one13:50
quiquellrlandy: promoter pass is for user ciuser13:50
panda|ruckzbr: yes, quiquell asked in #rdo13:50
panda|ruckquiquell: any replay13:50
quiquellrlandy: but at RDO periodic jobs user is review_rdoproject_org13:50
rlandyquiquell: let me find the ciuser password sec13:51
quiquellrlandy: I have that one13:51
quiquellrlandy: but ciuser is not the user13:51
* quiquell find the code13:51
zbrquiquell: i have the impression that the only tox job running on tripleo-ci is the linters one.13:52
*** ykarel|afk is now known as ykarel13:52
panda|ruckzbr: heat stacks are currently high but not in error.13:52
quiquellpanda|ruck: those tox failures are related to runc13:52
quiquellpanda|ruck: I think heat stack are not involve there13:52
panda|ruckquiquell: runc ?13:53
quiquellpanda|ruck: 0001702497-runc-centos-7-100-000047190613:54
quiquellthis is the error13:54
quiquellhttps://paste.fedoraproject.org/paste/HxNqbp0bkGygMe-ukaKVeA13:54
*** jpena|lunch is now known as jpena13:54
weshayarxcruz https://review.openstack.org/#/c/639794/13:54
quiquellrlandy: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/dlrnapi_report.sh#L913:54
panda|ruckquiquell: today I don't understand when you are adding typos or writing seriously13:54
quiquellrlandy: so for periodics user is review_rdoproject_org13:54
quiquellrlandy: I am creating new user (tripleo-internal-ci) to report from BM13:54
quiquellrlandy: No one find the password for review_rdoproject_org though13:55
rlandyquiquell: lol13:55
quiquellyep13:55
quiquell:Ã-/13:55
rlandyso I shouldn't bother looking for it13:55
rlandyhow are reporting then?13:55
quiquellrlandy: totally I have put a "TODO" in the creds doc13:55
rlandymystery13:56
quiquellrlandy: rdo secret is for review_rdoproject_org password13:56
quiquellrlandy: but internal factory secret is for ciuser password13:56
quiquellrlandy: at internal factory dlrn reporting was not working13:56
quiquellrlandy: I am taking a step buck and do noop job to test reporting13:56
rlandyquiquell; k - you can just recreate the secret for tripleo-ci-internal-config13:57
quiquellrlandy: but I don't have the password13:58
rlandyquiquell: ack - when you do13:58
quiquellsure sure13:58
rlandyquiquell: do yo have the secret create url for downstream?13:58
quiquellrlandy: yep no problem with that13:58
* rlandy will send13:58
quiquellknow that stuff13:59
quiquellfrom reproducer work13:59
quiquellrlandy: I have put in place a change in the role to make user configurable -> https://review.rdoproject.org/r/#/c/19064/13:59
quiquellrlandy: so we can use 'ciuser' now13:59
quiquellrlandy: or the new user jpena is creating13:59
rlandyquiquell: k - cool. we had a bunch of node failures last night so I couldn't test but previously, the job was not cleaning up so the old vm was still there14:01
rlandymight explain why we never connected to it14:01
rlandyretrying this morning14:01
quiquellrlandy: yap, before leave i show a lot of dirty stuff there14:02
quiquellrlandy: we need the cleanup14:02
rlandyquiquell: toci had skip-tags set14:02
rlandythere is a lot of voodoo there14:02
rlandywe undercover a lot of magic when we try to do something different14:02
quiquellhackgic14:03
panda|ruckquiquell: see what I meant ?14:03
quiquellpanda|ruck: about what ?14:04
panda|ruck13:54:36    +panda|ruck | quiquell: today I don't understand when you are adding typos or writing seriously14:05
rlandyquiquell: check it out ... https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/b75753d1b8194b99bcb50a34e966b0ca?logfile=console.log14:05
rlandywe're installing undercloud14:06
quiquellpanda|ruck: serious types I would say14:06
quiquellrlandy: \o/ !!!14:06
quiquellyey14:06
quiquellrlandy: another one, URL trigger is no good for us14:07
rlandyquiquell: we need a drink - both of us!!14:07
quiquellrlandy: we don't have the proper headers anywhere14:07
weshayquiquell rlandy NICE14:07
weshayso what was the deal w/ the key?14:07
rlandyweshay: keys were fine14:07
quiquellpermissions ?14:07
rlandyweshay: it just wan't running clean up14:07
rlandyso old vm new keys14:07
weshaylolz14:07
rlandyno match14:07
weshayright14:07
weshayk14:07
weshayrlandy easy :)14:07
quiquellrlandy: maybe docker container with clean toci :-)14:08
weshayquiquell told you :)14:08
quiquellweshay: old doc14:08
quiquelldog14:08
weshaylolz14:08
weshaythat's right.... young blookd14:08
weshayblood14:08
rlandyquiquell: bakc to trigger - no good?14:08
rfolco|roverquiquell, can you please join my bj to see if you have any clues on my centos reproducer error ?14:09
rlandyI think QE is using a URL trigger14:09
panda|rucktoday is officailly typo day14:09
quiquellrlandy: yep we don't ahve ETag or Last-Modified headers14:09
rlandypanda|ruck: everyday is typo day14:09
rfolco|roverquiquell, my fedora one worked, I just cannot make the centos reproducer work14:09
quiquellrlandy: also if they fail is difficult to debug14:09
rlandyI bet14:09
rlandyconnection etc.14:09
rlandyquiquell: what about the chained job idea?14:09
rlandyquery dlrn14:10
quiquellrlandy: so going to do a POC on DLRN polling14:10
*** holser_ has quit IRC14:10
quiquellrlandy: yep going to do that14:10
rlandyquiquell,: awesome14:10
quiquellrfolco|rover: let's take a look14:10
rlandyquiquell: if this job passes, I ma going to merge the base job in config14:10
rfolco|roverquiquell, https://redhat.bluejeans.com/u/rfolco/14:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages)14:10
quiquellweshay: do you know where do we have the password for rewiew_rdoproject_org ?14:10
rlandywe are getting a lot of errors about having a base job in jobs14:11
quiquellrlandy, weshay: What was the state of reproducer and OVB ?14:16
quiquellwe can run them out of the box ?14:16
*** ykarel_ has joined #oooq14:16
rlandyquiquell; ack - I added stuff to the script for ovb14:17
rlandyI think marios managed to do so14:17
quiquellmarios: reproducer + ovb works out of the box ?14:18
weshayrlandy can you please merge https://review.rdoproject.org/r/#/c/19055/14:18
mariosquiquell: rlandy ? i don't think i tried that?14:18
mariosoh14:18
mariosyou mean rdo cloud yah rlandy quiquell ?14:18
rlandymarios: oh - sorry14:18
*** ykarel has quit IRC14:19
rlandyweshay: done14:19
weshayrfolco|rover panda|ruck did sova ever end up show either the introspection or image build error?14:19
mariosrlandy: quiquell i mean yeah few nits as documented in https://tree.taiga.io/project/tripleo-ci-board/task/76514:20
mariosrlandy: quiquell but yes got to a complete run yesterday14:20
rfolco|roverweshay, I checked yesterday, the last time it happened was before sshnaidm|off changing the sova production server... will check it again14:20
weshayrfolco|rover please add a link / log to prove it's working.. it's just good practice https://bugs.launchpad.net/tripleo/+bug/181759814:20
openstackLaunchpad bug 1817598 in tripleo "introspection ( prepare images ) failing in 3rd party ovb jobs fs001/35" [Critical,Fix released]14:20
quiquellmarios: ack didn't remember it was OVB thanks14:20
*** holser_ has joined #oooq14:20
weshayrfolco|rover k.. thanks14:20
quiquellrfolco|rover: ^ looks like it should work https://review.rdoproject.org/r/#/c/19064/14:20
rfolco|roverok14:20
rlandymarios: you're the best taiga user out there :) everything is so well doc'ed14:21
quiquellrfolco|rover: https://tree.taiga.io/project/tripleo-ci-board/task/76514:21
mariosrlandy: thanks. i am liking the markdown support there its pretty easy to make it look pretty14:22
* rlandy needs to look into that14:22
*** dsneddon has joined #oooq14:23
*** vinaykns has joined #oooq14:23
bogdandoPTAL https://review.openstack.org/#/q/topic:ci_pipelines+(status:open+OR+status:merged) may be we should just give it a try and see?14:24
*** dtrainor_ has quit IRC14:26
*** dtrainor has joined #oooq14:26
chandankumarHappy weeekend guys, see ya on monday :-)14:26
*** amoralej is now known as amoralej|lunch14:26
*** chandankumar is now known as raukadah14:26
mariosrlandy: quiquell welp we can't run the reproducer on a kvm guest then "libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm"14:27
mariosrlandy: quiquell (its just what beaker gave me, getting another box now)14:28
mariosrlandy: quiquell yeah but that is silly, maybe use it for launching on ovb but not for libvirt o_O14:28
*** dsneddon has quit IRC14:29
ykarel_panda|ruck, is the RDO phase1 master/rocky tripleo issue is known? any bug?14:29
*** ykarel_ is now known as ykarel14:29
rlandymarios: you should be able to run libvirt14:29
mariosrlandy: on a kvm guest though?14:29
panda|ruckykarel: I'm looking at it, but looks like even the collect logs is failing14:29
panda|ruckykarel: fails during overcloud-prep-images14:30
ykarelpanda|ruck, ack, okk so you are already on it14:30
ykarelhmm without logs it will difficult to find it14:30
rlandymarios: never tried that - on a beaker box directly - yes14:31
panda|ruckykarel: https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/112/artifact/console.txt.gz14:31
panda|ruckykarel: missing14:31
panda|ruckykarel: I'm trying to understand where the collect logs fails14:31
weshaypanda|ruck can we merge this if it's working an patch on top for improvements?14:31
weshayhttps://review.rdoproject.org/r/#/c/19048/2/ci-scripts/infra-cleanup/ovb-tenant-cleanup.sh14:31
mariosrlandy: ack. yeah beaker just gave me a non vm box so will try that now.14:32
* weshay see's 3 servers in error14:32
*** ccamacho has quit IRC14:33
ykarelpanda|ruck, /me looking14:33
*** ccamacho has joined #oooq14:34
panda|ruckweshay: yes, works14:34
panda|ruckweshay: jsut deleted those three servers in error14:34
weshayrock14:34
panda|ruckweshay: removing -114:34
zbrweshay: we hace some serious discussions related to yum/dnf/bindeo on infra now... pabelanger opened pandora box14:35
weshayruh roh14:35
weshaymarios you have 5 min?14:35
mariosweshay: o/14:35
panda|ruckzbr: you're a troublemaker :)14:35
weshaymarios if you have 5min join my blue14:35
zbrit seems that official pkg manager in r8 is *yum*, not dnf, but that yum is a ~symlink for dnf.14:36
mariosweshay: k joining14:36
weshayyes14:36
zbransible reports pkg_mgr as dnf which suited us.14:36
weshayzbr that is correct14:36
panda|ruckzbr: timebox the time you want to spend on this please ...14:36
rlandyweshay: quiquell: SUCCESS - https://sf.hosted.upshift.rdu2.redhat.com/logs/49/163849/39/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/b75753d/ - with collect logs operational14:36
panda|ruckzbr: it could easily take the rest of the week14:36
weshayzbr does that mess up ansible or cli options?14:36
ykarelpanda|ruck, so possibly logs are failing since https://review.openstack.org/#/c/631067/ merged14:37
ykarelbefore that i can see logs14:37
ykarelmerged on 9th Feb14:37
panda|ruckykarel: oh, so it's rlandy's fault14:37
weshayrlandy++14:37
hubbot1weshay: rlandy's karma is now 4514:37
weshayrlandy++14:37
hubbot1weshay: rlandy's karma is now 4614:37
panda|ruck:)14:38
zbrnot messing ansible, but may mess us, and clearly is affecting the bindep change we wanted to make.14:38
weshayquiquell++14:38
hubbot1weshay: quiquell's karma is now 2014:38
ykarelrfolco|rover, hi you remember discussion related to fs02114:39
ykarel<rfolco|rover> ykarel, I'll open a bug for it, high prio, not critical14:39
ykarel^^ one?14:39
ykareli see job still failing14:39
rlandypanda|ruck: what did I do now?14:40
rfolco|roverykarel, ok, panda|ruck don't we have a bug for fs20 ?14:40
ykarel021?14:40
rfolco|roveroops14:40
rfolco|roverpanda|ruck, fs02114:40
panda|ruckfs20 ?14:41
zbrpanda|ruck: that issue is far more important than you may think as it could ruin a huge amount of effort we invested in preparing for the *8, is not something we want to ignore.14:41
panda|ruckrlandy: the reproducer script creation broke the logs collection in ci.centos14:42
panda|ruckrlandy: working on it14:43
panda|ruckykarel: yes, that's it14:43
rlandypanda|ruck: wow and nobody noticed for almost a month14:43
panda|ruckykarel: that part runs and misses a lot of files because assumes zuul14:43
ykarelyup14:44
panda|ruckrlandy: yep, well, ci.centos jobs where passing at a reasonable rate14:44
rlandypanda|ruck: k - let me know if you need help - we should be able to skip if user is not zuul14:44
panda|ruckzbr: which issue ?14:44
quiquellrlandy: SUCCESS ? you kidding ?14:46
quiquellyey !!!14:46
*** dsneddon has joined #oooq14:49
quiquellrfolco|rover: https://operations.cee.redhat.com/quickvm14:49
rlandyquiquell: well undercloud only - but it's a start14:50
quiquellrlandy: good one, ok have to drop now14:50
*** ccamacho has quit IRC14:51
*** quiquell is now known as quiquell|off14:51
rlandyquiquell|off:  have a good weekend14:52
*** ccamacho has joined #oooq14:53
panda|ruckrlandy: ykarel  https://review.openstack.org/64039314:53
ykarelpanda|ruck, ack14:54
panda|ruckrlandy: ykarel in CI we are currently creating a zuul dict based on the zuul vars content. If we'll ever move quickstart to the executor itself, the zuul variable should be present anyway.14:55
*** dsneddon has quit IRC14:56
rlandypanda|ruck: yep, should be fine14:57
ykarelpanda|ruck, ack14:57
panda|ruckweshay: we'll have to wait for https://review.openstack.org/640393  to have logs in ci.centos and undestand why it's not working. currently failing in overcloud-prep-images, but we have not logs to investigate14:59
weshaypanda|ruck++15:11
hubbot1weshay: panda|ruck's karma is now 115:11
*** dsneddon has joined #oooq15:16
*** dsneddon has quit IRC15:21
*** ratailor has joined #oooq15:23
weshayrfolco|rover k15:24
weshayready15:24
weshayzbr I need to speak w/ you in a bit15:24
rfolco|roverweshay, o/15:24
zbrsure.15:24
*** amoralej|lunch is now known as amoralej15:25
*** dsneddon has joined #oooq15:38
*** dsneddon has quit IRC15:46
weshaypanda|ruck sova is NOT fixed15:49
weshay https://logs.rdoproject.org/20/639620/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6283a94/job-output.txt.gz15:50
weshayhttps://logs.rdoproject.org/20/639620/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/6283a94/logs/bmc-console.log15:50
weshaypanda|ruck this is not working https://github.com/sshnaidm/sova/blob/master/tripleoci/data/patterns.yml#L57015:50
weshaycan you please take another look15:51
weshaypanda|ruck I wonder if sova is pulling that log15:51
rfolco|roverweshay, this --> https://github.com/sshnaidm/sova/blob/master/tripleoci/data/patterns.yml#L52915:52
weshayI would think this does it https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L24515:52
*** ratailor has quit IRC15:54
*** dsneddon has joined #oooq15:54
panda|ruckweshay: diving deeper ...15:57
*** dsneddon has quit IRC15:59
weshaythanks16:03
*** ykarel is now known as ykarel|away16:03
*** holser_ has quit IRC16:03
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages)16:10
panda|ruck2019-03-01 16:13:33,576 - watchcat - DEBUG - utils.get_regular_file:237 - Get regular file /logs/bmc-console.log16:13
panda|ruck2019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz was saved as 40416:14
panda|ruck2019-03-01 16:13:33,577 - watchcat - WARNING - analysis.analyze:105 - File /logs/bmc-console.log is not downloaded, skipping its patterns16:14
panda|ruckit fails to download it16:14
*** dsneddon has joined #oooq16:15
panda|ruckweshay: it tries to download tehe file assuming is a gzipped file, and there is an undocumented pattern to specify to get the plain file that we want ... trying to decode the syntax16:20
*** jfrancoa has quit IRC16:21
weshaypanda|ruck ah nice16:23
panda|ruckweshay: nope, nothing I can do without modifying some code here, sova assumes it has to download a gzipped file16:23
weshaythank you16:23
weshayah16:24
weshay:(16:24
panda|ruckweshay: it's hard-coding adding a .gz at the end16:24
weshayaye16:24
panda|ruckself.file_link).rstrip(".gz") + ".gz"16:24
panda|ruckso it strips the last char in the file, and adds .gx16:24
panda|ruckgz16:24
panda|ruckweshay: so you see 019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz w16:24
panda|ruckin the logs16:24
panda|ruckbmc-console.lo.gz16:24
panda|ruckweshay: one thing we could do could be to make the lgos gzipped from now one16:25
panda|ruckon*16:25
rlandypanda|ruck: weshay: psl review https://review.openstack.org/#/c/639824/16:26
*** gkadam has quit IRC16:27
rlandyweshay: one more https://code.engineering.redhat.com/gerrit/#/c/164089/16:27
panda|ruckweshay: in  fact the function to download the file is tring to handle a lot of exceptions, but none works. So w may trying at source by just renaming the bmc-console.log to bmc-console.log.gz.16:28
weshaypanda|ruck ya.. I had that thought16:30
weshayin collect-logs?16:30
panda|ruckweshay: yes, but we need to change sova too anyway, to change the file name there16:31
panda|ruckweshay: no, nevermind the last line16:31
weshaypanda|ruck I can merge on sova16:31
panda|ruckweshay: oh yes, mind the list line indeed16:31
weshaypanda|ruck ya16:31
panda|ruckweshay: ok, but sova will not be able to analyze the existing bmc_console logs16:32
weshayI think you can put up both changes and we can move forward on it16:32
weshaypanda|ruck join my blue16:32
weshaylet's knock this out16:32
weshaypanda|ruck I think the bmc.log is created by the ovb post playbook?16:34
weshaybut collect logs could rename it16:34
mariosrlandy: fyi got the same on a non kvm box i put a note here for now https://tree.taiga.io/project/tripleo-ci-board/task/766 will revisit next week16:38
mariosrlandy: so using --libvirt i get that http://pastebin.test.redhat.com/728999 - i mean makes sense on a kvm guest but this one (according to beaker anyway) is non virtualized.16:38
rlandymarios: ack - ok- thanks16:45
weshayrlandy ack to merge both16:45
*** ykarel|away has quit IRC16:45
rlandythanks16:45
*** ykarel|away has joined #oooq16:46
panda|ruckweshay:  https://review.rdoproject.org/r/1906916:49
*** irclogbot_0 has joined #oooq16:51
*** jtomasek has quit IRC16:52
weshaypanda|ruck https://github.com/sshnaidm/sova/blob/master/tripleoci/config.py#L23816:54
sshnaidm|offweshay, what is the point to change bmc log name? it should be *.log17:04
weshaysshnaidm|off greetings day off :)17:04
sshnaidm|offweshay, it was an error in yaml, I fixed it today, should be fine in next hours17:04
sshnaidm|offweshay, I get sova pull requests quickly :)17:04
* weshay looks at commits17:05
weshaysshnaidm|off the bmc.log is not getting downloaded via sova17:05
weshayafawct17:05
panda|rucksshnaidm|off: I saw this in sova container log 2019-03-01 16:13:33,577 - watchcat - DEBUG - utils.get_regular_file:244 - File /cidata/82885c3/bmc-console.lo.gz was saved as 40417:05
*** ykarel|away has quit IRC17:06
*** bogdando has quit IRC17:06
weshaywe're chatting in https://bluejeans.com/u/whayutin/17:06
weshaysshnaidm|off I think we need sova fixed to handle .log files that are not zipped17:11
sshnaidm|offweshay, panda|ruck https://github.com/sshnaidm/sova/commit/a0e7555c47c3d14c9613725faa902e525c9713c617:11
* weshay looks17:11
weshaysshnaidm|off sanitize the name.. to always have .gz?17:14
weshaythanks!17:15
*** kopecmartin is now known as kopecmartin|off17:15
weshayzbr you still around?17:19
zbryes, joining now.17:19
*** panda|ruck is now known as panda|ruck|off17:21
sshnaidm|offweshay, they're saved always gzipped to save disk space17:21
zbrhttps://review.openstack.org/#/c/636160/17:25
*** trown is now known as trown|lunch17:31
*** irclogbot_0 has quit IRC17:56
rfolco|roverrlandy, reproducer -l on centos is retrieving "There is no node to run jobs", just checking if this is something you hit before... any clues or this would require some debugging/investigation ?17:59
rlandyrfolco|rover: no, where are you running? on a baremetal machine?18:00
*** dtantsur is now known as dtantsur|afk18:00
rlandyI don;t know how libvirt would work on a node of rdocloud18:00
rlandyI have only tested on a mini-dell18:01
rlandyrfolco|rover: do you have logs/tmate to share?18:01
*** irclogbot_0 has joined #oooq18:02
rfolco|roverrlandy, on a beaker box, just fresh centos18:02
rfolco|roverrlandy, I am not sure I should spend time on this18:02
rlandyrfolco|rover: did you hit the same issues as marios?18:02
*** ykarel|away has joined #oooq18:03
* rfolco|rover not aware of marios issues18:03
rlandy<marios> rlandy: fyi got the same on a non kvm box i put a note here for now https://tree.taiga.io/project/tripleo-ci-board/task/766 will revisit next week18:03
rlandy<marios> rlandy: so using --libvirt i get that http://pastebin.test.redhat.com/728999 - i mean makes sense on a kvm guest but this one (according to beaker anyway) is non virtuali18:03
rlandyrfolco|rover: ^^18:03
*** jpena is now known as jpena|off18:03
rlandyrfolco|rover: if you can run ovb and standalone on rdocloud, you'rep robably good18:04
rfolco|roverrlandy, centos/ovb --> fail, centos/libvirt --> fail18:04
rfolco|roverrlandy, will move to f2818:04
rlandyweird ...18:04
*** jaosorior has quit IRC18:04
rlandyrfolco|rover: k - let me know18:05
rlandywill look into it18:05
rfolco|roverrlandy, thx18:05
*** irclogbot_0 has quit IRC18:05
weshayrlandy I'll put in a check on the script to test virt capabilities if -l is called18:08
weshayrfolco|rover you got The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm ?18:09
rlandyweshay: where did you test virt? bm box, right?18:09
weshayon your bm box?18:09
rlandyThere is no node to run jobs18:09
weshayrlandy matt young's old laptop :)18:09
rlandyweshay: ^^ rfolco|rover"s error18:09
rlandyweshay: lol18:09
weshayya.. I'll try it now18:09
weshayrlandy but a test up front to prevent folks from running -l on a vm would be good18:10
weshayI'll add that18:10
*** irclogbot_0 has joined #oooq18:10
*** amoralej is now known as amoralej|off18:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario002-multinode- (2 more messages)18:10
rlandyweshay: ack18:12
weshayrfolco|rover the only reason I can think of why you'd get that error is a missing kernel package18:17
weshaycan you cat /proc/cpu and see if the flags are there?18:17
rfolco|roverweshay, flags? but this is not nested virt, is it?18:19
weshay cat /proc/cpuinfo | grep vmx18:20
weshayif that returns as a flag, you should not hit The error was: libvirtError: invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm18:20
weshayif it's not there.. you need to enable it the bios18:21
*** derekh has quit IRC18:22
rfolco|roverafaik weshay this is required for nested virt18:22
rfolco|rovervmx I mean18:22
rfolco|roverI have the vms running look18:22
rfolco|rover[rfolco@rdo-ci-fx2-02-s5 ~]$ sudo virsh list --all18:22
rfolco|rover Id    Name                           State18:22
rfolco|rover----------------------------------------------------18:22
rfolco|rover 1     subnode-0                      running18:22
rfolco|rover 2     subnode-1                      running18:22
weshayoh ok18:22
weshayso that's good18:22
weshayhelp me understand where the error was hit?18:22
rfolco|roverfor some reason nodepool did not add to the pool or something?18:23
weshayrfolco|rover which job are you reproducing?18:23
rfolco|roverreproducing undercloud containers18:23
rfolco|roverhttp://logs.openstack.org/41/634241/1/check/tripleo-ci-centos-7-undercloud-containers/76e57c3/logs/18:23
weshayand what was the error?18:23
rfolco|rover /var/tmp/REPRODUCER/roles/ansible-role-tripleo-ci-reproducer/tasks/start.yaml:57 fatal: [localhost]: FAILED! => {"changed": false, "msg": "There is no node to run jobs"}18:24
weshayrlandy unfortunatlely undercloud jobs use the multindoe nodeset18:24
rfolco|roverweshay, there is no node18:24
weshayrfolco|rover are there any containers that are down?18:24
rfolco|roveryes18:25
rfolco|roverzuul18:25
rfolco|roverlol18:25
rfolco|roverce0512164585        rdoci/zuul:stable                 "/usr/bin/dumb-ini..."     36 minutes ago      Exited (0) 36 minutes ago18:25
rlandyhmmm - shouldn't18:25
weshaythat can be fine18:25
weshaywhile jobs are not running.. I've seen that go down and it comes back18:25
* weshay tries that same job http://logs.openstack.org/41/634241/1/check/tripleo-ci-centos-7-undercloud-containers/76e57c3/logs/reproducer-zuul-based-quickstart.tar18:26
rlandyweshay: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/undercloud-jobs.yaml#L22 undercloud job use the singlenode nodeset18:26
rlandyonly libvirt always deploys two nodes18:26
rlandythe fault is in the libvirt template18:27
weshaye0734f1a469e        rdoci/zuul:stable                 "/usr/bin/dumb-ini..."     45 hours ago        Exited (0) 45 hours ago                                                      tripleo-ci-reproducer_gerritconfig_118:27
weshayfor example18:27
weshayrlandy k18:27
weshayrfolco|rover trying now18:27
rfolco|roverweshay, good, I'm looking at the logs while my local f27-f28 upgrade goes in my local laptop env18:28
rfolco|roverhell docker-compose logs empty18:35
rfolco|roverdoing a manual [rfolco@rdo-ci-fx2-02-s5 tripleo-ci-reproducer]$ docker-compose up -d18:36
*** trown|lunch is now known as trown18:37
weshayrfolco|rover bbiab but it's working for me18:45
rfolco|roverof course :(18:46
rfolco|roverweshay, fedora?18:46
rfolco|roversay yes say yes18:46
*** ykarel|away has quit IRC18:48
rfolco|roverweshay, magically after running docker-compose up and then re-running the reproducer, I was able to run the job19:05
rfolco|roverrlandy, ^19:05
rfolco|roverhttp://rdo-ci-fx2-02-s5:9000/t/tripleo-ci-reproducer/status19:05
rlandyhow nice19:05
rfolco|roversomething happened with docker-compose I think19:06
*** rfolco|rover has quit IRC19:21
*** tosky has quit IRC19:42
*** tosky has joined #oooq19:43
*** dmellado has quit IRC19:44
*** dmellado has joined #oooq19:45
*** irclogbot_0 has quit IRC19:50
*** rlandy is now known as rlandy|brb19:57
*** irclogbot_0 has joined #oooq20:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages)20:10
*** derekh has joined #oooq20:26
*** rfolco has joined #oooq20:37
*** rlandy|brb is now known as rlandy20:51
rlandyrfolco: hoe's it going - need me to look at anything?21:16
rfolcorlandy, trying on laptop.... will see in a bit21:17
rlandyon your laptop - that is brave21:17
weshayback21:27
weshaythe docker-compose.yaml was empty?21:28
weshaymy undercloud failed to install21:29
weshayreally not a fan of the containre's prepare in tripleo-common21:31
rfolcorlandy, where do you run your repro? vm ?21:32
*** irclogbot_0 has quit IRC21:37
rlandyrfolco: what do mean by my repo?21:37
rlandyoh my reproducer21:37
rlandyon my mini-dell21:37
rlandyso you have a log or error?21:38
weshayrfolco fyi /me made some comments21:38
rfolcoweshay, ok, thanks.21:39
weshayrlandy fyi https://review.openstack.org/64053821:43
weshaypanda++21:45
weshaysshnaidm++21:45
rlandylooking21:46
rlandythere is such and env var as LIBVIRT_ERROR??21:48
rlandynot defined anywhere21:48
rlandyweshay: ^^21:48
rlandyoh I see what you did21:49
rlandyforget that21:49
weshay?21:49
weshayoh21:49
weshaythe ||21:49
rlandyI see21:51
rlandytested21:51
rlandylooks fine21:51
* rlandy needs to read the whole line in order21:51
rlandyit's not == true21:52
weshayaye21:54
weshay- pip conflicts: paramiko, requests, httpd2lib, etc.21:56
weshay?21:56
rfolcoif not too late, https://review.openstack.org/#/c/64053821:57
* weshay wonders if we should use a vm21:57
rfolcoweshay, ^21:57
weshayvirtenv21:57
weshayI mean21:57
* weshay looks21:57
rfolcoweshay, prevent -l on vm... so just block if you run from a vm... a vm may or not have vmx21:59
weshayrfolco ya.. I think for now we can limit the scope of what we support.. I need to see if we can test non vmx/smx on a regular basis21:59
rfolcok22:00
weshaynon virt capable boxes have to pretty old at this point I think22:01
weshayyou know better than me on this one.. though.. I would rather have someone update the bios to allow vmx/smx22:01
* weshay doesn't want to support the world.. want to get this working well22:02
weshayrfolco were you able to resolve the pip conflicts?22:02
rfolcoweshay, nah, I am moving to vm, f* my laptop22:03
* weshay isn't 100% clear on why we shouldn't use a virtenv.. but I think there was one22:03
rfolcoweshay, my point on your patch is: IMHO you should just check if running on a vm, and bock -l. The way you're checking vmx is not what you need. Try finding something else to check instead of vmx.22:06
weshayif it's a host w/o vmx/smx I'm not sure we want to allow it22:07
weshayas well as not allowing -l from a guest22:08
weshayif that is sane22:08
rfolcoso 2 conditions22:10
*** rlandy has quit IRC22:10
rfolco1) block host if not kvm capable22:10
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp- (3 more messages)22:11
rfolco2) block vm (even if nested virt capable)22:11
rfolcoI was focused on #222:11
weshayah k22:11
weshaythanks for clarifying that22:11
weshayI'll update it22:11
rfolcothats a command also... virt-host-validate22:11
rfolcomuch more complete22:11
weshayugh.. can you make two pull requests w/o them being stacked?22:21
weshaygithub sucks.22:24
weshayconfirmed.. it sucks22:34
*** vinaykns has quit IRC22:36
weshayrfolco fyi.. https://bugs.launchpad.net/tripleo/+bug/181830522:36
openstackLaunchpad bug 1818305 in tripleo "overcloud-full image fails to build calling mkfs -t xfs " [Critical,Triaged]22:36
weshaytwo patches added to sova https://github.com/sshnaidm/sova/pull/6422:36
weshayno action required22:37
*** ajo_ has joined #oooq22:45
*** Tengu_ has joined #oooq22:47
weshayrfolco just fyi.. undercloud install is failing twice on the repoducer at22:50
weshay2019-03-01 22:31:52.356 27500 WARNING tripleoclient.v1.tripleo_deploy.Deploy [  ] TASK [Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log] ***22:50
weshaywhich usually means the docker containers did not download22:50
weshaybut looking at it..22:51
weshayrfolco unless that patch really broke something22:52
weshayand it may have22:52
*** Tengu has quit IRC22:52
*** vkmc has quit IRC22:52
*** ajo has quit IRC22:52
*** ajo_ is now known as ajo22:52
weshayrfolco everythign failed on that patch22:53
weshayexcept amazing undercloud upgrades22:53
*** vkmc has joined #oooq22:53
weshaywhich makes me think the patch was not applied22:53
weshayrfolco you may want to try a recreate from a successful job22:53
weshayfirst22:53
*** rascasoft has quit IRC23:07

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!