| *** tosky has quit IRC | 00:23 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci- (2 more messages) | 01:03 |
|---|---|---|
| *** jaosorior has quit IRC | 01:20 | |
| *** jaosorior has joined #oooq | 01:32 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario001-multinode-oooq-container @ (2 more messages) | 03:03 |
| *** ykarel has joined #oooq | 03:19 | |
| *** rlandy has quit IRC | 03:28 | |
| *** skramaja has joined #oooq | 03:34 | |
| *** raukadah is now known as chandankumar | 03:41 | |
| *** udesale has joined #oooq | 03:54 | |
| *** ratailor has joined #oooq | 03:56 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo- (2 more messages) | 05:03 |
| *** saneax has joined #oooq | 05:36 | |
| quiquell | ykarel: o/ | 06:45 |
| quiquell | ykarel: next one for the docker login https://review.rdoproject.org/r/#/c/18993/ | 06:45 |
| quiquell | sshnaidm: ^ | 06:45 |
| quiquell | I think we don't need the python3-docker as zuul ansible_python_interpreter is harcoded to python2 (will see in the future) | 06:46 |
| ykarel | quiquell, so when u reproduced with local zuul, python-docker was installed there? | 06:55 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo- (2 more messages) | 07:03 |
| quiquell | ykarel: did have the same problem and do the same fix but though it was something missing | 07:04 |
| quiquell | But looks like it was really reproducing it | 07:04 |
| *** skramaja has quit IRC | 07:11 | |
| *** jfrancoa has joined #oooq | 07:19 | |
| *** kopecmartin|off is now known as kopecmartin | 07:23 | |
| chandankumar | kopecmartin: Hello | 07:24 |
| kopecmartin | chandankumar, hi | 07:24 |
| *** jtomasek has joined #oooq | 07:25 | |
| chandankumar | kopecmartin: regarding minimizing the os_tempest varshttps://review.openstack.org/#/c/639258/ does this approach looks ok? | 07:25 |
| quiquell | ykarel: What I don't know why this was not appearing with the old password | 07:25 |
| chandankumar | or you need something different | 07:25 |
| chandankumar | kopecmartin: it will help to put less vars in the playbook | 07:26 |
| quiquell | ykarel: we have no_logs: true but tristanC show a "unauthorize" error | 07:26 |
| quiquell | ykarel: let's merge this config project and see | 07:26 |
| quiquell | ykarel: if it works we can go back to the other secret | 07:26 |
| *** skramaja has joined #oooq | 07:26 | |
| ykarel | quiquell, ack | 07:27 |
| kopecmartin | chandankumar, better, let's see what other reviewers think | 07:33 |
| chandankumar | kopecmartin: I am think another approach also standardize the playbook in tqe itself and reuse that | 07:34 |
| chandankumar | *thinking | 07:34 |
| chandankumar | at all places | 07:34 |
| quiquell | ykarel: Can we rescheduler periodic-tripleo-centos-7-master-containers-build-push ? | 07:38 |
| ykarel | quiquell, you can run in check pipeline against testproject | 07:38 |
| quiquell | ykarel: secret will not work there | 07:39 |
| ykarel | why? | 07:39 |
| quiquell | ykarel: if have to be from the same project where the secret is stored | 07:39 |
| quiquell | ykarel: Or that I though | 07:39 |
| quiquell | ykarel: but let's try it | 07:39 |
| ykarel | quiquell, i don't think pipeline restrict using secrets | 07:40 |
| ykarel | it should work | 07:40 |
| quiquell | marios, ykarel: Let's merge this https://review.rdoproject.org/r/#/c/18913/ | 07:48 |
| quiquell | TO be able to run tripleo-build-containers-fedora-28 at third party check jobs | 07:48 |
| marios | quiquell: ack will check in bit | 07:48 |
| quiquell | marios: Let's see if we can start to flow the f28 stuff | 07:49 |
| quiquell | marios: RDO login working at check | 07:50 |
| quiquell | ykarel: ^ you were right secrets can be exercise at check | 07:50 |
| ykarel | quiquell, ack | 07:52 |
| ykarel | quiquell, that patch seems not ready to merge | 07:54 |
| marios | zbr|out: quiquell duplicate? https://tree.taiga.io/project/tripleo-ci-board/task/746?kanban-status=1447276 https://tree.taiga.io/project/tripleo-ci-board/task/743?kanban-status=1447276 | 07:54 |
| ykarel | https://review.rdoproject.org/r/#/c/18913/ | 07:54 |
| *** chem` has joined #oooq | 07:55 | |
| *** ykarel is now known as ykarel|lunch | 07:56 | |
| marios | quiquell: lets merge then https://review.openstack.org/#/c/637212/ ? | 07:56 |
| quiquell | marios: +w | 07:56 |
| quiquell | I don't know if we can +w but, let's push a little this thing | 07:56 |
| marios | quiquell: the one you pointed to isn't even in taiga can't see it | 07:56 |
| marios | https://review.rdoproject.org/r/#/c/18913/ i mean quiquell | 07:57 |
| *** chem has quit IRC | 07:57 | |
| quiquell | marios: is in the Depends-On of https://tree.taiga.io/project/tripleo-ci-board/task/725?kanban-status=1447276 | 07:57 |
| quiquell | https://review.openstack.org/636160 | 07:57 |
| marios | quiquell: rebase ? https://review.rdoproject.org/r/#/c/18976/ | 07:57 |
| marios | quiquell: i don't see it in https://tree.taiga.io/project/tripleo-ci-board/task/725 | 07:59 |
| marios | quiquell: added there | 07:59 |
| quiquell | marios: rebased | 08:00 |
| quiquell | marios: thanks for adding | 08:00 |
| marios | quiquell: moved it added comment to 691 | 08:01 |
| *** ccamacho has joined #oooq | 08:02 | |
| quiquell | zbr|out: I am going to remove the files section from https://review.rdoproject.org/r/#/c/18913/ | 08:02 |
| quiquell | zbr|out: we can do that at other review for more project if we want, the amount of stuff we check to exercise that is minimal | 08:02 |
| quiquell | Have to leave for a few | 08:03 |
| *** quiquell is now known as quiquell|brb | 08:03 | |
| *** holser_ has joined #oooq | 08:04 | |
| quiquell|brb | ykarel|lunch: rever to try previos rdo_registry secret but with python-docker https://review.rdoproject.org/r/18995 | 08:07 |
| *** rascasoft has joined #oooq | 08:08 | |
| marios | quiquell|brb: soft -1 https://review.rdoproject.org/r/#/c/18913/ cos i don't get it yet | 08:15 |
| marios | quiquell|brb: can you check comments when you get a sec please | 08:15 |
| marios | quiquell|brb: i mean i get we want to run f28 job in non vote in rdo fine | 08:15 |
| marios | quiquell|brb: don't follow the files: bit yet | 08:16 |
| marios | quiquell|brb: and a bit confusing wrt having a template called container-build but it only does f28 | 08:16 |
| *** quiquell|brb is now known as quiquell | 08:23 | |
| quiquell | marios: I am back, let's do a bluesync | 08:23 |
| marios | quiquell|brb: gimme few mins pls? lets say in 35 mins at the start of th ehour? | 08:23 |
| quiquell | marios: sure no problem | 08:24 |
| marios | quiquell: thx | 08:24 |
| *** holser_ has quit IRC | 08:32 | |
| *** holser_ has joined #oooq | 08:33 | |
| quiquell | ykarel|lunch: removed files section https://review.rdoproject.org/r/#/c/18913/ | 08:37 |
| *** tosky has joined #oooq | 08:39 | |
| *** jpena|off is now known as jpena | 08:42 | |
| *** amoralej|off is now known as amoralej | 08:46 | |
| *** dtantsur|afk is now known as dtantsur | 08:46 | |
| *** ccamacho has quit IRC | 08:58 | |
| marios | quiquell: call? | 09:00 |
| marios | quiquell: bluejeans on invite | 09:00 |
| quiquell | sure sure sorry | 09:00 |
| *** bogdando has joined #oooq | 09:02 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo- (2 more messages) | 09:03 |
| *** ykarel|lunch is now known as ykarel | 09:10 | |
| ykarel | quiquell, ack | 09:10 |
| ykarel | quiquell, btw is that job ready? i see failures in last run | 09:12 |
| ykarel | INFO:kolla.common.utils.base:file: file:///etc/yum.repos.d/delorean-master-testing.repo, line: 1 INFO:kolla.common.utils.base:'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n' | 09:12 |
| quiquell | ykarel: nope is like chicken egg | 09:20 |
| quiquell | ykarel: we need stuff upstream but upstream need the job to verify | 09:20 |
| quiquell | ykarel: so we have to merge this failing as non-voting so we can iterate the upstream missing piece | 09:20 |
| ykarel | quiquell, Depends-On on upstream patch not working? | 09:21 |
| quiquell | ykarel: But then it will not merge | 09:21 |
| quiquell | ykarel: I can do a testproject review with that | 09:21 |
| quiquell | ykarel: since we are not merging upstreawm without running this | 09:21 |
| quiquell | ykarel: this is the upstreawm change https://review.openstack.org/#/c/636160/ | 09:22 |
| quiquell | ykarel: I want to have this containers-build f28 job as third party there | 09:22 |
| quiquell | ykarel: to verify the review | 09:22 |
| marios | quiquell: probably worth adding anote about that in the commit message added a comment at https://review.rdoproject.org/r/#/c/18913/ | 09:22 |
| quiquell | marios, ykarel: now everything you need to merge this is in the comment I think | 09:29 |
| quiquell | creatd a test review with the upstream change (could be that is failing since upstream change is WIP) | 09:30 |
| ykarel | quiquell, yup good to get job ready | 09:33 |
| ykarel | it's good to not run a broken new job to save some ci resources | 09:33 |
| quiquell | ykarel: you mean we can merge this without testproject running it ? | 09:34 |
| quiquell | ykarel: so we iterate in the upstream change ? | 09:35 |
| quiquell | ykarel: removed test review let's merge the job https://review.rdoproject.org/r/18913 | 09:36 |
| ykarel | quiquell, i mean get the job ready with testproject, merge the change to enable the job, recheck upstream to show that job is working, get upstream merged | 09:36 |
| quiquell | well number of resources is going to be the same | 09:37 |
| quiquell | and having the job directly in the WIP review will help us | 09:38 |
| quiquell | ykarel: the job have a files section with only stuff from the WIP | 09:38 |
| ykarel | quiquell, broken job will run against other reviews as well | 09:38 |
| ykarel | whereever files: is modified | 09:38 |
| quiquell | at config yu mean | 09:38 |
| *** ccamacho has joined #oooq | 09:39 | |
| quiquell | sorry no we don't have at config | 09:39 |
| quiquell | ykarel: let me share the parent | 09:39 |
| quiquell | ykarel: files from the parent https://softwarefactory-project.io/zuul/t/rdoproject.org/job/tripleo-build-containers-base-with-files | 09:40 |
| quiquell | ykarel: if even with the files section it has you see is not restrictive enough so we take too much resources | 09:40 |
| quiquell | ykarel: then we just test it in testproject | 09:41 |
| ykarel | quiquell, ack | 09:44 |
| quiquell | ykarel: so it's restrictive enough an we can merge the job ? | 09:44 |
| *** derekh has joined #oooq | 09:46 | |
| ykarel | quiquell, so if we merge the job, f28 broken container build job will be running in all reviews which are changing files defined in files: section | 09:47 |
| quiquell | ykarel: yep | 09:47 |
| quiquell | at tripleo-ci | 09:48 |
| ykarel | quiquell, do you want to run the broken job in other reviews(apart from the WIP patch) | 09:48 |
| quiquell | ykarel: future reviews of course | 09:48 |
| ykarel | quiquell, i am asking do you want to run the broken job in reviews being proposed until job is ready? | 09:49 |
| ykarel | after job is ready it makes sense to run it | 09:49 |
| quiquell | ykarel: nah, ack | 09:50 |
| quiquell | ykarel: it will takes more time from us to merge the wip though, since we need to run two revies :-( | 09:50 |
| quiquell | ykarel: but yes is going to take more resources | 09:50 |
| quiquell | ykarel: let's see how bad the the state is though | 09:51 |
| ykarel | quiquell, i would suggest to avoid running jobs not necessary by modifying zuul.d | 09:51 |
| ykarel | quiquell, this way you will get early feedback and save resources too | 09:51 |
| quiquell | ykarel: ok | 09:52 |
| quiquell | ykarel: so going back to the test review, don't know why is not running the job https://review.rdoproject.org/r/18997 | 09:52 |
| ykarel | quiquell, can you try renaming file to .zuul.yaml | 09:56 |
| quiquell | sure | 09:56 |
| quiquell | ykarel: about the other issue with RDO registry secret -> to revert to use the previous secret https://review.rdoproject.org/r/#/c/18995/ | 09:59 |
| quiquell | ykarel: so we check if the python-docker was just the issue | 09:59 |
| panda|ruck | 'morning | 10:01 |
| quiquell | panda|ruck: o/ | 10:01 |
| ykarel | quiquell, ack, but it's still not clear, u said with local zuul the job worked, can't get why it worked there without python-docker | 10:02 |
| ykarel | or it was installed somehow? | 10:02 |
| quiquell | ykarel: I need to install there too, but I though it was something wrong with the reproducer it self since we didn't test this jobs there | 10:03 |
| quiquell | ykarel: so it discovered legit issue | 10:03 |
| ykarel | quiquell, ack | 10:04 |
| quiquell | ykarel: so local zuul was "python-docker + new secret" I cannot check old secret there | 10:04 |
| quiquell | ykarel: since the local zuul cannot decrypt it | 10:04 |
| ykarel | quiquell, ack, you didn't shared python-docker thing earlier, that was confusing me | 10:04 |
| ykarel | i +2ed the revert | 10:05 |
| quiquell | ykarel: I think I shared i but badly | 10:05 |
| quiquell | ykarel: htanks for +2, we wait for zuul to +w ? | 10:05 |
| ykarel | possibly | 10:05 |
| ykarel | panda|ruck, rfolco|rover can you check https://review.openstack.org/#/c/637196, it has broken ceph jobs | 10:06 |
| ykarel | for all releases | 10:07 |
| panda|ruck | ykarel: mmmhh | 10:10 |
| panda|ruck | marios: sorry I missed the meeting, I had to solve a bank problem this morning | 10:10 |
| panda|ruck | marios: anythin interesting ? | 10:10 |
| marios | panda|ruck: np yeah some plan forward we can catch up on the community call later | 10:16 |
| marios | panda|ruck: basically merge https://review.rdoproject.org/r/#/c/18913/ so we can see f28 as rdo job in https://review.openstack.org/#/c/636160 | 10:17 |
| quiquell | marios, panda|ruck: Well ykarel prefer that we fix that at testproject | 10:20 |
| quiquell | instead of merge a broken job to RDO | 10:20 |
| marios | quiquell: ok sorry wasn't following | 10:20 |
| marios | panda|ruck: ^ | 10:20 |
| quiquell | marios: ya it's new | 10:20 |
| marios | panda|ruck: and then for the authentication quiquell reckons it is som other unrelated error we missed i.e. missing package | 10:21 |
| quiquell | ykarel: zuul is ok now in the revert of secrets https://review.rdoproject.org/r/#/c/18995/ | 10:21 |
| marios | quiquell: so you mean, it is definitely not to do with the new secret | 10:21 |
| * marios looks | 10:21 | |
| quiquell | marios: no no yet, we have to revert and then test | 10:21 |
| quiquell | marios: since is config project we have to merge to test | 10:22 |
| ykarel | quiquell, +W | 10:22 |
| quiquell | ykarel: ack thanks, let's relaunch the test | 10:22 |
| quiquell | ykarel: btw test project for the f28 containers-build not working https://review.rdoproject.org/r/#/c/18997/ | 10:23 |
| quiquell | ykarel: don't know hy is not launching the job there | 10:23 |
| ykarel | quiquell, because of files: | 10:23 |
| ykarel | from parent | 10:23 |
| quiquell | ykarel: dam... right | 10:24 |
| quiquell | let make a fake one | 10:24 |
| panda|ruck | ykarel: I don't understand, what do you prefer to not merge now ? | 10:24 |
| panda|ruck | ykarel: the build containers job ? | 10:24 |
| ykarel | panda|ruck, yes quiquell said it's not ready yet, so i said it's good to not run broken job | 10:25 |
| ykarel | good to get it ready first, then run against all upstream changes | 10:25 |
| panda|ruck | ykarel: ok | 10:25 |
| quiquell | panda|ruck: make sense | 10:25 |
| quiquell | It's a little more work having two reviews but is ok | 10:26 |
| panda|ruck | quiquell: yesterday in the meeting I suggested to duplicate the job definition from upstream, and not parent it. So we can understand if the secrets problem is really about parenting | 10:26 |
| quiquell | panda|ruck: secrets is working | 10:26 |
| quiquell | panda|ruck: renting was not the issue I tested that at reproducer | 10:26 |
| quiquell | panda|ruck: but I think I was following a red herring with the secret | 10:26 |
| quiquell | panda|ruck: as you pointed out from the beginning python-docker is needed | 10:27 |
| quiquell | was failing there in the periodics | 10:27 |
| quiquell | panda|ruck: https://review.rdoproject.org/r/#/c/18994/ <- test review | 10:27 |
| quiquell | panda|ruck: we have revert to use the previous secet, will re-run now | 10:27 |
| *** skramaja_ has joined #oooq | 10:27 | |
| *** skramaja has quit IRC | 10:29 | |
| panda|ruck | quiquell: AH! | 10:34 |
| panda|ruck | quiquell: so how was the registry receiving the request without python docker ? ... mmhh to many information fro different person, maybe the log wass from something else | 10:35 |
| quiquell | panda|ruck: I think the log that they provide us with authentication error was other thing | 10:36 |
| quiquell | panda|ruck: and was the principal red herring | 10:36 |
| quiquell | panda|ruck: I am checking now with the old secret | 10:36 |
| panda|ruck | I always get suprised on how right I am each time. Despite that, nobody believes me in the first time. :) Cassandra complex. | 10:36 |
| quiquell | Passandra | 10:37 |
| quiquell | panda|ruck: RDO loging working with previous secret https://softwarefactory-project.io/zuul/t/rdoproject.org/stream/9d3390f78be243e4ab6ae2fa5571018b?logfile=console.log | 10:37 |
| quiquell | panda|ruck: so was just the python-docker thing <- I though reproducer was broken but it was yelling the right thing to me | 10:38 |
| quiquell | panda|ruck, marios: So now conatiner push jobs should work | 10:40 |
| panda|ruck | yay | 10:40 |
| quiquell | panda|ruck: now we need the tripleo-repo thing ? | 10:40 |
| *** holser_ is now known as holser|lunch | 10:52 | |
| panda|ruck | ykarel: is this new to you ? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/9e95dc8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-26_01_50_41 | 10:52 |
| panda|ruck | ykarel: I don't see any bugs opened | 10:52 |
| ykarel | panda|ruck, looking | 10:53 |
| ykarel | panda|ruck, this is fixed | 10:53 |
| ykarel | should not see it in recent run, fix got merged 15 hours ago | 10:54 |
| marios | quiquell: \o/ /me happy to see the logs in http://logs.rdoproject.org/94/18994/1/check/periodic-tripleo-centos-7-master-containers-build-push/09fe5dd/ from your test | 10:54 |
| panda|ruck | ykarel: do you have a bug number ? | 10:55 |
| panda|ruck | ykarel: this is from last periodic run | 10:55 |
| ykarel | panda|ruck, current run is passing | 10:55 |
| ykarel | panda|ruck, https://review.openstack.org/#/c/639021/ fixed it, don't think there is bug for it | 10:56 |
| quiquell | marios: we have to listen panda more :-) he was talking about the python-docker from the beginning | 10:56 |
| quiquell | marios: so next stuff is https://tree.taiga.io/project/tripleo-ci-board/task/773?kanban-status=1447275 ? | 10:57 |
| marios | quiquell: yeah i think is almost ready aim to remove wips today | 10:58 |
| panda|ruck | ykarel: ok thanks. | 10:58 |
| marios | quiquell: but first food | 10:59 |
| panda|ruck | 3 dashboards with the same information at different levels of update | 10:59 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo- (2 more messages) | 11:03 |
| panda|ruck | ykarel: for https://review.openstack.org/637196 sshnaidm already suggested to use release, but didn't work. Bu we need to use release, so I have to investigate what were they doing wrong | 11:05 |
| ykarel | panda|ruck, ack | 11:05 |
| panda|ruck | ah yeah, this is a role in the executor, we are not passing release at that stage | 11:05 |
| panda|ruck | damn | 11:06 |
| marios | quiquell: | 11:08 |
| quiquell | yes ? | 11:09 |
| panda|ruck | quiquell: he's speechless | 11:09 |
| panda|ruck | ykarel: dod you open a bug for this problem ? | 11:10 |
| ykarel | panda|ruck, nope | 11:10 |
| panda|ruck | ykarel: ok opening | 11:10 |
| ykarel | ack | 11:10 |
| panda|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1817688 | 11:14 |
| openstack | Launchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:14 |
| ykarel | panda|ruck, ack | 11:14 |
| quiquell | panda|ruck: I am looking into https://tree.taiga.io/project/tripleo-ci-board/task/595?kanban-status=1447274 | 11:15 |
| quiquell | panda|ruck: But don't get the URL trigger there | 11:16 |
| quiquell | panda|ruck: do you have some context ? | 11:16 |
| panda|ruck | quiquell: apparently there's a url trigger for zuul taht is going to be merged soon, and we were eveluating the possibility to use it. | 11:17 |
| quiquell | panda|ruck: cool ! | 11:17 |
| panda|ruck | quiquell: but everybody agrees that it may not be the best move | 11:17 |
| arxcruz | quiquell: http://paste.openstack.org/show/746154/ | 11:17 |
| quiquell | panda|ruck: but I don't see the relation between that and selectin the release file to use | 11:17 |
| panda|ruck | quiquell: because it's new, nobody know how it works, and in the end may not even provide the right functionality | 11:17 |
| panda|ruck | quiquell: maybe it's in the wrong user story ? | 11:18 |
| quiquell | arxcruz: you have to wait it take a little the first time | 11:18 |
| quiquell | panda|ruck: I see it here too https://tree.taiga.io/project/tripleo-ci-board/us/587?milestone=220350 | 11:18 |
| quiquell | panda|ruck: ^ that make sense | 11:18 |
| quiquell | panda|ruck: but the other is about using correct release file I think, have to ask rlandy | 11:18 |
| quiquell | marios: this was good to merge ? https://review.openstack.org/#/c/637212 | 11:21 |
| marios | quiquell: it merged | 11:22 |
| arxcruz | quiquell: it fails | 11:22 |
| quiquell | ack | 11:22 |
| arxcruz | wondering if i need to open the 9000 door internally | 11:22 |
| quiquell | marios: going to move those task to move they point to the same review | 11:22 |
| quiquell | arxcruz: tmux ? | 11:22 |
| quiquell | arxcruz: tmate I m ean | 11:22 |
| marios | quiquell: ? sorry don't follow | 11:22 |
| arxcruz | quiquell: can wait? lunching | 11:22 |
| quiquell | arxcruz: /me lunch too | 11:23 |
| quiquell | marios: https://tree.taiga.io/project/tripleo-ci-board/task/743?kanban-status=1447276 and https://tree.taiga.io/project/tripleo-ci-board/task/746?kanban-status=1447276 | 11:23 |
| quiquell | point to the same review that have just merge | 11:24 |
| marios | quiquell: yes i commented therethis morning about that | 11:25 |
| marios | quiquell: /me shrugs | 11:25 |
| quiquell | marios: ok, let's wait for zbr|out | 11:25 |
| marios | quiquell: same here https://tree.taiga.io/project/tripleo-ci-board/task/691 | 11:26 |
| *** udesale has quit IRC | 11:29 | |
| quiquell | marios, panda|ruck: Do you know why don't we download the tarball of kolla refspec like https://review.openstack.org/changes/624838/revisions/a09bc90769c4c976b2eb79e4dc4f82275ee11ef9/archive?format=tbz2 ? | 11:30 |
| quiquell | marios: instead of checking refspec ? | 11:30 |
| *** ratailor has quit IRC | 11:31 | |
| panda|ruck | quiquell: to just apply the patch I guess | 11:31 |
| quiquell | rfolco|rover: ping | 11:31 |
| panda|ruck | quiquell: and not the whole code | 11:31 |
| quiquell | panda|ruck: well apply fails | 11:31 |
| quiquell | I suppose base is different ? | 11:32 |
| panda|ruck | quiquell: it was updated probably and now conflicts | 11:32 |
| marios | quiquell: i do not | 11:33 |
| quiquell | panda|ruck: is not a bad thing at least it means that those reviews have to be rebased | 11:35 |
| panda|ruck | ykarel: ceph is not failing jsut in master | 11:36 |
| panda|ruck | ykarel: scenario1 and 4 failing too | 11:36 |
| panda|ruck | ykarel: I mean in stable branches | 11:36 |
| panda|ruck | ykarel: scen1/4 failing in master too | 11:36 |
| panda|ruck | '.join(holders))\\nceph_disk.main.Error: Error: Device /dev/loop3 is in use by a device-mapper mapping (dm-crypt?): dm-0,dm-1,dm-2\", \"stderr_lines\": [\"+/entrypoint.sh:19: case \\\"$KV_TYPE\\\" | 11:36 |
| ykarel | panda|ruck, i commented for that too, and linked a review | 11:36 |
| panda|ruck | ykarel: mmmhh | 11:37 |
| ykarel | panda|ruck, https://review.openstack.org/547682 should fix master | 11:37 |
| ykarel | i commented ^^ in the patch which caused issue | 11:37 |
| panda|ruck | ah yeah, the non voting sstep | 11:37 |
| panda|ruck | ok | 11:37 |
| panda|ruck | yep yep I remember now the blue store | 11:37 |
| panda|ruck | but again no promotion even if the barbican issue (or what was related) is solved now | 11:38 |
| ykarel | yup sequence of blockers | 11:39 |
| panda|ruck | I rucks at this job. | 11:39 |
| ykarel | panda|ruck, i seen fs020 is also broken since 19th or even before | 11:39 |
| ykarel | not sure if it's already known and being worked upon | 11:40 |
| ykarel | should be broken with ovn switch | 11:40 |
| panda|ruck | ykarel: running tempest now. | 11:40 |
| ykarel | it will fail eventually | 11:40 |
| ykarel | seeing last couple of runs | 11:41 |
| panda|ruck | ykarel: why so negative ? :) | 11:43 |
| panda|ruck | ykarel: I see 6 tempest failures there. | 11:43 |
| ykarel | yup dhcp agent related | 11:43 |
| ykarel | which i think don't exist in ovn | 11:43 |
| ykarel | but good to ask ovn guys | 11:43 |
| panda|ruck | ykarel: lol, from what I hear they've been bombarded with these kind of requests. | 11:44 |
| panda|ruck | ykarel: "my car didn't start this morning, can you check if it's OVN fault ?" | 11:44 |
| ykarel | :) | 11:45 |
| panda|ruck | ykarel: trying to see if there's a bug | 11:45 |
| ykarel | ack | 11:45 |
| quiquell | panda|ruck: do you know why we don't do just a checkout instead of cherry-pick ? | 11:48 |
| quiquell | arxcruz: I am in | 11:48 |
| *** ccamacho has quit IRC | 11:50 | |
| *** jpena is now known as jpena|lunch | 11:57 | |
| *** amoralej is now known as amoralej|lunch | 11:58 | |
| panda|ruck | quiquell: I think the basic idea there is to update the patch folliwing the kolla updates, otherwise we may miss important kolla updates. | 11:59 |
| panda|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1817705 | 11:59 |
| openstack | Launchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:59 |
| panda|ruck | marked as promotion blocker | 11:59 |
| quiquell | panda|ruck: ack | 11:59 |
| ykarel | panda|ruck, ack | 12:00 |
| quiquell | panda|ruck: cherry pick is not going to give you the parents | 12:00 |
| panda|ruck | said alfred to bruce wayne | 12:01 |
| panda|ruck | quiquell: what parents ? | 12:01 |
| quiquell | panda|ruck: https://review.openstack.org/#/c/624838/ | 12:01 |
| quiquell | is rebased already | 12:02 |
| quiquell | have to chek but could be that parents are the problem | 12:02 |
| panda|ruck | parents are always the problem. Kids are born with a lot o potential most of the time. | 12:03 |
| quiquell | jaja | 12:03 |
| quiquell | have just tested cherry picking the parents it works | 12:04 |
| quiquell | we need more logic there :-/ | 12:04 |
| *** quiquell is now known as quiquell|lunch | 12:04 | |
| *** rfolco|rover has quit IRC | 12:11 | |
| *** rfolco|rover has joined #oooq | 12:11 | |
| *** ccamacho has joined #oooq | 12:12 | |
| *** quiquell|lunch is now known as quiquell | 12:15 | |
| quiquell | rfolco|rover: o/ do you have a minute ? | 12:15 |
| rfolco|rover | quiquell, one sec | 12:20 |
| quiquell | sure | 12:20 |
| *** holser|lunch is now known as holser_ | 12:22 | |
| *** holser__ has joined #oooq | 12:25 | |
| *** sanjayu_ has joined #oooq | 12:26 | |
| *** rf0lc0 has joined #oooq | 12:26 | |
| rf0lc0 | quiquell, sorry, what's up man? | 12:26 |
| quiquell | rf0lc0: was testing the f28 containers build here http://logs.rdoproject.org/97/18997/14/check/tripleo-build-containers-fedora-28-test/d0b5143/job-output.txt.gz | 12:28 |
| sshnaidm | arxcruz, chandankumar guys, where is skip list defined? | 12:28 |
| sshnaidm | I mean for release | 12:28 |
| quiquell | rf0lc0: It fails at cherry-pick since the kolla review has unmerged parents | 12:28 |
| arxcruz | sshnaidm: in validate-tempest/vars | 12:28 |
| *** dmellado_ has joined #oooq | 12:28 | |
| quiquell | rf0lc0: SO question is, why don't we doo checkout + rebase ? | 12:28 |
| sshnaidm | arxcruz, thanks | 12:28 |
| rf0lc0 | quiquell, there are 3 kolla patches to apply, not sure if the other 2 merged. Just one won't merge, ever. | 12:29 |
| *** gouthamr has quit IRC | 12:30 | |
| *** rnoriega has quit IRC | 12:30 | |
| quiquell | rf0lc0: I am able to apply the three of them | 12:30 |
| *** hubbot1 has quit IRC | 12:30 | |
| *** dmellado has quit IRC | 12:30 | |
| *** rfolco|rover has quit IRC | 12:30 | |
| *** holser_ has quit IRC | 12:30 | |
| *** saneax has quit IRC | 12:30 | |
| *** dmellado_ is now known as dmellado | 12:30 | |
| quiquell | rf0lc0: but we are applying only the last one | 12:30 |
| quiquell | rf0lc0: but we we cannot do checkout + rebase ? we have to skip any of the parents ? | 12:30 |
| sshnaidm | arxcruz, which jobs are running full tempest? 021 and 020? | 12:31 |
| rf0lc0 | quiquell, I was doing checkout before, zbr|out may have string opinion about cherry-picking instead. | 12:31 |
| *** rnoriega has joined #oooq | 12:31 | |
| rf0lc0 | quiquell, do whatever you need. The only restriction is: apply the kolla patch that is in the playbook, and the other two you can test in the job, but these will merge at some point | 12:32 |
| *** gouthamr has joined #oooq | 12:33 | |
| rf0lc0 | quiquell, so if the question is: can we change to checkout+rebase? I'd say yes. marios suggested the same if I remember right. | 12:33 |
| *** rf0lc0 is now known as rfolco|rover | 12:33 | |
| arxcruz | sshnaidm: yes, 020 with skip list, and 021 without | 12:37 |
| quiquell | rfolco|rover, zbr|out, marios: updating to checkout+rebase to make it work | 12:41 |
| rfolco|rover | quiquell, gracias | 12:41 |
| quiquell | first make it work then we can change it back to whatever we want | 12:42 |
| rfolco|rover | quiquell, it worked before in previous patchset when we had one in depends-on, other with checkout | 12:43 |
| quiquell | rfolco|rover: should work with checkout | 12:45 |
| rfolco|rover | quiquell, if rebase does not work, we'll just ignore ? | 12:48 |
| quiquell | rfolco|rover: well if rebase does not work, koll review is screw too | 12:49 |
| quiquell | good that it fails | 12:49 |
| quiquell | so we notify Alex | 12:49 |
| rfolco|rover | yes, then we fix the kolla patch | 12:49 |
| rfolco|rover | buena, mr felix, arriba muchacho! | 12:50 |
| *** ccamacho has quit IRC | 12:52 | |
| weshay | panda|ruck I need to know which jobs are blocked if we have a promotion blocker on https://bugs.launchpad.net/tripleo/+bug/1817688 | 12:53 |
| openstack | Launchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:53 |
| quiquell | rfolco|rover: let's test it though | 12:54 |
| rfolco|rover | arxcruz, chandankumar kopecmartin: the duplicate option patch in barbican plugin has zuul +1, please review | 12:55 |
| rfolco|rover | https://review.openstack.org/#/c/638502/ | 12:55 |
| weshay | rfolco|rover panda|ruck looks like fs001 is moving back to normal | 12:57 |
| rfolco|rover | weshay, I like good news | 12:57 |
| weshay | sshnaidm you avail in 30min for quick sync? | 12:58 |
| sshnaidm | weshay, yeah | 12:58 |
| weshay | thank you | 12:58 |
| weshay | panda|ruck please retrigger this until it passes :) https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/ by "resuming" the job | 12:59 |
| rfolco|rover | quiquell, reproducer needs udp ports/traffic open somehow ? We ran a reproducer on rdo cloud vm, but then it did not start zuul for some reason... | 12:59 |
| sshnaidm | weshay, in your bluejeans? | 12:59 |
| quiquell | rfolco|rover: it opens it for you | 12:59 |
| quiquell | rfolco|rover: Can help to find the issue | 12:59 |
| rfolco|rover | quiquell, my repro is a rdo vm, security groups issue | 13:00 |
| weshay | in 30 yes, or now? I need to make coffee but avail in few min if you want to go early | 13:00 |
| weshay | I have a cal invite out | 13:00 |
| rfolco|rover | quiquell, did not run on my local laptop, ran inside a f28 vm in rdo cloud... so I did not open udp | 13:00 |
| sshnaidm | quiquell, on different host has problems with ssh keys in reproducer :o, with same keys as in my host... how could it be so? | 13:00 |
| sshnaidm | weshay, up to you | 13:01 |
| quiquell | sshnaidm: that's super weird we are not even mounting them | 13:01 |
| panda|ruck | weshay:https://bugs.launchpad.net/tripleo/+bug/1817688 jobs added in the description | 13:01 |
| openstack | Launchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:01 |
| weshay | panda|ruck can you show the lines in the logs so people can verify? | 13:05 |
| ykarel | panda|ruck, pike is also affected ^^ | 13:05 |
| *** ccamacho has joined #oooq | 13:05 | |
| quiquell | rfolco|rover: do you have the machine with the reproducer ? | 13:05 |
| rfolco|rover | quiquell, yes, do you have 5 min ? | 13:06 |
| sshnaidm | quiquell, nevermind.. messed with users | 13:07 |
| quiquell | sshnaidm: ack | 13:09 |
| panda|ruck | weshay: added logs error extract https://bugs.launchpad.net/tripleo/+bug/1817688 | 13:12 |
| openstack | Launchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:12 |
| ykarel | panda|ruck, pike fs016,019 jobs are also affected:- https://trunk-primary.rdoproject.org/api-centos-pike/api/civotes_detail.html?commit_hash=8b255a648c381bfcff0623ba97880855c7fa21eb&distro_hash=15fc9723a56b892872cc97d12d8bd3ebc047c280 | 13:14 |
| ykarel | so all stable releases ceph jobs | 13:14 |
| weshay | panda|ruck ykarel thanks | 13:17 |
| weshay | sshnaidm ok.. ready when ever | 13:19 |
| sshnaidm | weshay, joining | 13:20 |
| panda|ruck | someone taht has ci.centos credential can put it in the shared ruck/rover doc ? | 13:21 |
| *** trown|outtypewww is now known as trown | 13:24 | |
| weshay | rfolco|rover can you cherry pick this to the promotion branch of sova https://github.com/sshnaidm/sova/pull/58/commits/54b892e32c58d564ad79e5d3eeebf1972b77ec06 | 13:26 |
| rfolco|rover | weshay, ok | 13:27 |
| *** agopi has quit IRC | 13:29 | |
| *** amoralej|lunch is now known as amoralej | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** rlandy has joined #oooq | 13:33 | |
| rlandy | panda|ruck: hi ... these playbooks https://github.com/openstack-infra/tripleo-ci/tree/master/playbooks do you know offhand which ones we call are really needed at this point (no legacy jobs)? | 13:37 |
| rlandy | quiquell: hi ... weshay blocked https://tree.taiga.io/project/tripleo-ci-board/task/792?kanban-status=1447274 | 13:38 |
| *** dtantsur is now known as dtantsur|afk | 13:38 | |
| rlandy | quiquell: but connecting to the undercloud via a nodepool node is working | 13:38 |
| quiquell | rlandy: yep I saw was going to ask | 13:38 |
| quiquell | rlandy: static provider ? | 13:38 |
| rlandy | quiquell: yeah - sorry - he said too much potential for something to go wrong | 13:39 |
| quiquell | rlandy: ack | 13:39 |
| quiquell | rlandy: btw Depends-On here https://code.engineering.redhat.com/gerrit/#/c/163849/ | 13:39 |
| quiquell | rlandy: does not work | 13:39 |
| rlandy | quiquell: not this ( the static driver does work - tested that before) but here we are adding a nodepool node to do the zuul work and then connecting to the undercloud | 13:39 |
| quiquell | rlandy: since the base job is at trusted project | 13:39 |
| panda|ruck | rlandy: not off-hand, no | 13:39 |
| rlandy | quiquell: I know - it's jut for my merge order | 13:39 |
| rlandy | so I don't stupidly merge one w/o the other like I did yesterday | 13:40 |
| quiquell | rlandy: ack | 13:40 |
| quiquell | rlandy: so a nodepool vm that connects to the real hardware undercloud | 13:40 |
| quiquell | is that it ? | 13:40 |
| rlandy | panda|ruck: quiquell: I am trying to narrow down what we really need and only call that in baremetal https://code.engineering.redhat.com/gerrit/#/c/163848/2/zuul.d/jobs.yaml | 13:41 |
| rlandy | quiquell: ack - see https://sf.hosted.upshift.rdu2.redhat.com/logs/21/163821/21/check/test-connection-to-hardware/ef01fcc/job-output.txt.gz#_2019-02-25_20_18_05_174514 | 13:41 |
| rlandy | quiquell: that way we keep the zuul/nodepool stuff off the virthost | 13:42 |
| rlandy | quiquell: if you want to bj, I can try explain what I am testing out here https://code.engineering.redhat.com/gerrit/#/c/163848/ | 13:43 |
| quiquell | rlandy: And about cleanup after job ? | 13:43 |
| rlandy | quiquell: we never clean up the bm | 13:43 |
| rlandy | it's a libvirt vm | 13:43 |
| rlandy | it gets destroyed in the next run's teardown | 13:43 |
| quiquell | rlandy: you mean the undercloud, I suppose overcloud we rewrite with PXE ? | 13:43 |
| rlandy | quiquell: exactly | 13:43 |
| quiquell | ack | 13:44 |
| rlandy | we re-pxe boot them each time | 13:44 |
| quiquell | rlandy: I was looking where to help there | 13:44 |
| quiquell | rlandy: And I saw https://tree.taiga.io/project/tripleo-ci-board/task/595?kanban-status=1447274 | 13:44 |
| quiquell | rlandy: But I don't get the relation between URL trigger and that | 13:44 |
| quiquell | rlandy: this is not about using the correct release file ? | 13:44 |
| rlandy | quiquell; lets' bj and I'll explain - I added a pipeline - but just for testing | 13:45 |
| rlandy | this won't be triggered periodically per se | 13:46 |
| rlandy | it will be triggerde by the change in file | 13:46 |
| quiquell | ack | 13:46 |
| quiquell | going to your blue | 13:46 |
| arxcruz | rfolco|rover: need to take my daughter to doctor, brb | 13:55 |
| *** agopi has joined #oooq | 13:58 | |
| weshay | sshnaidm thinking to turn off atop in the upstream jobs | 13:59 |
| weshay | and leave it running in rdo | 13:59 |
| sshnaidm | weshay, ok | 13:59 |
| weshay | we have containers-multinode and ovb to diagnose if we come across a perf issue | 13:59 |
| weshay | k.. will put up a change | 13:59 |
| weshay | rfolco|rover panda|ruck fyi.. http://logs.openstack.org/96/637196/6/check/tripleo-ci-centos-7-containers-multinode/dc7dcd9/logs/log-size.txt | 13:59 |
| weshay | http://logs.openstack.org/96/637196/6/check/tripleo-ci-centos-7-containers-multinode/dc7dcd9/logs/undercloud/var/log/ | 14:00 |
| * chandankumar is relocating will back in 30 mins | 14:02 | |
| weshay | panda|ruck, please resume the build here to get a rdo promotion and watch the job.. https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/ | 14:08 |
| panda|ruck | weshay: I was looking for the credentials in the ruck/rover doc but can't find them | 14:08 |
| marios | zbr|out: please update taiga man i left some comments today but the f28 is still a bit unclear to me som eduplicate etc | 14:09 |
| weshay | they wouldn't be in there.. I'll put them in the infra doc | 14:09 |
| *** vinaykns has joined #oooq | 14:10 | |
| weshay | panda|ruck got it? | 14:11 |
| panda|ruck | weshay: yes | 14:12 |
| *** ykarel is now known as ykarel|away | 14:13 | |
| weshay | panda|ruck thank you! | 14:13 |
| weshay | rfolco|rover note the ci.centos user/pass is in the infra doc | 14:14 |
| weshay | panda|ruck is https://bugs.launchpad.net/tripleo/+bug/1817688 really a quickstart bug? | 14:21 |
| openstack | Launchpad bug 1817688 in tripleo "job that test ceph in periodic pipeline for stable branches are getting the wrong configuration" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 14:21 |
| weshay | and are you in touch w/ jfulton about the issue? | 14:22 |
| weshay | or just fixing it :) | 14:22 |
| panda|ruck | weshay: it depends on how we want to approach it. One solution is making the release available early in the playbook. Not in quickstart but on how tripleo-ci passes variables to quickstart | 14:22 |
| panda|ruck | weshay: yes I'm in touch | 14:22 |
| weshay | cool | 14:22 |
| weshay | aye.. maybe something to frame up in the upcoming call | 14:23 |
| panda|ruck | weshay: we have two options, one is backporting the bluestore to queens and rocky, but no pike. the other is moving release to be calulated earlier. | 14:23 |
| panda|ruck | I'm in UA meeting | 14:23 |
| weshay | ah k | 14:23 |
| * weshay triages https://trello.com/b/j4IcIomh/production-chain-escalation | 14:23 | |
| panda|ruck | if the meeting finishes earlie maybe I can introduce the problem | 14:24 |
| panda|ruck | but moving the release variable to be computed earlie it's opening a can of worms, means moving forward the translation to ansible of toci_gate_test | 14:24 |
| weshay | panda|ruck.. I'll try and intro for you.. via the bug and maybe fulton | 14:30 |
| sshnaidm | marios, do you still have problem with beaker machine and dns? | 14:33 |
| sshnaidm | marios, I hit the same issue, but seems like it's ok in second run.. | 14:33 |
| marios | sshnaidm: well latest issue was for ssh-keyscan and sems dns related issue 2 @ https://tree.taiga.io/project/tripleo-ci-board/task/765 | 14:34 |
| sshnaidm | marios, yeah, seems like it helps | 14:35 |
| weshay | marios did the test boxes get both localhost and 1.1.1.1 for dns? | 14:40 |
| weshay | in resolv.conf | 14:41 |
| marios | weshay: no | 14:43 |
| marios | weshay: i didn't check now but already checked before they had the normal beaker ones there | 14:43 |
| weshay | marios was it a singlenode, multinode, ovb? | 14:43 |
| marios | weshay: standalone scen 1 | 14:43 |
| weshay | ok.. I don't think it has to do w/ beaker | 14:43 |
| weshay | but .. we'll chat | 14:43 |
| *** ykarel|away is now known as ykarel | 14:46 | |
| marios | weshay: k thanks | 14:48 |
| quiquell | panda|ruck, marios: periodic-tripleo-centos-7-master-containers-build-push pass at last periodic run :-) | 14:55 |
| *** ykarel_ has joined #oooq | 14:59 | |
| *** ykarel has quit IRC | 15:00 | |
| marios | quiquell: nice :) thanks for info with tesproject also green here but i want to add more to logs so i can verify http://logs.rdoproject.org/00/19000/1/check/periodic-tripleo-centos-7-master-containers-build-push/6f45860/ | 15:11 |
| rlandy | quiquell: here we go .. https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/a6454d2a49b84d13a68b151104ebb650?logfile=console.log | 15:13 |
| rlandy | quiquell: our first error: https://sf.hosted.upshift.rdu2.redhat.com/logs/49/163849/4/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/a6454d2/job-output.txt.gz#_2019-02-26_15_13_31_391484 | 15:14 |
| quiquell | rlandy: yey ! | 15:15 |
| quiquell | rlandy: the build-ssh role is missing | 15:18 |
| rlandy | zuul_temp_ssh_key | 15:18 |
| quiquell | rlandy: add-build-sshke | 15:18 |
| quiquell | rlandy: you have to run that role | 15:19 |
| rlandy | that is missing | 15:19 |
| quiquell | is done at tripleo jobs | 15:19 |
| rlandy | https://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/add-build-sshkey | 15:19 |
| rlandy | weird, I am parenting off upstream | 15:19 |
| rlandy | shoudl pick it up | 15:19 |
| quiquell | rlandy: but is done at zuul base jobs at upstream and RDO | 15:19 |
| quiquell | rlandy: looks like we don't do that at internal sf base job | 15:19 |
| quiquell | rlandy: yep but its base is going to be internal sf base job | 15:20 |
| rlandy | yes- comparing base with rdo | 15:20 |
| quiquell | ack | 15:20 |
| chandankumar | kopecmartin: what do you think about this https://review.openstack.org/#/c/639310/ ? | 15:21 |
| rlandy | quiquell: hmmm ... wondering about mirrors as well | 15:22 |
| quiquell | rlandy: all tha is different at fs base job and rdo/upstream base can be a problem | 15:22 |
| kopecmartin | chandankumar, It's more transparent, I like it more | 15:22 |
| quiquell | rlandy: we did have same issues with zuul reproducer | 15:22 |
| chandankumar | kopecmartin: feel free to fix the wording I am lame in that | 15:23 |
| rlandy | quiquell: so you have experience here - what else do we need to add :) | 15:23 |
| rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/base-minimal/pre.yaml | 15:24 |
| chandankumar | kopecmartin: I think we can remove /home/zuul stuff let's ansible home user dir takes care of that? | 15:24 |
| quiquell | rlandy: give me the url to the config project downstream | 15:24 |
| rlandy | quiquell: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config - but I think I got it | 15:24 |
| kopecmartin | chandankumar, sure | 15:24 |
| panda|ruck | quiquell: \o/ | 15:25 |
| rlandy | quiquell: it's here http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/playbooks/base/pre.yaml | 15:25 |
| quiquell | panda|ruck: hello | 15:25 |
| quiquell | rlandy: only prepare-workspace is there | 15:25 |
| rlandy | quiquell: yep - but it is in the sf-config | 15:26 |
| quiquell | rlandy: we change that or add pres at ours | 15:26 |
| rlandy | so it must be callable from downstream | 15:26 |
| rlandy | quiquell: we can just merge a change to the base in tripleo-ci-internal-config | 15:26 |
| rlandy | sec - putting up patch | 15:26 |
| quiquell | rlandy: you can even add it to your the untrusted project | 15:27 |
| panda|ruck | quiquell: was just commenting the news about the periodic job | 15:28 |
| rlandy | quiquell: ack - but I think it should be there- we can start with our job | 15:28 |
| quiquell | panda|ruck: ack | 15:28 |
| quiquell | rlandy: ack, let's just add stuff and see if we go beyond | 15:28 |
| *** ykarel_ is now known as ykarel|away | 15:32 | |
| rlandy | quiquell: adding it to untrusted will be too late | 15:33 |
| quiquell | rlandy: also at trusted | 15:33 |
| quiquell | rlandy: now that I think of | 15:33 |
| rlandy | quiquell: let me try base in trusted | 15:34 |
| quiquell | rlandy: ack | 15:34 |
| *** sanjayu_ has quit IRC | 15:34 | |
| *** sanjayu_ has joined #oooq | 15:35 | |
| rlandy | quiquell: https://code.engineering.redhat.com/gerrit/#/c/163905/ | 15:36 |
| rlandy | will merge this as soon as linters pass | 15:37 |
| quiquell | rlandy: then we are affecting all the jobs at internal software factory | 15:37 |
| quiquell | rlandy: can this be a problem ? | 15:37 |
| rlandy | quiquell: nah - just our local project | 15:37 |
| rlandy | quiquell: if we change sf-config we mess with all internal | 15:37 |
| rlandy | openstack/tripleo-ci-internal-config is just ours | 15:38 |
| quiquell | ack | 15:38 |
| quiquell | I am going to merge that | 15:38 |
| quiquell | We need the mirro thing too | 15:38 |
| quiquell | to hvae /etc/mirror_info.sh | 15:38 |
| rlandy | quiquell: go ahead and add the mirror to that patch | 15:39 |
| quiquell | rlandy: ack | 15:39 |
| rlandy | quiquell: wrt your comment - let's leave it consistent with upstream unless we get failures | 15:39 |
| rlandy | quiquell: I agree we should have as little diff as possible with upstream | 15:40 |
| rlandy | quiquell: http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/mirrors/centos-base.yaml | 15:42 |
| quiquell | rlandy: ack let's forget about it | 15:42 |
| quiquell | let's merge your change as it is and see | 15:42 |
| rlandy | quiquell: k - let's live dangerously - bit I think you are right we will need the mirror eventually | 15:43 |
| rlandy | quiquell: the undercloud install will fail w/o it but that is down the line | 15:44 |
| quiquell | ok it's a rdo-jobs | 15:47 |
| quiquell | review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml | 15:47 |
| rlandy | quiquell: let's add that when we fix https://sf.hosted.upshift.rdu2.redhat.com/logs/49/163849/4/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/dffae90/job-output.txt.gz#_2019-02-26_15_46_45_962993 | 15:50 |
| quiquell | rlandy: those are the build keys | 15:51 |
| *** ccamacho has quit IRC | 15:51 | |
| rlandy | it got past the first failure | 15:52 |
| rlandy | 2019-02-26 15:46:22.344661 | TASK [add-build-sshkey : Make sure user has a .ssh] | 15:52 |
| rlandy | ran | 15:52 |
| quiquell | \o/ | 15:53 |
| rlandy | not sure why this failed...2019-02-26 15:46:46.299653 | primary | MODULE FAILURE: Item: id_rsa | 15:53 |
| rlandy | quiquell: unless the node is diff in some way | 15:53 |
| rlandy | missing something else? | 15:54 |
| fultonj | marios weshay during the call you referenced a change which made 001/004 not vote for stable periodic jobs; do you have that review handy? i cannot find it. | 15:54 |
| quiquell | rlandy: is not related to .ssh directory | 15:54 |
| *** ccamacho has joined #oooq | 15:55 | |
| rlandy | ZUUL_USER_SSH_PUBLIC_KEY: /var/lib/nodepool/.ssh/zuul_rsa.pub | 15:55 |
| arxcruz | and i'm back | 15:58 |
| *** skramaja_ has quit IRC | 16:01 | |
| sshnaidm | arxcruz, chandankumar can you please tempest extra config there? I think I mess there with syntax: https://review.openstack.org/#/c/639324/1/config/general_config/featureset039.yml | 16:03 |
| sshnaidm | arxcruz, chandankumar it fails with "AnsibleUndefinedVariable: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'items'" | 16:03 |
| arxcruz | sshnaidm: maybe is the lines? check https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset054.yml#L81 | 16:06 |
| arxcruz | sshnaidm: also if you are running it on tempest containerized, you don't need the tempest_plugins | 16:07 |
| marios | fultonj: wasnt me | 16:09 |
| quiquell | rlandy: | 16:09 |
| weshay | probably alex | 16:09 |
| quiquell | Error: [Errno 13] Permission denied: '/var/tmp/zuul-console/fa163e16-f647-2586-9a56-00000000000f-primary.log | 16:10 |
| sshnaidm | weshay, sent you | 16:10 |
| quiquell | rlandy: https://sf.hosted.upshift.rdu2.redhat.com/logs/49/163849/4/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/dffae90/job-output.json.gz | 16:10 |
| sshnaidm | arxcruz, it's not containerized on queens afaik.. | 16:11 |
| arxcruz | ok | 16:11 |
| quiquell | rlandy: I am going to add start-zuul-console | 16:12 |
| rlandy | quiquell: k | 16:12 |
| panda|ruck | any summary from the community meeting ? | 16:13 |
| quiquell | rlandy: have to drop now | 16:13 |
| weshay | rfolco|rover 639212 | 16:13 |
| rlandy | quiquell: k - patch up - merged? | 16:14 |
| chandankumar | arxcruz: feel free to join osa meeting we need to discuss os_tempest dependency stuff & where to keep vars! | 16:14 |
| quiquell | rlandy: yep | 16:14 |
| arxcruz | chandankumar: let's do it ! | 16:14 |
| rlandy | quiquell: k - thanks - see you tomorrow | 16:15 |
| arxcruz | chandankumar: however, i'm not familiar with the roles, so be patient | 16:15 |
| rfolco|rover | weshay, thx | 16:16 |
| chandankumar | arxcruz: I am always there mate to solve mestries :-) | 16:16 |
| chandankumar | *mysteries | 16:17 |
| arxcruz | quiquell: quick question after finishes the reproducer, will it delete the vm? | 16:18 |
| arxcruz | i remember it was the case before | 16:18 |
| quiquell | arxcruz: yep | 16:19 |
| quiquell | arxcruz: you can do an autohold on them | 16:19 |
| arxcruz | how do i do that? | 16:20 |
| quiquell | arxcruz: they will there after failure if you do autohold | 16:20 |
| *** jfrancoa has quit IRC | 16:20 | |
| quiquell | arxcruz: cd ~/tripleo-ci-reproducer | 16:21 |
| quiquell | arxcruz: docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job [job name] --project test1 --reason whatever | 16:22 |
| quiquell | That's it | 16:22 |
| arxcruz | jesus... | 16:22 |
| quiquell | yep :-/ | 16:22 |
| arxcruz | quiquell: returning nothing is okay right ? | 16:24 |
| quiquell | arxcruz: then youre good | 16:24 |
| quiquell | you can check with zuul autohold-list <- I think is like that | 16:24 |
| *** quiquell is now known as quiquell|off | 16:28 | |
| sshnaidm | marios, weshay, quiquell|off this should resolve problem on beaker machines: https://review.rdoproject.org/r/19007 | 16:30 |
| marios | sshnaidm: nice thanks | 16:31 |
| marios | sshnaidm: will check | 16:31 |
| weshay | panda|ruck https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/ | 16:41 |
| *** ykarel|away has quit IRC | 16:41 | |
| *** ykarel has joined #oooq | 16:42 | |
| panda|ruck | weshay: rekicked tripleo-quickstart-promote-rocky-rdo_trunk-minimal but it failed again | 16:44 |
| weshay | panda|ruck what you want to do is.. go to https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/106/ | 16:45 |
| weshay | and click "resume build" | 16:45 |
| weshay | basically enough times to get a node that will work.. \0/ ci.centos :) | 16:45 |
| weshay | it takes 3-6 times | 16:46 |
| panda|ruck | weshay: and I assume we cannot ask ci.centos to add https://wiki.jenkins.io/display/JENKINS/Naginator+Plugin so it does it by itself ? | 16:48 |
| weshay | panda|ruck I would rather ask.. how to ensure we only checkout boxes of a certain class rather than how to retrigger | 16:50 |
| rascasoft | weshay, don't be afraid to ask, you'll receive anything you'll ask the Father for in His name | 16:52 |
| weshay | rascasoft bless you | 16:54 |
| *** vinaykns has quit IRC | 16:59 | |
| *** bogdando has quit IRC | 17:00 | |
| *** ccamacho has quit IRC | 17:01 | |
| *** vinaykns has joined #oooq | 17:03 | |
| panda|ruck | rfolco|rover: weshay for https://bugs.launchpad.net/tripleo/+bug/1817705 I'm talking with OVN folks, they should add a note to the bug. In the meantime I have https://review.openstack.org/639386, waiting for comment on it from them too | 17:08 |
| openstack | Launchpad bug 1817705 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master fails tempest test related to DHCP agent" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 17:08 |
| marios | talking about logs weshay panda|ruck sshnaidm rfolco|rover i added /yum.repos.d @ v5 https://review.openstack.org/#/c/638652/ testing with https://review.rdoproject.org/r/#/c/19000/ which is about to report hoping to compare delorean.repo with https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/delorean.repo wdyt | 17:12 |
| marios | weshay: panda|ruck rfolco|rover moved ready reviews https://tree.taiga.io/project/tripleo-ci-board/task/773 i'll remove the WIPs from patches once we see good results at /r/19000 & any other discussion around how thats working | 17:14 |
| chandankumar | hello | 17:16 |
| *** sshnaidm is now known as sshnaidm|afk | 17:17 | |
| chandankumar | what is the ansible galaxy alternative they are working on? | 17:17 |
| sshnaidm|afk | chandankumar, https://galaxy.ansible.com/docs/mazer/index.html ? | 17:18 |
| chandankumar | sshnaidm|afk: I think i got it https://github.com/ansible/mazer | 17:19 |
| chandankumar | sshnaidm|afk: thanks :-) | 17:20 |
| panda|ruck | chandankumar: arxcruz https://review.openstack.org/639386 | 17:22 |
| *** holser__ has quit IRC | 17:48 | |
| *** trown is now known as trown|lunch | 17:57 | |
| *** kopecmartin is now known as kopecmartin|off | 18:14 | |
| weshay | rlandy fyi https://review.openstack.org/639407 | 18:15 |
| *** jpena is now known as jpena|off | 18:15 | |
| *** chandankumar is now known as raukadah | 18:16 | |
| *** chem` has quit IRC | 18:16 | |
| weshay | rlandy can you please merge https://review.rdoproject.org/r/#/c/18956/ for rfolco|rover | 18:20 |
| rfolco|rover | thanks | 18:21 |
| raukadah | weshay: rfolco|rover rlandy arxcruz what about this change https://review.openstack.org/639310 in order to solve kopecmartin|off problem :-) | 18:22 |
| arxcruz | raukadah: i have a question, put in the comments, waiting a response :) | 18:23 |
| raukadah | arxcruz: I will try to reply | 18:23 |
| arxcruz | ok | 18:23 |
| weshay | raukadah interesting :) | 18:24 |
| weshay | arxcruz when you have a sec.. I want to chat.. can be later this week too | 18:24 |
| arxcruz | weshay: can be now, your bj? | 18:24 |
| weshay | sure.. will just be a sec | 18:24 |
| *** derekh has quit IRC | 18:25 | |
| weshay | I'm in | 18:25 |
| raukadah | arxcruz: replied! | 18:25 |
| raukadah | arxcruz: thank you I learned a lot without going too much in ansible playbook like block usage | 18:27 |
| raukadah | arxcruz++ | 18:27 |
| raukadah | arxcruz: you helped me a lot to explore more intersting stuff without doing through the docs | 18:28 |
| raukadah | in deep | 18:28 |
| raukadah | while working on os_tempest | 18:28 |
| *** panda|ruck is now known as panda|ruck|off | 18:28 | |
| arxcruz | weshay: https://github.com/search?q=topic%3Apython+org%3Areportportal&type=Repositories | 18:31 |
| fultonj | weshay: is recheck the right thing for https://review.openstack.org/#/c/638364/ ? | 18:37 |
| rlandy | raukadah: I will +2 https://review.openstack.org/#/c/639310/ when you and arxcruz are happy with it - don't have much insight there | 18:44 |
| *** rfolco|rover has quit IRC | 18:46 | |
| *** amoralej is now known as amoralej|off | 18:49 | |
| weshay | fultonj ya.. looks like infra error http://logs.openstack.org/64/638364/2/check/tripleo-ci-centos-7-standalone/d708d93/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 18:57 |
| *** sanjayu_ has quit IRC | 18:59 | |
| *** rfolco has joined #oooq | 19:12 | |
| fultonj | weshay: ok, thanks recheck'd | 19:12 |
| *** trown|lunch is now known as trown | 19:18 | |
| raukadah | rlandy: I am ok with changes, waiting on arxcruz comments | 19:19 |
| rlandy | k | 19:19 |
| *** gouthamr has quit IRC | 19:20 | |
| *** gouthamr_ has joined #oooq | 19:20 | |
| *** rfolco is now known as rfolco|rover | 19:28 | |
| *** ykarel has quit IRC | 19:33 | |
| *** derekh has joined #oooq | 19:58 | |
| *** derekh has quit IRC | 20:09 | |
| *** gouthamr_ is now known as gouthamr | 20:42 | |
| *** derekh has joined #oooq | 21:24 | |
| *** derekh has quit IRC | 21:24 | |
| *** zbr|out has quit IRC | 21:27 | |
| *** jtomasek has quit IRC | 21:43 | |
| *** agopi has quit IRC | 22:59 | |
| *** rascasoft has quit IRC | 23:15 | |
| *** sanjayu_ has joined #oooq | 23:19 | |
| *** vinaykns has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!