| *** Vorrtex has quit IRC | 00:32 | |
| *** trown has quit IRC | 00:32 | |
| *** trown has joined #oooq | 00:33 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (2 more messages) | 00:35 |
|---|---|---|
| *** aakarsh|2 has joined #oooq | 01:24 | |
| *** ykarel|away has joined #oooq | 02:12 | |
| *** Goneri has quit IRC | 02:15 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb- (3 more messages) | 02:35 |
| *** ykarel|away has quit IRC | 03:44 | |
| *** ykarel|away has joined #oooq | 04:06 | |
| *** ykarel|away is now known as ykarel | 04:10 | |
| *** ykarel_ has joined #oooq | 04:13 | |
| *** ykarel has quit IRC | 04:16 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (3 more messages) | 04:35 |
| *** holser_ has joined #oooq | 04:52 | |
| *** ratailor has joined #oooq | 04:59 | |
| *** holser_ has quit IRC | 05:07 | |
| *** dsneddon has quit IRC | 05:17 | |
| *** dsneddon has joined #oooq | 05:30 | |
| *** quiquell|off is now known as quiquell|rover | 05:32 | |
| *** holser_ has joined #oooq | 05:34 | |
| *** dsneddon has quit IRC | 05:36 | |
| *** marios has joined #oooq | 05:38 | |
| *** holser_ has quit IRC | 05:47 | |
| *** jtomasek has quit IRC | 05:48 | |
| *** gouthamr has quit IRC | 05:56 | |
| *** gouthamr has joined #oooq | 06:00 | |
| *** udesale has joined #oooq | 06:03 | |
| *** kopecmartin|off is now known as kopecmartin | 06:09 | |
| *** dsneddon has joined #oooq | 06:11 | |
| *** ykarel_ is now known as ykarel | 06:11 | |
| *** dsneddon has quit IRC | 06:15 | |
| *** dsneddon has joined #oooq | 06:20 | |
| *** jfrancoa has joined #oooq | 06:23 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (3 more messages) | 06:35 |
| *** skramaja has joined #oooq | 06:44 | |
| *** dsneddon has joined #oooq | 06:45 | |
| *** holser_ has joined #oooq | 06:56 | |
| *** ccamacho has joined #oooq | 07:00 | |
| *** holser_ has quit IRC | 07:01 | |
| *** holser_ has joined #oooq | 07:04 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:06 | |
| *** ykarel is now known as ykarel|lunch | 07:32 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** tosky has joined #oooq | 07:39 | |
| *** amoralej|off is now known as amoralej | 07:40 | |
| *** dsneddon has quit IRC | 07:49 | |
| marios | quiquell|rover: * anyone interested and has time we'll discss standalone-upgrade job and validations in 5 mins with Tengu at https://redhat.bluejeans.com/7661925373/ https://review.openstack.org/#/c/651213/ | 07:55 |
| *** bogdando has joined #oooq | 08:01 | |
| zbr | jfrancoa: please revote on https://review.openstack.org/#/c/650921/ tx | 08:01 |
| *** dsneddon has joined #oooq | 08:02 | |
| *** jtomasek has joined #oooq | 08:06 | |
| *** dsneddon has quit IRC | 08:06 | |
| chandankumar | quiquell|rover: please have a look at this https://review.rdoproject.org/r/#/c/19916/ | 08:17 |
| chandankumar | thanks! | 08:17 |
| jfrancoa | zbr: ack | 08:18 |
| marios | Tengu: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset050.yml | 08:20 |
| marios | https://github.com/openstack-infra/tripleo-ci/blob/master/README.rst | 08:20 |
| marios | Tengu: ^ | 08:20 |
| *** panda has joined #oooq | 08:21 | |
| Tengu | perfect :) | 08:23 |
| *** ykarel|lunch is now known as ykarel | 08:24 | |
| panda | Tengu: thanks. | 08:24 |
| marios | Tengu: ack its all in there https://github.com/openstack-infra/tripleo-ci/tree/master/zuul.d https://docs.openstack.org/tripleo-docs/latest/ci/ci_primer.html#where-do-tripleo-ci-jobs-live | 08:25 |
| Tengu | marios: so basically, if I just "git grep 'voting: false'" in there and edit the related featureset, it should be fine.. ? | 08:27 |
| Tengu | at least on standalone + undercloud | 08:27 |
| marios | Tengu: yeah default is true https://zuul-ci.org/docs/zuul/user/config.html#attr-job.voting (yes) | 08:29 |
| *** yolanda_ has joined #oooq | 08:29 | |
| Tengu | good :) | 08:29 |
| Tengu | hmm. a featureset can be used in multiple job. maybe one job isn't voting while another is? | 08:31 |
| marios | Tengu: this is true. we can consider adding it to featureset_override | 08:32 |
| marios | Tengu: so note if you find such case (grep should help ) and we can discuss some more | 08:32 |
| Tengu | ok! | 08:32 |
| marios | Tengu: (cos featureset_override goes directly on the job definition ) | 08:32 |
| Tengu | marios: for instance, 052 is one of those cases. | 08:33 |
| Tengu | marios: maybe it's better to move the variable in the job definition using the override? | 08:33 |
| marios | Tengu: yes scenario 1-4 and vanila standalone | 08:33 |
| marios | (are using fs052_ | 08:33 |
| Tengu | woule be cleaner and clearer imho | 08:33 |
| *** derekh has joined #oooq | 08:34 | |
| marios | Tengu: lmme fetch you an example. only hesitation there is that the feature has proven controversion some folks panda sshnaidm and others have expressed concern at using featurest_override since featureset is meant to be the source of truth. anyway you can propose it and we discus in gerrit the change is pretty small | 08:35 |
| *** dtantsur|afk is now known as dtantsur | 08:35 | |
| Tengu | marios: pretty small and, hopefully, temporary since it should be enabled by default everywhere :) | 08:35 |
| *** dsneddon has joined #oooq | 08:35 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (3 more messages) | 08:35 |
| panda | yay, another override. | 08:35 |
| Tengu | or else we enable that validation by default and that's it XD | 08:36 |
| Tengu | panda: don't be grumpy :) | 08:36 |
| marios | Tengu: you need to add your thing into the allowed ones https://github.com/openstack-infra/tripleo-ci/blob/master/roles/run-test/tasks/main.yaml and then enable it on the job like https://github.com/openstack-infra/tripleo-ci/blob/8985516a3abb28f4c56ab739645afac907df6d11/zuul.d/standalone-jobs.yaml#L462-L475 | 08:37 |
| panda | I'm not grumpy. I've been trying to make my point for the last months. Using override is the quickest and easiest way to add a new variable to the jobs workflow. | 08:38 |
| Tengu | panda: oh, so you're all for it? didn't understand that, sorry :) | 08:39 |
| panda | Tengu: no, I still think it's wrong, but sometimes we have no choice | 08:40 |
| panda | Tengu: we can't offer a better way. yet. | 08:40 |
| panda | And I'm working on the background to try to find this better way. | 08:40 |
| Tengu | panda: ok :). Now I get it | 08:40 |
| *** dsneddon has quit IRC | 08:40 | |
| panda | for the meantime, I'm not blocking anything | 08:40 |
| *** saneax has joined #oooq | 08:41 | |
| panda | Tengu: I've lost the first part of the context, what's the variable ? I'll add it to my backgroud analysis | 08:45 |
| Tengu | panda: the one I've created for the new "service validation", "validate_services". You probably remember our quick chat about that new validation and the need to enable it on non-voting jobs first :) | 08:45 |
| Tengu | it makes the CI job fail if it finds either a crashed container, or a crashed systemd unit | 08:46 |
| panda | Tengu: yes, and you were thinking about putting it into the featureset overrides because ... ? | 08:46 |
| Tengu | panda: because I'd like to enable it on more non-voting jobs now. Issue is, some featuresets are used for both voting and non-voting jobs, meaning we must use some higher level param for that. | 08:47 |
| panda | Tengu: like a value that wins over everything else ? | 08:48 |
| Tengu | panda: in that precise case, yes | 08:48 |
| Tengu | basically, adding that validation only when we have a "voting: false" is nice | 08:49 |
| panda | Tengu: perfect. Thanks for your collaboration in building a better future for our CI. | 08:49 |
| Tengu | panda: well, I'm trying to do the right thing ^^' | 08:50 |
| panda | your call is really important for us :) | 08:50 |
| panda | please, stay online to not lose the priority | 08:50 |
| Tengu | https://review.openstack.org/651725 https://review.openstack.org/651726 | 08:50 |
| * Tengu slaps panda with a suspicious non-functional AT power source | 08:50 | |
| Tengu | :) | 08:50 |
| * panda suddenly feels nostalgic | 08:51 | |
| Tengu | marios: so, basically, once the two listed patches are merged, we can drop the validate_services from the featuresets. | 08:51 |
| Tengu | marios: I'll provide a patch with depends-on. | 08:52 |
| marios | Tengu: ack add me on reviews | 08:53 |
| Tengu | done. | 08:56 |
| arxcruz | chandankumar: please give me your ack on https://tree.taiga.io/project/tripleo-ci-board/task/987 | 09:03 |
| chandankumar | arxcruz: sure! | 09:04 |
| *** dsneddon has joined #oooq | 09:16 | |
| *** dsneddon has quit IRC | 09:21 | |
| *** dsneddon has joined #oooq | 09:22 | |
| quiquell|rover | Damn have screw it workflowing buildah periodic :-( | 09:23 |
| panda | quiquell|rover: unscrew | 09:24 |
| quiquell|rover | I am into it | 09:24 |
| quiquell|rover | they were working fine at test review | 09:24 |
| chandankumar | zbr: hello | 09:26 |
| quiquell|rover | un secrets | 09:26 |
| chandankumar | zbr: please have a look at this failure http://logs.openstack.org/54/650054/7/check/openstack-ansible-functional-tempestconf-centos-7/d0f58e9/job-output.txt.gz#_2019-04-11_09_15_44_528191 | 09:26 |
| chandankumar | zbr: I am not getting why it is failing at that handler | 09:26 |
| *** dsneddon has quit IRC | 09:30 | |
| zbr | chandankumar: looking... | 09:30 |
| panda | chandankumar: I see 'tempest_install_method' is undefined | 09:31 |
| zbr | right. | 09:32 |
| zbr | inside the handler | 09:32 |
| zbr | panda: any chance to wf the file removals https://review.openstack.org/#/c/650921/ ? | 09:33 |
| *** dsneddon has joined #oooq | 09:48 | |
| arxcruz | chandankumar: what was the other scenario ? | 09:49 |
| arxcruz | 004 right ? | 09:49 |
| arxcruz | for me ? | 09:49 |
| arxcruz | you did 1 and 3 | 09:49 |
| arxcruz | i was suppoed to work on 2 and 4 right ? | 09:49 |
| arxcruz | can't remember | 09:49 |
| arxcruz | supposed* | 09:49 |
| chandankumar | arxcruz: I did 1 3 and plain standalone | 09:49 |
| arxcruz | ok | 09:49 |
| chandankumar | arxcruz: https://tree.taiga.io/project/tripleo-ci-board/us/972 | 09:49 |
| *** dsneddon has quit IRC | 09:53 | |
| *** ratailor_ has joined #oooq | 10:08 | |
| *** ratailor has quit IRC | 10:11 | |
| zbr | is there a way to get an email/notif when a promotion happens? | 10:15 |
| zbr | rfolco: are you doing a followup on https://review.rdoproject.org/r/#/c/19481/ ? | 10:17 |
| quiquell|rover | panda: This make any sense to you https://review.rdoproject.org/r/20142 ? | 10:21 |
| quiquell|rover | panda: another fix for promotion blocker | 10:21 |
| quiquell|rover | looks like buildah push uses sude | 10:21 |
| quiquell|rover | to be able to use overlayfs | 10:21 |
| quiquell|rover | instead of fuse-overlayfs | 10:22 |
| zbr | i guess i can cross the "we dont need root" from the list of selling features ;) | 10:22 |
| quiquell|rover | zbr: looks like yes :-) | 10:22 |
| *** dsneddon has joined #oooq | 10:25 | |
| *** dsneddon has quit IRC | 10:30 | |
| panda | zbr: we still don't need root for the rest of the workflow | 10:32 |
| quiquell|rover | damn... | 10:32 |
| quiquell|rover | we have other jobs using docker push | 10:32 |
| quiquell|rover | we need to run this twice at become and non become | 10:32 |
| panda | quiquell|rover: that review is poiting at the wrong bug | 10:33 |
| quiquell|rover | nope | 10:33 |
| panda | quiquell|rover: there's now way to run buildah without root ? | 10:33 |
| quiquell|rover | is the right one | 10:34 |
| quiquell|rover | don't know | 10:34 |
| quiquell|rover | tripleo-common run buildah push with sudo | 10:34 |
| quiquell|rover | to use native overlayfs | 10:34 |
| quiquell|rover | instead of fuse-overlayfs | 10:34 |
| quiquell|rover | panda: updated to also do docker login to non root https://review.rdoproject.org/r/#/c/20142/ | 10:34 |
| panda | quiquell|rover: no way to put the pus setuid or add the user to a group that can use overlayfs directly ? | 10:35 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (3 more messages) | 10:35 |
| quiquell|rover | panda: have to investigate but it will take time and now master/stein is blocked at containers build | 10:36 |
| zbr | i dont understand why we have playbook for login, when this should be a role but that's another aspect. | 10:37 |
| quiquell|rover | panda: we can make sudo optional at tripleo-common and all | 10:38 |
| quiquell|rover | and pass it to client and so on and so on | 10:39 |
| panda | quiquell|rover: ok | 10:39 |
| *** yolanda_ has quit IRC | 10:40 | |
| quiquell|rover | panda: looks like answer to that is they implemented to use fuse-overlayfs https://github.com/containers/buildah/issues/386 | 10:42 |
| chandankumar | sshnaidm: hello | 10:43 |
| chandankumar | sshnaidm: http://paste.openstack.org/show/749167/ seen this issue with ovb | 10:43 |
| sshnaidm | chandankumar, hi | 10:43 |
| sshnaidm | chandankumar, your zuul.yaml in test1 project should look like that: paste.openstack.org/show/749118/ | 10:44 |
| quiquell|rover | panda: so we workflow this or not ? https://review.rdoproject.org/r/#/c/20142/ | 10:44 |
| sshnaidm | chandankumar, the error means you don't provide credentials to rdo cloud | 10:45 |
| panda | quiquell|rover: we discussed the alternatives, and this is the only way | 10:45 |
| chandankumar | sshnaidm: it means exported values are not working | 10:45 |
| *** ratailor__ has joined #oooq | 10:45 | |
| sshnaidm | chandankumar, instead of ${OS_USERNAME} and ${OS_PASSWORD} should be your real user and pass | 10:45 |
| chandankumar | sshnaidm: ok | 10:45 |
| quiquell|rover | panda: I am going to workflow it, since it's config project we cannot test it at reviews | 10:46 |
| panda | quiquell|rover: sorry, I'm retarded today. I meant to add a +2 ther | 10:47 |
| quiquell|rover | panda: ack, just workflow it | 10:47 |
| quiquell|rover | panda: it's already broken | 10:47 |
| *** ratailor_ has quit IRC | 10:48 | |
| panda | quiquell|rover: done | 10:48 |
| quiquell|rover | thanks | 10:49 |
| quiquell|rover | we will have to do the same with promoter push if we use buildah :-/ | 10:50 |
| *** yolanda_ has joined #oooq | 10:52 | |
| chandankumar | sshnaidm: collect-logs review is good to go https://review.openstack.org/#/c/650613/ | 10:52 |
| sshnaidm | chandankumar, commands should be on the same indentation level | 10:54 |
| *** jpena is now known as jpena|lunch | 10:56 | |
| *** amoralej is now known as amoralej|lunch | 10:56 | |
| *** udesale has quit IRC | 10:57 | |
| chandankumar | sshnaidm: thansk, it worked | 11:00 |
| chandankumar | I will update the patch | 11:00 |
| *** panda has quit IRC | 11:01 | |
| *** panda has joined #oooq | 11:02 | |
| *** panda is now known as panda|lunch | 11:03 | |
| *** dsneddon has joined #oooq | 11:03 | |
| chandankumar | sshnaidm: again failed at http://paste.openstack.org/show/749171/ | 11:04 |
| chandankumar | ovb stack create failed bmc-template': No images matching {'name': u'bmc-template'} | 11:05 |
| *** dsneddon has quit IRC | 11:08 | |
| *** dsneddon has joined #oooq | 11:25 | |
| *** dsneddon has quit IRC | 11:31 | |
| quiquell|rover | rfolco: ping | 11:32 |
| quiquell|rover | rfolco: periodic buildah failure does not prevent the other periodic jobs from running :-/ | 11:34 |
| quiquell|rover | rfolco: do you know something about it ? | 11:35 |
| arxcruz | sshnaidm: is there a way to recheck only the RDO Third Party CI check ? | 11:51 |
| sshnaidm | arxcruz, check-rdo | 11:51 |
| arxcruz | sshnaidm: thanks | 11:51 |
| *** dsneddon has joined #oooq | 11:55 | |
| *** rlandy has joined #oooq | 11:57 | |
| *** rlandy is now known as rlandy|ruck | 11:58 | |
| *** ykarel is now known as ykarel|afk | 11:59 | |
| *** dsneddon has quit IRC | 11:59 | |
| *** dsneddon has joined #oooq | 12:02 | |
| *** dsneddon has quit IRC | 12:09 | |
| rlandy|ruck | quiquell|rover: hi | 12:10 |
| quiquell|rover | rlandy|ruck: o/ | 12:10 |
| rlandy|ruck | quiquell|rover: the three new cards on escalation board ... | 12:10 |
| quiquell|rover | rlandy|ruck: promotion is broken | 12:10 |
| rlandy|ruck | quiquell|rover: what did we break now? | 12:11 |
| * rlandy|ruck moves cards | 12:11 | |
| quiquell|rover | Haven't check yet the fintrospection | 12:11 |
| quiquell|rover | have created the other two | 12:12 |
| rfolco | quiquell|rover, what you mean by 'buildah failures' ? | 12:12 |
| quiquell|rover | the build containers is suppose to be fixe dnow | 12:12 |
| quiquell|rover | rlandy|ruck, rfolco: periodic buildah fix https://review.rdoproject.org/r/20142 | 12:12 |
| panda|lunch | I'm still trying to not read "buildah" as a klingon word. | 12:13 |
| quiquell|rover | weird thing jobs are not skipped | 12:13 |
| rlandy|ruck | quiquell|rover: how was it working before? | 12:13 |
| weshay | quiquell|rover refering to the auth bug? | 12:13 |
| quiquell|rover | weshay: yep | 12:13 |
| quiquell|rover | weshay: also jobs are not skipped in the pipeline :-/ | 12:14 |
| *** panda|lunch is now known as panda | 12:14 | |
| weshay | rlandy|ruck quiquell|rover ah.. it was the change from docker to buildah | 12:15 |
| weshay | the check jobs don't push anything | 12:15 |
| quiquell|rover | weshay: yep | 12:15 |
| quiquell|rover | weshay: but even those jobs are periodic and fail the other periodic jobs are not skipped (that's new bug though) | 12:15 |
| weshay | wtf /me doesn't understand this https://review.rdoproject.org/r/#/c/20142/3/playbooks/tripleo-ci-periodic-base/containers-push-registry-login.yaml | 12:15 |
| weshay | noting I've 0 coffee | 12:15 |
| weshay | quiquell|rover | 12:16 |
| weshay | right | 12:16 |
| quiquell|rover | weshay: sudo buildah push | 12:16 |
| quiquell|rover | weshay: you have to do docker login as root | 12:16 |
| weshay | w/ items: -true -false | 12:16 |
| weshay | k | 12:16 |
| weshay | that I get | 12:16 |
| quiquell|rover | weshay: we use that for rocky and others | 12:16 |
| quiquell|rover | weshay: they do docke rpush | 12:16 |
| weshay | were is the second {{ item }} | 12:16 |
| weshay | OH | 12:17 |
| weshay | so what.. you run it twice? | 12:17 |
| quiquell|rover | one for root one for non-root | 12:17 |
| quiquell|rover | so both have the credentials | 12:17 |
| weshay | heh | 12:17 |
| weshay | quiquell|rover is tricky | 12:17 |
| quiquell|rover | yep | 12:17 |
| quiquell|rover | is this | 12:17 |
| weshay | https://images.app.goo.gl/RJvF7VcdfXXceicg6 | 12:18 |
| quiquell|rover | or use investigate if we have remove sudo from buildah push at tripleo-common | 12:18 |
| weshay | quiquell|rover totally fine to merge and try it | 12:18 |
| weshay | and my favorite meme https://images.app.goo.gl/197GEgZnamwMj5V59 | 12:19 |
| quiquell|rover | is merged already | 12:19 |
| *** panda has quit IRC | 12:19 | |
| quiquell|rover | weshay: but what worries me is jobs not being skipped | 12:19 |
| quiquell|rover | weshay: take a look at zuul status in opestack-periodic | 12:19 |
| weshay | hrm | 12:19 |
| weshay | k | 12:19 |
| rfolco | quiquell|rover, so the sudo patch would fix it ? | 12:19 |
| quiquell|rover | it should | 12:20 |
| quiquell|rover | rfolco: still jobs are not skipped after periodic buildah fail in the pipeline :-/ | 12:20 |
| rfolco | quiquell|rover, weird | 12:20 |
| quiquell|rover | yep | 12:21 |
| quiquell|rover | :-/ | 12:21 |
| weshay | ya.. something busted our pipeline | 12:21 |
| quiquell|rover | I thoug it was because buildah job is non-voting | 12:21 |
| quiquell|rover | but is not the case | 12:21 |
| weshay | ah crap... quiquell|rover to ensure this doesn't promote | 12:21 |
| weshay | we should add the build jobs to criteria | 12:21 |
| quiquell|rover | weshay: yep | 12:21 |
| quiquell|rover | the promote to dlrn though | 12:21 |
| quiquell|rover | I mean report | 12:22 |
| weshay | k.. /me puts up a review | 12:22 |
| quiquell|rover | then we have f28 standalone failing at ovs | 12:24 |
| quiquell|rover | but the skipping job thing is super weird | 12:24 |
| *** jpena|lunch is now known as jpena | 12:26 | |
| chandankumar | arxcruz: need some help here https://tree.taiga.io/project/tripleo-ci-board/task/898 there are 6 plugins whose tag is not created upstream | 12:27 |
| chandankumar | arxcruz: I have added 3 for you, and rest 3 on me | 12:28 |
| weshay | quiquell|rover rlandy|ruck https://review.rdoproject.org/r/20147 | 12:28 |
| weshay | so it's either a config bug or a zuul bug | 12:28 |
| *** panda has joined #oooq | 12:28 | |
| quiquell|rover | weshay: it's not zuul, look https://review.rdoproject.org/r/#/c/20143/ | 12:30 |
| weshay | suspect change https://github.com/rdo-infra/review.rdoproject.org-config/commit/d611d43ad7c338d8d7b3e9258b356acb40a4dd4d#diff-654a75b6d629a8b43d978089d4135353 | 12:30 |
| weshay | seems ok | 12:31 |
| quiquell|rover | that's ok | 12:32 |
| *** dsneddon has joined #oooq | 12:32 | |
| *** amoralej|lunch is now known as amoralej | 12:33 | |
| weshay | zbr hello | 12:33 |
| weshay | zbr anything you need to chat about? | 12:33 |
| zbr | weshay: neah. | 12:34 |
| rfolco | quiquell|rover, the push part is in run or post ? | 12:34 |
| quiquell|rover | run | 12:35 |
| rfolco | quiquell|rover, I think we never failed push only, we failed to build containers and it stopped the following jobs | 12:35 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (2 more messages) | 12:35 |
| quiquell|rover | build and push is done in the same openstack command | 12:36 |
| rfolco | quiquell|rover, it does not make sense, zuul should not run the other jobs then | 12:36 |
| quiquell|rover | I am going to ask at RDO guys | 12:37 |
| quiquell|rover | the only difference is the non-voting stuff | 12:37 |
| *** dsneddon has quit IRC | 12:38 | |
| *** trown has quit IRC | 12:39 | |
| weshay | quiquell|rover where are you talking to them? | 12:42 |
| quiquell|rover | not doing it yet | 12:43 |
| quiquell|rover | #rdo | 12:43 |
| weshay | sshnaidm fyi.. second attempt.. not ready to merge.. but wanted ur thoughts as you have time https://review.openstack.org/#/c/651654/ | 12:44 |
| rlandy|ruck | marios: hi ... | 12:44 |
| rlandy|ruck | marios: https://code.engineering.redhat.com/gerrit/#/c/167633/ - we would need to merge that if we want to use it | 12:45 |
| weshay | quiquell|rover in your time https://review.rdoproject.org/r/#/c/20147/ | 12:46 |
| rlandy|ruck | I can put nhicher on the review and get his take - it was my impression that the rhel-8 image would require more than that - just guessing | 12:46 |
| marios | rlandy|ruck: ack... so that is just copy/paste from the rhel-7 one.. i am trying to put skeleton in place. perhaps we can keep this review around, and change th job to use the cloud-rhel8 one if that works enough for a job run | 12:47 |
| rlandy|ruck | marios: I tried cloud-8 - no dice | 12:48 |
| marios | rlandy|ruck: and fix this review as we find out more | 12:48 |
| marios | rlandy|ruck: ah so then we definitely need this | 12:48 |
| rlandy|ruck | marios: ack - let's put nhicher on the review and let him comment and we can fix from there | 12:48 |
| weshay | marios rlandy|ruck fyi.. I heard they still don't have overcloud images for 15 | 12:48 |
| sshnaidm | weshay, I'm not sure it will work when mirror is off, need to test it | 12:48 |
| marios | rlandy|ruck: sounds good thanks | 12:48 |
| sshnaidm | weshay, but in general seems ok | 12:49 |
| rlandy|ruck | weshay: either way we need the rhel-8 image | 12:49 |
| weshay | aye | 12:49 |
| weshay | true | 12:49 |
| rlandy|ruck | and for standalone, we don't care | 12:49 |
| rlandy|ruck | marios could happily get by as we are | 12:49 |
| rlandy|ruck | with rhel image | 12:49 |
| weshay | aye | 12:49 |
| rlandy|ruck | marios: you can see the patch I tried with cloud-8 | 12:49 |
| quiquell|rover | weshay: I think it has to be the name of the job | 12:50 |
| rlandy|ruck | sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/4/check/tripleo-ci-rhel-8-standalone-rhos-15/5fb617c/job-output.txt.gz | 12:50 |
| quiquell|rover | weshay: not the toci_job_type | 12:50 |
| rlandy|ruck | quiquell|rover: ^^ also thought so but it's reporting to dlrn like that | 12:50 |
| weshay | rlandy|ruck marios probably want to install python selinux in pre.yaml playbook | 12:51 |
| weshay | should be on the image imho, but better to be explicit too | 12:52 |
| *** ykarel|afk is now known as ykarel | 12:52 | |
| weshay | arxcruz can you carve a little time for me today? | 12:52 |
| marios | weshay: ack you mean rhel8osp15 | 12:53 |
| *** ratailor__ has quit IRC | 12:55 | |
| weshay | marios re: overcloud images ya.. mburns was saying overcloud images are not yet built in brew | 12:55 |
| marios | weshay: k. i made the wording here about 'put the definitions in place' so for now doing just that https://tree.taiga.io/project/tripleo-ci-board/us/952?milestone=226267 | 12:56 |
| marios | weshay: thanks noted | 12:56 |
| quiquell|rover | weshay, rfolco: pipeline is working as expected | 12:56 |
| quiquell|rover | I don't know what we show at status | 12:56 |
| quiquell|rover | It could have being previous runs ? | 12:57 |
| rfolco | quiquell|rover, ok so false alarm | 12:58 |
| weshay | rlandy|ruck just saw this on the list.. not sure if we want to use it http://download-node-02.eng.bos.redhat.com/rel-eng/latest-RHEL-8/compose/BaseOS/x86_64/images/rhel-guest-image-8.0-1835.x86_64.qcow2 | 12:58 |
| quiquell|rover | yep | 12:58 |
| * rlandy|ruck looks | 12:58 | |
| *** quiquell|rover is now known as quiquell|lunch | 12:59 | |
| *** udesale has joined #oooq | 13:00 | |
| rfolco | panda, can you help me understand why zuul is complaning here ? https://review.openstack.org/#/c/651362/5/zuul.d/build-containers.yaml... then on patchset7, I re-add -buildah job and zuul is happy again. | 13:02 |
| rfolco | is it because I am removing a voting job ? | 13:02 |
| rlandy|ruck | marios: asking about your review | 13:04 |
| rlandy|ruck | let's get that through one way or another | 13:04 |
| panda | rfolco: mmhh | 13:05 |
| rfolco | panda, never seen this before, super weird | 13:06 |
| rlandy|ruck | weshay: to use that image, we'd need some base empty node | 13:07 |
| quiquell|lunch | rfolco: looks like comment on old patchset ? | 13:08 |
| rfolco | quiquell|lunch, what you mean? I removed -buildah job completely, and zuul complains -buildah is undefined | 13:09 |
| rfolco | quiquell|lunch, removed from layout | 13:09 |
| quiquell|lunch | zuul is working in the review | 13:09 |
| quiquell|lunch | at patchset 7 | 13:10 |
| *** dsneddon has joined #oooq | 13:10 | |
| rfolco | quiquell|lunch, I added it back | 13:10 |
| rfolco | quiquell|lunch, and parent it to the new one without -buildah | 13:10 |
| sshnaidm | where is defined container name like "centos"-binary-... ? | 13:12 |
| sshnaidm | and not fedora-binary or something else | 13:12 |
| quiquell|lunch | sshnaidm: you mean weher is composed ? | 13:13 |
| sshnaidm | quiquell|lunch, yeah | 13:13 |
| quiquell|lunch | sshnaidm: at tripleo-common | 13:13 |
| quiquell|lunch | in the builders | 13:13 |
| marios | rlandy|ruck: which one? /me reads back | 13:13 |
| quiquell|lunch | There is a jinja template | 13:13 |
| quiquell|lunch | with the names of all the containers | 13:13 |
| rlandy|ruck | marios; your review to add rhel-8 node - nhicher is commenting there | 13:14 |
| panda | this must be one of those bugs in zuul like "you can't use jinja tags in commit messages" or "you can't use quote in comments" | 13:14 |
| marios | rlandy|ruck: ah ack didn't see looking | 13:14 |
| marios | rlandy|ruck: don't see comment yet | 13:14 |
| *** dsneddon has quit IRC | 13:14 | |
| marios | rlandy|ruck: https://code.engineering.redhat.com/gerrit/#/c/167633/ | 13:14 |
| marios | ? | 13:14 |
| rlandy|ruck | he just started | 13:14 |
| *** Goneri has joined #oooq | 13:15 | |
| marios | rlandy|ruck: ah k thx | 13:15 |
| marios | rlandy|ruck: will check | 13:15 |
| marios | rlandy|ruck: later | 13:15 |
| rlandy|ruck | marios: you can join #prodchain-infra-dfg | 13:15 |
| quiquell|lunch | sshnaidm: https://github.com/openstack/tripleo-common/search?q=name_prefix&unscoped_q=name_prefix | 13:15 |
| sshnaidm | quiquell|lunch, thanks... something passes {{name_prefix}} as "redhat" in rhel jobs.. | 13:16 |
| rfolco | panda, yes, probably | 13:17 |
| quiquell|lunch | rdhat-binary | 13:17 |
| quiquell|lunch | or just redhat ? | 13:17 |
| quiquell|lunch | sshnaidm: also https://github.com/openstack/tripleo-common/blob/5fa9fadee43a9f235e13dc62bc62f34d66d8ca4b/tripleo_common/image/builder/buildah.py#L94 | 13:18 |
| panda | rfolco: wait, I see buildah was still mentioned in ps5, but zull complains on a different thing in PS6 | 13:18 |
| rfolco | panda, I tried a different approach on ps6 | 13:19 |
| rlandy|ruck | quiquell|lunch: btw .. here is our experiment on filtering containers https://review.openstack.org/#/c/651662/ | 13:19 |
| rfolco | panda, coz did not make sense | 13:19 |
| rlandy|ruck | quiquell|lunch: filtering takes no effect | 13:19 |
| panda | rfolco: but I still see the -buildah job in the project-template in PS5 | 13:20 |
| rfolco | panda, do you? | 13:20 |
| rfolco | looking again | 13:20 |
| quiquell|lunch | rlandy|ruck: that's not filtered ? | 13:20 |
| rlandy|ruck | quiquell|lunch: bildah maybe - docker not | 13:20 |
| rfolco | panda, no, ps5 does not have -buildah | 13:20 |
| quiquell|lunch | but --exclude is correctly constructed | 13:21 |
| quiquell|lunch | I think it excludes the push but not the build | 13:21 |
| rfolco | panda, zuul is caching its config and not reloading it I suspect | 13:21 |
| quiquell|lunch | rfolco: this will work ? http://logs.openstack.org/62/651662/4/check/tripleo-build-containers-centos-7/ba44641/logs/build_containers.sh.txt.gz | 13:22 |
| quiquell|lunch | like multiple exclude lines ? | 13:22 |
| quiquell|lunch | yep working alright excludes=[u'opendaylight', u'neutron-server-opendaylight'] | 13:22 |
| rfolco | yes, but should not have \ in the last line | 13:23 |
| *** dsneddon has joined #oooq | 13:30 | |
| amoralej | podman and buildah have been upload in master, we have tested it with several oooq scenarios but let us know if you observe anything abnormal | 13:31 |
| amoralej | s/upload/updated/ | 13:33 |
| quiquell|lunch | good I am ruck/rover let's the party start :-) | 13:36 |
| *** quiquell|lunch is now known as quiquell|rover | 13:37 | |
| amoralej | is designate included in any oooq deployment featureset? | 13:39 |
| amoralej | quiquell|rover, ^ | 13:39 |
| amoralej | any idea? | 13:39 |
| chandankumar | weshay: sshnaidm zbr https://tree.taiga.io/project/tripleo-ci-board/us/854 feel free to take a look at collect-logs, tasks also added | 13:39 |
| rfolco | jpena, have you seen this before ? zuul is apparently not refreshing its config, I deleted -buildah job and it complains the job is undefined | 13:40 |
| rfolco | jpena, https://review.openstack.org/#/c/651362/5/zuul.d/build-containers.yaml | 13:40 |
| * chandankumar headed home | 13:40 | |
| quiquell|rover | amoralej: checking | 13:40 |
| *** vinaykns has joined #oooq | 13:40 | |
| zbr | chandankumar: all tasks parked? nothing to review/ | 13:40 |
| amoralej | quiquell|rover, just found it in scenario003 | 13:41 |
| amoralej | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-stein/8f80d74/logs/undercloud/var/log/containers/designate/ | 13:41 |
| rlandy|ruck | quiquell|rover: correct - the push we are considering https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2 and we shouldn't be | 13:42 |
| rlandy|ruck | see discussion on ruck/rover eitherpas | 13:43 |
| rlandy|ruck | etherpad | 13:43 |
| rlandy|ruck | the failure we saw in push was actually from neutron-server-opendaylight | 13:43 |
| quiquell|rover | amoralej: I only see it at scenario003 at tht | 13:45 |
| quiquell|rover | fs 7/14/18/40 it is | 13:45 |
| *** d0ugal has quit IRC | 13:46 | |
| amoralej | quiquell|rover, good enough | 13:48 |
| amoralej | thanks | 13:48 |
| quiquell|rover | amoralej: ack will go back if we found some podman issue | 13:49 |
| quiquell|rover | rlandy|ruck: so I donÂ't get it, exclude is not working ? | 13:49 |
| rlandy|ruck | quiquell|rover: one sec - just sending panda creds | 13:50 |
| quiquell|rover | ack | 13:50 |
| zbr | quiquell|rover: rlandy|ruck : one older tox.ini fix in https://review.openstack.org/#/c/640463/ -- already reviewed but got outdated ... | 13:51 |
| quiquell|rover | +2 | 13:53 |
| rlandy|ruck | zbr: don | 13:55 |
| rlandy|ruck | done | 13:55 |
| zbr | thanks! | 13:56 |
| arxcruz | weshay: sure boss | 13:56 |
| rlandy|ruck | w+ed | 13:56 |
| jpena | rfolco: I can only think that the -buildah job is referenced elsewhere, maybe? Either that, or Zuul doesn't like deleting the job at the same time as its reference | 13:57 |
| zbr | wow, finally "I see colors!" in zuul logs... | 13:57 |
| weshay | zbr ooh.. let me see | 13:57 |
| zbr | https://zuul.openstack.org/stream/767131a6e0f14955b2619f953402c407?logfile=console.log | 13:57 |
| rfolco | jpena, second option, I m 100% sure -buildah is not in use anywhere else :) | 13:57 |
| zbr | zuul loger itself is not colored but the output from the commands finally works with ANSI colors. | 13:58 |
| rfolco | zbr, what? zuul is not colorblind anymore ? | 13:58 |
| rfolco | zbr, ok the stream is gone, can't see it for the one you just paste | 13:59 |
| zbr | rfolco: partially, console got new feathers.... lets hope people will not start abusing it with emoji | 13:59 |
| *** aakarsh|2 has quit IRC | 14:00 | |
| rfolco | zbr, its a good idea -- gifs !! | 14:00 |
| zbr | oops, closed browser by mistake. i will post a photo on next run. | 14:00 |
| rfolco | ping scrum marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, zbr, kopecmartin | 14:01 |
| *** d0ugal has joined #oooq | 14:01 | |
| rfolco | weshay, busy in another call ? | 14:02 |
| *** gkadam has joined #oooq | 14:02 | |
| zbr | one thing that I wished doing was to make the console log link persistent on zuul: here is the trick: when job starts a bootrap index.html file is uploaded to the log server (which is only used to load the live stream). When jobs ends, the same file is replaced by the final job output. The benefit: one url known from start. | 14:02 |
| rfolco | zbr, join scrum pls | 14:02 |
| rlandy|ruck | quiquell|rover: hi - you are holding all the bugs - do you want me to take some? | 14:02 |
| *** gkadam has quit IRC | 14:03 | |
| *** skramaja has quit IRC | 14:03 | |
| quiquell|rover | rlandy|ruck: well I am just with this now https://bugs.launchpad.net/tripleo/+bug/1824315 | 14:03 |
| openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 14:03 |
| quiquell|rover | rlandy|ruck: not looking at the others | 14:04 |
| rlandy|ruck | quiquell|rover: ack ok - just updating escalation board | 14:04 |
| quiquell|rover | Let me check the board | 14:04 |
| rlandy|ruck | quiquell|rover: I moved all the other cards - working on the last one | 14:05 |
| rlandy|ruck | quiquell|rover: pls ping before you log off and let me know if you want me to pick that last bug up | 14:05 |
| rlandy|ruck | quiquell|rover: I don;t think we are sure of the root case/ dfg there yet | 14:06 |
| quiquell|rover | rlandy|ruck: will do I think is the last promotion blocker for now | 14:06 |
| rlandy|ruck | will check in with arxcruz | 14:06 |
| rlandy|ruck | quiquell|rover++ | 14:06 |
| hubbot1 | rlandy|ruck: quiquell|rover's karma is now 7 | 14:06 |
| quiquell|rover | rlandy|ruck: upstream is working now | 14:06 |
| rlandy|ruck | good | 14:07 |
| rlandy|ruck | will watch | 14:07 |
| quiquell|rover | rlandy|ruck: i mean f28 standalone works at openstack infra | 14:07 |
| quiquell|rover | rlandy|ruck: noop https://review.openstack.org/#/c/560445 | 14:07 |
| quiquell|rover | rlandy|ruck: looks like br-ex and br-int are down | 14:08 |
| quiquell|rover | rlandy|ruck: nova-compute is not able to create a port | 14:08 |
| marios | weshay: https://review.openstack.org/#/c/651725/ | 14:10 |
| quiquell|rover | rlandy|ruck: ovb job failing pulling containers ? | 14:11 |
| weshay | marios ya.. I think that also makes sense | 14:11 |
| quiquell|rover | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/93bfd82/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 14:11 |
| rlandy|ruck | eek - just answering sprint question | 14:11 |
| quiquell|rover | is periodic containers are suppose to be there | 14:11 |
| quiquell|rover | ok | 14:11 |
| *** panda has quit IRC | 14:15 | |
| quiquell|rover | holy shit could be that an OVB job is runiing in the middle of consistent-to-tripleo-testing ? | 14:16 |
| quiquell|rover | and container push ? | 14:16 |
| quiquell|rover | like older ovb job periodic one | 14:16 |
| quiquell|rover | rlandy|ruck: drop now, you can take the bug if you want https://bugs.launchpad.net/tripleo/+bug/1824315 | 14:26 |
| openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 14:26 |
| quiquell|rover | rlandy|ruck: but openstack infra zuul is runnin f28 standalone alright | 14:27 |
| rlandy|ruck | oh man | 14:27 |
| * rlandy|ruck looks | 14:27 | |
| rlandy|ruck | parallel process | 14:27 |
| quiquell|rover | rlandy|ruck: let's wait for next spin | 14:31 |
| quiquell|rover | periodic is kicking in now | 14:31 |
| rlandy|ruck | quiquell|rover: ok - watching | 14:32 |
| rlandy|ruck | quiquell|rover:order look sok now | 14:33 |
| quiquell|rover | periodic-tripleo-centos-7-master-containers-buildah retry limit | 14:33 |
| rlandy|ruck | some skipped | 14:33 |
| rlandy|ruck | nothing started | 14:33 |
| * rlandy|ruck check node failure | 14:33 | |
| quiquell|rover | the others are running alright | 14:33 |
| rlandy|ruck | shoot - cloud may be overloaded | 14:33 |
| rlandy|ruck | problematic heat stacks | 14:33 |
| rlandy|ruck | cleaning up | 14:34 |
| *** dsneddon has quit IRC | 14:34 | |
| quiquell|rover | ok | 14:34 |
| quiquell|rover | droping now | 14:34 |
| *** quiquell|rover is now known as quiquell|off | 14:34 | |
| *** aakarsh|2 has joined #oooq | 14:34 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (2 more messages) | 14:35 |
| rlandy|ruck | quiquell|off: nah - tenant is clean | 14:36 |
| *** udesale has quit IRC | 14:38 | |
| *** ykarel is now known as ykarel|away | 14:38 | |
| rlandy|ruck | weshay: marios: one more for the osp-14 work ... adding the rocky job ... https://review.openstack.org/#/c/651247/ | 14:44 |
| rlandy|ruck | pls review/comment | 14:44 |
| weshay | rlandy|ruck sorry dumb question, is there a patch that runs standalone rocky? | 14:46 |
| * weshay wondering why it's not triggered in this review | 14:46 | |
| rlandy|ruck | weshay: that is part of the question in the review | 14:47 |
| rlandy|ruck | where should this run | 14:47 |
| rlandy|ruck | see commit message | 14:47 |
| *** ykarel|away has quit IRC | 14:47 | |
| weshay | rlandy|ruck follow the pattern in stein upstream, start w/ non-voting | 14:47 |
| *** aakarsh|2 is now known as aakarsh | 14:55 | |
| *** amoralej is now known as amoralej|mtg | 14:57 | |
| rlandy|ruck | weshay: next promotion kicked | 15:01 |
| arxcruz | rlandy|ruck: weshay sshnaidm panda zbr marios https://review.rdoproject.org/r/#/c/20140/ | 15:01 |
| arxcruz | this is for scenario004 | 15:01 |
| rlandy|ruck | I asked about retry failure - looks like infra thing | 15:01 |
| arxcruz | everything passing | 15:01 |
| weshay | arxcruz are you avail in 28min? | 15:02 |
| arxcruz | weshay: yes | 15:02 |
| *** Vorrtex has joined #oooq | 15:02 | |
| sshnaidm | arxcruz, "run_tempest: false" ? | 15:03 |
| sshnaidm | https://review.rdoproject.org/r/#/c/20140/2/zuul.d/standalone-jobs.yaml | 15:03 |
| arxcruz | sshnaidm: this is to skip the validate-tempest role | 15:03 |
| sshnaidm | arxcruz, I see.. but it's very confusing | 15:04 |
| marios | ack arxcruz will check | 15:04 |
| arxcruz | sshnaidm: yeah, it's the only sollution now... | 15:04 |
| arxcruz | once we deprecate validate-tempest this can be removed | 15:04 |
| sshnaidm | arxcruz, where is it tested/running? | 15:05 |
| arxcruz | sshnaidm: in the comment | 15:05 |
| arxcruz | sshnaidm: https://review.rdoproject.org/r/#/c/20141/ | 15:05 |
| sshnaidm | arxcruz, ack | 15:05 |
| arxcruz | rlandy|ruck: regarding f28 issue, i'm looking here because we had this before, but i can't find it atm, but i'll investigate :) | 15:07 |
| *** dsneddon has joined #oooq | 15:07 | |
| arxcruz | i remember i discus this with zbr | 15:07 |
| arxcruz | but can't remember right now | 15:07 |
| rlandy|ruck | arxcruz++ | 15:07 |
| rlandy|ruck | thanks | 15:07 |
| hubbot1 | rlandy|ruck: arxcruz's karma is now 16 | 15:07 |
| *** aakarsh has quit IRC | 15:11 | |
| *** dsneddon has quit IRC | 15:12 | |
| rlandy|ruck | rfolco: pls check latest promotions | 15:14 |
| rlandy|ruck | buildah jobs are passing | 15:14 |
| rlandy|ruck | pls check it is as you expect - so we are not missing anything | 15:15 |
| *** dsneddon has joined #oooq | 15:15 | |
| rlandy|ruck | looks like we pushed 22 mins ago ... https://console.registry.rdoproject.org/registry#/?namespace=tripleomaster | 15:16 |
| *** saneax has quit IRC | 15:17 | |
| *** ykarel|away has joined #oooq | 15:18 | |
| rfolco | rlandy|ruck, you mean the current one is working... latest finished one failed - https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-buildah | 15:18 |
| chandankumar | sshnaidm: weshay rlandy|ruck https://review.rdoproject.org/r/#/c/19916/ https://review.rdoproject.org/r/#/c/20037/ https://review.rdoproject.org/r/#/c/20061/ | 15:19 |
| chandankumar | needs reviews on these | 15:19 |
| rlandy|ruck | the master fail I know | 15:19 |
| rlandy|ruck | rfolco: more concerned out stein centos | 15:19 |
| rlandy|ruck | in registry | 15:19 |
| *** dsneddon has quit IRC | 15:20 | |
| sshnaidm | chandankumar, do these jobs run os_tempest on upstream too? | 15:23 |
| chandankumar | sshnaidm: currently standlone os_tempest tripleo is running aginst os_tempes | 15:25 |
| chandankumar | sshnaidm: goal was to first periodic then update upstream | 15:25 |
| chandankumar | *first port | 15:26 |
| sshnaidm | chandankumar, ook | 15:26 |
| weshay | arxcruz can you join my blue | 15:26 |
| arxcruz | weshay: yes | 15:26 |
| arxcruz | just a sec, grab something to drink | 15:26 |
| rlandy|ruck | chandankumar: https://review.rdoproject.org/r/#/c/20037/ +2'ed but will need rebase to merge | 15:29 |
| chandankumar | rlandy|ruck: done | 15:29 |
| rlandy|ruck | k - w'ed | 15:30 |
| chandankumar | rlandy|ruck: quiquell|off we need to keep an eye on scenario 1-4 jobs as they are moved to os_tempest | 15:30 |
| chandankumar | if anything breaks, let me & arxcruz know | 15:30 |
| *** jfrancoa has quit IRC | 15:30 | |
| chandankumar | then we can update the same in upstream | 15:30 |
| rlandy|ruck | chandankumar: same with https://review.rdoproject.org/r/#/c/20061/ | 15:30 |
| rlandy|ruck | rebase | 15:30 |
| rlandy|ruck | ok - will keep a watch | 15:30 |
| rlandy|ruck | rfolco: checked registry | 15:31 |
| rlandy|ruck | push looks ok now | 15:31 |
| rlandy|ruck | weshay: ^^ | 15:31 |
| chandankumar | rlandy|ruck: done | 15:31 |
| rfolco | rlandy|ruck, thanks for checking, need to take kid to school brb | 15:31 |
| chandankumar | rlandy|ruck: sshnaidm thanks :-) | 15:31 |
| rlandy|ruck | chandankumar: k - merged those - let's watch them | 15:32 |
| chandankumar | yup :-) | 15:33 |
| rlandy|ruck | ugh - undercloud install failures on ovb | 15:33 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein/5c570a7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-04-11_15_25_44 | 15:33 |
| rlandy|ruck | shoot can't find image | 15:33 |
| rlandy|ruck | rfolco: ^^ so close | 15:34 |
| chandankumar | rlandy|ruck: needs one time more https://review.rdoproject.org/r/#/c/20061/ +w | 15:34 |
| chandankumar | sshnaidm: one stupid question is it possible to add two nics to an openstack instances? | 15:35 |
| rlandy|ruck | https://console.registry.rdoproject.org/registry#/images/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 15:35 |
| rlandy|ruck | image is there | 15:35 |
| *** d0ugal has quit IRC | 15:35 | |
| rlandy|ruck | we have an issue | 15:36 |
| rlandy|ruck | rfolco: weshay; the jobs are not alright :( ... can't find images - logging bug | 15:37 |
| *** amoralej|mtg is now known as amoralej | 15:39 | |
| rlandy|ruck | docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 15:39 |
| rlandy|ruck | rfolco: ping - need some help here | 15:43 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1824388 | 15:44 |
| openstack | Launchpad bug 1824388 in tripleo "periodic jobs are failing undercloud install - Not found image" [Critical,Triaged] | 15:44 |
| chandankumar | sshnaidm: anything more needed on collect-logs changes https://review.openstack.org/#/c/650613/ | 15:46 |
| chandankumar | I have fixed shell indentation | 15:46 |
| *** hamzy has quit IRC | 15:47 | |
| *** ccamacho has quit IRC | 15:49 | |
| *** dsneddon has joined #oooq | 15:49 | |
| *** dsneddon has quit IRC | 15:55 | |
| *** panda has joined #oooq | 15:59 | |
| marios | ttyl folks | 16:00 |
| chandankumar | see ya #oooq warriors tomorrow :-) | 16:01 |
| *** chandankumar is now known as raukadah | 16:01 | |
| *** dsneddon has joined #oooq | 16:01 | |
| *** marios has quit IRC | 16:02 | |
| *** quiquell|off has quit IRC | 16:14 | |
| *** aakarsh has joined #oooq | 16:24 | |
| rlandy|ruck | panda: hey - around? | 16:27 |
| rfolco | rlandy|ruck, I am looking at the buildah push issue | 16:28 |
| rlandy|ruck | rfolco: thanks - do you see the bug logged? | 16:29 |
| rfolco | rlandy|ruck, I have seen that bug before | 16:29 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1824388 | 16:29 |
| openstack | Launchpad bug 1824388 in tripleo "periodic jobs are failing undercloud install - Not found image" [Critical,Triaged] | 16:29 |
| rfolco | rlandy|ruck, isn't it familiar ? https://bugs.launchpad.net/bugs/1819632 | 16:29 |
| openstack | Launchpad bug 1819632 in tripleo "[gate] Error pulling image: invalid character '<' parsing 404 response" [Medium,Fix released] - Assigned to Emilien Macchi (emilienm) | 16:29 |
| weshay | sshnaidm you still around? | 16:31 |
| sshnaidm | weshay, yep | 16:31 |
| weshay | sshnaidm question re: ansible tags on the standalone job in repro.. | 16:31 |
| * weshay looking for a reproducer override on that | 16:31 | |
| rlandy|ruck | rfolco: see ovb error | 16:31 |
| weshay | not seeing it | 16:31 |
| sshnaidm | weshay, what do you mean? | 16:31 |
| weshay | sshnaidm standalone in rdo-cloud is not running w/ the same ansible tags as upstream | 16:32 |
| rfolco | rlandy|ruck, unable to pull image: Error determining manifest MIME type | 16:32 |
| weshay | tempest is not running.. job defined in upstream uses tags=all | 16:32 |
| weshay | the reproducer job uses, build,standalone | 16:32 |
| sshnaidm | weshay, what is your zuul.yaml? | 16:32 |
| weshay | it wasn't referenced there | 16:33 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein/5c570a7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 16:33 |
| rlandy|ruck | rfolco: ^^ no MIME error here | 16:33 |
| rlandy|ruck | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 16:33 |
| rlandy|ruck | this is clearly brokem | 16:33 |
| rlandy|ruck | broken | 16:33 |
| *** zbr has quit IRC | 16:34 | |
| *** panda has quit IRC | 16:34 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (2 more messages) | 16:35 |
| sshnaidm | weshay, can you paste your zuul.yaml? | 16:36 |
| rfolco | rlandy|ruck, skopeo inspect returns ok for that image | 16:36 |
| weshay | sshnaidm well.. I would but I already moved on to try the os-ansible tempest job w/ the repro | 16:37 |
| rfolco | rlandy|ruck, perhaps missing tag? | 16:37 |
| rlandy|ruck | there are errors everywhere | 16:37 |
| rlandy|ruck | could be related | 16:37 |
| weshay | sshnaidm I have another box I can kick it on | 16:37 |
| * weshay does that | 16:38 | |
| weshay | few min | 16:38 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario001-standalone-stein/4bf87d6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-04-11_15_38_24 | 16:38 |
| rlandy|ruck | rfolco: we clearly have a problem here | 16:38 |
| rlandy|ruck | "2019-04-11 15:38:22,357 ERROR: 22066 -- ['/usr/bin/podman', 'start', '-a', u'container-puppet-rabbitmq'] run failed after unable to find container container-puppet-rabbitmq: no container with name or ID container-puppet-rabbitmq found: no such container", | 16:39 |
| rlandy|ruck | containers not being found everywhere | 16:39 |
| *** zbr has joined #oooq | 16:40 | |
| *** bogdando has quit IRC | 16:42 | |
| rlandy|ruck | we get the mime error when the container was pushed with errors | 16:43 |
| rlandy|ruck | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario001-standalone-stein/4bf87d6/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?level=ERROR | 16:44 |
| *** panda has joined #oooq | 16:45 | |
| rlandy|ruck | ImageNotFoundException: Not found image: https://trunk.registry.rdoproject.org/v2/tripleostein/centos-binary-qdrouterd/manifests/ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 16:45 |
| panda | randy|ruck: more or less | 16:46 |
| weshay | sshnaidm ok.. it's coming up now.. few more min | 16:47 |
| rfolco | rlandy|ruck, I'll bj w/ zbr for few min as its getting late for him.... and go investigate this issue after. | 16:47 |
| rlandy|ruck | root access? | 16:48 |
| sshnaidm | weshay, https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L155 | 16:48 |
| sshnaidm | weshay, I see 2 tags only | 16:48 |
| *** dtantsur is now known as dtantsur|afk | 16:50 | |
| weshay | sshnaidm hrm | 16:51 |
| weshay | sshnaidm ya.. that's odd | 16:51 |
| rlandy|ruck | rfolco: ack ok - you can ping me when done - still looking at it | 16:51 |
| weshay | sshnaidm how would it ever run tempest? | 16:52 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-standalone.yml#L61 | 16:53 |
| weshay | so it should be running | 16:53 |
| * weshay checks my run | 16:53 | |
| weshay | zuul coming up | 16:54 |
| weshay | sshnaidm and you have not seen that happen? | 16:55 |
| sshnaidm | weshay, I had tempest running | 16:55 |
| weshay | k. | 16:55 |
| sshnaidm | weshay, not sure what is the problem, need to see your zuul.yaml | 16:56 |
| *** hamzy has joined #oooq | 17:00 | |
| *** aakarsh has quit IRC | 17:02 | |
| rlandy|ruck | 2019-04-11 14:38:28 | Pushing trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f image with: sudo buildah push --tls-verify=False trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 17:08 |
| rlandy|ruck | ^^ no errors | 17:08 |
| *** jpena is now known as jpena|off | 17:10 | |
| *** derekh has quit IRC | 17:11 | |
| *** amoralej is now known as amoralej|off | 17:21 | |
| *** ykarel|away has quit IRC | 17:21 | |
| *** panda has quit IRC | 17:31 | |
| rfolco | rlandy|ruck, you want to bj ? have any thoughts or still looking? | 17:34 |
| *** jtomasek has quit IRC | 17:37 | |
| *** ykarel|away has joined #oooq | 17:39 | |
| *** kopecmartin is now known as kopecmartin|off | 17:40 | |
| rlandy|ruck | rfolco: yep we can bj | 17:44 |
| rlandy|ruck | got a few ideas | 17:44 |
| rlandy|ruck | rfolco: containers not accessible | 17:47 |
| rlandy|ruck | possibly due to root log in | 17:48 |
| *** sshnaidm_ has joined #oooq | 17:52 | |
| *** ykarel|away has quit IRC | 17:53 | |
| *** sshnaidm has quit IRC | 17:53 | |
| rlandy|ruck | weshay: hi ... re: https://review.rdoproject.org/r/#/c/20142/3/playbooks/tripleo-ci-periodic-base/containers-push-registry-login.yaml | 17:56 |
| weshay | hey | 17:56 |
| rlandy|ruck | do you know why we need both true and false there? | 17:56 |
| weshay | true is for podman, false is for docker | 17:56 |
| weshay | kind of hacky | 17:56 |
| rlandy|ruck | container access is completely hosed | 17:56 |
| weshay | container access? | 17:57 |
| rlandy|ruck | well that is not conditional - it's both | 17:57 |
| weshay | ya.. runs it twice | 17:57 |
| weshay | let's chat about it | 17:57 |
| rlandy|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1824388 | 17:57 |
| openstack | Launchpad bug 1824388 in tripleo "periodic jobs are failing undercloud install - Not found image" [Critical,Triaged] | 17:57 |
| rlandy|ruck | weshay: can we bj? | 17:57 |
| weshay | I'm on | 17:58 |
| rlandy|ruck | joining | 17:58 |
| rfolco | rlandy|ruck, weshay can I join ? | 18:01 |
| rlandy|ruck | rfolco: pls | 18:01 |
| rfolco | is this about buildah? | 18:01 |
| rfolco | k | 18:01 |
| rlandy|ruck | indeed | 18:01 |
| rfolco | rlandy|ruck, bj id pls | 18:03 |
| weshay | rfolco https://bluejeans.com/4113567798 | 18:03 |
| *** sshnaidm_ is now known as sshnaidm|off | 18:03 | |
| rlandy|ruck | weshay: Not found image: docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-cron:ef0e50851536fee32f34ce8f4a8d88f4c39b7ada_fd5c775f | 18:11 |
| weshay | https://bugs.launchpad.net/bugs/1819632 | 18:14 |
| openstack | Launchpad bug 1819632 in tripleo "[gate] Error pulling image: invalid character '<' parsing 404 response" [Medium,Triaged] - Assigned to Emilien Macchi (emilienm) | 18:14 |
| *** ykarel|away has joined #oooq | 18:18 | |
| *** tosky has quit IRC | 18:26 | |
| *** holser_ has quit IRC | 18:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ (2 more messages) | 18:35 |
| *** vinaykns has quit IRC | 18:43 | |
| *** vinaykns has joined #oooq | 18:47 | |
| sshnaidm|off | weshay, https://review.openstack.org/#/c/651696/ | 18:55 |
| sshnaidm|off | weshay, tested in https://review.rdoproject.org/r/#/c/20131/ | 18:56 |
| *** holser_ has joined #oooq | 19:09 | |
| rfolco | rlandy|ruck, weshay: revert patch ready https://review.rdoproject.org/r/#/c/20159 | 19:39 |
| weshay | rlandy|ruck https://review.rdoproject.org/r/#/c/20159 | 19:40 |
| *** ykarel|away has quit IRC | 20:07 | |
| *** dsneddon has quit IRC | 20:16 | |
| *** Vorrtex has quit IRC | 20:17 | |
| *** dsneddon has joined #oooq | 20:21 | |
| *** holser_ has quit IRC | 20:29 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (2 more messages) | 20:36 |
| *** holser_ has joined #oooq | 20:44 | |
| *** Goneri has quit IRC | 20:45 | |
| *** hamzy has quit IRC | 20:46 | |
| rlandy|ruck | rfolco: weshay: pls vote on https://review.openstack.org/#/c/651010/ | 21:07 |
| rlandy|ruck | required for pidone | 21:07 |
| *** tosky has joined #oooq | 21:29 | |
| weshay | rlandy|ruck let me know if you have a sec.. to give advice on repro | 21:42 |
| * weshay looks at patch | 21:42 | |
| *** holser_ has quit IRC | 21:53 | |
| *** aakarsh has joined #oooq | 22:07 | |
| *** tosky has quit IRC | 22:15 | |
| rlandy|ruck | weshay: what's up? | 22:27 |
| rlandy|ruck | weshay: rdocloud stuck? | 22:29 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (2 more messages) | 22:36 |
| rlandy|ruck | still a ton of nova commits | 22:36 |
| *** vinaykns has quit IRC | 22:44 | |
| *** aakarsh has quit IRC | 22:45 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:04 | |
| *** rlandy|ruck|bbl has quit IRC | 23:21 | |
| *** rlandy|ruck|bbl has joined #oooq | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!