| *** rascasoft has quit IRC | 00:02 | |
| rlandy | blockinfile will skip if the content is there | 00:06 |
|---|---|---|
| rlandy | it will add content but not delete it | 00:06 |
| *** rlandy is now known as rlandy|afk | 00:09 | |
| *** rascasoft has joined #oooq | 00:21 | |
| *** tosky has quit IRC | 00:27 | |
| *** rascasoft has quit IRC | 00:29 | |
| *** rfolco has joined #oooq | 00:43 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph- (2 more messages) | 00:49 |
| *** jbadiapa has quit IRC | 01:26 | |
| *** jbadiapa has joined #oooq | 01:31 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-ovb- (2 more messages) | 02:49 |
| *** apetrich has quit IRC | 03:15 | |
| *** gkadam has joined #oooq | 03:36 | |
| *** rlandy|afk is now known as rlandy | 03:45 | |
| *** rlandy has quit IRC | 03:49 | |
| *** ykarel|away has joined #oooq | 03:56 | |
| *** skramaja has joined #oooq | 04:02 | |
| *** udesale has joined #oooq | 04:02 | |
| *** ykarel|away is now known as ykarel | 04:17 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 04:49 |
| *** invincible has quit IRC | 04:58 | |
| *** chandan_kumar has joined #oooq | 04:59 | |
| *** chandan_kumar is now known as chkumar|ruck | 04:59 | |
| *** ykarel is now known as ykarel|afk | 05:34 | |
| *** ykarel|afk has quit IRC | 05:38 | |
| *** ykarel|afk has joined #oooq | 05:53 | |
| *** ykarel|afk is now known as ykarel | 06:10 | |
| *** quique|rover|off is now known as quiquell|rover | 06:15 | |
| quiquell|rover | chkumar|ruck: o/ | 06:24 |
| *** ratailor has joined #oooq | 06:25 | |
| chkumar|ruck | quiquell|rover: \o/ | 06:31 |
| chkumar|ruck | quiquell|rover: currently scenario4 standalone and fs019 still blocking master promotion | 06:31 |
| chkumar|ruck | due to manila issue | 06:31 |
| quiquell|rover | chkumar|ruck: this don't fix it ? https://review.openstack.org/#/c/630925/ | 06:33 |
| quiquell|rover | chkumar|ruck: this https://bugs.launchpad.net/tripleo/+bug/1813911 ? | 06:39 |
| openstack | Launchpad bug 1813911 in tripleo "Manilla tests are failing in featureset019 and scenario004" [Critical,Triaged] | 06:39 |
| chkumar|ruck | quiquell|rover: nope still not fixed | 06:39 |
| chkumar|ruck | quiquell|rover: that review is already picked up | 06:39 |
| chkumar|ruck | we need to check manila container share image version | 06:40 |
| chkumar|ruck | quiquell|rover: how we check the ceph version here ? | 06:41 |
| quiquell|rover | chkumar|ruck: but the error is different | 06:41 |
| quiquell|rover | chkumar|ruck: let me check | 06:42 |
| chkumar|ruck | quiquell|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/6d03096/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-02-01_01_23_56 | 06:42 |
| chkumar|ruck | quiquell|rover: no the test error is same | 06:42 |
| quiquell|rover | chkumar|ruck: ack | 06:43 |
| quiquell|rover | chkumar|ruck: ceph http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/afaa26c/logs/undercloud/var/log/extra/docker/containers/ceph-mds-upstream-centos-7-rdo-cloud-tripleo-0000444787/docker_info.log.txt.gz | 06:45 |
| quiquell|rover | 12.2.9 | 06:45 |
| quiquell|rover | It's not updated no | 06:45 |
| quiquell|rover | nah wait thi is not the place | 06:46 |
| chkumar|ruck | it all depends on promotion na? | 06:46 |
| quiquell|rover | but this is very old ceph version 12.2.9 | 06:46 |
| quiquell|rover | More than old totally different | 06:47 |
| quiquell|rover | chkumar|ruck: ahh here http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/afaa26c/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz | 06:48 |
| quiquell|rover | 3.2.0 | 06:48 |
| quiquell|rover | not updated | 06:48 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 06:49 |
| quiquell|rover | chkumar|ruck: we are missing https://cbs.centos.org/koji/taskinfo?taskID=694345 | 06:50 |
| chkumar|ruck | quiquell|rover: we need to update the version | 06:51 |
| quiquell|rover | release files I think | 06:51 |
| quiquell|rover | let me check | 06:51 |
| ykarel | quiquell|rover, you were checking old logs | 06:51 |
| quiquell|rover | ykarel: is working now | 06:52 |
| quiquell|rover | ? | 06:52 |
| ykarel | check logs in the link chkumar|ruck shared | 06:52 |
| ykarel | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/6d03096/logs/subnode-2/var/log/extra/docker/docker_allinfo.log.txt.gz | 06:52 |
| ykarel | 192.168.24.1:8787/ceph/daemon:v3.2.1-stable-3.2-luminous-centos-7-x86_64 | 06:52 |
| chkumar|ruck | ykarel: how to find that which version of ceph is used? | 06:53 |
| ykarel | chkumar|ruck, u can check ceph logs | 06:53 |
| ykarel | should be luminous latest | 06:53 |
| ykarel | package list in manila-share container image can be checked at https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/6d03096/logs/subnode-2/var/log/extra/docker/containers/openstack-manila-share-docker-0/docker_info.log.txt.gz | 06:55 |
| quiquell|rover | ykarel: yep all good with versions http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5a484b6/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz | 06:55 |
| quiquell|rover | latest ^ | 06:55 |
| ykarel | yup 3.2.1 | 06:56 |
| quiquell|rover | still failing http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/5a484b6/logs/tempest.html | 06:56 |
| ykarel | as per tom last comment in lp 12.2.10 is needed | 06:57 |
| ykarel | but it's not in centos mirror yet | 06:58 |
| quiquell|rover | ykarel: but areng we using the right container with the right RPM inside ? | 06:58 |
| chkumar|ruck | ykarel: quiquell|rover https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/6d03096/logs/subnode-2/var/log/containers/manila/manila-share.log.txt.gz?level=ERROR | 06:59 |
| ykarel | quiquell|rover, ceph container looks correct | 06:59 |
| chkumar|ruck | I am not sure it is related | 06:59 |
| ykarel | but see tom last comment it needs in manila share container too | 06:59 |
| chkumar|ruck | ykarel: quiquell|rover Do we query all the rpms installed with in each containers? | 07:00 |
| ykarel | if u see package list i shared, 12.2.5 is there | 07:00 |
| ykarel | chkumar|ruck, yes | 07:00 |
| ykarel | chkumar|ruck, it's related the error u shared | 07:00 |
| ykarel | it's the same error as before | 07:00 |
| ykarel | i can see https://github.com/ceph/ceph/pull/24839/commits/f5906585f3a7df823b904e86ba0a11ea81327e10 only in master | 07:02 |
| ykarel | not sure they picked it up in luminous yet | 07:03 |
| quiquell|rover | ykarel: this will update upstream https://cbs.centos.org/koji/taskinfo?taskID=694345 | 07:03 |
| quiquell|rover | ykarel: so the manila container will update with new version | 07:03 |
| quiquell|rover | ykarel: is that it ? | 07:03 |
| quiquell|rover | ykarel: they just need to update manila image ? | 07:03 |
| ykarel | quiquell|rover, if i am right https://github.com/ceph/ceph/pull/24839/commits/f5906585f3a7df823b904e86ba0a11ea81327e10 is needed | 07:04 |
| *** holser_ has joined #oooq | 07:04 | |
| ykarel | if https://cbs.centos.org/koji/taskinfo?taskID=694345 has included it then as soon as it's released | 07:04 |
| quiquell|rover | ykarel: so there is no ceph version with fix yet ? | 07:04 |
| ykarel | and with next container build all will be fine | 07:04 |
| ykarel | quiquell|rover, don't know if gfidente has pulled that ceph commit | 07:05 |
| quiquell|rover | ykarel: in the spec file ? | 07:05 |
| ykarel | yes | 07:05 |
| quiquell|rover | ykarel: where are the ceph spec files ? | 07:09 |
| quiquell|rover | ykarel: I don't see any patch applied here https://cbs.centos.org/kojifiles/work/tasks/4347/694347/build.log | 07:11 |
| *** jfrancoa has joined #oooq | 07:15 | |
| *** kopecmartin|off is now known as kopecmartin | 07:18 | |
| quiquell|rover | ykarel: got the spec from the src.rpm | 07:19 |
| quiquell|rover | and source code | 07:19 |
| quiquell|rover | Let check the change | 07:19 |
| ykarel | quiquell|rover, ack, looks like it's not there | 07:20 |
| quiquell|rover | ykarel: nope | 07:21 |
| quiquell|rover | ykarel: And they are not patching it | 07:21 |
| quiquell|rover | ykarel: new cep RPM is needed | 07:21 |
| quiquell|rover | and update all containers | 07:21 |
| ykarel | tom barron comment: We need to get this back to luminous (via mimic?) and downstream into 3.2. | 07:22 |
| ykarel | https://github.com/ceph/ceph/pull/24839 | 07:22 |
| ykarel | can't get that patch in tags so atleast it not's backported | 07:22 |
| ykarel | but tom or gfindente would know better what's the plan | 07:23 |
| quiquell|rover | ykarel: So they have to backport ? | 07:23 |
| ykarel | so ask them ^^ | 07:23 |
| ykarel | they know more, we can just guess | 07:23 |
| quiquell|rover | ykarel: ack will ask | 07:23 |
| quiquell|rover | damn... | 07:23 |
| quiquell|rover | ykarel: thanks ! | 07:23 |
| quiquell|rover | ykarel: btw promoter is working fine I think | 07:24 |
| quiquell|rover | ykarel: I don't see any infra error now | 07:24 |
| ykarel | master not promoted yet | 07:24 |
| quiquell|rover | ykarel: but is because this ceph thing | 07:24 |
| ykarel | no man | 07:24 |
| quiquell|rover | no ? | 07:24 |
| ykarel | we need to promote hash from 29th | 07:24 |
| ykarel | https://trunk.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=e4a542b9a3ea6f459605ffbaa3c8af97eb81921f&distro_hash=a9a57ed85ecff3d0ad0dac8f7107b94487e1bdb9 | 07:24 |
| quiquell|rover | Holy sh.. what I am looking at then | 07:24 |
| ykarel | all jobs passed but no promtion | 07:24 |
| quiquell|rover | ykarel: we have a good hash and it's not promoting | 07:25 |
| ykarel | yup exactly | 07:25 |
| quiquell|rover | after restart rocky promoted so at least something is good there | 07:25 |
| quiquell|rover | ykarel: thanks let me check | 07:25 |
| quiquell|rover | ykarel: Y did the staff in the middle of a promotion :-/ | 07:25 |
| quiquell|rover | ykarel: maybe it's all mess up there now | 07:26 |
| ykarel | quiquell|rover, currently pike is promoting | 07:26 |
| ykarel | let's see master picks up after it | 07:26 |
| ykarel | when did u restart? | 07:26 |
| quiquell|rover | ykarel: last two job failed | 07:26 |
| quiquell|rover | ykarel: at the link, they are part of the criteria ? | 07:26 |
| ykarel | nope | 07:26 |
| ykarel | fs021 is not part | 07:27 |
| quiquell|rover | yep, just checked | 07:27 |
| quiquell|rover | ok | 07:27 |
| quiquell|rover | Let me check, maybe I have to untag something | 07:27 |
| ykarel | you can check master.log to find why it's not picking up | 07:27 |
| ykarel | or it will run after pike | 07:27 |
| chkumar|ruck | quiquell|rover: sshnaidm|off panda|off https://review.rdoproject.org/r/#/c/18666/ -> port clean up script | 07:29 |
| chkumar|ruck | I am facing some problem at updated_at field need some help there | 07:30 |
| *** holser_ has quit IRC | 07:31 | |
| quiquell|rover | ykarel: I don't find commit hash at master.log e4a542b9a3ea6f459605ffbaa3c8af97eb81921f | 07:31 |
| ykarel | see yesterdays | 07:32 |
| quiquell|rover | ykarel: how much hashes do we get from DLRN | 07:32 |
| quiquell|rover | I think we get 5 or so | 07:32 |
| ykarel | i remember 5 | 07:32 |
| *** rascasoft has joined #oooq | 07:32 | |
| quiquell|rover | ykarel: there is a lot of hashes in top of that https://trunk.rdoproject.org/centos7-master-head/report.html | 07:33 |
| quiquell|rover | Is a very old one | 07:33 |
| quiquell|rover | Or we go back in time until we find one that passes ? | 07:34 |
| ykarel | nope, u need to check promoter code | 07:34 |
| ykarel | which 5 hashes it fetches | 07:34 |
| ykarel | i think it fetches which were used as tripleo-ci-testing | 07:34 |
| quiquell|rover | so we have a job that tag consistent to tripleo-ci-testing | 07:35 |
| quiquell|rover | then we test on tripleo-ci-testing | 07:35 |
| quiquell|rover | Let's ask DLRN directly | 07:35 |
| ykarel | yup it's at 7th number | 07:36 |
| quiquell|rover | ykarel: we can try to change promoter to take 10 or wait until we fix ceph | 07:36 |
| ykarel | i think it's good to promote we are 7 days behind | 07:37 |
| quiquell|rover | ykarel: it's part of the config | 07:39 |
| quiquell|rover | ykarel: just master https://review.rdoproject.org/r/#/c/18667/ | 07:41 |
| quiquell|rover | ykarel: I am going to change it manually at server | 07:42 |
| ykarel | quiquell|rover, ack | 07:42 |
| quiquell|rover | It's at pike now | 07:43 |
| quiquell|rover | Next iteration we will see | 07:43 |
| quiquell|rover | panda|off, sshnaidm|off -> 10 hashes instead of 5 at master promoter https://review.rdoproject.org/r/#/c/18667/ | 07:44 |
| *** saneax has joined #oooq | 07:46 | |
| ykarel | just temporary ^^ | 07:50 |
| *** ykarel is now known as ykarel|lunch | 07:50 | |
| *** panda|off is now known as panda | 07:53 | |
| quiquell|rover | panda: o/ | 07:53 |
| quiquell|rover | panda: https://review.rdoproject.org/r/#/c/18667/ | 07:53 |
| panda | wow | 07:53 |
| quiquell|rover | panda: so bad ? | 07:55 |
| quiquell|rover | panda: have change it manually at server will fallback after master promotion | 07:55 |
| panda | quiquell|rover: no no that's ok, be aware that promotion in dashboard will still show the date of the hash, not the date of the promotion | 07:57 |
| chkumar|ruck | quiquell|rover: do we need to care nv failing jobs against noop reviews? | 07:57 |
| quiquell|rover | chkumar|ruck: for example f28 is a nv | 07:59 |
| quiquell|rover | chkumar|ruck: Depends on the jobs | 07:59 |
| quiquell|rover | chkumar|ruck: for example upgrades usually is wip of upgrades team | 08:00 |
| quiquell|rover | chkumar|ruck: but if everythig else is ok is not bad to look at them | 08:00 |
| chkumar|ruck | quiquell|rover: I will take a look on them | 08:00 |
| chkumar|ruck | quiquell|rover: except fs19/scenario4 everything is ok in promotion pipeline on master | 08:00 |
| chkumar|ruck | quiquell|rover: periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-pike -> showing disk full error | 08:01 |
| chkumar|ruck | quiquell|rover: I am working on f20/f21 tempest result comparison | 08:14 |
| quiquell|rover | ykarel|lunch: master promoting | 08:16 |
| quiquell|rover | panda: do we merge the 10 hashes thing ? | 08:17 |
| chkumar|ruck | quiquell|rover: yup merged | 08:19 |
| *** apetrich has joined #oooq | 08:25 | |
| *** ccamacho has joined #oooq | 08:29 | |
| panda | quiquell|rover: that's ancient history now. | 08:31 |
| chkumar|ruck | panda: do we want to run os_tempest non-voting with all scenario job template? | 08:31 |
| panda | chkumar|ruck: the idea is to replace validate tempest completely, right ? | 08:32 |
| chkumar|ruck | panda: yes | 08:32 |
| panda | chkumar|ruck: and we are ready to do it | 08:33 |
| chkumar|ruck | panda: one work is still left, reusing the current skip list of validate-tempest | 08:33 |
| chkumar|ruck | arxcruz: ^^ Are you looking at this ? If not I can propose a patch? | 08:33 |
| panda | chkumar|ruck: and what skip list are you using now ? | 08:33 |
| chkumar|ruck | panda: currently we are running a set of selective tests coming from osa https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L97 | 08:34 |
| chkumar|ruck | panda: with no skip list | 08:34 |
| chkumar|ruck | let me propose a patch for skip list part | 08:35 |
| panda | chkumar|ruck: then I would wait until we have the skip list, if you prefer we can add the os tempest jobs in experimental | 08:35 |
| *** ykarel|lunch is now known as ykarel | 08:36 | |
| *** tosky has joined #oooq | 08:36 | |
| *** holser_ has joined #oooq | 08:37 | |
| jfrancoa | chkumar|ruck: hey, good morning. Is there any lp bug for the undercloud-upgrades job failing? dciabrin was preparing a fix for it, so to link it to the patch | 08:38 |
| chkumar|ruck | jfrancoa: https://bugs.launchpad.net/tripleo/+bug/1811450 | 08:39 |
| openstack | Launchpad bug 1811450 in tripleo "Undefined variable target_upgrade_version in tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades" [High,Triaged] | 08:39 |
| jfrancoa | chkumar|ruck: thanks a lot | 08:39 |
| *** rfolco has quit IRC | 08:41 | |
| jfrancoa | chkumar|ruck: mmm...but this is for the overcloud upgrades job, not for tripleo-ci-centos-7-containerized-undercloud-upgrades | 08:41 |
| chkumar|ruck | jfrancoa: I think we donot have one | 08:42 |
| chkumar|ruck | jfrancoa: I will create it right now | 08:42 |
| jfrancoa | chkumar|ruck: and in fact, its a duplicate bug of https://bugs.launchpad.net/tripleo/+bug/1812403 (I'll add it to the lp) | 08:42 |
| openstack | Launchpad bug 1812403 in tripleo "scenario000-multinode-oooq-container-upgrades fails with target_upgrade_version undefined" [Undecided,In progress] - Assigned to Jose Luis Franco (jfrancoa) | 08:42 |
| *** rfolco has joined #oooq | 08:42 | |
| jfrancoa | chkumar|ruck: thank you | 08:44 |
| jfrancoa | marios: quiquell|rover panda when you have some time, could you please review https://review.openstack.org/#/c/607525/ ? It would be helpful to merge it, otherwise I need to add depends-on patches on this one to debug the failing tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades job (I set a goal to have it green by next week) | 08:45 |
| quiquell|rover | jfrancoa: this will break upgrade jobs ? | 08:46 |
| *** jpena|off is now known as jpena | 08:47 | |
| jfrancoa | quiquell|rover: no, the opposite..it will unblock it (it will still fail, but for other bug https://bugs.launchpad.net/tripleo/+bug/1814104) | 08:47 |
| openstack | Launchpad bug 1814104 in tripleo "tripleo-container-tag image pulling failing in upgrade_tasks" [High,In progress] - Assigned to Jose Luis Franco (jfrancoa) | 08:47 |
| jfrancoa | quiquell|rover: I'm trying to make the job green, but this target_upgrade_version issue is blocking the way | 08:48 |
| quiquell|rover | jfrancoa: do you have a testing review with the Depends-On and all ? | 08:49 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 08:49 |
| jfrancoa | quiquell|rover: no, because last time I tried to add a depends-on a tht patch, the build-test-packages was failing constantly. Anyway, I'll add it | 08:50 |
| quiquell|rover | jfrancoa: well upgrade job is already broken let's workflow this | 08:51 |
| quiquell|rover | jfrancoa: ack ? | 08:51 |
| *** dsneddon has quit IRC | 08:51 | |
| quiquell|rover | jfrancoa: they are not part of the promotion criteria also | 08:52 |
| quiquell|rover | jfrancoa: workflowed | 08:52 |
| *** dsneddon has joined #oooq | 08:53 | |
| quiquell|rover | jfrancoa: ups workflowed cleand after deepends-on | 08:54 |
| jfrancoa | quiquell|rover: ups...there was not sync :-D I have just added the depends-on the other issue. well, let's leave the CI run and I'll ping you later to +1W | 08:54 |
| quiquell|rover | jfrancoa: ok there +1W is in the ether so just ping me | 08:54 |
| sshnaidm|off | quiquell|rover, centos-7 job fails here https://review.openstack.org/#/c/633444/ | 08:55 |
| sshnaidm|off | quiquell|rover, not sure why.. if it's ok, we can merge it today | 08:55 |
| ykarel | quiquell|rover, cool | 08:56 |
| chkumar|ruck | jfrancoa: https://bugs.launchpad.net/tripleo/+bug/1814223 | 08:57 |
| openstack | Launchpad bug 1814223 in tripleo "undercloud_upgrade container job is failing with error stopping containers " [Critical,Triaged] | 08:57 |
| quiquell|rover | sshnaidm|off: this is going to be sweet http://docs.grafana.org/guides/whats-new-in-v6-0/#explore | 08:57 |
| quiquell|rover | weshay: http://docs.grafana.org/guides/whats-new-in-v6-0/#explore | 08:57 |
| jfrancoa | chkumar|ruck: thanks a lot | 08:58 |
| sshnaidm|off | quiquell|rover, yeah, looks niiiice | 09:00 |
| quiquell|rover | sshnaidm|off: opens the door of proper stuff exploration | 09:02 |
| *** bogdando has joined #oooq | 09:02 | |
| *** dsneddon__ has joined #oooq | 09:06 | |
| *** dsneddon has quit IRC | 09:10 | |
| *** holser_ has quit IRC | 09:11 | |
| arxcruz | chkumar|ruck: i'll work on this | 09:11 |
| zbr|ssbarnea | @oooq: do we need to hardcode cirros image in tqe or we could make the code pick last release and avoid extra maintenance? | 09:20 |
| zbr|ssbarnea | related to https://review.openstack.org/#/c/633941/3 | 09:20 |
| panda | I want hardcore cirros image | 09:22 |
| quiquell|rover | zbr|ssbarnea: we have them here too https://review.openstack.org/#/c/634213/1/roles/validate-simple/defaults/main.yml | 09:22 |
| panda | if they die, they die | 09:22 |
| panda | quiquell|rover: I need you | 09:22 |
| zbr|ssbarnea | panda: to me it looks like a recipe for extra maintenance for us, to keep them updated. do we have repeated cases where newer versions broke us badly? | 09:24 |
| quiquell|rover | panda: ack blue ? | 09:24 |
| panda | quiquell|rover: ack | 09:25 |
| panda | quiquell|rover: my blue | 09:25 |
| chkumar|ruck | arxcruz: oh sorry | 09:25 |
| chkumar|ruck | arxcruz: https://review.openstack.org/#/c/634380/ | 09:25 |
| chkumar|ruck | arxcruz: I am not sure it will work, feel free to take it over from here | 09:26 |
| arxcruz | chkumar|ruck: jesus man, you said you want to be ruck because you were full of tempest... | 09:26 |
| chkumar|ruck | arxcruz: sorry for that one, no more patches | 09:27 |
| marios | jfrancoa: ack sorry was afk for a bit will check | 09:37 |
| zbr|ssbarnea | quiquell|rover: chkumar|ruck : can we merge https://review.rdoproject.org/r/#/c/18649/ ? | 09:37 |
| jfrancoa | marios: thanks a lo | 09:37 |
| zbr|ssbarnea | panda: need help with the te-broker role cleanup: https://review.rdoproject.org/r/#/c/18614/ -- is only about removing tasks. | 09:39 |
| marios | jfrancoa: is the job meant to be red there (upgrades still broken i mean ) | 09:44 |
| marios | jfrancoa: (is this part of the fix but not THE fix yet??) | 09:45 |
| jfrancoa | marios: yes yes, you know how this works when you try to resurrect such a dead job...when you fix one issue a new one appears | 09:45 |
| marios | jfrancoa: ack no is fine just checking you want to merge this one as is then | 09:45 |
| jfrancoa | marios: yes, because otherwise I have to do that mess with depends-on patches to debug the new issues | 09:46 |
| jfrancoa | marios: anyway, as I told quiquell|rover let's wait for the CI results adding the depends-on, then I'll remove the dependency and we can try to merge the patch (as otherwise we would need to merge first the one on the depends-on) | 09:47 |
| marios | jfrancoa: k just ping if you need revote later | 09:48 |
| jfrancoa | marios: I will, thanks for having a look | 09:55 |
| marios | jfrancoa: no worries, i didn't have time to check the depends on to be honest have meeting starting now | 09:55 |
| zbr|ssbarnea | panda: can you please add https://tree.taiga.io/project/tripleo-ci-board/us/680 to the sprint? trivial leftover. | 09:56 |
| marios | panda: no bluejeans on the invite | 09:59 |
| panda | marios: zbr|ssbarnea my bj | 09:59 |
| marios | panda: would you like to share it please :D | 09:59 |
| marios | dont say gcerami | 09:59 |
| marios | numbers! | 09:59 |
| panda | marios: zbr|ssbarnea 3492508669 | 10:00 |
| marios | panda: thx joining | 10:00 |
| chkumar|ruck | sshnaidm|off: Hey | 10:04 |
| chkumar|ruck | sshnaidm|off: at this https://review.rdoproject.org/r/#/c/18666/2/ci-scripts/infra-cleanup/ovb-tenant-cleanup.sh@171 at using updated_at jq fails | 10:04 |
| chkumar|ruck | sshnaidm|off: need some help here | 10:04 |
| *** ratailor_ has joined #oooq | 10:08 | |
| *** derekh has joined #oooq | 10:09 | |
| Tengu | #headshot. pffff. we're pretty doomed with those unmanaged rules in fact.... UNLESS running the plain "iptables" command, we're stuck with them. | 10:10 |
| *** ratailor has quit IRC | 10:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 10:49 |
| quiquell|rover | chkumar|ruck: gates are timing out | 10:51 |
| quiquell|rover | chkumar|ruck: ann no 3 days old | 10:51 |
| chkumar|ruck | quiquell|rover: please donot say that | 10:52 |
| quiquell|rover | chkumar|ruck: hehe sorry | 10:55 |
| panda | quiquell|rover: never cry wolf! | 10:55 |
| *** fmount has quit IRC | 11:05 | |
| *** fmount has joined #oooq | 11:06 | |
| zbr|ssbarnea | @oooq does anyone knows how to detect docker group at runtime? different docker blends may have docker or dockerroot and I need to detect it with ansible/cli. | 11:14 |
| quiquell|rover | zbr|ssbarnea: /etc/docker/daemon.json | 11:16 |
| quiquell|rover | have them | 11:16 |
| zbr|ssbarnea | quiquell|rover: this file may not exist at all. | 11:16 |
| zbr|ssbarnea | quiquell|rover: is not an easy task because i cannot even use the distribution/os version to assume that as you could have the docker service installed from various sources (docker, docker-io, docker-ce). | 11:20 |
| zbr|ssbarnea | i tried to look on disk to find more info, still i was not able to find where is this even defined. | 11:22 |
| *** udesale has quit IRC | 11:23 | |
| marios | brb | 11:26 |
| *** marios has quit IRC | 11:26 | |
| panda | zbr|ssbarnea: I think it's defined in the rpm spec | 11:26 |
| *** marios has joined #oooq | 11:26 | |
| quiquell|rover | chkumar|ruck: quuens noop has just failed http://logs.rdoproject.org/24/567224/178/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/e9f6d25/ | 11:28 |
| quiquell|rover | chkumar|ruck: Don't know how important this is | 11:28 |
| zbr|ssbarnea | panda: so you propose inspecting the spec of a rpm for which I do not even know the name? i hope to find something easier than that. -- i was hoping to find a one-linter kind of solution, not to write a full scripts. | 11:29 |
| chkumar|ruck | quiquell|rover: chekcing | 11:29 |
| quiquell|rover | Also in master | 11:31 |
| quiquell|rover | http://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/ | 11:31 |
| panda | zbr|ssbarnea: then it's easy, just look at the owner of /var/run/docker.sock | 11:32 |
| panda | zbr|ssbarnea: s/owner/group | 11:32 |
| panda | zbr|ssbarnea: that is the entry point | 11:32 |
| panda | zbr|ssbarnea: that is the reason why the user needs to be in that group | 11:32 |
| quiquell|rover | chkumar|ruck: in master is raise exceptions.DeploymentError("Overcloud configuration failed.") | 11:32 |
| panda | zbr|ssbarnea: otherwise you cannot open that socket | 11:32 |
| quiquell|rover | Object "route add 0.0.0.0/0 via 10.0.0.1 dev br-ex" is unknown | 11:33 |
| panda | quiquell|rover: it should be "dev br-ex-it" otherwise the packets will remain in EU | 11:34 |
| * panda hides | 11:34 | |
| zbr|ssbarnea | panda: funny aspect, I need the docker_group expecially for fixing the permissions on the sock, to allow other users to talk to it :D | 11:34 |
| marios | https://review.openstack.org/#/c/633771/3 any objections? me hovers over merge button | 11:34 |
| panda | zbr|ssbarnea: then it's arbitrary | 11:34 |
| panda | zbr|ssbarnea: you set the group therem then modify deamon.json accordingly | 11:34 |
| zbr|ssbarnea | panda: i am afraid you may be right about it. | 11:34 |
| quiquell|rover | marios: do it do it, do you want me to do it ? | 11:35 |
| panda | zbr|ssbarnea: as always | 11:35 |
| panda | marios: -2 | 11:35 |
| * panda now looks at the review | 11:35 | |
| marios | panda: its the docs index move | 11:36 |
| marios | i've been spamming the chan for couple days | 11:36 |
| marios | no one complained | 11:36 |
| marios | i merged | 11:36 |
| marios | wdyt folks moving ci into new section https://review.openstack.org/#/c/633771 http://logs.openstack.org/71/633771/3/check/openstack-tox-docs/0ea6f8e/html/ thanks. no new content just rearranges sections in the index and adds standalone ci index @ /ci/index.html | 11:36 |
| marios | panda: like that ^ | 11:36 |
| marios | quiquell|rover: thanks done | 11:37 |
| panda | marios: did you ever read "the hichhiker's guide to the galaxy" ? | 11:37 |
| marios | panda: actually no but i am aware of it (and saw the move ;) ) | 11:37 |
| quiquell|rover | chkumar|ruck: 053 master same error http://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/f928570/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:38 |
| quiquell|rover | queens ? | 11:38 |
| quiquell|rover | queens look different | 11:38 |
| panda | marios: well in the first chapter of the book, aliens arrive to destroy the earth. THe announcement was clearly shown in the board of the sublevel alpha of the planet epsilon. Nobody complained fomr the earth, so they were going to wipe it out anyway. | 11:39 |
| chkumar|ruck | quiquell|rover: master one looks like somehting gone wrong with os-client-config tiries to ping the ip | 11:39 |
| panda | marios: :) | 11:39 |
| chkumar|ruck | not ping apply the network config | 11:39 |
| marios | panda: haha | 11:39 |
| *** fmount has quit IRC | 11:40 | |
| quiquell|rover | chkumar|ruck: it fails adding the route to route table | 11:41 |
| zbr|ssbarnea | ... i am wondering if there is a way to stop a blocked interactive docker run that was started via ssh. it seems impossible to put a break to it. tried any escapes i ever knew. | 11:41 |
| chkumar|ruck | quiquell|rover: and queens one https://logs.rdoproject.org/24/567224/178/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/e9f6d25/logs/undercloud/var/log/mistral/executor.log.txt.gz?level=ERROR | 11:41 |
| *** fmount has joined #oooq | 11:41 | |
| chkumar|ruck | quiquell|rover: let me check with tengu | 11:41 |
| quiquell|rover | chkumar|ruck: ack | 11:42 |
| chkumar|ruck | Tengu: | 11:42 |
| chkumar|ruck | Tengu: hey | 11:42 |
| quiquell|rover | going to open lp | 11:42 |
| chkumar|ruck | Tengu: please have a look here https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-01_09_40_42 | 11:42 |
| *** fmount has quit IRC | 11:43 | |
| Tengu | hmm ? | 11:43 |
| Tengu | geeezzz.... we should really get some better logging in there -.- | 11:44 |
| chkumar|ruck | Tengu: overcloud deploy is failing while adding a route table | 11:44 |
| *** fmount has joined #oooq | 11:44 | |
| Tengu | Object \"route add 0.0.0.0/0 via 10.0.0.1 dev br-ex\" is unknown, try \"ip help ok | 11:45 |
| Tengu | you might want to ping neutron guys instead? | 11:45 |
| Tengu | I think I saw something about "br-ext not present" in LP. | 11:45 |
| quiquell|rover | Tengu: humm | 11:45 |
| quiquell|rover | we have just a master promotion | 11:45 |
| Tengu | https://bugs.launchpad.net/tripleo/+bug/1782317 | 11:45 |
| openstack | Launchpad bug 1782317 in tripleo "[master] scenario008 multinode job failing at undercloud giving Invalid local_interface specified. br-ex is not available." [Critical,Triaged] | 11:45 |
| quiquell|rover | maybe there were something fishy there | 11:46 |
| Tengu | oh. I just saw the date of the LP -.-. | 11:46 |
| ykarel | quiquell|rover, you mean promotion caused it? | 11:46 |
| Tengu | seriously, why is it still a thing ?! | 11:46 |
| quiquell|rover | ykarel: Don't know | 11:47 |
| chkumar|ruck | Tengu: that bug was filed long time and no body looked at that | 11:47 |
| ykarel | quiquell|rover, ack, i have seen this error from time to time, so should not be related to promotion | 11:47 |
| ykarel | may be doing a logstash query will help | 11:48 |
| quiquell|rover | let's check before LP | 11:48 |
| *** skramaja has quit IRC | 11:48 | |
| quiquell|rover | ykarel: we have two different fs failing same at master | 11:48 |
| quiquell|rover | fs001 and fs053 | 11:48 |
| ykarel | same error, same time, | 11:50 |
| quiquell|rover | Yep could be infra | 11:50 |
| Tengu | https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/undercloud/var/log/extra/network.txt.gz no br-ext btw | 11:50 |
| quiquell|rover | Tengu: so it's adding a route to a bridge that does not exists ? | 11:51 |
| Tengu | I would say so yeah. | 11:51 |
| Tengu | "ip -4 a" should show all interface, even if down. | 11:51 |
| Tengu | so basically, we don't have the bridge at that time for some reason. | 11:52 |
| panda | quiquell|rover: does the nodepool setup happen before or after starting zuul ? | 11:52 |
| quiquell|rover | panda: in parallel | 11:52 |
| ykarel | Tengu, but u are seeeing on undercloud | 11:52 |
| quiquell|rover | panda: the docker-compose.yaml.j2 | 11:52 |
| panda | quiquell|rover: oook | 11:52 |
| Tengu | ykarel: oh. sho. wait. | 11:53 |
| Tengu | didn't see it was overcloud_deploy. | 11:53 |
| Tengu | https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/overcloud-controller-1/var/log/extra/network.txt.gz (failed controller-1) vs https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/overcloud-controller-0/var/log/extra/network.txt.gz (success controller-0) | 11:54 |
| quiquell|rover | Tengu: >overcloud-controller-1 | 11:54 |
| Tengu | same constatation: overcloud-controller-1 has no br-ex nor br-tenant | 11:54 |
| Tengu | race condition? | 11:54 |
| ykarel | possibly ovs related | 11:55 |
| quiquell|rover | Tengu: they are different nodes, how can be a race condition ? | 11:55 |
| Tengu | quiquell|rover: maybe controller-1 is slower for one thing in the background? I don't really know how network is set up in there. | 11:56 |
| Tengu | you probably want hardprov or network dfg instead of a poor DF ;). | 11:56 |
| quiquell|rover | https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/overcloud-controller-1/var/log/journal.txt.gz | 11:56 |
| quiquell|rover | br-ex looks good there | 11:56 |
| Tengu | question is, why is that br-ex restarted? | 11:57 |
| quiquell|rover | Tengu: looks this is in the process of setting things up and is teard it down | 11:58 |
| quiquell|rover | Tengu: since it cannot add the route | 11:58 |
| quiquell|rover | Tengu: other controllers have it https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/overcloud-controller-0/var/log/extra/network.txt.gz | 11:59 |
| Tengu | maybe? I really don't know :/ | 11:59 |
| quiquell|rover | Don't know | 11:59 |
| Tengu | is it a consistent failure? | 11:59 |
| quiquell|rover | We can check timeings of ovs add the bridge | 12:00 |
| *** fmount has quit IRC | 12:00 | |
| *** fmount has joined #oooq | 12:00 | |
| quiquell|rover | Tengu: yep br-ex created and then deleted | 12:01 |
| quiquell|rover | https://logs.rdoproject.org/45/560445/236/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b5bc7e1/logs/overcloud-controller-1/var/log/openvswitch/ovs-vswitchd.log.txt.gz | 12:01 |
| Tengu | ok | 12:02 |
| quiquell|rover | Tengu: Maye you are right ovs take more time to add the br-ex we don't check and try to add the route | 12:03 |
| quiquell|rover | Tengu: but br-ex is not there so we tear down | 12:03 |
| quiquell|rover | Well or restart | 12:03 |
| Tengu | ^^ | 12:03 |
| Tengu | usually this kind of behavior (works on foo, fails on bar) is due to a race condition | 12:04 |
| panda | quiquell|rover: the jobs that test reproducer ar for host and livirt ? | 12:04 |
| Tengu | meaning: it's probably not a consistent failure, but depending on the race cause, it might hit more than we would accept. | 12:04 |
| quiquell|rover | panda: yep | 12:04 |
| quiquell|rover | Tengu: | 12:04 |
| quiquell|rover | comparing times | 12:04 |
| panda | quiquell|rover: no test for rdocloud | 12:04 |
| panda | quiquell|rover: ok | 12:04 |
| quiquell|rover | panda: not yet, we have to test nodepools sharing same tenant | 12:04 |
| quiquell|rover | panda: Can be a problem | 12:05 |
| quiquell|rover | Tengu: so at ovs we have 2019-02-01T09:37:20.408Z|00034|bridge|INFO|bridge br-ex: added interface br-ex on port 65534 | 12:05 |
| quiquell|rover | 03:37:20 | 12:05 |
| quiquell|rover | That's the creating time of the bridge | 12:06 |
| quiquell|rover | and failure is at 09:37:15 | 12:06 |
| quiquell|rover | Looks liek this thing is not waiting for the bridge to come up | 12:06 |
| Tengu | :) | 12:06 |
| quiquell|rover | 5 seconds difference | 12:06 |
| Tengu | that's a nice race. | 12:06 |
| quiquell|rover | Also the restart is not restarting or looks like :-) | 12:07 |
| quiquell|rover | This is huge http://git.openstack.org/cgit/openstack/os-net-config/tree/os_net_config/impl_ifcfg.py | 12:09 |
| chkumar|ruck | quiquell|rover: may be slaweq on help there related to ovn | 12:10 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| chkumar|ruck | weshay: quiquell|rover: I am logging out early, Will see if possible at night. | 12:39 |
| chkumar|ruck | see ya! | 12:39 |
| quiquell|rover | chkumar|ruck: by | 12:40 |
| *** panda is now known as panda|lunch | 12:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 12:49 |
| weshay | chkumar|ruck, thanks | 12:50 |
| weshay | have a good weekend | 12:51 |
| quiquell|rover | weshay: o/ | 12:51 |
| weshay | a hoy | 12:51 |
| quiquell|rover | weshay: I have to leave earlier today too, we will have to do 1-1 with a little kid around | 12:51 |
| *** gkadam has quit IRC | 12:52 | |
| *** ratailor_ has quit IRC | 12:52 | |
| weshay | quiquell|rover, we can skip it today | 12:52 |
| quiquell|rover | weshay: thanks, next week | 12:52 |
| quiquell|rover | weshay: Looks like bindep is not that powerful | 12:53 |
| weshay | lolz | 12:53 |
| weshay | I know | 12:53 |
| quiquell|rover | Give you the missing pieces | 12:53 |
| weshay | quiquell|rover, going to have a design session w/ panda|lunch in a few | 12:53 |
| weshay | if it were avail as an rpm itself, that would help | 12:54 |
| quiquell|rover | yep we need pip to check RPMs :-) | 12:54 |
| quiquell|rover | Was just an idea :-/ | 12:54 |
| weshay | quiquell|rover, once we have to deal w/ centos7, fedora rhel8, centos 8 | 12:54 |
| quiquell|rover | But maybe is not bad to have something like binde.txt | 12:54 |
| weshay | I think bindep will become more valuable | 12:55 |
| quiquell|rover | ack | 12:55 |
| quiquell|rover | we can have the file and parse ourself | 12:55 |
| weshay | quiquell|rover, from what I read, the file name doesn't matter | 12:55 |
| weshay | so I named it the file based on the python interp we need | 12:55 |
| *** apetrich has quit IRC | 12:56 | |
| quiquell|rover | weshay: can't we have only one bindep.txt with centos7 fedors28 filters ? | 12:56 |
| quiquell|rover | Don't know | 12:56 |
| weshay | hrm.. filters? | 12:57 |
| weshay | lint jobs? | 12:57 |
| weshay | not sure how they would enforce that | 12:57 |
| quiquell|rover | weshay: looks like you can have profiles too | 12:59 |
| quiquell|rover | like python2 and python3 profiles | 12:59 |
| quiquell|rover | so you mark them in the same bindep.txt | 12:59 |
| quiquell|rover | https://docs.openstack.org/infra/bindep/readme.html#writing-requirements-files | 13:00 |
| weshay | oh, if you can create your own profile/distro names we're fine | 13:00 |
| weshay | that will be second pass | 13:00 |
| quiquell|rover | python3-pip [python3] | 13:00 |
| weshay | quiquell|rover, I want ansible to read in the bindep file | 13:00 |
| quiquell|rover | python2-pip [python2] | 13:00 |
| quiquell|rover | and the like | 13:00 |
| quiquell|rover | then | 13:00 |
| quiquell|rover | bindep ... pytho2 | 13:00 |
| weshay | so that between the user script and the ansible pre.yaml we're using the same config for rpms | 13:00 |
| quiquell|rover | bindep ... python3 | 13:00 |
| quiquell|rover | so know | 13:01 |
| weshay | aye k | 13:01 |
| weshay | second pass I think we'll do that | 13:01 |
| *** apetrich has joined #oooq | 13:01 | |
| quiquell|rover | But don't know if platform filtering is powerful enough to express the python version we want | 13:01 |
| quiquell|rover | weshay: openstack have this https://github.com/openstack/ansible-role-bindep | 13:02 |
| weshay | ya.. was looking for the src, didn't find that | 13:03 |
| quiquell|rover | Haha 3 commits | 13:03 |
| weshay | ya.. let's use that https://github.com/openstack/ansible-role-bindep/blob/master/tasks/install.yaml#L20 | 13:03 |
| *** trown|outtypewww is now known as trown | 13:03 | |
| weshay | well maybe | 13:03 |
| weshay | quiquell|rover, naw.. we don't even need it | 13:04 |
| weshay | quiquell|rover, been talking to rlandy | 13:04 |
| weshay | so if we wrote pre.yaml such that it checks for the rpms or for the docker group prior to having to take action w/ become / sudo | 13:05 |
| weshay | that playbook can be idempoent | 13:05 |
| weshay | and then would not be a problme | 13:05 |
| quiquell|rover | sound like a plan | 13:05 |
| quiquell|rover | sshnaidm|off: still don't like i t | 13:05 |
| weshay | quiquell|rover, sshnaidm|off doesn't have to use it | 13:06 |
| quiquell|rover | but is pre.yaml going to consume bindep/requirements or call install-deps.sh ? | 13:06 |
| weshay | only the user script HAS to call it.. and we HAVE to run it in CI | 13:06 |
| weshay | pre.yaml will read bindeps, check if there are any missing.. if so register a variable, then become root and install | 13:06 |
| weshay | sagi is an expert right? so all the rpms and docker group work is done and would be skipped | 13:07 |
| weshay | we can also skip w/ tags | 13:07 |
| weshay | however it's still worth ensuring playbooks are idempotent | 13:07 |
| weshay | we're working with tripleo tooo long where that is not possible | 13:07 |
| weshay | that's how ansible is supposed to work | 13:07 |
| weshay | rerun 100 times doesn't matter | 13:07 |
| weshay | quiquell|rover, imho it would look pretty shitty if a user forgot to use the skip setup or forgot the tag.. and our script fails or messed up their box in some way | 13:08 |
| quiquell|rover | weshay: smell like molecule testing | 13:08 |
| weshay | ya.. we totally should have molecule | 13:08 |
| weshay | zbr|ssbarnea++ | 13:09 |
| hubbot1 | weshay: zbr|ssbarnea's karma is now 1 | 13:09 |
| quiquell|rover | weshay: I was thinking about the OVB stuff | 13:09 |
| weshay | k | 13:09 |
| quiquell|rover | weshay: or libvirt stuff with molecule so we use clean system | 13:09 |
| zbr|ssbarnea | weshay: btw, can you tell me to which extense the jenkins jobs are still needed? https://github.com/rdo-infra/ci-config/tree/master/jenkins/jobs | 13:09 |
| quiquell|rover | weshay: starting docker-compose at docker file is not possible | 13:09 |
| quiquell|rover | weshay: ansible-galaxy install git+https://github.com/openstack/ansible-role-bindep | 13:09 |
| quiquell|rover | weshay: bindep role works just fine | 13:11 |
| zbr|ssbarnea | quiquell|rover: be careful about galaxy, it was deprecated and any bugs is closed automatically. i would avoid adding anything related to it until we get its replacement in good shape. | 13:11 |
| weshay | zbr|ssbarnea, only in ci.centos | 13:11 |
| quiquell|rover | zbr|ssbarnea: just a snippet to test a role | 13:11 |
| quiquell|rover | zbr|ssbarnea: we are not going to integrate it | 13:11 |
| zbr|ssbarnea | weshay: this was my impresison too, this is why I proposed https://review.rdoproject.org/r/#/c/18649/ -- to separate jjb part from linting. | 13:11 |
| weshay | ya.. galaxy good in concept.. shitty in practice | 13:12 |
| zbr|ssbarnea | weshay: i think it will be sorted by the new tool that takes it place. still i prefer to give them time to polish it. | 13:12 |
| *** ccamacho has quit IRC | 13:13 | |
| quiquell|rover | weshay: there is a zuul role to use bindep.txt | 13:17 |
| quiquell|rover | weshay: https://zuul-ci.org/docs/zuul-jobs/roles.html | 13:17 |
| *** rlandy has joined #oooq | 13:17 | |
| *** holser_ has joined #oooq | 13:18 | |
| quiquell|rover | rlandy: hello there | 13:18 |
| weshay | morning rlandy :) | 13:19 |
| *** ccamacho has joined #oooq | 13:19 | |
| weshay | rlandy, I'm going to run https://tree.taiga.io/project/tripleo-ci-board/task/687 by panda|lunch this morning | 13:19 |
| weshay | hopefully we'll have a solid design after that | 13:19 |
| rlandy | quiquell|rover: weshay: hello | 13:19 |
| rlandy | quiquell|rover: this is a good find https://github.com/openstack/ansible-role-bindep/blob/master/bindep.txt | 13:21 |
| quiquell|rover | rlandy, weshay: btw have you see this ? https://review.openstack.org/#/q/topic:freeze_job | 13:21 |
| quiquell|rover | They are creating a zuul-runner | 13:21 |
| weshay | ya | 13:21 |
| weshay | I saw that | 13:21 |
| weshay | we requested that | 13:21 |
| weshay | zbr|ssbarnea, help me understand what's failing here http://logs.openstack.org/01/634301/2/check/openstack-tox-linters/c8ebd6f/job-output.txt.gz#_2019-01-31_23_23_38_242057 | 13:22 |
| quiquell|rover | akc | 13:22 |
| quiquell|rover | ack | 13:22 |
| rlandy | quiquell|rover: that could be a log way off | 13:22 |
| rlandy | and a lot of our stuff we will still need | 13:22 |
| weshay | totally | 13:23 |
| rlandy | so whatever - we just go with your role until we get something worth looking at | 13:23 |
| quiquell|rover | yep libvirt and all | 13:23 |
| rlandy | oh libvirt, yes dear libvirt | 13:23 |
| quiquell|rover | they don't manage nodes lifecycle yet also | 13:23 |
| rlandy | cause of all complications in lfe | 13:23 |
| panda|lunch | weshay: http://logs.openstack.org/01/634301/2/check/openstack-tox-linters/c8ebd6f/job-output.txt.gz#_2019-01-31_23_23_10_815057 | 13:23 |
| *** panda|lunch is now known as panda | 13:23 | |
| quiquell|rover | lunch time | 13:23 |
| weshay | oh ya.. | 13:23 |
| weshay | thanks | 13:23 |
| *** quiquell|rover is now known as quiquell|lunch | 13:23 | |
| weshay | panda, ok.. shall we design? | 13:25 |
| weshay | and dance | 13:25 |
| panda | after you | 13:26 |
| rlandy | weshay: ^^ can I listen in? | 13:28 |
| rlandy | would like to know what I am changing now | 13:28 |
| panda | rlandy: but you can't talk :) | 13:28 |
| weshay | rlandy, sure come on in | 13:29 |
| rlandy | panda: I have no intention of expressing my opinion | 13:29 |
| weshay | well.. panda and I will be going back and forth on it.. but come on it | 13:29 |
| weshay | rlandy, ur fine | 13:29 |
| weshay | get in here | 13:29 |
| zbr|ssbarnea | weshay: see http://logs.openstack.org/01/634301/2/check/openstack-tox-linters/c8ebd6f/job-output.txt.gz#_2019-01-31_23_23_10_815543 -- exec without shebang | 13:30 |
| zbr|ssbarnea | weshay: harder to spot because we do not have colors enabled in console, yet. | 13:31 |
| *** jpena|lunch is now known as jpena | 13:37 | |
| *** holser_ has quit IRC | 13:41 | |
| rlandy | marios: quiquell|lunch: I wanted to ask about the user/keys requirement | 13:50 |
| rlandy | marios: quiquell|lunch: we need the option to set how many users and keys? | 13:51 |
| *** quiquell|lunch is now known as quiquell | 13:51 | |
| quiquell | rlandy: I am back | 13:51 |
| quiquell | rlandy: we have three entry points for keys | 13:52 |
| quiquell | rlandy: user, upstream-gerrit, rdo-gerrit | 13:52 |
| rlandy | quiquell: k - stop one sec | 13:52 |
| rlandy | user | 13:52 |
| rlandy | is that always the current ansible user? | 13:52 |
| marios | o/ rlandy we only really need to override one for the id_rsa to use (the default in the role is to use that for upstream and rdo gerit too right quiquell | 13:52 |
| quiquell | yep | 13:52 |
| rlandy | is there a need to set that user? or just the related keys? | 13:53 |
| quiquell | no the user yes the key | 13:53 |
| quiquell | arxcruz: for example asked for it | 13:53 |
| quiquell | he didn't want to rewrite the id_rsa because is was not passwordless | 13:53 |
| rlandy | fine - so the scripts needs to allow the user to set the following: | 13:53 |
| rlandy | user_pri_key: "id_rsa" | 13:53 |
| rlandy | user_pub_key: "{{ user_pri_key }}.pub" | 13:53 |
| rlandy | ssh_path: "{{ ansible_user_dir }}/.ssh" | 13:53 |
| rlandy | upstream_gerrit_user: "{{ ansible_user }}" | 13:53 |
| rlandy | upstream_gerrit_key: "{{ user_pri_key }}" | 13:53 |
| rlandy | rdo_gerrit_user: "{{ ansible_user }}" | 13:54 |
| rlandy | rdo_gerrit_key: "{{ user_pri_key }}" | 13:54 |
| marios | rlandy: isn't that what you already have in your review? | 13:54 |
| arxcruz | quiquell: well, i had to generate a passwordless new one and add it on my gerrit account anyway | 13:54 |
| rlandy | marios: it's in the launcher playbook | 13:54 |
| rlandy | I need to add those options to the bash script | 13:55 |
| marios | rlandy: yeah i mean you added the keys in the last version i checked earlier anyway | 13:55 |
| quiquell | rlandy: Yep | 13:55 |
| rlandy | it's a lot of options | 13:55 |
| rlandy | for bash | 13:55 |
| quiquell | rlandy: yep I know | 13:55 |
| marios | rlandy: quiquell i am wondering if we should keep it simple. like one key. we are asking them to create a new key for pem stuff. we can maybe go one further ask them to add it to gerrit& rdo and the role just default uses that one | 13:55 |
| rlandy | quiquell: marios: so I thinking maybe we can be smart about it | 13:55 |
| marios | quiquell: ? wdyt? but we should merge and iterate anyway | 13:56 |
| quiquell | rlandy: yep let's go with one key | 13:56 |
| quiquell | rlandy: not sue about user for upstream/rdo gerrit | 13:56 |
| rlandy | quiquell: is that workable? | 13:56 |
| rlandy | it's fine | 13:56 |
| marios | rlandy: quiquell lets merge with all the things required right now, thats the stuff you have above rlandy | 13:56 |
| quiquell | rlandy: role has the flexibility in case we need them, but we can reduce that in the script | 13:56 |
| rlandy | we will offer all options | 13:56 |
| marios | rlandy: quiquell lets make it pretty later | 13:56 |
| marios | rlandy: quiquell to use one key you have to rework the role abit right | 13:56 |
| rlandy | marios: quiquell: if it's starts to looks overwhelming, we will reduce | 13:56 |
| quiquell | marios: no | 13:57 |
| quiquell | marios: role defaults to user_key | 13:57 |
| marios | quiquell: and we need to get user to add their new key to gerrit & rdo | 13:57 |
| marios | quiquell: ah ack good on the default | 13:57 |
| rlandy | marios: quiquell: what I was thinking is that we would not offer the option to change the pub key | 13:57 |
| quiquell | marios: it's already though | 13:57 |
| rlandy | has to be pri key.pub | 13:57 |
| marios | quiquell: so fine then one key | 13:57 |
| quiquell | rlandy: yep we can even generate pub keys no problem with that | 13:57 |
| marios | quiquell: we don't need to expose it in reproducer | 13:57 |
| marios | quiquell: i mean | 13:57 |
| rlandy | the rest we offer | 13:57 |
| marios | quiquell: keep the options in the role | 13:58 |
| quiquell | rlandy: I remember someone asked for the pub key | 13:58 |
| rlandy | quiquell: lol | 13:58 |
| quiquell | rlandy: didn't want to add it | 13:58 |
| rlandy | ok fine | 13:58 |
| marios | quiquell: rlandy but we don't need to expose them in the reproducer script | 13:58 |
| quiquell | rlandy: so user_key rdo_user upstream_user | 13:58 |
| quiquell | maybe that's enough | 13:58 |
| rlandy | let's see how bad all options look | 13:58 |
| rlandy | maybe it will be fine | 13:58 |
| quiquell | rlandy: role has default for everything so you don't have to put defaults there | 13:59 |
| quiquell | rlandy: just don't pass them if they are not set | 13:59 |
| rlandy | quiquell: yep - kind of just reminder note for myself to ask you today | 13:59 |
| rlandy | will will have to pass them all if we are offering the option now | 14:00 |
| weshay | quiquell, rlandy marios ya.. we need users to take care of their keys... | 14:00 |
| quiquell | so all the options | 14:00 |
| quiquell | we can reduce that later on | 14:00 |
| rlandy | yes | 14:00 |
| rlandy | I am going with that | 14:00 |
| weshay | running some sort of test up front like checking the key for pem and ssh'ing to gerrit systems and bailing if it fails would be nice | 14:00 |
| rlandy | let's see how bad it looks | 14:00 |
| quiquell | weshay: that kind of thin at role | 14:01 |
| weshay | sorry? | 14:01 |
| quiquell | checking keys | 14:01 |
| quiquell | so they go over CI | 14:01 |
| quiquell | Are you talking about checking keys ? | 14:01 |
| rlandy | I am not - just adding what the user wants - that is all | 14:01 |
| quiquell | Then we can in molecule do this kind of fast tests | 14:02 |
| rlandy | weshay: you have a bunch of review out there - would like to add the install stuff to the bash script - are they ready to use? | 14:02 |
| weshay | rlandy, we should blue/chat about them in a bit | 14:03 |
| rlandy | weshay: sure | 14:03 |
| panda | rlandy: if I understand correcly you don't have to, those step will be done by pre.yaml in the reproducer ? | 14:05 |
| quiquell | rlandy, weshay, panda: I am tented to add a review at the role using zuul role "bindep" to install bindep.txt that we have there | 14:05 |
| rlandy | panda: iiuc, we will be doing that install in bash - sagi opposed sudo in ansible | 14:06 |
| panda | aaaand I'm back at square one. | 14:06 |
| panda | quiquell: who's rover ? :) | 14:06 |
| weshay | rlandy, ugh | 14:07 |
| quiquell | dam nick | 14:07 |
| *** quiquell is now known as quiquell|rover | 14:07 | |
| rlandy | lol | 14:07 |
| quiquell|rover | panda: thanks | 14:07 |
| rlandy | no!!! | 14:07 |
| rlandy | we need quiquell|rover back | 14:07 |
| panda | then something else needs to rover | 14:07 |
| panda | someone | 14:07 |
| *** quiquell|rover is now known as quique|roverish | 14:07 | |
| rlandy | something? | 14:07 |
| quique|roverish | I am rovering man | 14:08 |
| *** quique|roverish is now known as quiquell|rover | 14:08 | |
| panda | quiquell|rover: I know, I hope you don't burn out looking at both things | 14:08 |
| rlandy | panda: k - ignore my comment until I chat with weshay about his reviews | 14:08 |
| rlandy | and how and where to incorporate | 14:08 |
| quiquell|rover | panda: is slow today, I am good | 14:09 |
| * rlandy just adds key options in the mean time | 14:09 | |
| panda | said him while an avalanche of bug was approaching for the north | 14:09 |
| quiquell|rover | Or incompetent at rovering :-) | 14:09 |
| jfrancoa | quiquell|rover: hey, this is what I was referring this morning when I said that I couldn't test the upgrades job with a depends-on a tht patch http://logs.openstack.org/25/607525/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/e2957a6/job-output.txt.gz#_2019-02-01_10_57_20_565541 | 14:11 |
| quiquell|rover | jfrancoa: let's take a look ad build logs from dlrn | 14:12 |
| quiquell|rover | jfrancoa: http://logs.openstack.org/25/607525/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/e2957a6/logs/delorean_logs/c6/eb/c6eb68496acd3351d46c54a6db4b0ff4f943e8af_dev/rpmbuild.log.txt.gz | 14:13 |
| panda | rlandy: maybe we can user pre.yaml anyway, and we can make the user pass --extra-vars "ansible_sudo_pass=yourPassword" | 14:13 |
| panda | ansible_become_pass | 14:14 |
| quiquell|rover | jfrancoa: Is like RPM spec file from tht does not expect some files | 14:14 |
| jfrancoa | quiquell|rover: mi no comprende...what does the error mean? | 14:14 |
| rlandy | panda: would like to see what weshay has to say first ... | 14:15 |
| rlandy | so we don;t get yet another option nobody likes | 14:15 |
| rlandy | the fun of everyone's conflicting opinions | 14:16 |
| quiquell|rover | jfrancoa: so the distgit from tht that contains the .spec is not compatible wiht the Depends-On maybe ? | 14:18 |
| quiquell|rover | But is just changin stuff no new files or the like | 14:18 |
| jfrancoa | quiquell|rover: exactly..the patch doesn't include anything new...it's just modifying some files | 14:18 |
| jfrancoa | quiquell|rover: and the same thing had happened with a different tht patch couple of days ago | 14:19 |
| quiquell|rover | triplo-common RPM is build ok | 14:19 |
| panda | rlandy: do you have clear what needs to be installed (even if not who/when ?) | 14:19 |
| quiquell|rover | jfrancoa: maybe releases ? | 14:19 |
| quiquell|rover | Let me look | 14:19 |
| jfrancoa | quiquell|rover: mmm..maybe..although we're upgrading from master to master | 14:20 |
| rlandy | panda: I think so - other than the fact that pre installs docker twice - once with rpm and once with pip | 14:20 |
| rlandy | quiquell|rover: ^^ ... fyi | 14:20 |
| quiquell|rover | jfrancoa: but the .spec file is the same | 14:21 |
| quiquell|rover | :-/ weird | 14:21 |
| quiquell|rover | rlandy: docker pip is not docker package | 14:21 |
| quiquell|rover | rlandy: we need both | 14:21 |
| quiquell|rover | rlandy: pip docker is just the python wrapper (Don't know if we install that with package) | 14:21 |
| panda | docker (3.7.0) - A Python library for the Docker Engine API. | 14:22 |
| quiquell|rover | jfrancoa: We can reproduce and rerun the build command | 14:23 |
| quiquell|rover | jfrancoa: but we can also merge the review | 14:23 |
| jfrancoa | quiquell|rover: i'm more for the second option in fact..I'll get rid of the depends-on, and let's try to merge this | 14:23 |
| quiquell|rover | jfrancoa: lets merge it, upgrade jobs are already broken | 14:24 |
| quiquell|rover | jfrancoa: let's unblock you guys | 14:24 |
| jfrancoa | quiquell|rover: thanks a lot, marios also, when you have some moment, could you give some review again on https://review.openstack.org/#/c/607525/ ? | 14:25 |
| marios | jfrancoa: no | 14:28 |
| jfrancoa | marios: ohh okey okey...first you abandon us, and now you renege from us..very nice | 14:30 |
| jfrancoa | :-D | 14:30 |
| marios | ;) | 14:31 |
| marios | scumbag marios | 14:31 |
| weshay | rlandy, ok.. ready | 14:33 |
| weshay | to speak to you and whomever else | 14:33 |
| rlandy | weshay: joining your bj | 14:34 |
| quiquell|rover | marios: Do we kick this guy ? | 14:34 |
| marios | quiquell|rover: nah hes good. he brings the cachopo | 14:36 |
| *** apetrich has quit IRC | 14:36 | |
| quiquell|rover | marios: XD | 14:37 |
| *** apetrich has joined #oooq | 14:38 | |
| quiquell|rover | marios: standalone scenarios job have increase in time :-( | 14:45 |
| quiquell|rover | scenario002 2H | 14:45 |
| *** quiquell|rover is now known as quiquell|off | 14:47 | |
| quiquell|off | Ok drop now, have a good weekend @oooq | 14:47 |
| *** quiquell|off is now known as quique|rover|off | 14:48 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 14:49 |
| marios | quique|rover|off: u 2 | 14:53 |
| *** ykarel has quit IRC | 14:53 | |
| *** agopi has quit IRC | 15:04 | |
| *** apetrich has quit IRC | 15:05 | |
| *** apetrich has joined #oooq | 15:06 | |
| zbr|ssbarnea | @oooq: any can recommend a tool to build presentations (from code/md/rst, something that can keep in git)? | 15:19 |
| *** ykarel has joined #oooq | 15:26 | |
| weshay | rlandy, panda I may be right back.. have a rhel 8 call | 15:31 |
| marios | rfolco: panda: weshay had a go at design & tasks for https://tree.taiga.io/project/tripleo-ci-board/us/652 fyi | 15:35 |
| *** agopi has joined #oooq | 15:36 | |
| weshay | marios, /me reads | 15:36 |
| marios | weshay: adding some more stuff but the tasks mainly | 15:38 |
| *** panda is now known as panda|braindead | 15:42 | |
| weshay | rfolco, I may kick out our 1-1 a 1/2 hour | 15:43 |
| weshay | depends on rlandy and panda|braindead | 15:43 |
| rfolco | weshay, ok | 15:44 |
| rlandy | weshay; we're done | 15:44 |
| rlandy | on your bj - coding up review changes | 15:44 |
| weshay | k.. will come back in a few | 15:44 |
| rlandy | weshay: sorry - that was a badly broken response - we are not on your bj - I am coding the changes now | 15:46 |
| weshay | ah k | 15:46 |
| marios | brb | 15:55 |
| *** marios has quit IRC | 15:55 | |
| *** marios has joined #oooq | 15:56 | |
| panda|braindead | rlandy: do you know what should create the DOCKER iptables chain ? | 15:57 |
| *** marios has quit IRC | 15:57 | |
| *** marios has joined #oooq | 15:57 | |
| rlandy | docker install? | 15:58 |
| weshay | rfolco, ready now | 15:59 |
| rfolco | weshay, ok joining | 15:59 |
| *** saneax has quit IRC | 16:00 | |
| zbr|ssbarnea | what was the stuff with RETRY_LIMIT? I clicked the link to logs and I didn't see any error. https://logs.rdoproject.org/27/18627/6/check/tox-molecule/3f27c3f/job-output.txt.gz | 16:00 |
| zbr|ssbarnea | weshay: rlandy panda|braindead : please help me merge the fix for ovb-tenant-cleanup: https://review.rdoproject.org/r/#/c/18517/ | 16:03 |
| rlandy | zbr|ssbarnea: looking | 16:05 |
| *** ykarel is now known as ykarel|away | 16:13 | |
| weshay | rfolco, http://file.rdu.redhat.com/~whayutin/ENGINEERING_REPORTS/ | 16:19 |
| zbr|ssbarnea | weshay: report is quite cool but i find it bit too verbose, i could probably combine tripleo-* repos into a single table. | 16:30 |
| *** kopecmartin is now known as kopecmartin|off | 16:37 | |
| *** vinaykns has joined #oooq | 16:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7 (2 more messages) | 16:49 |
| *** apetrich has quit IRC | 16:51 | |
| *** apetrich has joined #oooq | 16:53 | |
| *** bogdando has quit IRC | 16:54 | |
| zbr|ssbarnea | who is the zuul guru around? why I get this error: https://review.openstack.org/#/c/634452/ ? | 17:00 |
| weshay | zbr|ssbarnea, I think you cant hvae -job: and -project: check: jobs gate: jobs in the same yaml | 17:04 |
| weshay | that's how I read that | 17:04 |
| zbr|ssbarnea | weshay: and why https://review.rdoproject.org/r/#/c/18627/6/zuul.d/layout.yaml worked? | 17:05 |
| *** jfrancoa has quit IRC | 17:05 | |
| zbr|ssbarnea | i started to suspect whitespace issues.... | 17:05 |
| weshay | zbr|ssbarnea, you don't have project: template | 17:05 |
| weshay | the job is only defined in one place | 17:05 |
| panda|braindead | zbr|ssbarnea: you can't use zuul.d for anything that is not zuul configuration | 17:10 |
| panda|braindead | zbr|ssbarnea: you must put your playbook somewhere else | 17:10 |
| panda|braindead | zbr|ssbarnea: not under zuul.d | 17:10 |
| zbr|ssbarnea | panda|braindead: but on rdo it worked, i guees different version of zuul. | 17:11 |
| panda|braindead | zbr|ssbarnea: zuull tried to parse your playbook and is seeing that you have a list with 1 element which is a dictionary with two keys: hosts and tasks | 17:11 |
| zbr|ssbarnea | panda|braindead: i read it differntly but it makes sense now, moving it under playbooks. | 17:12 |
| panda|braindead | zbr|ssbarnea: I don't think it depends on the version of zuul, zuul is very picky on the zuul.d directory. Last time I tried for example you could not put an empty file there, or zuul would fail to parse the yaml | 17:12 |
| zbr|ssbarnea | a zuul linter could prove handy | 17:14 |
| panda|braindead | zbr|ssbarnea: zuul linting exist, but shows its output only to zuul administrators. | 17:15 |
| zbr|ssbarnea | panda|braindead: weshay : thanks sorted. | 17:18 |
| zbr|ssbarnea | panda|braindead: do you also happen to know what happens behind the RETRY_LIMIT error? | 17:21 |
| zbr|ssbarnea | example https://review.rdoproject.org/r/#/c/18627/ -- clicking on it didn't help me understand the issue. | 17:21 |
| weshay | rlandy, | 17:23 |
| weshay | sudo -n true && passwordless_sudo="1" || passwordless_sudo="0" | 17:23 |
| weshay | if [[ "$passwordless_sudo" == "1" ]]; then | 17:23 |
| zbr|ssbarnea | panda|braindead: i wonder if is not the same issue you told me about but with different side-effect. maybe i should move playbooks/ outside zuul.d | 17:23 |
| rlandy | looking in sec | 17:24 |
| panda|braindead | only zuul admin know exactly what happens, but it's usually an error in the ansible playbooks, which are silently failed. | 17:26 |
| rlandy | weshay: https://review.rdoproject.org/r/#/c/18664/ - testing - but here's the idea | 17:32 |
| rlandy | oh - adding the requirements - sec | 17:32 |
| rlandy | weshay: ok - https://review.rdoproject.org/r/#/c/18664/ | 17:35 |
| rlandy | just checking file path | 17:35 |
| rlandy | not sure if the server start/enable will skip if it is - will see | 17:36 |
| weshay | fyi https://review.openstack.org/#/c/634353/6/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 17:37 |
| *** jpena is now known as jpena|off | 17:37 | |
| rlandy | weshay: ^^ adding those changes to my review | 17:37 |
| weshay | rlandy, careful.. just fyi | 17:38 |
| rlandy | now that I have the pattern for pre | 17:38 |
| weshay | for now | 17:38 |
| rlandy | weshay: whatever - it's juts a review | 17:38 |
| weshay | I have some commented out lines.. :) | 17:38 |
| rlandy | weshay: I already have those changes | 17:38 |
| weshay | I'll test for a bit.. then ping you again | 17:38 |
| weshay | k k | 17:38 |
| weshay | he he | 17:38 |
| panda|braindead | rlandy: what does the option --gerrit-user do ? | 17:39 |
| rlandy | panda|braindead: I am changing that | 17:39 |
| rlandy | we need three diff users | 17:39 |
| rlandy | and keys | 17:39 |
| rlandy | see the launcher playbook | 17:39 |
| rlandy | you can have a diff upstream gerrit user | 17:39 |
| rlandy | rdo gerrit user | 17:39 |
| rlandy | and current user | 17:39 |
| rlandy | see note that all three need to be added in patch 41 | 17:40 |
| panda|braindead | that's why none of my attempts worked so far | 17:40 |
| panda|braindead | I'm gcerami locally, gabrielecerami in rdo and panda in openstack | 17:40 |
| rlandy | panda|braindead: maybe wait until monday | 17:40 |
| rlandy | these are wip patches now | 17:41 |
| rlandy | we are actively editing stuff | 17:41 |
| *** trown is now known as trown|lunch | 17:41 | |
| rlandy | panda|braindead: I am adding all those options | 17:41 |
| panda|braindead | rlandy: ok | 17:41 |
| *** panda|braindead is now known as panda | 17:46 | |
| *** derekh has quit IRC | 18:00 | |
| *** dtrainor has quit IRC | 18:05 | |
| rlandy | weshay: panda: http://pastebin.test.redhat.com/706085 - does this capture it? | 18:17 |
| weshay | rlandy, 2. Edit the launcher.yaml playbook to pass the sudo password | 18:21 |
| weshay | the rest looks good | 18:22 |
| weshay | 2 not yet | 18:22 |
| weshay | ansible-playbook playbook.yml -i inventory.ini --user=username \ | 18:23 |
| weshay | --extra-vars "ansible_sudo_pass=yourPassword" | 18:23 |
| weshay | Update 2017: Ansible 2.2.1.0 now uses var ansible_become_pass. Either seems to work. | 18:23 |
| rlandy | --extra-vars "ansible_sudo_pass=yourPassword | 18:23 |
| rlandy | ^^ thats clear enough | 18:23 |
| rlandy | I'll add that one | 18:23 |
| weshay | rlandy, when we have it merged | 18:24 |
| weshay | rlandy, would you be cool adding tq/install-deps.sh to the tar file? | 18:25 |
| rlandy | weshay: ack - we can do that | 18:26 |
| * rlandy thinks where to pull it from | 18:26 | |
| rlandy | the tar file gets created in the job | 18:26 |
| rlandy | tq is cloned in src | 18:27 |
| rlandy | so copy from src to logs | 18:27 |
| weshay | :) | 18:27 |
| * rlandy thinks about that | 18:27 | |
| rlandy | sorry - thinking out loud on irc | 18:27 |
| rlandy | well | 18:27 |
| rlandy | I am not sure tq is a required project on all cimaybe just libvirt | 18:28 |
| rlandy | - would have to check that | 18:28 |
| weshay | rlandy, I mean.. anywhere we have the reproducer we have tq ya? | 18:31 |
| ska | How does OOO allow access to services (5000 and 80) on the external network? I've configured OOO on KVM manually and it doesn't have any externally facing services although it did provision the external nics in overcloud. | 18:34 |
| rlandy | yeah | 18:35 |
| *** agopi is now known as agopi|transit | 18:37 | |
| zbr|ssbarnea | weshay: hurrah! i managed to make the upstream install-docker succeed on centos, now time to get the change accepted. | 18:41 |
| *** agopi|transit has quit IRC | 18:42 | |
| zbr|ssbarnea | https://review.openstack.org/#/c/633948/ -- install-docker patch | 18:43 |
| weshay | I need more context of what you are trying to do and why to fully appreciate what you are up to atm | 18:44 |
| *** panda is now known as panda|off | 18:47 | |
| weshay | zbr|ssbarnea, ur just fixing that role? | 18:47 |
| weshay | which is good | 18:47 |
| zbr|ssbarnea | weshay: all started when I discovered that on rdo i was not able to run molecule because there was no docker. upstream i used ubuntu-xenial image + install-docker role in pre.yaml in order to be able to use it. | 18:47 |
| weshay | oh k k | 18:48 |
| weshay | that makes sense | 18:48 |
| weshay | it's good to dogfood, so thanks for fixing centos | 18:48 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 18:49 |
| zbr|ssbarnea | this is because i was told too often to provide proof that the change works, not I will be able to provide proof withing the job itself. like refactoring around te-broker role. | 18:49 |
| weshay | OH zbr|ssbarnea hey | 18:49 |
| weshay | question | 18:49 |
| weshay | re: f28 | 18:49 |
| weshay | have a sec? | 18:49 |
| weshay | or too late? | 18:50 |
| zbr|ssbarnea | bj, yours? | 18:50 |
| weshay | sure | 18:50 |
| *** ykarel|away has quit IRC | 18:52 | |
| weshay | https://review.openstack.org/#/q/topic:generate-zuul-based-reproducer+(status:open+OR+status:merged) | 18:55 |
| *** trown|lunch is now known as trown | 18:58 | |
| weshay | rlandy, let's plan on talking to zbr|ssbarnea re: testing your script via molecule | 19:05 |
| weshay | for f28 and centos 7 python2/3 | 19:05 |
| weshay | rlandy, ok.. if you can get install-deps.sh into the tar I think we're pretty good atm | 19:06 |
| rlandy | weshay: ack - already asked zbr|ssbarnea to help | 19:06 |
| weshay | rlandy, aye cool.. look for a monday sync | 19:06 |
| rlandy | weshay: will add to tar - just adding rest of key options | 19:06 |
| rlandy | and will submit change | 19:06 |
| weshay | :) | 19:07 |
| rlandy | we may be getting there | 19:07 |
| *** agopi|transit has joined #oooq | 19:10 | |
| *** agopi|transit is now known as agopi | 19:12 | |
| *** apetrich has quit IRC | 19:17 | |
| *** apetrich has joined #oooq | 19:19 | |
| rlandy | ugh - there are now a lot of ssh key and user options | 19:24 |
| *** apetrich has quit IRC | 19:34 | |
| *** holser_ has joined #oooq | 19:53 | |
| ska | My public facing IP on Overclouds were setup as br-ex interfaces with dhcp. Can I convert those to fixed ip and create a public enpoint on them? | 20:03 |
| ska | endpoints that is. | 20:03 |
| rlandy | weshay: https://review.openstack.org/631067 and https://review.rdoproject.org/r/#/c/18664/ | 20:09 |
| rlandy | ^^ both under test - but that should include the ideas put forward | 20:09 |
| rlandy | all of them :) | 20:09 |
| vinaykns | weshay: I couldn't start tripleo_nova_compute service in the undercloud...it says No compute node record for host undercloud.localdomain: ComputeHostNotFound_Remote: Compute host undercloud.localdomain could not be found. | 20:14 |
| weshay | rlandy, /me looks | 20:15 |
| vinaykns | and I've tried manually to run the command using podman and it says failed to connect to container's attach socket | 20:15 |
| weshay | vinaykns, ok.. pass me the command you used.. and I'll run on my box | 20:16 |
| vinaykns | bash quickstart.sh -R master-tripleo-ci -v -n -I --tags all --teardown none --nodes config/nodes/1ctlr_1comp.yml -e enable_telemetry=true -e undercloud_undercloud_ntp_servers=clock.redhat.com --playbook quickstart-extras-overcloud.yml localhost | 20:17 |
| weshay | rlandy, we should have a merge party | 20:17 |
| vinaykns | well I'm doing step by step installation...the first two steps were succesful | 20:17 |
| rlandy | weshay: oh gosh - don;t merhe that - I am testing | 20:18 |
| rlandy | merge | 20:18 |
| * rlandy makes mistakes | 20:18 | |
| weshay | vinaykns, can you try w/o localhost and instead use 127.0.0.2 | 20:18 |
| *** holser_ has quit IRC | 20:18 | |
| weshay | vinaykns, we actually call out localhost as unsupported | 20:18 |
| weshay | because ansible treats 127.0.0.2 and localhost differently | 20:18 |
| vinaykns | Oops actually I run with ipaddress of the host | 20:18 |
| weshay | k k | 20:18 |
| weshay | vinaykns, changing playbook | 20:20 |
| vinaykns | sure.! | 20:20 |
| weshay | I'm outside the vpn | 20:21 |
| weshay | wfh | 20:21 |
| vinaykns | you changes that ntp thing. | 20:21 |
| vinaykns | changed | 20:21 |
| weshay | had to | 20:21 |
| weshay | [wes@localhost tripleo-quickstart]$ bash quickstart.sh -R master-tripleo-ci -v -n -I --tags all --teardown none --nodes config/nodes/1ctlr_1comp.yml -e enable_telemetry=true 127.0.0.2 | tee wes.log | 20:22 |
| weshay | rlandy, ok.. got +1's here from CI | 20:23 |
| * weshay updates commit messages as prepares for votes and merge | 20:23 | |
| weshay | SSH Error: data could not be sent to remote host "undercloud". Make sure this host can be reached over ssh | 20:24 |
| weshay | ha | 20:24 |
| weshay | I have to make sure ssh works | 20:24 |
| weshay | bah | 20:25 |
| weshay | ok.. that did NOT work | 20:25 |
| weshay | vinaykns, remove -I, --retain-inventory | 20:32 |
| weshay | vinaykns, why are you using no-clone? | 20:33 |
| weshay | you changing things? | 20:33 |
| vinaykns | before running the command i had the latest clone | 20:33 |
| vinaykns | so that's why i used no-clone flag | 20:33 |
| weshay | k.. let's keep this simple | 20:33 |
| weshay | bash quickstart.sh -R master-tripleo-ci -v --tags all --clean --teardown none --nodes config/nodes/1ctlr_1comp.yml -e enable_telemetry=true whayutin-testbox | 20:34 |
| vinaykns | no teardown..? | 20:34 |
| weshay | sorry | 20:34 |
| weshay | need --teardown all | 20:34 |
| weshay | bash quickstart.sh -R master-tripleo-ci -v --tags all --clean --teardown all --nodes config/nodes/1ctlr_1comp.yml -e enable_telemetry=true whayutin-testbox | 20:35 |
| weshay | jjez | 20:36 |
| weshay | jeez | 20:36 |
| weshay | every little thing | 20:36 |
| weshay | module failure, retry | 20:36 |
| weshay | fun right | 20:37 |
| vinaykns | yeah..lot of | 20:37 |
| weshay | geez | 20:43 |
| rlandy | how nice | 20:43 |
| rlandy | I didn;t mess up lint this time | 20:43 |
| rlandy | weshay: I incorporated your changes from https://review.openstack.org/#/c/634353/ | 20:44 |
| weshay | ah k | 20:44 |
| weshay | rlandy, /me merges https://review.rdoproject.org/r/#/c/18665/1 | 20:45 |
| weshay | won't hurt anything | 20:45 |
| weshay | rlandy, k.. abandoned https://review.openstack.org/#/c/634353/ | 20:46 |
| weshay | vinaykns, there it goes | 20:48 |
| rlandy | ok | 20:48 |
| weshay | libvirt.. is a little like windows | 20:48 |
| vinaykns | yeah..till here I'm able to get through | 20:49 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 20:49 |
| vinaykns | I'm breaking at overcloud containers prep | 20:49 |
| rlandy | weshay: wrt bindep comment - I want to see how ci deal with this | 20:49 |
| rlandy | may change it | 20:49 |
| rlandy | we can merge on monday | 20:50 |
| rlandy | it would safe the extra task | 20:50 |
| weshay | rlandy, ? | 20:59 |
| weshay | not sure what you mean | 20:59 |
| rlandy | I added the extra task - so I could use with_items over two actions | 20:59 |
| rlandy | anyways tox error - fixing | 21:00 |
| *** fultonj has quit IRC | 21:01 | |
| weshay | vinaykns, meh.. ipv6 locally broke that attempt.. retrying | 21:03 |
| weshay | w/ undercloud_undercloud_nameserver=1.1.1.1 | 21:03 |
| weshay | rlandy, k | 21:03 |
| vinaykns | okay.! | 21:03 |
| weshay | rlandy, any thoughts on https://review.openstack.org/#/c/565839/ | 21:04 |
| weshay | I think this has been causing issues for me recently on libvirt | 21:04 |
| rlandy | weshay: truth is that unbound reworks that anyways | 21:05 |
| rlandy | weshay: we can merge it if it might help | 21:05 |
| rlandy | unbound runs in zuul roles | 21:05 |
| weshay | k.. on my list | 21:05 |
| rlandy | 2019-02-01 21:06:07.360083 | rdo-centos-7 | [WARNING]: Unable to find 'bindep_python2.txt' in expected paths (use -vvvvv | 21:07 |
| rlandy | 2019-02-01 21:06:07.377761 | rdo-centos-7 | ERROR: InvocationError for command '/home/zuul/src/review.rdoproject.org/rdo-infra/ansible-role-tripleo-ci-reproducer/.tox/linters/bin/python -m pre_commit run -a' (exited with code 1) | 21:07 |
| weshay | rlandy, which path? | 21:07 |
| rlandy | https://logs.rdoproject.org/64/18664/7/check/tox-linters/1816b3b/job-output.txt.gz#_2019-02-01_21_06_07_377761 | 21:08 |
| rlandy | var/ssh/id_rsa: No such file or directory | 21:08 |
| rlandy | ^^ we never added that | 21:08 |
| rlandy | hmmm .... | 21:09 |
| rlandy | not in that path in ci | 21:09 |
| weshay | did it fail on 2019-02-01 21:06:07.355099 | rdo-centos-7 | [0;34mplaybooks/tripleo-ci-reproducer/pre.yaml[0m:[0;36m34[0m: [[1;31mE206[0m] [0;31mVariables should have spaces before and after: {{ var_name }}[0m | 21:10 |
| rlandy | let's try this again | 21:15 |
| weshay | ok.. running locally | 21:18 |
| weshay | [WARNING]: Unable to find 'bindep_python2.txt' in expected paths (use -vvvvv | 21:18 |
| weshay | is what I get too | 21:18 |
| weshay | tbh.. rlandy | 21:18 |
| weshay | - name: Read bindep file contents | 21:18 |
| weshay | set_fact: | 21:18 |
| weshay | bindep_contents: "{{ lookup('file', 'bindep_python2.txt') }}" | 21:18 |
| weshay | - include: install-packages.yaml package="{{item}}" | 21:18 |
| weshay | with_items: "{{ bindep_contents }}" | 21:18 |
| weshay | - name: Read requirements.txt file contents | 21:18 |
| weshay | set_fact: | 21:18 |
| weshay | requirements_contents: "{{ lookup('file', 'requirements.txt') }}" | 21:18 |
| weshay | - name: Install python dependencies | 21:18 |
| weshay | pip: | 21:18 |
| weshay | name: "{{ requirements_contents }}" | 21:18 |
| weshay | does not make sense to me | 21:18 |
| weshay | you have no base dir | 21:18 |
| weshay | rlandy, that worked w/ {{ playbook_dir }} for me | 21:19 |
| weshay | rlandy, oh.. man | 21:20 |
| weshay | rlandy, it's not a depends on | 21:20 |
| weshay | https://review.rdoproject.org/r/#/c/18664/7 | 21:20 |
| weshay | sorry | 21:20 |
| weshay | rlandy, rebase your review on top of it | 21:20 |
| rlandy | will do | 21:22 |
| weshay | files/playbooks/setup-gerrit.yaml:102: ssh_key: "{{ lookup('file', '/var/ssh/id_rsa.pub') }}" | 21:23 |
| weshay | rlandy, this is failing on another patch | 21:24 |
| weshay | you want to blue? | 21:24 |
| weshay | http://pastebin.test.redhat.com/706163 | 21:25 |
| weshay | rlandy, k.. got past some of it w/ https://review.rdoproject.org/r/18694 | 21:26 |
| weshay | that is a throw away review | 21:26 |
| weshay | rlandy, dang it | 21:28 |
| weshay | https://review.rdoproject.org/r/#/c/17981/ | 21:28 |
| rlandy | that's not out error | 21:39 |
| *** trown is now known as trown|outtypewww | 21:42 | |
| rlandy | weshay: playbooks_dir is not right | 21:45 |
| rlandy | it's one back | 21:45 |
| rlandy | will fix on sunday | 21:50 |
| *** rlandy has quit IRC | 21:50 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 22:49 |
| *** jtomasek has quit IRC | 23:48 | |
| *** jtomasek has joined #oooq | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!