| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (1 more message) | 00:37 |
|---|---|---|
| *** pliu_ has joined #oooq | 02:17 | |
| *** sanjayu_ has joined #oooq | 02:34 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo- (1 more message) | 02:37 |
| *** myoung has quit IRC | 02:39 | |
| *** sanjayu_ has quit IRC | 03:18 | |
| *** links has joined #oooq | 03:52 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, (1 more message) | 04:37 |
| *** jaganathan has joined #oooq | 04:45 | |
| *** jaganathan has quit IRC | 04:47 | |
| *** jaganathan has joined #oooq | 04:47 | |
| *** jfrancoa has joined #oooq | 05:12 | |
| *** pgadiya has joined #oooq | 05:16 | |
| *** pgadiya has quit IRC | 05:16 | |
| *** skramaja has joined #oooq | 05:20 | |
| *** quiquell has joined #oooq | 05:41 | |
| quiquell | marios: Good morning | 05:42 |
| *** kopecmartin has joined #oooq | 05:49 | |
| *** bogdando has joined #oooq | 06:15 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, (1 more message) | 06:37 |
| *** amoralej|off is now known as amoralej | 06:50 | |
| quiquell | jfrancoa: Looks like toci_quickstart just fail in the playbook and doesn't continue so collect_logs is not executed | 06:51 |
| *** brault has joined #oooq | 06:51 | |
| quiquell | also exit_value is not printed | 06:51 |
| *** tesseract has joined #oooq | 06:52 | |
| jfrancoa | quiquell: then try to recheck, let's see if next execution brings logs with it | 06:52 |
| quiquell | jfrancoa: We have them at the RDO | 06:52 |
| jfrancoa | quiquell: meanwhile I am running the job with reproducer (it's giving me issues too) | 06:52 |
| quiquell | jfrancoa: Reproducer is still usgin the old toci scripts | 06:53 |
| quiquell | jfrancoa: So it kind of confirm that it's not related, just a defect passing the gates | 06:53 |
| quiquell | jfrancoa: Gates are broken now | 06:53 |
| quiquell | jfrancoa: Also RDO is not using the new framwork, so confirmed again | 06:54 |
| jfrancoa | quiquell: need a coffee..can't think with empty stomach :-D | 06:55 |
| quiquell | jfrancoa: That's why logs appear there | 06:55 |
| quiquell | jfrancoa: Found it... | 07:00 |
| quiquell | jfrancoa: Missing "\" that's why we don't have logs | 07:00 |
| jfrancoa | quiquell: do you have the log somewhere? | 07:18 |
| quiquell | jfrancoa: Running now | 07:18 |
| quiquell | jfrancoa: Found new stuff | 07:18 |
| quiquell | look at update jobs | 07:18 |
| quiquell | update bug | 07:18 |
| quiquell | https://bugs.launchpad.net/tripleo/+bug/1783866 | 07:18 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 07:18 |
| quiquell | - - TRIPLEO_DEPLOY_IDENTIFIER=1532451032 | 07:18 |
| quiquell | + - TRIPLEO_DEPLOY_IDENTIFIER= | 07:18 |
| quiquell | at docker_config.yaml | 07:19 |
| *** ccamacho has joined #oooq | 07:20 | |
| *** dtantsur|afk is now known as dtantsur | 07:21 | |
| *** pliu_ has quit IRC | 07:24 | |
| *** zoli|gone is now known as zoli | 07:30 | |
| *** jtomasek has joined #oooq | 07:49 | |
| *** gkadam has joined #oooq | 07:49 | |
| *** dtantsur is now known as dtantsur|bbl | 08:00 | |
| jfrancoa | quiquell: man, I am comparing both docker_config.yaml FAILED: https://logs.rdoproject.org/71/584771/14/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/c796079/logs/undercloud/var/lib/mistral/5cf4bb48-2415-4575-95ea-7459eb55929e/Controller/docker_config.yaml.txt.gz vs PASSED: https://logs.rdoproject.org/28/585528/7/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-maste | 08:03 |
| jfrancoa | r/b21ba7a/logs/undercloud/var/lib/mistral/10a104c9-55c7-474e-92ff-db31c9660b41/Controller/docker_config.yaml.txt.gz and they are exactly the same | 08:03 |
| quiquell | jfrancoa: Have copy the same :-( | 08:04 |
| quiquell | jfrancoa: Updated | 08:04 |
| quiquell | rlandy: Run it with the reproducer and fixing that pass the updates | 08:04 |
| jfrancoa | quiquell: my bad, the second docker_config is also from a failed job. I will take your log with the depends-on | 08:05 |
| quiquell | jfrancoa: Upgrade job has fail with the Depends-On job can you take a look ? | 08:05 |
| jfrancoa | quiquell: sure, I'll check if after comparing these two files | 08:06 |
| quiquell | jfrancoa: Depends-On running now tripleo-upgrade : run overcloud minor update in each of the roles/hostgroups | 08:06 |
| quiquell | jfrancoa: I have paste Ronelle's new stuff in the bug of updates | 08:08 |
| quiquell | https://bugs.launchpad.net/tripleo/+bug/1783866 | 08:08 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 08:08 |
| quiquell | jfrancoa: She has add the logs flags, and it looks ok | 08:08 |
| quiquell | rlandy: Run it with the reproducer and fixing that pass the updates | 08:15 |
| quiquell | jfrancoa: Patch with the Depends-On have pass the bad step :-/ | 08:15 |
| jfrancoa | quiquell: total black magic | 08:15 |
| quiquell | jfrancoa: There have to be some dependency trigger by it, that have the fix, doesn't have to be hth | 08:16 |
| quiquell | jfrancoa: Let's compare yum packages after it finishes | 08:18 |
| quiquell | jfrancoa: We also have the logs of the failing one at EmilienM Fix | 08:19 |
| quiquell | The patch with the Depends-On updates this three packages | 08:27 |
| quiquell | Updated: 3:mariadb-libs-10.1.20-2.el7.x86_64 | 08:27 |
| quiquell | Updated: openstack-tripleo-heat-templates-9.0.0-0.20180726060839.7425b7d.el7.noarch | 08:27 |
| quiquell | Updated: python-ipaddress-1.0.16-3.el7.noarch | 08:28 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ (1 more message) | 08:38 |
| *** jaosorior has quit IRC | 08:38 | |
| panda|rover | status ? | 08:52 |
| quiquell | panda|rover: Updates fails again, also there were a missing "\" in the latest patchset on fix | 08:52 |
| quiquell | panda|rover: That prevents to get logs of failing update job | 08:52 |
| quiquell | panda|rover: Also tested Depends-On an its working :-/ | 08:53 |
| quiquell | panda|rover: Ronelle have found the issue but we don't know where it comes frome | 08:53 |
| quiquell | panda|rover: All here https://bugs.launchpad.net/tripleo/+bug/1783866 | 08:53 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 08:53 |
| panda|rover | the misformed yaml ? | 08:53 |
| quiquell | panda|rover: Again the passing Depends-On has it alright | 08:54 |
| quiquell | panda|rover: We need people from tht to give us any clue | 08:55 |
| quiquell | panda|rover: Want to sync with bj ? | 08:56 |
| panda|rover | quiquell: ok bj/gcerami | 08:57 |
| *** jaosorior has joined #oooq | 08:58 | |
| quiquell | panda|rover: Let's sync in a few | 09:01 |
| quiquell | panda|rover: Maybe at this moment we have to merge the fix, so we don't get more defect... don't know if we can force it | 09:04 |
| panda|rover | quiquell: don't even know if it's the right thing, the fix is going to make the update fail. Right now it passes as false positive | 09:06 |
| quiquell | panda|rover: That opens the door for more defects... don't know really... | 09:07 |
| quiquell | panda|rover: Also RDO is not using the new workflow and is failing too in the same point | 09:07 |
| quiquell | panda|rover: And reproducer too | 09:07 |
| quiquell | panda|rover: So it's clear is not related to the workflow | 09:07 |
| *** d0ugal has joined #oooq | 09:09 | |
| quiquell | panda|rover: Passing one https://review.openstack.org/#/c/586444/ | 09:12 |
| quiquell | RDO cloud is down :-( | 09:17 |
| panda|rover | tis gun be a great sprint. | 09:21 |
| *** pliu_ has joined #oooq | 09:22 | |
| quiquell | panda|rover: "Retrospective: one bug fixed" | 09:22 |
| panda|rover | quiquell: you're optimist | 09:25 |
| quiquell | panda|rover: The only good scenario is no more defects passing gates :-/ | 09:26 |
| *** quiquell has quit IRC | 09:27 | |
| *** quiquell has joined #oooq | 09:27 | |
| *** holser_ has joined #oooq | 09:31 | |
| *** zoli is now known as zoli|lunch | 09:32 | |
| quiquell | jfrancoa: We have logs again http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/c57633c/logs/ | 09:45 |
| jfrancoa | quiquell: thanks, /me checking | 09:47 |
| *** jaosorior has quit IRC | 09:49 | |
| quiquell | panda|rover: Dirty stuff would be to merge it with the Depends-On to the tht top | 09:57 |
| *** holser_ has quit IRC | 09:59 | |
| panda|rover | quiquell: I bet we'd start seeing this fail again after it becomes package | 10:00 |
| quiquell | panda|rover: For sure... | 10:00 |
| quiquell | This whole think is like a perfect storm | 10:00 |
| *** sshnaidm|afk has quit IRC | 10:09 | |
| *** holser_ has joined #oooq | 10:15 | |
| *** holser_ has quit IRC | 10:15 | |
| *** holser_ has joined #oooq | 10:17 | |
| *** holser_ has joined #oooq | 10:19 | |
| *** holser_ has quit IRC | 10:20 | |
| *** bogdando has quit IRC | 10:26 | |
| *** bogdando_ has joined #oooq | 10:26 | |
| *** bogdando_ is now known as bogdando | 10:29 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ (1 more message) | 10:38 |
| *** dtantsur|bbl is now known as dtantsur | 10:49 | |
| *** brault_ has joined #oooq | 10:50 | |
| quiquell | panda|rover: Lat's day of this sprint is 15 of august ? | 10:52 |
| quiquell | last | 10:52 |
| *** brault has quit IRC | 10:53 | |
| panda|rover | quiquell: yes | 10:53 |
| panda|rover | 15 is retrospective day | 10:53 |
| quiquell | 15 I leave on PTO for three weeks | 10:53 |
| quiquell | GOing to change the card | 10:54 |
| panda|rover | quiquell: no you don't | 10:54 |
| panda|rover | quiquell: I'll come to madrid and I'll chain you to the chair | 10:54 |
| quiquell | panda|rover: Good to know, will change my location | 10:54 |
| panda|rover | quiquell: no need to change the card | 10:55 |
| panda|rover | quiquell: retrospective day is not counted as team time | 10:55 |
| quiquell | panda|rover: ack | 10:55 |
| quiquell | panda|rover: Feels good to reduce the days :-P | 10:55 |
| *** zoli|lunch is now known as zoli | 11:02 | |
| *** bogdando has quit IRC | 11:16 | |
| *** bogdando has joined #oooq | 11:16 | |
| *** quiquell has quit IRC | 11:21 | |
| *** quiquell has joined #oooq | 11:22 | |
| *** quiquell has quit IRC | 11:24 | |
| *** quiquell has joined #oooq | 11:24 | |
| *** skramaja has quit IRC | 11:31 | |
| *** quiquell has quit IRC | 11:40 | |
| *** d0ugal has quit IRC | 11:41 | |
| *** quiquell has joined #oooq | 11:42 | |
| *** quiquell has quit IRC | 11:43 | |
| *** quiquell has joined #oooq | 11:43 | |
| EmilienM | quiquell: what's up? | 11:44 |
| *** atoth has joined #oooq | 11:47 | |
| *** quiquell has quit IRC | 11:48 | |
| *** quiquell has joined #oooq | 11:48 | |
| *** quiquell_ has joined #oooq | 11:49 | |
| *** quiquell has quit IRC | 11:49 | |
| *** quiquell has joined #oooq | 11:52 | |
| *** quiquell_ has quit IRC | 11:52 | |
| *** amoralej is now known as amoralej|lunch | 11:54 | |
| *** quiquell_ has joined #oooq | 11:54 | |
| *** rfolco|off is now known as rfolco|ruck | 11:55 | |
| *** quiquell has quit IRC | 11:58 | |
| *** quiquell_ has quit IRC | 12:01 | |
| *** quiquell has joined #oooq | 12:02 | |
| *** sshnaidm has joined #oooq | 12:08 | |
| *** sshnaidm is now known as sshnaidm|off | 12:08 | |
| quiquell | Leaving for lunch | 12:13 |
| quiquell | Read you in a few | 12:13 |
| *** quiquell has quit IRC | 12:13 | |
| *** holser_ has joined #oooq | 12:25 | |
| *** trown|outtypewww is now known as trown | 12:34 | |
| *** agopi has quit IRC | 12:37 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ (1 more message) | 12:38 |
| *** amoralej|lunch is now known as amoralej | 12:47 | |
| *** rlandy has quit IRC | 12:48 | |
| *** rlandy has joined #oooq | 12:48 | |
| *** holser_ has quit IRC | 12:48 | |
| rfolco|ruck | arxcruz, known issue ? http://logs.openstack.org/09/582609/3/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/fb5afda/logs/undercloud/home/zuul/tempest.log.txt.gz | 12:48 |
| *** holser_ has joined #oooq | 12:49 | |
| arxcruz | rfolco|ruck: checking | 12:50 |
| rlandy | panda|rover: hi - all set with that bug? looking at jistr patch | 12:51 |
| panda|rover | rlandy: we're testing in the fix patch | 12:51 |
| panda|rover | rlandy: job still haven't completed | 12:51 |
| arxcruz | rfolco|ruck: so, overcloud deployment fail, i wonder why it continue to run tempest | 12:51 |
| rlandy | I left notes last night - hope it helped | 12:51 |
| arxcruz | rfolco|ruck: http://logs.openstack.org/09/582609/3/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/fb5afda/job-output.txt.gz#_2018-07-27_10_21_48_948707 | 12:51 |
| arxcruz | rfolco|ruck: so, the issue here isn't related to tempest itself failing, but why did the playbook tried to run tempest even though the overcloud deployment fails | 12:52 |
| rlandy | panda|rover: I can try it in my reproducer | 12:52 |
| arxcruz | rfolco|ruck: http://logs.openstack.org/09/582609/3/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/fb5afda/job-output.txt.gz#_2018-07-27_10_21_47_298863 | 12:52 |
| arxcruz | it's ignoring the deployment failure | 12:52 |
| *** holser_ has quit IRC | 12:53 | |
| rfolco|ruck | arxcruz, can you add a when clause there ? | 12:53 |
| panda|rover | rlandy: your reporducer proved that that was the only issue. Updates job it in overcloud deploy ATM, crossing fingers | 12:54 |
| arxcruz | rfolco|ruck: the overcloud deployment failure shouldn't be ignored | 12:54 |
| rfolco|ruck | arxcruz, by who | 12:55 |
| rlandy | panda|rover: just for my education (so I know how to follow the workflow here) how did you/jistr know the breaking change was in tripleo-common? | 12:56 |
| * rfolco|ruck wants to know this too ^ | 12:56 | |
| * rlandy messed around looking at changes in THT | 12:57 | |
| panda|rover | rfolco|ruck: rfolco|ruck jistr had to deal with a similar bug 2 years ago, and it left scars so deep that he still remembers the fix | 12:58 |
| panda|rover | rlandy: ^ | 12:58 |
| rlandy | oh I see, I feel for jistr | 12:58 |
| rfolco|ruck | déjà vu | 12:58 |
| panda|rover | rlandy: rfolco|ruck anyway, the key is to know exaclty what writes that docker_config.yaml, which is not tht, but is apparently done in config download | 12:58 |
| *** jfrancoa has quit IRC | 12:59 | |
| *** jfrancoa has joined #oooq | 12:59 | |
| rlandy | panda|rover: yep asked that exact question to two people - bth of whom ignored it | 12:59 |
| rlandy | now we know, good | 12:59 |
| rfolco|ruck | panda|rover, do you know the offending/breaking patch in treipleo-common ? | 13:00 |
| rfolco|ruck | I am just curious if this happened during gate breakage | 13:01 |
| arxcruz | rfolco|ruck: i would need to dig into the code | 13:01 |
| arxcruz | rfolco|ruck: from what i see, toci is ignoring the failure and running the other playbooks in sequence, but i need to understand what scenario003 do etc, etc | 13:02 |
| rfolco|ruck | arxcruz, np... just field for improvement... if overcloud deploy failed, no reason to try tempest... | 13:02 |
| *** quiquell has joined #oooq | 13:02 | |
| rfolco|ruck | arxcruz, ahhhhh the false positive bug | 13:02 |
| rfolco|ruck | arxcruz, nm, ignore. We'll fix this. | 13:03 |
| arxcruz | rfolco|ruck: ok | 13:03 |
| EmilienM | quiquell: I saw the patchset, let's see how it goes | 13:05 |
| rlandy | rfolco|ruck: panda|rover: I have it running in my reproducer | 13:08 |
| quiquell | EmilienM: lets see... I am crossing fingers | 13:08 |
| quiquell | rlandy: nice the findings you did, it was the issue | 13:14 |
| rlandy | quiquell: thanks! | 13:14 |
| rlandy | only took me two days ?! | 13:15 |
| quiquell | rlandy: not much, without logs and all | 13:15 |
| quiquell | rlandy: so we break the gates and the defect comes in | 13:16 |
| quiquell | rlandy: it was hide waitting for it | 13:16 |
| rlandy | really rfolco|ruck found the correct failing logs, just followed from there | 13:16 |
| rfolco|ruck | quiquell, I did not find any offending patch in t-common while gate was broken, did you? | 13:16 |
| quiquell | rfolco|ruck didn't check | 13:17 |
| rfolco|ruck | rlandy, I wish I had your eyes to find so tiny diff in the logs | 13:17 |
| quiquell | rfolco|ruck there is no new stuff? | 13:18 |
| quiquell | Maybe its a cluster interaction of varios changes elswhere | 13:19 |
| rfolco|ruck | quiquell, not in tripleo-common... don't know if other change would indirectly break it.... | 13:19 |
| rfolco|ruck | yep | 13:19 |
| quiquell | That get fixed with it | 13:19 |
| quiquell | The silly depends-on was making the job pass | 13:19 |
| rfolco|ruck | you'll have to explain me that magic | 13:20 |
| quiquell | Told us something ablut changing the lines of docker_configm.yaml | 13:20 |
| quiquell | Lucky as hell | 13:21 |
| rlandy | hmmm ... change failed in my hacked up reproducer - but that means nothing - it's **very** hacked up at this point | 13:21 |
| rlandy | will see what zuul reveals | 13:21 |
| quiquell | Have to be finishing | 13:21 |
| *** agopi has joined #oooq | 13:22 | |
| quiquell | If it passes we can make ruck rover dashboard demo to celebrate | 13:22 |
| rlandy | ^^ me looking forward | 13:23 |
| rlandy | rasca; going to do some work on your reviews today | 13:27 |
| rlandy | I'll fix sshnaidm|off's comments etc. | 13:27 |
| rlandy | rfolco|ruck: no fs073 ran with this job or multinode000? | 13:29 |
| rlandy | update | 13:29 |
| rfolco|ruck | rlandy, parsing you question... can you please elaborate ? | 13:30 |
| rlandy | rfolco|ruck: how will we know it fixed the problem if https://review.openstack.org/#/c/586499/ doesn;t run the broken tests? | 13:30 |
| rfolco|ruck | rlandy, quiquell depends-on this patch on that other | 13:31 |
| rlandy | rfolco|ruck: ah ok | 13:31 |
| *** quiquell_ has joined #oooq | 13:32 | |
| *** quiquell has quit IRC | 13:32 | |
| *** myoung has joined #oooq | 13:35 | |
| *** quiquell_ has quit IRC | 13:35 | |
| *** quiquell has joined #oooq | 13:35 | |
| quiquell | rlandy, rfolco|ruck: Job is at the critical task... | 13:36 |
| panda|rover | upgrades failed | 13:37 |
| quiquell | panda|rover: Is legit, is a upgrade fail, | 13:38 |
| rfolco|ruck | is upgrade job working/functional/reliable? its non-voting... | 13:38 |
| quiquell | rfolco|ruck: They have some fixes for it already | 13:39 |
| quiquell | rfolco|ruck: Supose that's why is nv | 13:39 |
| *** bogdando has quit IRC | 13:43 | |
| quiquell | rfolco|ruck, panda|rover: Looks like we are still missing some logs at undercloud/var/lib/ | 13:45 |
| quiquell | in the upgrade job | 13:45 |
| rfolco|ruck | you mean collect logs not copying it to logs ? | 13:45 |
| quiquell | rfolco|ruck: Fixed some stuff in the patch but still for the upgrade jobs /var/lib/ only have unbound | 13:45 |
| quiquell | rfolco|ruck: Previous failing update jobs have it all | 13:46 |
| quiquell | rfolco|ruck: Passing jobs are storing undercloud/var/lib correctly | 13:48 |
| *** jaganathan has quit IRC | 13:48 | |
| rfolco|ruck | quiquell, show me example and I can fill a bug | 13:48 |
| rfolco|ruck | please | 13:48 |
| quiquell | rfolco|ruck: http://logs.openstack.org/28/585528/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/b328978/logs/undercloud/var/lib/ | 13:49 |
| quiquell | rfolco|ruck: From the uber fix ^ | 13:49 |
| quiquell | rfolco|ruck: It's missing mistral and all | 13:49 |
| rfolco|ruck | uber ? | 13:49 |
| rfolco|ruck | uber deliver enchilada in spain ? | 13:50 |
| rfolco|ruck | does ^ | 13:50 |
| quiquell | rfolco|ruck: Also fixes | 13:50 |
| quiquell | rfolco|ruck: http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/c57633c/logs/undercloud/var/lib/ | 13:50 |
| panda|rover | updates passed ? | 13:50 |
| rlandy | looks good | 13:50 |
| quiquell | rfolco|ruck: Failing 'update' job have it all | 13:50 |
| panda|rover | or am I dreaming ? | 13:50 |
| quiquell | panda|rover: [[ 0 == 0 ]] | 13:51 |
| quiquell | It has pass | 13:51 |
| panda|rover | but it has really passed ? and really updated ? | 13:51 |
| panda|rover | jfrancoa: pinch us | 13:51 |
| jfrancoa | panda|rover: the job failed, but I need to see the logs in /var/lib/mistral to see if the file is missing. But there are no logs stored in the job | 13:53 |
| jfrancoa | panda|rover: I am talking about the upgrades job quiquell pasted the logs from | 13:53 |
| quiquell | jfrancoa: Talking about update jobs, looks like it's passing, can we ensure that it really updating ? | 13:54 |
| jfrancoa | quiquell: we're now in a DFG retro. I will have a look at it as soon as it finish | 13:54 |
| quiquell | jfrancoa: ack | 13:55 |
| panda|rover | it may be already too late | 13:55 |
| quiquell | panda|rover: Maybe we have another defect ? | 13:55 |
| quiquell | panda|rover: We also need to merge the tripleo-common stuff first | 13:56 |
| quiquell | jfrancoa: Where do we have to look ? yum packages ? containers ? | 13:58 |
| quiquell | panda|rover: What do we doo ? | 13:58 |
| panda|rover | quiquell: what do we do for what ? | 13:59 |
| panda|rover | quiquell: I +1 jistr change, now we announce that it works and ask for reviews. I'll +2 Emilien's patch in the meantime | 14:00 |
| panda|rover | quiquell: for the upgrade job, we'll fix them after this | 14:00 |
| panda|rover | well , ruck and rover | 14:00 |
| quiquell | panda|rover: So we try to workflow the fixes, ok. | 14:00 |
| *** kopecmartin has quit IRC | 14:01 | |
| rlandy | I voted on those two reviews as well | 14:03 |
| quiquell | Yep, me too | 14:04 |
| quiquell | panda|rover: Going to post the Jiri review a #tripleo to get reviews on it | 14:05 |
| quiquell | panda|rover: let's hope no more defects pass the gates | 14:06 |
| panda|rover | hope is not a strategy :) | 14:07 |
| quiquell | panda|rover: They are waitting for it | 14:08 |
| *** links has quit IRC | 14:10 | |
| rfolco|ruck | arxcruz, are you aware of http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-3nodes-multinode/cd65ccf/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-07-27_09_21_23 | 14:10 |
| quiquell | panda|rover, rfolco|ruck, rlandy, ssbarnea: Want a little demo of the ci dashboard ? | 14:11 |
| rfolco|ruck | arxcruz, TestNetworkBasicOps.test_network_basic_ops failing almost every job for tripleo-ci-centos-7-3nodes-multinode | 14:11 |
| rlandy | sure | 14:11 |
| ssbarnea | yep | 14:12 |
| quiquell | arxcruz: Going to do a little demo of the new rr dashboard | 14:16 |
| quiquell | https://bluejeans.com/7891065232 | 14:17 |
| quiquell | rlandy, ssbarnea: ^ | 14:18 |
| *** links has joined #oooq | 14:29 | |
| myoung | quiquell: mind if i join? | 14:29 |
| quiquell | myoung: Join join, didn't know if you were up | 14:29 |
| myoung | kk | 14:30 |
| myoung | will be there in a few, don't wait, finishing up another call | 14:30 |
| myoung | thx! | 14:30 |
| quiquell | ok | 14:30 |
| quiquell | no problem I can repeat it | 14:30 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, legacy-tripleo-ci-centos-7 (1 more message) | 14:38 |
| jfrancoa | Hey folks, something we're discussing in the retro is about adding a bot in rhosp-upgrades channel. Who does take care of the bot mantainence in the oooq channel? or who did set it up, so we could ask him/her for support on this? | 14:40 |
| panda|rover | jfrancoa: what kind of bot ? | 14:40 |
| jfrancoa | panda|rover: something like the hubbot you have, that would notify us for example if the number of bugs is over a threshold or if the upgrades job would be failing | 14:41 |
| panda|rover | jfrancoa: also, I forgot to metion earlier, is tripleo-upgrade actually checking that the start packages were updated and the update/upgrade really took place ? | 14:42 |
| jfrancoa | panda|rover: what others are there? (I know also about the kudos ++ one) | 14:42 |
| panda|rover | jfrancoa: openstack bot resolves the bugs description for example | 14:43 |
| jfrancoa | panda|rover: it does check if the upgrade/update ended succesfuly, but if the tripleo-upgrade doesn't get to run (like it was the case recently) then there is no way to verify it | 14:43 |
| panda|rover | jfrancoa: hubbot was created by adarasz, I think knowledge has been passed to myoung now | 14:43 |
| jfrancoa | panda|rover: we were discussing in the retro, that we should add some validations in tripleo-validations which would run during the validation step at the end of the job and check if the upgrade/update was really performed | 14:43 |
| jfrancoa | myoung: is there any site/document where I could start checking how to set it up? or do you mind if I set up a meeting next week to learn about it? | 14:47 |
| myoung | jfrancoa: looking for it now :) there's an etherpad | 14:48 |
| jfrancoa | myoung: great, thanks! | 14:48 |
| myoung | TLDR, on the tripleo-ci teams' infra tenant in RDO cloud we have an instance running the boty | 14:48 |
| myoung | bot | 14:48 |
| myoung | #config plugins.GateStatus.changeIDs | 14:48 |
| myoung | %config plugins.GateStatus.changeIDs | 14:48 |
| hubbot | myoung: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab If12c8fe9bd0bea98a4842f279399285344f22246 | 14:48 |
| myoung | %config plugins.GateStatus.jobFilter | 14:49 |
| hubbot | myoung: .*-nv$ | 14:49 |
| myoung | %config plugins.GateStatus.changeIDs | 14:49 |
| hubbot | myoung: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab If12c8fe9bd0bea98a4842f279399285344f22246 | 14:49 |
| myoung | %printusers | 14:50 |
| hubbot | Current userFilter: ['zuul', 'rdo-ci', 'rdothirdparty']; all users and comments within the timeLimit: zuul (12), rdo-ci (14), rdothirdparty (12) | 14:50 |
| myoung | %gatestatus | 14:50 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, legacy-tripleo-ci-centos-7 (1 more message) | 14:50 |
| myoung | jfrancoa: ^^ you can use a '%', or send private messages to 'hubbot' | 14:50 |
| * myoung digs up the etherpad | 14:50 | |
| myoung | jfrancoa: https://etherpad.openstack.org/p/tripleo-ci-hubbot-configuration is the etherpad, and https://bluejeans.com/s/KQWY3 is a BJ recording of Atilla (no longer with us) walking through how it works | 14:51 |
| * myoung makes a note to move this to a google doc or an MD in source control | 14:51 | |
| jfrancoa | myoung: thanks a lot! | 14:52 |
| myoung | jfrancoa: (might be too much info :)) - if you are curious about implementation and how things are wired into our playbooks that can reprovision and/or set up $infraThings, the reviews (now merged) are archived from our team's sprint 12 cards: https://trello.com/c/vdDrtoee/50-hubbot-is-private-code-running-on-a-private-server-lets-open-this-up-and-run-on-a-shared-instance, https://trello.com/c/iWAz4ONC/73-hubbot-bot-add-two-dummy-changes-on-tht | 14:55 |
| myoung | -to-watch | 14:55 |
| myoung | jfrancoa: it's basically an instance of https://github.com/ProgVal/Limnoria with some plugin/conf file tweaks for our usage | 14:56 |
| jfrancoa | myoung: that's really useful. I can follow the reviews, thanks for all the info, I think I have quite a lot to start with | 14:59 |
| myoung | jfrancoa: ack, no worries. note: I'm not an expert, just the TC with a side of OCD around notes and links hahaha. | 15:00 |
| *** links has quit IRC | 15:02 | |
| quiquell | jfrancoa: We have also a grafan + irc bot thing, so you set the tresholds in grafana and you can check it in IRC | 15:05 |
| panda|rover | myoung: you have time to chat ? | 15:06 |
| jfrancoa | quiquell: I'll set up a bj call on monday with you, if you don't mind, and you can show me how it's done | 15:06 |
| myoung | panda|rover: I do, in about 10 mins | 15:06 |
| myoung | panda|rover: or perhaps 78 | 15:07 |
| myoung | 7 | 15:07 |
| myoung | :) | 15:07 |
| quiquell | jfrancoa: Deal, spanish redhat meeting, that's weird :-) | 15:07 |
| quiquell | This is the ci-dashboard code https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit | 15:09 |
| jfrancoa | quiquell: thanks, appointment set ;-) | 15:17 |
| quiquell | jfrancoa: Will bring cookies | 15:18 |
| panda|rover | jfrancoa: request the GDPR | 15:19 |
| quiquell | panda|rover: What's that ? | 15:19 |
| panda|rover | quiquell: the policy to sign for the cookies | 15:20 |
| quiquell | panda|rover: Damn... I am slow | 15:20 |
| jfrancoa | panda|rover: another one? I am fed up of having it popped up everywhere | 15:21 |
| quiquell | jfrancoa: non-stop :-) | 15:21 |
| myoung | panda|rover: free now if you still have time to chat | 15:21 |
| myoung | panda|rover: (was hoping to get a few mins today as well, i have a few specific Q's to chat about) | 15:21 |
| panda|rover | myoung: ok, going to your root | 15:22 |
| panda|rover | room | 15:22 |
| quiquell | jfrancoa: I found this regardin failing upgrade and mistral, don't know if it's of any help http://logs.openstack.org/28/585528/10/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/b328978/logs/undercloud/tmp/ansible-mistral-actionfI0kwt/ | 15:30 |
| quiquell | Shi... we are so unlucky there are openstack infra issues at zuul.o.o | 15:34 |
| quiquell | we have retry_limit errors at the fix | 15:34 |
| -openstackstatus- NOTICE: A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes | 15:35 | |
| quiquell | rfolco|ruck, panda|rover: ^ What can go wrong ... | 15:36 |
| panda|rover | yay | 15:36 |
| *** quiquell is now known as quique|luckyluck | 15:36 | |
| rfolco|ruck | :-/ | 15:37 |
| quique|luckyluck | Don't know what else can fail :-( | 15:37 |
| *** quique|luckyluck is now known as quiquell | 15:37 | |
| *** gkadam has quit IRC | 15:38 | |
| quiquell | Drop now, have a good weekend | 15:39 |
| *** quiquell has quit IRC | 15:39 | |
| *** pliu_ has quit IRC | 15:43 | |
| *** gkadam has joined #oooq | 15:53 | |
| *** gkadam has quit IRC | 15:54 | |
| *** gkadam has joined #oooq | 15:55 | |
| *** jfrancoa has quit IRC | 16:02 | |
| *** jfrancoa has joined #oooq | 16:03 | |
| *** zoli is now known as zoli|gone | 16:09 | |
| *** zoli|gone is now known as zoli | 16:09 | |
| *** amoralej is now known as amoralej|off | 16:11 | |
| *** links has joined #oooq | 16:14 | |
| *** trown is now known as trown|lunch | 16:15 | |
| *** links has quit IRC | 16:17 | |
| *** links has joined #oooq | 16:17 | |
| rlandy | https://review.openstack.org/#/c/584508/ | 16:18 |
| rlandy | rfolco|ruck: ^^ still needed? | 16:18 |
| rlandy | rfolco|ruck: panda|rover: I need to retest https://review.openstack.org/#/c/583195 for marios once the job fixing patches merge | 16:20 |
| rlandy | I removed my -1 | 16:21 |
| *** yolanda has quit IRC | 16:22 | |
| panda|rover | rlandy: ok | 16:22 |
| *** links has quit IRC | 16:23 | |
| rlandy | rasca: you around? | 16:29 |
| *** yolanda has joined #oooq | 16:29 | |
| *** tesseract has quit IRC | 16:32 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ (1 more message) | 16:38 |
| *** jfrancoa has quit IRC | 16:40 | |
| *** holser_ has joined #oooq | 16:40 | |
| *** panda|rover is now known as panda|rover|off | 17:03 | |
| *** trown|lunch is now known as trown | 17:05 | |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| rfolco|ruck | rlandy, sorry was @lunch, looking | 17:16 |
| rlandy | rfolco|ruck: no worries | 17:16 |
| rfolco|ruck | rlandy, ok so 584508 is a patch that replaces TAGS with ansible workflow. It is still valid , may need rebasing | 17:18 |
| rlandy | rfolco|ruck:ok - I am working on rasca's stuff in the mean time - waiting for the first two patches to merge | 17:23 |
| *** atoth has quit IRC | 17:54 | |
| *** atoth has joined #oooq | 17:55 | |
| *** myoung is now known as myoung|lunch | 18:09 | |
| *** jtomasek has quit IRC | 18:10 | |
| *** sshnaidm|off has quit IRC | 18:32 | |
| hubbot | FAILING CHECK JOBS on stable/queens: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb- (1 more message) | 18:38 |
| *** myoung|lunch is now known as myoung | 19:18 | |
| *** jtomasek has joined #oooq | 19:19 | |
| *** jtomasek has quit IRC | 19:19 | |
| *** rlandy is now known as rlandy|brb | 19:55 | |
| *** ccamacho1 has joined #oooq | 20:00 | |
| *** ccamacho has quit IRC | 20:01 | |
| *** rlandy|brb is now known as rlandy | 20:17 | |
| *** trown is now known as trown|outtypewww | 20:31 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7 (2 more messages) | 20:38 |
| *** gkadam has quit IRC | 20:58 | |
| *** brault_ has quit IRC | 21:02 | |
| panda|rover|off | and nothing will merge again for another day | 21:23 |
| *** rfolco|ruck is now known as rfolco|off | 21:33 | |
| *** holser_ has quit IRC | 21:38 | |
| agopi | rlandy, rook https://review.openstack.org/583717 this should ensure browbeat run | 21:57 |
| agopi | sorry for taking a lot of time, had troubles getting reproducer run to verify failing commands in rdo cloud | 21:57 |
| agopi | have a happy weekend yall | 21:57 |
| *** sshnaidm|off has joined #oooq | 22:04 | |
| rlandy | agopi: thanks - will review that | 22:06 |
| agopi | rlandy++ | 22:06 |
| hubbot | agopi: rlandy's karma is now 16 | 22:06 |
| agopi | fingers crossed | 22:06 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch, tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7 (2 more messages) | 22:38 |
| *** agopi has quit IRC | 22:38 | |
| *** myoung has quit IRC | 23:22 | |
| *** rlandy has quit IRC | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!