| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 00:14 |
|---|---|---|
| *** jtomasek has quit IRC | 01:10 | |
| *** jtomasek has joined #oooq | 01:45 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containers- (1 more message) | 02:14 |
| *** jtomasek has quit IRC | 03:05 | |
| *** skramaja has joined #oooq | 03:48 | |
| *** udesale has joined #oooq | 04:12 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (1 more message) | 04:14 |
| *** ykarel has joined #oooq | 04:42 | |
| *** ratailor has joined #oooq | 04:47 | |
| *** udesale_ has joined #oooq | 05:37 | |
| *** udesale has quit IRC | 05:38 | |
| *** udesale__ has joined #oooq | 05:38 | |
| *** udesale_ has quit IRC | 05:41 | |
| *** marios has joined #oooq | 05:42 | |
| *** quiquell|off is now known as quiquell | 05:49 | |
| quiquell | sshnaidm|off: +1w here ? https://review.openstack.org/#/c/569120 Add newton->ocata undercloud upgrades to unsupported list | 05:50 |
| quiquell | For the sprint | 05:50 |
| *** links has joined #oooq | 05:57 | |
| *** saneax-_-|AFK is now known as saneax | 05:58 | |
| *** saneax is now known as saneax-_-|AFK | 06:00 | |
| *** jfrancoa has joined #oooq | 06:03 | |
| *** skramaja has quit IRC | 06:05 | |
| *** skramaja_ has joined #oooq | 06:05 | |
| *** kopecmartin has joined #oooq | 06:06 | |
| *** anande has joined #oooq | 06:12 | |
| *** anande has quit IRC | 06:13 | |
| *** anande has joined #oooq | 06:14 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (1 more message) | 06:14 |
| *** saneax-_-|AFK is now known as saneax | 06:15 | |
| *** jrist has quit IRC | 06:24 | |
| *** jrist has joined #oooq | 06:38 | |
| *** holser__ has joined #oooq | 06:41 | |
| *** tosky has joined #oooq | 06:53 | |
| *** bogdando has joined #oooq | 06:58 | |
| *** jbadiapa has joined #oooq | 07:01 | |
| *** ccamacho has joined #oooq | 07:10 | |
| *** skramaja_ is now known as skramaja | 07:10 | |
| *** tesseract has joined #oooq | 07:10 | |
| *** ccamacho has quit IRC | 07:15 | |
| *** ccamacho has joined #oooq | 07:16 | |
| *** udesale__ has quit IRC | 07:30 | |
| *** udesale__ has joined #oooq | 07:30 | |
| *** holser__ has quit IRC | 07:43 | |
| *** kopecmartin has quit IRC | 07:45 | |
| *** holser__ has joined #oooq | 07:45 | |
| *** kopecmartin has joined #oooq | 07:45 | |
| *** anande has quit IRC | 08:03 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 08:04 | |
| *** jaosorior has joined #oooq | 08:04 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 08:14 |
| *** lucas-afk is now known as lucasagomes | 08:20 | |
| *** jaosorior has quit IRC | 08:26 | |
| *** anande has joined #oooq | 08:30 | |
| quiquell | sshnaidm|rover: Can you +1w here ? https://review.openstack.org/#/c/ for the sprint to progress | 08:40 |
| quiquell | sshnaidm|off: This is the correct one | 08:41 |
| quiquell | https://review.openstack.org/#/c/569120 | 08:41 |
| *** jaosorior has joined #oooq | 08:49 | |
| *** anande has quit IRC | 08:53 | |
| *** anande has joined #oooq | 08:53 | |
| *** myoung has quit IRC | 08:55 | |
| *** rnoriega has quit IRC | 08:55 | |
| *** rnoriega has joined #oooq | 08:56 | |
| *** myoung has joined #oooq | 08:57 | |
| *** moguimar has quit IRC | 09:05 | |
| *** moguimar has joined #oooq | 09:06 | |
| *** jbadiapa_ has joined #oooq | 09:32 | |
| *** zoli is now known as zoli|lunch | 09:32 | |
| *** jbadiapa has quit IRC | 09:35 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 10:14 |
| sshnaidm|rover | quiquell, done | 10:18 |
| quiquell | sshnaidm|rover: Cool thanks | 10:18 |
| *** udesale__ has quit IRC | 10:25 | |
| *** udesale has joined #oooq | 10:26 | |
| *** udesale has quit IRC | 10:29 | |
| *** udesale has joined #oooq | 10:30 | |
| quiquell | panda: Is rlandy already testing this https://trello.com/c/cbzgLxCA/736-move-fs37-job-to-directly-use-the-new-releases-framework ? | 10:43 |
| quiquell | Don't know what task to take | 10:43 |
| *** zoli|lunch is now known as zoli | 10:47 | |
| panda | quiquell: we should be in th testing phase, so what's left to do is test the effect of the script and the new workflow on the existing jobs | 10:48 |
| quiquell | panda: But do we need to take the related cards ? | 10:49 |
| quiquell | panda: Do you have a bj minute ? | 10:54 |
| panda | quiquell: yes | 10:58 |
| quiquell | Calling you | 11:00 |
| panda | quiquell: I don't see you | 11:03 |
| quiquell | https://bluejeans.com/7891065232 | 11:05 |
| quiquell | Call me please, don't know what's up calling you | 11:05 |
| *** anande has quit IRC | 11:05 | |
| *** udesale has quit IRC | 11:24 | |
| panda | quiquell: you may want to look at this | 11:27 |
| panda | http://logs.openstack.org/65/566565/11/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/3bae60f/job-output.txt.gz#_2018-05-18_16_53_54_473071 | 11:27 |
| panda | quiquell: the script is run, but there is no output, and no logs. The script reports no error. | 11:28 |
| quiquell | Was just reproducing it | 11:28 |
| panda | quiquell: to be honest, I think the script has too many catchall exception | 11:30 |
| *** lucasagomes is now known as lucas-hungry | 11:30 | |
| quiquell | panda: A lot of refactoring can be done in the script | 11:31 |
| quiquell | Also more declarative | 11:31 |
| *** sshnaidm|rover has quit IRC | 11:34 | |
| panda | quiquell: I'm not talking about refactoring, I'm talking about the try: except Exception: pass | 11:35 |
| panda | stuff like this usually catch all the exception and mute them | 11:35 |
| quiquell | panda: It's printing it | 11:38 |
| weshay | panda, quiquell need core? | 11:38 |
| quiquell | But yes it's weird | 11:38 |
| quiquell | we are wrapping | 11:38 |
| quiquell | The failure into a RuntimeError late ron | 11:38 |
| *** zoli is now known as zoli|afk-TPB | 11:39 | |
| quiquell | weshay: https://review.openstack.org/#/c/569120 already at the gates | 11:39 |
| quiquell | panda: Going for lunch now, let's go back later at the issue | 11:40 |
| *** quiquell is now known as quiquell|lunch | 11:40 | |
| bogdando | folks, why do we keep running tripleo-ci-centos-7-3nodes-multinode, it has not success story, at least as I can see it from elastic recheck | 11:51 |
| bogdando | has no* :) | 11:52 |
| weshay | bogdando, I'll ask Alex how he wants to handle that. | 11:54 |
| weshay | bogdando, I'm more worried about containers-multinode on queens tbh | 11:55 |
| *** rfolco has joined #oooq | 11:56 | |
| weshay | I think we missed a cherry-pick in deployed-server config but can't figure out what | 11:56 |
| *** tcw1 is now known as tcw | 12:01 | |
| *** trown|outtypewww is now known as trown | 12:03 | |
| *** tcw has quit IRC | 12:04 | |
| bogdando | weshay: can I found the last success run of the latter for stable/queens? | 12:09 |
| bogdando | like http://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Ftripleo-ci?searchJob=tripleo-ci-centos-7-containers-multinode&duration=P1M | 12:09 |
| *** sshnaidm|rover has joined #oooq | 12:11 | |
| weshay | bogdando, the deployment never gets started on that job.. I have a local recreate for about a week now | 12:11 |
| weshay | I can tmate you in if you like | 12:11 |
| bogdando | weshay: what do you mean, there is no success logs available? | 12:12 |
| *** ratailor has quit IRC | 12:12 | |
| bogdando | I wanted to try logreduce on it | 12:12 |
| weshay | fairly old at this point | 12:12 |
| weshay | ah | 12:12 |
| weshay | bogdando, I suspect you are picking up branches != queens in openstack-health | 12:13 |
| *** jfrancoa is now known as jfrancoa|lunch | 12:13 | |
| *** lucas-hungry is now known as lucasagomes | 12:13 | |
| bogdando | I couldn't find branches choices there | 12:13 |
| bogdando | is it groupKey=build_branch ? | 12:14 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (1 more message) | 12:14 |
| weshay | bogdando, going back to 5/14 I have no record of it passing | 12:14 |
| weshay | bogdando, it passed prior to the centos 7.4 -> 7.5 outtage | 12:15 |
| bogdando | btw, did you folks know of that brilliant logreduce tool? Workshop - Anomaly Detection in CI logs | 12:15 |
| bogdando | https://fedorapeople.org/~tdecacqu/opendevconf.git | 12:15 |
| weshay | bogdando, yes.. but have not used it yet | 12:16 |
| weshay | it is brilliant | 12:16 |
| bogdando | so I'm trying... :) | 12:16 |
| weshay | I think sagi has used it | 12:16 |
| sshnaidm|rover | bogdando, yeah, we saw it, but still didn't find a proper usage for it | 12:17 |
| bogdando | right, not sure if it fits well the fully automated cases, but should be extremely helpful for manual logs debugging tasks | 12:19 |
| weshay | sshnaidm|rover, thought you were off | 12:20 |
| sshnaidm|rover | weshay, yeah, yesterday | 12:21 |
| weshay | ah ok | 12:21 |
| weshay | sshnaidm|rover, can you focus 100% on the containers-multinode (queens) today? | 12:21 |
| sshnaidm|rover | weshay, it's what I'm doing now... | 12:21 |
| weshay | sshnaidm|rover, thank you sir | 12:21 |
| sshnaidm|rover | bogdando, do you know maybe why undercloud installation sets 127.0.0.1 as dns server? | 12:22 |
| *** quiquell|lunch is now known as quiquell | 12:22 | |
| bogdando | sshnaidm|rover: nope | 12:23 |
| bogdando | sshnaidm|rover: but it comes from t-h-t DnsServers logic | 12:24 |
| *** myoung is now known as myoung|ruck | 12:27 | |
| *** jfrancoa|lunch is now known as jfrancoa | 12:30 | |
| *** zoli|afk-TPB is now known as zoli | 12:32 | |
| *** tcw has joined #oooq | 12:34 | |
| *** jtomasek has joined #oooq | 12:35 | |
| rfolco | sshnaidm|rover, coz it relies on unbound? | 12:39 |
| rfolco | if we're talking about the same thing, unbound uses forwarding.conf for dns... | 12:42 |
| rfolco | forward-addr: 208.67.222.222 | 12:42 |
| rfolco | forward-addr: 8.8.8.8 | 12:42 |
| *** jbadiapa_ has quit IRC | 12:43 | |
| *** jbadiapa_ has joined #oooq | 12:59 | |
| *** jtomasek has quit IRC | 13:10 | |
| *** udesale has joined #oooq | 13:13 | |
| sshnaidm|rover | rfolco, where do you see it? in unbound config? | 13:18 |
| sshnaidm|rover | rfolco, unbound serer sets up before running a job, so I'm not sure unbound is guilty.. | 13:18 |
| quiquell | Ok sprint13 commit merged | 13:18 |
| quiquell | https://review.openstack.org/#/c/569120/ | 13:18 |
| sshnaidm|rover | weshay, can you vote please? https://review.openstack.org/#/c/569779/ | 13:18 |
| quiquell | weshay, panda, sshnaidm|rover: Some cores https://review.openstack.org/#/c/569115/ | 13:19 |
| weshay | sshnaidm|rover, we could probably just workflow that | 13:20 |
| sshnaidm|rover | weshay, yep | 13:20 |
| weshay | sshnaidm|rover, /me wonders why the linter is not catching that | 13:20 |
| sshnaidm|rover | weshay, yeah... me too | 13:21 |
| rfolco | sshnaidm|rover, I am talking about this... https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/nodepool-base/finalise.d/89-unbound#L119 | 13:24 |
| *** udesale has quit IRC | 13:24 | |
| rfolco | if your job has this nodepool-base element, it overrides resolv.conf :) | 13:25 |
| *** udesale has joined #oooq | 13:32 | |
| quiquell | weshay: Do you know if we have a public smtp server to use ? | 13:32 |
| myoung|ruck | arxcruz: ^^ (arx has done some smtp stuff w/ tempest results and might have plumbed this out) | 13:34 |
| * myoung|ruck will brb | 13:34 | |
| quiquell | myoung|ruck: Thanks ! | 13:36 |
| arxcruz | quiquell: myoung|ruck there isn't, you must use google api to do it (if your mail is already in gsuite) | 13:38 |
| quiquell | arxcruz: what user:password are you using for tempest mail tool ? | 13:39 |
| rfolco | If I remember right he uses a personal gmail account ... might be wrong | 13:43 |
| quiquell | Ok, will create a fake one | 13:43 |
| myoung|ruck | quiquell: maybe make it generic (tripleo-ci@gmail.com) or something, so we can reuse it. I think it's suboptimal to have arx's stuff going thru his personal email creds | 13:44 |
| quiquell | myoung|ruck: I will it's not related to tempest though | 13:45 |
| *** skramaja has quit IRC | 13:45 | |
| myoung|ruck | quiquell, arxcruz: I only mean if we're having to (generally) send email from tripleo-ci owned services/jobs (tempest or otherwise), it might make sense to solve the problem (jobs need a way to send email cleanly) once and reuse in future. {shrug} | 13:59 |
| myoung|ruck | have same issue in rdo2 | 13:59 |
| weshay | scrum time | 14:01 |
| *** tesseract-RH has joined #oooq | 14:09 | |
| * myoung|ruck is nose down in rdo2 outage debug/fix, weshay covering for me. thanks weshay! | 14:10 | |
| myoung|ruck | weshay++ | 14:10 |
| hubbot | myoung|ruck: weshay's karma is now 3 | 14:10 |
| *** tesseract has quit IRC | 14:12 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 (1 more message) | 14:14 |
| weshay | rfolco, how is software factory treating ya? | 14:25 |
| weshay | need anything? | 14:25 |
| panda | marios: is not only when you have acess to the undercloud. If I put a patch in tripleo-ci with a env command at the end, I can see with 0 effort and no privilege every variable in the evnironment | 14:25 |
| marios | panda: ack, yeah ci can do it for you :) | 14:25 |
| panda | marios: yeah, and we we did export a password, then with the env comand already dumped maybe someone can realize we are leaking password before someone else | 14:26 |
| rfolco | weshay, no, nicolas is helping me with rhel7 image elements while I fix undercloud and multinode jobs on centos7 as a baseline | 14:26 |
| panda | if we* | 14:26 |
| rfolco | weshay, I'm good | 14:26 |
| weshay | rfolco, rockin | 14:26 |
| rfolco | weshay, my next challenge will be adjusting repos with rhos-release... | 14:27 |
| weshay | rfolco, k.. review the release files in code.eng/tripleo-environments | 14:28 |
| rfolco | will do | 14:29 |
| *** quiquell is now known as quiquell|off | 14:38 | |
| rasca | weshay, myoung|ruck, the problem we were getting earlier apparently was caused by a general slowness of the entire HA datacenter, that now seems fixed | 14:41 |
| *** saneax is now known as saneax-_-|AFK | 14:42 | |
| myoung|ruck | rasca: gotcha. going thru this weekend's runs in rdo2 we had jobs fail attempting to access https://images.rdoproject.org/master/rdo_trunk/e8480e465826874c7851e6bc5672c9cbdee1cdbb_c19a0cf9 and the like...might not have just been the pidone datacenter | 14:44 |
| rasca | myoung|ruck, yeah and the fact that it recovered by itself make me think that someone just re attached the cable | 14:44 |
| rasca | :D | 14:44 |
| myoung|ruck | rasca: on friday evening I was seeing transfer rates of 30kb / sec, then sat failures with wholesale timeouts | 14:45 |
| myoung|ruck | just rekicking things now to get promotions rolling again, while I sort the slave issues we talked about this morning | 14:45 |
| *** links has quit IRC | 14:51 | |
| chandankumar | myoung|ruck: weshay I will be little late for standup | 14:52 |
| * chandankumar headed home | 14:52 | |
| *** trown is now known as trown|brb | 14:56 | |
| weshay | chandankumar, k | 14:57 |
| *** trown|brb is now known as trown | 15:00 | |
| myoung|ruck | chandankumar: ack, weshay is covering standups for me today | 15:00 |
| *** tesseract-RH has quit IRC | 15:16 | |
| *** tesseract has joined #oooq | 15:31 | |
| *** panda is now known as panda|bbl | 15:33 | |
| *** bogdando has quit IRC | 15:33 | |
| weshay | chandankumar, you can skip the sprint14 mtg, since you have it lined up | 15:57 |
| weshay | panda|bbl, ready when you are | 15:57 |
| chandankumar | weshay: sure | 15:57 |
| *** zoli is now known as zoli|gone | 16:03 | |
| *** zoli|gone is now known as zoli | 16:03 | |
| *** ykarel is now known as ykarel|away | 16:06 | |
| myoung|ruck | panda|bbl: you coming? | 16:08 |
| *** marios has quit IRC | 16:09 | |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 16:10 | |
| *** ykarel|away has quit IRC | 16:13 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 16:14 |
| *** holser__ has quit IRC | 16:15 | |
| *** tosky__ has joined #oooq | 16:32 | |
| *** tosky is now known as Guest12220 | 16:32 | |
| *** tosky__ is now known as tosky | 16:32 | |
| *** trown is now known as trown|lunch | 16:37 | |
| *** dougbtv_ has quit IRC | 16:43 | |
| *** udesale has quit IRC | 17:00 | |
| *** kopecmartin has quit IRC | 17:11 | |
| *** tesseract has quit IRC | 17:28 | |
| *** saneax-_-|AFK is now known as saneax | 17:36 | |
| *** trown|lunch is now known as trown | 17:49 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containers- (1 more message) | 18:14 |
| weshay | myoung|ruck, what's the progress on bm? | 18:19 |
| *** jfrancoa has quit IRC | 18:48 | |
| *** dtrainor has joined #oooq | 19:02 | |
| myoung|ruck | weshay: I have a summary mail coming, do you want TLDR here? | 19:28 |
| weshay | myoung|ruck, what's the TLDR | 19:30 |
| *** dtrainor has quit IRC | 19:32 | |
| myoung|ruck | weshay: re: pip install concurrency issue, fixed for BM nodes, made same fix to global jenkins config (so pidone nodes and others are also covered) - that seems to have worked (still checking job results) | 19:32 |
| myoung|ruck | weshay: re envD being unreachable, it needed a hard reboot from console, jobs now run on it, and resolving libvirt package issues on base node. | 19:33 |
| myoung|ruck | re: periodic OVB jobs (not rhos-12/13 but the RDO ones), they seem to have been failing since history is available (back to april?) on a missing centos image, that one is lower priority as it's not part of current promo criteria | 19:34 |
| myoung|ruck | re: current new hash from weekend on master, jobs still running (fs 20 takes forever...full tempest) - but BM OC deploy fails (not root caused yet). previous hash deployed OC but was failing in validate | 19:35 |
| myoung|ruck | re: queens promotion, about to rekick those now that we've resolved pip issue, same hash as before so we just need a BM fix | 19:36 |
| myoung|ruck | re: rasca's BMU jobs, now that https://review.openstack.org/#/c/569826 just merged those shoudl be good to go | 19:36 |
| * myoung|ruck notes that it's TLDR still...just one line per iron in the fire | 19:37 | |
| myoung|ruck | finally re: pike, ocata, and rhos-12/13 gates for {TQ,TQE}, no change since friday...only one of me heh...but plan to start in once the above stuff is in a better place | 19:37 |
| *** dtrainor has joined #oooq | 19:38 | |
| * myoung|ruck backs away slowly and returns to rdo2, he's online ~ 3 more hours | 19:38 | |
| weshay | myoung|ruck, master fs20 OC is working | 19:38 |
| myoung|ruck | weshay: yes, it's 4:20 into the run, executing tempest. it has another hour or so till done | 19:39 |
| weshay | ya | 19:39 |
| * myoung|ruck expects that to pass | 19:39 | |
| * weshay thought you were saying | 19:40 | |
| weshay | <myoung|ruck> re: current new hash from weekend on master, jobs still running (fs 20 takes forever...full tempest) - but BM OC deploy fails (not root caused yet). previous hash deployed OC but was failing in validate | 19:40 |
| weshay | previous hash deployed OC but was failing in validate | 19:41 |
| weshay | see that | 19:41 |
| weshay | now | 19:41 |
| myoung|ruck | i haven't dived into it yet, nearly done with envD (I think) and rekicking. It's failing on OC deploy here: https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans-121/undercloud/home/stack/overcloud_deploy.log.txt.gz#_2018-05-21_17_53_44 | 19:43 |
| myoung|ruck | ^^ "Error: /Stage[main]/Cinder/Cinder_config[DEFAULT/api_paste_config]/ensure: change from absent to present failed: Could not set 'present' on ensure: No such file or directory - /etc/cinder/cinder.conf at /etc/puppet/modules/cinder/manifests/init.pp:460", | 19:44 |
| myoung|ruck | (well not necessarily failing THERE but that's as far as I've chased that failure) | 19:44 |
| myoung|ruck | thus far | 19:44 |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 19:49 | |
| weshay | sshnaidm|rover, FYI https://review.openstack.org/#/c/569831/2 | 19:49 |
| sshnaidm|rover | weshay, yeah.. exactly was looking at that today, seems like it broke queens, but didn't get why | 19:50 |
| sshnaidm|rover | weshay, but how ovb was passing? | 19:52 |
| * weshay looks at config | 19:54 | |
| weshay | s | 19:54 |
| weshay | sshnaidm|rover, hrm https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml | 19:57 |
| weshay | sshnaidm|rover, so fs001 must just be taking the defaults | 19:57 |
| *** dtrainor has quit IRC | 20:02 | |
| sshnaidm|rover | weshay, ya.. I think adding multinode branchful jobs would be too much)) | 20:03 |
| sshnaidm|rover | weshay, or could we? | 20:04 |
| weshay | oh.. I had the same thought | 20:04 |
| weshay | sshnaidm|rover, I think n-1, n-2 is worth it.. w/ container-multinode | 20:04 |
| weshay | so starting off small | 20:04 |
| weshay | keep what we have | 20:04 |
| weshay | but any change to CI repos kicks container-multinode on master, n-1, n-2 | 20:05 |
| weshay | wdyt? | 20:05 |
| weshay | sshnaidm|rover, this still may have gone undetected | 20:05 |
| weshay | as it kind of did w/ fs001 queens | 20:05 |
| weshay | sshnaidm|rover, see #tripleo | 20:05 |
| weshay | but I still like the idea | 20:05 |
| weshay | if CI is not branched, then we have to cover more in ci | 20:06 |
| sshnaidm|rover | weshay, at least for quickstart | 20:06 |
| weshay | right.. | 20:07 |
| weshay | sshnaidm|rover, I think tripleo-ci, tq,tqe | 20:07 |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 20:14 |
| sshnaidm|rover | weshay, I'll need a few days off next week.. will send you | 20:19 |
| weshay | k np | 20:19 |
| weshay | sshnaidm|rover, hrm.. so looking at zuul | 20:21 |
| weshay | if we override branches in the layout.yml.. maybe that would kick a master, queens, pike? | 20:22 |
| weshay | sshnaidm|rover, or would we have to define duplicate jobs upstream | 20:25 |
| weshay | sshnaidm|rover, ya.. that's what we would have to do | 20:25 |
| sshnaidm|rover | weshay, we can do a template actually.. | 20:26 |
| sshnaidm|rover | weshay, like it's in 3party, but let's try if it's ok at all | 20:26 |
| weshay | k k | 20:26 |
| myoung|ruck | weshay: (quickie) https://code.engineering.redhat.com/gerrit/#/c/139404 (stop eyes bleeding) | 20:30 |
| *** dtrainor has joined #oooq | 20:33 | |
| weshay | sshnaidm|rover, no idea what I'm doing https://review.openstack.org/569855 | 20:34 |
| sshnaidm|rover | weshay, https://review.openstack.org/#/c/569854/ | 20:35 |
| weshay | ah branch-override: 'stable/queens' | 20:35 |
| sshnaidm|rover | weshay, it's running, need to check if it deploys right branch | 20:36 |
| weshay | the doc just sucks https://docs.openstack.org/infra/zuul/user/config.html | 20:36 |
| weshay | sshnaidm|rover, thanks!!! | 20:36 |
| weshay | tripleo-ci-centos-7-containers-multinode-queens | 20:37 |
| weshay | tripleo-ci-centos-7-containers-multinode-pike | 20:37 |
| weshay | sshnaidm|rover, where did you find the branch-override var | 20:38 |
| sshnaidm|rover | weshay, https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run.yaml#L42 | 20:40 |
| weshay | thanks | 20:45 |
| weshay | makes sense | 20:45 |
| myoung|ruck | weshay: FYI envD is back from the dead (or it's long vacation) https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envD-single_nic_vlans/70/consoleFull | 20:51 |
| myoung|ruck | ^^ uc install now | 20:51 |
| *** trown is now known as trown|outtypewww | 20:57 | |
| *** dtrainor has quit IRC | 21:08 | |
| weshay | sshnaidm|rover, no luck | 21:12 |
| weshay | I see --release master on the queens job | 21:12 |
| sshnaidm|rover | weshay, oops, typo | 21:13 |
| weshay | ah nice | 21:16 |
| myoung|ruck | arxcruz: is this a known issue already tracked by an LP / card somewhere/anywhere? https://thirdparty.logs.rdoproject.org/jenkins-periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb-50/console.txt.gz#_2018-05-19_04_53_03_527 | 21:23 |
| * myoung|ruck makes a duplicate potentially | 21:32 | |
| myoung|ruck | arxcruz: fyi https://bugs.launchpad.net/tripleo/+bug/1772522 | 21:44 |
| openstack | Launchpad bug 1772522 in tripleo "tempestmail.py fails in rdo phase 2 jobs" [Medium,Triaged] | 21:44 |
| myoung|ruck | weshay: fyi as expected the fs020 job on new hash is green https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/53/ | 21:45 |
| hubbot | FAILING CHECK JOBS on stable/ocata: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-3nodes- (1 more message) | 22:14 |
| myoung|ruck | weshay: you still here? | 23:13 |
| *** tosky has quit IRC | 23:15 | |
| *** sanjay__u has joined #oooq | 23:26 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!