| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message) | 00:37 |
|---|---|---|
| *** hamzy has joined #oooq | 00:38 | |
| *** vinaykns has joined #oooq | 01:18 | |
| *** dougbtv has quit IRC | 02:26 | |
| *** dougbtv has joined #oooq | 02:30 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci- (1 more message) | 02:37 |
| *** dougbtv has quit IRC | 02:39 | |
| *** openstack has joined #oooq | 02:47 | |
| *** vinaykns has quit IRC | 02:59 | |
| *** udesale has joined #oooq | 03:48 | |
| *** gkadam has joined #oooq | 03:49 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 04:37 |
| *** ykarel|away has joined #oooq | 04:47 | |
| *** jaganathan has joined #oooq | 04:48 | |
| *** jaganathan has quit IRC | 04:49 | |
| *** jaganathan has joined #oooq | 04:49 | |
| *** quiquell has joined #oooq | 05:08 | |
| *** skramaja has joined #oooq | 05:18 | |
| *** ratailor has joined #oooq | 05:30 | |
| *** pgadiya has joined #oooq | 05:35 | |
| *** pgadiya has quit IRC | 05:35 | |
| *** agopi has quit IRC | 05:38 | |
| *** gkadam has quit IRC | 05:39 | |
| quiquell | marios: Good morning sir | 05:46 |
| *** ratailor_ has joined #oooq | 05:47 | |
| *** kopecmartin has joined #oooq | 05:47 | |
| *** links has joined #oooq | 05:48 | |
| *** ratailor has quit IRC | 05:50 | |
| marios | o/ quiquell | 06:03 |
| *** udesale has quit IRC | 06:04 | |
| *** udesale has joined #oooq | 06:04 | |
| *** bogdando has joined #oooq | 06:11 | |
| quiquell | marios: Have to leave in a few to do some paperwork, I wil update with tags problems progress in a few | 06:12 |
| quiquell | marios: So you can help out | 06:12 |
| quiquell | marios: ok ? | 06:12 |
| marios | quiquell: ack | 06:13 |
| quiquell | marios: The basis is that we have break tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades | 06:16 |
| quiquell | marios: So EmilienM Has a fix for it here https://review.openstack.org/#/c/585528/ | 06:16 |
| marios | quiquell: right the tags fix | 06:17 |
| quiquell | marios: But it breaks tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades | 06:20 |
| quiquell | marios: Sorry tripleo-ci-centos-7-scenario000-multinode-oooq-container-<updates> | 06:20 |
| quiquell | marios: Folcos change that remove TAGS totally fix it but it depends on a lot of changes | 06:21 |
| quiquell | marios: So I have put in place a commit that is works in top of tripleo-ci right now https://review.openstack.org/#/c/585964/ | 06:21 |
| quiquell | marios: So task is monitor this last patch to check that it works | 06:22 |
| quiquell | marios: Also in the meantime compare EmilienM patch https://review.openstack.org/#/c/585528/ and folcos patch to check what's the difference https://review.openstack.org/#/c/584508/ | 06:22 |
| quiquell | marios: I have already compare the ansible-playbook calls and they are exactly the same :-( | 06:23 |
| quiquell | marios: I will be around for a few | 06:23 |
| quiquell | marios: Let's see if we can found the issue | 06:23 |
| marios | quiquell: ack. so Folcos top-level change at https://review.openstack.org/#/c/584508/10 fixes it? | 06:24 |
| quiquell | marios: Also I have add a Depends-On on EmilienM Patch to see the logs of the failing task | 06:24 |
| marios | tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates SUCCESS in 1h 58m 18s | 06:24 |
| quiquell | marios: Yep, but we cannot merge it as it is, so I have squash the dependencies into one | 06:24 |
| marios | yeah | 06:24 |
| quiquell | marios: If it works (we have also to check that OVB works are working fine too) then we can dirrectly merge that | 06:25 |
| quiquell | marios: Is this one https://review.openstack.org/#/c/585964/ | 06:25 |
| marios | quiquell: right. well the chain is not horrible | 06:25 |
| marios | quiquell: i mean the first 2 are +2A | 06:25 |
| *** jfrancoa has joined #oooq | 06:25 | |
| marios | quiquell: so we only need 2, https://review.openstack.org/#/c/584465/2 and then folco change | 06:25 |
| quiquell | marios: From the tip of tripleo-ci there are like three before folcos | 06:27 |
| quiquell | marios: One is in the gates and the other two are not | 06:27 |
| quiquell | marios: If we can merge them all soon and fix upgrades problem would be the perfect solution | 06:28 |
| quiquell | marios: If the squashed review works, and the other are not merged we have a side solution | 06:28 |
| marios | quiquell: yeah i know but the first two, periodic/dryrun and your one are +2AA | 06:28 |
| marios | quiquell: so remaining for attention are https://review.openstack.org/#/c/584465/2 and then folco change | 06:29 |
| quiquell | marios: Yep, so it's still possible that we get all of them merged | 06:29 |
| quiquell | marios: Squashed one is a plan B | 06:29 |
| marios | quiquell: yah i don't think we need it if we can merge these | 06:30 |
| quiquell | We can event pu Clean up one after TAGs | 06:30 |
| quiquell | marios: So TAGs one is next one after my +2A | 06:30 |
| quiquell | marios: Agree | 06:30 |
| quiquell | marios: Let's try to review the pending ones so they get +2 and similar, you can fix something directly | 06:31 |
| quiquell | And I think that's all | 06:31 |
| marios | quiquell: ack | 06:32 |
| marios | quiquell: now we just need the a-team music and a van | 06:32 |
| quiquell | marios: Thanks a lot man, sorry I call you | 06:32 |
| marios | quiquell: no worries :) i'll check the reviews in a bit | 06:32 |
| quiquell | marios: Haha you are murdock ? | 06:32 |
| marios | quiquell: y u apologise :) | 06:32 |
| quiquell | marios: Good to have you at my tz | 06:32 |
| quiquell | marios: One more thing, don't want to re-arrange, maybe the clean up is fixing suff up | 06:34 |
| marios | quiquell: sure lets try it like that i think between all of us we should be able to merge it by end of US day | 06:35 |
| quiquell | marios: Yep, maybe some review+fix is needed, not big deal though zuul queues are good at this hour | 06:35 |
| marios | quiquell: but /win 11 | 06:35 |
| quiquell | marios: The failing jobs at gates is high, maybe the queue get big soon http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 06:36 |
| quiquell | marios: 18 is a lot | 06:36 |
| quiquell | marios: But that's from yesterday | 06:37 |
| quiquell | marios: Maybe is fixed today | 06:37 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 06:37 |
| quiquell | sshnaidm|afk: https://review.rdoproject.org/r/#/c/15026/ | 06:43 |
| *** ratailor_ has quit IRC | 06:43 | |
| *** ratailor_ has joined #oooq | 06:44 | |
| *** holser_ has joined #oooq | 06:56 | |
| *** florianf has joined #oooq | 07:01 | |
| *** amoralej|off is now known as amoralej | 07:01 | |
| quiquell | marios: Have recheck clean up tag and Replate TAGs folco's change | 07:03 |
| quiquell | marios: The replace TAGs has some noice, I forgot to change Change-Id in the squashed one, now the review is back to normal | 07:04 |
| *** gkadam has joined #oooq | 07:04 | |
| *** saneax has joined #oooq | 07:20 | |
| *** kopecmartin has quit IRC | 07:22 | |
| quiquell | marios: One merged https://review.openstack.org/#/c/582387 | 07:27 |
| *** ccamacho has joined #oooq | 07:27 | |
| marios | quiquell: nice | 07:27 |
| quiquell | marios: Why this is not in the gates ? https://review.openstack.org/#/c/582466 | 07:27 |
| marios | quiquell: hm not sure it says "cannot merge" | 07:28 |
| marios | quiquell: but it can't be rebase since it is in series | 07:28 |
| quiquell | marios: But it wil lose the +2A | 07:28 |
| marios | yeah that not the end of world we can add panda|rover|off sshnaidm|afk around too | 07:29 |
| marios | quiquell: but lets think about it i don't think it is rebase/shouldn't be | 07:29 |
| quiquell | marios: We don't have to rebase it ? | 07:30 |
| quiquell | marios: I mean, I don't see any cannot merge message | 07:30 |
| marios | quiquell: not sure | 07:30 |
| marios | quiquell: i see it | 07:30 |
| quiquell | marios: where ? | 07:30 |
| marios | quiquell: Strategy Merge if Necessary | 07:30 |
| marios | Cannot Merge https://review.openstack.org/#/c/582466/ | 07:31 |
| quiquell | marios: Damn... where ? I don't see it | 07:31 |
| quiquell | Ahhh ok Merge IIf Necessary | 07:32 |
| quiquell | But I don't see Cannot Merge | 07:32 |
| marios | quiquell: srsly man | 07:33 |
| quiquell | marios: I am ofuscated... I only see the Merge if Necessary | 07:33 |
| marios | quiquell: spot the "Cannot Merge" maybe easier here https://i.imgur.com/r5PoAMh.png | 07:34 |
| marios | :D | 07:34 |
| quiquell | I don't have that.-. | 07:34 |
| marios | quiquell: refresh? | 07:34 |
| quiquell | marios: noper is not there | 07:35 |
| marios | wut | 07:35 |
| quiquell | Maybe is the tamper monkey script | 07:35 |
| *** zoli|gone is now known as zoli | 07:35 | |
| *** chem has quit IRC | 07:36 | |
| *** tosky has joined #oooq | 07:38 | |
| marios | quiquell: lemme try +A but don't think it will do anything | 07:38 |
| quiquell | marios: Firefox show it to me, chrome don't ... | 07:38 |
| marios | quiquell: waaat | 07:38 |
| quiquell | yep | 07:38 |
| quiquell | marios: Now!!! after your +A | 07:38 |
| quiquell | I think something get cached at chrome | 07:39 |
| quiquell | marios: OK going to rebase | 07:39 |
| quiquell | Cool only +1w | 07:39 |
| quiquell | needed | 07:39 |
| quiquell | rebased | 07:39 |
| quiquell | marios: Add a +1w there | 07:40 |
| marios | quiquell: ok | 07:40 |
| marios | quiquell: it kept the votes which was nice of it | 07:41 |
| marios | :) | 07:41 |
| quiquell | marios: Don't know on what depends, sometimes it remove it | 07:41 |
| quiquell | marios: Maybe depends if there are changed at the patches files | 07:41 |
| marios | quiquell: yah i expected them removed | 07:41 |
| quiquell | marios: Or if it's a ff merge | 07:41 |
| quiquell | marios: Maybe gerrit is being nice with us, it feels the pain | 07:42 |
| quiquell | marios: Have to leave now | 07:42 |
| marios | quiquell: ack | 07:42 |
| marios | ttyl | 07:42 |
| quiquell | marios: You are in control of this now, will be back in a few hours | 07:42 |
| *** quiquell is now known as quiquell|bbl | 07:42 | |
| marios | quiquell: ;) well we just need some more votes on https://review.openstack.org/#/c/584465/2 | 07:42 |
| marios | quiquell|bbl: and https://review.openstack.org/#/c/584508/10 | 07:42 |
| marios | panda|rover|off: sshnaidm|afk ** please ^^^ | 07:43 |
| marios | to unbreak the upgrades tags | 07:43 |
| quiquell|bbl | the pending reviews are | 07:43 |
| quiquell|bbl | https://review.openstack.org/#/c/584465/ | 07:43 |
| quiquell|bbl | https://review.openstack.org/#/c/584508/10 | 07:43 |
| quiquell|bbl | https://review.openstack.org/#/c/584508/ | 07:43 |
| quiquell|bbl | I mean | 07:43 |
| *** brault has joined #oooq | 07:44 | |
| marios | quiquell|bbl: thats what i said :) | 07:44 |
| marios | quiquell|bbl: but since we're spamming | 07:44 |
| marios | 10:42 < marios> quiquell: ;) well we just need some more votes on https://review.openstack.org/#/c/584465/2 | 07:44 |
| quiquell|bbl | Ahh ok yes jeje | 07:44 |
| marios | 10:42 < marios> quiquell|bbl: and https://review.openstack.org/#/c/584508/10 | 07:44 |
| marios | and you said 10:43 < quiquell|bbl> https://review.openstack.org/#/c/584465/ | 07:44 |
| marios | 10:43 < quiquell|bbl> https://review.openstack.org/#/c/584508/10 | 07:44 |
| marios | panda sshnaidm|afk please reviews ^^^ | 07:44 |
| marios | :D | 07:44 |
| quiquell|bbl | marios: ack, see you in a frew | 07:45 |
| quiquell|bbl | few | 07:45 |
| marios | o/ | 07:45 |
| quiquell|bbl | marios: I expect not much tabasco for me " | 07:45 |
| marios | quiquell|bbl: enchilada cannot be without tabasco | 07:45 |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| *** quiquell|bbl has quit IRC | 07:49 | |
| *** quiquell has joined #oooq | 08:02 | |
| quiquell | marios: Have check before leave | 08:02 |
| quiquell | marios: EmilienM Patch is passing now the updates :-/, http://logs.openstack.org/28/585528/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/2e91dbe/ | 08:03 |
| quiquell | marios: https://review.openstack.org/#/c/585528/ | 08:03 |
| quiquell | marios: I really leave now | 08:03 |
| *** quiquell has quit IRC | 08:03 | |
| *** zoli is now known as zoli|dentist | 08:25 | |
| *** kopecmartin has joined #oooq | 08:30 | |
| *** quiquell has joined #oooq | 08:34 | |
| quiquell | Marios: emilien fix works beacause the Depens-On it forces to get latest tht | 08:34 |
| quiquell | Marios: the problem have to be at tht and we have just bad luck | 08:35 |
| quiquell | Marios: dlrn is pointing yo a broken one | 08:35 |
| quiquell | After our merge they sure tht has a false positive for updates | 08:36 |
| quiquell | So people have being merging broken stuff that appear after emilien fix | 08:37 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: (1 more message) | 08:37 |
| *** ykarel|away has quit IRC | 08:38 | |
| *** ykarel| has joined #oooq | 08:38 | |
| quiquell | We can check the difference between latest tht and DLRN one | 08:39 |
| *** chem has joined #oooq | 08:41 | |
| quiquell | I will be on read only mode in and out | 08:42 |
| *** tesseract has joined #oooq | 08:42 | |
| *** udesale_ has joined #oooq | 08:44 | |
| quiquell | This is last DLRN tht | 08:46 |
| quiquell | http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9cea1b9c782610587d265cd6b74a2139dad8d7fb | 08:46 |
| *** udesale has quit IRC | 08:46 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:51 | |
| sshnaidm | https://review.openstack.org/#/c/584508/ requires a rebase | 08:51 |
| quiquell | Done, also rebases the parent | 08:54 |
| quiquell | This is the tht where DLRN is pointing http://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=db72fcd2e480694b4f21482f118fab28485acbea | 08:58 |
| quiquell | Tht will be fixed in the next periodic | 09:01 |
| *** florianf has quit IRC | 09:06 | |
| quiquell | sshnaidm: have pinned the version of grafana you can merge the review | 09:11 |
| panda|rover|off | quiquell: is the depends-on for https://review.openstack.org/585528 going to be merged ? | 09:13 |
| quiquell | Panda: we are trying folcos changes as they fixed it too | 09:13 |
| quiquell | I still dont know why updates was not failing at folcos changes, tht version should be the same... | 09:14 |
| quiquell | Feel like we were oculting an issue after our merge | 09:15 |
| panda|rover|off | quiquell: which folco's change ? https://review.openstack.org/#/c/584508 ? | 09:16 |
| quiquell | Yep | 09:16 |
| quiquell | But since it doesnt hace a depends on it should have use the same tht version... | 09:17 |
| quiquell | So still dont know why | 09:17 |
| quiquell | Panda maybe you can dig a little about it | 09:17 |
| panda|rover|off | quiquell: update me on the status | 09:19 |
| quiquell | Ok, so we broke nv upgrade job | 09:20 |
| panda|rover|off | quiquell: bj/gcerami | 09:20 |
| quiquell | Panda cannot do bj now | 09:20 |
| quiquell | Waiting for some paperwork at the administration | 09:21 |
| panda|rover|off | oh, the capital letters, you'r on your phone | 09:21 |
| panda|rover|off | ok | 09:21 |
| quiquell | Yep hehe thats a good signal | 09:21 |
| *** d0ugal has joined #oooq | 09:22 | |
| quiquell | The thing is emilien fix fails at updates job and this is a voting job | 09:22 |
| quiquell | Was unrelated to our changes | 09:22 |
| quiquell | So after i add a depends on for debugging purposes | 09:23 |
| quiquell | It passes | 09:23 |
| quiquell | It forces to get the latest tht | 09:23 |
| *** ratailor_ has quit IRC | 09:23 | |
| quiquell | But then i dont know why folcos change is not failing at updates | 09:24 |
| quiquell | Maybe was using and old tht version | 09:24 |
| quiquell | So we are tryng to get merged the new enchilada until last folcos change | 09:24 |
| quiquell | Also, we have to get the periodics get the latest tht | 09:25 |
| quiquell | Panda: that its all | 09:25 |
| ssbarnea | fyi, i will be away for a couple of hours, doctor appt. | 09:29 |
| quiquell | ssbnarnea: ack hope is nothing | 09:30 |
| quiquell | panda: this is the bug https://bugs.launchpad.net/tripleo/+bug/1783399 | 09:30 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 09:30 |
| panda|rover|off | yes I know | 09:31 |
| panda|rover|off | quiquell: Why there are no tests result in folco's change | 09:35 |
| quiquell | We did a rebase | 09:37 |
| quiquell | Look in the comments in the old ones | 09:37 |
| quiquell | Panda: we need to find if it was a tht problem | 09:41 |
| *** jfrancoa has quit IRC | 09:42 | |
| *** jfrancoa has joined #oooq | 09:42 | |
| panda|rover|off | jfrancoa: how do I understand what went wrong here ? http://logs.openstack.org/28/585528/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/e1f371f/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz | 09:48 |
| *** zoli|dentist is now known as zoli | 09:50 | |
| jfrancoa | panda|rover|off: no clue, I saw it yesterday too and tried to debug further but it doesn't give much info. And as the previous tasks are running with no_log, you get even less context | 09:50 |
| jfrancoa | panda|rover|off: I saw that quiquell ran the same job with a dependency which set the no_log to False | 09:51 |
| panda|rover|off | jfrancoa: and that passed | 09:51 |
| quiquell | Yep its taking newer tht i suspect | 09:51 |
| jfrancoa | panda|rover|off: however, it looks like after his recheck it's working fine: http://logs.openstack.org/28/585528/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/2e91dbe/ | 09:51 |
| quiquell | 2018-07-25 16:58:07 | u'TASK [Set docker_startup_configs_with_default fact | 09:51 |
| panda|rover|off | jfrancoa: so the job is shy, if you threaten to show its inside, it will decide to pass | 09:51 |
| quiquell | Hehe | 09:52 |
| jfrancoa | panda|rover|off: there should be a log from the ansible execution in /var/log/mistral, I'm going to check if it has further info | 09:52 |
| *** d0ugal has quit IRC | 09:52 | |
| quiquell | Panda we have to diff DLRN hash with latest hash at tht | 09:52 |
| panda|rover|off | quiquell: if that's the case, something is really wrong, we should not need a depends-on, to get the latest code in tht | 09:52 |
| quiquell | Panda tht is taken from dlrn | 09:53 |
| quiquell | And they are promoted by periodic | 09:53 |
| quiquell | A depends on forces build-test-packages | 09:53 |
| quiquell | That forces to use latest thr | 09:53 |
| quiquell | Tht | 09:54 |
| jfrancoa | panda|rover|off: nah, same shy result http://logs.openstack.org/28/585528/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/e1f371f/logs/undercloud/var/log/containers/mistral/package_update.log.txt.gz#_2018-07-25_16_58_05_539 | 09:54 |
| panda|rover|off | quiquell: tripleo-heat-templates in our chage is always taken from the latest delorean package, even if it's not promoted | 09:56 |
| panda|rover|off | s/chage/checks | 09:56 |
| panda|rover|off | quiquell: we are taking it from delorean-current | 09:56 |
| quiquell | But maybe there were no dlrn package yet | 09:56 |
| quiquell | We can do a recheck wirhout the depends on | 09:57 |
| quiquell | Depends on really forces it | 09:57 |
| panda|rover|off | quiquell: dlrn creates a new hash automatically whenever a new merge in any of the project is detected | 09:57 |
| quiquell | Timing issue ? | 09:57 |
| panda|rover|off | jfrancoa: what does that mean ? | 09:58 |
| panda|rover|off | quiquell: anyway, the hashes in the two jobs differ | 10:01 |
| *** quiquell has quit IRC | 10:01 | |
| *** quiquell has joined #oooq | 10:03 | |
| quiquell | panda|rover|off: I am back | 10:03 |
| quiquell | Do you want some bj sync ? | 10:03 |
| chandankumar | arxcruz: it this review good to go https://review.openstack.org/#/c/580384/ ? | 10:04 |
| panda|rover|off | quiquell: remove the depends-on | 10:04 |
| quiquell | panda|rover|off: Now https://review.openstack.org/#/c/582466/ this is passing upgrades... holy molly | 10:04 |
| arxcruz | chandankumar: yup | 10:04 |
| jfrancoa | panda|rover|off: I don't know what the error is about, with such a short log. It's in deploy-step-tasks https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L77 but there are really no info to deduce where the problem is. maybe marios knows anything about it? | 10:05 |
| chandankumar | panda|rover|off: bogdando please have a look at this review https://review.openstack.org/#/c/580384/ and https://review.openstack.org/#/c/577780/ | 10:05 |
| panda|rover|off | quiquell: what's that ? | 10:06 |
| panda|rover|off | jfrancoa: there's some analysis in the bug | 10:06 |
| panda|rover|off | jfrancoa: https://bugs.launchpad.net/tripleo/+bug/1783399 | 10:06 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 10:06 |
| quiquell | panda|rover|off: That is the latest from our enchilada | 10:06 |
| panda|rover|off | jfrancoa: some errors in mistral maybe ? | 10:07 |
| panda|rover|off | quiquell: the problems was never that the upgrades didn't pass, it's that they were not doing the upgrade | 10:07 |
| quiquell | panda|rover|off: And upgrades job is passing :-/ but upgrades tags are not there | 10:07 |
| quiquell | panda|rover|off: Yep, now upgrades pass without doing upgrades :-/ | 10:07 |
| *** udesale__ has joined #oooq | 10:08 | |
| panda|rover|off | so now we merged this, and we need to rebase all the fixes on top of this ? | 10:08 |
| marios | jfrancoa: o/ which job/issue? | 10:08 |
| quiquell | panda|rover|off: They depend on this from the beginning | 10:08 |
| quiquell | panda|rover|off: No problem with that | 10:08 |
| marios | jfrancoa: nm (upgrades tags) | 10:09 |
| marios | quiquell: panda|rover|off you mean we have a problem even after https://review.openstack.org/#/c/584508/11 ? | 10:09 |
| *** apetrich has quit IRC | 10:09 | |
| panda|rover|off | quiquell: marios please bj | 10:10 |
| panda|rover|off | bj/gcerami | 10:10 |
| marios | panda|rover|off: ack gimme sec | 10:10 |
| quiquell | marios: nop | 10:10 |
| quiquell | panda|rover|off: Going to connect there too | 10:10 |
| quiquell | ok ? | 10:10 |
| *** udesale_ has quit IRC | 10:10 | |
| jfrancoa | marios: yes, right. The thing is that Emilien's patch was failing yesterday for few jobs (minor updates and some others), and the minor updates one was failing with this error: http://logs.openstack.org/28/585528/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/e1f371f/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz but it's really hard to know what went wrong | 10:10 |
| panda|rover|off | quiquell: yes please | 10:11 |
| panda|rover|off | jfrancoa: you have time to join too ? | 10:11 |
| jfrancoa | panda: sure, why not. joining | 10:11 |
| marios | jfrancoa: ack but https://review.openstack.org/#/c/584508/11 apparently fixes the issue (and if you see diff it obsoletes EmilienM change) | 10:12 |
| marios | assuming it works :) | 10:12 |
| *** sshnaidm is now known as sshnaidm|afk | 10:13 | |
| panda|rover|off | marios: https://bugs.launchpad.net/tripleo/+bug/1783399 | 10:23 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 10:23 |
| chandankumar | kopecmartin: arxcruz please have a look at the epic card https://trello.com/c/yAnDETzJ/878-sprint-17-tempest-clear-technical-debts | 10:24 |
| chandankumar | since tomorrow I will be not available we are planning to do sprint planning on monday if it is ok? | 10:24 |
| chandankumar | kopecmartin: arxcruz if anything is not clear feel free to comment on the card | 10:28 |
| *** d0ugal has joined #oooq | 10:33 | |
| quiquell | So going to rebase, EmilienM Changes in top of last one near gates | 10:33 |
| *** d0ugal has quit IRC | 10:34 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 10:37 |
| quiquell | panda|rover|off, marios, jfrancoa: just missing the +1w https://review.openstack.org/#/c/585528 | 10:39 |
| quiquell | :-) | 10:39 |
| panda|rover|off | quiquell: have you removed the depends-on ? | 10:39 |
| quiquell | panda|rover|off: yep | 10:40 |
| panda|rover|off | quiquell: we need to understand if all passes before +W | 10:40 |
| quiquell | panda|rover|off: sure, agree just saying that it has already a 2 +2 | 10:40 |
| panda|rover|off | quiquell: ok | 10:41 |
| quiquell | panda|rover|off: Looking at the releases.sh file from failing update job this is the hash 959e1d7f755ee681b6f23b498d262a9e4dd6326f_4cbb1814 | 10:41 |
| quiquell | panda|rover|off: And this is the repo https://trunk.rdoproject.org/centos7-master-head/95/9e/959e1d7f755ee681b6f23b498d262a9e4dd6326f_4cbb1814/ | 10:44 |
| quiquell | again 20th of july | 10:44 |
| quiquell | panda|rover|off: the tht we are using have to be that or I am missing something ? | 10:44 |
| quiquell | Jul 25 15:19:14 Installed: openstack-tripleo-heat-templates-9.0.0-0.20180725034409.db72fcd.el7.noarch is the version installed | 10:49 |
| quiquell | panda|rover|off: So it's 25th of fuly | 10:50 |
| quiquell | passing updates job uses Jul 26 06:05:27 Installed: openstack-tripleo-heat-templates-9.0.0-0.20180726032504.bc64f12.el7.noarch | 10:51 |
| quiquell | So it's newer | 10:51 |
| quiquell | jfrancoa, marios: I think this is the fix at THT https://github.com/openstack/tripleo-heat-templates/commit/b5bb72987df0949431b2c2397a5d539798531542 | 10:52 |
| quiquell | Is the only commit in the middle of fail and pass update job | 10:52 |
| quiquell | But another stuf could have being updates, we have the gate broken after our merge :-/ | 10:53 |
| *** brault_ has joined #oooq | 10:59 | |
| *** brault has quit IRC | 11:00 | |
| *** amoralej is now known as amoralej|lunch | 11:01 | |
| *** quiquell is now known as quiquell|bbl | 11:05 | |
| *** skramaja_ has joined #oooq | 11:13 | |
| *** skramaja has quit IRC | 11:14 | |
| *** skramaja_ is now known as skramaja | 11:17 | |
| *** udesale__ has quit IRC | 11:25 | |
| sshnaidm|afk | quiquell|bbl, seems like we need to separate jobs stats by rdo cloud and tripleo ci. Currently all problems of rfo cloud hide problems of tripleo ci and break statistics with 100% failures.. | 11:26 |
| quiquell|bbl | sshnaidm|afk: ack, a review coming in a few | 11:30 |
| sshnaidm|afk | quiquell|bbl, let's make then 2 upper rows, one with upstream and other with rdo cloud | 11:31 |
| sshnaidm|afk | quiquell|bbl, how do you extract json? via "sharing"? | 11:31 |
| EmilienM | quiquell|bbl: hey, so what's up? | 11:32 |
| *** quiquell|bbl is now known as quiquell | 11:32 | |
| quiquell | EmilienM: Hello there | 11:32 |
| quiquell | EmilienM: At the end update job issue was non related, we have rebase your change in top of the one in the gates | 11:32 |
| quiquell | EmilienM: Possible suspect THT, as we were giving false possitives at updates jobs, checks were not checking | 11:33 |
| EmilienM | quiquell: ok so my patch should work pass the check jobs this time? | 11:34 |
| quiquell | EmilienM: We think so, after adding a Depends-On on it, update jobs was passing | 11:34 |
| quiquell | EmilienM: Now we are checking the rebase | 11:34 |
| EmilienM | cool! | 11:35 |
| EmilienM | thanks | 11:35 |
| quiquell | EmilienM: Will ping you with anything, sorry about the defect | 11:35 |
| quiquell | sshnaidm|afk: Going to add two singlestats one with Upstream and one with RDO | 11:36 |
| *** panda|rover|off is now known as panda|rover | 11:37 | |
| *** zoli is now known as zoli|lunch | 11:38 | |
| EmilienM | quiquell: it happens ;-) | 11:42 |
| *** agopi has joined #oooq | 11:45 | |
| quiquell | sshnaidm|afk: https://review.rdoproject.org/r/#/c/15032/, you can review it with just a docker-compose up over the change | 11:50 |
| quiquell | sshnaidm|afk: It will be at localhost | 11:50 |
| quiquell | sshnaidm|afk: Can you open this ? http://10.36.116.72 | 11:51 |
| sshnaidm|afk | quiquell, yep | 11:51 |
| quiquell | sshnaidm|afk: that's my laptop | 11:51 |
| sshnaidm|afk | quiquell, how about something like that? http://38.145.35.97/d/cEEjGFFmz/cockpit?orgId=1 | 11:51 |
| quiquell | sshnaidm|afk: Just two things | 11:52 |
| quiquell | sshnaidm|afk: Can I do a little change over it ? | 11:53 |
| quiquell | sshnaidm|afk: I like it, just a layout change | 11:53 |
| sshnaidm|afk | quiquell, of course, it's testing | 11:53 |
| quiquell | sshnaidm|afk: admin password ? | 11:53 |
| sshnaidm|afk | quiquell, but noop upstream jobs still have legacy jobs.. | 11:54 |
| sshnaidm|afk | quiquell, the same as in tripleo-infra | 11:54 |
| sshnaidm|afk | quiquell, tell me if you log in | 11:54 |
| quiquell | Ok, I am in | 11:56 |
| sshnaidm|afk | quiquell, and I filtered by name "/legacy-*/ which is not correct, names will change.. | 11:56 |
| quiquell | sshnaidm|afk: "type" = 'upstream' | 11:56 |
| sshnaidm|afk | quiquell, need to have more strict differentiation | 11:56 |
| quiquell | "type" = 'rdo' | 11:56 |
| sshnaidm|afk | ok | 11:56 |
| quiquell | Let me change this and the layout | 11:56 |
| quiquell | sshnaidm|afk: But like it better the two rows | 11:56 |
| sshnaidm|afk | quiquell, yeah, also want to add table of periodic jobs only (like we have table of noop and gate failures) | 11:57 |
| quiquell | Yep that's a good idea, periocis are the gate version of RDO | 11:57 |
| *** ccamacho1 has joined #oooq | 11:58 | |
| sshnaidm|afk | quiquell, so on one row will be noop upstream and gates, on other - noop rdo and periodics | 11:58 |
| *** ccamacho has quit IRC | 11:58 | |
| quiquell | sshnaidm|afk: Make sense totally | 11:58 |
| sshnaidm|afk | so we'll have also separate rows for different clouds | 11:58 |
| quiquell | sshnaidm|afk: The periodicity of a job in zuul is configured at pipeline or at job ? | 12:01 |
| sshnaidm|afk | quiquell, pipeline | 12:01 |
| sshnaidm|afk | openstack-periodic and openstack-periodic-24hr | 12:02 |
| quiquell | sshnaidm|afk: Going to filter by pipeline more thatn job_name | 12:03 |
| sshnaidm|afk | quiquell, yeah, jobs names change | 12:04 |
| *** rfolco|ruck|off is now known as rfolco|ruck | 12:05 | |
| quiquell | sshnaidm|afk: You have change it directly in the dashboard-ci ? | 12:07 |
| quiquell | sshnaidm|afk: Going to revert dashboard-ci to the previous version | 12:08 |
| quiquell | Until we get right the other | 12:08 |
| *** zoli|lunch is now known as zoli | 12:11 | |
| sshnaidm|afk | quiquell, I didn't save anything in dashboard-ci | 12:13 |
| sshnaidm|afk | quiquell, that was my testing vm | 12:14 |
| *** hamzy has quit IRC | 12:14 | |
| quiquell | sshnaidm|afk: ok | 12:14 |
| *** hamzy has joined #oooq | 12:19 | |
| quiquell | sshnaidm|afk: Check it now, I have fix the queries | 12:24 |
| quiquell | sshnaidm|afk: And also add periodic table | 12:24 |
| quiquell | sshnaidm|afk: http://38.145.35.97/d/cEEjGFFmz/cockpit?panelId=61&orgId=1 | 12:24 |
| sshnaidm|afk | quiquell, I see it in dashboard-ci too. I'm just not sure my vm has latest changes, better to try on dashboard-ci | 12:28 |
| *** rlandy has joined #oooq | 12:28 | |
| quiquell | sshnaidm|afk: Sure is not the same vm ? | 12:29 |
| rlandy | rfolco: ruck: hello | 12:29 |
| rlandy | rfolco|ruck, | 12:29 |
| quiquell | I have revert them i can revert it again | 12:29 |
| rfolco|ruck | rlandy, o/ | 12:29 |
| quiquell | sshnaidm|afk: Is the same IP... | 12:30 |
| quiquell | sshnaidm|afk: dashboard-ci.tripleo.org is 38.145.35.97 | 12:30 |
| sshnaidm|afk | quiquell, f$%k | 12:30 |
| quiquell | sshnaidm|afk: :-/ | 12:30 |
| sshnaidm|afk | quiquell, I was sure it's my vm | 12:30 |
| quiquell | sshnaidm|afk: no problem, is that in 30 minutes we have the planning and I was going to show it, going to revert this ok ? | 12:31 |
| sshnaidm|afk | quiquell, ok, so we are testing on production :D | 12:31 |
| sshnaidm|afk | quiquell, sure | 12:31 |
| quiquell | sshnaidm|afk: Like big bosses :-) | 12:31 |
| quiquell | sshnaidm|afk: Good thing about grafana it has a "versions" option so you can revert from it | 12:31 |
| sshnaidm|afk | quiquell, but why to revert if you did it already? | 12:32 |
| quiquell | sshnaidm|afk: To be sure it's working ? | 12:32 |
| rlandy | marios: hi | 12:32 |
| rlandy | marios: need to pick your updates brain | 12:32 |
| quiquell | sshnaidm|afk: If you are ok with the changes, I will import them to code | 12:32 |
| marios | rlandy: just joined a call sory | 12:33 |
| marios | sorry | 12:33 |
| marios | rlandy: talk in a bit/after planning | 12:33 |
| sshnaidm|afk | quiquell, yeah, totally | 12:33 |
| sshnaidm|afk | quiquell, and all the rest after demo | 12:34 |
| rlandy | marios: ok - one question ... (pls answer when you have time)... this file ... https://logs.rdoproject.org/71/584771/14/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/c796079/logs/undercloud/var/lib/mistral/5cf4bb48-2415-4575-95ea-7459eb55929e/Controller/docker_config.yaml.txt.gz - look at the last start_irder - does that look to be in the right place to you? | 12:34 |
| *** amoralej|lunch is now known as amoralej | 12:35 | |
| quiquell | sshnaidm|afk: https://review.rdoproject.org/r/15032 | 12:36 |
| *** quiquell is now known as quiquell|lunch | 12:37 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 12:37 |
| rlandy | quiquell: hi - newbie dashboard question | 12:37 |
| rlandy | quiquell|lunch: oh - when you get back from lunch, pls ^^ | 12:38 |
| sshnaidm|afk | quiquell|lunch, will it build container too? | 12:38 |
| sshnaidm|afk | rlandy, ask, maybe I can help | 12:39 |
| quiquell|lunch | sshnaidm|afk: Yep, all is prepare for it, don't wory | 12:40 |
| quiquell|lunch | sshnaidm|afk: Just merge the commit | 12:40 |
| quiquell|lunch | sshnaidm|afk: For sure it will blow up before show it to the team :-P | 12:40 |
| quiquell|lunch | rlandy: shoot, will answer it | 12:40 |
| sshnaidm|afk | quiquell|lunch, no, no way :D | 12:41 |
| quiquell|lunch | sshnaidm|afk: Let's merge it after :-) | 12:42 |
| rlandy | quiquell|lunch: not urgent - but I was wondering how I could trace a failure/job history - of that was possible. I can wait until you demo | 12:42 |
| quiquell|lunch | rlandy: We have an exploration dashboard http://dashboard-ci.tripleo.org/d/-UEjGKFmz/exploration?orgId=1 | 12:42 |
| quiquell|lunch | rlandy: click on influxdb filter you can filter all the fields there | 12:42 |
| rlandy | quiquell|lunch: cool - that's what I am looking for | 12:43 |
| rlandy | oops - Sorry for the inconvenience | 12:43 |
| rlandy | Please go back to your home dashboard and try again. | 12:43 |
| quiquell|lunch | rlandy: Also you can shere the specific search | 12:43 |
| rlandy | 404 | 12:43 |
| quiquell|lunch | rlandy: for example http://dashboard-ci.tripleo.org/d/-UEjGKFmz/exploration?orgId=1&var-influxdb_filter=change%7C%3D%7C572312&var-influxdb_filter=pipeline%7C%3D%7Copenstack-check | 12:44 |
| quiquell|lunch | rlandy: the search is in the url so you can share it | 12:44 |
| sshnaidm|afk | rlandy, Influxdb filter is in top left corner | 12:44 |
| sshnaidm|afk | quiquell|lunch, btw, look what I found: https://docs.ansible.com/ansible/latest/plugins/callback/grafana_annotations.html | 12:46 |
| *** links has quit IRC | 12:46 | |
| rfolco|ruck | quiquell|lunch, I don't see a depends-on at https://review.openstack.org/#/c/585528 | 12:46 |
| sshnaidm|afk | quiquell|lunch, I don't know what it can do, but seems promising..\ | 12:46 |
| rlandy | quiquell|lunch: thanks - will poke around | 12:46 |
| * rlandy is very attached to job histories | 12:47 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:48 | |
| myoung | o/ | 12:51 |
| *** quiquell|lunch is now known as quiquell | 12:51 | |
| quiquell | sshnaidm: Was thinking that maybe we are doing too much at collect logs regarding grafana, we can move that offline | 12:52 |
| quiquell | sshnaidm: And reduce time jobs a little | 12:52 |
| quiquell | sshnaidm: Thats pull vs push :-) | 12:52 |
| sshnaidm | quiquell, yeah, need to think about it | 12:53 |
| quiquell | sshnaidm: yep both options have tradeoffs | 12:53 |
| sshnaidm | quiquell, actually we already do infludb sending with pull, just making files in the job | 12:53 |
| sshnaidm | quiquell, then a script pull these files | 12:53 |
| quiquell | sshnaidm: Just generate ara jsons at jobs, but pulling them from telegraf ? | 12:54 |
| quiquell | sshnaidm: Right now is injecting to influxdb at collect logs ? | 12:54 |
| quiquell | sshnaidm: Also graphite is there I think | 12:54 |
| *** saneax has quit IRC | 12:55 | |
| quiquell | sshnaidm: We can even generate the json offline, just getting the ansible.sqlite | 12:55 |
| EmilienM | quiquell: https://review.openstack.org/#/c/585528/ failing on updates, see in zuul | 12:55 |
| EmilienM | http://logs.openstack.org/28/585528/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8a2e24a/ ---- the logs | 12:55 |
| sshnaidm | quiquell, we make files in collect logs, and then we pull these files with a script which sends them to influxdb.. This is because ports of influxdb in rdo are opened only to rdo cloud.. | 12:55 |
| panda|rover | quiquell: when we first debated pull vs push, we realized that pull approach will take time and should be designed well | 12:55 |
| EmilienM | :( | 12:55 |
| *** saneax has joined #oooq | 12:55 | |
| panda|rover | /o\ | 12:55 |
| quiquell | EmilienM: http://logs.openstack.org/28/585528/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8a2e24a/job-output.txt.gz#_2018-07-26_11_19_27_439555 | 12:56 |
| quiquell | EmilienM: TL;DR ssh: connect to host 10.4.70.63 port 22: No route to host | 12:56 |
| quiquell | EmilienM: Going to recheck | 12:56 |
| *** kopecmartin has quit IRC | 12:56 | |
| *** saneax has quit IRC | 12:56 | |
| panda|rover | yep | 12:57 |
| quiquell | EmilienM: upgrades is failing too for the same reason, doesn't make sense | 12:58 |
| *** saneax has joined #oooq | 12:58 | |
| quiquell | Ups my recheck was there already | 12:59 |
| panda|rover | quiquell: is this change using the the new bootstrap method ? | 12:59 |
| quiquell | panda|rover: nope | 13:00 |
| quiquell | panda|rover: toci_gate_test.sh:262): subnodes_scp_deploy_env | 13:00 |
| *** sanjayu_ has joined #oooq | 13:00 | |
| rasca | myoung, hi there! I got an alert about a "followup: quickstart older version and upgrade support" meeting, but I don't see anyone on the call, is this topic still valid? | 13:01 |
| *** ykarel| has quit IRC | 13:01 | |
| quiquell | EmilienM, panda|rover: maybe I broke something with the rebase | 13:01 |
| myoung | rasca: i think not, we're in sprint planning so I can't attend. we haven't had a specific agenda for some time, but weshay_PTO still owns the meeting so I can't cancel | 13:02 |
| myoung | o/ morning folks, sprint planning starts now | 13:02 |
| panda|rover | quiquell: the other multinode jobs are workingf | 13:02 |
| rasca | myoung, ack | 13:02 |
| *** ykarel| has joined #oooq | 13:03 | |
| *** saneax has quit IRC | 13:03 | |
| quiquell | panda|rover: Think so https://zuul.openstack.org/stream.html?uuid=0a3451b7c8cb4783bbd87fe455e07bb6&logfile=console.log | 13:04 |
| quiquell | panda|rover: that's tripleo-ci-centos-7-containers-multinode | 13:04 |
| panda|rover | yep | 13:04 |
| quiquell | also multinode queens and pike looks like working too | 13:04 |
| quiquell | Maybe I fucked it up with the featuresets | 13:05 |
| EmilienM | folks sorry to ask again but... can we revert what rfolco|ruck did | 13:06 |
| EmilienM | and maybe rework on these patches | 13:07 |
| EmilienM | can we stash the changes that we merged and revert in one patch | 13:07 |
| rfolco|ruck | EmilienM, rdo jobs fail the same way, I don't know how reverting it will fix updates job | 13:07 |
| EmilienM | rfolco|ruck: OVB jobs you mean? | 13:08 |
| marios | rlandy: i will have a look after our call | 13:08 |
| rfolco|ruck | EmilienM, RDO 3rd party CI -> legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master | 13:09 |
| rfolco|ruck | EmilienM, I agree I broke updates TAGS but the error in overcloud controller updates (docker) seems to be unrelated. | 13:10 |
| quiquell | rfolco|ruck, EmilienM: Looks the issue go through the gates bcause we broke them | 13:11 |
| EmilienM | what's the problem with rdo jobs, what's the trace | 13:11 |
| rfolco|ruck | https://logs.rdoproject.org/28/585528/2/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/de300dc/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz#_2018-07-25_09_59_56 | 13:12 |
| *** ykarel| has quit IRC | 13:12 | |
| rfolco|ruck | EmilienM, correct me if I am wrong... but this is exactly the same error seen in zuul ci (updates) job.... but there the job is marked as SUCCESS somehow for my TAGS patch... | 13:13 |
| quiquell | rfolco|ruck: The version of EmilienM fix that passed updates have newer THT version | 13:15 |
| quiquell | EmilienM, rfolco|ruck: Don't know if THT can have break updates but they didn't detect it because we broke the gates | 13:16 |
| quiquell | so they have a false positive update job | 13:16 |
| rfolco|ruck | quiquell, show me please when the job really updated without this error | 13:16 |
| EmilienM | quiquell: show me the trace | 13:17 |
| quiquell | rfolco|ruck: EmilienM fix passing updates http://logs.openstack.org/28/585528/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/2e91dbe/ | 13:17 |
| quiquell | rfolco|ruck, EmilienM: 585528,3 | 13:17 |
| quiquell | https://review.openstack.org/#/c/585528/3 | 13:18 |
| quiquell | It has the Depends-On | 13:18 |
| quiquell | In the history you can find the last zuul and it's passing | 13:18 |
| EmilienM | http://logs.openstack.org/28/585528/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8a2e24a/job-output.txt.gz#_2018-07-26_11_19_27_439555 | 13:19 |
| EmilienM | it fails here ^ | 13:19 |
| quiquell | failing with tht openstack-tripleo-heat-templates-9.0.0-0.20180725034409.db72fcd.el7.noarch | 13:19 |
| EmilienM | I don't think it's THT | 13:20 |
| marios | rlandy: yeah looks like it might be a nit like it should be indented you mean? | 13:20 |
| EmilienM | see the failure ^ | 13:20 |
| quiquell | EmilienM: ack | 13:20 |
| marios | rlandy: wrt https://logs.rdoproject.org/71/584771/14/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/c796079/logs/undercloud/var/lib/mistral/5cf4bb48-2415-4575-95ea-7459eb55929e/Controller/docker_config.yaml.txt.gz | 13:20 |
| EmilienM | quiquell: it fails to configure subnode | 13:21 |
| quiquell | EmilienM: Talking about the previous failure, this one doesn't make sense the other multinode jobs are working | 13:21 |
| quiquell | EmilienM: 10.4.70.63 is not suppose to be there ' | 13:21 |
| quiquell | ? | 13:21 |
| EmilienM | you're asking me? | 13:22 |
| EmilienM | I don't know | 13:22 |
| quiquell | The ip is ok we should be able to ssh it | 13:22 |
| rfolco|ruck | quiquell, I don't understand why with the depends-on it works | 13:23 |
| rfolco|ruck | in patchset 3 | 13:23 |
| quiquell | EmilienM: Let's wait to see if recheck goes beyond that | 13:23 |
| quiquell | rfolco|ruck: My only theory is newer tht | 13:24 |
| quiquell | rfolco|ruck: Depends-On make build-test-package kick in | 13:24 |
| quiquell | rfolco|ruck: Have check the openstack-tripleo-heat-templates and is newer in the working one | 13:24 |
| rlandy | marios: yep - not sure it's a nit though ... causing failure | 13:25 |
| rlandy | http://logs.openstack.org/08/584508/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/7125c41/job-output.txt.gz#_2018-07-25_04_58_43_670952 | 13:26 |
| rlandy | marios: ^^ | 13:26 |
| EmilienM | quiquell: can I update https://review.openstack.org/#/c/585528/ (commit message) to trigger CI again? | 13:26 |
| EmilienM | so we don't want for all jobs to finish | 13:26 |
| EmilienM | wait* | 13:27 |
| quiquell | EmilienM: Totally, is what I am trying to do misserably | 13:27 |
| EmilienM | quiquell: done | 13:28 |
| quiquell | EmilienM: thanks man | 13:28 |
| quiquell | EmilienM: Let's see if they go beyond that | 13:28 |
| *** kopecmartin has joined #oooq | 13:29 | |
| chandankumar | panda|rover: https://review.openstack.org/#/c/586213/ urgent fix to unblock promotion | 13:32 |
| *** skramaja has quit IRC | 13:33 | |
| *** chem has quit IRC | 13:33 | |
| *** vinaykns has joined #oooq | 13:38 | |
| *** vinaykns has quit IRC | 13:41 | |
| *** rook-- is now known as rook | 13:45 | |
| panda|rover | chandankumar: approved | 13:46 |
| quiquell | sshnaidm: We have to duplicate RDO/Upstream alarms if we want to have them splittted | 13:48 |
| sshnaidm | quiquell, why to split them..? | 13:48 |
| quiquell | sshnaidm: To have alerts for Upstream and alerts for RDO, but maybe is better to have them joined | 13:49 |
| panda|rover | ... | 13:49 |
| sshnaidm | quiquell, oh, I see, if we have alerts for noop jobs, it will include both rdo and upstream, right? | 13:50 |
| quiquell | sshnaidm: Yep... | 13:50 |
| quiquell | sshnaidm: Leet's continue after the meeting or tomorrow, will explain it in the demo | 13:51 |
| sshnaidm | quiquell, ok, but I don't think it's so necessary to split them | 13:51 |
| quiquell | sshnaidm: It suppose to normalize one day or another | 13:52 |
| sshnaidm | quiquell, exactly | 13:52 |
| quiquell | sshnaidm: But still we want stats splitted | 13:52 |
| *** dougbtv has joined #oooq | 14:03 | |
| *** rnoriega has quit IRC | 14:07 | |
| *** myoung has quit IRC | 14:07 | |
| *** weshay_PTO has quit IRC | 14:07 | |
| *** rnoriega has joined #oooq | 14:07 | |
| *** weshay has joined #oooq | 14:08 | |
| *** vinaykns has joined #oooq | 14:10 | |
| sshnaidm | rfolco|ruck, panda|rover https://trello.com/c/hkvfxAdX/667-cixtripleoci-rdo-software-factory-3rd-party-jobs-failing-due-to-instance-nodefailure | 14:11 |
| sshnaidm | rfolco|ruck, panda|rover this is still ongoing, just fyi | 14:11 |
| *** myoung has joined #oooq | 14:12 | |
| rfolco|ruck | sshnaidm, thanks. I thought it was resolved, then saw some errors this morning, then saw your chat about nodepools not spawning vms, then got confused... | 14:13 |
| sshnaidm | rfolco|ruck, also master phase 1 passed today \o/ | 14:14 |
| sshnaidm | rfolco|ruck, phase2 should pass too. All that remains is master in tripleo ci, after chandankumar has his patches mrged | 14:15 |
| rfolco|ruck | how do you know its only chandankumar patches ? | 14:15 |
| rfolco|ruck | sshnaidm, ^ | 14:15 |
| sshnaidm | rfolco|ruck, afaik yes, maybe also problem with scenario007, but I think it should have been resolved already.. | 14:16 |
| rfolco|ruck | good info thx for the update sshnaidm | 14:16 |
| *** jfrancoa has quit IRC | 14:28 | |
| *** sanjayu_ has quit IRC | 14:29 | |
| *** jfrancoa has joined #oooq | 14:30 | |
| *** bogdando has quit IRC | 14:34 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: (1 more message) | 14:37 |
| *** florianf has joined #oooq | 14:38 | |
| quiquell | EmilienM, rfolco|ruck, panda|rover: The jobs are going beyond the problem, let's cross fingers | 14:40 |
| quiquell | Holy sh, | 14:42 |
| quiquell | tripleo-ci-centos-7-containers-multinode-pike network failure downloading delorean.repo | 14:42 |
| quiquell | Failed to retrieve repo file from https://trunk.rdoproject.org/centos7-pike/current-tripleo/delorean.repo after 10 retries | 14:42 |
| quiquell | going to leave it just to check that upgrade/updates jobs are workin g | 14:44 |
| rfolco|ruck | quiquell, what does this have with the problem? | 14:44 |
| quiquell | rfolco|ruck: Nothing... we are very unlucky :-/ | 14:44 |
| quiquell | rfolco|ruck: Let keep it running to see that the other jobs works | 14:44 |
| *** holser_ has quit IRC | 14:55 | |
| *** tbarron has quit IRC | 15:01 | |
| *** tbarron has joined #oooq | 15:08 | |
| *** holser_ has joined #oooq | 15:10 | |
| *** hamzy has quit IRC | 15:41 | |
| *** hamzy has joined #oooq | 15:41 | |
| *** vinaykns has quit IRC | 15:43 | |
| *** vinaykns has joined #oooq | 15:43 | |
| *** zoli is now known as zoli|gone | 15:46 | |
| *** ykarel| has joined #oooq | 15:57 | |
| *** myoung is now known as myoung|lunch | 16:00 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:00 | |
| *** florianf has quit IRC | 16:10 | |
| *** holser_ has quit IRC | 16:14 | |
| chandankumar | rfolco|ruck: scenario7 issue resolved in the mroning | 16:16 |
| *** quiquell has quit IRC | 16:16 | |
| chandankumar | rfolco|ruck: https://review.openstack.org/585908 scenario7 fix | 16:17 |
| *** ykarel| has quit IRC | 16:18 | |
| *** verdurin has quit IRC | 16:21 | |
| *** verdurin has joined #oooq | 16:25 | |
| rfolco|ruck | chandankumar, thx for the update | 16:27 |
| chandankumar | rfolco|ruck: regaring pike delorean repo issue https://bugs.launchpad.net/tripleo/+bug/1783540/comments/1 | 16:29 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 16:29 |
| chandankumar | rfolco|ruck: I will be out tomorrow, feel free to take over the bugs | 16:29 |
| *** ykarel| has joined #oooq | 16:30 | |
| rfolco|ruck | chandankumar, why are you assigned to this bug, is this something you can fix ? | 16:31 |
| *** kopecmartin has quit IRC | 16:31 | |
| chandankumar | rfolco|ruck: we need to sync with software factory dfg or tristinc can help | 16:31 |
| chandankumar | join #sf-dfg for software and #rhos-ops for rdocloud | 16:32 |
| chandankumar | both are internal to RH | 16:32 |
| rfolco|ruck | chandankumar, already in both... | 16:33 |
| panda|rover | pike is failing is emilien\s patch | 16:33 |
| panda|rover | but was working in PS3 | 16:33 |
| rfolco|ruck | panda|rover, I think its the same error as https://bugs.launchpad.net/tripleo/+bug/1783540 | 16:34 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 16:34 |
| chandankumar | 2018-07-26 14:36:49.302409 | primary | ssh: connect to host 10.4.70.63 port 22: No route to host | 16:34 |
| chandankumar | http://logs.openstack.org/13/586213/1/check/tripleo-ci-centos-7-containers-multinode-pike/825d7c0/job-output.txt.gz#_2018-07-26_14_36_49_302409 | 16:34 |
| chandankumar | it is recent failure | 16:34 |
| panda|rover | rfolco|ruck: yes, I'm not counting it as blocking | 16:35 |
| panda|rover | rfolco|ruck: if everything else passes, I'm going to merge it | 16:36 |
| rfolco|ruck | May the gods of CI be with you panda|rover | 16:36 |
| panda|rover | they never are | 16:36 |
| rfolco|ruck | panda|rover, wait | 16:37 |
| chandankumar | we donot know who will screw rdo cloud or upstream zuul nodepool | 16:37 |
| panda|rover | the only god that favours me is the rotten luck | 16:37 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: (1 more message) | 16:37 |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| chandankumar | last 2 weeks both screwed us | 16:38 |
| panda|rover | chandankumar: I've seen this in other jobs, starting a couple of hours ago | 16:38 |
| rfolco|ruck | panda|rover, are you talking about merging emilien's fix ? | 16:38 |
| panda|rover | rfolco|ruck: yes | 16:38 |
| *** tesseract has quit IRC | 16:39 | |
| *** amoralej is now known as amoralej|off | 16:39 | |
| rfolco|ruck | panda|rover, PS5 did not report yet, finishing jobs, including updates one. And pike failed (voting). Whats the plan? | 16:40 |
| panda|rover | rfolco|ruck: that's what I was talking about | 16:41 |
| panda|rover | rfolco|ruck: I've seen pike pass in PS3, so I'm conting it as a success. it's failing with the infra problems. If everything else passes, I'm going to merge it. | 16:41 |
| *** ykarel| has quit IRC | 16:42 | |
| rfolco|ruck | panda|rover, it passes with the depends-on, yes. Now if it passes without depends-on, so tht does not matter. Agreed to merge if it passes... we'll need one more recheck as pike is voting. | 16:43 |
| *** vinaykns has quit IRC | 16:44 | |
| panda|rover | dammit | 16:45 |
| panda|rover | tempest passed | 16:46 |
| chandankumar | panda|rover: on the tempest patch pike failed | 16:46 |
| panda|rover | chandankumar: sorry, nevermind | 16:47 |
| *** chandankumar is now known as chkumar|trekk | 16:47 | |
| panda|rover | rfolco|ruck: updates failed | 16:47 |
| rfolco|ruck | panda|rover, sure? I am not finding it in console | 16:48 |
| rfolco|ruck | http://zuul.openstack.org/stream.html?uuid=ee17474ca5f14c1bba484529a824cafd&logfile=console.log | 16:48 |
| panda|rover | rfolco|ruck: 2018-07-26 16:37:23.788644 | primary | +(./toci_quickstart.sh:298): echo 'Playbook run of multinode-overcloud-update.yml failed' | 16:48 |
| rfolco|ruck | panda|rover, I made a noop change on top of the tags fix (the one you -2) and it is running updates job only here https://review.openstack.org/586235 | 16:51 |
| panda|rover | rfolco|ruck: at this point on safe change could be to just unmigrate the job to the old parent | 16:52 |
| panda|rover | ??? | 16:53 |
| panda|rover | rfolco|ruck: the job is marked as success | 16:54 |
| rfolco|ruck | false positive | 16:55 |
| rfolco|ruck | http://logs.openstack.org/28/585528/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/ee17474/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz | 16:55 |
| panda|rover | and this has nothing to do with the tags | 16:55 |
| *** d0ugal has joined #oooq | 16:55 | |
| *** d0ugal has quit IRC | 16:55 | |
| *** d0ugal has joined #oooq | 16:55 | |
| panda|rover | the playbook clearly fails, but we are not registering this as failuyre | 16:55 |
| rfolco|ruck | exactly what was trying to explain in the call | 16:55 |
| rfolco|ruck | panda|rover, ignore_errors in the task | 16:56 |
| panda|rover | rfolco|ruck: still it's our reponsibility, the playbook failed | 16:56 |
| panda|rover | no no | 16:56 |
| panda|rover | rfolco|ruck: it failed, with no ignore | 16:56 |
| panda|rover | rfolco|ruck: the playbook failed completely, and we are marking it as success. | 16:56 |
| rfolco|ruck | yep... ignore my comment above | 16:57 |
| rfolco|ruck | Set docker_startup_configs_with_default fact | 16:57 |
| rfolco|ruck | this is failing consistently with the same error, same point.... | 16:57 |
| panda|rover | dammit | 16:57 |
| rfolco|ruck | except on PS3 when we depends-on a newer tht review | 16:58 |
| panda|rover | that patch from quique was merged | 16:58 |
| rfolco|ruck | patch ? | 16:58 |
| panda|rover | I'm going to hit the red button | 16:58 |
| panda|rover | https://review.openstack.org/582466 | 16:59 |
| panda|rover | this broke us completely | 16:59 |
| panda|rover | I knew it, using the for loop was unnecessary, and it's causing us trouble | 17:00 |
| panda|rover | rfolco|ruck: bj/gcerami | 17:01 |
| panda|rover | rfolco|ruck: we need to take action | 17:01 |
| rfolco|ruck | sec | 17:01 |
| *** jfrancoa has quit IRC | 17:01 | |
| *** d0ugal has quit IRC | 17:21 | |
| rfolco|ruck | panda|rover, https://bugs.launchpad.net/tripleo/+bug/1783857 | 17:26 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Undecided,New] - Assigned to Gabriele Cerami (gcerami) | 17:26 |
| *** jtomasek has quit IRC | 17:29 | |
| *** trown is now known as trown|lunch | 17:29 | |
| chkumar|trekk | panda|rover: http://logs.openstack.org/13/586213/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/85287a1/job-output.txt.gz#_2018-07-26_17_14_28_700362 | 17:33 |
| chkumar|trekk | mirror timed out | 17:33 |
| panda|rover | dammit | 17:34 |
| chkumar|trekk | panda|rover: paul is looking into it on #openstack-infra | 17:34 |
| panda|rover | chkumar|trekk: ok, at this moment a lot is broken | 17:35 |
| *** myoung|lunch is now known as myoung | 17:35 | |
| chkumar|trekk | panda|rover: mirror fixed, i have done a recheck | 17:40 |
| rlandy | panda|rover: rfolco|ruck: https://review.openstack.org/#/c/583195/ is the patch marios wants merged ... looking at the rdocloud errors - if I recheck will that help at all? needs rebase? | 17:43 |
| rlandy | https://logs.rdoproject.org/95/583195/11/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata-branch/3a5b0c0/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 17:44 |
| rlandy | introspection timeout on ocata | 17:45 |
| rlandy | "Overcloud minor update execution step failed..." | 17:45 |
| rlandy | upgdate failure on 37 | 17:46 |
| rlandy | status on that? | 17:46 |
| panda|rover | rlandy: it's complicated | 17:47 |
| panda|rover | rlandy: talking again about tests and quality https://bugs.launchpad.net/tripleo/+bug/1783857 so even whe updates fail, we are showing a success | 17:50 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 17:50 |
| panda|rover | rlandy: but in other way, updates is broken, and not because of us | 17:50 |
| rlandy | panda|rover: right so we are n case 2 | 17:50 |
| panda|rover | rlandy: the failure we are seeing in updates is unrelated | 17:50 |
| rlandy | panda|rover: ack | 17:51 |
| rlandy | we can w+1 with theses failures | 17:51 |
| panda|rover | rlandy: we need someone for upgrades to help us understand what's happening | 17:51 |
| rlandy | but it depends if we should | 17:51 |
| rlandy | panda|rover: I did some investigation yesterday | 17:51 |
| rlandy | into the updates failure | 17:52 |
| panda|rover | rlandy: maybe the updates fail for rdo network issues too, it fails on pakcage update, baye it's failing to download packages | 17:52 |
| rlandy | getting ... | 17:52 |
| rlandy | https://logs.rdoproject.org/95/583195/11/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/696681e/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz#_2018-07-26_16_24_44 | 17:54 |
| rlandy | panda: note the ansible error | 17:54 |
| rlandy | u'fatal: [upstream-centos-7-rdo-cloud-0000075097]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}', | 17:54 |
| rlandy | panda|rover: ^^ | 17:54 |
| rlandy | 2018-07-26 16:24:44 | u'An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ^', | 17:55 |
| rlandy | rfolco|ruck and I looked into that yesterday | 17:55 |
| rlandy | I thought it was cleaned up today? | 17:55 |
| panda|rover | rlandy: it's not | 17:56 |
| panda|rover | rlandy: the updates jobs are failing with this http://logs.openstack.org/28/585528/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/ee17474/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz#_2018-07-26_16_37_22 | 17:56 |
| rlandy | https://logs.rdoproject.org/95/583195/11/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/696681e/logs/undercloud/var/lib/mistral/e29e9ccd-2725-4522-9db6-32034e3971d2/Controller/docker_config.yaml.txt.gz | 17:57 |
| panda|rover | rlandy: and since it's an error in package update step, it may be related to network errors in the repo | 17:57 |
| rlandy | but them why the ansible error? | 17:57 |
| panda|rover | rlandy: bj/gcerami ? I'm talking with rfolco|ruck too about this | 17:58 |
| rlandy | was asking marios about the misplaces start_order in the link above | 17:58 |
| rlandy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps-tasks.yaml#L77 | 18:02 |
| *** jaganathan has quit IRC | 18:16 | |
| rlandy | https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2 | 18:18 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy- (1 more message) | 18:37 |
| rook | rlandy: how can I get onto something that has ran featureset53? | 18:51 |
| rook | I want to look at why this is failing : | 18:51 |
| rook | 2018-07-25 21:35:15.305816 | tripleo-ovb-centos-7 | failed: [undercloud] (item=centos7) => {"changed": false, "cmd": ". /home/zuul/overcloudrc; openstack image list | grep 'centos7'", "delta": "0:00:04.091176", "end": "2018-07-25 21:35:15.286721", "item": "centos7", "msg": "non-zero return code", "rc": 1, "start": "2018-07-25 21:35:11.195545", "stderr": "volume version 3.52 is not in supported versions: 1, 2, | 18:51 |
| rook | 3", "stderr_lines": ["volume version 3.52 is not in supported versions: 1, 2, 3"], "stdout": "", "stdout_lines": []} | 18:51 |
| rlandy | rook: I can't get you onto those nodes | 18:52 |
| rlandy | infra might be able to | 18:52 |
| rlandy | ask on #rdo | 18:52 |
| rlandy | we usually use the reproducer | 18:52 |
| rlandy | to do that sort of debug worl | 18:52 |
| rlandy | work | 18:52 |
| rlandy | then it is your won node and you can work on it as you need | 18:53 |
| rook | agopi: ^ didn't you have issues with the reproducer? | 18:54 |
| *** tosky has quit IRC | 18:57 | |
| *** atoth has quit IRC | 18:57 | |
| agopi | yeah rdo cloud was down rook | 18:59 |
| agopi | will try again now | 18:59 |
| agopi | and if i get it to work | 18:59 |
| agopi | you can hop on | 18:59 |
| agopi | https://gist.github.com/aakarshg/10f3e189330a555270f369e4e0943e4c rlandy | 19:00 |
| agopi | i was seeing this | 19:00 |
| rlandy | agopi: I noticed that on other check jobs | 19:00 |
| rlandy | not your fault | 19:01 |
| agopi | ack rlandy retrying reproducer now | 19:01 |
| agopi | will ping if i see any other issue | 19:01 |
| rlandy | agopi: feel free to follow/ask on #rhos-ops | 19:01 |
| rlandy | as well | 19:01 |
| agopi | ack rlandy | 19:01 |
| *** hamzy has quit IRC | 19:02 | |
| *** hamzy_ has joined #oooq | 19:02 | |
| *** brault_ has quit IRC | 19:08 | |
| *** brault has joined #oooq | 19:10 | |
| *** agopi is now known as agopi|brb | 19:11 | |
| rlandy | rfolco|ruck: if I want to run a reproducer on fs037, what's the a latest updated job that has shown that | 19:25 |
| rlandy | I have https://review.openstack.org/#/c/583195/ | 19:25 |
| rlandy | but it's rebased on a milion changes | 19:25 |
| rfolco|ruck | rlandy, not sure what you mean | 19:35 |
| rfolco|ruck | https://review.openstack.org/#/c/585528/ | 19:35 |
| rfolco|ruck | need a reproducer for ^ latest ps | 19:36 |
| rfolco|ruck | the one that fails, isnt't it ? | 19:36 |
| *** trown|lunch is now known as trown | 19:43 | |
| *** ccamacho1 has quit IRC | 19:43 | |
| rasca | rlandy, you still around by any chance? | 19:48 |
| rlandy | rasca: yes | 19:48 |
| rasca | rlandy, hey hi, so I think that the latest review on the bm-ha-ovb things fix all that I need about rdocloud but... | 19:49 |
| rasca | rlandy, I'm still getting the error "Error: Failed to apply catalog: Execution of '/usr/bin/getcert list' returned 1: Error org.freedesktop.DBus.Error.TimedOut" | 19:49 |
| rlandy | rasca: can you link full log? | 19:50 |
| rlandy | or show where this is happening? | 19:50 |
| rook | agopi|brb: ack | 19:51 |
| rasca | rlandy, sure | 19:51 |
| *** agopi|brb is now known as agopi | 20:00 | |
| rlandy | panda|rover: rfolco|ruck: have you see this error lately in ci? https://bugs.launchpad.net/tripleo/+bug/1772807 | 20:37 |
| openstack | Launchpad bug 1772807 in tripleo "default containerized undercloud install with local CA fails with "Error org.freedesktop.DBus.Error.TimedOut"" [High,Triaged] | 20:37 |
| rlandy | rasca is hitting it | 20:37 |
| rasca | panda|rover, rfolco|ruck, I wonder why we're not giving importance on this ^^^ | 20:37 |
| rlandy | pretty consistently | 20:37 |
| rasca | it is consistent on rdocloud | 20:37 |
| rasca | (I've just increased the importance to high) | 20:37 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch @ https://review.openstack.org/560445, (1 more message) | 20:37 |
| rfolco|ruck | rasca, too many critical problems, I'll take a look | 20:38 |
| rasca | rfolco|ruck, don't want to harass you, I'll continue my tests on Monday | 20:39 |
| rasca | rlandy, restarting dbus resolved the problem | 20:39 |
| rlandy | rasca: hmm - at least that springs you for the moment | 20:40 |
| rlandy | we need to raise that at escalation meeting | 20:40 |
| rasca | rlandy, yeah but I've two choices | 20:40 |
| rasca | 1) I go manual (it's an option, but it will not help us in automating rdocloud jobs) | 20:41 |
| rasca | 2) I add a workaround patch (but don't like workarounds, do we?) | 20:41 |
| rlandy | rasca: or get it escalated and fixed | 20:42 |
| rlandy | you're on those meetings | 20:42 |
| rasca | I AM THE OWNER | 20:43 |
| rasca | :D | 20:43 |
| rlandy | rasca: cool -pull rank there | 20:43 |
| rasca | rlandy, I've added the alert tag, the card should be created and then I'll escalate it | 20:45 |
| rasca | rlandy, leaving now, thanks rlandy for your constant support | 20:45 |
| rlandy | rasca: ack - let's touch base tomorrow to talk about sprint work | 20:55 |
| *** brault has quit IRC | 20:55 | |
| *** trown is now known as trown|outtypewww | 21:03 | |
| *** dougbtv_ has joined #oooq | 21:11 | |
| *** rfolco|ruck is now known as rfolco|off | 21:11 | |
| *** dougbtv has quit IRC | 21:12 | |
| *** bandini has quit IRC | 21:48 | |
| *** bandini has joined #oooq | 21:48 | |
| *** gkadam has quit IRC | 22:30 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch @ https://review.openstack.org/560445, (1 more message) | 22:37 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
