| *** Goneri has joined #oooq | 00:12 | |
| *** agopi|brb has joined #oooq | 00:20 | |
| *** agopi|brb is now known as agopi | 00:23 | |
| *** Goneri has quit IRC | 00:28 | |
| rlandy | bbl | 00:31 |
|---|---|---|
| *** rlandy has quit IRC | 00:31 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-3nodes- (1 more message) | 00:39 |
| *** rfolco|off is now known as rfolco|ruck | 01:01 | |
| *** vinaykns has joined #oooq | 01:16 | |
| *** rfolco|ruck is now known as rfolco|off | 01:55 | |
| *** vinaykns has quit IRC | 01:57 | |
| *** vinaykns has joined #oooq | 01:57 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445 | 02:39 |
| *** links has joined #oooq | 03:15 | |
| *** skramaja has joined #oooq | 03:50 | |
| *** skramaja_ has joined #oooq | 04:00 | |
| *** skramaja has quit IRC | 04:00 | |
| *** jaganathan has joined #oooq | 04:00 | |
| *** jaganathan has quit IRC | 04:02 | |
| *** jaganathan has joined #oooq | 04:02 | |
| *** jaganathan has quit IRC | 04:03 | |
| *** jaganathan has joined #oooq | 04:03 | |
| *** gkadam has joined #oooq | 04:05 | |
| *** yolanda has quit IRC | 04:09 | |
| *** vinaykns has quit IRC | 04:14 | |
| *** ykarel|away has joined #oooq | 04:23 | |
| *** yolanda has joined #oooq | 04:24 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 04:39 |
| *** udesale has joined #oooq | 04:50 | |
| *** bogdando has joined #oooq | 05:10 | |
| *** skramaja_ is now known as skramaja | 05:21 | |
| *** ykarel|away has quit IRC | 05:26 | |
| *** jfrancoa has joined #oooq | 05:29 | |
| *** ratailor has joined #oooq | 05:35 | |
| *** ykarel|away has joined #oooq | 05:55 | |
| *** gkadam has quit IRC | 06:04 | |
| *** gkadam has joined #oooq | 06:05 | |
| *** quiquell has joined #oooq | 06:18 | |
| *** ccamacho has joined #oooq | 06:21 | |
| quiquell | Going to cry we have merge the fix :-) | 06:22 |
| quiquell | marios: On friday I made a little dashboard-ci demo, in case you are interested I can do it again for you | 06:23 |
| marios | quiquell: o/ | 06:24 |
| marios | quiquell: which fix | 06:24 |
| marios | quiquell: tags? | 06:24 |
| marios | quiquell: dashboard-ci memo? | 06:24 |
| marios | oh demo | 06:24 |
| marios | quiquell: cool sure maybe save it for the community call this afternoon? | 06:24 |
| quiquell | marios: Already did for the rest of the team | 06:26 |
| quiquell | Tag fix yes :-) | 06:26 |
| *** ykarel|away is now known as ykarel | 06:35 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 06:39 |
| *** bogdando has quit IRC | 06:44 | |
| *** quiquell is now known as quiquell|bbl | 07:04 | |
| *** jaganathan has quit IRC | 07:09 | |
| *** jaganathan has joined #oooq | 07:09 | |
| *** links has quit IRC | 07:13 | |
| *** bogdando has joined #oooq | 07:24 | |
| *** amoralej|off is now known as amoralej | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:36 | |
| *** tesseract has joined #oooq | 07:42 | |
| *** florianf has joined #oooq | 07:43 | |
| *** udesale has quit IRC | 07:49 | |
| *** udesale has joined #oooq | 07:49 | |
| *** tosky has joined #oooq | 07:55 | |
| quiquell|bbl | marios: Looking at your bootstrap patch | 08:07 |
| *** quiquell|bbl is now known as quiquell | 08:07 | |
| quiquell | We don't want to have a pre.yaml with all the tasks ? | 08:07 |
| *** udesale has quit IRC | 08:13 | |
| *** links has joined #oooq | 08:13 | |
| *** udesale has joined #oooq | 08:16 | |
| *** ykarel|lunch is now known as ykarel | 08:18 | |
| marios | quiquell: you mean https://review.openstack.org/583195 ? | 08:21 |
| quiquell | marios: Yep | 08:22 |
| marios | quiquell: it was renamed to 'ceph.yaml' after discussion https://review.openstack.org/#/c/583195/10/playbooks/tripleo-ci/pre.yaml | 08:23 |
| marios | quiquell: but it is still wired up the same into pre.yaml @ https://review.openstack.org/#/c/583195/12/zuul.d/base.yaml | 08:24 |
| quiquell | marios: Don't know if we will need it for the reproducer rafactoring | 08:24 |
| *** gkadam has quit IRC | 08:31 | |
| *** gkadam has joined #oooq | 08:37 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ (1 more message) | 08:39 |
| *** gkadam has quit IRC | 08:41 | |
| *** brault has joined #oooq | 08:55 | |
| *** sshnaidm|afk has quit IRC | 09:02 | |
| *** sshnaidm has joined #oooq | 09:04 | |
| quiquell | sshnaidm: Are you there ? have some question with some toci bash variables used at influxdb | 09:12 |
| *** ratailor has quit IRC | 09:20 | |
| *** ratailor has joined #oooq | 09:24 | |
| sshnaidm | quiquell, yep | 09:25 |
| *** zoli is now known as zoli|lunch | 09:32 | |
| quiquell | sshnaidm: They are REMAINING_TIME and STATS_TESTENV | 09:35 |
| quiquell | sshnaidm: I see that REMAINING_TIME is used for the run_with_timeout | 09:36 |
| quiquell | sshnaidm: But we are not running this with reproducer and we will use ansible stuff for that purpose | 09:36 |
| quiquell | sshnaidm: Ans zuul has it's own mechanism for timeouts... so maybe we can remove it ? | 09:36 |
| quiquell | sshnaidm: Even the use of run_with_timeout | 09:37 |
| quiquell | sshnaidm: Zuul is already timing out | 09:37 |
| sshnaidm | quiquell, run_with_timeout was for stopping job before zuul kill it to collect logs, but when we started to collect logs in post.yaml.. | 09:38 |
| sshnaidm | quiquell, maybe we can remove it | 09:38 |
| sshnaidm | quiquell, but it will be a problem for OVB jobs | 09:39 |
| sshnaidm | quiquell, because right now OVB hosts are erased right after zuul kills the job (when it's timeouted) and we can't collect logs from them | 09:39 |
| sshnaidm | quiquell, remaining time we get from devstack I think.. need to check if we get it in any way in new zuul way | 09:41 |
| quiquell | sshnaidm: Sure we have, I have a review tha prints all the vars | 09:41 |
| quiquell | sshnaidm: You can find the variable here https://review.openstack.org/#/c/581313/ | 09:41 |
| quiquell | sshnaidm: Where do we do the cleanup of OVB nodes ? | 09:42 |
| sshnaidm | quiquell, we don't really control cleanup of OVB jobs, it's something between zuul and te-broker | 09:43 |
| sshnaidm | quiquell, I've already checked once if it's possible to hold them, seems like not really | 09:43 |
| quiquell | sshnaidm: We don't send the delete order from toci ? | 09:43 |
| quiquell | sshnaidm: So I don't get the relation with REMAINING_TIME | 09:43 |
| sshnaidm | quiquell, no, all managements of OVB stack is done in te-broker | 09:43 |
| sshnaidm | quiquell, blue? | 09:44 |
| quiquell | sshnaidm: Sure | 09:44 |
| sshnaidm | https://bluejeans.com/u/sshnaidm/ | 09:44 |
| *** panda|rover|off is now known as panda|rover | 09:48 | |
| *** ratailor has quit IRC | 09:51 | |
| *** ratailor has joined #oooq | 09:52 | |
| *** jaosorior has quit IRC | 09:53 | |
| *** jaosorior has joined #oooq | 09:54 | |
| *** ratailor has quit IRC | 09:56 | |
| *** ratailor has joined #oooq | 09:57 | |
| ssbarnea | can we please merge the pause bugfix? https://review.openstack.org/#/c/583965/ | 10:08 |
| ssbarnea | it breaks all the time if console is redirected. | 10:09 |
| panda|rover | ssbarnea: are you sure connection: local is an effective replacement for local_action ? | 10:11 |
| panda|rover | ssbarnea: any reason why you're not using local_action directly ? | 10:11 |
| ssbarnea | panda|rover: because code is much harder to read when using local_action | 10:13 |
| ssbarnea | they have same functionality, but 2nd one is easier to read (and lint) | 10:13 |
| panda|rover | ssbarnea: then use delegate_to: 127.0.0.1 | 10:14 |
| panda|rover | ssbarnea: I'm reading docs and connection: has some implications | 10:14 |
| panda|rover | ssbarnea: https://docs.ansible.com/ansible/2.6/user_guide/playbooks_delegation.html#local-playbooks | 10:14 |
| panda|rover | ssbarnea: the note | 10:14 |
| panda|rover | ssbarnea: also if you look at the last example here https://docs.ansible.com/ansible/2.6/user_guide/playbooks_delegation.html#local-playbooks, you can use the args per line syntax | 10:16 |
| quiquell | panda|rover: Good morning, do you have some time for sprint sync ? | 10:16 |
| panda|rover | just add the argument module: | 10:16 |
| ssbarnea | reading it, i need to test to be sure, i know that these nuances can make a big difference. | 10:16 |
| ssbarnea | sure, will do. | 10:17 |
| panda|rover | ssbarnea: thanks | 10:17 |
| panda|rover | quiquell: sure | 10:18 |
| ssbarnea | panda|rover: there is another way to fix the bug, bumping ansible to 2.5.7 https://review.openstack.org/#/c/587371/ | 10:19 |
| ssbarnea | sadly jobs seem to be broken for some weird missing --requirements error. | 10:20 |
| quiquell | panda|rover: Already at your blue | 10:20 |
| quiquell | sshnaidm: Can I do the same with STATS_OOOQ ? | 10:21 |
| panda|rover | ssbarnea: yeah, too much to test before we can bump, yours seems really a good solution, instead of pausing blindly | 10:21 |
| panda|rover | ssbarnea: I don't want to block too much, if you don't have time to test the suggestion, I can take it back. | 10:22 |
| ssbarnea | panda|rover: i will rework it, also fixing the 35s instead of 30 (marios comment) | 10:25 |
| ssbarnea | clearly quickstart.sh looks borken to me: documentation states --requirements argument but implementation accepts only --requirements-file | 10:32 |
| ssbarnea | sshnaidm: I think you should be able to help me with that https://github.com/openstack/tripleo-quickstart/commit/0a30e04efe41c8e1579731c848e4d78f4a3768da | 10:36 |
| ssbarnea | you added "-file" suffix, but documentation line was not updated, also these jobs were not updated. Am I wrong to believe that -file should be removed? | 10:37 |
| ssbarnea | what i do not understand is how this was not catched by any gate, mainly is an API change that would break any job passing the --requirements parameter. | 10:38 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ (1 more message) | 10:39 |
| ssbarnea | panda|rover: ^ tell me if I am correct/wrong, so I would know if I shoud make a CR to remove -file or something else. | 10:39 |
| panda|rover | ssbarnea: sorry, context ? | 10:42 |
| *** zoli|lunch is now known as zoli | 10:47 | |
| ssbarnea | panda|rover: look at https://github.com/openstack/tripleo-quickstart/commit/0a30e04efe41c8e1579731c848e4d78f4a3768da#diff-8846dd18c9ee9c09dadeee541156c2b8L194 -- and line 270 | 10:52 |
| ssbarnea | this caused all rdo builds to fail, example: https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/6550/console | 10:54 |
| ssbarnea | i just want to know if I should remove the -file suffix, or just add back the one that does not have the suffix. | 10:55 |
| sshnaidm | quiquell, yep | 10:57 |
| sshnaidm | ssbarnea, yeah, I think you can remove it | 10:58 |
| sshnaidm | ssbarnea, I mean -file suffix | 10:58 |
| ssbarnea | doing it right now, thanks. | 10:58 |
| sshnaidm | and also remove these "--requirements" from ci.centos jobs | 10:59 |
| sshnaidm | ssbarnea, ^^ | 10:59 |
| sshnaidm | it's completely redundant there | 10:59 |
| sshnaidm | oh, we use there another requirements file, ok.. could be replaced by "-r" then | 11:00 |
| ssbarnea | sshnaidm: panda|rover https://review.openstack.org/#/c/587384/ -- fix for --req* | 11:01 |
| panda|rover | ssbarnea: is there a bug for the rdo failures ? | 11:01 |
| ssbarnea | nope, btw, where should I raise the bugs? | 11:03 |
| panda|rover | ssbarnea: in launchpad | 11:03 |
| ssbarnea | ok, doing it, refreshing review after. | 11:04 |
| panda|rover | ssbarnea: yep, please add the close bug tag in the review | 11:05 |
| panda|rover | ssbarnea: paste the bug here, so I can review it and eventually assign tags | 11:05 |
| ssbarnea | https://bugs.launchpad.net/tripleo-quickstart/+bug/1784608 | 11:07 |
| openstack | Launchpad bug 1784608 in tripleo-quickstart "quickstart.sh: ERROR: unknown option: --requirements" [Undecided,New] | 11:07 |
| ssbarnea | i am kinda happy to use LP instead of storyboard :) | 11:08 |
| panda|rover | ssbarnea: don't get too happy, the plan is to move | 11:11 |
| ssbarnea | yeah, i know... hopefully not all plans need to materialize. | 11:12 |
| ssbarnea | sshnaidm: panda|rover : now you can vote on https://review.openstack.org/#/c/587384/ again, i mentioned the bug. | 11:13 |
| panda|rover | ssbarnea: I said all the rdo jobs were failing for this bug ? | 11:14 |
| panda|rover | ssbarnea: do you have example logs ? | 11:14 |
| ssbarnea | panda|rover: yep, check this https://review.openstack.org/#/c/587371/ -- and look at failures, all of them for the same reason which is unrelated to the change. | 11:15 |
| panda|rover | sshnaidm: now I'm confused :) | 11:20 |
| panda|rover | ssbarnea: do you have time to chat ? | 11:26 |
| ssbarnea | sure | 11:27 |
| ssbarnea | panda|rover: https://bluejeans.com/u/ssbarnea | 11:29 |
| panda|rover | ssbarnea: give me 2 minutes | 11:31 |
| panda|rover | ssbarnea: I'm there | 11:33 |
| quiquell | damn: sshnaidm and ssbarnea have even the same number of letters | 11:33 |
| panda|rover | quiquell: duh | 11:33 |
| ssbarnea | try https://bluejeans.com/2655417928 | 11:33 |
| panda|rover | quiquell: they are both standard unix usernames | 11:34 |
| panda|rover | 8 letters | 11:34 |
| panda|rover | like the old times | 11:34 |
| quiquell | panda|rover: Unix etiquette | 11:34 |
| panda|rover | ssbarnea: I'm there | 11:34 |
| quiquell | sshnaidm: Question, do you have a env of dashboard-ci running ? looks like the bot is not legit | 11:35 |
| panda|rover | ssbarnea: ok, let's talk here | 11:45 |
| ssbarnea | https://bluejeans.com/2655417928 | 11:47 |
| *** amoralej is now known as amoralej|lunch | 11:55 | |
| *** rfolco|off is now known as rfolco|ruck | 12:12 | |
| *** panda|rover is now known as panda|rover|lunc | 12:18 | |
| *** ratailor has quit IRC | 12:32 | |
| weshay | morning | 12:34 |
| weshay | panda|rover|lunc, rfolco|ruck hey.. new kolla bug on horizon.. but good news is that jobs are kicking :) | 12:34 |
| *** rlandy has joined #oooq | 12:35 | |
| panda|rover|lunc | new ? so it's still in horizon, but a different one from sunday ? | 12:35 |
| weshay | panda|rover|lunc, ya | 12:35 |
| weshay | panda|rover|lunc, https://trello.com/c/qGAKp5Yw | 12:36 |
| weshay | panda|rover|lunc, rfolco|ruck also I noticed a bug in the script that creates escalations | 12:36 |
| weshay | should be fixed now | 12:36 |
| quiquell | weshay: gm, fyi rr dashboard is now at dashboard-ci.tripleo.org the migration to tripleo-infra is done | 12:37 |
| weshay | quiquell, nice | 12:37 |
| weshay | quiquell, what is up w/ the RDO CI stats | 12:37 |
| weshay | we're saying 90% of those jobs are failing? | 12:38 |
| weshay | that is coming from the zuul json? | 12:38 |
| weshay | quiquell, ? | 12:39 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container- (1 more message) | 12:39 |
| weshay | quiquell, wouldn't we see more fs001/35 failures ^ | 12:39 |
| *** ykarel is now known as ykarel|away | 12:40 | |
| *** agopi has quit IRC | 12:40 | |
| rlandy | rfolco|ruck: panda|rover|lunc: weshay: when you get a chance, pls see reviews from https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration - didn't kick the browbeat job. I think we may have to merge the some of these? | 12:40 |
| weshay | quiquell, where would you like bugs written for the ruck/rover cockpit? | 12:42 |
| quiquell | weshay: I am back | 12:42 |
| weshay | panda|rover|lunc, rfolco|ruck btw.. http://cistatus.tripleo.org/promotion/ is back | 12:43 |
| quiquell | weshay: For bugs FIXME section at https://docs.google.com/document/d/1MHflTy9krTFGrZ4nL_PG4AJDlcynWVkau9Mjvnkq094/edit is enough for now | 12:43 |
| weshay | quiquell, k.. thanks | 12:43 |
| weshay | quiquell, so what's up w/ the rdo jobs? | 12:43 |
| weshay | 90% fail? | 12:43 |
| quiquell | weshay: Let me check | 12:43 |
| weshay | quiquell, ya.. what is the data source on that | 12:44 |
| quiquell | weshay: The zuulv3 builds API | 12:45 |
| quiquell | Have to document all that at the panels | 12:45 |
| quiquell | weshay: Exploring them here http://dashboard-ci.tripleo.org/d/-UEjGKFmz/exploration?orgId=1&var-influxdb_filter=type%7C%3D%7Crdo | 12:47 |
| quiquell | weshay: They look pretty broken yes | 12:48 |
| quiquell | weshay: Going to verify, doesn't make sense | 12:49 |
| weshay | quiquell, sorry.. what is the url for the json? | 12:51 |
| quiquell | weshay: https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds | 12:52 |
| panda|rover|lunc | I'm seeing some of these https://logs.rdoproject.org/94/587394/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/caf9600/job-output.txt.gz#_2018-07-31_12_42_39_921617 in the latest jobs | 12:53 |
| quiquell | weshay: for tqe https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds?project=openstack/tripleo-quickstart-extras | 12:53 |
| weshay | quiquell, thanks | 12:53 |
| weshay | rfolco|ruck, you see https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds and search for FAILURE | 12:54 |
| panda|rover|lunc | rfolco|ruck: ^ | 12:54 |
| weshay | sshnaidm, how we doing on getting collaborators on sova? | 12:55 |
| panda|rover|lunc | rfolco|ruck: asking in #sf about these, but seems we have similar failures in post pipeline too | 12:55 |
| sshnaidm | weshay, cores were added | 12:55 |
| sshnaidm | weshay, and panda|rover|lunc was collaborator there before | 12:55 |
| sshnaidm | weshay, you should have mail about it | 12:55 |
| rlandy | sshnaidm++ | 12:57 |
| hubbot | rlandy: sshnaidm's karma is now 4 | 12:57 |
| rlandy | I missed that promotions/rdocloud job page | 12:57 |
| sshnaidm | rlandy, yeah, should work now | 12:58 |
| rlandy | sshnaidm: now we are going to change the jobs names again :) | 12:58 |
| weshay | sshnaidm, thank you!! | 12:58 |
| sshnaidm | weshay, also moved ovb jobs form promotion page to "check" jobs | 12:59 |
| rlandy | yay | 12:59 |
| weshay | sshnaidm, ya.. that was smart | 12:59 |
| weshay | thank you | 12:59 |
| rlandy | marios: hi - how are we going on the pike failure? | 13:00 |
| *** agopi has joined #oooq | 13:01 | |
| weshay | quiquell, sshnaidm so looking at the results from sova and the ruck/rover cockpit.. judging http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?panelId=63&fullscreen&orgId=1 | 13:01 |
| weshay | doesn't seem right to me | 13:01 |
| marios | rlandy: waiting on the latest run, it is going to hang/timeout again (form console) almost done watching zuul run | 13:02 |
| *** ykarel|away has quit IRC | 13:02 | |
| marios | rlandy: it should include logs this time | 13:03 |
| marios | rlandy: https://trello.com/c/GvjcnJB2/850-translate-tripleosh-bootstrap-subnodes-into-a-series-of-tasks#comment-5b60384c3e7a3cc8bd6f0819 | 13:03 |
| quiquell | weshay: What's the major difference you see ? | 13:04 |
| sshnaidm | weshay, I don't count nodes failures in sova | 13:04 |
| rlandy | marios: my reproducer did just fine :( | 13:04 |
| sshnaidm | weshay, they don't have logs, so nothing to analyze there | 13:04 |
| marios | rlandy: yeah thanks very much i got the update | 13:04 |
| rlandy | which means I didn't reproduce anything | 13:05 |
| rlandy | weird | 13:05 |
| quiquell | sshnaidm, weshay: It's still good to have node_failures at dashboard-ci ? | 13:05 |
| quiquell | weshay: We 'skip' SKIPPED builds at dashboard-ci | 13:05 |
| weshay | quiquell, depends.. | 13:05 |
| sshnaidm | weshay, and sova skip "skipped" too | 13:06 |
| quiquell | sshnaidm, weshay: The only differente is the NODE_FAILURE we can ignore them if you see it appropiate | 13:06 |
| sshnaidm | quiquell, I think worth to have them, otherwise we'll miss it | 13:06 |
| weshay | sshnaidm, quiquell yes.. it's worth it.. if that is the final job state | 13:07 |
| weshay | quiquell, noting that paul put in a change to have zuul retry | 13:08 |
| quiquell | weshay: are NODE_FAILURE going to disappear in the future ? | 13:08 |
| *** links has quit IRC | 13:09 | |
| sshnaidm | quiquell, maybe it's worth to split "last jobs" to upstream and rdo-ci too | 13:10 |
| sshnaidm | so that we'll not be spammed with these node failures | 13:10 |
| quiquell | sshnaidm: Yep agree, we missed that part in the last split exercise | 13:11 |
| quiquell | weshay: are you ok with that ? | 13:11 |
| weshay | ya.. last jobs should be split | 13:12 |
| weshay | agree | 13:12 |
| quiquell | weshay, sshnaidm: review coming | 13:12 |
| chandankumar | weshay: trown|outtypewww sshnaidm please have a look at this etherpad when you are free https://etherpad.openstack.org/p/devconfin2018 I need to prepare demo tomorrow | 13:15 |
| weshay | sorry my connection dropped | 13:17 |
| weshay | panda|rover|lunc, rfolco|ruck you guys want/need to sync? | 13:17 |
| rfolco|ruck | yes... I am looking at master container build failure... want to know if this kolla error is a new one... | 13:18 |
| rfolco|ruck | nERROR:kolla.common.utils.ec2-api:Unknown error when pushing\nTraceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/kolla/image/build.py\", line 322, in run\n self.push_image(image)\n File \"/usr/lib/python2.7/site-packages/kolla/image/build.py\", line 348, in push_image | 13:18 |
| rfolco|ruck | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-master-containers-build/e824a88/job-output.txt.gz | 13:18 |
| sshnaidm | quiquell, hmm.. if we split "check jobs" and "ci stats" too, maybe it's worth to have a different row for "rdo jobs" | 13:18 |
| *** Goneri has joined #oooq | 13:19 | |
| quiquell | sshnaidm: Doing a review for last jobs | 13:19 |
| weshay | rfolco|ruck, /me looking | 13:19 |
| weshay | rfolco|ruck, don't see errors in https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-master-containers-build/e824a88/logs/kolla/ | 13:20 |
| weshay | so things built ok | 13:21 |
| weshay | and I can close the horizon bug | 13:21 |
| sshnaidm | quiquell, btw, do you have graphs for "RDO cloud performance"? for me all is empty except "nodes status" graph | 13:21 |
| quiquell | sshnaidm: Nope, they don't appear I have add a todo/fixme list https://docs.google.com/document/d/1MHflTy9krTFGrZ4nL_PG4AJDlcynWVkau9Mjvnkq094/edit | 13:22 |
| weshay | rfolco|ruck, here's the error https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-master-containers-build/e824a88/logs/kolla/logs/ec2-api.log | 13:22 |
| weshay | panda|rover|lunc, rfolco|ruck I'm not sure if the periodic jobs are kicking often enough | 13:23 |
| weshay | quiquell, are you filtering for tripleo jobs btw? | 13:24 |
| weshay | or all jobs | 13:24 |
| weshay | looks like tripleo | 13:24 |
| weshay | sshnaidm, quiquell join my blue for a minute | 13:25 |
| chandankumar | rfolco|ruck: is this one a known issue 2018-07-31 10:52:42 | AmbiguousAuthSystem: Must provide Keystone credentials or user-defined endpoint, error was: cannot import name universaldetector | 13:25 |
| quiquell | weshay: Filtering by projects, all the projects that belong to tripleo | 13:25 |
| chandankumar | http://logs.openstack.org/68/584368/8/check/tripleo-ci-centos-7-undercloud-oooq/1670e29/logs/undercloud/home/zuul/undercloud_reinstall.log.txt.gz#_2018-07-31_10_52_42 | 13:25 |
| quiquell | weshay: ack | 13:25 |
| rfolco|ruck | chandankumar, apparently yes | 13:25 |
| quiquell | weshay, sshnaidm: split last jobs https://review.rdoproject.org/r/15089 | 13:26 |
| *** links has joined #oooq | 13:26 | |
| rfolco|ruck | weshay, master container build for example... 2018-07-30T22:57:50 than... 2018-07-31T04:57:12 --> 6 hours.... looks right? | 13:27 |
| panda|rover|lunc | rfolco|ruck: there should be another at 11 | 13:30 |
| panda|rover|lunc | more or less | 13:30 |
| panda|rover|lunc | not sure what's the time zone fo the log server | 13:31 |
| *** myoung has joined #oooq | 13:37 | |
| *** skramaja has quit IRC | 13:38 | |
| *** chuck_ has joined #oooq | 13:41 | |
| *** chuck_ is now known as zul | 13:42 | |
| weshay | rfolco|ruck, ya.. so I think we need to have the job run again | 13:42 |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| weshay | rfolco|ruck, that error seems transient | 13:43 |
| weshay | I think | 13:43 |
| weshay | panda|rover|lunc, rfolco|ruck anything else I can help w/? | 13:43 |
| rfolco|ruck | weshay, I did not open a new bug for container build master coz ... yes, not consistent | 13:43 |
| weshay | rfolco|ruck, ack.. ya that was the right thing imho .. agree w/ you | 13:44 |
| myoung | panda|rover|lunc, rfolco|ruck: (optional) please update https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @ L62 with any additional ruck/rover status items of note. Promotion status and alerts have already been added. | 13:44 |
| weshay | rfolco|ruck, are you using http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 ? | 13:45 |
| rfolco|ruck | weshay, I tried to. Realized it was a bit unstable yet | 13:47 |
| sshnaidm | to everybody who work with rdo cloud: https://pasteboard.co/Hx0I6sH.jpg | 13:48 |
| rfolco|ruck | sshnaidm, node_failure is gone? | 13:49 |
| weshay | lolz | 13:49 |
| weshay | rfolco|ruck, it's worth using and fixing :) | 13:49 |
| weshay | rfolco|ruck, ok.. the only other things I see are | 13:49 |
| weshay | rfolco|ruck, rdo phase 1 queens and pike | 13:50 |
| weshay | please get promotion blockers up on those two builds | 13:50 |
| weshay | and tripleo-ci, pike is at 3days, please check that out | 13:50 |
| weshay | panda|rover|lunc, anything I can help you out w/? | 13:50 |
| weshay | ssbarnea, ping.. want to sync up? | 13:51 |
| *** vinaykns has joined #oooq | 13:51 | |
| quiquell | sshnaidm, weshay: just skipping node_failures https://review.rdoproject.org/r/15091 | 13:53 |
| rfolco|ruck | myoung, do you care about promotion status for the squad mtg ? | 13:55 |
| weshay | quiquell, is the rdo cloud performance data updating for you? | 13:55 |
| weshay | I only see node status | 13:55 |
| quiquell | weshay: There is already a FIXME item for it | 13:56 |
| quiquell | weshay: Didn't have time to work this out | 13:56 |
| weshay | heh. k | 13:56 |
| weshay | np | 13:56 |
| quiquell | rfolco|ruck: What are the issues you found ? | 13:56 |
| myoung | rfolco|ruck: for the #tripleo meeting every week we document general tripleo CI status in our etherpad (like the other squads do) | 13:57 |
| myoung | rfolco|ruck: i've already populated it with promotion status, and tripleo bugs with 'alert' tag. Just meant if there's anything above/beyond that you think we should communicate from ruck standpoint, feel free to add it there. | 13:57 |
| myoung | #tripleo meeting starts in 120 sec (ish) | 13:58 |
| rfolco|ruck | myoung, k | 13:58 |
| marios | rlandy: come on timeout already http://zuul.openstack.org/stream.html?uuid=734965f901ce459f89ec4024d8a040fe&logfile=console.log | 13:59 |
| marios | ! | 13:59 |
| weshay | quiquell, so for rdo ci stats total | 13:59 |
| marios | rlandy: (pike job on https://review.openstack.org/#/c/583195/ should be done soon) | 13:59 |
| weshay | should be success / success + failures I suppose? | 13:59 |
| marios | rlandy: 'done' i mean timout but hoping to see logs lets see | 14:00 |
| rlandy | marios: looking | 14:00 |
| quiquell | weshay: for this one ? http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=1532872852639&to=1533045652639&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&panelId=203&fullscreen | 14:00 |
| quiquell | Ok have to leave for a few | 14:01 |
| rlandy | marios: and there we have a queens branch failure? | 14:01 |
| *** panda|rover|lunc is now known as panda|rover | 14:01 | |
| *** quiquell is now known as quiquell|bbl | 14:01 | |
| weshay | quiquell|bbl, aye | 14:01 |
| marios | rlandy: no the queens one is green at http://zuul.openstack.org/ tripleo-ci-centos-7-containers-multinode-queenssuccess | 14:02 |
| rlandy | ok - that failure was an image issue | 14:03 |
| *** bogdando has quit IRC | 14:11 | |
| marios | panda|rover: looks like your logs patch works \o/ | 14:13 |
| marios | panda|rover: added comment https://review.openstack.org/#/c/587103/4 | 14:13 |
| panda|rover | \o/ | 14:16 |
| weshay | <mwhahaha> the healthy bits are captured in the docker info log so we can probably check logstash | 14:18 |
| weshay | re: health checks.. | 14:18 |
| weshay | panda|rover, can I get you to look into that please? | 14:18 |
| *** quiquell|bbl is now known as quiquell | 14:18 | |
| quiquell | I am back | 14:18 |
| panda|rover | weshay: what do you want me to look ? if we have the docker health check logs in logstash ? | 14:21 |
| weshay | panda|rover, I don't think we do, I'd like them added.. also if possible to send that data to https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=1 | 14:23 |
| ssbarnea | sshnaidm: please recast vote on https://review.openstack.org/#/c/587384/ i lost it when I updated the message. | 14:24 |
| *** holser_ has joined #oooq | 14:24 | |
| weshay | rlandy, oy.. we have to duplicate all playbooks and templates? | 14:24 |
| weshay | rlandy, /me added paul to the review | 14:24 |
| rlandy | weshay: it looks that way to me - I was hoping panda|rover or rfolco|ruck would say I could do something else | 14:24 |
| weshay | :( | 14:25 |
| rlandy | weshay: I thought it would pick up a base test from tripleo | 14:25 |
| weshay | rlandy, it's worth double checking w/ pb | 14:25 |
| rlandy | dup'ed that as well :( | 14:25 |
| panda|rover | as far as I know, roles can be imported, playbooks must be copied | 14:25 |
| rlandy | weshay: absolutely - will add him | 14:25 |
| panda|rover | and I remember Paul first saying it. | 14:25 |
| weshay | panda|rover, k.. panda|rover so maybe we get the templates into a role? | 14:25 |
| quiquell | weshay, sshnaidm: the NODE_FAILURE graph https://review.rdoproject.org/r/15092 | 14:26 |
| quiquell | drop now, read you tomorrow | 14:26 |
| weshay | panda|rover, or that doesn't matter so much as it's temporary? | 14:26 |
| *** quiquell is now known as quiquell|off | 14:26 | |
| weshay | quiquell, thanks | 14:26 |
| rfolco|ruck | yes, playbooks have to belong to the same repo where the job is defined | 14:26 |
| sshnaidm | ssbarnea, you need to use "Closes-Bug: #111111" in commit message | 14:26 |
| openstack | bug 111111 in tepache (Ubuntu) "Tepache doesn't create a working code" [Undecided,Confirmed] https://launchpad.net/bugs/111111 | 14:26 |
| sshnaidm | ssbarnea, not "fixes lp" | 14:26 |
| panda|rover | lol | 14:26 |
| panda|rover | ssbarnea: and please fix that nasty Tepache bug. | 14:27 |
| sshnaidm | bug 0000001 | 14:27 |
| openstack | bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] https://launchpad.net/bugs/1 - Assigned to MFauzilkamil Zainuddin (apogee) | 14:27 |
| sshnaidm | heh | 14:27 |
| sshnaidm | maybe we can close this one ^ | 14:27 |
| panda|rover | WONTFIX | 14:28 |
| weshay | lolz | 14:28 |
| weshay | I love that bug | 14:28 |
| weshay | ssbarnea, ya.. please read through that openstack doc on commit messages | 14:28 |
| ssbarnea | sshnaidm: I copied this text from our documentation... but I just realised that I copied the "bad example".... bad idea to document the how not to do it before the correc tone. | 14:28 |
| weshay | that I sent yesterday.. it will save you time in the future | 14:28 |
| ssbarnea | I read, ..... but not the entire document. | 14:29 |
| weshay | ya. it's big :) | 14:29 |
| ssbarnea | is not that, i find the idea of putting BAD example before correct ones really,.... unfortunate. | 14:29 |
| sshnaidm | ssbarnea, https://docs.openstack.org/infra/manual/developers.html | 14:29 |
| panda|rover | weshay: can we get any design on the dashboard world ? The number, what they do ? these seems always injected out of sprint from individual contributions. We have currently tre different dashboard in three different places. | 14:30 |
| ssbarnea | I did google, find the doc, search in it, found text and copied and replaced bug number,... can you blame me for not reading line by line,... maybe a little bit ;) | 14:30 |
| weshay | ssbarnea, there are only a few important things to know | 14:30 |
| weshay | ssbarnea, I'll cover them w/ you in a minute | 14:31 |
| ssbarnea | funny part is that gerrit did recognize the bug and linked it. | 14:31 |
| sshnaidm | panda|rover, three? | 14:31 |
| sshnaidm | ssbarnea, but not sure it will close LP automatically with that message, should be "Closes" in it | 14:31 |
| ssbarnea | i will not touch it now, but I updated my note to use correct syntax. | 14:32 |
| sshnaidm | ssbarnea, if it doesn't solve bug completely, you can use "Related-Bug: #222222" | 14:32 |
| openstack | bug 222222 in linux (Ubuntu) "Sony VAIO VGN-SZ430N and other models; Stamina mode doesn't let Ubuntu boot up" [Undecided,Invalid] https://launchpad.net/bugs/222222 | 14:32 |
| panda|rover | sshnaidm: https://review.rdoproject.org/grafana/dashboard/db/tripleo-ci?orgId=1 , http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1, http://cistatus.tripleo.org/ | 14:34 |
| sshnaidm | panda|rover, also http://grafana.openstack.org/ | 14:34 |
| weshay | rasca, what's the bug # again? | 14:36 |
| rasca | weshay, looking for it sec | 14:37 |
| rasca | weshay, myoung, there https://bugs.launchpad.net/tripleo/+bug/1772807 | 14:38 |
| openstack | Launchpad bug 1772807 in tripleo "default containerized undercloud install with local CA fails with "Error org.freedesktop.DBus.Error.TimedOut"" [High,Triaged] | 14:38 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container- (1 more message) | 14:39 |
| *** links has quit IRC | 14:41 | |
| rfolco|ruck | sshnaidm, --requirements rings a bell? https://ci.centos.org/job/tripleo-quickstart-promote-pike-rdo_trunk-minimal/168/console | 14:42 |
| sshnaidm | rfolco|ruck, yeah. ssbarnea has a fixing patch | 14:42 |
| rfolco|ruck | sshnaidm, I remember seeing something, thought was you | 14:43 |
| rfolco|ruck | thx | 14:43 |
| sshnaidm | rfolco|ruck, https://review.openstack.org/#/c/587384/ | 14:43 |
| sshnaidm | rfolco|ruck, yeah, I'm this who broke it :D | 14:43 |
| ssbarnea | sshnaidm: it still needs Verified label to pass: https://review.openstack.org/#/c/587384/ | 14:44 |
| rfolco|ruck | sshnaidm, I don't know but I feel good hearing that after breaking gate... | 14:44 |
| rfolco|ruck | :) | 14:44 |
| sshnaidm | rfolco|ruck, everybody does it, no worries :) | 14:44 |
| chandankumar | sshnaidm: weshay this https://review.openstack.org/#/c/580384/ and this https://review.openstack.org/#/c/577780/5 | 14:45 |
| *** ccamacho has quit IRC | 14:48 | |
| arxcruz | sshnaidm: thanks for the review, can you please also review https://review.openstack.org/#/c/577780/ ? | 14:49 |
| arxcruz | :D | 14:49 |
| arxcruz | it's working now | 14:49 |
| sshnaidm | chandankumar, why undercloud, and not overcloud? https://review.openstack.org/#/c/577780 | 14:50 |
| sshnaidm | arxcruz, ^^ | 14:50 |
| *** tcw has quit IRC | 14:50 | |
| arxcruz | sshnaidm: because tempest runs in undercloud | 14:50 |
| sshnaidm | can't we run tempest container when overcloud is containerized? | 14:50 |
| *** tcw has joined #oooq | 14:51 | |
| arxcruz | sshnaidm: the run itself doesn't matter if overcloud is containerized or not | 14:51 |
| arxcruz | only undercloud | 14:51 |
| arxcruz | sshnaidm: for tempest is transparent, everything is tested via api | 14:51 |
| sshnaidm | arxcruz, ok | 14:52 |
| sshnaidm | arxcruz, does containerized_undercloud have a default value? | 14:53 |
| arxcruz | sshnaidm: to false iirc | 14:53 |
| arxcruz | sshnaidm: sorry, i don't know | 14:53 |
| sshnaidm | arxcruz, worth to check | 14:54 |
| sshnaidm | arxcruz, also I'd prefer this tempest config to be always in the end of file, so that people won't add undercloud_containerized settings *after* tempest config | 14:54 |
| arxcruz | sshnaidm: i added closer to other tempest stuff | 14:54 |
| sshnaidm | arxcruz, worth to have it in the end imho | 14:54 |
| arxcruz | okay, i'll update the files | 14:54 |
| sshnaidm | arxcruz, if somebody adds unvercloud containers settings after - this settings won't work | 14:54 |
| *** jfrancoa has quit IRC | 14:55 | |
| arxcruz | sshnaidm: fair enough | 14:55 |
| rfolco|ruck | panda|rover, can we create an etherpad with thoughts/suggestions/ideas/issues for cockpit ? | 14:57 |
| panda|rover | rfolco|ruck: https://trello.com/c/dyvGWJps/890-dashboards-maintenance | 14:57 |
| rfolco|ruck | ok will update there | 14:58 |
| panda|rover | this way this sill officially take off sprint time, but if quiquell|off needs to address the fixes it will have less time for the sprint, and this must be taken into consideration | 14:58 |
| *** jfrancoa has joined #oooq | 14:58 | |
| *** links has joined #oooq | 15:00 | |
| *** holser_ has quit IRC | 15:03 | |
| marios | rlandy: i think this is the change weshay was just referring to about the changes being applied from soren https://review.openstack.org/#/c/582963/ | 15:03 |
| weshay | ssbarnea, sorry you avail? | 15:06 |
| weshay | now | 15:06 |
| rasca | rlandy, https://bluejeans.com/9579113890 | 15:06 |
| weshay | https://bluejeans.com/4113567798 | 15:06 |
| ssbarnea | yep | 15:06 |
| *** holser_ has joined #oooq | 15:09 | |
| *** links has quit IRC | 15:12 | |
| rlandy | weshay: rfolco|ruck: panda|rover: meeting with paul this afternoon re: zuul v3 in rdocloud | 15:18 |
| panda|rover | rlandy: when ? | 15:19 |
| rfolco|ruck | rlandy, can I join ? | 15:19 |
| rlandy | panda|rover: not sure - will ping him later - and collect you all | 15:19 |
| rlandy | rfolco|ruck: of course | 15:19 |
| panda|rover | rlandy: collect us before time out, or post-run will fail. | 15:20 |
| rlandy | panda|rover" ^^??? | 15:20 |
| panda|rover | rlandy: a lousy joke, nevermind. | 15:21 |
| rlandy | panda|rover: ok - then I get it | 15:21 |
| rlandy | when is timeout? | 15:21 |
| * rfolco|ruck understands this flavor of jokes | 15:21 | |
| panda|rover | rlandy: for me is in ~2 hours | 15:22 |
| rlandy | panda|rover;will be after your timeout | 15:22 |
| rlandy | understand if you can't make it | 15:22 |
| chandankumar | weshay: sshnaidm I have moved the content for devconf demo to here https://docs.google.com/document/d/1PAhNsxf30m5ZRR--CX3GqB-FsDUj8M5oyedB5rLFaVc/edit | 15:23 |
| *** udesale has quit IRC | 15:23 | |
| panda|rover | rlandy: ok ping me anyway even if I'm off, I'll join if I can | 15:24 |
| rfolco|ruck | panda|rover, added promotion-blocker tag to https://bugs.launchpad.net/tripleo/+bug/1784608 | 15:29 |
| openstack | Launchpad bug 1784608 in tripleo "quickstart.sh: ERROR: unknown option: --requirements" [Critical,Fix released] - Assigned to Sorin Sbarnea (ssbarnea) | 15:29 |
| rfolco|ruck | coz you know... blocks promotion | 15:29 |
| panda|rover | rfolco|ruck: which jobs does it block ? I don't remember which jobs still use quickstart.sh | 15:29 |
| rfolco|ruck | https://ci.centos.org/job/tripleo-quickstart-promote-queens-rdo_trunk-minimal/119/console | 15:30 |
| rfolco|ruck | "rdo_trunk-promote-queens-current-tripleo" | 15:30 |
| panda|rover | rfolco|ruck: phase1 promotions | 15:30 |
| rfolco|ruck | yes | 15:30 |
| panda|rover | mmmhh | 15:30 |
| rfolco|ruck | still applies right ? | 15:30 |
| panda|rover | I'm not sure if tags applies for anything else than the trripleo-ci promotion | 15:31 |
| panda|rover | I've never used it ofr this | 15:31 |
| panda|rover | rfolco|ruck: keep it for now | 15:31 |
| rfolco|ruck | ok, if you don't add, it won't show up in cockpit for example | 15:32 |
| rfolco|ruck | in the alerts | 15:32 |
| rfolco|ruck | and promotion-blockers | 15:32 |
| *** holser_ has quit IRC | 15:33 | |
| panda|rover | ... | 15:33 |
| *** sshnaidm is now known as sshnaidm|off | 15:36 | |
| *** links has joined #oooq | 15:49 | |
| *** links has quit IRC | 15:51 | |
| *** links has joined #oooq | 15:52 | |
| *** links has quit IRC | 15:55 | |
| *** links has joined #oooq | 15:55 | |
| *** links has quit IRC | 15:58 | |
| *** zoli is now known as zoli|gone | 15:58 | |
| *** links has joined #oooq | 15:58 | |
| *** zoli|gone is now known as zoli | 15:58 | |
| *** links has quit IRC | 16:01 | |
| *** links has joined #oooq | 16:01 | |
| *** jfrancoa has quit IRC | 16:07 | |
| *** marios is now known as marios|gonehome | 16:08 | |
| *** marios|gonehome is now known as marios | 16:09 | |
| *** links has quit IRC | 16:10 | |
| weshay | rfolco|ruck, ping | 16:18 |
| weshay | rfolco|ruck, holla at me when you have time | 16:19 |
| *** links has joined #oooq | 16:21 | |
| weshay | rfolco|ruck, /me kicking jobs on https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/ | 16:22 |
| weshay | queens phase 1 is at 2 days | 16:22 |
| rfolco|ruck | weshay, lunch... will ping you asap | 16:23 |
| weshay | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-pike-current-tripleo/200/ | 16:23 |
| weshay | oh | 16:23 |
| weshay | 08:51:01 ERROR: unknown option: --requirements | 16:23 |
| weshay | ssbarnea, fixed this | 16:23 |
| rfolco|ruck | weshay, yep, was waiting this to merge | 16:24 |
| *** links has quit IRC | 16:24 | |
| weshay | rfolco|ruck, ok.. that should have a bug.. for future notice | 16:24 |
| weshay | and marked promotion blocker | 16:24 |
| *** myoung is now known as myoung|lunch | 16:24 | |
| *** links has joined #oooq | 16:25 | |
| rfolco|ruck | weshay, it has | 16:25 |
| rfolco|ruck | https://review.rdoproject.org/etherpad/p/ruckrover-sprint17 | 16:25 |
| rfolco|ruck | tags:added: promotion-blocker | 16:26 |
| weshay | rfolco|ruck, what line? | 16:26 |
| rfolco|ruck | 34 | 16:26 |
| *** links has quit IRC | 16:27 | |
| weshay | rfolco|ruck, thanks | 16:27 |
| weshay | :) | 16:27 |
| rfolco|ruck | yw | 16:27 |
| *** links has joined #oooq | 16:28 | |
| weshay | rfolco|ruck, didn't see it on http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit | 16:28 |
| weshay | under Promotion Blockers | 16:28 |
| rfolco|ruck | weshay, coz it did not have the tag one hour ago | 16:28 |
| weshay | now I do | 16:28 |
| weshay | yes | 16:28 |
| rfolco|ruck | weshay, gave some suggestions for the main tab https://trello.com/c/dyvGWJps/890-dashboards-maintenance | 16:29 |
| *** links has quit IRC | 16:30 | |
| rfolco|ruck | big charts just to show zuul queue is overkill | 16:30 |
| rfolco|ruck | IMHO | 16:30 |
| *** links has joined #oooq | 16:31 | |
| rfolco|ruck | alerts should show at the top, it gives overall status of ci | 16:31 |
| panda|rover | rfolco|ruck: anything critical to pass to me before I go ? | 16:31 |
| rfolco|ruck | panda|rover, no... I did not see any progress on the tempest bug for 3-node job | 16:32 |
| rfolco|ruck | https://bugs.launchpad.net/tripleo/+bug/1784017 | 16:32 |
| openstack | Launchpad bug 1784017 in tripleo "TestNetworkBasicOps.test_network_basic_ops failures" [Critical,Triaged] | 16:32 |
| weshay | rfolco|ruck, it's a pretty good indicatior of health imho | 16:33 |
| rfolco|ruck | non-voting, but we need to fix | 16:33 |
| weshay | the big pie graphs? | 16:33 |
| *** links has quit IRC | 16:33 | |
| rfolco|ruck | doesn't need to be big like that | 16:33 |
| *** links has joined #oooq | 16:33 | |
| rfolco|ruck | 2.5 hours queue and a small graph is enough | 16:33 |
| rfolco|ruck | anyways, this is my perspective | 16:34 |
| weshay | k k | 16:34 |
| weshay | rfolco|ruck, which jobs does this affect? https://bugs.launchpad.net/tripleo/+bug/1784017 | 16:34 |
| openstack | Launchpad bug 1784017 in tripleo "TestNetworkBasicOps.test_network_basic_ops failures" [Critical,Triaged] | 16:34 |
| weshay | rfolco|ruck, can you run that through an elastic recheck query | 16:34 |
| rfolco|ruck | tripleo-ci-centos-7-3nodes-multinode | 16:35 |
| rfolco|ruck | all times failing in check | 16:35 |
| *** links has quit IRC | 16:35 | |
| rfolco|ruck | its non-voting but annoying | 16:36 |
| *** links has joined #oooq | 16:36 | |
| chandankumar | rfolco|ruck: regarding above bug reason is here http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-3nodes-multinode/cd65ccf/logs/subnode-3/var/log/extra/errors.txt.gz#_2018-07-27_09_21_09_022 | 16:37 |
| weshay | rlandy, ping me when you chat w/ paul please | 16:38 |
| weshay | my afternoon is clear | 16:38 |
| *** links has quit IRC | 16:38 | |
| rlandy | weshay:ack | 16:39 |
| *** links has joined #oooq | 16:39 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 16:39 |
| chandankumar | weshay: which script generates extra/errors.txt file? | 16:39 |
| rfolco|ruck | weshay, we changed from qemu to kvm recently | 16:39 |
| rfolco|ruck | chandankumar, ^ | 16:39 |
| weshay | rfolco|ruck, correct.. | 16:41 |
| weshay | chandankumar, it's in collect logs | 16:41 |
| *** links has quit IRC | 16:41 | |
| weshay | chandankumar, oh interesting | 16:42 |
| *** links has joined #oooq | 16:42 | |
| weshay | wth.. does that show up on other jobs? | 16:42 |
| chandankumar | weshay: if that is the case, we can move collect-logs to use it all zuulv3 devstack jobs | 16:42 |
| chandankumar | that would be too much interesting | 16:43 |
| *** sshnaidm|off has quit IRC | 16:43 | |
| weshay | chandankumar, not sure what you are saying in that latest comment | 16:43 |
| weshay | chandankumar, I am planning on breaking out the collect-logs role | 16:43 |
| weshay | chandankumar, however it's just one simple command that creates the errors.txt file | 16:44 |
| weshay | if you are looking for that | 16:44 |
| weshay | I can show.. if you want to add it to devstack | 16:44 |
| weshay | chandankumar, need to thank Sagi for the idea though :) | 16:44 |
| *** links has quit IRC | 16:44 | |
| chandankumar | weshay: I mean to say in tripleo we use collect logs to gather all errors at one place at each node if we can add to all devstack jobs it will be much easier for other people | 16:44 |
| weshay | chandankumar, sure | 16:45 |
| weshay | you want the code snip? | 16:45 |
| *** links has joined #oooq | 16:45 | |
| chandankumar | weshay:yup | 16:45 |
| weshay | chandankumar, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml#L250 | 16:46 |
| weshay | chandankumar, then it's added to logstash | 16:46 |
| weshay | too | 16:46 |
| weshay | pretty nice | 16:46 |
| chandankumar | weshay: i will take a look | 16:47 |
| *** links has quit IRC | 16:48 | |
| *** amoralej is now known as amoralej|off | 16:48 | |
| *** links has joined #oooq | 16:48 | |
| *** panda|rover is now known as panda|rover|off | 16:49 | |
| *** links has quit IRC | 16:51 | |
| weshay | chandankumar, rfolco|ruck http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22could%20not%20find%20capabilities%20for%20domaintype%5C%5C%3Dkvm%5C%22%20AND%20build_status%3AFAILURE | 16:51 |
| *** links has joined #oooq | 16:51 | |
| weshay | chandankumar, rfolco|ruck really odd.. just the 3node and scenario008 job | 16:52 |
| weshay | rfolco|ruck, maybe we're not setting up the right packages on all the nodes in 3node | 16:53 |
| weshay | marios, ^ is there some diff w/ regards to the package prep in --boostrap w/ 3node? | 16:53 |
| rfolco|ruck | weshay, I was trying to find nova virt_type for the sub-nodes.... perhaps it uses qemu default for one subnode only ?? | 16:53 |
| rfolco|ruck | not finding it in etc | 16:53 |
| *** links has quit IRC | 16:54 | |
| *** links has joined #oooq | 16:54 | |
| *** links has quit IRC | 16:56 | |
| *** links has joined #oooq | 16:57 | |
| weshay | rfolco|ruck, this appears to run on all three http://logs.openstack.org/57/586057/10/check/tripleo-ci-centos-7-3nodes-multinode/a25e19b/logs/undercloud/var/log/bootstrap-subnodes.log.txt.gz | 16:57 |
| rfolco|ruck | weshay, why we don't save nova config under etc/docker/nova ? | 16:59 |
| weshay | ask in #tripleo | 17:00 |
| *** links has quit IRC | 17:00 | |
| weshay | thanks chandankumar | 17:00 |
| weshay | chandankumar++ | 17:00 |
| hubbot | weshay: chandankumar's karma is now 5 | 17:00 |
| chandankumar | weshay: sorry what have i done | 17:00 |
| weshay | chandankumar, found that kvm issue | 17:01 |
| chandankumar | weshay: you know Aziza and Sachin from Pune? | 17:01 |
| weshay | chandankumar, ya.. I love them :) | 17:01 |
| *** links has joined #oooq | 17:01 | |
| weshay | say hi for me | 17:01 |
| chandankumar | weshay: Aziza used to report to you and you hired sachine | 17:01 |
| chandankumar | weshay: will say it tomorrow | 17:01 |
| weshay | chandankumar, ya. neither really reported to me, I was just the team lead | 17:01 |
| weshay | those were fun days | 17:02 |
| weshay | easier than openstack | 17:02 |
| chandankumar | weshay: and I came to know your one's daughter name is Devvi :-) | 17:02 |
| chandankumar | *Devi | 17:02 |
| weshay | chandankumar, ha.. yes! | 17:02 |
| weshay | indian name :) | 17:02 |
| chandankumar | weshay: yes, one of the godess name :-) | 17:02 |
| weshay | yup | 17:03 |
| chandankumar | weshay: it is quite a small world :-) | 17:03 |
| weshay | heh.. small company | 17:03 |
| weshay | or it was | 17:03 |
| *** tesseract has quit IRC | 17:03 | |
| *** links has quit IRC | 17:03 | |
| weshay | chandankumar, ya.. tell Aziza and Sachin I miss them both :) | 17:03 |
| chandankumar | weshay: sure | 17:03 |
| *** links has joined #oooq | 17:04 | |
| weshay | chandankumar, you and I need to chat about the next few months of tempest work | 17:04 |
| weshay | when you have time | 17:04 |
| weshay | rfolco|ruck, thanks for logging that bug | 17:04 |
| chandankumar | weshay: Can we talk tomorrow ? | 17:05 |
| weshay | chandankumar, sure | 17:05 |
| chandankumar | weshay: let me know when you are free, I will schedule a meeting | 17:05 |
| *** links has quit IRC | 17:06 | |
| *** links has joined #oooq | 17:07 | |
| chandankumar | weshay: rfolco|ruck https://review.openstack.org/#/c/570892/ anf https://review.openstack.org/570884 can we merge this? | 17:07 |
| weshay | rfolco|ruck, SUCCESS https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-master-containers-build/db7e819/ | 17:09 |
| weshay | chandankumar, /me looks | 17:09 |
| rfolco|ruck | chandankumar, will review it asap | 17:09 |
| *** links has quit IRC | 17:09 | |
| rfolco|ruck | weshay, \o/ | 17:09 |
| *** links has joined #oooq | 17:10 | |
| weshay | chandankumar, why use the diff parent? https://review.openstack.org/#/c/570892/9 | 17:11 |
| *** links has quit IRC | 17:12 | |
| chandankumar | weshay: i will fix that | 17:12 |
| *** links has joined #oooq | 17:12 | |
| chandankumar | does someone looking at this job https://ci.centos.org/job/tripleo-quickstart-extras-gate-newton-delorean-full-minimal/6760/ | 17:13 |
| chandankumar | it always fails | 17:13 |
| *** links has quit IRC | 17:15 | |
| *** links has joined #oooq | 17:15 | |
| *** links has quit IRC | 17:18 | |
| *** links has joined #oooq | 17:18 | |
| *** links has quit IRC | 17:21 | |
| *** links has joined #oooq | 17:21 | |
| *** links has quit IRC | 17:24 | |
| *** links has joined #oooq | 17:24 | |
| *** links has quit IRC | 17:27 | |
| *** links has joined #oooq | 17:27 | |
| rfolco|ruck | chandankumar, this has been fixed by https://bugs.launchpad.net/tripleo/+bug/1784608 | 17:28 |
| openstack | Launchpad bug 1784608 in tripleo "quickstart.sh: ERROR: unknown option: --requirements" [Critical,Fix released] - Assigned to Sorin Sbarnea (ssbarnea) | 17:28 |
| *** links has quit IRC | 17:30 | |
| *** links has joined #oooq | 17:30 | |
| *** links has quit IRC | 17:33 | |
| *** links has joined #oooq | 17:33 | |
| rfolco|ruck | weshay, do you think this should be kvm instead ? https://github.com/openstack/tripleo-quickstart-extras/blob/ee03ae932a012f1eeede89c54248322f8538eab8/roles/overcloud-deploy/files/hardware_environments/virt/hw_settings.yml#L3 | 17:34 |
| *** myoung|lunch is now known as myoung | 17:34 | |
| weshay | rfolco|ruck, ya | 17:34 |
| rfolco|ruck | weshay, will make a quick test ok ? | 17:35 |
| rfolco|ruck | run 3-node job only with kvm fix | 17:35 |
| *** links has quit IRC | 17:36 | |
| *** links has joined #oooq | 17:37 | |
| weshay | rfolco|ruck, I think the issue the rpms required for kvm are not installed on all the nodes in 3node | 17:37 |
| weshay | but maybe I'm wrong | 17:37 |
| rfolco|ruck | weshay, qemu-kvm should be enough afaik | 17:38 |
| *** vinaykns has quit IRC | 17:38 | |
| *** links has quit IRC | 17:39 | |
| *** links has joined #oooq | 17:40 | |
| *** links has quit IRC | 17:42 | |
| *** links has joined #oooq | 17:42 | |
| *** links has quit IRC | 17:45 | |
| *** links has joined #oooq | 17:46 | |
| *** florianf has quit IRC | 17:46 | |
| *** links has quit IRC | 17:52 | |
| *** florianf has joined #oooq | 18:01 | |
| rlandy | rasca: ping | 18:05 |
| rlandy | rasca: 3ctlr_1comp or 1? | 18:05 |
| weshay | rlandy, 3 | 18:05 |
| rlandy | I'm going with 3 | 18:05 |
| weshay | :) | 18:05 |
| rlandy | object on the patch pls | 18:05 |
| rlandy | if I did the wrong thing | 18:05 |
| rlandy | hack, hack, hack .... | 18:06 |
| rlandy | agopi: rook: weshay: any issue with merging this review? https://review.openstack.org/#/c/583717/ | 18:07 |
| rlandy | I can remove the depends on | 18:07 |
| rlandy | and we can make a test patch for triggering | 18:08 |
| rlandy | not a blocker - just asking | 18:08 |
| agopi | rook patched it up rlandy, waiting for it finish running in our CI | 18:09 |
| rlandy | agopi: ack - no rush | 18:10 |
| agopi | should be good to go by tomorrow | 18:10 |
| rlandy | nice | 18:10 |
| rlandy | thanks | 18:10 |
| agopi | rlandy++ | 18:10 |
| hubbot | agopi: rlandy's karma is now 18 | 18:10 |
| rlandy | weshay: ^^ | 18:10 |
| weshay | k | 18:11 |
| *** holser_ has joined #oooq | 18:12 | |
| rook | rlandy: yeah we hit a couple of snags... but should be good after some patches from today. | 18:18 |
| *** vinaykns has joined #oooq | 18:27 | |
| *** holser_ has quit IRC | 18:33 | |
| *** jaganathan has quit IRC | 18:36 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 18:39 |
| rfolco|ruck | rlandy, keep me in the loop, I can cover you on this work on your PTO | 18:51 |
| rfolco|ruck | if you want, of course | 18:51 |
| rlandy | rfolco|ruck++ | 18:52 |
| hubbot | rlandy: rfolco|ruck's karma is now 1 | 18:52 |
| rlandy | rfolco|ruck: on the upside, it might get you out of ruck/rover shift :) | 18:54 |
| rfolco|ruck | rlandy, shhhh walls have ears | 18:55 |
| rfolco|ruck | :) | 18:55 |
| rlandy | rfolco|ruck; I plan for everything to go wrong while I am hiking some glacier with no internet access | 18:56 |
| rfolco|ruck | rlandy, haha | 18:56 |
| myoung | rlandy: are you headed to glaciers? | 18:59 |
| rlandy | myoung: yep - next week | 18:59 |
| myoung | coooooooollll | 18:59 |
| myoung | where? | 18:59 |
| myoung | (north obviously) | 18:59 |
| rlandy | Alaska | 18:59 |
| myoung | wowzers! | 18:59 |
| myoung | i got to fly over one a few weeks ago. amazing. | 19:00 |
| myoung | I suspect hiking one will be more of a workout :) | 19:00 |
| rlandy | well, the glacier hike is our last day - so we may be less brave by then | 19:01 |
| rlandy | https://review.openstack.org/587603 WIP: DNM: Enable upstream testing of tripleo-ha-utils | 19:03 |
| rlandy | rasca: ^^ | 19:03 |
| rfolco|ruck | weshay, I think my thoery has good chances of being correct: overcloud deploy uses --libvirt_type kvm when deploying nodes but for tempest what counts is what nova.conf has, should be virt_type = kvm | 19:12 |
| rfolco|ruck | if notthing there, uses qemu as default | 19:12 |
| rfolco|ruck | rlandy, wow I thought you were kidding... enjoy your hiking / skiing at alaska | 19:19 |
| weshay | rfolco|ruck++ | 19:19 |
| hubbot | weshay: rfolco|ruck's karma is now 2 | 19:19 |
| weshay | ya.. | 19:19 |
| weshay | rfolco|ruck, so is that read in from hw_env? | 19:19 |
| weshay | I was looking for that | 19:19 |
| rfolco|ruck | weshay, still looking... I think I'll try to add etc/nova logs for debugging | 19:20 |
| rfolco|ruck | weshay, if i am reading your mind, you thinking "don't go too deep"... ok will add my comments to the bug | 19:22 |
| weshay | rfolco|ruck, well.. I like where you going w/ this.. tbh however keep in mind we have to get master going | 19:24 |
| weshay | rfolco|ruck, https://review.rdoproject.org/zuul/status.htm | 19:24 |
| weshay | looks like fs001 and 35 failed | 19:24 |
| weshay | rfolco|ruck, I'm rekicking pike/queens rdo p1 jobs quickstart.sh should be fixed | 19:25 |
| rfolco|ruck | weshay, thx for doing this | 19:25 |
| rfolco|ruck | weshay, will look fs001 and 35 | 19:25 |
| rlandy | rfolco|ruck: wrt your comments on https://review.openstack.org/#/c/587228/2/playbooks/tripleo-ci/templates/toci_gate_test.sh.j2@229 | 19:44 |
| rlandy | ^^ I am nit sure | 19:44 |
| rlandy | not | 19:44 |
| rlandy | upgrades still has this included | 19:44 |
| rfolco|ruck | I think upgrades job has been added before we moved to zuulv3 with required-projects and etc | 19:45 |
| rfolco|ruck | if zuul already does that, why you have to manually gate it ? | 19:46 |
| rlandy | rfolco|ruck: afaict from before, the changes were not tested without it | 19:46 |
| rlandy | irrespective of the fact that the local repo is there | 19:46 |
| rlandy | the roles need to be copied correctly | 19:46 |
| rfolco|ruck | rlandy, probably browbeat is expecting to be run from there instead of the zuul src place | 19:47 |
| rlandy | rfolco|ruck: how would we fix that? | 19:47 |
| rfolco|ruck | I might be wrong, would need to understand how browbeat runs and change its call to the new workspace where we copy browbeat that zuul clones | 19:48 |
| rfolco|ruck | rlandy, please paste me links again for the patches or merged code that runs browbeat and I'll make more accurate comments there | 19:49 |
| rlandy | all roles and playbooks etc, are copied via the setup.cfg | 19:49 |
| rlandy | rfolco|ruck: should be the same as extras | 19:49 |
| * rlandy will paste in a bit | 19:49 | |
| rfolco|ruck | thx | 19:49 |
| rfolco|ruck | brb | 19:49 |
| rlandy | just setting up a reproducer for rasca's work | 19:49 |
| weshay | rfolco|ruck, this probably infra https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0ec0f44/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_03_30 | 19:52 |
| weshay | rlandy, have you seen this? | 20:03 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/80c01cd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_01_07 | 20:03 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/fa46166/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_02_56 | 20:04 |
| * rlandy looks | 20:05 | |
| rlandy | no - did something change? | 20:05 |
| rlandy | overcloud deploy | 20:06 |
| rlandy | oh | 20:06 |
| rlandy | weshay: this only in periodic? | 20:06 |
| rlandy | so ... resources.StorageSubnet.properties.allocation_pools[0].start: "172.18.0.10" does not validate ip_addr (constraint not found) comes from network_environment | 20:07 |
| rlandy | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/network/multiple-nics/network-environment.yaml#L17 | 20:08 |
| rlandy | weshay: ^^ | 20:08 |
| rlandy | line 15 actually | 20:09 |
| weshay | rlandy, ya.. it's across multiple jobs | 20:09 |
| weshay | and ipv6 | 20:09 |
| rlandy | so that's the complaint | 20:09 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/fa46166/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_02_56 | 20:09 |
| weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0ec0f44/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-31_19_03_30 | 20:10 |
| rlandy | state chHeat Stack create failed. | 20:13 |
| rlandy | chHeat? | 20:13 |
| rlandy | ok - so it's not any particular subnet | 20:14 |
| rlandy | it's the verification thereof | 20:14 |
| rfolco|ruck | weshay, tempest tests runs nested virt, so kvm-intel module must be present here http://logs.openstack.org/28/585528/8/check/tripleo-ci-centos-7-3nodes-multinode/cd65ccf/logs/undercloud/var/log/extra/lsmod.txt.gz | 20:30 |
| weshay | rfolco|ruck, k | 20:30 |
| weshay | rfolco|ruck, can you check if it's present in a job that is passing that test | 20:31 |
| rfolco|ruck | I got the undercloud lsmod, tempest runs from there... I guess | 20:32 |
| rfolco|ruck | or should check subnode (ctrller) | 20:32 |
| rfolco|ruck | weshay, yes! http://logs.openstack.org/55/587155/1/check/tripleo-ci-centos-7-3nodes-multinode/dff7051/logs/subnode-2/var/log/extra/lsmod.txt.gz | 20:35 |
| weshay | rfolco|ruck, and that's from a working job? | 20:36 |
| rfolco|ruck | exactly, working job has kvm-intel module | 20:36 |
| rfolco|ruck | https://docs.openstack.org/devstack/latest/guides/devstack-with-nested-kvm.html | 20:36 |
| rfolco|ruck | need to check why 1st level vms are not loading it in 80% of the jobs... perhaps this carries from host | 20:37 |
| *** zul has quit IRC | 20:37 | |
| *** zul has joined #oooq | 20:38 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 20:39 |
| rfolco|ruck | weshay, working job ran on a Intel Core Processor (Haswell, no TSX) | 20:39 |
| rfolco|ruck | failed one on Intel Xeon E312xx (Sandy Bridge) | 20:40 |
| weshay | ah | 20:40 |
| rfolco|ruck | weshay, https://bugs.launchpad.net/tripleo/+bug/1784017/comments/7 | 20:47 |
| openstack | Launchpad bug 1784017 in tripleo "Build of instance was re-scheduled: invalid argument: could not find capabilities for domaintype=kvm" [Critical,In progress] - Assigned to Rafael Folco (rafaelfolco) | 20:47 |
| weshay | hrm | 20:49 |
| rfolco|ruck | believe me nested virt runs 30x slower where I come from, this won't make our jobs much faster I believe | 20:49 |
| weshay | rfolco|ruck, it would be better if we could be smarter about when we add that setting | 20:49 |
| rfolco|ruck | thats why devstack kept qemu as default | 20:49 |
| rfolco|ruck | weshay, enable kvm when module kvm-intel is present... this ? | 20:50 |
| weshay | ya.. something along those lines | 20:50 |
| *** vinaykns has quit IRC | 20:52 | |
| weshay | rfolco|ruck, what ya think? | 20:56 |
| weshay | do-able? | 20:56 |
| weshay | should be right? | 20:56 |
| rfolco|ruck | weshay, yes, think so, trying to implement | 20:56 |
| weshay | rfolco|ruck, thanks man! | 20:56 |
| rlandy | doc/source/feature-configuration.rst will kill me in the end | 20:56 |
| rlandy | merge nightmare | 20:56 |
| weshay | rlandy, leave that for me | 20:56 |
| weshay | rlandy, I'll do it, don't waste ur time | 20:57 |
| rlandy | weshay: I have to fix it - I need the review for the reproducer | 20:57 |
| * rlandy is just complaining - ignore me | 20:57 | |
| *** florianf has quit IRC | 20:59 | |
| *** jtomasek has joined #oooq | 21:01 | |
| *** jtomasek_ has quit IRC | 21:03 | |
| *** Goneri has quit IRC | 21:08 | |
| *** rfolco|ruck is now known as rfolco|off | 21:09 | |
| *** yolanda has quit IRC | 21:25 | |
| *** myoung has quit IRC | 21:28 | |
| *** agopi has quit IRC | 21:30 | |
| weshay | rfolco|off, can you cover the program call tomorrow morning? | 21:31 |
| weshay | rfolco|off, status is RED for master, that's all we care about | 21:31 |
| weshay | rfolco|off, blockers include https://bugs.launchpad.net/tripleo/+bug/1784712 | 21:31 |
| openstack | Launchpad bug 1784712 in tripleo "ExternalNetwork, InternalApiNetwork, StorageNetwork fail to validate ip_addr" [Critical,Triaged] | 21:31 |
| weshay | https://trello.com/c/hkvfxAdX/667-cixtripleoci-rdo-software-factory-3rd-party-jobs-failing-due-to-instance-nodefailure | 21:32 |
| rfolco|off | weshay, ack | 21:39 |
| weshay | rfolco|off, anything you see w/ this filter https://trello.com/b/j4IcIomh/production-chain-escalation?menu=filter&filter=label:TripleO-master | 21:39 |
| *** vinaykns has joined #oooq | 21:48 | |
| rlandy | weshay: ok, so I'll put in a job to parallel https://softwarefactory-project.io/r/#/c/12967/ ... not sure about the test definition though | 21:55 |
| weshay | aye | 21:56 |
| *** vinaykns has quit IRC | 22:05 | |
| *** jtomasek has quit IRC | 22:21 | |
| rlandy | weshay: https://softwarefactory-project.io/r/#/c/13256/ - seem correct? | 22:32 |
| rlandy | adding depends to job | 22:32 |
| rlandy | https://review.rdoproject.org/r/#/c/15074/ - didn't work :( | 22:36 |
| *** vinaykns has joined #oooq | 22:38 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 22:39 |
| *** vinaykns has quit IRC | 22:44 | |
| rlandy | weshay: https://review.rdoproject.org/r/15097 - sanity check pls | 22:44 |
| rlandy | fixing required projects ... | 22:51 |
| *** agopi has joined #oooq | 23:16 | |
| *** tosky has quit IRC | 23:35 | |
| *** vinaykns has joined #oooq | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
