| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 01:55 |
|---|---|---|
| *** hamzy__ is now known as hamzy | 03:36 | |
| *** saneax has joined #oooq | 03:52 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 03:55 |
| *** ratailor has joined #oooq | 04:51 | |
| *** ratailor has quit IRC | 04:52 | |
| *** saneax has quit IRC | 05:02 | |
| *** agopi has joined #oooq | 05:05 | |
| *** ratailor has joined #oooq | 05:30 | |
| *** gkadam has joined #oooq | 05:45 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 05:55 |
| *** zul has quit IRC | 06:04 | |
| *** ykarel has joined #oooq | 06:08 | |
| *** apetrich has quit IRC | 06:25 | |
| *** apetrich has joined #oooq | 06:40 | |
| ykarel | marios_|ruck, ssbarnea|rover is this ovb issue known: RuntimeError: Found different numbers of baremetal and bmc ports. seeing in 3 environments: http://38.145.33.166/testenv-worker.log | 06:47 |
| ykarel | tracedback https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/24f2fdd/job-output.txt.gz#_2018-12-03_05_09_29_708328 | 06:48 |
| *** kopecmartin|off is now known as kopecmartin | 06:53 | |
| marios_|ruck | ykarel: doesn't sound familiar and i haven't seen it mysefl let me check the etherpad for any updates from weekend | 06:57 |
| ykarel | marios_|ruck, ack | 07:00 |
| marios_|ruck | ykarel: looks like a new one tho looking in http://cistatus.tripleo.org/promotion/ it doesn't show those fails yet | 07:01 |
| marios_|ruck | ykarel: i'll file the bug in a bit thanks for ping | 07:01 |
| ykarel | marios_|ruck, ack | 07:01 |
| marios_|ruck | ykarel: looks like it had one green run yesterday for that so it is new | 07:02 |
| marios_|ruck | ykarel: is this blocking you? | 07:02 |
| ykarel | marios_|ruck, not yet, | 07:02 |
| ykarel | but if it affects most of the ovb job then it's critical | 07:03 |
| marios_|ruck | ykarel: ack | 07:03 |
| *** jfrancoa has joined #oooq | 07:09 | |
| *** quiquell|off is now known as quiquell | 07:10 | |
| quiquell | marios_|ruck, ykarel: o/ | 07:10 |
| ykarel | o/ | 07:10 |
| *** skramaja has joined #oooq | 07:11 | |
| marios_|ruck | o/ | 07:12 |
| quiquell | ykarel, marios_|ruck: scenario002 for standalone, env file https://review.openstack.org/#/c/618537 | 07:19 |
| quiquell | ykarel:, marios_|ruck: Onlythis one is missing the others are already merged | 07:20 |
| marios_|ruck | quiquell: ack | 07:20 |
| marios_|ruck | quiquell: in a sec | 07:20 |
| quiquell | marios_|ruck: We have to find someone else to the other +2 | 07:20 |
| marios_|ruck | quiquell: k go pimp it in tripleo | 07:20 |
| marios_|ruck | :) | 07:20 |
| ykarel | quiquell, ack | 07:21 |
| quiquell | marios_|ruck: I am super bad selling shit | 07:21 |
| quiquell | marios_|ruck: le'ts merge non votings jobs too | 07:22 |
| quiquell | removing Depends-On | 07:22 |
| quiquell | Ahhh no tempest stuff we have to get tempest stuff right first | 07:23 |
| *** rascasoft has joined #oooq | 07:30 | |
| *** quiquell is now known as quiquell|brb | 07:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 07:55 |
| *** jtomasek has joined #oooq | 08:15 | |
| *** jtomasek has quit IRC | 08:15 | |
| *** jtomasek has joined #oooq | 08:16 | |
| *** quiquell|brb is now known as quiquell | 08:18 | |
| *** amoralej|off is now known as amoralej | 08:23 | |
| *** ykarel is now known as ykarel|lunch | 08:26 | |
| marios_|ruck | ykarel|lunch: fyi https://bugs.launchpad.net/tripleo/+bug/1806346 but from a quick survey just now i couldn't find more examples, did you say you had 3? | 08:30 |
| openstack | Launchpad bug 1806346 in tripleo "periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens job failed to retrieve environment - testenv-client - ERROR - Couldn't retrieve env" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 08:30 |
| *** holser_ has joined #oooq | 08:35 | |
| arxcruz | rascasoft: i'm very angry with you :( | 08:37 |
| ykarel|lunch | marios_|ruck, yes 3, look for error i shared in http://38.145.33.166/testenv-worker.log | 08:38 |
| rascasoft | arxcruz, be kind man, Christmas is near :) | 08:39 |
| arxcruz | rascasoft: and which gift you gave me? a good bye | 08:39 |
| rascasoft | arxcruz, LOL there are goodbyes and goodbyes ;) | 08:40 |
| *** ccamacho has joined #oooq | 08:45 | |
| ssbarnea|rover | marios: morning! | 08:47 |
| *** tosky has joined #oooq | 08:48 | |
| marios | o/ ssbarnea|rover | 08:48 |
| ssbarnea|rover | quiquell: we need workflow on https://review.openstack.org/#/c/621259/ -- as this is part of the timeout fix on upgrades. | 08:50 |
| quiquell | ssbarnea|rover: ack, looking | 08:53 |
| *** holser_ has quit IRC | 08:53 | |
| *** holser_ has joined #oooq | 08:54 | |
| *** holser_ has quit IRC | 08:54 | |
| *** holser_ has joined #oooq | 08:55 | |
| quiquell | ssbarnea|rover: done | 08:56 |
| ssbarnea|rover | thanks. | 08:56 |
| ssbarnea|rover | marios: which meetings do we need to attend? | 08:57 |
| marios_|ruck | ssbarnea|rover: quick sync call now? | 08:58 |
| marios_|ruck | (or in bit if you want coffee first:) | 08:59 |
| ssbarnea|rover | marios_|ruck: https://bluejeans.com/2655417928 - already have the coffee. | 08:59 |
| marios_|ruck | ssbarnea|rover: joining | 08:59 |
| *** bogdando has joined #oooq | 09:09 | |
| *** chem has joined #oooq | 09:10 | |
| ssbarnea|rover | marios_|ruck: those errors were from periodic jobs and are visible logs.rdo... not loaded to logstash. i wonder if we can start feeding logstash logs from periodic as it would very useful. | 09:31 |
| ssbarnea|rover | i wonder if this is because we didn't had time to do it or because that is not desired | 09:32 |
| ssbarnea|rover | quiquell: you were the primary contact for grafana? | 09:35 |
| quiquell | ssbarnea|rover: dashboard-ci yep | 09:36 |
| ssbarnea|rover | quiquell: have you considered spliting the cockpit into multiple boards? it grew too much and is confusing to load it. my personal view is that we could split it in 3-4 ones, each targeted for one specific area of investigation. | 09:39 |
| quiquell | ssbarnea|rover: more than dashboard we have do collapsable ones so they are in the same dashboard | 09:41 |
| quiquell | ssbarnea|rover: right now most of the cockpit is in main | 09:41 |
| *** ykarel|lunch is now known as ykarel | 09:41 | |
| ssbarnea|rover | also scrolling through page is awful on mac because of the iframes but i doubt this is somethign you can do, that being a design issue of grafana lists. | 09:42 |
| quiquell | ssbarnea|rover: we have Main, Zuul Jobs, Promotions, Issues, RDO cloud performance | 09:42 |
| quiquell | ssbarnea|rover: maybe 8 first ones have to be main, and create more collapsable sections for the other stuff | 09:42 |
| quiquell | ssbarnea|rover: Having multiple dashboard means multiple URLs means people forget about them | 09:43 |
| *** sshnaidm|off is now known as sshnaidm | 09:43 | |
| quiquell | ssbarnea|rover: Did you try to collapse stuff so you only see Main ? | 09:43 |
| quiquell | ssbarnea|rover: Main is not that big | 09:43 |
| ssbarnea|rover | i *want* mulltiple URLs so I can share a link that points to the a meaningful page, not one with 100 sub-windows where I can write instruction on how to find the right window. | 09:47 |
| quiquell | ssbarnea|rover: grafana have the share option | 09:48 |
| quiquell | ssbarnea|rover: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=1543657722281&to=1543830522282&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=rocky&var-releases=queens&var-releases=pike&panelId=144&fullscreen | 09:48 |
| quiquell | For example | 09:48 |
| ssbarnea|rover | quiquell: is not big, is *huge*: is ~12 screens long on my 5K screen. A dashboard is supposed to not even have a scrollbar as their primary use was to have them on TV screens. | 09:49 |
| quiquell | ssbarnea|rover: but maybe you are right, let's comment today with the rest of the team, about what they prefer | 09:49 |
| ssbarnea|rover | i see a "promotions" section but there is no page that contains only promotions, no way to send a link to someone to look at promotions, only. | 09:50 |
| ssbarnea|rover | the only option I see is to colapse promotions, nothing else. | 09:50 |
| ssbarnea|rover | quiquell: just to be clear, don't get my remarks in a negative way, I LOVE what you did | 09:51 |
| quiquell | ssbarnea|rover: I see there is something call Folders, at grafana | 09:51 |
| sshnaidm | ssbarnea|rover, I don't understand - what's your issue with grafana? | 09:51 |
| ssbarnea|rover | not sure how these are defined by I wonder if we can have both: split dashboards and still having a "kitchensink" one | 09:51 |
| quiquell | ssbarnea|rover: Maybe organizing with the we have separate URLs but the same entry point too | 09:52 |
| quiquell | ssbarnea|rover: that would be perfect | 09:52 |
| sshnaidm | ssbarnea|rover, what are you trying to do | 09:52 |
| sshnaidm | ssbarnea|rover, do you use laptop screen to look at grafana? | 09:55 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 09:55 |
| ssbarnea|rover | my problem: too much information in a single page, if I want to send an url to one graph to someone and ask him take a look at... i cannot do it. i cannot have one browser tab focused on promotions and another one focus on rdo cloud performance. At this moment all this information is in a single page/url. --- I do think we need to be able split it few focused areas. (clearly not one per graph) | 09:56 |
| sshnaidm | ssbarnea|rover, give you some time as ruck and rover, you'll see that you don't need to focus on rdo cloud performance, all you need is in Main. | 09:57 |
| *** derekh has joined #oooq | 09:57 | |
| sshnaidm | ssbarnea|rover, this config is based on experience of generations of rucks and rovers, just give it a chance | 09:58 |
| ssbarnea|rover | sshnaidm: that's what I was trying to say, lots of info i dont need added to default, provably also adding extra load on server client. and no way to pin sections to different browser tabs. | 09:59 |
| sshnaidm | ssbarnea|rover, you can share graphs, just click "share" | 09:59 |
| ssbarnea|rover | because is a ... one page web application. | 09:59 |
| ssbarnea|rover | sshnaidm: a single graph, not a section | 09:59 |
| ssbarnea|rover | the idea is to be able to have one web page (url) per section. | 10:00 |
| sshnaidm | ssbarnea|rover, you'd be surprised that you DO need this info later | 10:00 |
| sshnaidm | ssbarnea|rover, just before changing something try to use it in your r&r weeks, and then let's talk about it in retrospection | 10:01 |
| sshnaidm | ssbarnea|rover, all this is done not randomly, you can believe me | 10:01 |
| quiquell | ssbarnea|rover: Yep agree with sshnaidm is the way to do it | 10:02 |
| ssbarnea|rover | sshnaidm: i don't ask about doing any change now, i just wanted to give some feedback and see what others think abotu ti. | 10:02 |
| ssbarnea|rover | yep, i will make a note for retro. | 10:02 |
| sshnaidm | ssbarnea|rover, we'd like to have your feedback after heavy trying it, in introspection | 10:02 |
| ssbarnea|rover | also multiple dashboards is not in conflict with keeping the current one as it is. one will all and few others which are subsets. i bet we can doing without duplicating code. | 10:03 |
| quiquell | we can investigate Directories | 10:03 |
| quiquell | ssbarnea|rover: Good news, linting merged !!! | 10:04 |
| quiquell | :-) | 10:04 |
| ssbarnea|rover | lets go back to current issues: testenv-client - ERROR - Couldn't retrieve env on https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/24f2fdd/job-output.txt.gz#_2018-12-03_05_11_07_677638 | 10:05 |
| ssbarnea|rover | looks like infra to me but I don't know what needs to be done, who to ping. | 10:05 |
| quiquell | ssbarnea|rover: That could be that we have problems with RDO nodepool | 10:06 |
| sshnaidm | ssbarnea|rover, it's SINGLE point of information and it's intentionally, and it should be like single point, not a thousand pages like it was before you got to the team, you're lucky now | 10:06 |
| quiquell | ssbarnea|rover: best contack is kforde | 10:06 |
| *** dtantsur has joined #oooq | 10:07 | |
| sshnaidm | quiquell, ssbarnea|rover before contacting kforde need to check what's wrong in te-broker | 10:07 |
| quiquell | sshnaidm: ack | 10:07 |
| sshnaidm | ssbarnea|rover, http://38.145.33.166/testenv-worker.log | 10:08 |
| quiquell | ssbarnea|rover: have fix stuff, re-run tox -e linters and it's not taking changes | 10:08 |
| quiquell | ssbarnea|rover: Do I have to git review it ? | 10:08 |
| ssbarnea|rover | quiquell: you need to commit change before running the linting task, that's the only trick. | 10:16 |
| ssbarnea|rover | just commit, not git-review | 10:16 |
| ssbarnea|rover | because by default it does not include local unmerged changes | 10:16 |
| quiquell | ssbarnea|rover: ack | 10:17 |
| quiquell | ssbarnea|rover: thanks | 10:17 |
| quiquell | ssbarnea|rover: make sense, maybe we can even for git review so you don't forget about adding changes | 10:17 |
| quiquell | marios_|ruck, sshnaidm: for the zuul repro we need this https://review.openstack.org/#/c/619488/ | 10:37 |
| quiquell | humm wait have to fix it | 10:37 |
| marios_|ruck | quiquell: ack will check in a bit added to reviews queue | 10:41 |
| sshnaidm | quiquell, omg | 10:42 |
| sshnaidm | quiquell, seems like having this files in job config wasn't the best idea | 10:42 |
| quiquell | sshnaidm: ctx ? | 10:43 |
| sshnaidm | quiquell, what's that? | 10:44 |
| quiquell | sshnaidm: I mean I need context about last stuff you wrote | 10:44 |
| quiquell | sshnaidm: You mean the review ? | 10:44 |
| sshnaidm | quiquell, yeah | 10:44 |
| quiquell | sshnaidm: What did you found ? | 10:45 |
| quiquell | sshnaidm: You mean that do standalone_custom_env_files is problematic somehow ? | 10:45 |
| sshnaidm | quiquell, I mean this logic there brings complexity | 10:46 |
| sshnaidm | and -10 to maintainability | 10:46 |
| quiquell | sshnaidm: Yep :-( | 10:46 |
| quiquell | sshnaidm: It's just adding a prefix to a list | 10:46 |
| quiquell | :-) | 10:46 |
| quiquell | ansible way | 10:46 |
| quiquell | the review just "simplify" it :-) replacing ^ | 10:47 |
| *** panda|pto is now known as panda | 10:47 | |
| quiquell | sshnaidm, marios_|ruck: now it's reviewable https://review.openstack.org/619488 | 10:49 |
| quiquell | ssbarnea|rover: fixed the regex stuff ^ | 10:49 |
| sshnaidm | quiquell, I think you need a jinja there | 10:53 |
| sshnaidm | quiquell, otherwise it'll be just text multiline variable | 10:53 |
| sshnaidm | quiquell, try to test it locally | 10:53 |
| quiquell | sshnaidm: I have just test something similar in other review, let me test exactly this | 10:54 |
| marios_|ruck | ssbarnea|rover: : did you get a chance to look into that tempest issue. about to do so | 10:56 |
| marios_|ruck | ssbarnea|rover: otherwise i'll check the other one | 10:56 |
| marios_|ruck | quiquell: ack thanks in bit | 10:56 |
| ssbarnea|rover | marios_|ruck: look at tempest | 10:57 |
| sshnaidm | quiquell, just add {{ }} and will be fine | 10:59 |
| marios_|ruck | ssbarnea|rover: ack | 11:00 |
| quiquell | sshnaidm: yep, fixing | 11:06 |
| quiquell | sshnaidm: thanks | 11:06 |
| *** sshnaidm has quit IRC | 11:16 | |
| *** sshnaidm has joined #oooq | 11:16 | |
| arxcruz | marios_|ruck: correct me if i'm wrong, but in https://review.openstack.org/620596 does it need the featureset? Since in vars.featureset is already set to 052 ? | 11:17 |
| *** sshnaidm has quit IRC | 11:18 | |
| *** rfolco has joined #oooq | 11:18 | |
| *** sshnaidm has joined #oooq | 11:19 | |
| *** quiquell is now known as quiquell|brb | 11:21 | |
| marios_|ruck | arxcruz: ack thanks please add to review? | 11:26 |
| arxcruz | marios_|ruck: sure, i didn't on review because i might be wrong that's why i wanted to check with you first :) | 11:26 |
| arxcruz | done | 11:28 |
| marios_|ruck | arxcruz: i didn't check to be honest but will do you could be right | 11:28 |
| arxcruz | panda: do you have the url to featureset number reservation ? | 11:34 |
| *** ratailor has quit IRC | 11:41 | |
| *** hamzy_ has joined #oooq | 11:42 | |
| *** hamzy has quit IRC | 11:42 | |
| *** quiquell|brb is now known as quiquell | 11:45 | |
| panda | arxcruz: https://etherpad.openstack.org/p/quickstart-featuresets | 11:50 |
| arxcruz | panda: thanks | 11:50 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 11:55 |
| *** ykarel is now known as ykarel|afk | 12:03 | |
| marios | ssbarnea|rover: for the tempest i didn't file it because i can only find one case of it on the 2nd. i prepared the text here http://paste.openstack.org/raw/736569/ if we see it again | 12:07 |
| marios | ssbarnea|rover: just back from food, did you check the container prep issue ? or should i look there | 12:08 |
| ssbarnea|rover | marios: does "failed to reach SAVING state" reproduce? always/often ? | 12:10 |
| marios | ssbarnea|rover: where is that one? not following | 12:11 |
| *** ykarel|afk is now known as ykarel | 12:11 | |
| ssbarnea|rover | you posted it two lines above, anyway same problem: rdo errors which we don't know how spread are because we don't have log stash enabled. i will try to get the ok from infra to allow them to upload logs. | 12:13 |
| ykarel | ssbarnea|rover, for logstash, rdo jobs(third party, periodic etc) have logstash enabled https://review.rdoproject.org/app/kibana | 12:13 |
| ssbarnea|rover | ykarel: thanks! | 12:14 |
| ykarel | ssbarnea|rover, and to debug such tempest issue i think we need to get logs from overcloud | 12:15 |
| ykarel | can u have look at clearing those | 12:15 |
| ykarel | ssbarnea|rover, we need https://review.openstack.org/#/c/618669/ for overcloud logs | 12:15 |
| ykarel | panda, can we get ^^ | 12:16 |
| marios | ssbarnea|rover: ah you mean the new issue from ykarel today | 12:16 |
| marios | ssbarnea|rover: apparently three times in http://38.145.33.166/testenv-worker.log | 12:17 |
| marios | ssbarnea|rover: so i'm gonna poke at the container prep thing (added note on the etherpad about the tempest for now fyi) | 12:17 |
| ykarel | marios, ssbarnea|rover seems to be talking about fs020 tempest issue u posted: http://paste.openstack.org/raw/736569 | 12:18 |
| *** ykarel is now known as ykarel|afk | 12:19 | |
| marios | thanks ykarel|afk ssbarnea|rover i could only find one, that is why i didn't file the bug | 12:21 |
| ssbarnea|rover | ykarel|afk: marios : i tried to search for text from http://paste.openstack.org/raw/736569/ at https://review.rdoproject.org/app/kibana but nothing found, so is useless to me. now the question is why is not there. | 12:22 |
| ykarel|afk | ssbarnea|rover, /me leaving for a while, will check later, u can ask jpena on #rdo, it should work | 12:23 |
| ssbarnea|rover | thanks, i will do. | 12:23 |
| sshnaidm | How to debug environment failure of OVB job - https://www.youtube.com/watch?v=BQG6Il-MxeU&hd=1&cc_load_policy=1 | 12:27 |
| sshnaidm | ssbarnea|rover, marios ^^ | 12:27 |
| quiquell | sshnaidm: Who install roles here ? workspace/.quickstart//usr/local/share/ansible/roles/ | 12:28 |
| quiquell | sshnaidm: I think they not in sync with ZUUL_CHANGES :-/ | 12:28 |
| sshnaidm | quiquell, quickstart.sh or just pip | 12:28 |
| marios | ssbarnea|rover: thanks | 12:28 |
| marios | ssbarnea|rover: nice | 12:28 |
| marios | er sshnaidm ^ | 12:28 |
| sshnaidm | quiquell, need to check if they're installed from cloned directory | 12:29 |
| sshnaidm | quiquell, and if cloned directory has these changes | 12:30 |
| *** ykarel|afk has quit IRC | 12:30 | |
| quiquell | sshnaidm: I think the requirements.txt from tq is broken | 12:30 |
| sshnaidm | quiquell, you mean quickstart-extras-requirement.txt? | 12:31 |
| sshnaidm | quiquell, requirement is just for python modules | 12:31 |
| quiquell | sshnaidm: yep git+https://git.openstack.org/openstack/tripleo-quickstart-extras/#egg=tripleo-quickstart-extras | 12:31 |
| sshnaidm | quiquell, where do you see it? | 12:31 |
| quiquell | sshnaidm: Well at the new tripleo-ci-reproducer | 12:31 |
| sshnaidm | quiquell, need to check what's problem with reproducer.. | 12:32 |
| quiquell | sshnaidm: We are suppose to replace that with the home/zuul/src ... whatever | 12:32 |
| sshnaidm | quiquell, yep | 12:32 |
| quiquell | sshnaidm: I think it's general issue | 12:32 |
| sshnaidm | quiquell, ? | 12:33 |
| quiquell | sshnaidm: Cannot be problem would be too big, like fixes not executing | 12:33 |
| quiquell | sshnaidm: Don't we need this ? https://review.openstack.org/621562 | 12:40 |
| sshnaidm | quiquell, nope, it worked without that | 12:40 |
| sshnaidm | quiquell, which job do you reproduce | 12:42 |
| quiquell | sshnaidm: But this is wrong http://logs.openstack.org/56/618056/10/check/tripleo-ci-fedora-28-standalone/42d5b20/job-output.txt.gz#_2018-11-28_13_06_01_156732 | 12:43 |
| quiquell | sshnaidm: well I see this later on http://logs.openstack.org/56/618056/10/check/tripleo-ci-fedora-28-standalone/42d5b20/job-output.txt.gz#_2018-11-28_13_07_26_324964 | 12:45 |
| sshnaidm | quiquell, this should work: http://logs.openstack.org/59/621259/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/a4f796b/job-output.txt.gz#_2018-12-01_14_27_16_927737 | 12:46 |
| quiquell | sshnaidm: I think I am missing some role | 12:48 |
| quiquell | sshnaidm: at the tripleo-ci-reproducer | 12:48 |
| quiquell | sshnaidm: I don't see the orkspace path set to: /home/zuul/src/git.openstack.or... | 12:48 |
| sshnaidm | quiquell, I think it's in one of legacy playbooks, isn't it? | 12:49 |
| quiquell | sshnaidm: is the zuul cloner | 12:49 |
| quiquell | sshnaidm: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/fetch-zuul-cloner/templates/zuul-cloner-shim.py.j2 | 12:49 |
| quiquell | sshnaidm: If we fix the requirements we don't need this | 12:49 |
| sshnaidm | quiquell, well, it has some history.. | 12:50 |
| quiquell | sshnaidm: I didn't add it to our base job for it, do we need it ? | 12:50 |
| sshnaidm | quiquell, do we need what? | 12:51 |
| quiquell | sshnaidm: execute "fetch-zuul-cloner" at tripleo-ci-reproducer base job | 12:51 |
| quiquell | sshnaidm: I excluded it in purpose | 12:51 |
| quiquell | sshnaidm: If we fix the tqe requirements maybe it's not needed to add to base job | 12:51 |
| sshnaidm | quiquell, maybe we need to discuss it.. hacking reqs is not really a fix, it could affect a lot of things around | 12:53 |
| sshnaidm | quiquell, would be great to find a better way to handle changes in requirements | 12:54 |
| quiquell | sshnaidm: ack, but I think the review is legit | 12:54 |
| *** rlandy has joined #oooq | 12:54 | |
| quiquell | sshnaidm: https://review.openstack.org/#/c/621562/ or it could be problematic ? | 12:54 |
| quiquell | rlandy: o/ | 12:54 |
| rlandy | quiquell: hello | 12:55 |
| quiquell | rlandy: So you have a working zuul repro ? | 12:55 |
| rlandy | quiquell: I do indeed | 12:56 |
| rlandy | quiquell: I have a libvirt working situation as well | 12:56 |
| quiquell | rlandy: btw found issue with fetch-zuul-cloner | 12:56 |
| quiquell | rlandy: no kidding | 12:56 |
| quiquell | :-) | 12:56 |
| rlandy | quiquell: left notes on that now scary etherpad | 12:57 |
| quiquell | rlandy: ack | 12:57 |
| rlandy | quiquell: what was wrong with zuul-cloner? | 12:57 |
| rlandy | quiquell: after meeting we should chat about how to go forward | 12:57 |
| quiquell | rlandy: I was not adding it at base job | 12:57 |
| quiquell | rlandy: I can, no problem, excluded that in purpose | 12:57 |
| quiquell | rlandy: but fi we don't have fetch-zuul-cloner tqe is not correctly installed | 12:58 |
| rlandy | quiquell: hmm ... I would like to use the same jobs upstream is using | 12:58 |
| sshnaidm | ssbarnea|rover, who is a ruck? | 12:58 |
| quiquell | rlandy: Going to add it | 12:58 |
| rlandy | quiquell: the less we deviate the better | 12:58 |
| sshnaidm | ssbarnea|rover, any known problem with ovb jobs? Seems like all fail | 12:58 |
| rlandy | even if it runs unnecessary things | 12:58 |
| rlandy | quiquell: ^^ we should change upstream then rather than just the reproducer workflow | 12:59 |
| rlandy | quiquell: anyways, let's chat after meeting - it's looking good I think | 12:59 |
| rlandy | sshnaidm: marios_|ruck | 13:00 |
| sshnaidm | marios, can you please add "|ruck"? | 13:01 |
| sshnaidm | marios, also better to join #rhos-ops in internal irc | 13:07 |
| *** ade_lee has quit IRC | 13:07 | |
| weshay | rlandy++ | 13:09 |
| hubbot1 | weshay: rlandy's karma is now 39 | 13:09 |
| weshay | quiquell++ | 13:09 |
| hubbot1 | weshay: quiquell's karma is now 15 | 13:09 |
| weshay | sshnaidm, they are failing again? | 13:09 |
| weshay | they were all passing last night | 13:09 |
| sshnaidm | weshay, yeah, talking with kforde | 13:10 |
| weshay | sshnaidm, failing at the testenv_broker | 13:10 |
| weshay | we even had two promotions over the weekend | 13:10 |
| sshnaidm | weshay, fail in various points | 13:10 |
| sshnaidm | weshay, yeah, we know it works good in weekends | 13:11 |
| weshay | lolz | 13:11 |
| weshay | ya | 13:11 |
| sshnaidm | weshay, maybe let's run promotions in weekends only :) | 13:11 |
| weshay | maybe let's only work weekends | 13:11 |
| sshnaidm | weshay, I'm kinda doing it :D | 13:11 |
| marios_|ruck | sshnaidm: am here | 13:12 |
| marios_|ruck | o/ | 13:12 |
| *** holser_ has quit IRC | 13:13 | |
| sshnaidm | weshay, what did you use to create a screencast? | 13:13 |
| weshay | oh.. https://asciinema.org/ | 13:15 |
| ssbarnea|rover | sshnaidm: ovb jobs should work now after router reboot. | 13:16 |
| weshay | marios_|ruck, ssbarnea|rover so.. sounds like rdo-cloud buckled and is now failing.. keep status on that w/ kforde. thanks sshnaidm. rdo phase 1.. failed deploy w/ "no hosts" could be infra.. rekicking https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ | 13:16 |
| *** ykarel|afk has joined #oooq | 13:18 | |
| marios | weshay: ack added this one this morning from yatin ping https://bugs.launchpad.net/tripleo/+bug/1806346 (ovb) in the etherpad fyi | 13:18 |
| openstack | Launchpad bug 1806346 in tripleo "periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens job failed to retrieve environment - testenv-client - ERROR - Couldn't retrieve env" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 13:18 |
| *** ykarel|afk is now known as ykarel | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:22 | |
| *** agopi has quit IRC | 13:24 | |
| ssbarnea|rover | sshnaidm: is http://cistatus.tripleo.org/ again outdated? https://review.openstack.org/#/c/621259/ failed with timeout (the one you did recheck) but that's not visible on cistatus. | 13:26 |
| sshnaidm | ssbarnea|rover, not so fast.. it takes time | 13:27 |
| rlandy | sshnaidm: can we talk a few minutes about ovb? | 13:28 |
| sshnaidm | rlandy, sure | 13:28 |
| rlandy | quiquell: ^^ you can join | 13:28 |
| quiquell | rlandy: I will want to also ask you guys about fetch-zuul-cloner | 13:28 |
| rlandy | https://bluejeans.com/u/rlandy | 13:28 |
| rlandy | sshnaidm: quiquell; ^^ | 13:29 |
| *** agopi has joined #oooq | 13:30 | |
| *** udesale has joined #oooq | 13:30 | |
| *** zul has joined #oooq | 13:35 | |
| *** holser_ has joined #oooq | 13:40 | |
| *** holser_ has quit IRC | 13:44 | |
| *** holser_ has joined #oooq | 13:44 | |
| *** jaosorior has joined #oooq | 13:47 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 13:55 |
| quiquell | rlandy: Nah forget about te extra_vars at jobs, that was another reeview I cannot find, the one that is present is "extra_tags" :-) | 13:57 |
| rlandy | quiquell: k - I'll hack around a bit | 13:58 |
| rlandy | and see what we can do | 13:58 |
| *** amoralej|lunch is now known as amoralej | 14:00 | |
| *** quiquell is now known as quiquell|lunch | 14:01 | |
| marios_|ruck | ssbarnea|rover: https://bugs.launchpad.net/tripleo/+bug/1806403 i found 2 of those container prep for same image filed it and added to pad | 14:02 |
| openstack | Launchpad bug 1806403 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens fails during container image prepare with missing image tripleoqueens/centos-binary-rabbitmq" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 14:02 |
| marios_|ruck | rfolco: quiquell|lunch wana tlk? about the linting? | 14:02 |
| marios_|ruck | rfolco: quiquell|lunch am gonna look there now | 14:02 |
| marios_|ruck | maybe when quiquell|lunch is back rfolco | 14:03 |
| quiquell|lunch | marios_|ruck: the "environment: " solution has to work | 14:03 |
| marios_|ruck | i'll reply to your comment agree if we don't find better way | 14:03 |
| quiquell|lunch | marios_|ruck: Maybe it's missing son {%- stuff | 14:03 |
| marios_|ruck | ack lets check | 14:03 |
| rfolco | quiquell|lunch, marios_|ruck I left my comment there.... why not just an export in the shell | 14:04 |
| quiquell|lunch | marios_|ruck: Do some local runs of partial code, is the fastest way | 14:04 |
| quiquell|lunch | rfolco: we don't want ":/usr..." | 14:04 |
| marios_|ruck | rfolco: ack it was like that on v3 as we discussed on the call earlier | 14:04 |
| marios_|ruck | rfolco: replying with pointer | 14:04 |
| rfolco | quiquell|lunch, marios_|ruck I'll look at the tempest configs after my doctor appt this afternoon | 14:06 |
| marios_|ruck | rfolco: ack don't worry we'll sort out today | 14:06 |
| marios_|ruck | quiquell|lunch: let me know when you're back we can sync maybe quick call | 14:06 |
| quiquell|lunch | marios_|ruck: ack | 14:06 |
| marios_|ruck | quiquell|lunch: no rush is still early | 14:06 |
| *** weshay is now known as weshay_traveling | 14:09 | |
| marios_|ruck | quiquell|lunch: so actually the only one with a log to point to is v1 https://review.openstack.org/#/c/620556/1//COMMIT_MSG@7 where i had environment but no {% j2 https://review.openstack.org/#/c/620556/1/roles/standalone/tasks/main.yml | 14:15 |
| marios_|ruck | quiquell|lunch: (passing i mean) | 14:17 |
| marios_|ruck | http://logs.openstack.org/20/619520/10/check/tripleo-ci-centos-7-scenario004-standalone/3f3b430/job-output.txt.gz#_2018-11-29_10_48_48_355645 | 14:18 |
| quiquell|lunch | rlandy: Has a crazy idea about libvirt images lifecycle, have added a point to Test Req | 14:23 |
| rlandy | quiquell|lunch: sure - bring on the crazy idea | 14:24 |
| rlandy | Use docker images to start/stop libvirt images? | 14:24 |
| rlandy | quiquell|lunch: ^^?? | 14:24 |
| quiquell|lunch | rlandy: meaning, a docker node, that also runs the nodepool playboks to startup libvirt node :-) | 14:25 |
| quiquell|lunch | rlandy: so everyting can be start stop with docke compose or even scale it up | 14:25 |
| rlandy | quiquell|lunch: sshnaidm: have you guys looked at https://review.rdoproject.org/r/#/c/17633/? | 14:26 |
| rlandy | quiquell|lunch: sshnaidm: was in tristan's comment in our card | 14:28 |
| quiquell|lunch | rlandy: Same stuff but software factory + rdo config | 14:32 |
| rlandy | quiquell|lunch: wdyt? | 14:32 |
| rlandy | quiquell|lunch: will help us? duplicate? | 14:32 |
| quiquell|lunch | rlandy: We have to take into acount libvirt + upstream | 14:33 |
| quiquell|lunch | rlandy: For the rdo config I think | 14:33 |
| quiquell|lunch | rlandy: I mean OVB jobs and all | 14:33 |
| sshnaidm | rfolco, interesting.. | 14:34 |
| rlandy | quiquell|lunch: yep - wanted to see if it helped sshnaidm's issues | 14:34 |
| rlandy | rfolco? or me? | 14:34 |
| rfolco | sshnaidm, what? | 14:34 |
| rfolco | oh | 14:34 |
| sshnaidm | rfolco, oops, to rlandy | 14:34 |
| quiquell|lunch | rlandy: we can dockerize sf X-D | 14:34 |
| sshnaidm | quiquell|lunch, isn't it still? :) | 14:35 |
| sshnaidm | quiquell|lunch, but with podman and buildah! | 14:35 |
| rlandy | sshnaidm: does that review help you at all? | 14:35 |
| quiquell|lunch | sshnaidm: yey !!! :-) | 14:35 |
| sshnaidm | rlandy, yeah, might be helpful, will check it deeper | 14:35 |
| rlandy | marios_|ruck: have any time? would stil like to do something towards scenarios work | 14:36 |
| *** skramaja has quit IRC | 14:37 | |
| *** quiquell|lunch is now known as quiquell | 14:40 | |
| quiquell | marios_|ruck: I am back | 14:40 |
| rlandy | quiquell was never really away :) | 14:41 |
| quiquell | rlandy: I have also being doing the beds | 14:42 |
| rlandy | quiquell: anyways ... when you talk to marios_|ruck, I wanted to do some work on the scenarios - so maybe I'll take one of yours | 14:44 |
| quiquell | rlandy: totally, I am leaving scenario002 | 14:44 |
| * rlandy was not involved at all last sprint | 14:44 | |
| quiquell | rlandy: what's missing is the linting | 14:44 |
| quiquell | rlandy: and adapt tempest from multinode to them | 14:45 |
| rlandy | quiquell: k - will talk with marios_|ruck when he has time | 14:45 |
| quiquell | rlandy: ack | 14:45 |
| rlandy | whatever task seems the most appropriate | 14:45 |
| rlandy | otherwise I'll take the doc | 14:45 |
| rlandy | panda: you around?? | 14:45 |
| rlandy | panda: need to chat with you about card separation for https://tree.taiga.io/project/tripleo-ci-board/task/451 | 14:46 |
| rlandy | quiquell, sshnaidm and I all have numerous tasks on that card | 14:46 |
| rlandy | can that card become the user story? | 14:47 |
| rlandy | rfolco: ^^ pls advise as well | 14:47 |
| * rlandy needs cards there by wednesday | 14:47 | |
| sshnaidm | quiquell, can we please keep the current structure of the reproducer role patch? Because it changes every patchset and have headache when rebasing it :) | 14:50 |
| quiquell | sshnaidm: Did I change it ? | 14:50 |
| sshnaidm | quiquell, oh yes! :D | 14:51 |
| sshnaidm | quiquell, it moves to various paths each time | 14:51 |
| quiquell | sshnaidm: last ps is adding fetch-zuul-cloner | 14:51 |
| quiquell | sshnaidm: but same structure as "#14" ps | 14:51 |
| quiquell | sshnaidm: the one at the etherpad | 14:51 |
| sshnaidm | quiquell, I mean the general path to the role | 14:51 |
| quiquell | sshnaidm: Not going to more stuff | 14:51 |
| quiquell | sshnaidm: what ps do you use ? | 14:52 |
| rfolco | rlandy, I agree in promoting that task to user story | 14:52 |
| sshnaidm | quiquell, it was in infra-setup, then in playbooks/.., now in tripleo-ci-reproducer | 14:52 |
| quiquell | sshnaidm: tripleo-ci-reproducer is the final stuff | 14:53 |
| sshnaidm | quiquell, great | 14:53 |
| quiquell | sshnaidm: is teh patchset at etherpad | 14:53 |
| quiquell | sshnaidm: sorry :-( | 14:53 |
| rlandy | stuff moved? | 14:53 |
| rlandy | since 14? | 14:53 |
| sshnaidm | quiquell, np | 14:53 |
| quiquell | rlandy: same as etherpad | 14:53 |
| * rlandy reads again | 14:53 | |
| quiquell | rlandy: we are ok | 14:53 |
| rlandy | quiquell: ok - same as friday | 14:54 |
| quiquell | sshnaidm: moved everything to a dir to isolated from the repo and used pipenv so we don't do tox stuff but put python at requirements together | 14:54 |
| marios_|ruck | quiquell: ack wana catch up in 5 mins? | 14:54 |
| quiquell | sure, give a min | 14:54 |
| rlandy | quiquell: ok - probably will only rerun once figure out dlrn_hash stuff | 14:55 |
| quiquell | marios_|ruck: ready | 14:55 |
| quiquell | rlandy, rfolco: do you want to join so I do also handover of scenario002 ? | 14:56 |
| marios_|ruck | quiquell: https://redhat.bluejeans.com/7661925373 | 14:56 |
| marios_|ruck | rlandy: did you want to catch up about scenarios? | 14:57 |
| quiquell | marios_|ruck: Connected | 14:59 |
| rlandy | joining | 15:00 |
| marios_|ruck | is it me? | 15:01 |
| marios_|ruck | rlandy: quiquell ? ^ | 15:01 |
| rlandy | i think so | 15:01 |
| marios_|ruck | rejoining | 15:01 |
| sshnaidm | rlandy, quiquell we need to talk with Tristan about his patch.. I think it can be a part of solution for our future ovb jobs | 15:04 |
| sshnaidm | rlandy, quiquell because if no running te-broker we'll have to use secrets, and we can't decrypt them in running zuul locally | 15:05 |
| rlandy | sshnaidm: sec - just chatting with marios | 15:05 |
| sshnaidm | rlandy, quiquell so currently no any job that uses secrets.yaml can be reproduced for now | 15:06 |
| quiquell | sshnaidm: but to repro OVB we don't use tebroker we emulate it | 15:12 |
| quiquell | sshnaidm: we can generate or own secrets and config project | 15:12 |
| sshnaidm | quiquell, that's what Tristan does in his patch | 15:12 |
| sshnaidm | quiquell, seems like I need to do it too.. maybe just copy config repo and remove all secrets | 15:14 |
| quiquell | sshnaidm: you can exclude stuff | 15:14 |
| quiquell | sshnaidm: from main.yaml | 15:14 |
| quiquell | sshnaidm: "secrets" is one of the element | 15:14 |
| ssbarnea|rover | marios_|ruck: sshnaidm : alex fix related to yum updates for upgrade job failed the 2nd time in the gate with timeout on the same job: tripleo-ci-centos-7-containers-multinode what can we do? | 15:15 |
| marios_|ruck | ssbarnea|rover: in call re scenarios right now will check in a sec | 15:24 |
| rlandy | sshnaidm: k - ready to talk with Tristan when you are | 15:28 |
| rlandy | quiquell: ^^ you still have time today? | 15:29 |
| quiquell | rlandy: like half hour now | 15:29 |
| rlandy | sshnaidm: k - up to you | 15:29 |
| quiquell | rlandy, sshnaidm: but you can talk with tristan about ovb and repro I will catch up with you guys tomorrow | 15:32 |
| *** dtrainor has joined #oooq | 15:34 | |
| marios | rlandy: quiquell as discussed https://review.openstack.org/#/c/620556/6/roles/standalone/tasks/main.yml | 15:44 |
| marios | trying recheck on scen 1 | 15:44 |
| quiquell | marios: ack if working, we can simplify it a little removing the fact | 15:45 |
| marios_|ruck | done | 15:45 |
| marios_|ruck | quiquell: i updated | 15:46 |
| *** ykarel has quit IRC | 15:53 | |
| *** ykarel has joined #oooq | 15:53 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 15:56 |
| *** quiquell is now known as quiquell|off | 15:58 | |
| sshnaidm | ssbarnea|rover, you'd better to give links, I don't know what are you talking about | 15:58 |
| sshnaidm | rlandy, ready to talk if relevant | 15:59 |
| rlandy | sshnaidm: only if you want to chat | 15:59 |
| rlandy | really it's between you and tristan | 15:59 |
| ssbarnea|rover | sshnaidm: sure, https://review.openstack.org/#/c/621259/ -- twice in a raw timeout in the same place during the gate | 16:00 |
| rlandy | arxcruz: ping re: https://review.openstack.org/#/c/607077/ | 16:01 |
| rlandy | arxcruz: have you tried it out at all? | 16:01 |
| * rlandy looks at it to get dlrn_hash | 16:01 | |
| sshnaidm | ssbarnea|rover, which place?? links, please | 16:03 |
| ssbarnea|rover | sshnaidm: tripleo-ci-centos-7-containers-multinode job : timeout tempest | 16:04 |
| sshnaidm | ssbarnea|rover, I see only one.. do you know to copy/paste URL (links)? :) | 16:05 |
| ssbarnea|rover | http://logs.openstack.org/59/621259/5/gate/tripleo-ci-centos-7-containers-multinode/cd70f4c/ -- but apparently is not tempest fault. | 16:05 |
| sshnaidm | ssbarnea|rover, yeah, it's timeout, too few time for tempest | 16:07 |
| sshnaidm | ssbarnea|rover, need to check which part of it took too much time | 16:07 |
| ssbarnea|rover | yep, but we cannot afford to "press recheck"... | 16:07 |
| * ssbarnea|rover is no longer accidental, it kinda become the rule. | 16:08 | |
| ssbarnea|rover | this is why i am asking you: what can we do? | 16:09 |
| ssbarnea|rover | we have especially as this directly affects the gate | 16:09 |
| sshnaidm | ssbarnea|rover, sshnaidm> ssbarnea|rover, need to check which part of it took too much time | 16:09 |
| ssbarnea|rover | i am trying to do the same now,... lets see what we find. | 16:10 |
| ssbarnea|rover | ara report is useless in this case as longest runtime is 17min, clearly does not add up to 3h 20m time. | 16:11 |
| sshnaidm | ssbarnea|rover, did it happen twice or more? | 16:11 |
| ssbarnea|rover | sshnaidm: twice in a raw and exactly in the same place (tempest run) which takes only a couple of minutes, but I assume is only chance that made it happen now, probably it was going to fail in a different place. | 16:13 |
| sshnaidm | ssbarnea|rover, you can try this: docker run --net=host sshnaidm/jcomparison | 16:13 |
| sshnaidm | ssbarnea|rover, in "Good job" paste a link to some good same job, take one from sova | 16:14 |
| sshnaidm | ssbarnea|rover, in "bad job" use one of failed | 16:14 |
| ssbarnea|rover | trying now.... | 16:14 |
| arxcruz | rlandy: no, didn't have chance yet :) | 16:16 |
| arxcruz | :( | 16:16 |
| *** dtantsur is now known as dtantsur|afk | 16:17 | |
| sshnaidm | ssbarnea|rover, from graphs seems like containers prepare took too much time, worth to check what are they doing: http://logs.openstack.org/59/621259/5/gate/tripleo-ci-centos-7-containers-multinode/cd70f4c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 16:19 |
| sshnaidm | ssbarnea|rover, seems like all tasks take more time than usual, maybe cloud - need to check if it's the same one, grafana may help | 16:21 |
| rlandy | arxcruz: k, looking at it now | 16:24 |
| arxcruz | rlandy: sorry, i should do that for you :( | 16:24 |
| arxcruz | i was too lazy on that :( | 16:25 |
| ssbarnea|rover | sshnaidm: i cannot use jcompare now, i will go manual and see later. it reports as starting but it does not respond on port 5000 and it has no docs, but i will look into it later. | 16:27 |
| rlandy | arxcruz: no worries - I am just trying to see if the dlrn_hash is accessible anyways | 16:29 |
| rlandy | the api won't help if it's not | 16:29 |
| arxcruz | rlandy: i'll try to work on an env i'm also interested on this, and i want to contribute on zuul | 16:29 |
| ssbarnea|rover | sshnaidm: | 16:30 |
| ssbarnea|rover | sshnaidm: Configure Ironic pxe_ssh driver took 1h 27m ! -- this does not seem right to me. | 16:30 |
| sshnaidm | ssbarnea|rover, where is it? | 16:31 |
| ssbarnea|rover | http://logs.openstack.org/59/621259/5/gate/tripleo-ci-centos-7-containers-multinode/cd70f4c/job-output.txt.gz#_2018-12-03_10_32_31_516435 | 16:31 |
| ssbarnea|rover | sorry, wrong line, this was deploy time | 16:31 |
| ssbarnea|rover | so install-undercloud and deploy-overcloud take each ~1h 30m -- so no wonder that the job fails reaching the 3h timeout. | 16:32 |
| ssbarnea|rover | during install under-cloud the slowest tasks appear to be starting the containers ones, especially "step 3" once which took over 14m. | 16:38 |
| ssbarnea|rover | another weird delay can be seen at http://logs.openstack.org/59/621259/5/gate/tripleo-ci-centos-7-containers-multinode/cd70f4c/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-12-03_09_48_22_508 -- not sure what took ~23m there. | 16:39 |
| *** bogdando has quit IRC | 16:41 | |
| *** trown is now known as trown|lunch | 16:41 | |
| ssbarnea|rover | so tripleo-container-image-prepare is the expensive part. | 16:43 |
| marios_|ruck | ssbarnea|rover: /me almost off for the day soren. did you find out more about the gate issue on https://review.openstack.org/#/c/621259/5 | 16:51 |
| marios_|ruck | ssbarnea|rover: otherwise i'll check in the morning | 16:51 |
| marios_|ruck | ssbarnea|rover: anything else please add to etherpad before you go? | 16:52 |
| ssbarnea|rover | marios_|ruck: yeah, apparently timeouts cause by more or less natural long duration.... | 17:02 |
| *** udesale has quit IRC | 17:02 | |
| rfolco | sshnaidm, rlandy marios_|ruck panda: I have already one +2... would you review please https://review.openstack.org/619337 | 17:06 |
| marios_|ruck | ssbarnea|rover: so timout not error/? will check in the morning | 17:11 |
| marios_|ruck | rfolco: really have to go now but if it is still around will check it in morning | 17:11 |
| rfolco | marios_|ruck, np man, have a good one | 17:12 |
| ssbarnea|rover | sure. nothing you can do right now. | 17:12 |
| *** kopecmartin is now known as kopecmartin|off | 17:40 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 17:56 |
| *** derekh has quit IRC | 18:02 | |
| *** apetrich has quit IRC | 18:18 | |
| *** dsneddon has joined #oooq | 18:19 | |
| *** amoralej is now known as amoralej|off | 18:24 | |
| *** udesale has joined #oooq | 18:26 | |
| *** apetrich has joined #oooq | 18:30 | |
| *** ykarel is now known as ykarel|away | 18:36 | |
| *** holser_ has quit IRC | 18:40 | |
| *** ykarel|away has quit IRC | 18:47 | |
| *** trown|lunch is now known as trown|outtypewww | 18:59 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 19:56 |
| *** udesale has quit IRC | 20:30 | |
| *** holser_ has joined #oooq | 20:44 | |
| rlandy | sshnaidm: still around? | 20:44 |
| sshnaidm | rlandy, yep | 20:44 |
| rlandy | sshnaidm: I am getting an error trying to schedule a job on my rdocloud tenant | 20:44 |
| rlandy | sshnaidm: http://pastebin.test.redhat.com/677859 | 20:45 |
| rlandy | the job never schedules | 20:46 |
| sshnaidm | rlandy, where is the error though? | 20:47 |
| rlandy | you seen that before? | 20:47 |
| rlandy | hmm ... wonder of it is the job def | 20:47 |
| rlandy | retrying | 20:47 |
| sshnaidm | there are no errors in this log | 20:48 |
| sshnaidm | rlandy, look at scheduler logs, usually there is something if job isn't scheduled | 20:48 |
| rlandy | sshnaidm: just see a whole bunch of lines like: | 20:52 |
| rlandy | scheduler_1 | 0046dc4b6d68c7ea387f5aa0822bc8dd47fd7118ba04 refs/changes/88/584088/3 | 20:52 |
| sshnaidm | rlandy, that's config part, it's ok, tail logs right after you submit a patch | 20:53 |
| *** apetrich has quit IRC | 20:58 | |
| *** udesale has joined #oooq | 20:58 | |
| rlandy | sshnaidm: no error that I can see - how long does it usually take to see the job running with rdocloud? | 20:59 |
| sshnaidm | rlandy, could take minutes | 20:59 |
| rlandy | sshnaidm: so you see instances created? | 21:00 |
| sshnaidm | rlandy, look at launcher logs also | 21:00 |
| sshnaidm | rlandy, usually yes | 21:00 |
| rlandy | launcher logs are red | 21:02 |
| rlandy | sshnaidm: http://pastebin.test.redhat.com/677874 | 21:04 |
| rlandy | no quota | 21:04 |
| rlandy | I have only one instance | 21:05 |
| rlandy | must be all of rdocloud | 21:05 |
| rlandy | launcher_1 | 2018-12-03 20:56:19,540 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for rdo-cloud: {'compute': {'cores': 32, 'instances': 10, 'ram': 65536}} | 21:05 |
| rlandy | sshnaidm: some node failures in CI as well | 21:09 |
| sshnaidm | rlandy, need more logs to see | 21:10 |
| sshnaidm | rlandy, do you have in your tenant network and router? | 21:10 |
| rlandy | sshnaidm: yes | 21:13 |
| rlandy | it doesnt say no network connection | 21:14 |
| rlandy | launcher_1 | File "/usr/local/lib/python3.7/site-packages/nodepool/driver/openstack/provider.py", line 201, in unmanagedQuotaUsed | 21:14 |
| rlandy | launcher_1 | flavor = flavors.get(server.flavor.id) | 21:14 |
| rlandy | flavors are standard | 21:14 |
| *** apetrich has joined #oooq | 21:18 | |
| *** udesale has quit IRC | 21:23 | |
| rlandy | sshnaidm: it finally scheduled | 21:26 |
| rlandy | ha | 21:26 |
| sshnaidm | cool | 21:27 |
| rlandy | now I can try passing dlrn_hash_tag | 21:27 |
| *** dsneddon has quit IRC | 21:32 | |
| *** jfrancoa has quit IRC | 21:37 | |
| *** holser_ has quit IRC | 21:39 | |
| *** amoralej|off is now known as amoralej | 21:46 | |
| *** amoralej is now known as amoralej|off | 21:47 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 21:56 |
| *** jaosorior has quit IRC | 22:13 | |
| *** dsneddon has joined #oooq | 22:16 | |
| *** jaosorior has joined #oooq | 22:17 | |
| *** udesale has joined #oooq | 22:23 | |
| *** udesale has quit IRC | 22:25 | |
| *** udesale has joined #oooq | 22:26 | |
| *** udesale has quit IRC | 22:46 | |
| *** irclogbot_0 has quit IRC | 22:47 | |
| *** jaosorior has quit IRC | 22:53 | |
| *** irclogbot_0 has joined #oooq | 23:08 | |
| *** jtomasek has quit IRC | 23:10 | |
| *** gkadam has quit IRC | 23:48 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario003-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!