| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:39 |
|---|---|---|
| *** ccamacho has quit IRC | 00:43 | |
| weshay | rlandy|rover|bbl, ok.. when I see green on that job, I'll start on converting the other env.. and you can look at the jjb prototype | 01:26 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:08 | |
| *** ccamacho has joined #oooq | 02:17 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 02:39 |
| *** rlandy|rover has quit IRC | 02:49 | |
| *** ccamacho has quit IRC | 03:13 | |
| *** skramaja has joined #oooq | 03:18 | |
| *** agopi has joined #oooq | 03:30 | |
| *** agopi has quit IRC | 03:40 | |
| *** udesale has joined #oooq | 03:52 | |
| *** agopi has joined #oooq | 03:57 | |
| *** agopi has quit IRC | 04:08 | |
| *** ykarel|away has joined #oooq | 04:09 | |
| *** ykarel_ has joined #oooq | 04:21 | |
| *** ykarel|away has quit IRC | 04:23 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 04:39 |
| *** pgadiya has joined #oooq | 04:48 | |
| *** pgadiya has quit IRC | 04:48 | |
| *** pliu has quit IRC | 05:10 | |
| *** pliu has joined #oooq | 05:11 | |
| *** rasca has quit IRC | 05:11 | |
| *** jschlueter has quit IRC | 05:11 | |
| *** myoung|off has quit IRC | 05:11 | |
| *** faceman has quit IRC | 05:11 | |
| *** lucasagomes has quit IRC | 05:11 | |
| *** rnoriega has quit IRC | 05:11 | |
| *** lhinds has quit IRC | 05:11 | |
| *** lhinds has joined #oooq | 05:12 | |
| *** faceman has joined #oooq | 05:13 | |
| *** rnoriega has joined #oooq | 05:14 | |
| *** rasca has joined #oooq | 05:14 | |
| *** lucasagomes has joined #oooq | 05:16 | |
| *** jschlueter has joined #oooq | 05:17 | |
| *** myoung has joined #oooq | 05:17 | |
| *** quiquell|off is now known as quiquell | 05:32 | |
| *** ratailor has joined #oooq | 05:36 | |
| *** tcw has quit IRC | 06:03 | |
| *** jtomasek has joined #oooq | 06:11 | |
| *** jtomasek has quit IRC | 06:11 | |
| *** tcw has joined #oooq | 06:12 | |
| *** jtomasek_ has joined #oooq | 06:13 | |
| *** jtomasek_ has quit IRC | 06:16 | |
| *** udesale has quit IRC | 06:18 | |
| *** udesale has joined #oooq | 06:18 | |
| *** saneax has joined #oooq | 06:29 | |
| *** florianf has joined #oooq | 06:34 | |
| *** jtomasek has joined #oooq | 06:38 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 06:39 |
| *** bogdando has joined #oooq | 06:43 | |
| quiquell | sshnaidm: What do we need to run this at an RDO job ? | 06:55 |
| quiquell | https://review.rdoproject.org/r/#/c/14084/1 | 06:55 |
| *** bogdando has quit IRC | 07:06 | |
| *** tosky has joined #oooq | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:11 | |
| *** bogdando has joined #oooq | 07:12 | |
| *** kopecmartin has joined #oooq | 07:12 | |
| *** ykarel_ is now known as ykarel | 07:15 | |
| *** gkadam has joined #oooq | 07:15 | |
| *** florianf has quit IRC | 07:18 | |
| *** florianf has joined #oooq | 07:18 | |
| *** ccamacho has joined #oooq | 07:25 | |
| *** holser__ has joined #oooq | 07:37 | |
| *** holser__ has quit IRC | 08:00 | |
| *** jbadiapa_ is now known as jbadiapa | 08:00 | |
| *** holser__ has joined #oooq | 08:00 | |
| quiquell | sshnaidm: You there ? | 08:18 |
| *** ykarel_ has joined #oooq | 08:21 | |
| *** ykarel has quit IRC | 08:23 | |
| *** ykarel_ is now known as ykarel | 08:28 | |
| arxcruz|ruck | chandankumar: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/50abb54/undercloud/home/jenkins/tempest.log.gz | 08:30 |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 08:39 |
| *** moguimar has quit IRC | 08:51 | |
| chandankumar | arxcruz|ruck: yup | 08:55 |
| chandankumar | arxcruz|ruck: is it poping only in periodic jobs? | 08:56 |
| arxcruz|ruck | chandankumar: so far yes, i'm trying to reproduce on rdocloud | 08:56 |
| panda | quiquell: how's going ? | 08:57 |
| panda | quiquell: something left to review ? | 08:57 |
| panda | quiquell: should I take the card in backlog ? | 08:57 |
| quiquell | panda: Was starting with that card | 08:58 |
| chandankumar | arxcruz|ruck: I have reproducer ready just now | 08:58 |
| chandankumar | arxcruz|ruck: from fs30 | 08:58 |
| arxcruz|ruck | chandankumar: master ? | 08:59 |
| arxcruz|ruck | chandankumar: can you check ? | 08:59 |
| quiquell | panda: Have upload small fix to emit_release_script, prevent check job from working | 08:59 |
| chandankumar | arxcruz|ruck: sure | 08:59 |
| quiquell | panda: https://review.openstack.org/#/c/572420/ | 08:59 |
| quiquell | panda: Let me upload the new job as a DNM and we can play with it | 08:59 |
| quiquell | panda: The new job https://review.openstack.org/#/c/574665/ | 09:01 |
| quiquell | panda: It's not integrated with the emit release, just print the toci variable | 09:02 |
| quiquell | panda: Humm the job job doesn't appear in zuuls, I am missing something | 09:04 |
| quiquell | panda: Na forget about it | 09:04 |
| quiquell | panda: Testing patch https://review.openstack.org/574671 | 09:09 |
| quiquell | panda: We can try to merge emit_releases_script changes first https://review.openstack.org/#/c/572420/ | 09:12 |
| quiquell | panda: About the promoter your change has been merged, we can close https://bugs.launchpad.net/tripleo/+bug/1768090 | 09:15 |
| openstack | Launchpad bug 1768090 in tripleo "promoter script is not comparing timestamps correctly when folding hashes" [High,In progress] - Assigned to Gabriele Cerami (gcerami) | 09:15 |
| chandankumar | arxcruz|ruck: http://paste.openstack.org/show/723283/ | 09:25 |
| chandankumar | arxcruz|ruck: https://github.com/openstack/python-tempestconf/blob/master/config_tempest/services/identity.py#L65 | 09:25 |
| chandankumar | it is returning 500 | 09:25 |
| chandankumar | arxcruz|ruck: we need to take a look at keystone logs why it doing 500 | 09:25 |
| arxcruz|ruck | chandankumar: i'm opening a bug anyway | 09:31 |
| chandankumar | arxcruz|ruck: the bug is alrady there | 09:33 |
| sshnaidm | quiquell, need to add a job here: https://github.com/rdo-infra/review.rdoproject.org-config/blob/8f5408dc753ff072b03961b35304da9bd50b4c64/zuul/projects.yaml#L3853 | 09:33 |
| ssbarnea | out of curiosity, did anyone had any success (or attempt) of using ansible-review? | 09:33 |
| sshnaidm | quiquell, and configure it there https://github.com/rdo-infra/review.rdoproject.org-config/blob/8f5408dc753ff072b03961b35304da9bd50b4c64/jobs/rdoinfra.yaml#L2 | 09:33 |
| chandankumar | arxcruz|ruck: https://bugs.launchpad.net/tripleo/+bug/1776301 | 09:33 |
| openstack | Launchpad bug 1776301 in tripleo "[master promotion] Tempest is failing with " KeyError: 'resources' "errors - Connection refused" [Critical,Triaged] | 09:33 |
| *** moguimar has joined #oooq | 09:33 | |
| *** dtantsur|afk is now known as dtantsur | 09:35 | |
| arxcruz|ruck | chandankumar: meanwhile, could we add a try catch ? | 09:36 |
| arxcruz|ruck | nah, silly me | 09:36 |
| quiquell | sshnaidm: Ok, will try to add tox-py27 | 09:37 |
| chandankumar | arxcruz|ruck: there is but we need to make some modification for the lower portion of the code | 09:37 |
| arxcruz|ruck | sshnaidm: what do we need to do to have this merged https://review.openstack.org/#/c/574270/ ? | 09:49 |
| arxcruz|ruck | recheck ? | 09:49 |
| arxcruz|ruck | sshnaidm: nevermind, it's on the gates already | 09:50 |
| chandankumar | arxcruz|ruck: http://paste.openstack.org/show/723289/ does something like this work? | 09:55 |
| arxcruz|ruck | chandankumar: i'm thinking here, adding a try / catch is just hiding the problem... | 09:56 |
| arxcruz|ruck | i'm trying to understand the flask stuff on keystone | 09:56 |
| quiquell | panda: You there ? | 10:02 |
| ykarel | arxcruz|ruck, chandankumar i think /v3 needs to be appended in service_url, | 10:03 |
| ykarel | with ^^ i am not seeing the issue can you try | 10:04 |
| arxcruz|ruck | ykarel: https://github.com/openstack/keystone/blob/master/keystone/version/controllers.py#L43 | 10:05 |
| ykarel | arxcruz|ruck, yup, but this seems to be not working | 10:05 |
| panda | quiquell: sorry, got some problems | 10:05 |
| panda | quiquell: here now | 10:05 |
| chandankumar | ykarel: one min trying | 10:05 |
| quiquell | panda: Ok np, what do you want first reviews, new job or promoter ? | 10:06 |
| arxcruz|ruck | chandankumar: can you please give me access to your env ? | 10:06 |
| arxcruz|ruck | github.com/arxcruz.keys | 10:06 |
| panda | quiquell: new job, so it can roll whilewe do other things | 10:06 |
| chandankumar | arxcruz|ruck: it worked | 10:07 |
| arxcruz|ruck | hmmmm | 10:09 |
| chandankumar | adding v3 at service_url | 10:09 |
| arxcruz|ruck | well i'm waiting your patch on python-tempestconf | 10:09 |
| arxcruz|ruck | chandankumar: can you verify at your keystone if this code is there https://github.com/openstack/keystone/blob/master/keystone/version/controllers.py#L43 ? | 10:09 |
| ykarel | arxcruz|ruck, chandankumar so what i can say definitely there is an issue in keystone with unversioned url(/v3 not appended), adding /v3 in tempestconf can workaround that | 10:10 |
| ykarel | arxcruz|ruck, what i noticed is that when we append /v3 in url https://github.com/openstack/keystone/blob/master/keystone/version/controllers.py#L43 is not hit | 10:11 |
| ykarel | so we don't see issue in this case | 10:11 |
| panda | quiquell: but you already tested fs50 n -> n + 1 ? | 10:12 |
| panda | quiquell: becasuse I don't see much left to do for this card .. | 10:13 |
| *** zoli is now known as zoli|lunch | 10:15 | |
| arxcruz|ruck | ykarel: yeah, i think i know the problem... | 10:16 |
| arxcruz|ruck | chandankumar: how was the auth_url ? | 10:16 |
| arxcruz|ruck | was it ending with / ? | 10:16 |
| arxcruz|ruck | before the error in keystone log there's this 2018-06-11 20:16:29.821 216 INFO keystone.common.wsgi [req-591e2ecd-8088-4d2e-a5ae-c23a1624187d - - - - -] GET http://192.168.24.9:5000// | 10:17 |
| arxcruz|ruck | with // at the end | 10:17 |
| quiquell | panda: The WIP of the new job https://review.openstack.org/#/c/574665/ | 10:23 |
| quiquell | panda: It just print the toci variable, have to integrate it with the script | 10:23 |
| *** holser__ has quit IRC | 10:23 | |
| *** holser__ has joined #oooq | 10:24 | |
| arxcruz|ruck | chandankumar: is possible to give me access ? | 10:26 |
| chandankumar | arxcruz|ruck: zuul@38.145.34.92 | 10:27 |
| chandankumar | arxcruz|ruck: run sh tempest-setup.sh | 10:28 |
| chandankumar | you will be directly taken into containers | 10:28 |
| quiquell | panda: We also need some reviewing on the patches | 10:29 |
| chandankumar | arxcruz|ruck: able to ssh? | 10:30 |
| arxcruz|ruck | chandankumar: yes | 10:30 |
| arxcruz|ruck | chandankumar: may i add a print on keystone just to check what's going on ? | 10:31 |
| chandankumar | arxcruz|ruck: feel free to do anything, | 10:31 |
| chandankumar | arxcruz|ruck: I am preparing a aptch | 10:31 |
| ykarel | arxcruz|ruck, may be // is the problem, but i have no idea yet | 10:33 |
| panda | quiquell: uploaded new patchset to test fs50 | 10:33 |
| panda | quiquell: not liking too much how I changed stuff :/ | 10:33 |
| panda | quiquell: which patches ? | 10:34 |
| panda | quiquell: all the dependencies ? | 10:34 |
| arxcruz|ruck | ykarel: from urljoin isn't the problem | 10:34 |
| quiquell | panda: What do you mean ? | 10:34 |
| arxcruz|ruck | >>> req = webob.Request.blank( | 10:35 |
| arxcruz|ruck | ... '/v3', headers={'Accept': 'application/json-home'}) | 10:35 |
| arxcruz|ruck | >>> req | 10:35 |
| arxcruz|ruck | <Request at 0x10affcc10 GET http://localhost/v3> | 10:35 |
| arxcruz|ruck | chandankumar: i've added some log.info in keystone, now restarting the docker | 10:36 |
| arxcruz|ruck | and it seems isn't working D: | 10:36 |
| *** bogdando has quit IRC | 10:37 | |
| panda | quiquell: uhm I was about to ruin the rebases .. | 10:37 |
| quiquell | panda: This new job is going to run only at queens ? | 10:38 |
| quiquell | panda: Because n - n + 1 doesn't make sense in master | 10:38 |
| panda | quiquell: we can always run n -> n + 1 from master to the FUTURE | 10:39 |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 10:39 |
| quiquell | panda: Sounds very DLRN | 10:40 |
| quiquell | panda: Maybe we can start to merge trown's work, it's working fine | 10:40 |
| quiquell | panda: First this https://review.openstack.org/#/c/572419 | 10:41 |
| quiquell | panda: Then this https://review.openstack.org/#/c/572420 | 10:41 |
| *** quiquell has quit IRC | 10:48 | |
| chandankumar | arxcruz|ruck: i think keystone died | 10:58 |
| arxcruz|ruck | chandankumar: checking | 10:59 |
| *** quiquell_phone has joined #oooq | 10:59 | |
| arxcruz|ruck | chandankumar: i'm restarting the docker | 11:00 |
| quiquell_phone | Panda: they have switch of the lights at my building | 11:00 |
| quiquell_phone | Going for lunch, lets talk later | 11:01 |
| quiquell_phone | Quiquell|lunch | 11:01 |
| *** quiquell_phone is now known as quiquell|lunch | 11:01 | |
| *** quiquell|tmp has joined #oooq | 11:06 | |
| *** quiquell|lunch has quit IRC | 11:07 | |
| quiquell|tmp | panda: Ok I am back, have some minutes | 11:07 |
| panda | quiquell|tmp: lunch already done ? | 11:09 |
| panda | quiquell|tmp: need to rebase all the changes | 11:09 |
| quiquell|tmp | panda: Nope, will do the lunch later, I am connecting with my mobile phone | 11:11 |
| quiquell|tmp | sshnaidm, marios: reviews for trown's changes https://review.openstack.org/#/c/572419/173978 | 11:12 |
| panda | quiquell|tmp: can you check the new rebase under 574665 | 11:15 |
| panda | ? | 11:15 |
| arxcruz|ruck | ykarel: so it seems there's nothing to do, the problem is the /v3 because i'm running tempestconf on chandankumar env, and i get the error in python-tempestconf, but nothing in the logs from keystone as you pointed here | 11:17 |
| arxcruz|ruck | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/fbc106b/subnode-2/var/log/containers/keystone/keystone.log.txt.gz?level=ERROR#_2018-06-11_20_16_29_824 | 11:17 |
| marios | quiquell|tmp: ack will check | 11:17 |
| ykarel | arxcruz|ruck, i can see the logs in keystone | 11:18 |
| arxcruz|ruck | ykarel: i'm running the code, and not getting the error, i believe it's related to whatever else, at least in chandankumar environment | 11:19 |
| ykarel | arxcruz|ruck, python -c "import requests,json;r=requests.get('http://192.168.24.1:5000',verify=False,headers={'Accept': 'application/json-home'});print(r.content)" | 11:20 |
| ykarel | when i run ^^, i see same logs as CI in keystone.log | 11:20 |
| ykarel | here:- /var/log/containers/keystone/keystone.log | 11:20 |
| ykarel | i am testing against undercloud | 11:21 |
| arxcruz|ruck | ykarel: you're right | 11:21 |
| arxcruz|ruck | weird.. | 11:21 |
| quiquell|tmp | panda: Have rebased 574665 now it's ok, let's recheck the test again | 11:23 |
| quiquell|tmp | panda: And now I go for lunch | 11:23 |
| *** quiquell|tmp is now known as quiquell|lunch | 11:24 | |
| *** quiquell|lunch has quit IRC | 11:28 | |
| *** quiquell has joined #oooq | 11:31 | |
| *** quiquell is now known as quiquell|lunch | 11:31 | |
| arxcruz|ruck | ykarel: it seems the problem is the headers | 11:32 |
| *** zoli|lunch is now known as zoli | 11:32 | |
| arxcruz|ruck | if i remove the headers 'Accept': 'application/json-home' it pass | 11:32 |
| ykarel | arxcruz|ruck, but without this you will not get extensions | 11:32 |
| arxcruz|ruck | ykarel: yes | 11:33 |
| arxcruz|ruck | and i think our code was just wrong all the time | 11:33 |
| arxcruz|ruck | that's the right behavior | 11:33 |
| arxcruz|ruck | because | 11:33 |
| arxcruz|ruck | if you do a curl on the :500 | 11:33 |
| arxcruz|ruck | :5000 | 11:33 |
| arxcruz|ruck | it will return the expected | 11:33 |
| arxcruz|ruck | the list of endpoints | 11:33 |
| arxcruz|ruck | then if you do :5000/v3 | 11:33 |
| arxcruz|ruck | it will return the info for the v3 | 11:34 |
| arxcruz|ruck | i believe this is the right behavior | 11:34 |
| *** quiquell|lunch has quit IRC | 11:36 | |
| *** amoralej is now known as amoralej|lunch | 11:48 | |
| weshay | panda, ready? | 12:01 |
| panda | weshay: already there | 12:02 |
| weshay | panda, can't hear you | 12:02 |
| *** rlandy has joined #oooq | 12:12 | |
| *** rlandy is now known as rlandy|rover | 12:12 | |
| rlandy|rover | arxcruz|ruck: hello! | 12:14 |
| arxcruz|ruck | rlandy|rover: hello, i'm already aware of the failures :) | 12:15 |
| arxcruz|ruck | rlandy|rover: chandankumar is working on a patch | 12:15 |
| rlandy|rover | arxcruz|ruck: lol - not chasing you :) - I think I messed up the bug I logged yesterday | 12:15 |
| arxcruz|ruck | rlandy|rover: lol, no, didn't meant that, just to let you know :) | 12:16 |
| rlandy|rover | arxcruz|ruck: I may have put two problems in the same bug :( | 12:17 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1776301 | 12:17 |
| openstack | Launchpad bug 1776301 in tripleo "[master promotion] Tempest is failing with " KeyError: 'resources' "errors - Connection refused" [Critical,Triaged] | 12:17 |
| rlandy|rover | There was a tempest error and and tempestmail error | 12:18 |
| rlandy|rover | idk if they are the same problem | 12:18 |
| arxcruz|ruck | rlandy|rover: it's the same, problem is, keystone change (again) the url | 12:18 |
| rlandy|rover | arxcruz|ruck; ok - just checking - I know very little about these things | 12:18 |
| arxcruz|ruck | so, python-tempestconf was trying in an endpoint that no longer works | 12:18 |
| rlandy|rover | which is why I ruck/rover with you :) | 12:18 |
| *** ratailor has quit IRC | 12:19 | |
| *** ykarel_ has joined #oooq | 12:21 | |
| *** trown|outtypewww is now known as trown | 12:21 | |
| *** ykarel has quit IRC | 12:24 | |
| *** ykarel_ is now known as ykarel | 12:26 | |
| ykarel | arxcruz|ruck, rlandy|rover rdo zuul queue is too large currently: 27 hours | 12:28 |
| ykarel | as we are aware of the current issues, should we kill the current run until we get the tempestconf fix | 12:29 |
| ykarel | if want queens can be run as queens fix has already landed | 12:29 |
| rlandy|rover | ykarel: we can kill the containers build on master | 12:32 |
| rlandy|rover | weshay: arxcruz|ruck: ^^ ok? | 12:32 |
| arxcruz|ruck | rlandy|rover: fine by me, this run will fail anyway | 12:33 |
| arxcruz|ruck | only queens for now, we still working on the patch for master | 12:33 |
| rlandy|rover | arxcruz|ruck: ykarel: hmmm - I don;t have access to kill it - do you? | 12:33 |
| arxcruz|ruck | turns out we need to test both / and /v3 because packstack still use / to get the extensions and also we need to care about previous releases as well | 12:33 |
| rlandy|rover | looking at jenkins | 12:34 |
| rlandy|rover | otherwise will ask on rdo | 12:34 |
| arxcruz|ruck | weshay: let me know when you'r done with panda :) | 12:34 |
| weshay | arxcruz|ruck, 3min | 12:34 |
| ykarel | rlandy|rover, i don't have access, amoralej|lunch jpena can do that | 12:34 |
| ykarel | but i think both master/queens killed together | 12:34 |
| arxcruz|ruck | ykarel: and i'm still not sure if the fix will solve the problem, perhaps we need to set the env to skip ssl verify | 12:34 |
| ykarel | arxcruz|ruck, in tempest-conf that's already set, | 12:35 |
| arxcruz|ruck | ykarel: i meant in the get-overcloud-nodes script | 12:35 |
| arxcruz|ruck | that runs before the tempest-conf | 12:36 |
| ykarel | arxcruz|ruck, you mean the version check that just merged | 12:36 |
| weshay | arxcruz|ruck, ready | 12:37 |
| arxcruz|ruck | ykarel: yeah | 12:37 |
| ykarel | arxcruz|ruck, let's see how it goes in queens, have you tried any reproducer with that patch? | 12:37 |
| ykarel | i think it will work | 12:37 |
| arxcruz|ruck | still working | 12:37 |
| rlandy|rover | checking the tenant | 12:38 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 12:39 |
| rlandy|rover | ykarel: arxcruz|ruck: we got some stacks in create failed - cleaning up | 12:40 |
| ykarel | rlandy|rover, ack | 12:41 |
| rlandy|rover | ykarel: is there some other issue on rdocloud today? odd tests within long waiting job queues are not being scheduled | 12:47 |
| rlandy|rover | the tenant is not oversubscribed | 12:47 |
| ykarel | rlandy|rover, am not aware of any issue for today on rdocloud | 12:47 |
| rlandy|rover | I'll ask on rdo - just in case - more the scheduling than the cloud itself | 12:48 |
| ykarel | rlandy|rover, i think they are just waiting for the vms | 12:48 |
| ykarel | as upperlimit is set to 80 or 85 | 12:48 |
| ykarel | yup jpena can give more insight on it | 12:48 |
| rlandy|rover | we are not short of them - maybe we should up the limit | 12:48 |
| rlandy|rover | I think we can oversubscribe more - will ask | 12:49 |
| ykarel | ack also i remember there were talks of 4 new compute nodes, | 12:49 |
| ykarel | but don't know for what was those and what's the progress there | 12:50 |
| ykarel | weshay, any idea ^^? | 12:50 |
| *** quiquell has joined #oooq | 12:58 | |
| quiquell | weshay: Are you there ? | 13:00 |
| *** tcw has quit IRC | 13:00 | |
| weshay | ykarel, quiquell sec.. coming out of a 1-1 | 13:00 |
| ykarel | ack | 13:00 |
| quiquell | weshay: np | 13:01 |
| *** amoralej|lunch is now known as amoralej | 13:02 | |
| amoralej | ykarel, what do you need? | 13:02 |
| amoralej | to cancel some job? | 13:02 |
| ykarel | amoralej, jpena cleared up | 13:02 |
| amoralej | ok | 13:03 |
| *** tcw has joined #oooq | 13:03 | |
| trown | quiquell: thanks for fixing up the backwards store_true thing... we probably need some basic functional test that actually passes arguments to script and checks results... since our unit tests cant catch that | 13:05 |
| rlandy|rover | amoralej: we're chatting on rhos-ops regarding the queues | 13:06 |
| rlandy|rover | quiquell: hey - did you get the message re: your reprovisioned box? | 13:06 |
| quiquell | trown: yw, would be nice to mock ansiple-playbook to do integration testing of TOCI | 13:07 |
| quiquell | rlandy|rover: Yep, thanks ! :-) | 13:07 |
| quiquell | trown: Your changes are good to merge, maybe we can squash the two commits | 13:07 |
| trown | quiquell: that is more than what I am thinking ... more like a test that just does `python emit-releases.py ...` and checks the output matches | 13:08 |
| trown | quiquell: ya maybe it is better... not even sure why I split those to begin with... | 13:08 |
| quiquell | trown: in that case, passing ARGV and still runing pytest is possible, so we don't have to run the script | 13:08 |
| quiquell | trown: we jus test the function main() | 13:08 |
| quiquell | trown: and mock the calling functions | 13:09 |
| trown | quiquell: oh, ya adding a test or 2 like that would be good | 13:09 |
| quiquell | trown: If you do an squahs, do it over the last patch, all the Depends-On are pointing to it | 13:10 |
| weshay | ykarel, I think the new hardware was added, we're going to get 5 new nodes over the next 3 quarters I think | 13:10 |
| weshay | ykarel, rlandy|rover we're also going to get the working nodes from rh1 | 13:10 |
| trown | quiquell: k | 13:10 |
| ykarel | weshay, ack | 13:11 |
| weshay | quiquell, what's up | 13:11 |
| quiquell | weshay: two things I misunderstood | 13:12 |
| quiquell | weshay: for fs037 (undercloud updates) tripleo-upgrades doesn't make sense ? | 13:13 |
| rlandy|rover | nice | 13:13 |
| *** skramaja has quit IRC | 13:13 | |
| quiquell | weshay: meaning adding jobs for fs037 at tripleo-upgrade project | 13:13 |
| weshay | quiquell, 37 is the update workflow | 13:14 |
| weshay | it needs to be check/gate on tq/tqe/tci tripleo-heat-templates, python-triploclient, tripleo-common on master/queens | 13:14 |
| quiquell | weshay: Ok brain fart will abandon the change | 13:15 |
| quiquell | weshay: And the other, if fs050 is master only how re we going to use it to test n -> n + 1 ? | 13:15 |
| weshay | quiquell, additionally I'm trying to help get fs51 the upgrade workflow | 13:15 |
| weshay | in as check, non-voting | 13:16 |
| weshay | on queens/master | 13:16 |
| quiquell | weshay: Yep, but that's not part of the fs037 sprint14 task, I was confused yesterday | 13:16 |
| quiquell | weshay: Sorry about the confusion | 13:16 |
| weshay | fs037 on queens was a sprint task I think | 13:18 |
| weshay | quiquell, https://trello.com/c/flI683EI/774-ci-job-create-job-37-work-on-queens-and-calls-tripleo-upgrade-updates-workflow | 13:18 |
| rlandy|rover | rfolco: hey - you're a zuul/SF expert these days, right :) ... how come other check jobs get scheduled before the one job left in a change set waiting 28 hr 24 min? | 13:18 |
| weshay | that should just read.. updates workflow.. not upgrade | 13:18 |
| * rlandy|rover does not understand zuul's logic here | 13:18 | |
| rlandy|rover | should be fifo, iiuc | 13:19 |
| weshay | not sure who wrote that card | 13:19 |
| *** kopecmartin has quit IRC | 13:19 | |
| quiquell | weshay: K, and about the other point fs050 and n -> n + 1 ? | 13:19 |
| quiquell | weshay: if fs050 is master only, we cannot do a n -> n + 1 with it | 13:19 |
| rfolco | rlandy|rover, parsing what you said... | 13:20 |
| rlandy|rover | rfolco: pls see https://review.rdoproject.org/zuul/ | 13:20 |
| rlandy|rover | we have jobs waiting over 20 hours | 13:20 |
| rlandy|rover | with one missing scheduled job | 13:20 |
| weshay | quiquell, jump on my blue | 13:20 |
| *** kopecmartin has joined #oooq | 13:20 | |
| rlandy|rover | and then other, new jobs running | 13:20 |
| weshay | quiquell, that is what emit_release_file.py should handle | 13:21 |
| weshay | quiquell, https://bluejeans.com/u/whayutin/ | 13:21 |
| quiquell | Ok | 13:21 |
| rlandy|rover | rfolco: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master on the 17 hr job got scheduled | 13:22 |
| rlandy|rover | and our 24+ job did not | 13:22 |
| rlandy|rover | neither did the 28+ hr job | 13:22 |
| rfolco | rlandy|rover, for those with more than 20hr, I see at least one voting job that failed... and there is one job queued. I am not 100% sure this is the case, if zuul is smart enough to favor new green jobs and leave failed ones to the end (queued) | 13:26 |
| rlandy|rover | rfolco: if that is the case we should NOT merge any more failing jobs | 13:26 |
| rlandy|rover | weshay: ^^ | 13:27 |
| rfolco | rlandy|rover, it should run yet, I am not saying zuul won't run those | 13:27 |
| marios | folks is this a known thing "Error in build_rpm_wrapper for openstack-tripleo-common" e.g. at http://logs.openstack.org/86/571186/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/0e1c43b/job-output.txt.gz#_2018-06-12_12_11_48_543361 | 13:28 |
| rlandy|rover | rfolco: interestingly the 24 hr periodic also has 1 queued job | 13:28 |
| rlandy|rover | now those have lower priority | 13:28 |
| rlandy|rover | but still | 13:28 |
| rlandy|rover | one job? | 13:28 |
| rlandy|rover | one little job? | 13:28 |
| *** Goneri has joined #oooq | 13:29 | |
| rlandy|rover | those jobs are all green | 13:29 |
| rlandy|rover | there is some zuul logic in here we're not optimizing | 13:29 |
| rfolco | hmmm | 13:29 |
| rfolco | true | 13:29 |
| rfolco | there is one in periodic with 33 hr 29 min | 13:30 |
| rfolco | all green | 13:30 |
| rfolco | and one queued | 13:30 |
| rlandy|rover | I know right .... | 13:30 |
| rfolco | rlandy|rover, gimme a sec, let me get some thoughts from sf-dfg | 13:31 |
| rlandy|rover | rfolco: cool | 13:31 |
| rlandy|rover | thanks | 13:31 |
| rfolco | yw | 13:31 |
| rlandy|rover | oh dear - the upstream gate is also sitting at 17 hours??? | 13:36 |
| *** bogdando has joined #oooq | 13:41 | |
| *** atoth has joined #oooq | 13:43 | |
| rlandy|rover | marios: checked the build logs on that job ... error: File not found: /builddir/build/BUILDROOT/openstack-tripleo-common-9.1.1-0.20180612121053.7adbe75.el7.x86_64/usr/bin/container-check | 13:44 |
| rlandy|rover | did we package container-check? | 13:44 |
| rlandy|rover | http://logs.openstack.org/86/571186/3/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/0e1c43b/logs/delorean_logs/7a/db/7adbe755a74692a346124440c179bcf169c94d79_dev/build.log.txt.gz | 13:45 |
| rlandy|rover | marios: as a side note, you can check sova for job results comparison: http://cistatus.tripleo.org/ | 13:45 |
| rlandy|rover | ah - not bad on thr rhos-13 job - got to container-prep | 13:47 |
| *** bogdando has quit IRC | 13:48 | |
| *** kopecmartin has quit IRC | 13:50 | |
| marios | rlandy|rover: thanks gonna check that build log | 13:51 |
| marios | rlandy|rover: is that a new thing from today (container-check file not found) i think so/haven't seen it before | 13:51 |
| rfolco | rlandy|rover, fyi I am on #sf-dfg listening to paul... we'll have to fix precedence there looks like. I can work on that | 13:52 |
| quiquell | panda, myoung: need another eyes in this error http://logs.openstack.org/71/574671/2/check/tripleo-ci-centos-7-containerized-undercloud-from-release-upgrades/e336ec3/job-output.txt.gz | 13:53 |
| rlandy|rover | rfolco++ | 13:54 |
| hubbot | rlandy|rover: rfolco's karma is now 1 | 13:54 |
| quiquell | panda, myoung: ok, found the issue | 13:55 |
| rlandy|rover | marios: I don't know the latest status on container-check but we were pip installing it - expecting one day it would get packaged - maybe today should have been that day. It's Ian Main's work | 13:56 |
| marios | rlandy|rover: ack thanks | 13:56 |
| myoung | quiquell: just looking at it now... http://logs.openstack.org/71/574671/2/check/tripleo-ci-centos-7-containerized-undercloud-from-release-upgrades/e336ec3/job-output.txt.gz#_2018-06-12_11_44_07_999468 | 13:56 |
| myoung | line 34: syntax error: unexpected end of file | 13:57 |
| rlandy|rover | rfolco: so you want a LP bug for the zuul scheduling issue? | 13:57 |
| rlandy|rover | do you | 13:57 |
| rfolco | rlandy|rover, I will throw the question back to you... do we need it ? | 13:58 |
| rlandy|rover | rfolco; idk how the sf dfg works | 13:59 |
| rlandy|rover | what their bug tracking system is | 13:59 |
| rfolco | rlandy|rover, alan is struggling to delete slaves on nodepool | 13:59 |
| rlandy|rover | rfolco: also depends how soon it can get fixed | 13:59 |
| rfolco | many on delete state | 13:59 |
| rlandy|rover | I know | 13:59 |
| rlandy|rover | been watching them struggle with that all morning | 14:00 |
| rfolco | rlandy|rover, on me... relax and enjoy your rovering | 14:00 |
| rlandy|rover | stacks are clean though | 14:00 |
| rfolco | I can take care of this issue for you | 14:00 |
| *** bogdando has joined #oooq | 14:00 | |
| rlandy|rover | rfolco: man, happy to have you on board! | 14:00 |
| rlandy|rover | one less thing to worry about | 14:00 |
| rfolco | I promise work, I did not promise fixing | 14:01 |
| rfolco | lol | 14:01 |
| rfolco | I will report back to you asap | 14:01 |
| *** kopecmartin has joined #oooq | 14:02 | |
| myoung | arxcruz|ruck: please update status matrix https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @ L33 | 14:06 |
| myoung | arxcruz|ruck: for weekly #tripleo squad status | 14:06 |
| myoung | arxcruz|ruck: minimal the 4x4 matrix of jobs/days from rhos-release dashboard, augment with anything else notable for the #tripleo wider team | 14:06 |
| myoung | arxcruz|ruck: please :) | 14:06 |
| arxcruz|ruck | myoung: ack | 14:07 |
| *** kopecmartin has quit IRC | 14:07 | |
| *** kopecmartin has joined #oooq | 14:07 | |
| *** kopecmartin has quit IRC | 14:07 | |
| *** kopecmartin has joined #oooq | 14:08 | |
| *** ccamacho has quit IRC | 14:20 | |
| *** ccamacho has joined #oooq | 14:20 | |
| *** quiquell is now known as quiquell|off | 14:22 | |
| *** arxcruz|ruck is now known as arxcruz|brb | 14:25 | |
| rfolco | https://review.rdoproject.org/r/14195 Set precedence to normal for openstack-periodic | 14:27 |
| ykarel | rfolco, is ^^ temporary? | 14:28 |
| ykarel | as we need to promote packages from promotion jobs | 14:28 |
| rfolco | no, it should be definite | 14:28 |
| rfolco | and it should be low not normal | 14:28 |
| myoung | marios, sshnaidm, and I gave the community meeting +10 mins an no one showed, if folks want/need to chat about $allThingsCI, the room is open, happy to return, else cancelled. | 14:28 |
| ykarel | rfolco, not sure how it would go as we might skip most of the cron runs for this | 14:29 |
| rfolco | myoung, I think its hard to join right after community meeting when you are busy with something.... it would be much easier to attend in a specific time | 14:30 |
| rlandy|rover | weshay: panda: ^^ pls weigh in here | 14:30 |
| rlandy|rover | I am not sure how we want to solve this issue | 14:30 |
| myoung | rfolco: I concur re: a scheduled time, vs. "after the #tripleo meeing ends" - which is a variable start time and harder to plan for. The counterpoint is that as #tripleo assembles for their weekly meeting and it's scheduled for a full hour (but rarely is) - it's 'open time'. I'll bring this up in retrospective tomorrow (or marios will, he has some thoughts on the topic as well). | 14:31 |
| myoung | I'll pop in now as well in case folks see the internal calendar invite start time of now | 14:32 |
| weshay | rfolco, did you attend the #tripleo mtg? | 14:33 |
| panda | community meeting overlaps with UA sync if tripleo meeting ends early | 14:33 |
| weshay | rlandy|rover, /me reads | 14:33 |
| weshay | how far back? | 14:33 |
| rfolco | weshay, no, sorry, was discussing rdo nodepool issue and working on a fix. | 14:34 |
| weshay | rfolco, k.. reminder.. attending the #tripleo mtg is mandatory | 14:35 |
| weshay | for this group | 14:35 |
| rfolco | weshay, ack | 14:36 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 14:40 |
| weshay | rfolco, rlandy|rover before we lower precedence we should consider other options because that has a large negative effect on pipelines | 14:40 |
| rlandy|rover | weshay: rfolco; yep - that is why I wanted you to weigh in | 14:42 |
| rlandy|rover | we have a throughput issue | 14:42 |
| rlandy|rover | and the suggestion was just to let the queues empty out | 14:42 |
| rlandy|rover | but we have the 24hr pipeline job and two checks jobs | 14:42 |
| rlandy|rover | that just are never getting scheduled | 14:43 |
| rlandy|rover | the pipeline job worries me more | 14:43 |
| rlandy|rover | because the next 24 hr job will get queued behind it | 14:43 |
| rlandy|rover | if it does not run before then | 14:43 |
| rlandy|rover | weshay: I asked if we could just get rid of the jobs that are as yet unscheduled - ... | 14:56 |
| rlandy|rover | <pabelanger> no, you need to stop zuul to clear the queue | 14:57 |
| rlandy|rover | <pabelanger> I'd just leave it to eventually catch up | 14:57 |
| rlandy|rover | so we're sitting with jobs that are just going to be longer- and longer running | 14:58 |
| *** rfolco has quit IRC | 14:58 | |
| *** rfolco has joined #oooq | 14:58 | |
| * rlandy|rover proposed to give all of openstack dev two days off and clear the queues | 14:58 | |
| weshay | rlandy|rover, rfolco sorry.. bluejeans for 4min? | 14:59 |
| weshay | https://bluejeans.com/u/whayutin/ | 14:59 |
| rlandy|rover | joined | 15:00 |
| weshay | sorry.. I thought I was in | 15:00 |
| *** trown is now known as trown|lunch | 15:02 | |
| marios | rlandy|rover: fyi that issue with container check i found this https://review.rdoproject.org/r/#/c/14143/ and https://review.openstack.org/#/c/573699/ | 15:10 |
| marios | rlandy|rover: (so indeed looks like it is being packaged. maybe a new run on that failing https://review.openstack.org/#/c/571186/3 will pass | 15:10 |
| rlandy|rover | marios; in meeting - will look in a bit | 15:12 |
| *** ccamacho has quit IRC | 15:14 | |
| marios | rlandy|rover: ack was just fyi as you were wondering if we packaged container-check) | 15:17 |
| marios | rlandy|rover: thanks | 15:17 |
| ykarel | marios, it's removed from package | 15:20 |
| ykarel | i mean reverted: https://review.rdoproject.org/r/#/c/14190 | 15:20 |
| ykarel | the updated tripleo-common-containers package is in repo, so recheck should work | 15:21 |
| marios | ykarel: thank you! | 15:21 |
| *** ykarel is now known as ykarel|away | 15:22 | |
| *** saneax has quit IRC | 15:36 | |
| rlandy|rover | ykarel|away: thanks for answering as usual | 15:36 |
| rlandy|rover | marios: all set? let me know if there is still something to look into | 15:37 |
| *** jtomasek is now known as jtomasek|bbl | 15:37 | |
| marios | rlandy|rover: thanks | 15:42 |
| *** bogdando has quit IRC | 15:43 | |
| weshay | quiquell|off, thanks https://review.openstack.org/#/c/574417/ | 15:45 |
| *** hamzy has quit IRC | 15:48 | |
| *** kopecmartin has quit IRC | 16:02 | |
| weshay | sshnaidm, you still around? | 16:08 |
| weshay | sshnaidm, I need the trend on tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades | 16:08 |
| weshay | in sova | 16:09 |
| *** zoli is now known as zoli|gone | 16:14 | |
| *** zoli|gone is now known as zoli | 16:15 | |
| *** trown|lunch is now known as trown | 16:15 | |
| *** agopi has joined #oooq | 16:20 | |
| *** ykarel_ has joined #oooq | 16:22 | |
| *** ykarel|away has quit IRC | 16:24 | |
| weshay | sshnaidm, you have 2 +2's on https://review.openstack.org/#/c/572798/1 | 16:33 |
| weshay | trown, where did we end up w/ 3node support on the libvirt repro? | 16:36 |
| weshay | not supported I thought | 16:36 |
| weshay | but wanted to confirm | 16:36 |
| trown | weshay: ya not supported | 16:38 |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:40 |
| trown | weshay: hardcoded subnode-{0,1}: https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml#L149 so not at all trivial to add support | 16:40 |
| weshay | trown, thanks | 16:41 |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| myoung | weshay, trown, during the libvirt reproducer sprint I did this, which is some of the building blocks to move from "hard coded 2 nodes" to an arbitrary # of nodes --> https://github.com/halcyondude/ansible-role-virtvars | 16:50 |
| myoung | I also have/had patches to use it but we decided it was out of scope and punted it | 16:51 |
| myoung | if it's a goal I have a good starting point already done... | 16:51 |
| myoung | related goal was to use guest agent, which would fit nicely into existing prototype | 16:51 |
| * weshay wishes we had CI on the libvirt repro | 16:54 | |
| rlandy|rover | myoung: weshay: who is our rhos-13 contact? missing overcloud_container_image_prepare.yaml on rhos-13 | 16:54 |
| weshay | rlandy|rover, we are missing a tht? | 16:55 |
| weshay | file | 16:55 |
| rlandy|rover | http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/2018-06-11.2/ | 16:55 |
| *** udesale has quit IRC | 16:55 | |
| ykarel_ | rlandy|rover, jschlueter | 16:55 |
| weshay | OH | 16:55 |
| rlandy|rover | thanks - asking | 16:56 |
| *** ykarel_ is now known as ykarel|away | 16:56 | |
| myoung | rlandy|rover: jschlueter, but jjoyce also can help. likely #rhos-delivery is a good place to start as well | 16:56 |
| jschlueter | rlandy|rover: bad puddle don't use | 16:56 |
| rlandy|rover | jschlueter; ok - last known good puddle? | 16:56 |
| rlandy|rover | nvm | 16:57 |
| rlandy|rover | this looks better | 16:57 |
| rlandy|rover | http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/2018-06-12.1/ | 16:57 |
| rlandy|rover | thanks | 16:57 |
| rlandy|rover | will rerun gate | 16:57 |
| weshay | rlandy|rover, wait | 16:57 |
| jschlueter | rlandy|rover: either use 2018-06-12.1 or 2018-06-11.1 ... last passed ci was 2018-06-08.3 | 16:57 |
| weshay | rlandy|rover, are you picking a random puddle? | 16:57 |
| rlandy|rover | 2018-06-12.1 is ok | 16:57 |
| weshay | rlandy|rover, http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/passed_phase1/ | 16:57 |
| myoung | we should ber picking up passed_phase_1 | 16:57 |
| rlandy|rover | weshay: latest afaict ... | 16:57 |
| rlandy|rover | getting | 16:57 |
| rlandy|rover | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n65 | 16:58 |
| jschlueter | rlandy|rover: 11.2 puddle never passed phase 1 ci | 16:58 |
| myoung | ahh those jobs aren't using the trigger-getbuild.sh script | 16:58 |
| rlandy|rover | also ... | 16:58 |
| weshay | rlandy|rover, you must must must must must must | 16:58 |
| rlandy|rover | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n72 | 16:58 |
| weshay | ONLY | 16:58 |
| weshay | ONLY | 16:58 |
| weshay | ONLY | 16:58 |
| weshay | ONLY | 16:58 |
| weshay | use passed_phase_1 | 16:59 |
| rlandy|rover | don;t thk we need that line | 16:59 |
| rlandy|rover | above all the ONLYs | 16:59 |
| weshay | k | 16:59 |
| myoung | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n7 should be getting current_build, which if we're using the script that the rest uses should always be passed_phase_1 | 16:59 |
| rlandy|rover | libguestfs_kernel_override: 3.10.0-693.5.2.el7.x86_64 - can we kill that? | 17:00 |
| weshay | rlandy|rover, http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n35 | 17:00 |
| weshay | GDM | 17:00 |
| weshay | all these configs should default to passed_phase_1 | 17:00 |
| myoung | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/scripts/trigger-getbuild.sh#n108 | 17:00 |
| weshay | myoung, any implications to making that the default in the release configs | 17:00 |
| weshay | vs.. using a workaround script? | 17:00 |
| rlandy|rover | {{ rhos_release_args }} | 17:00 |
| myoung | weshay: so if current_build is defined, it'll call rhos-release with a puddle id | 17:01 |
| jschlueter | rlandy|rover, weshay: also need to be using rhceph-3-rhel7:latest for OSP 13 | 17:01 |
| weshay | that script should be the exception | 17:01 |
| myoung | if it's not defined, it'll silently take the latest puddle. I'm not crazy about this but it's what we have. guessing those specific jobs are not passing the variable that the rest are passing. rlandy|rover is this the TQ gate job? | 17:01 |
| weshay | imho not the rule | 17:01 |
| weshay | myoung, rlandy|rover we can default to passed_phase_1 in the release config | 17:02 |
| weshay | and not bother defining it w/ the script | 17:02 |
| weshay | others using that script are welcome to keep it in | 17:02 |
| myoung | jschlueter: does rhos-release have the ability to chase a symlink? | 17:02 |
| myoung | e.g. get passed_phase_1 passed vs. a puddle id? | 17:02 |
| weshay | I think it does | 17:03 |
| myoung | yeah then we can bake it into release configs. | 17:03 |
| jschlueter | myoung: yes it does just fine ... provide -P -p <symlink|puddle_id> | 17:03 |
| myoung | the goal for all these when designed was to have any/all jobs take a parameter of what puddle/hash to look at, and be fed it | 17:03 |
| myoung | if we want to bake in phase1 to yml's also fine | 17:04 |
| myoung | can just change the default to use that | 17:04 |
| rlandy|rover | rhceph-3-rhel7:latest - where would we define that | 17:04 |
| rlandy|rover | jschlueter: ^^? | 17:04 |
| jschlueter | rlandy|rover: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n69 | 17:05 |
| *** ykarel|away has quit IRC | 17:05 | |
| jschlueter | docker_ceph_container_config and docker_ceph_deploy_params | 17:05 |
| myoung | rlandy|rover: patch to change default incoming in 1m | 17:06 |
| myoung | (note: it's already done for the other jobs afaict) | 17:06 |
| myoung | just via current_build param | 17:06 |
| myoung | too much confusion over this. agree with weshey that we should just default it in the configs as well | 17:06 |
| *** holser__ has quit IRC | 17:06 | |
| *** hamzy has joined #oooq | 17:09 | |
| myoung | rlandy|rover, weshay, jschlueter: https://code.engineering.redhat.com/gerrit/141358 Default all release configs to passed_phase1 | 17:11 |
| rlandy|rover | looking | 17:11 |
| myoung | rlandy|rover, jschlueter, weshay, arxcruz|brb (fyi/history: ospphase0 jobs that trigger on puddles automagically on internal jobs were updated 17-may to trigger off phase0 (https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=commit;h=381abbed975b86c1b3f0bfd907702f8c6a13eb45) | 17:19 |
| myoung | the release configs updated by patch above (https://code.engineering.redhat.com/gerrit/141358) are afaict only used by TQ/TQE gating jobs. | 17:19 |
| * myoung mumbles "sorry TMI" and goes back to work | 17:20 | |
| jschlueter | :-) | 17:21 |
| rlandy|rover | thanks for the info | 17:22 |
| *** amoralej is now known as amoralej|off | 17:22 | |
| rlandy|rover | myoung: weshay: would like to merge this https://code.engineering.redhat.com/gerrit/#/c/141358/ so I can out another patch in to update rhceph-3-rhel7:lates | 17:24 |
| * myoung nods affermative at rlandy|rover | 17:25 | |
| rlandy|rover | ok - let's go with it | 17:25 |
| myoung | rlandy|rover: i should have updated that last month | 17:25 |
| rlandy|rover | see what happens | 17:25 |
| rlandy|rover | np | 17:25 |
| myoung | w/ changed script defaults | 17:25 |
| rlandy|rover | next patch coming up ... | 17:26 |
| rlandy|rover | and here we go again on rdocloud queens/master promotion | 17:26 |
| rlandy|rover | all queued | 17:26 |
| myoung | rlandy|rover, weshay: (probably more TMI) confirmed that the ovb gate jobs don't use the script to fetch the current IP. the script predates the dlrnapi when multijobs were the success criteria vs. promoter. We had the very real problem of jobs picking up different puddle ID's in each multijob run. They should all get passed_phase1 now. | 17:28 |
| rlandy|rover | myoung: good to know - thanks | 17:34 |
| *** holser__ has joined #oooq | 17:34 | |
| *** holser___ has joined #oooq | 17:36 | |
| *** holser__ has quit IRC | 17:40 | |
| weshay | myoung, rlandy|rover <sigh> | 17:43 |
| weshay | oh wait.. that will be ok | 17:43 |
| * weshay just going through reviews | 17:43 | |
| weshay | rlandy|rover, so just checking you are not passing a build to the jobs though right? | 17:44 |
| rlandy|rover | https://code.engineering.redhat.com/gerrit/141362 | 17:44 |
| rlandy|rover | weshay; ^^ one more to check | 17:45 |
| rlandy|rover | weshay: we should not - just take latest known good | 17:45 |
| rlandy|rover | ie: passed_phase_1 | 17:45 |
| weshay | rlandy|rover, +2, you can merge | 17:45 |
| myoung | weshay: per your feedback, https://code.engineering.redhat.com/gerrit/#/c/141358 slams the default to passed_phase1 in the configs. It was already using that for all the jobs except OVB gates, which don't use the script as part of a "get build" multijob phase | 17:46 |
| rlandy|rover | ok - let's try the rhos-13 gate again | 17:46 |
| rlandy|rover | then back to bm | 17:46 |
| *** marios has quit IRC | 17:49 | |
| *** marios has joined #oooq | 17:49 | |
| rlandy|rover | %gatestatus | 18:04 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:04 |
| *** agopi has quit IRC | 18:15 | |
| weshay | trown, https://review.openstack.org/#/c/574417/ | 18:32 |
| weshay | rlandy|rover, fyi https://review.openstack.org/#/c/574794/ fixes the above hubbot issue | 18:32 |
| weshay | rlandy|rover, although I'm hoping emit_releases_file.py does as well | 18:32 |
| rlandy|rover | looking | 18:33 |
| *** holser__ has joined #oooq | 18:33 | |
| rlandy|rover | i understand that queens needs to kick but how does that stop p->q? | 18:34 |
| rlandy|rover | weshay: ^^ | 18:35 |
| rlandy|rover | not understanding the commit message explanation | 18:35 |
| weshay | rlandy|rover, that's removing queens branches | 18:36 |
| weshay | from kicking the upgrade job | 18:36 |
| rlandy|rover | With the new emit_releases_file.py in play when queens is triggered we should see queens -> master kick vs.. pike to queens for this job. | 18:36 |
| weshay | so only master kicks.. and queens -> master is executed | 18:36 |
| rlandy|rover | I'm +2 on the change | 18:36 |
| *** holser___ has quit IRC | 18:36 | |
| weshay | rlandy|rover, ya.. I'll retest queens kicking by reverting that after fs51 is added to the exception list | 18:37 |
| *** jtomasek|bbl is now known as jtomasek | 18:37 | |
| weshay | rlandy|rover, w/ https://review.openstack.org/#/c/574417/ | 18:37 |
| rlandy|rover | anyways +2'ed the change | 18:37 |
| weshay | thanks | 18:37 |
| rlandy|rover | I get the basic idea | 18:37 |
| rlandy|rover | weshay: nit pick comment on https://review.openstack.org/#/c/574417 - if we need to merge this, I'll +2 now | 18:39 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:40 |
| rlandy|rover | weshay; in other news - there are two pocs running now downstream - envE with fs001 and rhos-13 gate - of we get a pass on either, will make changes to jjb | 18:40 |
| rlandy|rover | if | 18:40 |
| *** agopi has joined #oooq | 18:53 | |
| *** atoth has quit IRC | 19:21 | |
| rlandy|rover | weshay: you around? pls see conversation on #sf-dfg | 19:32 |
| weshay | myoung, can you please ensure all cards have QE | 19:35 |
| weshay | https://trello.com/c/ZPNYHG3F/775-ci-job-make-job-50-gate e.g. | 19:35 |
| weshay | rlandy|rover, /me looks | 19:42 |
| rlandy|rover | weshay: on the upside we're down to 14 hr 50 min on check jobs in rdocloud - which is practically speed results compared with 28 hrs earlier | 19:46 |
| weshay | rfolco, qe on https://trello.com/c/flI683EI/774-ci-job-create-job-37-work-on-queens-and-calls-tripleo-upgrade-updates-workflow | 19:48 |
| weshay | ? | 19:48 |
| weshay | rfolco, can you update the check boxes if you are all set | 19:49 |
| rfolco | weshay, yes, I assume I can move to complete as well | 19:49 |
| weshay | rfolco, ya.. if you are signing off on it | 19:49 |
| weshay | myoung, 0% complete on test https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow | 19:50 |
| *** rfolco is now known as rfolco_doctor | 20:05 | |
| rlandy|rover | %gatestatus | 20:14 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:14 |
| weshay | rlandy|rover, the upstream gate is fairly long, atm due to a packaging bug | 20:24 |
| weshay | rlandy|rover, I've checked it out.. nothing more we can do, I -1'd the appropriate review | 20:25 |
| weshay | rlandy|rover, tempest errors on fs18 | 20:27 |
| rlandy|rover | weshay: k - thanks | 20:29 |
| sshnaidm | weshay, sova is fixed | 20:29 |
| rlandy|rover | checking out the fs035 failures | 20:29 |
| weshay | sshnaidm, thank god | 20:30 |
| weshay | sshnaidm, we sure we have scen 5,6,7,8,9,10? | 20:30 |
| weshay | I know it will only show up if a job runs | 20:30 |
| sshnaidm | weshay, I see 7 8 10 were running | 20:31 |
| sshnaidm | weshay, yeah, it will show up if will run | 20:31 |
| sshnaidm | weshay, every time rdo cloud has problems with networking it causes problems to dockers on sova host.. | 20:32 |
| sshnaidm | weshay, also ruck rover dashboard won't work in this case.. maybe need to find more reliable place | 20:33 |
| weshay | that's nice of it | 20:33 |
| weshay | sshnaidm, got to work w/ what we have | 20:33 |
| *** brault has quit IRC | 20:39 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:40 |
| *** gkadam has quit IRC | 20:42 | |
| *** hamzy has quit IRC | 20:45 | |
| *** bandini has quit IRC | 20:52 | |
| *** bandini has joined #oooq | 20:54 | |
| weshay | rlandy|rover, does arxcruz|brb have a fix for the tempest config issue? | 20:57 |
| weshay | chandankumar, arxcruz|brb https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/9558a2d/undercloud/home/jenkins/tempest.log.txt.gz#_2018-06-12_20_32_25 | 20:57 |
| weshay | ? | 20:57 |
| *** trown is now known as trown|outtypewww | 20:57 | |
| rlandy|rover | weshay: afaik, he chandankumar did | 20:59 |
| rlandy|rover | at least the bug is assigned to him | 20:59 |
| weshay | https://review.openstack.org/#/c/574691/ | 21:00 |
| weshay | rlandy|rover++ | 21:01 |
| hubbot | weshay: rlandy|rover's karma is now 6 | 21:01 |
| weshay | rlandy|rover, nice job on envE | 21:01 |
| rlandy|rover | getting there | 21:01 |
| rlandy|rover | weshay; don't worry about the other configs, I'm on it | 21:01 |
| weshay | rlandy|rover, I don't mind if you want to start on the jjb | 21:02 |
| rlandy|rover | it's all pretty much related | 21:02 |
| rlandy|rover | the jjb is going to be one quickstart call | 21:03 |
| rlandy|rover | also holding out hope for rhos-13 | 21:04 |
| rlandy|rover | weshay: OMG ... 24 pipeline last job - scheduled!!!! | 21:04 |
| rlandy|rover | we're saved | 21:04 |
| rlandy|rover | oh happiness is | 21:05 |
| weshay | HA | 21:05 |
| weshay | rlandy|rover, that job better pass ;) | 21:05 |
| rlandy|rover | 41 hr 5 min | 21:05 |
| rlandy|rover | weshay; even if it doesn't, the next one will kick and not get delayed behind this one | 21:06 |
| myoung | weshay, sshnaidm: I've updated the logstash file list review (https://review.openstack.org/#/c/570896) per your feedback re: adding update files. I also made it depend on a new review to add timestamps to the update logfiles as afaik this is required --> https://review.openstack.org/#/c/574886 | 21:08 |
| weshay | rlandy|rover, https://review.rdoproject.org/r/#/c/13706/ | 21:08 |
| rlandy|rover | voted | 21:11 |
| myoung | weshay: long lines...ok | 21:14 |
| chandankumar | weshay: tosky can +2 on this one https://review.openstack.org/#/c/574691/ | 21:18 |
| arxcruz|brb | chandankumar: i did the +2+w already | 21:19 |
| tosky | weshay, arxcruz|brb, weshay : I disagree with that patch | 21:20 |
| tosky | it's different from what it was discussed | 21:21 |
| tosky | I don't see the results from the RDO CI | 21:21 |
| myoung | weshay: what's the right way to continue a line in yaml, in a shell block using ansible's "|" operator ? | 21:22 |
| arxcruz|brb | tosky: sorry about that, i'll provide another patch with what we agree right after we have this fixed, sounds good for you? | 21:22 |
| arxcruz|brb | tomorrow morning, first thing i'll do, because it's pretty late for both of us today | 21:23 |
| myoung | I can use "shell: >" but now every multiline command (which is all of them) needs to be updated to use "&&" | 21:23 |
| tosky | arxcruz|brb: no, it's not good for me | 21:24 |
| tosky | I hate when people says agrees on one thing and that things are changed completely | 21:24 |
| myoung | weshay: I didn't think 80 col line lengths for this stuff was an issues as most of the rest of TU and our stuff doesn't abide by 80 col... | 21:24 |
| tosky | it's pretty late and it should not have happened, because this patch was discussed several hours ago | 21:24 |
| tosky | and agreed upon | 21:24 |
| arxcruz|brb | tosky: ok, just -w and i'll submit the new one, please bear with me | 21:25 |
| tosky | you can -w, no need for me | 21:25 |
| tosky | and yes, it's late | 21:25 |
| *** tosky has quit IRC | 21:25 | |
| sshnaidm | myoung, you can use regular "\" | 21:32 |
| sshnaidm | myoung, btw, we added logs to master branch only, need to do it for quuens, newton, ocata, pike... | 21:33 |
| myoung | sshnaidm: aye commented in the card to that effect, we need the backport patches | 21:36 |
| rlandy|rover | weshay: fyi ... this is all we will be left with per env in hardware https://code.engineering.redhat.com/gerrit/#/c/141378 - note nodes settings are moving to https://review.openstack.org/#/c/574894/ | 21:45 |
| rlandy|rover | testing this piece out on env E while adding other env settings | 21:47 |
| *** agopi has quit IRC | 21:52 | |
| rlandy|rover | and rhos-13 going for overcloud deploy | 21:52 |
| *** jbadiapa has quit IRC | 21:55 | |
| *** Goneri has quit IRC | 21:56 | |
| *** agopi has joined #oooq | 22:02 | |
| *** myoung is now known as myoung|bbl | 22:07 | |
| *** holser__ has quit IRC | 22:11 | |
| *** holser__ has joined #oooq | 22:31 | |
| *** florianf has quit IRC | 22:36 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 22:40 |
| *** dtrainor has quit IRC | 22:52 | |
| *** holser__ has quit IRC | 22:54 | |
| *** hamzy has joined #oooq | 23:11 | |
| rlandy|rover | close but not yet on rhos-13 | 23:39 |
| rlandy|rover | deploy failed | 23:39 |
| rlandy|rover | | Controller | cadd461d-2cc4-482c-8787-05aafaed3d70 | OS::Heat::ResourceGroup | CREATE_FAILED | 2018-06-12T22:00:00Z | | 23:42 |
| *** dougbtv_ has joined #oooq | 23:45 | |
| *** dougbtv_ has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!