Tuesday, 2019-05-21

*** sshnaidm|afk has quit IRC00:04
*** sshnaidm has joined #oooq00:06
*** panda|rover has quit IRC00:10
*** panda has joined #oooq00:11
*** Vorrtex has quit IRC00:33
*** ykarel|away has joined #oooq02:29
*** ykarel|away is now known as ykarel02:44
*** saneax has quit IRC03:09
*** udesale has joined #oooq03:47
*** yolanda_ has quit IRC03:49
*** ratailor has joined #oooq04:46
*** ykarel is now known as ykarel|away05:03
*** raukadah is now known as chandankumar05:06
*** ykarel|away has quit IRC05:08
*** marios_|ruck has joined #oooq05:16
*** marios has joined #oooq05:16
*** marios_|ruck has quit IRC05:17
*** marios_|ruck has joined #oooq05:18
*** ykarel|away has joined #oooq05:20
chandankumarmarios_|ruck: Good morning05:21
chandankumarmarios_|ruck: for rhel8 standalone tempest-> cirros image cache https://code.engineering.redhat.com/gerrit/#/c/170843/ and https://review.opendev.org/#/c/660246/ -> tempest image fs overrirde05:22
marios_|ruckchandankumar: morning thanks will check in bit05:25
*** ykarel|away is now known as ykarel05:28
ykarelmarios_|ruck, can you also check comment in https://review.opendev.org/#/c/660150/05:28
ykareland i also reverted https://review.rdoproject.org/r/#/c/20859/05:29
ykarelfor more context, just forwarded mail to you05:30
marios_|ruckykarel: thanks05:30
ykareland testing blocking ansible 2.8, and seems my guess is correct05:31
chandankumarmarios_|ruck: holy shit https://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/34/check/tripleo-ci-rhel-8-standalone-rhos-15/e101979/logs/tempest.html.gz05:32
chandankumarmarios_|ruck: it passed05:32
chandankumarnow time to make cirros image cache nodepool part05:32
chandankumarsorry wrong language05:33
marios_|ruckchandankumar: cool :D05:33
chandankumaris it possible to test sf-config changes with tripleo-ci internal jobs?05:33
chandankumarmarios_|ruck: I have one more idea, we can create two standalone jobs05:34
chandankumarone which will test cirros image and another rhel 8 image during tempest run05:34
chandankumaras customers will be spinning RHEL 8 instances so05:34
chandankumarin the cloud05:35
*** quiquell has joined #oooq05:38
marios_|ruckchandankumar: no if you mean cos its a trusted project?05:38
*** quiquell has quit IRC05:38
marios_|ruckchandankumar: you have to wait for it to merge before you can run it (sf-config... i *think* is trusted)(05:38
chandankumarmarios_|ruck: no idea, I donot like the idea of trusted project,05:39
marios_|ruckchandankumar: :D05:39
chandankumarnothing can be trusted everything needs to be tested in order to make it trusted05:39
*** quiquell has joined #oooq05:39
*** ykarel_ has joined #oooq05:42
*** udesale has quit IRC05:44
*** ykarel has quit IRC05:44
*** udesale has joined #oooq05:44
*** saneax has joined #oooq05:51
*** ykarel__ has joined #oooq06:01
*** ykarel_ has quit IRC06:03
*** ykarel__ is now known as ykarel06:04
*** zbr_ has quit IRC06:05
*** skramaja has joined #oooq06:25
*** holser_ has joined #oooq06:33
*** jfrancoa has joined #oooq06:45
*** ccamacho has joined #oooq06:56
*** tesseract has joined #oooq07:06
marios_|ruckchandankumar: any idea why wes set that to fix-released? which patches i mean? https://bugs.launchpad.net/tripleo/+bug/182886007:11
openstackLaunchpad bug 1828860 in tripleo "queens, tempest.api.volume multiple failures" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)07:11
chandankumarmarios_|ruck: the jobs started passing from yesterday07:12
chandankumarno idea what changes entered into the system07:12
*** kopecmartin|off is now known as kopecmartin07:12
marios_|ruckchandankumar: k thx07:12
chandankumarmarios_|ruck: we need to fix the logserver logging07:13
chandankumarhttps://review.rdoproject.org/r/#/c/20840/ needs to be merged07:14
marios_|ruckchandankumar: looks like apevec just did07:17
chandankumarmarios_|ruck: yes thanks :-)07:21
*** apetrich has joined #oooq07:28
chandankumarpanda: Hello, needs +2 and +w on https://review.opendev.org/#/c/660246/ for rhel-8 tempest proof: https://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/34/check/tripleo-ci-rhel-8-standalone-rhos-15/e101979/logs/tempest.html.gz07:29
chandankumarpanda: thanks :-)07:29
*** amoralej|off is now known as amoralej07:35
*** ykarel is now known as ykarel|lunch07:35
*** udesale has quit IRC07:35
*** udesale has joined #oooq07:36
*** tosky has joined #oooq07:41
*** apahim_ has joined #oooq07:53
*** jpena|off is now known as jpena07:56
*** bogdando has joined #oooq08:03
*** skramaja_ has joined #oooq08:05
*** skramaja has quit IRC08:07
Tenguhello there! quick question: would the CI consider adding a job for molecule tests? I'm thinking about adding molecule unit testing to the tripleo-validations (https://molecule.readthedocs.io/en/stable/)08:13
TenguI'm pretty sure this will add value to the validations, and make possible to actually CI-test the whole work.08:13
marios_|ruckTengu: you should talk to soren zbr he is away still but has some wip on that08:18
* marios_|ruck going for food biab08:18
marios_|ruckTengu: i can find you pointers when i'm back if you dont find in meantime08:18
*** dtantsur|afk is now known as dtantsur08:20
Tengumarios_|ruck: great, thanks! I'm currently reading molecule doc.08:20
*** d0ugal has quit IRC08:21
*** ykarel|lunch is now known as ykarel08:30
*** derekh has joined #oooq08:43
*** d0ugal has joined #oooq08:56
pandaykarel: sorry for the ignorance what does the buildsys-tag do ?09:05
ykarelpanda, it pushes the build to delorean deps repo09:05
*** apahim_ has quit IRC09:05
ykarelpackage build09:05
*** dtantsur is now known as dtantsur|brb09:06
pandaykarel: thanks09:06
chandankumarmarios_|ruck: Hello09:06
chandankumarmarios_|ruck: since rhel-8 standalone base is working fine, do we want to move other standalone scenario jobs or we wait for weshay to come online for RHEL8 and RHOS15?09:07
pandaykarel: updating to ansible 2.8 is a definitive no-no for us. It removes support for the roles: keyword, that we have still pretty much everywhere.09:07
ykarelpanda, no no for quickstart or tripleo?09:08
pandaykarel: quickstart for sure, for tripleo I'd check the playbooks first09:10
pandaif I can find where they are ...09:10
*** marios has quit IRC09:11
*** marios_|ruck has quit IRC09:11
ykarelpanda, but quickstart uses venv for ansible09:11
ykareland there it must be what it's defined in requirements.txt09:11
pandaykarel: yes, do you know where the tripleo playbooks are ?09:11
*** marios_|ruck has joined #oooq09:11
marios_|ruckchandankumar: ack except it isn't really 'fine'09:12
marios_|ruckchandankumar: the depends on are hacks09:12
marios_|ruckchandankumar: which we need to tidy up today work out how to carry the rest09:12
*** marios has joined #oooq09:12
ykarelpanda, they are distributed, one u can find in tht/common/deploy-tasks.yml09:12
marios_|ruckchandankumar: fantastic we got the green run but we can't merge stuff as is09:13
ykarelpanda, for this particual issue(container build job which mixes ansible package and ansible zuul executor) we just reverted ansible 2.8 bump in Train, atleast some TripleO standalone/multinode jobs passed with ansible 2.809:13
ykarelmarios_|ruck, ^^09:13
ykarelhttps://review.rdoproject.org/r/#/q/topic:ansible28+(status:open+OR+status:merged)09:13
chandankumarmarios_|ruck: let's hope cirros image nodepool also gets merged by evening09:13
*** openstackstatus has quit IRC09:13
*** openstackstatus has joined #oooq09:14
*** ChanServ sets mode: +v openstackstatus09:14
marios_|ruckchandankumar: ack#09:14
ykarelpanda, mainly the playbooks are build on the fly09:15
ykarelyou can see those in jobs logs09:15
pandaykarel: ... I see some roles: still09:16
ykarelpanda, so only two possibilities: those roles executed and passed or not executed at all09:17
ykarelhttps://review.rdoproject.org/r/#/c/20755/09:18
ykarelceph-ansible also passed09:19
pandaykarel: mmhh, re reading documentation, I think I have outdated information. There is no mention anymore on the removal of roles: . Maybe they changed their plans.09:22
ykarelhmm may be, /me not aware of any such plans though09:24
pandamarios_|ruck: what are you working on ?09:29
marios_|ruckpanda: looking at https://bugs.launchpad.net/tripleo/+bug/1829736 but about to switch to the rhel809:30
openstackLaunchpad bug 1829736 in tripleo "periodic-tripleo--master-containers-build is failing with "Failed to import docker-py - cannot import name __version__"" [Critical,In progress]09:30
marios_|ruckpanda: what do you need09:30
marios_|ruckpanda: things look 'calm' in the sense i haven't filed new bugs today09:31
marios_|ruckpanda: up to date on https://etherpad.openstack.org/p/ruckroversprint10 and less on fire with the warlock thing gone09:31
marios_|ruckpanda: whatsup09:31
*** skramaja has joined #oooq09:42
marios_|ruckchandankumar: can you check comment #5 when you can please https://bugs.launchpad.net/tripleo/+bug/1828859 still seeing the same in the periodic run today09:42
openstackLaunchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)09:42
*** skramaja_ has quit IRC09:43
marios_|ruckykarel: so that should fix us for now https://review.rdoproject.org/r/#/c/20860/ periodics containers build i mean09:48
marios_|ruckykarel: the ansible 2.8 revert09:48
ykarelmarios_|ruck, looking09:48
ykarelthe repo09:48
*** holser_ is now known as holser|lunch09:49
ykarelwe also need to remove that from repo09:49
ykarelamoralej, can you clear it ^^09:49
amoralejyes09:49
amoralejgive me a while09:49
marios_|ruckykarel: you mean ansible 2.8 from the repo09:49
marios_|ruckykarel: ok thanks amoralej let us know if we need to do something to help09:50
ykarelmarios_|ruck, yes,09:50
marios_|ruckok thanks ykarel amoralej09:50
amoralejmarios_|ruck, ykarel done10:01
ykarelack10:01
amoralejnot it should fetch 2.710:01
marios_|ruckamoralej: thanks10:01
ykarelyes, trired locally, ansible-2.7 is pulled10:02
amoralejykarel, it only works with ansible-2.8.0 in the venv10:08
amoralejfails with 2.5, 2.6 and 2.710:09
amoralejmaybe we could uninstall ansible before running the docker_login ?10:09
amoraleji guess we don't need it?10:10
amoralejan that stage10:10
ykarelamoralej, ack, /me also noticed the same venv and package part10:10
ykareland yes ansible package should not be needed before docker_login10:11
amoraleji think the problem is related to PYTHONPATH and versions mess10:12
amoralejwhen tries to import docker10:12
amoralejinstead of using /usr/lib/python2.7/site-packages/docker10:12
amoralejit loads /usr/lib/python2.7/site-packages/ansible/module_utils/docker/10:12
chandankumarmarios_|ruck: How can I know here https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/d657319/job-output.txt.gz at which commit tqe master is clonned?10:20
ykarelamoralej, hmm, with earlier version i see docker_common and with 2.8 it's docker under module_utils10:23
amoralejyeah, i think that's the issue10:24
amoralejtogether with some change in load order or something in 2.810:24
amoralejthat's why it works with 2.8 i guess10:24
marios_|ruckchandankumar: can see some info there ? https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/d657319/logs/zuul-variables.yaml10:24
ykarelamoralej, https://github.com/ansible/ansible/commit/0c2bb3da043e38e779f91b6037d0999c1e60b862#diff-5f3137ef16fa6f5a89b619f82d4e6c5710:25
marios_|ruckchandankumar: for the commit not sure10:25
marios_|ruckchandankumar: so you think it need another run maybe?10:25
marios_|ruckchandankumar: can you add a comment at least on the bug for now?10:25
chandankumarmarios_|ruck: yes10:25
chandankumardone!10:28
marios_|ruckpanda: rfolco can you folks check out this https://code.engineering.redhat.com/gerrit/#/c/167407/ when rlandy and weshay come online i think we can try merge that one10:30
ykarelchandankumar, marios i see tempest_roles in fs017, shouldn't that be cleaned instead of the patch that merged?10:30
*** dtantsur|brb is now known as dtantsur10:30
ykarelchandankumar, marios_|ruck i mean https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset017.yml#L17110:31
sshnaidmykarel, do you know if centos7 nodepool image comes with ansible installed?10:32
marios_|ruckykarel: not sure question for chandankumar but i don't see any changes in the fs from reviews we can post one though unless chandankumar disagrees10:32
amoralejykarel, re-tested, so we need removing ansible from the primary host before running the docker_login or get ansible updated to 2.8 in zuul executors10:33
amoralejor running the login as a nested playbook with local ansible10:33
ykarelamoralej, getting ansible 2.8 in zuul executor will take time, so 1st option seems valid for now10:33
amoralejas done in the build-containers10:33
amoralejbut i'd say that's uggly10:33
amoralejyes, i think removing ansible from there is better10:33
chandankumarykarel: https://logs.rdoproject.org/32/20832/3/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/a8164a6/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-20_10_58_0610:34
ykarelsshnaidm, /me doubt on ansible package installed, but can check nodepool log to confirm10:34
marios_|ruckykarel: wondering if that is used elsewhere other jobs i mean fs1710:34
marios_|ruckykarel: other than queens... only thing i could think of but i posted your comments on the bug just now anyway10:34
chandankumarykarel: We need to figoure out which commit is tqe is pointing whne clonned in fs017 job10:34
ykarelauth.tempest_roles 'Member, creator'10:34
ykareli see ^^ in fs017 file10:34
chandankumarykarel: it is needed10:35
marios_|ruckykarel: chandankumar there is also fs17 rocky/queens/pike10:35
*** udesale has quit IRC10:46
*** holser|lunch is now known as holser_10:55
*** amoralej is now known as amoralej|lunch11:01
ykarelmarios_|ruck, yes that job runs in other releases as well11:08
ykarelmarios_|ruck, in short: more work is needed apart from the merged patch to resolve fs017-queens issue11:09
sshnaidmpanda, marios_|ruck please review https://review.opendev.org/#/c/660138/ it's required for ovb internal jobs11:10
marios_|rucksshnaidm: ack11:12
marios_|ruckykarel: thanks11:12
*** panda is now known as panda|rover|eat11:13
sshnaidmmarios_|ruck, for context: https://code.engineering.redhat.com/gerrit/#/c/170796/11:27
sshnaidmmarios_|ruck, it's in internal gerrit, I can't post it in upstream11:27
weshaymarios_|ruck good afternoon11:31
marios_|ruckweshay: o/11:32
marios_|ruckthanks sshnaidm11:32
marios_|ruckweshay: did you see the green job chandankumar++ (and obv rlandy++ )@ https://code.engineering.redhat.com/gerrit/#/c/167540/3411:33
marios_|ruckweshay: but we need to tidy up the hacks11:33
marios_|ruckweshay: or work out where to carry them11:33
marios_|ruckweshay: been trying to do that a bit but also waiting to sync with rlandy and you (waiting to see if python_interpreter works in the config at https://code.engineering.redhat.com/gerrit/#/c/167407/9 now with recheck in the job11:34
marios_|ruckweshay: and updated rlandy's review at v9 @  https://review.opendev.org/66015111:34
marios_|ruckcurrently looking at yours https://review.opendev.org/#/c/659879/511:34
marios_|ruckweshay: been commenting on them as i go11:34
Tenguyay! we get the common_function back in tripleo-ci \o/11:35
chandankumarweshay: https://code.engineering.redhat.com/gerrit/#/c/170843/ needs +w on this for rhel 8 cirros imahe11:38
chandankumaronce merge we can test it directly there11:38
chandankumar*image11:38
weshaychandankumar marios_|ruck ++ I sure did :)11:38
weshaymarios_|ruck chandankumar now we just have to remove hacks11:38
weshaymarios_|ruck you added python_interp to the release.. so we can remove https://review.opendev.org/#/c/660189/5 from the depends on and retry?11:39
marios_|ruckweshay: but it fails because i removed the become /nodes https://review.opendev.org/#/c/660151/9/playbooks/openstack-zuul-jobs/legacy/pre.yaml11:39
weshaychandankumar can you ask the packaging team what we should do re: the mock rpm for rhel8?11:41
weshayI don't see any builds11:41
weshaychandankumar perhaps we can build it and add it to dlrn-deps so we don't need epel11:41
weshaychandankumar context...  we install epel -> install mock -> remove epel immediately11:42
weshayatm11:42
weshaymarios_|ruck so we have to set the interp on the playbook w/ a when clause I guess?11:42
weshaymarios_|ruck should we get on blue?11:43
weshaythis is too slow for me11:43
marios_|ruckweshay: yah i think so11:43
weshaychandankumar join w/ us11:43
weshayhttps://bluejeans.com/u/whayutin/11:44
chandankumarweshay: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=87180111:44
chandankumarweshay:  we can do something like this dnf -y install http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/mock/1.4.14/2.el8eng/noarch/mock-1.4.14-2.el8eng.noarch.rpm11:47
*** marios_|ruck is now known as marios_|ruck|cal12:00
*** jpena is now known as jpena|lunch12:00
*** marios_|ruck|cal is now known as marios_|ruckcall12:00
*** marios_|ruckcall is now known as marios_|ruck12:01
marios_|ruckweshay: chandankumar done v10 https://code.engineering.redhat.com/gerrit/#/c/167407/12:02
*** ratailor has quit IRC12:09
chandankumarweshay: marios_|ruck we can get the mock from rcm-tools repo https://mojo.redhat.com/docs/DOC-1058262#jive_content_id_Distribution_and_availability12:11
*** udesale has joined #oooq12:16
chandankumarweshay: marios_|ruck https://code.engineering.redhat.com/gerrit/#/c/170843/12:19
marios_|ruckchandankumar: https://tree.taiga.io/project/tripleo-ci-board/task/988?kanban-status=144727512:19
weshaymarios_|ruck going to get you the starlink spaceX internet service to avoid the 1.5 second delay :)12:21
marios_|ruckweshay: ;)12:22
weshayradez is in our channel.. love that guy..  go punch toure for me12:22
weshayjust cause12:22
*** amoralej|lunch is now known as amoralej12:25
*** panda|rover|eat is now known as panda|rover12:27
chandankumarmarios_|ruck: please rebase this https://code.engineering.redhat.com/gerrit/#/c/167540/12:29
chandankumarweshay: marios_|ruck updated the patch https://code.engineering.redhat.com/gerrit/167407 for mock12:29
weshaypanda|rover would I be interupting you for a small request? http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/aa9ce85/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz12:29
rfolcodoes anybody have a demo for today's community call ? weshay sshnaidm marios_|ruck arxcruz ?12:29
weshaypanda|rover the tripleo-upgrade job in periodic is trying to pull containers from docker.io vs.. rdo reg12:30
marios_|ruckchandankumar: ack gimme few12:30
weshayshould be a easy fix12:30
arxcruzrfolco: I can12:30
weshay2019-05-21 10:52:50 | 2019-05-21 10:52:50.687 110948 ERROR tripleoclient.v1.tripleo_upgrade.Upgrade [-] Exception: Not found image: docker://docker.io/tripleostein/centos-binary-cinder-api:tripleo-ci-testing: ImageNotFoundException: Not found image: docker://docker.io/tripleostein/centos-binary-cinder-api:tripleo-ci-testing[00m12:30
rfolcoarxcruz, nice! thank you12:30
panda|roverweshay: ok12:31
weshaypanda|rover thanks man12:32
sshnaidmrfolco, mm.. not yet, I think firstly this patch should be merged https://review.rdoproject.org/r/#/c/20548/12:32
rfolcosshnaidm, ack12:32
rfolcosshnaidm, but you don't necessarily need to talk about merged stuff. You can even go through your patch in the demo :)12:33
rfolcosshnaidm, arxcruz will make it today, please consider next week.12:33
sshnaidmrfolco, ack12:34
chandankumarmarios_|ruck: card updated12:35
*** rlandy has joined #oooq12:37
rlandyweshay: hi ... saw https://code.engineering.redhat.com/gerrit/#/c/170843/ just merged. Was an image build job kicked as well?12:39
rlandyotherwise that only happens periodically12:39
weshayrlandy there is one more change to make..  I think marios_|ruck is updating the image so we don't have to create and chmod things like /etc/nodepool12:40
rlandyoh cool - thanks12:40
weshayrlandy if you can trying to work out install-deps for rhel-8 that would be pushing it forward12:40
weshayrlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/34/check/tripleo-ci-rhel-8-standalone-rhos-15/e101979/12:40
*** aakarsh has quit IRC12:40
weshaycheck it out12:41
marios_|ruckweshay: rlandy here ... not sure if we want to create the /private and the rest of the files or just /etc nodepool https://code.engineering.redhat.com/gerrit/17092012:42
marios_|ruckweshay: rlandy maybe worth adding before request image build12:42
weshayagree12:43
*** Goneri has joined #oooq12:43
chandankumarmarios_|ruck: it needs rebase12:43
weshaymarios_|ruck you sure that will have the right perm?12:43
chandankumarnodepool patch12:43
marios_|ruckchandankumar: ack that was quick :D12:43
marios_|ruckchandankumar: but gonna update now anyway thanks12:43
marios_|ruckweshay: no but i don't think we can set 777 :D12:44
marios_|ruckweshay: like https://review.opendev.org/#/c/660151/8..9/playbooks/openstack-zuul-jobs/legacy/pre.yaml12:44
chandankumarrlandy: I tried the rdo log upload from sf hosted https://code.engineering.redhat.com/gerrit/#/c/170519/7/zuul.d/standalone-jobs.yaml does nto worked12:44
rlandychandankumar: k - I'll look at it later today12:46
chandankumarsure, thanks!12:46
rlandymarios_|ruck, really we do run the role to create all those dir12:46
rlandythe issue is the order we ran the roles in12:46
marios_|ruckrlandy: shouldn't hurt to have few mkdirs in https://code.engineering.redhat.com/gerrit/#/c/170920/1/nodepool/elements/virt-customize/rhel8-cloud.yaml if it means we can iterate quickly12:47
marios_|ruckrlandy: ?12:47
marios_|ruckrlandy: or you think not necessary12:47
rlandymarios_|ruck: well, imho, it should match the upstream-rhel-7 image as closely as possible12:47
rlandybut really I am not sure what is in that image :)12:48
rlandythe less rhel-8 specific changes the better12:48
rlandyeither way the job inherits from upstream so the role will run12:48
rlandymarios_|ruck: do you know what was build in upstream-rhel-7?12:48
rlandyhttps://review.opendev.org/#/c/660151/9/zuul.d/base.yaml12:49
rlandy^^ we are switching the order there12:49
panda|roverweshay: is there a bug to close for that standalone upgrade error ?12:49
rlandybecause https://github.com/openstack/tripleo-ci/blob/master/playbooks/nodepool-provider/pre.yaml#L12 didn't work12:50
rlandymarios_|ruck: so as you think - as long as the upstream playbook can still run w/o error12:50
weshaypanda|rover not atm no.. just saw it12:50
*** jpena|lunch is now known as jpena12:52
*** saneax has quit IRC12:53
panda|roverweshay: ok, trying with https://review.opendev.org/66037912:53
panda|roverweshay: I'll try to launch a test on this in rdo12:53
ykarelweshay, panda|rover also notice tripleo-ci-testing tag is used instead of hash12:53
ykarelwhich is not pushed to registry with new container build job12:54
ykarelso need to consider that as well12:54
weshaypanda++ thanks12:55
ykarelweshay, u checked my comment in https://review.opendev.org/#/c/660150/?12:56
ykarelif not can you check please12:56
marios_|ruckrlandy: ok lets see i'll update for now ... wrt minimal changes this is the 'fake' rhel8 element we're talking about here so we should just make it what we need until we switch to the 'real' one12:56
marios_|ruckrlandy: assuming it works always...12:57
rlandymarios_|ruck: removed setting python interp from https://review.opendev.org/#/c/660189/5..7/playbooks/multinode-standalone.yml12:57
rlandy^^ is it set elsewhere12:57
rlandylooks like weshay did that12:58
marios_|ruckrlandy: i added it to the config12:58
rlandythe job config?12:58
marios_|ruckrlandy: here https://code.engineering.redhat.com/gerrit/16740712:58
marios_|ruckrlandy: we don't know if that works yet12:58
panda|roverykarel: mmhh, I'll try to replace dlrn_hash_tag with container_build_id, but I'm not sure if it's valued in this job12:59
ykarelpanda|rover, can you check my comment in https://review.opendev.org/#/c/660150/, and then think about the solutin to fix it12:59
ykareli think u need to go some steps back to fix the job completely13:00
marios_|ruckrlandy: wrt https://review.opendev.org/#/c/660151/8..9/playbooks/openstack-zuul-jobs/legacy/pre.yaml ... i can't set 777 on the directory do you think we need to override default permissions?13:00
marios_|ruckrlandy: i mean in creating on the image for /etc/nodepool/sub_nodes etc13:00
marios_|ruckrlandy: those have 766 but we won't do that... should we try default see what happens13:00
weshayarxcruz not sure if you'll be able to do anything w/ this..  seeing basicops fails in fs001 tests as well in check.. so could be real http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-stein/58a70d7/logs/tempest.html.gz13:00
marios_|ruckrlandy: means iterating image builds though... :/13:01
arxcruzweshay: it's network problem imho, but i'll take a look13:01
rlandymarios_|ruck: yep - hard to tell - we just hacked until it worked13:01
rlandymarios_|ruck: my best guess is to ask admins what is in rhel-713:01
rfolcoweshay, ci community mtg needs to be updated in rhos mtg calendar, can you please update it ? It shows as canceled for me.13:01
rlandyand do the same13:01
rlandysave the build again issue13:02
rlandymarios_|ruck: also - if they can tell us how to kick the build job, that would be helpful13:02
rfolcoweshay, I don't have write access to rhos mtg calendar.13:02
weshayarxcruz fs001 example fyi http://logs.rdoproject.org/33/614633/12/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/a2a6539/logs/tempest.html.gz13:02
rlandythen we don;t have to bug them13:02
sshnaidmrfolco, when is community meeting?13:02
rfolcoin 30 min from now13:02
arxcruzweshay:this is different13:03
rfolco30 min before tripleo irc mtg13:03
radezweshay: home today, license to for tomorrow when I see him :)13:03
arxcruzweshay: same test, different errors, i'll check both13:03
weshayarxcruz thank you sir13:04
weshayrfolco I can't help you.. ask our admin eliska13:05
marios_|ruckrlandy: gona chown zuul:zuul on /etc/nodepool there should work13:05
marios_|rucklets see13:05
rfolcoweshay, arxcruz chandankumar panda|rover marios_|ruck rlandy sshnaidm: ci community mtg starts in 25 min from now. arxcruz will give a 10 min demo on zuul-runner today.13:05
marios_|ruckthanks rfolco13:05
rfolcoso ignore what calendar says, will send email to eliska to fix it.13:06
*** skramaja has quit IRC13:06
marios_|ruckrfolco: in my calendar the time is correct 'rhos team meetings' cal13:06
rfolcohmm, weird13:07
rfolcook then13:07
rlandymarios_|ruck: you can remove the changes in https://review.opendev.org/#/c/660151/9/zuul.d/base.yaml13:07
rlandyand13:08
rlandyhttps://review.opendev.org/#/c/660189/7/playbooks/multinode-standalone.yml13:08
marios_|ruckrlandy: k wdyt https://code.engineering.redhat.com/gerrit/#/c/170920/2/nodepool/elements/virt-customize/rhel8-cloud.yaml   did i miss anything otherwise will reach out for rebuild13:09
marios_|ruckrlandy: looking13:09
marios_|ruckrlandy: k can you comment in gerrit otherwise it is impossible to track13:10
rlandymarios_|ruck: ha ... https://github.com/openstack/project-config/tree/master/nodepool/elements/zuul-worker13:10
rlandyhttps://github.com/openstack/project-config/blob/master/nodepool/elements/zuul-worker/install.d/60-zuul-worker13:10
marios_|ruckrlandy: nice that might be worth looking into if it fails here13:10
marios_|ruckrlandy: looking (creates the user etc?)13:10
rlandymarios_|ruck: see the install and the zuul_user piece13:11
marios_|ruckrlandy: ack it does keys but not directories13:11
marios_|ruckor i missed it13:11
rlandymarios_|ruck: ttps://github.com/openstack/project-config/blob/master/nodepool/elements/zuul-worker/extra-data.d/60-zuul-user13:11
rlandy^^ keys13:11
marios_|ruckrlandy: 16:12 < marios_|ruck> rlandy: ack it does keys but not directories13:12
*** aakarsh has joined #oooq13:12
marios_|ruckrlandy: like /etc/nodepool stuff i added in virt-customize just now13:12
rlandymarios_|ruck: ack - I that was some legacy stuff we still carry around with us13:13
rlandymarios_|ruck: wrt https://code.engineering.redhat.com/gerrit/#/c/170920/2/nodepool/elements/virt-customize/rhel8-cloud.yaml13:14
rlandylooks ok - not sure if we need lines 159-162 but we will find out13:15
rlandyif we can run out own image builds, we can just rerun and take it out if no good13:15
marios_|ruckrlandy: ack... talking of which13:15
marios_|ruckjpena: o/ hello we please need new image build for https://code.engineering.redhat.com/gerrit/#/c/170920/ rhel8-cloud. is it something we can do instead of harrassing you every day?13:15
marios_|ruckjpena: we know there is periodic but we are trying to iterate as fast as we can13:16
marios_|ruckjpena: please thanks sorry13:16
marios_|ruck:D13:16
jpenamarios_|ruck: besides periodic, you need an SF admin to do it, so feel free to ping me13:16
jpenait's just a simple command, no big deal13:16
marios_|ruckjpena: k :/ sorry about that thanks whenever you next can then we need to test that https://code.engineering.redhat.com/gerrit/#/c/170920/13:16
marios_|ruckjpena: thank you very much13:16
marios_|ruckchandankumar: weshay rlandy  fyi ^^^13:17
rlandymarios_|ruck: k- let's try it13:17
rlandyif it's no good, we'll know before jpena EOD13:17
rlandyand we can revert13:17
jpenarlandy: in the US afternoon, you can count on nhicher13:18
marios_|ruckweshay: can't find your username to add you to this review https://code.engineering.redhat.com/gerrit/#/c/170920/Add creation of /etc/nodepool into the fake rhel-8 image for tripleo13:18
marios_|ruckhttps://code.engineering.redhat.com/gerrit/#/c/170920/13:19
marios_|ruckjpena: you're right updating thanks13:19
rlandyjpena: thank you13:19
jpenanp13:20
marios_|ruckjpena: well13:20
marios_|ruckjpena: i think we're gonna try with owner zuul13:20
marios_|ruckjpena: and default perms for now13:21
marios_|ruckjpena: i already have chown zuul:zuul13:21
marios_|ruckjpena: should be OK?13:21
marios_|ruckjpena: problem is i don't know what the 'correct' permissions are there but hoping that making zuul the owner will overcome the issue and defaults should be OK13:22
marios_|ruckjpena: replying on the review i think its ok13:22
rlandymarios_|ruck: I have some concerns about adding ansible_python_interpreter to the settings file. I shoudl work for standalone but may impact bm. If it's a problem when I get to bm, I'll do something about it then13:23
rlandythe virthost and the undercloud are not always the same distro13:24
marios_|ruckjpena: replied https://code.engineering.redhat.com/gerrit/#/c/170920/3 thanks please have a look too rlandy weshay chandankumar wdyt13:24
marios_|ruckrlandy: reading now13:25
marios_|ruckrlandy: ack13:25
chandankumarit is already workflowed13:28
chandankumarmarios_|ruck: let's rebase the main patch13:28
marios_|ruckchandankumar: ack, getting to it now13:29
marios_|ruckchandankumar: but we have a call to join13:29
marios_|ruckbrb13:29
rlandymarios_|ruck: ack - lets try this!13:29
chandankumarmarios_|ruck: weshay rlandy do we want to merge https://code.engineering.redhat.com/gerrit/#/c/167407/ this13:29
chandankumarmock changes stuff13:29
chandankumarfeel free to cleanup the depends on which got merged13:29
rlandychandankumar: not yet - I'd like to see if the python interp change is correct13:29
chandankumarok13:30
rlandyif it works this run, yeah13:30
rlandychandankumar: thanks for fixing the missing tempest image13:30
rlandyrfolco: bj for meeting?13:31
chandankumarrlandy: we also enabled gerritbot for dowstream reviews on sf-ops13:31
weshaychandankumar ack.. I see https://download.devel.redhat.com/rel-eng/RCMTOOLS/latest-RCMTOOLS-2-RHEL-8/compose/BaseOS/x86_64/os/Packages/mock-1.4.14-2.el8eng.noarch.rpm   perhaps we only want to install the one rpm13:32
weshaynot sure if the rest would polute the build?13:32
weshaychandankumar we can create the repo properly and do an includes13:32
weshayif you look at other release file examples13:32
chandankumarweshay: nope it will install only mock13:32
chandankumarand it's dependencies13:32
rfolcoweshay, ci community call13:33
weshayoh ya13:33
weshayyou remove it13:33
rfolcohttps://bluejeans.com/411356779813:33
weshayk13:33
weshayrlandy based on ^ we can remove the exception on the build-test-packages ( maybe )13:33
chandankumarweshay: https://code.engineering.redhat.com/gerrit/#/c/167407/10..11/config/release/rhos-15.yml just removing rcm repo after installing mock13:33
weshaychandankumar aye.. looks good13:34
chandankumardownstream standalone jobs does not have ara also13:35
rlandyweshay: ack - definitely13:35
chandankumarykarel: ping13:37
chandankumarykarel: https://logs.rdoproject.org/32/20832/4/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/a6f38bb/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-21_12_24_0113:39
chandankumarrfolco: what is the bj link for community call?13:39
rfolcochandankumar, https://bluejeans.com/411356779813:39
rfolcowes room13:39
panda|roverjfrancoa: ping13:40
jfrancoapanda|rover: pong, hello o/13:40
panda|roverjfrancoa: I have question regarding standalone upgrade13:41
jfrancoapanda|rover: sure, shoot.13:41
chandankumarrfolco: please add me to the community calender invite13:43
chandankumarykarel: https://logs.rdoproject.org/32/20832/4/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/a6f38bb/logs/undercloud/var/log/extra/rpm-list.txt.gz13:43
jpenarlandy, marios_|ruck the updated image is now ready13:44
rlandyjpena: thanks13:44
marios_|ruckjpena: ty13:45
marios_|ruckjpena++13:45
ykarelchandankumar, yes tempestconf 2.2.0 picked13:52
ykarelbut job passed , how, /me looking ur patch now13:53
chandankumarykarel: ok13:54
chandankumarykarel: will polish the commit message then13:54
ykarelchandankumar, ack13:55
ykarelbut strange removing Member role fixed it13:55
*** holser__ has joined #oooq13:56
*** rf0lc0 has joined #oooq13:57
*** Vorrtex has joined #oooq13:58
weshaysshnaidm sorry13:59
marios_|ruckchandankumar: actually going to try the rebase now...14:02
marios_|ruckchandankumar: fyi14:02
marios_|ruckon the 15 job i mean14:03
chandankumarmarios_|ruck: I am fixing queens now14:03
marios_|ruckchandankumar: cool14:04
marios_|ruckchandankumar: thx14:04
*** tosky__ has joined #oooq14:05
marios_|ruckpanda|rover: isn't the standup in half hour?14:05
panda|rover...14:06
*** derekh has quit IRC14:06
*** tosky has quit IRC14:06
*** holser_ has quit IRC14:06
*** rfolco has quit IRC14:06
*** dalvarez has quit IRC14:06
*** amoralej has quit IRC14:06
panda|rovermarios_|ruck: yyes, my font is very little14:06
marios_|ruckpanda|rover: or should i join14:06
*** udesale has quit IRC14:06
*** sshnaidm has quit IRC14:06
*** ksambor has quit IRC14:06
*** mjturek has quit IRC14:06
marios_|ruckpanda|rover: ack thx14:06
*** udesale has joined #oooq14:07
*** amoralej` has joined #oooq14:08
*** rf0lc0 is now known as rfolco14:12
marios_|ruckrebased chandankumar rlandy https://code.engineering.redhat.com/gerrit/#/c/167540/14:12
rlandythanks14:12
rlandyrun again?14:12
*** sshnaidm has joined #oooq14:12
chandankumarykarel: marios_|ruck: panda|rover here is the queens fix https://review.opendev.org/#/c/660351/214:14
ykarelack14:15
panda|rovergod save the queens.14:15
rfolcoI want to break free14:15
rfolcoGod knows14:15
chandankumarpanda|rover: call the King,14:16
rfolcofix the code coz I want to break free14:16
*** vinaykns has joined #oooq14:17
rfolco(molecule is needed for testing :D)14:25
rfolcomarios_|ruck, sorin from desert smiles14:25
rlandymarios_|ruck: if this run works, I'd like to remove some hack from the reviews so we are only left with what we really need14:27
*** ykarel is now known as ykarel|afk14:27
mariospanda|rover: reminder call in 1 min14:28
mariospanda|rover: ;)14:28
mariosrlandy: yes well thats what i've been trying to do14:28
mariosrlandy: joining call14:28
mariosrlandy: ack14:29
chandankumarmarios: rlandy regarding nodepool changes http://paste.openstack.org/show/751876/14:31
rlandyyeah - we have a lot of legacy stiff to ditch14:32
rlandybut that is a much larger project than this14:32
*** marios_|ruck is now known as marios|ruck|call14:33
*** amoralej` is now known as amoralej14:38
chandankumarweshay: rlandy marios I am logging out, feel free to mail us for tomorrow rhos-15 debugging, thanks, See ya tomorrow :-)14:46
rlandysure14:46
*** chandankumar is now known as raukadah14:46
marios|ruck|callraukadah: thanks14:46
weshayraukadah I'm trying to get an OSP phase 2 build up now14:46
*** tosky__ is now known as tosky14:59
*** hamzy has quit IRC15:07
*** dalvarez has joined #oooq15:12
rlandymarios|ruck|call: marios++ looking good - on tempest15:12
marios|ruck|callrlandy: ack i didn't want to say but it didn't fail immediately so...15:13
marios|ruck|callrlandy:  waiting15:13
marios|ruck|callrlandy: on the dfg:all call15:13
marios|ruck|callhopign this is green cos going to drop after the call15:13
marios|ruck|callrlandy: i will continue tomorrow to try and remove the hacks depending on where you leave off in your day today15:13
rlandymarios|ruck|call: ack - please ping before your EOD so we can confirm what still needs to be done15:13
marios|ruck|callrlandy: sorry didn't make huge progess was ruck half of day15:13
marios|ruck|callrlandy: ack15:13
marios|ruck|callrlandy: but compared to 2 days ago a LOT of patches and hacks there and there has actually been a green job so15:14
marios|ruck|callrlandy++15:14
rlandyteam effort here15:14
*** ksambor_ has joined #oooq15:18
rlandy2019-05-21 11:16:43.862423 | primary | Traceback (most recent call last):15:22
rlandy2019-05-21 11:16:43.862427 | primary |   File "./tempestmail.py", line 22, in <module>15:22
rlandy2019-05-21 11:16:43.862435 | primary |     import requests15:22
rlandy2019-05-21 11:16:43.862440 | primary | ImportError: No module named requests15:22
rlandy2019-05-21 11:16:43.862444 | primary |15:22
rlandy^^ real issue?15:23
marios|ruck|callhttps://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/35/check/tripleo-ci-rhel-8-standalone-rhos-15/a323169/15:24
marios|ruck|call green ?15:25
marios|ruck|callrlandy: so lets merge that weshay  https://code.engineering.redhat.com/gerrit/#/c/167407/ rlandy15:29
marios|ruck|calland this https://review.opendev.org/#/c/660151/15:29
weshaymarios|ruck|call agree, rlandy need review on https://review.opendev.org/#/c/660151/15:32
rlandyweshay: marios|ruck|call: I am not sure if we need https://review.opendev.org/#/c/660151/9/zuul.d/base.y15:33
marios|ruck|callrlandy: ack you wana comment we can flip and recheck15:34
*** hamzy has joined #oooq15:35
weshayya.. worth checking w/o the zuul change w/ the new image.. agree15:37
rlandyweshay: yeah - would like to make fewer changes that impact upstream15:39
rlandyone more run w/o change15:39
rlandythen merge15:39
rlandyok?15:39
rlandyweshay: marios: let's also enable build ... https://review.opendev.org/#/c/660189/7/playbooks/multinode-standalone.yml15:41
weshayrlandy go 4 it15:44
rlandyin progress15:44
*** ykarel|afk is now known as ykarel|away15:45
marios|ruck|callhttps://review.opendev.org/#/c/660151/ rlandy just hit the update button but you beat me to it15:46
rlandymarios: updating the job file15:46
weshayrlandy started anything w/ install-deps.sh? Just curious because if not I'll fire up a vm15:50
*** ykarel|away has quit IRC15:51
rlandyweshay: no - that still needs works and clean up15:51
rlandyweshay: I have the rhel8 box from yesterday if you want15:52
rlandymarios|ruck|call: weshay: kicked a jobs with reduced depends-ons https://code.engineering.redhat.com/gerrit/#/c/167540/3615:52
marios|ruck|callrlandy: nice thanks15:52
rlandyif that works, we have just the install-deps stuff to hash out15:52
*** marios|ruck|call is now known as marios|ruck15:52
weshaynice.. down to two15:53
rlandyyep - https://review.opendev.org/660151 is pretty harmless15:53
rlandyso really one major15:53
rlandyweshay: want to work on install-deps together or you have that sorted? if so, will move on to osp-15 on bm15:54
marios|ruckweshay: rlandy /me begins shutdown sequence need something else?15:55
marios|ruckrlandy: thanks for picking up looks like you're well in15:55
rlandymarios|ruck: not really - will just move the card on taiga out of blocked15:55
rlandyas I think we are close15:55
*** ccamacho has quit IRC15:57
* marios|ruck shutdown sequence 15:57
weshaymarios|ruck \0 thanks man15:58
raukadahweshay: will take a look once jobs finishes15:58
marios|ruckweshay: thanks you raukadah rlandy team ++15:58
raukadahmarios|ruck: you are welcome :-)15:59
marios|ruckrlandy: update taiga/email/fax whatever you prefer for me to pickup tomorrow16:01
* rlandy finds fax16:02
rlandyto update marios|ruck16:02
marios|ruck:D16:02
rlandycarrier pigeon?16:02
rlandyweshay: :( ... still timing out on bm in periodic ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-stein/6e9666a/job-output.txt.gz16:03
rlandyok if  increase job timeout on our tenant?16:03
rlandyweshay:^^16:03
raukadahbetter go for telegraphy16:03
rlandyovercloud deploy is not stuck - it just gets only an hour to go ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-stein/6e9666a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz16:04
*** marios|ruck has quit IRC16:09
*** marios has quit IRC16:09
*** panda|rover has quit IRC16:09
*** holser__ has quit IRC16:09
*** bogdando has quit IRC16:10
*** panda has joined #oooq16:10
*** holser_ has joined #oooq16:11
*** jfrancoa has quit IRC16:11
weshayrlandy you using the very latest centos?16:13
rlandyweshay:on the virthost?16:14
rlandythe image for sure16:14
rlandyI still need to virt-customize the centos image16:15
rlandyand build overcloud images16:15
rlandytakes time16:15
weshayrlandy ovb is down on fs001 master.. would be cool if I could see a bm run on it16:20
weshayrlandy http://logs.rdoproject.org/79/660379/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/62b25d7/logs/tempest.html.gz16:20
weshayso I can determine if it's the infra.. arxcruz fyi ^16:21
rlandychecking last master run16:22
rlandyweshay: timing out again ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/a0966d1/job-output.txt.gz16:23
rlandyI asked nhicher before if he had an issue with my increasing the job timeout, he said no problem from admins16:24
rlandyso we can do thta16:24
rlandythat16:24
rlandyotherwise no way we will finish these jobs16:24
* rlandy puts up review16:24
*** holser__ has joined #oooq16:29
*** holser_ has quit IRC16:30
*** holser__ has quit IRC16:30
*** holser_ has joined #oooq16:31
weshayrlandy if you can have bm fs001 by EOD that would be awesome16:34
weshaybaremetal16:34
weshaycontext http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&fullscreen&panelId=20716:34
rlandyweshay: pls vote on these two before I merge:  https://code.engineering.redhat.com/gerrit/170968 Increase tripleo-ci-internal job timeout16:35
rlandyand16:35
rlandyhttps://code.engineering.redhat.com/gerrit/170970 Increase BM job timeout16:36
rlandyotherwise no show16:36
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/170968/ needs to go first16:37
*** dtantsur is now known as dtantsur|afk16:48
rlandyweshay: thanks merged - kicking bm master - you want tripleo-ci-testing or current-tripleo?16:50
rlandycheck16:50
weshayrlandy current-tripleo16:50
rlandyso current-tripleo16:50
rlandygot it16:50
weshaythanks16:50
*** udesale has quit IRC16:53
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/9997adb7d1314687b4ce4c8503fbcbf1?logfile=console.log16:54
rlandy^^ osp-15 standalone ... looks good with minimal depends-on16:55
rlandyweshay: ^^16:55
weshayaye.. saw16:55
rlandyweshay: https://code.engineering.redhat.com/gerrit/165815 Test master17:00
rlandyjust checking that gets the right release file17:01
*** quiquell has quit IRC17:10
rlandysshnaidm: ping re: https://code.engineering.redhat.com/gerrit/#/c/170910/17:22
rlandysshnaidm: we don;t want ovb config on all rhel jobs17:23
sshnaidmrlandy, I'd say it doesn't hurt17:23
rlandysomething up with bm config17:24
*** jpena is now known as jpena|off17:24
* rlandy checks for unintended changes17:24
rlandysshnaidm: no need17:24
sshnaidmrlandy, I can extract it to different playbook17:24
rlandyhow did baremetal.yml become baremetal-internal????17:27
sshnaidmrlandy, maybe this caused? https://review.opendev.org/#/c/660138/17:31
rlandyhttps://github.com/openstack/tripleo-ci/blob/745ee233679b8f8b08ab36f7f84fc99f24b87e17/roles/common/vars/main.yaml#L5017:31
*** kopecmartin is now known as kopecmartin|off17:31
*** saneax has joined #oooq17:32
rlandysshnaidm: ^^ that is causing it to look for a file called baremetal-internal17:32
rlandyhttps://github.com/openstack/tripleo-ci/blob/master/roles/common/vars/main.yaml#L5017:32
sshnaidmrlandy, and before this rhcloud wasn't defined, right?17:33
rlandydoes not exist17:33
rlandycorrect17:33
rlandynow ...17:33
rlandy  {%- elif nodepool_provider == 'tripleo-ci' -%}17:33
rlandy  internal17:33
rlandy  {%- endif -%}17:33
sshnaidmrlandy, yeah17:33
rlandyif we ran bm on rdoclud before we would have hit this17:33
rlandysshnaidm: can you case that on ovb?17:33
sshnaidmrlandy, I can change to "elif nodepool_provider == 'tripleo-ci' and environment_type == 'ovb'"17:34
sshnaidmrlandy, although a little bit ugly17:34
rlandysshnaidm: ok by me17:35
rlandywhen you have the review, I'll try it with the current test run17:35
sshnaidmrlandy,  https://review.opendev.org/66047417:37
*** ykarel|away has joined #oooq17:37
sshnaidmrlandy, also https://code.engineering.redhat.com/gerrit/17097717:37
rlandysshnaidm: https://code.engineering.redhat.com/gerrit/170977 looks good17:38
rlandytrying tripleo-ci patch - thanks17:39
*** saneax has quit IRC17:41
*** saneax has joined #oooq17:42
*** amoralej is now known as amoralej|off17:46
sshnaidmrlandy, seems like introspection fails in upshift cloud we have: https://sf.hosted.upshift.rdu2.redhat.com/logs/61/168161/23/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-internal/b0c4cc2/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-05-20_22_14_3517:49
sshnaidmor maybe problem with bmc17:50
rlandyok - bm looks better with pacth18:02
rlandylooking at introspection18:02
rlandysshnaidm: can you get on the undercloud and see if you have ipmi control?18:03
rlandyis the instackenv.json correct?18:03
sshnaidmrlandy, not sure I can get on undercloud..18:12
sshnaidmwill try tomorrow to do it manually18:12
rlandysshnaidm: k - ping me tomorrow if you want to work on it together18:13
*** panda has quit IRC18:14
*** panda has joined #oooq18:15
*** holser_ has quit IRC18:16
*** tcw has quit IRC18:19
rlandyweshay: are you working on bindep stuff or can I edit https://review.opendev.org/#/c/659879/?18:51
* rlandy also wants to fix containers-prepare-parameters18:52
weshayrlandy happy to tmate w/ you18:52
weshayit was all kinds of f'd18:52
weshaywhy break a tradition18:52
* weshay gets a soda18:52
rlandyok- give me 5 - I'll ping you with the tmate session18:52
weshayrlandy I have it going18:53
rlandyk18:54
sshnaidmweshay, images were built: https://sf.hosted.upshift.rdu2.redhat.com/logs/61/168161/25/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-internal/0c9b5ad/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-05-21_14_17_2618:56
sshnaidmweshay, just problems with ipmi on cloud, will look tomorrow..18:56
*** sshnaidm is now known as sshnaidm|afk18:56
rlandyweshay:   sudo $(package_manager) install shyaml || true # install when available19:14
rlandyhttps://review.opendev.org/#/c/660151/10/roles/run-test/templates/toci_gate_test.sh.j219:18
rlandyweshay: ^^19:18
*** phyrox has joined #oooq19:32
*** ykarel|away has quit IRC19:36
*** vinaykns has quit IRC19:38
*** vkapalav has joined #oooq19:38
weshayrlandy k.. cool -y is included btw https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/oooq_common_functions.sh.j2#L23219:43
rlandyoh - I see19:43
rlandythat's why it worked19:43
*** Goneri has quit IRC19:48
*** saneax has quit IRC19:51
*** Goneri has joined #oooq20:02
*** Vorrtex has quit IRC20:02
*** dsneddon has quit IRC20:04
rlandyweshay: need to run out for about 20 mins - biab20:05
rlandythere are two master bm jobs running20:06
rlandyone on tripleo-ci and and one on tripleo-ci-testing20:06
*** rlandy is now known as rlandy|biab20:06
weshayk.. thanks20:06
*** rlandy|biab is now known as rlandy20:39
rlandyweshay: pls vote  - fixes change breaking bm https://review.opendev.org/#/c/66047420:43
weshayrlandy fyi.. rhel-8 install-deps works.. I borked something on centos20:44
weshayupdated the code.eng review fyi20:44
rlandyha - you win some you lose some20:44
*** hamzy has quit IRC20:45
phyroxI just saw that tripleo ui is deprecated starting in stein, is there anything to replace it?20:53
weshayphyrox for tripleo-ui I'm afaid not.. horizon will still be avail20:54
phyroxi see20:55
weshayrlandy the latest run got through tempest, so I think we're good.. the odd thing is that the job is canceled when I update a dependency21:19
rlandyweshay: I rechecked it21:20
rlandyit's running again now21:20
rlandyit will get canceled if you update the dep21:20
weshayI saw.. but hold off as I'm updated the install-deps a bit21:20
weshayya.. that is weird right21:21
weshayI think I added the default python to just the test instance... and not my source :)21:21
weshayeasy21:21
weshaylooks like it's working now21:21
weshayso I think I'm probably good on that patch21:21
weshayhttps://review.rdoproject.org/zuul/stream/36a5edf10bc2466f982cc3f9b6d32bde?logfile=console.log21:22
rlandyweshay: not sure if you saw my question on #prodchain-infra-dfg ...21:22
* weshay rereads21:22
rlandy do you want to export all your log on review.rdo ? If yes, perhaps change the post job is the best solution21:22
rlandyweshay: ^^21:22
weshayrlandy where would that change be made, previously it was env config21:24
weshaynot sure now that we're all zuul'd out21:24
rlandyweshay: so I was testing with raukadah on a test-basis21:25
rlandybut we'd make the change on config21:25
rlandytripleo-ci-internal-config21:25
rlandywe'll need this https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml21:25
rlandyequivalent thereof21:25
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/playbooks/base/post.yaml#n1421:26
rlandy^^ change the values here21:26
rlandyon base21:26
weshayah cool21:28
weshaythanks rlandy21:28
weshayugh21:33
weshaywhat's wrong w/ me21:33
rlandyoh - what now?21:33
weshayrlandy just forgot all the changes you helped me correct21:33
rlandyweshay: should we tmate again?21:34
rlandysorry - was busy watching th bm job21:34
weshayrlandy don't think so21:34
weshayrlandy let me resubmit and have you look21:34
weshaydoing so now21:34
rlandyweshay: ack21:34
weshayrlandy https://review.opendev.org/#/c/660508/4/install-deps.sh21:35
rlandylooking21:35
rlandyweshay: https://review.opendev.org/#/c/660508 - two questions/comments21:39
*** aakarsh has quit IRC21:42
weshayrlandy I don't think oooq_common is used locally21:59
weshayor for the repro21:59
*** tesseract has quit IRC22:07
*** vkapalav has quit IRC22:24
*** aakarsh has joined #oooq22:29
*** phyrox has quit IRC22:34
rlandyweshay: looks good with what's running now - so ok22:39
*** tosky has quit IRC22:42
rlandyweshay: so bm finished with failing overcloud deploy on master ... https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/52/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cf99e1e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz23:07
rlandyiiuc, OVB was only failing tempest?23:07
rlandyweird - passed but failed?23:09
rlandyweshay: wait, I lie ... we have it - the tempest failure !!23:09
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/52/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cf99e1e/logs/undercloud/home/zuul/tempest.log.txt.gz23:10
rlandyha - there we go23:10
rlandymachine is yours or arxcruz if you still want it23:10
rlandy2019-05-21 22:44:31.382903 | primary | TASK [validate-tempest : Execute tempest] **************************************23:11
rlandy2019-05-21 22:44:31.467143 | primary | Tuesday 21 May 2019  22:44:31 +0000 (0:00:00.157)       0:01:01.572 ***********23:11
rlandy2019-05-21 22:46:51.612512 | primary | fatal: [undercloud]: FAILED! => {23:11
rlandyperfect!!23:11
weshaynice23:24
weshaybbl have the kids on tues :)23:24
rlandysure23:28
*** Goneri has quit IRC23:52
*** rlandy has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!