Friday, 2019-03-15

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @  (3 more messages)00:20
*** hamzy has joined #oooq00:22
*** vinaykns has joined #oooq01:21
*** rlandy|bbl is now known as rlandy01:41
*** dsneddon has joined #oooq01:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @  (3 more messages)02:20
*** vinaykns has quit IRC02:26
*** ccamacho has quit IRC02:50
*** skramaja has joined #oooq03:39
*** udesale has joined #oooq03:50
*** chkumar246 is now known as chandankumar03:52
*** ykarel has joined #oooq03:58
*** dsneddon has quit IRC04:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298,  (3 more messages)04:20
*** ykarel has quit IRC04:20
*** ykarel_ has joined #oooq04:20
*** dsneddon has joined #oooq04:50
*** dsneddon has quit IRC05:04
*** dsneddon has joined #oooq05:26
*** chandankumar has quit IRC05:58
*** saneax has joined #oooq06:14
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298,  (3 more messages)06:20
*** chandankumar has joined #oooq06:22
*** dsneddon has quit IRC07:09
*** saneax has quit IRC07:16
*** dsneddon has joined #oooq07:21
*** apetrich has joined #oooq07:25
*** yolanda has joined #oooq07:30
*** kopecmartin|off is now known as kopecmartin07:57
*** jfrancoa has joined #oooq08:08
*** amoralej|off is now known as amoralej08:09
*** jpena|off is now known as jpena08:11
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298,  (3 more messages)08:20
*** ykarel_ is now known as ykarel|lunch08:38
*** jtomasek has joined #oooq08:41
*** derekh has joined #oooq08:47
*** ccamacho has joined #oooq08:48
*** bogdando has joined #oooq08:48
*** ccamacho has quit IRC08:52
*** saneax has joined #oooq08:59
*** dsneddon has quit IRC08:59
*** holser_ has joined #oooq09:03
*** rfolco has quit IRC09:07
*** dsneddon has joined #oooq09:14
zbro/09:24
zbrmarios_|ruck: awake?09:24
marios_|ruckdefine 'awake'09:24
marios_|ruckzbr: i've been on for hours mate09:25
zbrjust asking, it was too quiet here09:25
marios_|ruckcheck em all up to date https://etherpad.openstack.org/p/ruckroversprint7 and everything09:25
zbrmaybe you can help me wf https://review.openstack.org/#/c/640463/09:26
marios_|ruckzbr: review request?09:26
marios_|ruckzbr: or something specific you want me to comment on there09:26
zbrnope, i do not expect you to revie things without checking, it should be self explanatory09:27
zbralso no risk09:27
marios_|ruckzbr: so its a review request?09:28
* marios_|ruck confused :D09:28
zbryes.09:28
marios_|ruckzbr: ack but not immediately - just done a round of those and i need to do something other than stare at gerrit09:28
marios_|rucknoted for next run promise cross my heart etc09:28
* marios_|ruck really need to fix that today https://review.openstack.org/#/c/64309509:29
marios_|ruckzbr: did you see btw http://logs.openstack.org/11/643511/2/check/tripleo-ci-fedora-28-standalone/dca2cab/logs/undercloud/home/zuul/tempest/tempest.html.gz09:29
marios_|ruckgreen09:29
marios_|rucki moved that bug to fix released09:30
marios_|ruckwes bumped the hash last night09:30
zbrif i remember well, we have being told to drop f28 scen jobs, right? https://review.openstack.org/#/c/641447/09:30
marios_|ruckhttps://bugs.launchpad.net/tripleo/+bug/1819877 that bug09:30
openstackLaunchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Fix released] - Assigned to Sorin Sbarnea (ssbarnea)09:30
zbrmarios_|ruck: yes, i know it passed last night.09:30
marios_|rucks w e e e t t09:31
zbri do not expect it to last one week without breaking, but that's why we are here, to fix them.09:31
zbrpanda: https://review.openstack.org/#/c/613795/ ara duplic install removal, already confirmed by sshnaidm|afk09:34
*** ykarel|lunch is now known as ykarel09:40
*** dtantsur|afk is now known as dtantsur09:47
pandazbr: is the commit message related to the change ?09:51
zbrpanda: mostly yes, see sagi comment.09:53
zbrpanda: if you want to have some fun... you can read about https://review.openstack.org/#/c/639951/ -- it seems that reaching consensus on openstack,... is futile.... or why democracy may be bit broken ;) N people, N+1 ideas.09:57
*** saneax has quit IRC10:01
*** saneax has joined #oooq10:01
*** rfolco has joined #oooq10:05
*** rfolco has quit IRC10:09
*** rfolco has joined #oooq10:11
*** dtantsur is now known as dtantsur|brb10:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario004-standalone, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario001-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298,  (3 more messages)10:20
*** dsneddon has quit IRC10:32
arxcruzzbr: so, did fedora 28 job pass ?10:35
*** skramaja has quit IRC10:40
marios_|ruckarxcruz: https://bugs.launchpad.net/tripleo/+bug/1819877/comments/210:40
openstackLaunchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Fix released] - Assigned to Sorin Sbarnea (ssbarnea)10:40
*** udesale has quit IRC10:56
*** udesale has joined #oooq10:57
*** dsneddon has joined #oooq10:58
*** dsneddon has quit IRC11:03
*** saneax has quit IRC11:07
*** saneax has joined #oooq11:07
chandankumararxcruz: kopecmartin do we want to keep the agenda for today's os_tempest meeting on etherpad for easy accessibilty?11:16
arxcruzchandankumar: up to you11:16
arxcruzhonestly, i'm not feeling well to have meeting today, but...11:16
arxcruzmarios_|ruck: WOOOOOOOOTTTTT!!!!11:17
weshayarxcruz something good happened.. tell me!11:28
arxcruzweshay: fedora 28 job pass11:28
weshaythat was so yesterday11:28
arxcruzweshay: why did you send a new patch set yesterday https://review.openstack.org/#/c/642420/ with an old code?11:28
weshayur like omg like so uncool11:28
arxcruzlol11:28
weshayI did?11:29
weshaycrap11:29
weshayI thought it was just rebased11:30
weshaymarios_|ruck anything I can assit w/?11:31
*** weshay is now known as weshay|rover11:31
weshay|roverwhere is that etherpad /me looks11:31
marios_|ruckweshay|rover: o/ etherpad up to date https://etherpad.openstack.org/p/ruckroversprint711:31
marios_|ruckweshay|rover: crossed some stuff etc11:31
marios_|ruckweshay|rover: re the tempest&retagging one i added some comments rocky fails on collect logs like 2019-03-15 05:08:07.372029 | primary | OSError: [Errno 2] No such file or11:32
*** rascasoft has quit IRC11:32
marios_|ruckdirectory: '/home/zuul/tempest'11:32
marios_|ruckhttps://review.openstack.org/#/c/643095/1211:32
weshay|roverya.. arxcruz you nail that down yet? ^11:33
weshay|roverarxcruz or did you stupid boss fuck you up?11:33
* weshay|rover grabs coffee11:34
*** rascasoft has joined #oooq11:34
arxcruzweshay|rover: pretty much, you send on ps 17 a old patch11:36
arxcruzmarios_|ruck: the rocky failure should be handled by chandankumar patch11:37
arxcruzmarios_|ruck: https://review.openstack.org/#/c/640742/ next run should work11:37
*** dsneddon has joined #oooq11:38
* marios_|ruck food biab11:38
marios_|ruckack arxcruz will check in a bit11:38
weshay|roverarxcruz has it worked for you locally?11:39
weshay|rovermarios_|ruck so I'm not so worried about rocky11:41
weshay|rovermarios_|ruck looks like fs35 deployment worked where fs001 does not..11:41
weshay|roverfs35 is very similar to fs001 but ipv611:41
*** tosky has joined #oooq11:43
weshay|roverapetrich ping.. ronelle may need your eyes today or tomorrow on an issue we think may be related to mistral11:43
*** dsneddon has quit IRC11:43
weshay|roverapetrich we're double checking11:43
weshay|rovermarios_|ruck have you reviewed https://review.openstack.org/#/c/643187/  I recall now what the design was around container-prep11:46
weshay|rovermarios_|ruck basically it was to never set container_build_id directly11:48
weshay|roverand only set it in container-prep11:48
*** saneax has quit IRC11:49
marios_|ruckweshay|rover: ack i see green fs1 there http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/7cd376d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz11:49
marios_|ruckweshay|rover: sorry wrong log11:49
weshay|rovermarios_|ruck that may just be luck11:49
weshay|rovervs.. the other review11:49
*** saneax has joined #oooq11:49
marios_|ruckweshay|rover: but http://logs.rdoproject.org/87/643187/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/d66b3d7/11:49
*** saneax has quit IRC11:50
*** saneax has joined #oooq11:51
weshay|roverlooks like fs35 timed out11:51
weshay|rover2019-03-15 05:48:14 | Ansible timed out at 3373 seconds.11:51
*** dsneddon has joined #oooq11:55
*** jpena is now known as jpena|lunch11:59
*** dsneddon has quit IRC12:00
*** dtantsur|brb is now known as dtantsur12:01
rfolcopanda, which us is about modifying dlrn promoter and phase1 jobs for stein ?12:02
rfolcopanda, #maybe #848 ?12:03
pandarfolco: promoter doens't need to be modified for stein. https://tree.taiga.io/project/tripleo-ci-board/us/848?milestone=22304912:04
rfolcopanda, just ini ?12:04
weshay|rovermarios_|ruck we may want to push 1 of the 2 changes for the container_build_id today.. I see we're close to a promotion in master12:04
pandarfolco: probably yes12:06
pandarfolco: you have to be sure dlrn end point for stein exists, but I think INfra is done there.12:06
rfolcopanda, cool12:06
marios_|ruckweshay|rover: ack i was hoping we can get the tempest thing sorted and consider merging https://review.openstack.org/#/c/643095/1212:07
marios_|ruckweshay|rover: gonna see why master fails there12:07
weshay|rovermust be related to containerized tempest options as the tempest bug is not impacting all the jobs.. /me needs to look at that12:08
marios_|ruckhm also collect logs12:08
*** dsneddon has joined #oooq12:09
*** matbu has quit IRC12:09
*** saneax has quit IRC12:11
*** saneax has joined #oooq12:12
*** dsneddon has quit IRC12:13
*** derekh has quit IRC12:14
*** panda is now known as panda|lunch12:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)12:20
weshay|rovermarios_|ruck perhaps that's why sshnaidm|afk wanted to get collect_logs in post12:24
marios_|ruckweshay|rover: was there a review? i see errors in http://logs.rdoproject.org/95/643095/12/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/c95a70d/logs/quickstart_collect_logs.log12:24
*** holser_ has quit IRC12:25
*** rlandy has joined #oooq12:26
apetrichweshay|rover, I'm up to it if needed. But I don't ronelle online.12:26
weshay|roverapetrich oh hey12:29
weshay|roverapetrich thanks for getting back..12:29
weshay|roverI see a mistral container build error12:29
weshay|roverthat will need to be fixed.12:29
weshay|roveryou up for that  one? lolz12:29
* weshay|rover writing it up now12:29
weshay|roveroh joy right?12:30
apetrichYAY CONTAINERS!! said no one ever12:30
apetrichbut yeah I'm up to it :)12:30
weshay|roverthe best use I've seen is to host old fucking versions of java so you can use ancient bmc consoles12:30
weshay|roverapetrich probably just a packaging bug12:31
apetrichI saw someone installing booting a win 98 in 1:30 mins12:31
rlandyapetrich: hi12:31
apetrichweshay|rover, no worries.12:31
apetrichI will look after it12:32
weshay|roverhttps://bugs.launchpad.net/tripleo/+bug/182027012:32
openstackLaunchpad bug 1820270 in tripleo "ERROR:kolla.common.utils.mistral-base:The command '/bin/sh -c dnf -y install openstack-mistral-common openstack-tacker redis openstack-tripleo-common python-shade ansible skydive-ansible && dnf clean all && rm -rf /var/cache/dnf' returned a non-zero code: 1" [Critical,Triaged]12:32
apetrichweshay|rover, cheers12:32
weshay|roverapetrich so you'll have to use the latest dlrn hash.. tripleo-ci-testing12:32
weshay|roverto recreate that12:32
apetrichcool12:32
apetrichrlandy, do you have a different issue that I can help with or is this ^^12:33
rlandyweshay|rover: rocky ran passed the point of exporting the plan12:33
rlandyapetrich: I do :)12:33
apetrichrlandy, yay! :)12:33
rlandyapetrich: I don;t think it's a mistral bug but I would like some help figuring out what it's complaining about12:34
apetrichrlandy, no worries. How can I help?12:35
rlandyapetrich: the one weshay|rover created is probably more urgent ... but if you can decipher this error, that would be helpful ... https://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/3/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/ac01acf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:35
rlandyfail only in master12:36
rlandyI have removed almost all configs passed and the error remains12:36
apetrichrlandy, I know that causes that.12:37
rlandyapetrich; you do ?? pls tell me so I can fix it12:38
apetrichrlandy, so mistral stores in the database the signatures of the python side of the actions. like what parameters they need.12:38
weshay|rovercomputers, computers cause that12:38
weshay|roverjust turn them off12:38
apetrichrlandy, did you install a patch by hand? or that was from a ci run12:38
apetrichrlandy, because that shows that there's an "validate" parameter that was passed that mistral wasn't expecting.12:39
rlandyapetrich: it has a dependent patch I need a fix for bm undercloud12:39
rlandybut it's from ci12:39
rlandywhere could the parameter be passed from?12:40
rlandyI didn't patch it by hand12:40
apetrichrlandy, usually from the workflow.12:40
rlandyapetrich: ok - so ovb with similar parameters works12:41
apetrichrlandy, the thing that happens is that you add a parameter for the workflow and for the python action and mistral doesn't understand until you run a mistral-db-manage upgrade12:41
rlandythe additional piece I pass is the network-environment12:41
apetrichbut if it was from CI it is weird because it should have done that before the run12:42
rlandycould it be something in here: https://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/3/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/ac01acf/logs/undercloud/home/zuul/network-environment.yaml.txt.gz?12:42
apetrichunlikely12:42
weshay|roverapetrich perhaps we're in between patches landing in master?12:42
weshay|roverI would think ovb and multinode would be complaining12:42
rlandyweshay|rover: I have hadt his happen twoce on master in bm12:42
weshay|roverapetrich Tengu is doing some validation work.. not sure if the overcloud was recently updated12:43
weshay|rovermr Tengu care to dance?12:43
weshay|roverfor a sec12:43
Tenguhmm ?12:43
Tenguweshay|rover: what's up?12:44
apetrichweshay|rover, I saw something there. let me dig12:44
weshay|roverthere is an overcloud validation flag that mistral doesn't like, just wondering if there are any recent patches re: that12:44
rlandyright but ovb does not complain - so the error is likely still on my side - although I don't know where12:44
Tenguweshay|rover: don't think so - care to share the flag?12:44
weshay|roverTengu tbh, not 100% if the validation work you are doing is related at all other than in name12:44
Tenguoh, wait12:44
Tenguthat's rabi12:44
apetrichyay codesearch is down12:45
apetrichthat helps12:45
weshay|roverWHAT12:45
weshay|roverhow am I supposed to work12:45
apetrichhttp://codesearch.openstack.org/?q=validate&i=nope&files=&repos=12:45
Tenguapetrich weshay|rover rabi knows more about that.12:45
apetrichback12:45
Tenguhave to find back, wait.12:45
weshay|roverapetrich I just shit my pants12:45
weshay|roverTengu thanks man12:45
apetrichtoo much open files. someone has closed a file and now it works12:45
weshay|roverlolz12:45
Tenguweshay|rover: https://bugs.launchpad.net/tripleo/+bug/1818926  that one?12:47
openstackLaunchpad bug 1818926 in tripleo "deploy fails with Invalid input [name=tripleo.parameters.update, class=tripleo_common.actions.parameters.UpdateParametersAction, unexpected=[\'validate\']" [High,Fix released] - Assigned to Rabi Mishra (rabi)12:47
weshay|roverTengu ya12:48
weshay|roverthat looks like what rlandy was hitting12:48
Tengu:)12:48
apetrichrlandy, here is the parameter that is it complaning https://github.com/openstack/tripleo-common/blame/master/tripleo_common/actions/parameters.py#L13112:48
weshay|roverThis issue was fixed in the openstack/tripleo-common 10.5.0 release.12:48
Tenguso apparently, fix released, maybe just an update somewhere?12:49
weshay|roverrlandy can you get the tripleo-common rpm12:49
rlandyyes- that is it12:49
weshay|roverversion12:49
apetrichoh great12:49
* Tengu loves his memory and his irc logs12:49
*** dsneddon has joined #oooq12:49
Tengu:)12:49
rlandyyes - logging back in12:49
rlandyMar 14 15:30:34 Installed: openstack-tripleo-common-containers-10.4.1-0.20190314010350.bba82c7.el7.noarch12:51
Tenguthat's it :)12:51
weshay|roverrlandy hrm12:52
weshay|roverwas 10.5 released?12:52
rlandyTengu: thank you12:52
Tengunp :)12:52
Tengupleasure to help ;)12:52
rlandywhy does ovb not get this error?12:52
weshay|roverTengu so 10.4 is the latest12:52
weshay|roverin dlrn12:52
weshay|roverOH wait12:53
weshay|roverrlandy confirm you have dlrn-consistent enabled please12:53
rlandyweshay|rover: I did confirm that yesterday ...12:53
weshay|roverhttps://trunk.rdoproject.org/centos7-master/consistent/openstack-tripleo-common-10.5.1-0.20190314170112.62bbac3.el7.noarch.rpm12:53
rlandygetting12:53
weshay|roverrlandy I think you must be wrong12:53
Tenguhttp://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-master/62/bb/62bbac3d6987aeb70888bab2b636f28e377b4344_bee30a4c_b16f4cf4/ according to http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-master/report.html12:54
weshay|roverwell.. that package was from yesterday12:54
Tenguoh, sorry, that's for rhel8 -.-'. mixing things....12:54
weshay|roverTengu we're not using rhel-8 you bastich12:54
*** dsneddon has quit IRC12:54
weshay|roverrlandy paste me the log again12:55
Tenguyeah.... well... :) can't be perfect, can I?12:55
rlandyTengu: well it is the same error trace12:55
weshay|roverTengu we love you just the way you are12:55
Tenguweshay|rover: awww cute ;)12:55
rlandythe exact same12:55
Tengurlandy: so with the 10.4 package? as the fix is in 10.5...12:56
weshay|roverrlandy Tengu this should get the right packages12:56
weshay|roverhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/master.yml#L6512:56
weshay|roveragree w/ Tengu12:56
weshay|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/3/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/ac01acf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:58
rlandyovb has the same 10.4.1 package12:58
rlandyso I have a package mismatch?12:58
weshay|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/3/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/ac01acf/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz12:58
weshay|roverrlandy Tengu the issue is when rlandy ran it.. it was not available12:59
weshay|roverhttps://trunk.rdoproject.org/centos7/0b/b8/0bb84d576246d50aefbd10f42197ab9f42f5d07e_01511b06/12:59
weshay|roverthat's the dlrn_current repo ^12:59
weshay|roverthis literally was built yesterday12:59
Tengucaching or something?12:59
weshay|roverhowever WHY THE F>> don't more jobs fail on this12:59
weshay|roverTengu no12:59
chandankumarweshay|rover: arxcruz kopecmartin os_tempest meeting time https://redhat.bluejeans.com/1571313919/13:00
weshay|roverjoining...13:00
weshay|roverrlandy rerun.. it will work13:00
rlandyTengu: weshay|rover: ack - let me rerun ... thanks for your help debugging this13:00
Tengurlandy: no problem ;).13:01
*** saneax has quit IRC13:01
weshay|roverrlandy we'll need to get in the habit of comparing packages w/ ovb jobs13:01
chandankumarkopecmartin: arxcruz weshay|rover https://tree.taiga.io/project/tripleo-ci-board/epic/416 epic card13:02
*** saneax has joined #oooq13:02
*** jpena|lunch is now known as jpena13:04
weshay|roverapetrich Tengu thanks guys13:04
weshay|roverrlandy would be interesting to look at an ovb job from the same time and check the tripleo-common rpm version13:06
marios_|ruckdo we have rdo logstash13:07
*** dsneddon has joined #oooq13:10
*** dsneddon has quit IRC13:14
weshay|rovermarios_|ruck don't think so13:15
marios_|ruckweshay|rover: ack filed it anyway with pointers there seen in 3 different ovb master13:16
marios_|ruckhttps://bugs.launchpad.net/tripleo/+bug/182028013:16
openstackLaunchpad bug 1820280 in tripleo "periodic ovb master jobs failing "Stderr: u'Cannot find device "br-ex" & error: Could not connect to cluster" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)13:16
rlandyweshay|rover: the ovb job I looked at had the same tripleo-common13:16
ykarelrlandy, the mistral ovb 'validate' issue is i think because of different tripleo-common on host(from delorea-current) vs tripleo-common on mistral container(from delorean repo)13:23
ykarelcontainers are from 14 days back13:24
ykarellast promotion13:24
weshay|roverya. that is also a good point13:25
chandankumarweshay|rover: https://docs.openstack.org/openstack-ansible-os_tempest/latest/user/configuration.html and https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/os_tempest.html13:25
weshay|roverbut they should get updated13:25
marios_|ruckweshay|rover: so do you think we should instead do https://review.openstack.org/#/c/643187/13:26
weshay|rovermarios_|ruck the only reason why I would think so is because it's a smaller change13:27
weshay|roverbut I'm not 100% sure tbh13:27
marios_|ruckweshay|rover: i mean does it achieve same thing. i kinda liked that we are tidying up the prepare-parameter vs prepare-parameters.yaml for example in https://review.openstack.org/#/c/643095/12 but ovb is still very red there so if https://review.openstack.org/#/c/643187/ is working then lets do that srsly13:27
marios_|ruckweshay|rover: maybe lets recheck https://review.openstack.org/#/c/643187/13:27
marios_|ruckhttp://logs.rdoproject.org/87/643187/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/d66b3d7/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz       tag: "56eedaf13617f1880ce5b1b268db53a774f453c1_a77b7752"13:28
marios_|rucklgtm13:28
marios_|ruckoops13:28
marios_|ruckship it13:28
marios_|ruck:D13:28
marios_|ruckshit it13:28
marios_|ruckwow13:28
marios_|ruckweshay|rover: well this fails in same way as on 'my' one http://logs.rdoproject.org/87/643187/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/9174ca3/job-output.txt.gz 2019-03-15 06:07:55.485428 | primary | sort: write failed: standard output: Broken pipe (collect logs)13:29
marios_|ruckbut the others are green there13:29
marios_|ruckweshay|rover: lets do that https://review.openstack.org/#/c/643187/ do we want to rebase it onto arxcruz patch https://review.openstack.org/#/c/642420/13:40
marios_|ruckweshay|rover: i will abandon https://review.openstack.org/#/c/643095/ wdyt?13:40
weshay|roverin mtg w/ tempest folks13:41
marios_|ruckk tty in a bit then13:41
*** dsneddon has joined #oooq13:43
rlandyykarel; interesting then that we don't hit this more often13:48
rlandyrfolco: hello - how are you doing?13:48
*** dsneddon has quit IRC13:48
rfolcorlandy, I read the docs and need clarification / more context to work on bm13:49
ykarelrlandy, you mean some jobs passing?13:49
rlandyrfolco: k - do you want to bj?13:49
rfolcorlandy, hmmm 1x1 w/ wes now in 10 min13:49
rfolcorlandy, can i ping you after?13:49
rlandyrfolco: sure13:49
rlandyykarel: ack - why so few jobs failed with that error13:50
rfolcorlandy, thanks. Meanwhile I am working on branching us13:50
ykarelrlandy, any ovb or multinode job passed?13:51
rlandyovb13:51
ykarelall should fail13:51
ykarelif they carry package/container difference13:51
ykarelrlandy, good to check passed ovb logs to see what containers/packages are there, i can check if you share logs13:52
weshay|roverrlandy in theory the containers are always yum updated13:52
weshay|roverexcept in promotion jobs13:52
weshay|roverthey would not be13:52
rlandyweshay|rover: so the jobs are named periodic13:53
weshay|roverthe expectation is promtion jobs are built w/ the latest13:53
rlandybut I am kicking them manually to test atm13:53
weshay|roverykarel we resolved the issue w/ ovb.. just debating which path to take13:53
rlandymaybe things will even out when we start triggering properly13:53
ykarelweshay|rover, cool13:53
weshay|roverykarel to many variables too many people working out of context13:54
ykarelhmm agree13:54
*** dtantsur is now known as creepy_owlet13:59
rfolcoweshay|rover, knock knock. May I get in for 1x1?14:00
*** amoralej is now known as amoralej|lunch14:01
arxcruzzbr: why did you abandon?14:02
arxcruzzbr: the qemu patch14:02
arxcruzdid we were able to fix it in another way ?14:02
zbrarxcruz: you will see lots more abandons from me, i started to drop stuff where i do not have time to chase14:03
arxcruz(╯°□°)╯︵ ┻━┻14:03
zbrarxcruz: i can no longer follow up on all open CRs, so I am dropping those that not really urgent. I can restore, also cores can restore if they want.14:04
weshay|roverzbr I need to chat w/ you in about 30 min just for 5 min14:04
weshay|roverrfolco please join, thanks for waiting14:04
rfolcok14:04
zbrsure14:04
kopecmartinarxcruz, chandankumar when you have a moment, please, have a look https://review.openstack.org/#/c/641368/14:05
arxcruzkopecmartin: done14:09
arxcruzkopecmartin: nicely done, love when i see this kind of reusability :)14:10
kopecmartinarxcruz, thanks14:10
*** dsneddon has joined #oooq14:16
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)14:20
*** dsneddon has quit IRC14:20
*** vinaykns has joined #oooq14:20
rlandyykarel: weshay|rover: so master deploy still fails with that same error ...14:35
rlandyMar 15 13:40:09 Installed: openstack-tripleo-common-containers-10.5.1-0.20190315111148.e7103f9.el7.noarch14:36
rlandyMar 15 13:40:09 Installed: openstack-tripleo-common-10.5.1-0.20190315111148.e7103f9.el7.noarch14:36
*** jtomasek has quit IRC14:36
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/7/check/periodic-tripleo-ci-centos-7-baremetal-single_nic-3ctlr_1comp-featureset001-master/1e3d7ee/logs/undercloud/14:36
rlandylogs: ^^14:36
rlandymaybe the jobs are missing a yum update14:37
ykarelrlandy, see tripleo-common https://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/7/check/periodic-tripleo-ci-centos-7-baremetal-single_nic-3ctlr_1comp-featureset001-master/1e3d7ee/logs/undercloud/var/log/extra/podman/containers/mistral_engine/podman_info.log.txt.gz14:37
ykarelit's old14:37
ykareland containers and not updated, you can check container-prepare-parameter.yaml14:38
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/25/165325/7/check/periodic-tripleo-ci-centos-7-baremetal-single_nic-3ctlr_1comp-featureset001-master/1e3d7ee/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz14:39
ykarelyes no update params ^^14:39
marios_|ruckweshay|rover: i think all those f28 bugs in https://etherpad.openstack.org/p/ruckroversprint7 are the same14:39
rlandymaybe it's my job naming - checking cases14:39
marios_|ruckweshay|rover: they are all missing the same packages  sysvinit-tools  python-shade  skydive-ansible  python-novajoin  python2-networking-ansible  python2-networking-baremetal14:40
rlandyupdate_containers: false - default14:40
*** agopi has joined #oooq14:42
rlandyykarel: ok - so he update_containers param is false by default and set in toci testenv https://github.com/openstack-infra/tripleo-ci/blob/05cb96dc298981639de642523d6a47979b9297c9/toci-quickstart/config/testenv/ovb-rdocloud.yml#L26 which baremetal jobs do not pick up14:44
rlandythanks for pointing me to that14:44
rlandyI will add to baremetal envs14:44
ykarelack14:45
*** saneax has quit IRC14:49
*** saneax has joined #oooq14:49
*** dsneddon has joined #oooq14:56
*** jaosorior has quit IRC14:56
weshay|rovermarios_|ruck sorry for the delay14:57
weshay|rovermarios_|ruck I think they are all missing different packages14:57
weshay|rovermaybe I'm wrong14:57
*** amoralej|lunch is now known as amoralej14:57
weshay|rovermarios_|ruck want to sync before you leave?14:58
marios_|ruckweshay|rover: sure sync . i saw the same missing packages in all of them like in the ehtperad14:58
marios_|rucketherpad14:58
weshay|roverk /me looks and joins blue14:58
marios_|ruckweshay|rover: k gimme 2 mins pls and joining sec brb14:59
weshay|rovermarios_|ruck https://review.rdoproject.org/r/#/c/19342/1/fedora_constraints15:02
*** ykarel is now known as ykarel|away15:12
*** ykarel_ has joined #oooq15:15
weshay|roverarxcruz marios_|ruck rlandy is this ready for workflow ??? https://review.openstack.org/#/c/642420/1815:16
arxcruzweshay|rover: yup15:17
marios_|ruckweshay|rover: checking15:17
weshay|roverrocky fs001 is working http://logs.rdoproject.org/20/642420/18/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/19fd71a/logs/stackviz/#/testrepository.subunit15:17
arxcruzweshay|rover: then need https://review.openstack.org/#/c/643267 to make standalone run tempest containerized15:18
*** ykarel|away has quit IRC15:18
weshay|roverrlandy with which fs?15:19
weshay|roverupdate containers false15:20
*** ykarel_ is now known as ykarel|away15:20
marios_|ruckarxcruz: weshay|rover do we need to ad to gate too in https://review.openstack.org/#/c/642420/18/zuul.d/layout.yaml otherwise lgtm15:21
marios_|ruckrlandy: ^15:21
rlandymarios_|ruck: decent question15:23
rlandyweshay|rover: fs00115:24
rlandyit is not set in fs15:24
rlandyit's in the toci env files15:25
zbrtaiga seems very slow today15:32
rlandyzbr: as opposed to every other day when it's never speedy?15:33
*** saneax has quit IRC15:46
*** saneax has joined #oooq15:47
*** udesale has quit IRC15:59
*** agopi has quit IRC16:00
rfolcorlandy, ping bm chat16:06
rlandyrfolco: k - joining bj16:07
rfolcorlandy, pls gimme number again16:07
rlandyhttps://bluejeans.com/116727655616:07
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)16:20
*** kopecmartin is now known as kopecmartin|off16:22
*** bogdando has quit IRC16:24
*** matbu has joined #oooq16:26
*** agopi has joined #oooq16:27
*** agopi_ has joined #oooq16:30
*** agopi has quit IRC16:32
weshay|roverrlandy can you +2 https://review.openstack.org/#/c/643187/  I know it says wip, we want to get it in so promotion jobs have a chance over the weekend16:54
weshay|roverI may update the commit msg.. once I get some job feedback16:54
rlandyyep - just finishing meeting16:54
*** ykarel|away has quit IRC16:55
*** ykarel|away has joined #oooq16:55
rlandyweshay|rover: done - I take your work it's good16:57
rlandyword16:57
*** rlandy is now known as rlandy|biab17:00
*** ykarel|away has quit IRC17:00
*** ykarel|away has joined #oooq17:08
ykarel|awaymarios_|ruck, weshay|rover, to clear promotion in weekend you need couple of more things other than https://review.openstack.org/#/c/643187/17:14
ykarel|awayjust in case u missed17:14
ykarel|away1. mistralclient release17:15
ykarel|away2. container-push docker.io should not rely on tripleo-ci-testing tag17:15
ykarel|away^^ two things atleast17:15
*** dsneddon has quit IRC17:15
ykarel|away1. i think will not happen soon, so you can pin mistralclient in rdoinfo17:16
weshay|rover2. promotion script does not17:16
ykarel|awayhttps://github.com/rdo-infra/ci-config/blob/63214f36f97463d0692b3200dd5ec1997c2a6626/ci-scripts/container-push/container-push.yml#L2517:19
ykarel|awayhttps://github.com/rdo-infra/ci-config/blob/63214f36f97463d0692b3200dd5ec1997c2a6626/ci-scripts/container-push/container-push.yml#L4917:20
*** dsneddon has joined #oooq17:20
*** jpena is now known as jpena|off17:21
*** creepy_owlet is now known as dtantsur|afk17:22
weshay|roveroh no17:25
* weshay|rover looks17:25
weshay|roverykarel|away that was dumb17:26
* weshay|rover looks at who did that17:26
weshay|roverprobably attila17:26
ykarel|awayanyway needs to be fixed17:26
weshay|roveraye17:27
weshay|roverwill put something up shortly17:28
weshay|roverlame17:28
weshay|roverykarel|away have a good weekend!!17:28
weshay|roverand thank you for the eyes!!17:28
ykarel|awayalso pin mistralclient that is also required for promotion17:29
* ykarel|away leaving17:29
ykarel|awayhttps://github.com/redhat-openstack/rdoinfo/blob/master/tags/stein-uc.yml#L30417:30
*** rfolco is now known as rfolco|doctor17:31
weshay|roverhrm.. I don't think that tag is used17:31
ykarel|awayneed to pin to the fix https://github.com/openstack/python-mistralclient/commit/0cd6b282920214d2995d1203aa0ad7d33c825b1017:31
ykarel|awaytag is used17:31
weshay|roverah ya it is17:31
weshay|roverbut easily fixed17:32
weshay|roverk17:32
* ykarel|away out17:41
*** ykarel|away has quit IRC17:47
*** rlandy|biab is now known as rlandy17:51
apetrichweshay|rover, so https://bugs.launchpad.net/tripleo/+bug/1820270 is a duplicate of other bugs that happens for missing packages. it will be fixed when https://review.openstack.org/#/c/642772/5 merges18:03
openstackLaunchpad bug 1820268 in tripleo "duplicate for #1820270 Fedora28 container builds broken and reporting missing packages" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)18:03
weshay|roverapetrich aye18:10
weshay|roverugh rlandy can you try to push a fake git-review to rdo18:10
apetrichweshay|rover, I'm closing the lid now. Have a great weekend18:11
*** amoralej is now known as amoralej|off18:12
*** saneax has quit IRC18:13
*** jfrancoa has quit IRC18:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7 (2 more messages)18:20
rlandyyes18:24
*** dsneddon has quit IRC18:24
rlandyweshay|rover: https://review.rdoproject.org/r/1941418:26
*** dsneddon has joined #oooq18:29
weshay|roverrlandy hey there18:29
weshay|roverrlandy can you take quick lok at https://review.rdoproject.org/r/1941518:29
rlandyweshay|rover: hello18:29
weshay|roverrlandy sorry .. getting my car worked on18:29
rlandynp - looking18:30
weshay|roverrlandy sorry it was  the wifi at the shop18:30
rlandywow - forgot we still had scripts from rdo-infra/ci-config18:32
rlandyweshay|rover: looks ok at review- unfortunately not active tests to confirm - need that merged?18:33
weshay|roverrlandy so.. let me paste something18:34
weshay|roverrlandy http://pastebin.test.redhat.com/73834118:35
weshay|roverso you can see full_hash in the rocky log18:35
weshay|roverand you can test the other w/ the playbook bit18:35
rlandyweshay|rover: ack ok- I see18:36
* rlandy likes some evidence if I merge with not much background knowledge18:37
rlandypromote_name': 'tripleo-ci-testing'18:37
rlandyweshay|rover: done18:38
weshay|roverrlandy I don't see that, what line?18:40
rlandyin your paste18:41
rlandy'distro_hash': 'ad69facaaade0755238c4e7502d409cf4c2f870b', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org'18:41
weshay|roveroh..  weird I don't see that key18:51
*** tosky has quit IRC18:53
weshay|roverrlandy so that must only be from prior to change19:01
rlandymerged anyways19:02
weshay|roverok.. thank you.. I'll revert if I see something funny19:03
weshay|roverhave to drive home now.. bbiab19:03
weshay|rovergood shabbos19:03
rlandygood shabbos19:03
*** hubbot1 has quit IRC19:17
*** gouthamr has quit IRC19:17
*** dmellado has quit IRC19:17
zbrrlandy: see if you can wf https://review.openstack.org/#/c/565215/ - it seems as something useful to me.19:29
rlandylooking19:33
rlandyrdo mirror down??19:39
rlandy"http://mirror.regionone.rdo-cloud.rdoproject.org timed out"19:39
*** openstack has joined #oooq19:41
*** ChanServ sets mode: +o openstack19:41
*** gouthamr has joined #oooq19:53
rlandyfyi ...19:53
rlandy<alhernan> jdennis I noticed the alert and am working on it.19:53
rlandy<alhernan> it should online soon.19:53
* weshay|rover back19:56
weshay|roverrlandy ha.. thought something was going on19:57
weshay|roverwas about to say up19:57
weshay|roverbut that is the wrong term19:57
rlandyweshay|rover; yeah - been trying to get this mater job to get to deploy stage19:57
weshay|rovermater?19:58
weshay|rovermaster19:58
weshay|roverah ya19:58
weshay|roverthat will be tough atm19:58
weshay|rovercan you include a patch?19:58
weshay|rovermaster ovb may be buried w/ bugs19:59
weshay|roverbecause everyone is ignoring ovb19:59
weshay|roveratm19:59
weshay|roverstill19:59
weshay|roverand now upstream infra seems borked20:00
weshay|roveris it friday?20:00
rlandyinclude which patch?20:01
rlandyI can  - I have included already20:02
rlandyand yes, it's friday20:02
rlandybefore big family event and everyone is crazy here20:02
rlandyweshay|rover: you want a patch tested?20:03
weshay|roverall of rdo is down20:03
weshay|roverrlandy naw20:03
weshay|roverthinking about merging this as is20:03
rlandycharming20:03
rlandyweshay|rover: you are right - we need to pick another mirror for internal20:04
weshay|roverre: the repro?20:04
rlandyre anything that doesn't explicitly run on rdo cloud20:04
weshay|roverif it runs outside of rdo20:06
weshay|roverwe just shouldn't use a mirror I think20:06
weshay|roverthat's what plain old quickstart does20:06
weshay|roverwe can pull patches directly from docker.io20:06
weshay|roverinstall from pip mirrors20:07
weshay|roverand install packages from centos and rdo repos20:07
rlandyright - the direct approahc20:07
rlandyapproach20:07
rlandyweshay|rover: so - kforde pinged me again re: psi-tripleo-ci-service and if we can log in20:08
weshay|roverguess I can try that now20:09
rlandyif it's up20:09
* weshay|rover is so out of context there20:09
weshay|roverhow much time do you have in ur day?20:09
rlandyanother hour20:09
weshay|roverwant to try now?20:09
weshay|roversince the world is down20:10
rlandyone advantage of daylights saving20:10
rlandysure20:10
* rlandy gets url20:10
* weshay|rover checks my keys20:10
weshay|rovergpg2 --fingerprint                                                                                                                     master✔ thinkdoe ⌚ 14:10:4020:11
weshay|rover---------------------------------20:11
weshay|roverpub   rsa2048 2018-10-08 [SC] [expires: 2021-10-07]20:11
weshay|roverah crap20:13
weshay|roverthat's not match the server20:13
*** dmellado has joined #oooq20:29
*** gouthamr_ has joined #oooq20:29
*** hubbot1 has joined #oooq20:33
*** rlandy has quit IRC21:12
*** rfolco|doctor has quit IRC21:15
*** gouthamr has quit IRC21:22
*** vinaykns has quit IRC21:27
*** agopi__ has joined #oooq22:00
*** agopi_ has quit IRC22:02
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario001-multinode-oooq-container @  (2 more messages)22:21
*** agopi__ has quit IRC22:38
*** rascasoft has quit IRC23:07
*** agopi__ has joined #oooq23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!