Tuesday, 2018-10-30

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb- (4 more messages)00:40
*** agopi is now known as agopi|brb01:57
*** agopi|brb has quit IRC02:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @  (4 more messages)02:40
*** agopi|brb has joined #oooq02:42
*** ykarel|away has joined #oooq02:55
*** ykarel|away is now known as ykarel02:56
*** udesale has joined #oooq03:49
*** ccamacho has quit IRC04:26
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)04:40
*** d0ugal has joined #oooq05:18
*** ykarel has quit IRC05:20
*** d0ugal has quit IRC05:30
*** jaosorior has quit IRC05:32
*** jaosorior has joined #oooq05:32
*** ykarel has joined #oooq05:42
*** d0ugal has joined #oooq05:44
*** ratailor has joined #oooq05:48
*** jbadiapa has quit IRC06:10
*** hamzy_ has joined #oooq06:17
*** hamzy has quit IRC06:18
*** jfrancoa has joined #oooq06:31
*** quiquell|off is now known as quiquell06:34
*** saneax has joined #oooq06:39
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000 (3 more messages)06:40
*** d0ugal has quit IRC07:01
*** jfrancoa has quit IRC07:05
*** jfrancoa has joined #oooq07:06
*** holser_ has joined #oooq07:17
chandankumararxcruz: Hello07:30
chandankumararxcruz: regarding tempestconf failure https://review.openstack.org/#/c/612968/ I think there is a bug in tempestconf here https://github.com/openstack/python-tempestconf/blob/master/config_tempest/main.py#L43507:31
chandankumararxcruz: it is returning none, that's why it went to that set_cloud_config_values method07:32
*** ykarel is now known as ykarel|lunch07:35
*** jbadiapa has joined #oooq07:36
*** holser_ has quit IRC08:00
*** holser_ has joined #oooq08:01
*** ccamacho has joined #oooq08:04
*** gkadam has joined #oooq08:05
*** kopecmartin|off is now known as kopecmartin08:08
quiquellssbarnea: needed to fix confliext at tqe py3 review :-( https://review.openstack.org/#/c/610491/08:09
*** quiquell has quit IRC08:12
*** ykarel|lunch is now known as ykarel08:28
*** huynq has joined #oooq08:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000 (3 more messages)08:40
*** chem has joined #oooq08:42
*** quiquell has joined #oooq08:44
arxcruzchandankumar: do you have a patch for it ?08:48
chandankumararxcruz: nope still digging out08:48
arxcruzkopecmartin: ^08:48
arxcruzkopecmartin: you were the one who wrote that piece of code if i recall correctly08:48
*** d0ugal has joined #oooq08:50
kopecmartinarxcruz, what code?08:51
*** quiquell has quit IRC08:51
chandankumarkopecmartin: it is basically related to this failurehttp://logs.openstack.org/68/612968/6/check/openstack-ansible-functional-centos-7/594c310/job-output.txt.gz#_2018-10-29_12_11_19_65779808:54
chandankumarkopecmartin: https://review.openstack.org/61296808:54
arxcruzkopecmartin: https://github.com/openstack/python-tempestconf/blob/master/config_tempest/main.py#L43508:57
kopecmartinchandankumar, arxcruz not a bug .. you're missing OS_PROJECT_NAME08:57
kopecmartinI commented in the review08:57
arxcruzkopecmartin: diký moc08:58
arxcruzkopecmartin: os_project_name is the same tenant_name right ?08:58
kopecmartinarxcruz, yes, os_project_name is the old tenant_name08:59
arxcruzkopecmartin: awesome, thanks! I already update the patch :)09:00
*** ccamacho has quit IRC09:01
kopecmartinarxcruz, you're welcome09:01
*** sshnaidm|bbl is now known as sshnaidm|off09:02
*** d0ugal has quit IRC09:05
*** bogdando has joined #oooq09:05
*** dsneddon has quit IRC09:14
*** quiquell has joined #oooq09:16
ssbarneawho can help me understand how testenv-client works? ...and if there is a way run it locally.09:18
ssbarneaasking because atm i have no clue about investigating errors like https://logs.rdoproject.org/20/613920/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/5aa760a/job-output.txt.gz09:19
quiquellssbarnea: What's the issue ?09:21
ssbarneaexecution fails and i have no idea what is the error09:21
quiquellssbarnea: btw we have a first deploy successful of f28 at zuul, but failing later on so we are near09:21
quiquellssbarnea: Could be just RDO09:21
quiquellssbarnea: They are having network issues in the nodepool09:22
ssbarneaquiquell : in this case lets review/merge https://review.openstack.org/#/c/613920/ ASAP as nothing on RDO will work without it.09:23
ssbarneaarxcruz kopecmartin ^^ please, is a simple one.09:23
ssbarneamainly fixing faulty logic of using python3 if it was installed.09:23
chandankumarssbarnea: is gear package not available from rpm?09:24
quiquellssbarnea: we have lingering stacks at te-broker09:25
quiquellssbarnea: The error just say that it cannot get a node from te-broker09:25
arxcruzeverytime i see a shell script, with so many if and else...09:26
pandabecause te-broker changed ip09:27
quiquellpanda: How so ?09:27
pandaquiquell: apparently it was rebuilt09:27
chandankumarssbarnea: it is only available in Fedora, if somehow we can get in Centos then we donot need to install from pip09:27
chandankumaras we using sudo there09:28
*** dsneddon has joined #oooq09:29
ssbarneaquiquell : made me stupid, but I am unable to "read" an error message saying this, maybe we should add a clear error message. maybe even a link to a statuspage, if we have any.09:41
ssbarneaanyway, i have a cool idea about how to install gear at runtime, i did it in the past successfully.09:42
quiquellssbarnea: We at least has a error message :-)09:47
*** ccamacho has joined #oooq10:11
quiquellssbarnea: fixing the tqe patch with your comment10:20
*** amoralej|off is now known as amoralej10:24
*** jtomasek has quit IRC10:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)10:40
ssbarneadoes anyone knows why this fails to merge? https://review.openstack.org/#/c/613920/ -- is listed as depends-on on https://review.openstack.org/#/c/613797/10:59
*** jbadiapa has quit IRC11:04
ssbarnearfolco|rover : what's the status on tebroken? should be fixed soon?11:06
panda??11:08
*** ratailor has quit IRC11:11
*** gkadam has quit IRC11:12
*** udesale has quit IRC11:24
quiquellssbarnea: you have to rebase them gerrit rebase or git review not always work11:30
quiquellssbarnea: hard way is git rebase origin/master and git review11:31
ssbarneaquiquel: i did rebase twice, once i even didn't had what to rebase, it was on top. i was wondering if it could be something else that cause the same error.11:31
quiquelland pull --rebae11:33
quiquellssbarnea: let me try11:33
quiquellssbarnea: done, I think now we are good11:38
*** huynq has quit IRC11:45
*** dtantsur|afk is now known as dtantsur11:50
*** amoralej is now known as amoralej|lunch11:56
*** dtrainor has joined #oooq12:01
*** jtomasek has joined #oooq12:01
rfolco|roverssbarnea, do you have a link for tebroker issue ?12:02
ssbarnearfolco|rover sure, https://logs.rdoproject.org/20/613920/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/5aa760a/job-output.txt.gz12:03
weshayquiquell, rdo cloud is down12:07
weshayquiquell, let's chat when you have a sec12:07
*** agopi|brb is now known as agopi12:17
chandankumarweshay: let me know when you are free we can talk today12:18
weshaychandankumar, you have a good connection atm ?12:19
chandankumarweshay: yes I am currently in RH office12:19
weshaychandankumar, k.. give me a minute and I'll jump online12:19
chandankumarweshay: sure12:19
weshaychandankumar, ok.. thanks for waiting..12:25
weshayready12:25
chandankumarweshay: same bj?12:26
weshayhttps://bluejeans.com/411356779812:26
*** zul has quit IRC12:27
*** ykarel is now known as ykarel|afk12:29
*** trown|outtypewww is now known as trown12:33
*** rlandy has joined #oooq12:35
*** radez has joined #oooq12:37
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)12:40
*** ykarel|afk has quit IRC12:44
rlandymarios; hello12:50
rlandymarios: had some success yesterday with saving out the toci and bash files12:50
rlandyhttps://logs.rdoproject.org/78/613678/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/f15e80b/logs/undercloud/home/zuul/src/git.openstack.org/openstack/tripleo-ci/scripts/12:51
rlandymarios: will have to look through how much of this is usable outside of CI12:52
rlandybut at least the templated vars are included12:52
rlandyrfolco|rover: ^^ thanks - got it to work eventually12:52
mariosrlandy: cool will check it out, am having another go at the variables just updated https://review.openstack.org/613571 . i was also trying to get something ara related into there for debug but i realised that getting that (ara) will have to come later, not in run-v3 like my review12:53
rfolco|roverrlandy, I did nothing, glad to see you made it12:53
mariosrlandy: i.e. in collect-logs here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/publish.yml12:53
rlandymarios; nice - looking at your review12:53
rlandyI'll also have to edit v3 to not created the templated scripts but to copy them if we are in the reproducer12:54
mariosrlandy: nothing fancy, just trying to grab the relevant things we need. still didn't work out how to demarshal that required projects am hoping ara will work out for th eplaybooks12:54
rlandysounds good12:54
mariosrlandy: will continue to poke on that collect-logs for the playbooks but i ran out of time to post that wip for testing before the call in 1012:55
marioswell 5 now12:55
rlandymarios: sorry - which collect logs for playbooks?12:55
mariosrlandy: no i meant, in collect-logs, we are also getting the ara stuff and writing out reproducer, so use that playbook for getting the ara playbooks12:56
rlandyah ok12:56
mariosrlandy: https://github.com/openstack/tripleo-quickstart-extras/blob/882071f27439c3695206e3fc2e2fb237b7f1dde3/roles/collect-logs/tasks/publish.yml#L1812:56
mariosrlandy: for example12:56
mariosrlandy: and in that same file later on we are writing reproducer12:56
rlandyok - my work is also going to need some commands to remove the CI pieces if we use the templates rendered like this12:57
rlandyquiquell: ^^ fyi ... let me know what you think ... out in a review to save the toci files etc. after they are rendered with the cars in run-v3 and use those in the reproducer .... https://logs.rdoproject.org/78/613678/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/f15e80b/logs/undercloud/home/zuul/src/git.openstack.org/openstack/tripleo-ci/12:58
rlandysaves us collecting those vars12:58
rlandynot all will be usable outside ci12:58
rlandylike te-broker piece12:59
rlandyrascasoft: sorry about yesterday - things went a bit crazy :) ... looks like we have some success in OVB again - I will try your test job again13:01
rascasoftrlandy, oh that is good to know, but when you say success on OVB where you mean exactly? Inside the reviews?13:01
rlandyyeah - example ... https://review.openstack.org/#/c/613678/13:02
rlandylet's see13:02
rlandyI'll try again  later today13:02
*** jbadiapa has joined #oooq13:02
weshayquiquell, ping13:03
weshayanyone need anything from me?13:03
*** ykarel has joined #oooq13:05
*** udesale has joined #oooq13:08
quiquellweshay: pong13:10
quiquellrlandy: ack, will look13:11
*** jtomasek_ has joined #oooq13:13
*** amoralej|lunch is now known as amoralej13:13
*** jtomasek has quit IRC13:17
quiquellrlandy: Do we want the reproducer to download stuff from the logs dir ? or they will be used to generate the reproducer ?13:17
rlandyquiquell; well, I kind of think the reproducer will call all the playbooks13:20
rlandybut those playbooks can be called with ready vars13:20
quiquellrlandy: ready bars you mean embedded in the reproducer itself ?13:22
quiquells/bars/vars/13:22
rlandyyes13:22
rlandyin the producer itself and13:22
rlandywith toci_gate_test already formed13:23
weshayrfolco|rover, I'm your ruck today13:25
weshayrfolco|rover, let me know if you need help13:25
rfolco|roverweshay, need someone to cover community mtg, conflicts with LT call13:25
weshaymarios, rlandy can you please cover the community mtg today? We have a chat w/ leadershipo13:26
weshayshipO13:26
rlandyweshay: ack13:26
quiquellrlandy: Hummm that's good one, rendered toci_gate_test has the variables, cool !13:29
quiquellrlandy: like it13:29
rlandyquiquell: thanks13:29
quiquellrlandy: is the reproducer going to has the toci_gate_test.sh embeded or it will download it ?13:29
rlandyquiquell: I'm thinking of that problem :)13:29
rlandywe will now how have the saved file13:30
rlandywe can reference it13:30
quiquellrlandy: It's good that we don't depend on log.o.o this stuff is volatile13:30
quiquellrlandy: we can always download a tarball13:30
quiquellrlandy: reproducer and friends13:30
mariosweshay: sure np13:31
quiquellrlandy: even at autoexecutable tarball13:31
rlandyquiquell: idk - really - I'll put in a few diff patches and you guys can let me know what you think is best13:31
rlandywe will need a few files13:31
mariosrlandy: folks you know there's all hands now right?13:31
quiquellrlandy: sure sure, humm autoexecutable tarball maybe is not a bad idea13:31
quiquellmarios: do we ?13:31
rlandymarios: ack - very informative13:31
mariosrlandy: ack13:32
*** vinaykns has joined #oooq13:45
pandashame they called it RHEL all-hands. I lost the first half13:50
rascasoftpanda, so it's just "hands" without the "all-" first half?14:01
pandarascasoft: It's all, without hands14:02
weshayrfolco|rover, have some notes about the gates in sprint21 etherpad14:02
weshayrfolco|rover, you have status in the etherpad for tripleo14:05
weshayFOLKS14:05
weshayare you in #tripleo?14:05
weshayplease follow both the westford mtg and tripleo :)14:05
*** d0ugal has joined #oooq14:06
pandaweshay: rfolco|rover : LT meeting is being canceled14:10
weshayha.. called it14:11
rfolco|roverok14:11
pandaweshay: yes but for the wrong reason ...14:17
pandaweshay: it was actually for the holiday in Israel :)14:17
weshaypanda, ah.. election14:18
*** gkadam has joined #oooq14:21
*** d0ugal has quit IRC14:28
rlandyweshay: lol - "holiday in Israel" - I was thinking what yom tov I missed14:32
weshayright14:32
weshaytu-bish-vote14:32
rlandy:)14:32
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)14:40
weshayhttps://review.rdoproject.org/r/#/c/17102/514:46
weshayrlandy, panda14:46
*** dtantsur is now known as dtantsur|brb14:50
ssbarnearlandy weshay marios : now I need your help https://review.openstack.org/#/c/613250/  - f28 and critical to get in.15:31
ssbarneasorry, wrong link... https://review.openstack.org/#/c/613920/15:32
*** agopi is now known as agopi|brb15:32
ssbarneawhat is even worse is that i think zuul execution will fail, one job timeouted, didn't post the result yet.15:33
quiquellyep we need some core/review effor for new reviews for helping with f28 effort15:34
ssbarneai guess even zuul was distracted by the news15:34
quiquellweshay, rlandy, panda, marios: https://review.openstack.org/#/q/topic:standalone-f28+(status:open+OR+status:merged)15:34
ssbarnearlandy https://review.openstack.org/#/c/613920/ is sorting the problem you observed that python3 was used instead of python2. is a copy/paste form qs.sh (because we didn't had time to implement code deduplication).15:36
ssbarneapanda ^ i think know well the subject.15:36
*** agopi|brb has quit IRC15:37
rlandyssbarnea: ok - I'll test again15:38
rlandyssbarnea: how many patches do  need - just 613920?15:38
ssbarnearlandy and https://review.openstack.org/#/c/613797/15:39
pandassbarnea: yes, this function will be put in quickstart itself and used mainly by the installer15:39
ssbarneathese two are top prio one, with them we should be in a good spot.15:39
rlandyok - looking15:40
quiquellrlandy: ssbarnea ones are about fixing phase1 so they have priority15:40
quiquellrlandy: the others are to continue with f28 effort, we are near overcloud deploys ok with them15:40
quiquellrlandy: I mean standalone15:40
rlandyquiquell: I have bm jobs that are broken atm15:42
rlandytesting there15:42
weshaychandankumar, arxcruz can you guys raise a blocking bug on https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/a0519a4/logs/undercloud/home/zuul/tempest.log.txt.gz15:43
weshayplease15:43
arxcruzjesos, what is that?15:44
quiquellrlandy: with the changes ?15:44
quiquellrlandy: ssbarnea review should fix that15:45
rlandyhere's hoping15:46
*** nhicher has joined #oooq15:47
quiquellSee you tomorrow15:48
*** quiquell is now known as quiquell|off15:48
*** ccamacho has quit IRC15:48
*** ykarel is now known as ykarel|away15:50
*** quiquell|off has quit IRC15:52
*** saneax has quit IRC15:52
chandankumarweshay: arxcruz looking into it15:56
chandankumarlocally it is working fine that might be some problem with other tempest plugin15:57
arxcruzchandankumar: seems to be packating problem in some tempest plugin15:57
arxcruzneed to identify which one15:57
arxcruzweshay: do you have a env up ?15:57
weshaynhicher, ^15:58
chandankumararxcruz: just do podman pull docker.io/tripleomaster/centos-binary-tempest:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b15:58
weshaynhicher, can we source vexxhost creds and bring up a reproducer?15:58
chandankumararxcruz: enter into container tempest run -l will tell what plugin has messed up15:58
nhicherarxcruz, weshay: I can set autohold on nodepool to get env if needed15:58
weshaynhicher, that would rock15:59
weshayarxcruz, you may have better luck in a personal tenant btw15:59
chandankumararxcruz: feel free to take a look, left the finding on irc, I can take a look on the morning15:59
weshaythan we have in the openstack-nodepool tenant15:59
weshaychandankumar, arxcruz it's odd that we're only seeing that in ovb jobs btw16:00
weshayany idea wy tempest upstream is fine in discovery?16:01
arxcruzweshay: need to check16:01
arxcruzweshay: quickstart-extra version in ovb are the same as upstream right ?16:01
weshayya.. both use master16:02
nhicherweshay: I never tested reproducer, because vexxhost was for validating we are able to run ovb job on it, but I can have a look if needed16:02
rlandyssbarnea: ok if I rebase https://review.openstack.org/#/c/613797/?16:02
ssbarnearlandy no because it will never merge, we would loose another 8h+ hours.16:03
rlandynot sure the change will test w/o rebase - I'll try16:03
chandankumarweshay: arxcruz couple of things16:04
ssbarneait would be ok if you could find others to workflow it after rebase, so it would merge.16:04
chandankumarweshay: enable_tempest = False in https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/a0519a4/logs/undercloud/home/zuul/undercloud.conf.txt.gz16:04
chandankumarweshay: docker.io/tripleomaster/centos-binary-tempest:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b - image is pulled from docker registry16:05
ssbarneai am getting bit tired of that game, ~day to just get the +1 from zuul and meanwhile it would need rebase, so is almost impossible to reach the gate.16:05
chandankumarnot local undercloud registery16:05
weshaychandankumar, I don't recall which is proper?16:06
chandankumararxcruz: weshay: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/general_config/featureset010.yml#n90 missing from fs0116:06
weshayit should be the same version16:06
chandankumarweshay: if it is breaking here, then it should also break in all jobs where tempest container is used might be16:06
chandankumarweshay: that is my first guess but I am digging into the container16:07
weshaychandankumar, hold.. why does it matter which registry the container is coming from16:07
weshaychandankumar, ah k16:07
rlandy16:04:34 No match for argument: python*-libselinux16:08
rlandyssbarnea: I can install python2-libselinux16:08
rlandywell it's looking for it16:08
rlandyno - can't find match16:08
ssbarneaweshay rlandy : i need you magic hand for a trick on https://review.openstack.org/#/c/613920/ -- press the rebase and put the votes back, it fail the check due to one timeout, but it want to make +W today.... so hopefully it will merge till tomorrow.16:09
weshaychandankumar, arxcruz hrm.. wonder why we werent consistent on that setting https://review.openstack.org/#/c/577780/9/config/general_config/featureset001.yml16:09
rlandyssbarnea: go ahead - will vote16:10
chandankumararxcruz: http://paste.openstack.org/show/733652/ -> may be ironic tempest plugin16:10
chandankumarneeded to go for food16:10
ssbarnearlandy : will to the same on the other one, me wondering if it would not be better to remove the dependency, as that dependency is needed only for non voting jobs. what do you think?16:12
ssbarnearlandy i already missed one merge due to fail-to-merge-dependency this morning.16:12
*** jtomasek_ has quit IRC16:12
rlandyssbarnea: rebase will help testing16:12
rlandyif the dep breaks any job, better to keep it16:12
arxcruzweshay: the undercloud_enable_tempest was there before, I just move all tempest stuff down the featureset file, probably i miss that...16:13
weshayya.. I would love to reorder all the settings alphebetcially16:13
*** dsneddon has quit IRC16:14
ssbarnearlandy: yep, i rebased both, lost the votes. now i doubt i will get +W on them before end of day :(16:14
rlandy python2-libselinux16:15
rlandygiving me issues16:15
arxcruzchandankumar: not sure if that's the case, the paste you point, it's missing credentials, which is okay if you didn't source or create tempest.conf16:15
arxcruzweshay: https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/a0519a4/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-30_09_10_1816:15
arxcruz2018-10-30 09:10:18 | IOError: [Errno 13] Permission denied: '/var/log/tempest/tempest.log'16:15
arxcruzit seems it's failing because can't write on /var/log/tempest/tempest.log16:15
chandankumararxcruz: Let me dig later on, I need to run16:17
weshayya.. that's what I saw16:18
weshayarxcruz, but why in ovb?16:18
arxcruzweshay: ¯\_(ツ)_/¯ need to digg into it16:18
arxcruzweshay: so, multinode, the tempest.log is generated in tempest homedir16:20
arxcruzweshay:16:21
arxcruzsudo docker run --net=host -i -v /home/zuul/tempest_data/:/home/zuul -e 'PYTHONWARNINGS=ignore:Certificate has no, ignore:A true SSLContext object is not available' -e CURL_CA_BUNDLE= -v /var/log/containers/tempest:/var/log/tempest -v /opt/cache/files/cirros-0.3.5-x86_64-disk.img:/opt/cache/files/cirros-0.3.5-x86_64-disk.img16:21
arxcruzdocker.io/tripleomaster/centos-binary-tempest:3ed8ac0e93367a02ad53d9fa93467057724b6621_fd8eb74b /usr/bin/bash -c 'set -e; /home/zuul/tempest_container.sh'16:21
weshayarxcruz, vs http://logs.openstack.org/00/611800/2/gate/tripleo-ci-centos-7-containers-multinode/f14c4c8/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-30_10_47_3616:23
arxcruzweshay: fair point16:24
arxcruzthis is why i HATE this big shell script polluted with a lot of jinja ifs and elses16:24
arxcruzfreaking hard later to track these things16:24
arxcruzweshay: can we change /var/log/tempest permission just to test ?16:25
weshayarxcruz, happy to hear your thoughts about it16:25
weshayarxcruz, sure16:26
*** dtantsur|brb is now known as dtantsur16:26
arxcruzweshay: I think the problem here is that docker running as root, is creating the /var/log/tempest and it's failing because of that when zuul tries to run and write the logs there16:27
arxcruzit's a guess16:27
arxcruzi'll try to run the reproducer and create an env16:27
weshayarxcruz, thanks.. nhicher ^16:27
weshayarxcruz++16:28
hubbot1weshay: arxcruz's karma is now 716:28
weshaypanda, https://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-email16:29
nhicherweshay, arxcruz: I created the bug report https://bugs.launchpad.net/tripleo/+bug/180068216:30
openstackLaunchpad bug 1800682 in tripleo "tempest: import errors on master" [Undecided,New]16:30
*** dsneddon has joined #oooq16:30
*** trown is now known as trown|lunch16:33
pandaweshay: thanks, added some comments16:39
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)16:40
pandaweshay: http://paste.openstack.org/show/733644/16:40
*** agopi|brb has joined #oooq16:41
*** udesale has quit IRC16:43
*** ykarel has joined #oooq16:46
*** ykarel|away has quit IRC16:48
*** jrist has quit IRC16:51
*** agopi|brb is now known as agopi16:53
*** jrist has joined #oooq16:54
*** quiquell has joined #oooq16:56
*** quiquell is now known as quiquell|off16:56
weshayrlandy, so we need the patch for ovb in upshift?17:01
rlandyweshay: according to kforde, yes17:01
rlandyweshay: alex somehow worked around it17:01
rlandybut his workaround didn't work for me on the one time I tested it17:01
rlandyalso I have not worked on it for a few days17:02
rlandybm broken from python changes and reproducer stuff17:02
rlandyweshay: wrt python changes, we need to upgrade  the bm slave17:03
rlandychatting with ssbarnea17:03
rlandyssbarnea> rlandy bug confirmed on fc2617:03
rlandyweshay: any objection to my dnf upgrading the slave from f26 to f28?17:03
rlandyno match for  python*-libselinux17:04
weshayrlandy, ack +117:06
*** bogdando has quit IRC17:06
weshayrlandy, but it should be there in 2617:06
weshayweird17:06
rlandyweshay: feel free to try :)17:06
weshayrlandy, panda rfolco|rover ok... /me going to try and create a plan around ovb17:06
rlandyI am on 10.16.150.21217:06
rfolco|roverweshay, sounds like a plan17:07
rlandyweshay: ^^17:07
rlandyweshay: I can find libselinux-python17:07
rlandybut not python*-libselinux17:07
weshayoh lord17:08
*** gkadam has quit IRC17:08
pandassbarnea: you still around ?17:08
ssbarneapanda: yes. working on gettting a f26 instance17:08
pandassbarnea: it may be a bit out of date17:08
pandassbarnea: can you cleanup the standalone-f28 topic ?17:09
ssbarneai am downloading it with vagrant now, eta 16min.17:09
pandassbarnea: Abandon what's not  current17:09
pandassbarnea: I think you may want to download something newer than that17:09
ssbarneasure. we can revive them later if needed, let me send you the list, i cannot abandon/restore them myself.17:10
pandassbarnea: ok17:10
*** ykarel is now known as ykarel|away17:11
ssbarnearlandy is right: proposed code in qs.sh fails to install rpms on f26, it was *not* metioned as a supported platform...17:11
rlandyssbarnea: can you point to bug re:  python*-libselinux?17:11
ssbarnearlandy should create a bug for it, i doubt she did.17:12
ssbarneapanda : abandon https://review.openstack.org/591077 https://review.openstack.org/595374 https://review.openstack.org/60083617:12
pandassbarnea: anything else taht we can clean up ?17:15
weshayrfolco|rover, what are you poking at atm?17:15
pandassbarnea: like remove DNM POC WIP NOOOO tags in the title ?17:15
rfolco|roverweshay, looking at the open bugs. Need to sync. Can you bj?17:15
*** ykarel|away has quit IRC17:16
weshayrfolco|rover, can I pause you ..17:16
weshayrfolco|rover, there is some disturbance in the force atm17:16
weshayhttp://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen17:16
rfolco|roverweshay, I am paused. I need to play17:16
* weshay wondering wtf17:17
rlandyssbarnea: no - thought you had - I'm upgrading17:17
weshayrfolco|rover, ya.. we have a trend now http://cistatus.tripleo.org/gates/17:17
rfolco|roverweshay, so this is diff from the bugs you paste in the ruck notes ?17:18
ssbarneapanda: https://review.openstack.org/#/c/610728/ - to abandon: it does not make sense, "when" does not exist at playbook level.17:19
ssbarneadanm, quiquell|off is already off, he has two of the DNM ones.17:21
ssbarneai will try to rebased them to see if there is anything of use remaining in them17:21
ssbarneabut i need to do it manually17:22
*** ccamacho has joined #oooq17:23
ssbarnearlandy : if you are doing the upgrade, I will stop trying to make the code work with f26 and focus on other areas. ok?17:23
rlandyssbarnea: ack - doing the upgrade17:24
rlandyon 26->27 now17:24
rlandythen will run 27-> 2817:24
rlandyie: slave will be 2817:24
ssbarnearlandy try only with 27, i am curious if this is enough17:24
weshayssbarnea, rfolco|rover we have other slaves at f2617:24
rlandyack ok17:24
weshaymost are f2617:24
ssbarneaif we put 28 we will hit py3 issues....17:24
rlandyok - I'll stop at 27 for the moment17:25
rlandystill busy17:25
ssbarneaok, i will have a look at f26 later, after cleaning the list of reviews17:25
*** ykarel|away has joined #oooq17:26
*** gkadam has joined #oooq17:29
*** kopecmartin is now known as kopecmartin|off17:30
rlandyweshay: do you have console access to 10.16.150.212?17:31
weshaywhat is that?17:32
rlandyI started the dnf system-upgrade reboot17:32
rlandyweshay: bm slave17:32
weshayrlandy, so we need to get on the host17:32
rlandybut now I have no idea if it's working/failing/stalling17:32
weshayit's a libvirt guest17:32
weshayrlandy, you know where the host is?17:32
rlandyslave to run BM jobs root@10.16.150.21217:33
rlandyaccording to jenkins17:33
weshayrlandy, ya. .t aht is the virt guest17:33
weshayrlandy,  jump on my blue and we'll go hunting17:33
rlandyhosted on virthost:rhev-i8c-02.mpc.lab.eng.bos.redhat.com17:33
rlandyweshay: I think I may be able to find it ... sec17:33
weshayk17:34
*** panda is now known as panda|off17:37
*** gkadam has quit IRC17:40
ssbarneamaybe someone can explain why this went in limbo for one week https://review.openstack.org/#/c/591540/17:40
ssbarnearlandy weshay panda|off ^^17:41
rlandyweshay: I can access drac but that console does not opens for me17:42
ssbarneaweshay : please just abandon https://review.openstack.org/#/c/595374/17:43
weshayssbarnea, limbo?17:43
ssbarneaweshay : as in nothing happening (no reviews...)17:43
weshayssbarnea, who did you ask for a review?17:43
rlandybut the slave came back to life as f2717:43
weshayssbarnea, we're short on cores man.. so there is that bit of tribal knowledge17:44
ssbarneaweshay i know, not trying to blame anyone :(17:45
weshayssbarnea, http://paste.openstack.org/show/733658/17:46
ssbarneavery bad mix of: few cores x high gate/check random failure rate x big number of changes in parallel (conflicting sometimes) = very low productivity.17:46
weshayssbarnea, you nailed it17:46
rlandyssbarnea: running now with f2717:46
weshaythat is exactly right17:46
ssbarneaweshay maybe you can setup a cron to send such email every friday, could motivate us maybe.17:47
weshayssbarnea, not a bad idea17:48
rlandyssbarnea: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans/37/console17:49
rlandyno show17:49
*** trown|lunch is now known as trown17:50
ssbarnearlandy: working now to update the change to make the yum/dnf command more portable17:51
*** jfrancoa has quit IRC17:53
*** dtantsur is now known as dtantsur|afk17:57
chandankumarweshay: arxcruz testing the changes here https://review.rdoproject.org/r/#/c/13943/ my first guess with this one https://review.openstack.org/#/c/614321/18:05
chandankumarweshay: arxcruz I will take a look tommorrow18:05
*** chandankumar is now known as chkumar|off18:05
weshayk18:05
weshaychkumar|off, thanks18:05
weshayhttps://tree.taiga.io/project/tripleo-ci-board/wiki/notes-from-email18:13
*** holser_ has quit IRC18:15
*** dsneddon has quit IRC18:21
arxcruzchkumar|off: how so if your patch isn't merged yet...18:22
*** dsneddon has joined #oooq18:32
*** amoralej is now known as amoralej|off18:39
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch,  (3 more messages)18:41
ssbarneaapparently my "smart" logic  of using yum/dnf with wildcards fails as the behaviour i rely on is not consistent across platforms18:42
ssbarneamainly i used the fact that yum/dnf was not failing when a wildcard pattern failed to had matches, which was true for tested platforms. aparently not for f26 -- time to build to distintive lists of rpms.18:43
ssbarnealibselinux python bindings is the problematic one because both variants have no file-path in common so we cannot use the provides trick to install the desired one.18:45
ssbarnearlandy weshay : not sure if you still have time, i rebased my molecule change on top of the qs.sh change to test that change cross platform. if you can get https://review.openstack.org/#/c/613672/ and run "make" it would be great.19:11
*** weshay is now known as weshay_doctor19:15
rlandyssbarnea: on it19:17
rlandytrying https://review.openstack.org/#/c/613797/19:19
*** ykarel|away has quit IRC19:32
*** ykarel|away has joined #oooq19:32
*** ykarel|away has quit IRC19:38
rlandyssbarnea: I'm updating your patch19:45
ssbarnearlandy, with what? asking because I already have lots of local fixes for it.19:45
rlandyssbarnea: ok - if you have it under control19:46
rlandyleft a comment19:46
rlandywrt error I got19:46
ssbarnearlandy perfect! probably is one i already know about.19:46
rlandywe"ll find out19:47
ssbarnearlandy the big question is if you managed to make molecule run19:47
rlandydidnt try that yet19:47
rlandywas testing other patch19:47
rlandysince rebase patch failed19:48
*** saneax has joined #oooq19:52
*** rascasoft has quit IRC20:13
*** rnoriega has quit IRC20:13
*** rnoriega has joined #oooq20:14
*** saneax has quit IRC20:24
ssbarnearlandy i updated that change and contains multiple changes, clearly need a new look at it. passed my 3 platform test with molecule.20:33
*** agopi is now known as agopi|brb20:33
*** vinaykns has quit IRC20:34
rlandyyep20:35
*** vinaykns has joined #oooq20:35
rlandyok20:36
rlandytesting again20:36
rlandyssbarnea: 20:37:41 Complete!20:38
rlandy20:37:41   File "<string>", line 120:38
rlandy20:37:41     import pip; print('pip:%s ' % pip.__version__, end='')20:38
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages)20:41
*** agopi|brb has quit IRC20:41
nhicherarxcruz, chkumar|off, weshay_doctor: I have a node on hold on vexxhost, which ssh key should I add on the instance for debugging ?20:50
*** weshay_doctor is now known as weshay20:51
ssbarnearlandy not sure what you are trying to say, that line worked correctly for me.20:51
arxcruznhicher: https://github.com/arxcruz.keys20:51
rlandywhen I included your patch, it failed for me there20:51
nhicherarxcruz: ssh zuul@162.253.55.12520:52
ssbarnearlandy: ok, I know. fixing now.20:53
rlandyok - working on reproducer stuff now - will look again later20:53
nhicherarxcruz: last log are https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/3a527f7/20:54
arxcruznhicher: checking20:54
nhicherarxcruz: this one https://logs.rdoproject.org/66/16566/10/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/3a527f7/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-30_20_34_5020:55
ssbarnearlandy fixed: missing and import from the future...20:56
arxcruznhicher: are you sure it's up and running this box?20:57
arxcruz++ openstack floating ip list -c ID -f value20:58
arxcruzFailed to discover available identity versions when contacting https://10.0.0.5:13000. Attempting to parse version from URL.20:58
arxcruznhicher: ^20:58
nhicherarxcruz: there is only the undercloud node, the others nodes were deleted by te-broker20:59
nhicherarxcruz: nodepool manage only the undercloud20:59
arxcruz:(21:00
arxcruznhicher: need the other nodes...21:00
arxcruznhicher: i'm setting a env21:00
nhicherarxcruz: ok, I didn't know that, I think it should be possible to debug tempest import without overcloud nodes, sorry for that21:01
arxcruznhicher: np, thanks for get the env anyway :)21:02
arxcruznhicher: mine is installing now21:02
nhicherarxcruz: I will delete it if you don't use it, good luck for debugging =)21:02
*** trown is now known as trown|outtypewww21:09
weshaynhicher, lolz21:20
weshaythat's a new one21:20
nhicherweshay: which one ?21:21
weshaydebugging tripleo w/o the overcloud21:21
nhicherweshay: it's only tempest, but I agree =). I don't know how to keep stack created during the run when set node on hold21:23
weshayya.. that is just a node pool thang21:23
weshaywe do need a ovb plugin for nodepool21:23
weshaytoo bad we're all just chasing our tails21:23
nhicherweshay: yes really =)21:23
rlandyssbarnea: we got a bit further22:09
*** agopi|brb has joined #oooq22:33
*** agopi|brb is now known as agopi22:34
*** vinaykns has quit IRC22:37
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-quick-basic, tripleo- (3 more messages)22:41
*** rlandy has quit IRC23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!