Monday, 2018-10-22

*** agopi_ is now known as agopi00:04
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb- (4 more messages)00:34
*** ykarel|away has joined #oooq02:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7 (4 more messages)02:34
*** rnoriega has quit IRC02:36
*** skramaja has joined #oooq03:02
*** saneax has joined #oooq03:08
*** ykarel|away is now known as ykarel03:50
*** udesale has joined #oooq03:55
*** udesale has quit IRC03:55
*** udesale has joined #oooq03:56
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)04:34
*** apetrich has quit IRC05:23
*** apetrich has joined #oooq05:39
*** ratailor has joined #oooq05:50
*** quiquell|off is now known as quiquell06:04
*** gvrangan has joined #oooq06:12
*** jfrancoa has joined #oooq06:33
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)06:34
*** jbadiapa has joined #oooq06:42
*** chkumar|off is now known as chandankumar06:45
*** ccamacho has joined #oooq07:03
*** rascasoft has joined #oooq07:08
*** jtomasek has joined #oooq07:12
*** udesale has quit IRC07:33
*** udesale has joined #oooq07:33
*** tosky has joined #oooq07:42
quiquellssbarnea: Good morning07:45
*** bogdando has joined #oooq07:57
*** amoralej|off is now known as amoralej08:09
*** panda has joined #oooq08:13
ykarelpanda, hi08:22
*** sshnaidm|off is now known as sshnaidm|ruck08:22
ykarelpanda, sshnaidm|ruck can u check https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/multinode-jobs.yaml#L38508:23
pandaykarel: hi08:23
ykareli think this should be moved to singlenode08:23
ykarelundercloud only job08:23
sshnaidm|ruck marios , can you please confirm it's singlenode? ^^08:24
ykarelearlier it used to work because of https://review.rdoproject.org/r/#/c/16876/9/playbooks/legacy/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset050-upgrades-master/run.yaml@10808:24
ykarelsshnaidm|ruck, in rocky it's added correctly https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/undercloud-jobs.yaml#L25-L2608:25
sshnaidm|ruckit's  featureset050 there08:25
sshnaidm|ruckykarel, oh, I see08:25
*** jaganathan has joined #oooq08:26
pandamechanic translation08:26
pandawe trusted the original jobs names08:27
*** holser_ has joined #oooq08:27
mariossshnaidm|ruck: what is the question if undercloud is singlenode?08:27
sshnaidm|ruckmarios, featureset05008:28
mariossshnaidm|ruck: https://github.com/openstack-infra/tripleo-ci/blob/ab0f0ae95d882fef6a5adeeafe207df9f5badf1f/zuul.d/undercloud-jobs.yaml#L47-L4808:29
chandankumarpanda: https://tree.taiga.io/project/tripleo-ci-board/us/168 all tasks populated, feel free to take a look08:29
mariossshnaidm|ruck: that what you mean?08:29
*** chem` is now known as chem08:30
pandassbarnea: ping08:30
sshnaidm|ruckmarios, yeah08:31
ssbarneamorning08:31
chandankumarquiquell: Hello08:32
chandankumarquiquell: can we add a field in ruck/rover dashboard to track failed standalone jobs gated against puppet-%{project}?08:33
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)08:34
pandachandankumar: wow thanks.08:35
pandassbarnea: you available to discuss https://tree.taiga.io/project/tripleo-ci-board/us/196?milestone=203564 ?08:36
ssbarneapanda: sure, i am really interested about the subject, even if i do not know it very well. 5mins to grab a coffee.08:38
quiquellchandankumar: We are not suppose to worry about them I think08:39
quiquellssbarnea, panda: We are suppose to run ansible over python3 at subnodes with the f28 thing ?08:47
pandaquiquell: guess so. ansible is run in undercloud, and undercloud is supposed to have python3 only. As far as I know, ansible run on python3 too08:48
quiquellssbarnea: Are we reviving this https://review.openstack.org/#/c/591540/  ?08:48
quiquellpanda: ack08:49
ssbarneaquiquell: yep, we are supposed to use a pure py3 approach, no py2 anywhere,.... py2 would only if we fail08:49
pandado or do, there's no fail.08:49
quiquellssbarnea: Ok, looks like for fedora28 the only way is to do a pip3 install ansible, there is no python3-ansible stuff, I am testing with it08:51
ssbarneaquiquell: no problem with that, i already used the pip fallback for another package. and if we do try yum fallback to pip we are ok even for the moment when we will have a rpm.08:52
quiquellssbarnea: Cool them I am testing the right thing, thanks !08:53
ssbarneaquiquell: the funny part will be about other python packages that you will need for using ansible.... like hidden dependencies.08:53
quiquellssbarnea: I have found already stuff a tripleo-ha-utils :-/08:53
ssbarneabut the rule is kinda simple: if there is a rpm for it, it has to be installed and pip must be avoided for that package, we use pip only for that without rpm.08:54
quiquellssbarnea: Simple and failr rule08:54
pandammhh python2-ansible is targeted for fedora 2908:56
pandapython308:56
quiquellpanda: yep08:56
ssbarneapanda: i am sure we will have it for "f28"08:56
quiquellssbarnea: They backport those things ?08:56
*** ykarel_ has joined #oooq08:57
ssbarneaquiquell: likely that a backport will happen.08:59
ssbarneaI have on controversion line that I do not like: ln -s /usr/bin/python3 /usr/bin/python08:59
ssbarneai find this one very dangereus and i plan to remove it from that review, even if it will break few things, we will have to fix them without this hack.09:00
*** ykarel has quit IRC09:00
pandassbarnea: bj ?09:00
quiquellssbarnea: reproducer + new workflow + f28 it's working to iterate09:01
quiquell:-)09:01
pandammhh09:01
pandaso we are going to discuss exaclty this now09:02
pandamaybe we don't need to run ansible on python309:02
quiquellpanda: That's another possibility, at least we can give it a try and see how far we get09:02
quiquellpanda: Soon or later we will ahve to use python3 so we will discover stuff09:03
quiquellpanda: The issue is that all the tripleo packages have to be adapted to it09:03
*** ykarel_ is now known as ykarel09:08
pandassbarnea: bj ?09:09
ssbarneapanda https://bluejeans.com/265541792809:09
*** ykarel is now known as ykarel|lunch09:14
sshnaidm|ruckmarios, hi, do you have 5 min to talk about prev ruck/rover cycle?09:24
ssbarneapanda: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2#L22409:28
*** dtantsur|afk is now known as dtantsur09:30
quiquellssbarnea, panda: Can I join ?09:31
mariossshnaidm|ruck: sure man but can it be in like half hour or 1 hour?09:33
mariossshnaidm|ruck: and you can have longer than 5 mins :)09:33
sshnaidm|ruckmarios, sure, ping me as you're available09:38
*** dsneddon has quit IRC09:39
quiquellssbarnea: They are reverting python3 stuff at tq https://review.openstack.org/#/c/612289/09:44
quiquell-/09:44
*** skramaja has quit IRC09:44
pandaquiquell: yes09:51
ssbarneaquiquell: yep give it a go, I will rework the change to avoid it.09:59
pandabogdando: where did https://review.openstack.org/#/c/612289/ failed ?10:00
*** ykarel|lunch is now known as ykarel10:02
rascasoftpanda, who other core can we involve to have this https://review.openstack.org/#/c/612289 merges ASAP?10:04
rascasoft(morning, btw)10:04
pandarascasoft: where does it fail ?10:05
rascasoftpanda, in all the rdophase2 and ospphase0 jobs10:05
rascasoftpanda, do you have an idea how in the world that commit passed all the gates if it's failing consistently everywhere?10:07
mariossshnaidm|ruck: pvt ready when you are10:07
pandarascasoft: because we don't have any automatic test for rdophase210:07
pandasshnaidm|ruck: can you take a look ?10:07
ssbarnearascasoft: we assumed that py3 is *not* installed on centos-7 machines.10:08
rascasoftpanda, ok, but what is described in https://review.openstack.org/#/c/612289 is not related to rdophase2... right?10:08
rascasoftssbarnea, I'm hitting it consistently in RHEL 7.510:09
ssbarnearascasoft: i can create a new change to improve the detection to make it work, but it will take 1-2h do do it and if we are lucky we can have it merged till tomorrow. is ok?10:10
*** rfolco|rucker has joined #oooq10:13
*** rfolco|rucker is now known as rfolco|rover10:13
chandankumarpanda: rfolco|rover I have kept you in cc on the email related porting py2/p3 porting spec file , each dfg should take care of their tempest plugin10:18
sshnaidm|ruckpanda, at what?10:18
rfolco|roverchandankumar, thanks10:18
pandasshnaidm|ruck:  https://review.openstack.org/#/c/61228910:19
sshnaidm|ruckrfolco|rover, hey10:22
rfolco|rovero/ sshnaidm|ruck10:22
sshnaidm|ruckrfolco|rover, wanna be ruck or rover? I just choose randomly10:23
rfolco|roverfine to me10:23
rfolco|roversshnaidm|ruck, can you pls look at https://review.openstack.org/#/c/611717/10:24
rascasoftssbarnea, what's the difference in having the revert merged? I will be quicker, isn't it? Btw please add me as a reviewer for the new review so I can test it10:24
rfolco|roversshnaidm|ruck, attempts to fix the bug where job fails to install ara deps across multiple branches10:25
rfolco|roversshnaidm|ruck, the idea is to disable ara and then add it later by pinning version10:26
rfolco|roverlet me know if this sounds reasonable10:26
rascasoftpanda, sshnaidm|ruck, can we +workflow https://review.openstack.org/#/c/612289 ?10:26
sshnaidm|ruckrfolco|rover, pip and setuptools are not really ara deps, I just update them to prevent various setuptools errors, not sure we need to check its status10:27
sshnaidm|ruckrascasoft, will look10:27
sshnaidm|ruckrfolco|rover, if we can't install setuptools, we have much bigger problem than ara setup..10:27
sshnaidm|ruckrfolco|rover, also you need "when: undercloud_ara|bool" in "Install ara task" too10:28
rfolco|roversshnaidm|ruck, context I have is that ara is giving more trouble than benefit10:28
sshnaidm|ruckrfolco|rover, that's fine, let's make it conditional and disabled by default for now10:28
sshnaidm|ruckrfolco|rover, but no need to check "dependencies"10:29
sshnaidm|ruckrfolco|rover, commented on patch10:30
rfolco|roverstill not quite sure what is the point here ^, ok to see it in the patch, yes,  pls10:30
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)10:34
ssbarneaquiquell https://docs.openstack.org/infra/bindep/ ?11:00
quiquellssbarnea: Let me check11:02
mariossshnaidm|ruck: https://bugs.launchpad.net/tripleo/+bug/179852511:06
openstackLaunchpad bug 1798525 in tripleo "The undercloud-upgrades job is failing during the upgrade with "error was: 'ironic_api_short_bootstrap_node_name' is undefined" " [Undecided,Triaged] - Assigned to Marios Andreou (marios-b)11:06
*** sanjayu_ has joined #oooq11:14
*** saneax has quit IRC11:16
*** sanjayu_ has quit IRC11:23
*** sanjayu_ has joined #oooq11:24
kopecmartinmarios, sorry, i had a lunch break, therefor I've missed the mail and the meeting, anything you need me to do now?11:27
marioskopecmartin: ack11:28
mariosnp11:28
marioswas late notice just added folks incase available11:28
*** udesale has quit IRC11:30
*** sanjayu_ has quit IRC11:34
*** sanjayu_ has joined #oooq11:35
sshnaidm|ruckrfolco|rover, did you see this before? http://paste.openstack.org/show/732579/11:35
sshnaidm|ruckykarel, ^^11:36
*** sanjayu_ has quit IRC11:36
rfolco|roverno11:36
*** sanjayu_ has joined #oooq11:36
ykarelsshnaidm|ruck, nope11:37
*** dtantsur is now known as dtantsur|brb11:37
ykarelsshnaidm|ruck, where are u seeing it11:38
ykarelsshnaidm|ruck, may be mrunge knows if it's failing recently11:38
sshnaidm|ruckykarel, Rocky containers build job11:38
sshnaidm|ruckykarel, yeah, just now failed11:38
ykarelsshnaidm|ruck, there is some update today: http://mirror.centos.org/centos/7/opstools/x86_64/sensu/11:39
ykarelso some new deps are broken11:39
*** sanjayu_ has quit IRC11:39
*** sanjayu_ has joined #oooq11:40
sshnaidm|ruckykarel, rfolco|rover fyi https://bugs.launchpad.net/tripleo/+bug/179921011:41
openstackLaunchpad bug 1799210 in tripleo "Rocky promotion containers build job fails because sensu dependencies" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)11:41
rfolco|roversshnaidm|ruck, ack11:42
ykarelrfolco|rover, sshnaidm|ruck, ack, good to ping mrunge he might have some idea if there is already some progress on the fix11:42
ykareland seems it will affect master as well11:43
ykarelis it affecting there as well11:43
ykarel?11:43
*** amoralej is now known as amoralej|lunch11:44
sshnaidm|ruckykarel, yes, renamed the bug11:44
ykarelsshnaidm|ruck, ack11:44
ykarelsshnaidm|ruck, re. the revert11:45
ykarelansible 2.5.7[1] is not compatible with python 3.4 which is in CentOS7.11:45
ykarelpython3.4 in centos?11:45
sshnaidm|ruckykarel, yeah11:45
ykarelsshnaidm|ruck, what is blocked?11:46
sshnaidm|ruckykarel, I don't understand - why did it use py3 on centos7?11:46
ykarelme too11:46
ykarelwhere it's reproduced can't get11:46
sshnaidm|ruckssbarnea, panda what is background of https://review.openstack.org/#/c/612289/ ?11:46
ykarelmay be rascasoft knows11:46
ykarelPlease let's have this merged soon, we're blocked on it.11:47
sshnaidm|ruckrascasoft, ^^11:47
*** sanjayu_ has quit IRC11:47
*** sanjayu_ has joined #oooq11:47
arxcruzmarios: quiquell hey guys, i want to help with the reproducer, but i'm pretty lost, it's been a while since i touch on CI stuff what can I help?11:51
ssbarneasshnaidm|ruck: i think that rascasoft can explain it better, he found that it break on centos-7 sytems where py3 was installed. I will work today on fixing it and assure it will also work with this. Still testing will take some time so is better to revert.11:54
mariosarxcruz: take one of https://tree.taiga.io/project/tripleo-ci-board/us/53?milestone=203564 tasks11:54
sshnaidm|ruckrfolco|rover, why not to set "when: undercloud_ara|bool" here? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-setup/tasks/main.yml#L3211:55
*** panda is now known as panda|lunch11:55
sshnaidm|ruckssbarnea, why does it run with py3 on centos7? it should use py2 there11:56
rfolco|roversshnaidm|ruck, I was unsure if that playbook was entirely for ara11:56
sshnaidm|ruckrfolco|rover, yeah, seems like it is11:56
rfolco|roversshnaidm|ruck, sounds to be a good switch then11:56
quiquellssbarnea, panda|lunch: Another crazy tool https://github.com/f-prime/Coffer11:57
mariosarxcruz: there is an example in rlandy one i'm looking into the common case now so i can discuss in scrum11:57
*** quiquell is now known as quiquell|lunch11:57
rascasoftsshnaidm|ruck, ykarel, I didn't get problems with centos7 but with RHEL711:57
sshnaidm|ruckrascasoft, you mean it works on centos7, but not on rhel7?11:58
arxcruzmarios: yeah, i saw those tasks, and that's the problem, i don't know where to start, i wasn't working on zuul v3 migration, in fact, i was away from ci stuff for several sprints, only working on tempest11:58
arxcruzalthough I have a pretty idea how zuul v3 works...11:59
rascasoftsshnaidm|ruck, I mean I tested it just on rhel 7 and the problem is the one described in the review11:59
rascasoftand of course, with the review (so the revert) in place, it works11:59
*** ratailor has quit IRC12:00
mariosarxcruz: yah task for now is to take one of those and analyze the things that happen in the jobs deifnition. we will then transefer that to reprododucer as a task. but not yet. right now just folllow the job definition and make some notes12:05
*** sanjayu__ has joined #oooq12:06
quiquell|lunchmarios, arxcruz: Understanding the new tripleo-ci workflow for zuulv3 would be like the first thing12:06
arxcruzquiquell|lunch: indeed, do you have a start point ? or can you spend a few minutes with me to enlight me ?12:07
arxcruz:D12:07
quiquell|lunchmarios: this is more or les tripleo-ci/playbooks/tripleo-ci/run-v3.yaml12:07
quiquell|lunchCan help you while the food is heating :-)==12:08
*** kopecmartin has quit IRC12:08
quiquell|luncharxcruz: tripleo-ci/zuul.d/base.yaml is the place for job definitions12:08
mariosquiquell|lunch: am currently looking into the https://tree.taiga.io/project/tripleo-ci-board/task/19812:09
quiquell|luncharxcruz: There you can find the pre/run/post hooks if you follow the hoocks you end in the final code12:09
mariosquiquell|lunch: i want to update before scrum t - 51 mins12:09
*** sanjayu_ has quit IRC12:09
quiquell|lunchmarios: Yep this is like the preparations, we have to try to emulate as much as possible that12:10
quiquell|lunchmarios: Now we can even read the yaml directly to retrieve the info12:10
quiquell|lunchmarios: Or re-implement zuulÂ's job config mechanism12:10
quiquell|lunchmarios: port of12:10
quiquell|lunchpart of12:10
*** kopecmartin has joined #oooq12:11
panda|lunchmarios: when is you next release ?12:13
*** rnoriega has joined #oooq12:19
sshnaidm|ruckpanda|lunch, do you remember in which dir kolla logs are located in containers build job?12:20
rfolco|roversshnaidm|ruck, I watch the discussion on sensu, not sure what is the fix12:21
panda|lunchsshnaidm|ruck: no, and I don't remember ever lookign at the logs, I remember looking for errors only on the console.12:21
sshnaidm|ruckrfolco|rover, me trying to understand if we built this container before at all\12:21
*** quiquell|lunch is now known as quiquell12:24
sshnaidm|ruckrascasoft, it should not use py3 on centos12:26
mariosrfolco|rover: sshnaidm|ruck fyi https://tree.taiga.io/project/tripleo-ci-board/issue/219?kanban-status=2027733 cc panda|lunch12:26
sshnaidm|ruckrascasoft, can you please submit a lp bug with logs?12:27
mariosrfolco|rover: sshnaidm|ruck i'll bring it up on scrum12:27
rascasoftsshnaidm|ruck, I'm not understanding why you keep repeating centos12:27
*** zul has joined #oooq12:27
rascasoftsshnaidm|ruck, my problem comes while using RHEL12:27
rfolco|rovermarios, ack12:27
sshnaidm|ruckrascasoft, it should be the same in centos, otherwise you tried it and saw it passing on centos712:27
rascasoftsshnaidm|ruck, not following at all12:28
sshnaidm|ruckrascasoft, ok, let's sync after a mtg12:28
ykarelsshnaidm|ruck, centos is released after rhel, so it's possible there is some mismatch12:28
sshnaidm|ruckykarel, yeah, but then I'd like to know which exactly12:28
rascasoftsshnaidm|ruck, ykarel, I know one rule: if before it was working, and after it is not, than the patch have a problem, so we need a revert12:29
rascasoftbut we can discuss for sure about why this is happening12:29
rascasoftin the meantime I'll apply the revert manually12:29
ykarelrascasoft, agree. there seems some other issue not just ansible version12:29
sshnaidm|ruckrascasoft, well, we need to find out why exactly it fails before reverting12:29
mariospanda|lunch: what kind of release are you talking about :)12:30
rascasoftsshnaidm|ruck, fine by me, I'll do the manual patch12:31
sshnaidm|ruckrascasoft, but can we access env or logs for debug?12:31
sshnaidm|ruckrascasoft, just want to see what happens12:31
rascasoftsshnaidm|ruck, sure, let's have the meeting12:32
sshnaidm|ruckcool12:32
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)12:34
*** rlandy has joined #oooq12:35
sshnaidm|ruckmarios, do you know what was(is) te-broker issue in ovb?12:35
mariossshnaidm|ruck: no i don't think anyone knows for sure possibly ovs related which is why weshay_pto posted the debug patches12:42
sshnaidm|ruckarxcruz, chandankumar what is that about? https://trello.com/c/qiKrS3Nc/783-cixlp1797931tripleociproa-tempest-pip-install-fails-on-rocky-minimal-config12:43
chandankumarsshnaidm|ruck: checking12:43
quiquellssbarnea: Can you help me ?12:44
mariossshnaidm|ruck: ah so about those debug patches, they are tracked here  https://bugs.launchpad.net/tripleo/+bug/179752612:44
openstackLaunchpad bug 1797526 in tripleo "Failed to get power state for node FS01/02" [Critical,Triaged]12:44
sshnaidm|ruckmarios, ACK12:45
*** kopecmartin has quit IRC12:47
*** kopecmartin has joined #oooq12:55
*** panda|lunch is now known as panda12:56
pandamariOS: when's your next release ?12:56
quiquellpanda: You need a sprint issue card about jokes :-)12:57
quiquellpanda: Just to read them at the retro12:57
*** gvrangan has quit IRC12:58
arxcruzrfolco|rover: is the scrum now, and on your bj ?13:00
rlandy22:04:13 /usr/bin/python3: No module named virtualenv13:00
rlandyssbarnea: ^^ showing up everywhere in phase 2/ jenkins jobs13:00
ssbarnearlandy: send me alink13:00
ssbarneain fact I know, let me make the CR to fix it, i was in meeting all the mornign.13:01
rlandyssbarnea: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-14-baremetal-hp_dl360_envE-single_nic_vlans/19/console13:01
rlandysame thing in upshift13:01
rlandyand bm testing introspection13:01
*** ykarel_ has joined #oooq13:03
rlandypanda: meeting13:04
*** dtantsur|brb is now known as dtantsur13:04
rlandywe are waiting for your UA leadership13:04
pandarfolco|rover: can you paste the meeting id here ?13:05
rfolco|roverhttps://www.google.com/url?q=https%3A%2F%2Fbluejeans.com%2F587845809713:05
rfolco|roveroops13:05
rfolco|roversh*t13:05
rfolco|roverhttps://bluejeans.com/587845809713:06
pandagood enough13:06
rfolco|roversorry13:06
*** ykarel has quit IRC13:06
*** weshay_pto has quit IRC13:07
rlandyssbarnea: thanks -  killing all my jobs13:09
ssbarnearlandy: i added https://tree.taiga.io/project/tripleo-ci-board/taskboard/sprint-21-175  working on it now. rascasoft already added a revert which is +W, just to unblock jobs.13:10
rfolco|roversshnaidm|ruck, sensu bug is assigned to me, are you or mrunge or ykarel_ looking at that bug ?13:13
ykarel_rfolco|rover, mrunge is fixing it, u can ask him to add details to the bug13:13
sshnaidm|ruckrfolco|rover, just need to check that it started to work13:13
ykarel_sshnaidm|ruck, that should not work until packages are in mirror.centos.org13:14
ykarel_and that should be by 9:00 UTC if those are tagged correctly13:14
ykarel_tomorrow 9:00 UTC13:14
sshnaidm|ruckrfolco|rover, ^^13:14
ykarel_http://mirror.centos.org/centos/7/opstools/x86_64/13:15
rfolco|roverykarel_, sshnaidm|ruck thanks13:23
*** ykarel_ is now known as ykarel13:23
sshnaidm|ruckrlandy, ping me please after mtg, wrt https://review.openstack.org/#/c/610078/313:24
bogdandopanda: I think as ramishara replied https://github.com/openstack/tripleo-quickstart/blob/f2f6dcb209df33d063382f89a4a5c7620a3c91ac/quickstart.sh#L4113:31
rlandymarios:hi - so the bm jobs shows that on 0/18 we reproduce the failure - and the job gets to tempest on 10/19 when the revert goes through13:42
rlandyso that job picks up the changes correctly13:42
pandabogdando: I meant to ask in which environments it was failing, but I understand now what is the problem13:42
rlandychecking which templates we are picking up13:42
rlandysshnaidm|ruck; ping re: https://review.openstack.org/#/c/610078/313:43
mariosrlandy: thanks once you're done digging (don't spend all day on this... !) would be great if you added some brief note so i can pickup my morning?13:43
sshnaidm|ruckrlandy, yeah, I don't understand - what is it for? where is htis file used?13:43
rlandysshnaidm|ruck: that impacts the reproducer13:43
rlandyjust for testing13:44
sshnaidm|ruckrlandy, is it used in reproducer?13:44
rlandysshnaidm|ruck: honestly I think your -2 is justifed13:44
rlandythe bmc node, yeah13:44
rlandysshnaidm|ruck: we are trying to unify those workflows in this sprint13:44
rlandybut yes, that is what is used in the reproducer13:44
rlandysshnaidm|ruck: but you are right - I don;t actualy think we should merge it13:45
rlandyit was part of an experiment to get to the bottom of all these ovb failures in rdocloud13:45
rlandynot a solution imho13:45
rlandywe discussed changing the code on the te-broker instance itself13:45
rlandyto try this diff in ci13:45
rlandyderek seemed to think it may help13:46
sshnaidm|ruckrlandy, I'm not sure we even use it in reproducer..13:46
sshnaidm|ruckrlandy, in ovb-manage-stack bmc_flavor is medium by default, do we add this file anywhere?13:46
rlandy    -e @$WORKSPACE/tripleo-quickstart-extras/config/environments/rdocloud.yml \13:46
rlandyexample:13:47
rlandyhttps://logs.rdoproject.org/74/611674/8/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/7da02d8/logs/reproducer-quickstart.sh13:47
rlandywe do override it13:47
rlandysshnaidm|ruck: my objection is that it won;t prove anything13:48
*** ratailor has joined #oooq13:48
sshnaidm|ruckrlandy, I see... well, then maybe we need to merge it to match CI13:48
sshnaidm|ruckrlandy, if it's only for reproducer, I don't know too what it should to prove.. but better have it same as in CI I think13:48
rlandysshnaidm|ruck: we are in process of working on all that13:49
rlandysshnaidm|ruck: it will all be addressed this sprint ...13:49
sshnaidm|ruckrlandy, ok, then won't block it13:49
rlandysshnaidm|ruck: no - keep blocking it13:49
rlandysshnaidm|ruck: I will talk with weshay tomorrow about it13:50
sshnaidm|ruckrlandy, then can you please block it with your comment?13:50
rlandyyeah13:50
sshnaidm|ruckthanks13:50
*** amoralej|lunch is now known as amoralej13:57
*** vinaykns has joined #oooq13:58
*** vinaykns has quit IRC13:58
rfolco|roversshnaidm|ruck, what should I attack first? gate timeouts investigation, baremetal introspection debugging, ... ?14:00
*** vinaykns has joined #oooq14:01
*** agopi has quit IRC14:03
rlandysshnaidm|ruck: in other news  ... [undercloud-setup : Install python-pip] is falling over on rhel14:05
*** ykarel is now known as ykarel|afk14:13
sshnaidm|ruckrlandy, are there logs from this failure?14:19
sshnaidm|ruckrfolco|rover, I think in this order you mentioned14:19
rfolco|roversshnaidm|ruck, ok, let me know if anything urgent14:20
sshnaidm|ruckrfolco|rover, sure14:20
rlandysshnaidm|ruck: yeah - sorry - got distracted on #tripleo ... https://paste.fedoraproject.org/paste/aBw3ZCQeWGiBuPfA4Hhciw <- some background14:21
rlandysshnaidm|ruck: basically rhel minimal does ship with python-pip14:22
rlandy17:36:05 task path: /home/rhos-ci/jenkins/workspace/upshift-rhos-14-ci-rhos-ovb-featureset001/usr/local/share/ansible/roles/undercloud-setup/tasks/ansible_cfg.yml:514:22
rlandy17:36:05 Friday 19 October 2018  17:37:10 +0000 (0:00:00.374)       0:08:26.996 ********14:22
rlandy17:36:09 fatal: [undercloud]: FAILED! => {"changed": false, "msg": "No package matching 'python-pip' found available, installed or updated", "rc": 126, "results": ["No package matching 'python-pip' found available, installed or updated"]}14:22
sshnaidm|ruckrlandy, yeah, but what it the failure??14:22
sshnaidm|ruckrlandy, ok, python-pip is not in rhel repos I suppose14:22
sshnaidm|ruckrlandy, but it should be in dlrn repos, right?14:22
rlandysshnaidm|ruck: I tried to install python pip because of ara ...14:23
rlandyhttps://thirdparty.logs.rdoproject.org/jenkins-upshift-rhos-14-ci-rhos-ovb-featureset001-177/console.txt.gz#_2018-10-18_23_46_45_80714:23
sshnaidm|ruckrlandy, the ara task is not supposed to install pip, we do it somewhere before this in CI, ara task just updates the pip14:24
sshnaidm|ruckrlandy, ok, so we don't have pip installed there, but we do have it in upstream CI14:24
rlandysshnaidm|ruck, https://github.com/openstack/tripleo-quickstart-extras/blob/14243ab99825cf240d3a2d836062200e57654208/roles/undercloud-setup/tasks/ansible_cfg.yml#L1014:26
pandakopecmartin: do you have time to chat ?14:26
rlandywhy do we pip install pi14:26
rlandypip14:26
kopecmartinpanda, sure, what's up?14:26
pandakopecmartin: in bj ?14:26
rlandysshnaidm|ruck: on rhel it's not there to begin with14:26
rlandyno python-pip, no pip14:26
kopecmartinpanda, sure, your or mine?14:27
pandakopecmartin: 349250866914:27
sshnaidm|ruckrlandy, it's not install, it's update of pip to latest version14:27
rlandyand the pip install w/o virtualenv is what is being objected to14:27
pandakopecmartin: mine14:27
pandakopecmartin: KABOOM14:27
panda(mine)14:27
rlandysshnaidm|ruck: ok - so for rhel we need to install this somewhere14:27
rlandyor do this work in a virtualenv14:27
sshnaidm|ruckrlandy, in upstream CI pip is preinstalled in the image14:27
rlandywell not everywhere14:27
rlandysshnaidm|ruck: and when I mentioned it to rhel-del14:28
rlandythey said well don't pip install w/o venv14:28
rlandyso ...14:28
sshnaidm|ruckrlandy, actually for ara I think it's better to use virtualenv for it..14:28
rlandysshnaidm|ruck: why no venv?14:28
sshnaidm|ruckrlandy, I agree, better to use venv14:29
rlandysshnaidm|ruck: ok - I was going to put in change for it when the python3 problem hit14:29
rlandyanyeays - blocker for rhel atm14:29
*** ykarel_ has joined #oooq14:29
rlandyunless I disable the ara install on rhel14:30
sshnaidm|ruckrlandy, rfolco|rover has a patch to disable it for now, should be fine https://review.openstack.org/#/c/611717/14:30
rlandysshnaidm|ruck: cool - thanks14:31
rlandycan work with that for now14:31
* rlandy just trying to qualify upshift14:31
rlandynot going well :(14:31
sshnaidm|ruckrlandy, yeah, we need to rewrite it with venv14:31
sshnaidm|ruckbut for ansible venv module you need a pip.. arghh14:31
rlandysshnaidm|ruck: I was kind of having that fight on friday with rhel del14:32
rlandywrt not having pip on base image14:32
rlandyI didn't win14:32
*** ykarel|afk has quit IRC14:32
rlandypython-virtualenv should be on rhel base though14:32
sshnaidm|ruckrlandy, it's one of two biggest mysteries in the world - if there are aliens and why the hell no pip in rhel and centos by default!14:33
sshnaidm|ruckand the same chance to get answers for them..14:33
rfolco|roverof course the alien one will be answered first14:33
rlandysshnaidm|ruck: lol - I think I'll tackle the aliens question because I lost the fight on pip14:34
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb- (4 more messages)14:34
sshnaidm|ruckrlandy, don't forget to create a user story14:34
rlandyon the aliens?14:34
sshnaidm|ruckrlandy, sure14:34
rlandyor using a venv?14:34
rlandyyeah - panda likes jokes14:35
sshnaidm|ruckrlandy, no, venv is not so interesting14:35
* rlandy goes back to reviews14:35
sshnaidm|ruckrlandy, is there virtualenv package in default rhel repos?\14:40
rlandysshnaidm|ruck: according to apevec, yes14:41
sshnaidm|ruckrlandy, ok14:41
rlandywe are talking about rhel minimal14:41
rlandywhat is used in ci14:41
rlandywhen they say minimal, they mean it14:41
pandarlandy: do you have time to discuss https://tree.taiga.io/project/tripleo-ci-board/us/226?no-milestone=1 ?14:42
rlandypanda: sure14:43
rlandyit's my new mission14:43
rlandypanda: updated with more info ... https://tree.taiga.io/project/tripleo-ci-board/us/226?no-milestone=114:47
rlandysshnaidm|ruck:^^ done14:49
sshnaidm|ruckrlandy++14:50
hubbot1sshnaidm|ruck: rlandy's karma is now 3314:50
arxcruzpanda: ping14:57
*** quiquell is now known as quiquell|off14:57
pandaarxcruz: ping14:58
pandaarxcruz: oh, sorry wrong ICPM code14:58
arxcruzpanda: want to chat? i've create the tasks/userstories/epics14:58
pandaarxcruz: that was fast14:58
pandaarxcruz: my bj ?14:59
arxcruzpanda: sure, what is your bj ?14:59
pandaarxcrux: 349250866915:00
arxcruzpanda: logged15:01
*** ratailor has quit IRC15:07
arxcruzpanda: https://etherpad.openstack.org/p/openstack-ansible-tempest15:11
sshnaidm|ruckarxcruz, chandankumar can you please look what is it about? https://trello.com/c/qiKrS3Nc/783-cixlp1797931tripleociproa-tempest-pip-install-fails-on-rocky-minimal-config15:34
sshnaidm|ruckarxcruz, chandankumar it's in CI escalation board15:34
sshnaidm|ruckand I have no idea what is it15:34
arxcruzsshnaidm|ruck: checking15:35
rlandymarios: we don;t capture /usr/share in the logs - which is where the tht templates would be :(15:42
mariosrlandy: yeah i've thought before we should add that to /extra15:42
rlandyhmmm ...15:42
rlandymaybe it is ... checking15:42
*** jfrancoa has quit IRC15:43
mariosrlandy: yah maybe it is somewhere there15:43
rlandyseems like a big missing piece if we don't15:43
rlandythat is the key here15:43
rlandyare the new templates used in the change job15:43
rlandymarios: ha - we have it in bm https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-master-baremetal-dell_fc430_envB-single_nic_vlans-8/undercloud/usr/share/openstack-tripleo-heat-templates/15:48
rlandywhich is not useful15:48
rlandywe know it registers there15:48
mariosrlandy: /me off ttyl have a good one15:53
rlandymarios: have a good night15:54
panda/dev/ttyl15:57
*** sanjayu_ has joined #oooq16:06
*** sanjayu__ has quit IRC16:08
*** vinaykns has quit IRC16:14
ykarel_sshnaidm|ruck, is this ovb issue known:- https://logs.rdoproject.org/17/612417/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/a63be0f/job-output.txt.gz16:15
chandankumarpanda: hello still around?16:16
ykarel_sshnaidm|ruck, in testenv-broker log i see: ERROR (CommandError): No server with a name or ID of '83380d36-f510-43c2-9545-230d009b3eea' exists.16:16
ykarel_rfolco|rover, ^^16:17
sshnaidm|ruckykarel_, there are so a lot of them16:18
sshnaidm|ruckykarel_, and everything is because of network failures of rdo cloud16:18
chandankumarykarel_: sshnaidm|ruck can we move to tqe vm to aws to avoid network failure and use cloud creds to do the operation from there?16:19
sshnaidm|ruckykarel_, not sure it even makes sense to collect all these symptoms when we know what is the problem..\16:19
chandankumaror something stable infra16:19
sshnaidm|ruckchandankumar, move what?16:20
chandankumarsshnaidm|ruck: I mean tqe broker16:20
sshnaidm|ruckchandankumar, the problem is not with te broker16:20
sshnaidm|ruckchandankumar, the problem is with connectivity on rdo cloud, and we still have all vms run there16:21
chandankumarsshnaidm|ruck: sorry I misread the problem16:21
ykarel_sshnaidm|ruck, ack but can't get how the Error resembles to network issue:  No server with a name or ID of '83380d36-f510-43c2-9545-230d009b3eea' exists.16:22
sshnaidm|ruckchandankumar, when network is ok everything works like a charm..16:22
ssbarneapanda: check https://seashells.io/v/NqnEmjAm -- i think I managed to make the detection to work16:22
pandassbarnea: where did you get the r8 machine ?16:25
ssbarneapanda: intranet.16:25
*** ykarel__ has joined #oooq16:25
*** sshnaidm|ruck is now known as sshnaidm|afk16:27
*** ykarel_ has quit IRC16:28
ssbarneapanda here is the code: https://github.com/ssbarnea/me/blob/master/distros.yml -- is bash but inside ansible for testing purposes.16:29
ssbarneaas I wanted to detect them the same way as ansible did.16:29
ssbarneai guess that the size of that script explains why we may want to avoid keeping that code in more than one place16:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7 (4 more messages)16:34
*** holser_ has quit IRC16:40
*** kopecmartin is now known as kopecmartin|off16:41
*** ykarel__ is now known as ykarel|away16:42
*** bogdando has quit IRC16:44
*** ykarel|away has quit IRC16:50
*** dtantsur is now known as dtantsur|afk17:06
rfolco|roverrlandy, I am looking for a recent change where we started reusing venvs... am I crazy about it ?17:15
rlandyrfolco|rover: not sure what you mean17:16
rlandyreusing venv?17:16
rfolco|roverrlandy, well, I don't recall but I saw a recent change where current venv was reused.... the reason i am looking for this is that venv python binary is changed while it is running17:18
rfolco|roverhttps://logs.rdoproject.org/58/609858/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/22b7d84/job-output.txt.gz#_2018-10-22_13_54_48_11531417:18
rfolco|roverso an i/o error raises17:18
rlandyah17:20
rlandyhttps://review.openstack.org/#/c/604757/6/roles/standalone/tasks/main.yml17:21
rlandyrfolco|rover: ^^ this?17:22
rfolco|roverrlandy, hmm I am not sure how this correlates to what I am facing17:23
rfolco|roverrlandy, thanks anyway for trying :)17:24
*** sanjayu_ has quit IRC17:25
*** irclogbot_0 has joined #oooq17:25
*** sanjayu__ has joined #oooq17:25
*** dsneddon has joined #oooq17:25
*** irclogbot_0 has quit IRC17:31
*** agopi has joined #oooq17:32
*** irclogbot_0 has joined #oooq17:34
*** dsneddon has quit IRC17:38
*** dsneddon has joined #oooq17:39
*** ccamacho has quit IRC17:41
*** irclogbot_0 has quit IRC17:56
*** irclogbot_0 has joined #oooq18:06
*** amoralej is now known as amoralej|off18:09
ssbarnearfolco|rover rlandy : i am not sure if this is related to venv reusal but I got a funny error locally: https://gist.github.com/ssbarnea/a5edb1d9ad16d6f8a8380616b8be4a1018:12
rfolco|roverhmm yeah looks similar18:13
rfolco|roverI am trying to understand what is going on here18:14
ssbarneaand another funny story is that I see numpy being installed when I try "quickstart.sh -b" ... didn't know we went into bigdata.18:14
rfolco|roverlol18:15
rfolco|rovermachine learning is mocked everywhere18:15
*** agopi has quit IRC18:15
*** apetrich has quit IRC18:17
*** apetrich has joined #oooq18:30
ssbarneabrowbeat has numpy listed as dependency and I really don't like the fact that  "-b" does get numpy. any idea on how to avoid it?18:30
ssbarneanumpy is the godzilla of python packages and i would prefer not to get it installed, if possible.18:31
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024, tripleo-ci-centos-7 (4 more messages)18:34
rfolco|roveragreed, we should add numpy only for browbeat jobs ssbarnea18:44
rfolco|roverrlandy, would you w+ this one? it got stuck on gate for some reason last sprint... rebased, got zuul +1 again, two +2...18:46
rfolco|roverhttps://review.openstack.org/#/c/602425/18:46
*** kopecmartin|off has quit IRC19:03
ssbarneait seems that ansible not longer supports py34 but official py3 on centos-7 is py34, not another one.... which is a problem.19:04
rfolco|roverthats the problem in living in the edge and trying to stay stable at the same time19:08
ssbarnearfolco|rover : there are two edges the deprecation ones and the bleading edge ones... i think that ansible team made a mistake to drop py34 in this case19:10
ssbarneai wonder if I can tell tox to avoid using py3419:12
*** irclogbot_0 has quit IRC19:14
*** irclogbot_0 has joined #oooq19:15
*** vinaykns has joined #oooq19:18
*** rlandy has quit IRC19:35
*** rlandy has joined #oooq19:43
*** agopi has joined #oooq19:48
*** agopi_ has joined #oooq19:50
*** agopi has quit IRC19:53
rlandymarios: you still around or is that just your computer active?20:07
*** sanjayu__ has quit IRC20:09
*** saneax has joined #oooq20:09
*** agopi_ is now known as agopi20:12
*** saneax has quit IRC20:24
*** sanjayu_ has joined #oooq20:24
*** irclogbot_0 has quit IRC20:27
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb- (3 more messages)20:34
*** sanjayu_ has quit IRC20:38
*** saneax has joined #oooq20:39
*** saneax has quit IRC20:39
*** saneax has joined #oooq20:40
*** agopi has quit IRC20:52
*** kopecmartin has joined #oooq21:07
*** hubbot1 has quit IRC21:30
*** dmellado has quit IRC21:32
*** gouthamr has quit IRC21:32
*** dsneddon_ has joined #oooq22:41
*** vinaykns has quit IRC22:46
*** agopi has joined #oooq22:47
*** tosky has quit IRC23:09
*** saneax has quit IRC23:16
*** saneax has joined #oooq23:17
*** ssbarnea has quit IRC23:40

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!