Thursday, 2018-09-06

rlandyexpecting pike, queens  to fail - need to backport the change00:00
rlandyback later to backport change and run again00:04
*** rlandy is now known as rlandy|bbl00:04
*** tosky has quit IRC00:07
*** agopi|brb is now known as agopi00:56
*** thrash is now known as thrash|g0ne01:53
*** rlandy|bbl is now known as rlandy01:54
rlandyugh - rebase time01:54
weshayrlandy, is that osp-1302:14
weshayOH02:15
weshayremoval of lagacy playbook02:15
rlandywhich part?03:12
rlandymultinode is legacy work finally moving along03:12
*** rlandy has quit IRC03:13
*** skramaja has joined #oooq03:27
*** gkadam has joined #oooq03:31
*** jaganathan has quit IRC04:14
*** ykarel|away has joined #oooq04:17
*** ratailor has joined #oooq05:15
*** ykarel|away is now known as ykarel05:18
*** ykarel has quit IRC05:26
*** udesale has joined #oooq05:38
*** gkadam_ has joined #oooq05:41
*** gkadam has quit IRC05:43
*** ykarel has joined #oooq05:45
*** gkadam__ has joined #oooq05:45
*** quiquell has joined #oooq05:46
quiquellchandankumar: good morning05:47
*** gkadam_ has quit IRC05:47
chandankumarquiquell: Good morning :-)05:51
quiquellWhat  role do you want to do the first half ?05:56
quiquellruck or rover ?05:56
chandankumarquiquell: ruck will be ok, in 2nd week we can switch the roles05:58
quiquellchandankumar: ack05:58
*** quiquell is now known as quiquell|rover05:58
*** chandankumar is now known as chkumar|ruck05:58
*** marios|rover has quit IRC06:02
*** kopecmartin has joined #oooq06:08
*** udesale has quit IRC06:29
*** jtomasek has joined #oooq06:33
*** apetrich has quit IRC06:48
*** sshnaidm|off is now known as sshnaidm07:06
*** quiquell|rover has quit IRC07:06
*** quiquell has joined #oooq07:07
*** quiquell has quit IRC07:07
*** quiquell has joined #oooq07:12
*** quiquell has quit IRC07:13
*** quiquell has joined #oooq07:13
*** quiquell has quit IRC07:13
*** ccamacho has joined #oooq07:14
*** quiquell has joined #oooq07:14
*** quiquell has quit IRC07:14
*** quiquell has joined #oooq07:15
*** quiquell is now known as quiquell|rover07:16
*** jfrancoa has joined #oooq07:26
*** udesale has joined #oooq07:27
*** ssbarnea|bkp2 has quit IRC07:27
quiquell|roversshnaidm: Doing small changes at dashbaord-ci to reduce the notification noisce to RDO admins07:29
quiquell|roversshnaidm: workflowing them07:29
*** ssbarnea|bkp has joined #oooq07:40
*** tosky has joined #oooq07:41
*** holser_ has joined #oooq07:44
quiquell|roverssbarnea|bkp, marios: The last ruck rover etherpad is updated ?07:59
mariosquiquell|rover: i was updating this one https://review.rdoproject.org/etherpad/p/ruckrover-sprint1808:00
mariosquiquell|rover: really the main issue last night was the bug on the queens scn 4 job at https://bugs.launchpad.net/tripleo/+bug/179085708:00
openstackLaunchpad bug 1790857 in tripleo "Queens tripleo-ci-centos-7-scenario004-multinode-oooq-container failing tempest manila_tempest_tests.tests.api.test_shares.SharesCephFSTest " [Critical,In progress] - Assigned to chandan kumar (chkumar246)08:00
quiquell|roverLet me know when you finish to create the new one whit the content08:00
quiquell|rovermarios: ack08:01
mariosquiquell|rover: all of the ones on the etherpad are crossed out/means fix committed so they should be good08:01
mariosquiquell|rover: i think we will do a call with weshay later (he emailed us ) to do a status/handover08:02
quiquell|rovermarios: So everything is at UPDATES FROM RUCK/ROVER SPRINT 18 ?08:02
quiquell|roverok08:02
mariosquiquell|rover: yeah but sec i have something which might help for now08:02
quiquell|rovermarios: ok, just let me know when you finish08:02
mariosquiquell|rover: where is the 'right place' i mean its teh first time i did ruck/rover08:03
mariosquiquell|rover: should i add notes to there ? https://review.rdoproject.org/etherpad/p/ruckrover-sprint1808:03
quiquell|rovermarios: Let's do one thing, just add the an update from today here https://review.rdoproject.org/etherpad/p/ruckrover-sprint1908:04
quiquell|roverSo chkumar|ruck and I have the latest08:05
quiquell|roverHumm we don't have a noop change for rocky ?08:05
quiquell|roversshnaidm: ^ ?08:05
mariosquiquell|rover: does it help :)08:11
marios?08:11
mariosquiquell|rover: its just the main things i was looking at yesterday, as the 'main ongoing issues'08:11
mariosquiquell|rover: only one of them is an actual bug in progress08:11
mariosquiquell|rover: ssbarnea|bkp ^^^ please update too for any other stuff you were looking at (I think fs35 you were poking at yesterday too)08:12
quiquell|rovermarios: Depends have different is main from rocky right now08:12
quiquell|rovermarios: Thanks so much08:12
mariosquiquell|rover: i have no idea how to parse that sentense08:12
marios11:12 < quiquell|rover> marios: Depends have different is main from rocky right now08:12
marioswhat did he mean by this08:12
quiquell|roverI mean having a noop change to test rocky, only make sense if rocky is very different from master08:13
mariosquiquell|rover: ack08:13
*** ratailor_ has joined #oooq08:14
*** ratailor has quit IRC08:17
quiquell|rovermarios, ssbarnea|bkp: Any suggestion for the dashboard-ci now that you have finish your task ?08:24
ssbarnea|bkpquiquell|rover marios : i updated the ruckrover document yesterday, crossing all fixe dbugs.08:24
quiquell|roverssbarnea|bkp: Can you look at the sprint19 to see if nothing is missing ? https://review.rdoproject.org/etherpad/p/ruckrover-sprint1908:25
ssbarnea|bkpquiquell|rover : yep, its size grew too much and it puts Safari to its knees (100%cpu), better to start splitting it multiple boards, interlinked.08:25
mariosquiquell|rover: i really like it so far but was only using the check/gate fails list and the % for now08:25
*** dtantsur|afk is now known as dtantsur08:25
mariosquiquell|rover: super useful anyway08:26
quiquell|rovermarios, ssbarnea|bkp: Any suggestion or idea can be added here https://trello.com/c/dyvGWJps/890-dashboards-maintenance?menu=filter&filter=label:Sprint%2017%20CI08:26
ssbarnea|bkpalso found few links that go nowhere, which can be confusing. but overall the dashboard is AWESOME !08:26
quiquell|rovercool08:26
ssbarnea|bkpi hope it will replace the other ones, by implementing their features. I like the idea of using grafana instead of creating a new product from zero.08:27
ssbarnea|bkpit should be easier to upgrade and maintain, i guess.08:27
quiquell|roverssbarnea|bkp: If you have some free brain cycle, would nice to point it to the broken links in the trello card08:28
sshnaidmquiquell|rover, we don't, need to make rocky gate check08:28
quiquell|roversshnaidm: Ok, going too, ok ?08:28
sshnaidmquiquell|rover, sure08:28
quiquell|roversshnaidm: Similar to queens is good enoough ? or we need something new ?08:28
ssbarnea|bkpi don't right now, but be sure I will try in the future. having too many dashboards is not good.08:28
quiquell|roverssbarnea|bkp: ack, thanks.08:29
sshnaidmquiquell|rover, is https://review.openstack.org/#/c/589448/ ready ot merge?08:29
quiquell|roversshnaidm: Has just rebased it, let's wait for CI, but it shoudl be, also the WIP with Ronelle's rebase is working fine too08:29
sshnaidmquiquell|rover, ok.. let's try to merge it today08:30
quiquell|roversshnaidm: Agree08:30
quiquell|rovermarios: ^ we will need your core help to merge the bash to ansible commit08:31
quiquell|rovermarios: To deliver it today08:31
*** sshnaidm is now known as sshnaidm|afk08:31
*** udesale has quit IRC08:47
*** udesale has joined #oooq08:47
*** udesale has quit IRC08:48
*** udesale has joined #oooq08:48
*** d0ugal has joined #oooq08:49
chkumar|ruckmarios: quiquell|rover https://review.rdoproject.org/etherpad/p/ruckrover-sprint19 l18 tempest issue is fixed na?08:49
quiquell|roversshnaidm|afk: hubbot has unregistered user, he is living at #openstack-unregistered now... :-/08:50
marioschkumar|ruck: well queens @ tripleo-ci-centos-7-scenario004-multinode-oooq-container https://review.openstack.org/#/c/567224/100 still failed last time did it all merge already?08:51
chkumar|ruckmarios: the fix got merged but not yet picked up this review08:52
mariosmanila_tempest_tests.tests.api.test_shares.SharesCephFSTest  chkumar|ruck yeah08:52
marioschkumar|ruck: ^^ http://logs.openstack.org/24/567224/100/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/f225f42/job-output.txt.gz#_2018-09-06_02_14_08_18516008:52
chkumar|ruckmarios: i checked it from here http://logs.openstack.org/24/567224/100/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/f225f42/logs/undercloud/var/log/extra/rpm-list.txt.gz08:52
marioschkumar|ruck: so thats the one was watching anyway well find out on next run08:52
marioschkumar|ruck: ack08:53
chkumar|ruckquiquell|rover: please pass me the ruckrover dashboard link or better we can add it to channel topic08:55
quiquell|roverdashboard-ci.tripleo.org08:56
quiquell|roverchkumar|ruck: Cannot change the topic08:57
quiquell|roverI am not an operator08:57
quiquell|roversshnaidm|afk: I have no rights to add rocky noop https://review.openstack.org/#/c/600387/ to hubbot09:00
*** udesale has quit IRC09:16
*** sshnaidm|afk is now known as sshnaidm09:16
sshnaidmquiquell|rover, I'll look at it09:16
quiquell|roversshnaidm: Have create the patch for rocky, it's a matter of add it at hubbot, it's living at #openstack-unregistered09:17
quiquell|roversshnaidm: Also register the hubbot user, and add the logic to login09:17
quiquell|roversshnaidm: But most important part is to register the Change-Id for rocky I9c40e97a9fbc850962605b2a46420e6c645d5c2309:18
sshnaidmquiquell|rover, yeah, I tried to bring it back yesterday. I've already registered it, but didn't know they want real mail to confirm it :)09:18
sshnaidmnow need to re-register it to different mail..09:18
quiquell|roversshnaidm: We can use the ruck/rover gmail09:18
sshnaidmquiquell|rover, what is it?09:18
quiquell|rovertripleo.ci.ruck.rover@gmail.com09:18
quiquell|rovercreated for the dashboard-ci09:18
sshnaidmquiquell|rover, cool, and pass?09:19
quiquell|roversshnaidm: It's at the passwords doc09:19
sshnaidmquiquell|rover, ack09:19
quiquell|roversshnaidm: What's the best place to change gating_repo priority ?09:37
quiquell|roverbuild-test-packages ?09:37
sshnaidmquiquell|rover, change to what?09:38
quiquell|roversshnaidm: To have more priority than delorean-current09:38
sshnaidmquiquell|rover, it has priority 1, the highest09:39
sshnaidmquiquell|rover, what it the problem?09:39
quiquell|roversshnaidm: https://bugs.launchpad.net/tripleo/+bug/175693609:39
openstackLaunchpad bug 1756936 in tripleo "rdo jobs ignore depends-on tripleo-common" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)09:39
quiquell|roverTrying the Alex approach09:40
sshnaidmquiquell|rover, it always had the highest priority, the problem was there that we just didn't apply our build patches in CI09:41
quiquell|roversshnaidm: Yesterday I read that you fixed it ?09:42
sshnaidmquiquell|rover, because zuul automatically rebases patches in CI we should always have the higher version than in delorean-current09:42
quiquell|roverso maybe this is closed ?09:42
sshnaidmquiquell|rover, yeah, now I think we can close it..09:42
quiquell|roverDo you have the review with the fix ?09:42
quiquell|roverWant to try it09:42
sshnaidmquiquell|rover, it's shortest patch in the world :) https://review.openstack.org/#/c/598089/09:43
quiquell|roveryu kidding me09:44
quiquell|rover:-)09:44
sshnaidmyeah, little "s" blocked CI for 2 weeks09:44
quiquell|roverclazzzy09:44
quiquell|roverAnd the u ?09:45
quiquell|roverDon't remeber what the u mod do09:45
quiquell|roversshnaidm: So we can close it '09:46
quiquell|rover?09:46
sshnaidmquiquell|rover, I'd ask @bandini if he has still the problem before09:47
quiquell|roversshnaidm: ack09:47
quiquell|roverbandini: Do you still suffer from https://bugs.launchpad.net/tripleo/+bug/175693609:47
openstackLaunchpad bug 1756936 in tripleo "rdo jobs ignore depends-on tripleo-common" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)09:47
ssbarnea|bkpsshnaidm thanks for merging https://review.rdoproject.org/r/#/c/15914/ -- seeing ANSI on console really improves my workday :)09:48
sshnaidmssbarnea|bkp, glad to help09:48
*** panda has quit IRC09:48
bandiniquiquell|rover: I don't (but I have not done a lot of work where I would trigger it lately). feel free to close it and I can reopen if I stumble on it09:49
quiquell|roverbandini: ack, will do09:49
bandinithanks09:49
ssbarnea|bkpi wonder if we do have a way to perform a gate-check on jenkins jobs because they do not vote and is too easy to miss checking them, not to mention that is hard to spot an empty CR in the logs.09:50
*** d0ugal has quit IRC09:51
*** udesale has joined #oooq09:53
ssbarnea|bkpcan someone explain me the purpose of the "Verify Sphinx build" task?09:57
*** quiquell|rover is now known as quiquell|rover|a09:57
*** quiquell|rover|a is now known as quique|rover|afk09:57
*** d0ugal has joined #oooq10:02
cgoncalvessshnaidm, hi. could you please revisit https://review.openstack.org/#/c/596428 ?10:11
sshnaidmcgoncalves, will look10:12
sshnaidmcgoncalves, I need to know what is running there too, can you please add full command and output? it should be in _quickstart.log in directory you run from10:13
sshnaidmcgoncalves, because I didn't meet such problem ever10:13
cgoncalvessshnaidm, yeah, weshay was also puzzled when he connected to my box. I even reinstalled the OS just to be sure10:14
*** d0ugal has quit IRC10:14
sshnaidmcgoncalves, ok, waiting for you output then10:17
sshnaidmcgoncalves, I also don't understand this fix - ansible "copy" copies from local host to remote, shell "cp" copies files on the remote host, it's not equivalent at all10:18
*** quique|rover|afk is now known as quiquell|rover10:19
ssbarnea|bkpmarios i found interesting bug: getting user id fails just after succesfull creation of user, see https://ci.centos.org/job/tripleo-quickstart-extras-gate-master-delorean-quick-basic/341/consoleFull "https://ci.centos.org/job/tripleo-quickstart-extras-gate-master-delorean-quick-basic/341/consoleFull"10:22
*** d0ugal has joined #oooq10:23
ssbarnea|bkpfiles as https://bugs.launchpad.net/tripleo/+bug/179105310:26
openstackLaunchpad bug 1791053 in tripleo "delorean: Get the non-root user UI fails after user is created" [Undecided,New]10:26
mariosssbarnea|bkp: ack i'm looking at failing scen 1 on rocky gate check fails like http://logs.openstack.org/16/599216/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/3d16b52/logs/subnode-2/var/log/extra/errors.txt.gz#_2018-09-06_03_27_18_02810:26
marios2018-09-06 03:27:18.028 ERROR /var/log/containers/ceilometer/agent-notification.log: 28 ERROR ceilometer.publisher.gnocchi [-] fail to retrieve filtered project : NoUniqueMatch: ClientException10:26
ssbarnea|bkpsure, i am digging on mine for now.10:27
*** ratailor__ has joined #oooq10:27
*** ratailor_ has quit IRC10:30
quiquell|rovermarios, ssbarnea|bkp: Do you know why we don't have master promotions ?10:37
mariosquiquell|rover: no should have clear on promotion blockers afaik10:38
mariosquiquell|rover: or its another issue10:38
quiquell|roverno promotion to phase110:38
quiquell|roverat master10:38
mariosquiquell|rover: pointer jobs/logs?10:38
quiquell|roverJust looking at grafan's promotions10:39
mariosquiquell|rover: k am looking at some other issue now (rocky scen 1 fail)10:39
mariosquiquell|rover: i need to find at least one more of it to file a bug :)10:40
quiquell|rovermarios: No worries let me dig deeper10:40
*** ratailor has joined #oooq10:41
*** ratailor__ has quit IRC10:41
*** ratailor has quit IRC10:42
chkumar|ruckweshay: myoung sprint-19 tempest topics https://etherpad.openstack.org/p/tripleo-tempest-sprint19 can we do the planning meeting a little early today as I need to catch the flight10:53
quiquell|roverykarel: Do you know when is the next run for https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ ?10:59
ykarelquiquell|rover, when centos7-master/current-tripleo updates11:00
quiquell|roverykarel: humm, but we have change promoter to bypass one job... so there is no change at tripleo-current11:01
quiquell|roverwe have to wait for a tripleo-current promotion ?11:01
*** kopecmartin has quit IRC11:01
*** kopecmartin has joined #oooq11:02
ykarelquiquell|rover, do u want to test some fix?11:03
quiquell|roverykarel: We commented phase1 promote job at master.ini in promoter11:03
ykarelokk, so?11:04
quiquell|roverykarel: To be able to have a phase1 promotion that fix a problem with virtualbmc11:04
quiquell|roverbtw tripleo-current is not promoteion because of this https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/26c2305/job-output.txt.gz#_2018-08-29_19_00_26_03867211:04
quiquell|rovermarios, ssbarnea|bkp: Did you find something similar in previous sprint ?11:04
ykarelquiquell|rover, but still don't get why u wanted to rerun rdo_trunk-promote-master-current-tripleo11:06
ssbarnea|bkpquiquell|rover first time when I see this error,11:06
ykarelquiquell|rover, as phase 1 would have been promoted by skipping that job in master.ini11:07
*** ccamacho has quit IRC11:07
quiquell|roverykarel: https://bugs.launchpad.net/tripleo/+bug/179096911:07
openstackLaunchpad bug 1790969 in tripleo "Could not find the requested service virtualbmc: host" [Critical,Fix released] - Assigned to Quique Llorente (quiquell)11:07
quiquell|roverykarel: The fix https://review.rdoproject.org/r/#/c/16158/1 is not running11:08
mariosquiquell|rover: new for me too11:08
quiquell|roverykarel: Because there have not being any change at tripleo-current even with phase1 behind tripleo-current is not re-running ?11:08
ykarelthen probably issue in promoter, looking http://38.145.34.55/master.log11:10
ykarelquiquell|rover, is https://review.rdoproject.org/r/#/c/16158/1 applied on promoter server?11:10
ykarelbecause as per logs it's not11:11
quiquell|roverykarel: ack, let me check11:11
mariosquiquell|rover: ssbarnea|bkp i rechecked the rocky gate check11:14
mariosthat was so satisfying to write11:14
quiquell|rovermarios: sshnaidm is activating it in hubbot so we have it automatically11:14
mariosquiquell|rover: sshnaidm i didn't know that11:14
mariosquiquell|rover: thanks11:15
quiquell|rovermarios: Next I will add it to dashboard-ci11:15
mariosquiquell|rover: ssbarnea|bkp anyway the point is i was looking into the failing rocky-scen1 fail: http://logs.openstack.org/16/599216/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/3d16b52/11:15
mariosquiquell|rover: ssbarnea|bkp and went digging but in the end i see the fails were all from yesterday. Today it passes at master gate check https://review.openstack.org/#/c/560445/133 has green tripleo-ci-centos-7-scenario001-multinode-oooq-containerSUCCESS in 3h 03m 58s BUT patchset 132 had failed tripleo-ci-centos-7-scenario001-multinode-oooq-container TIMED_OUT in 3h 20m 36s11:16
quiquell|roversshnaidm: Someone has change rocky.ini config manually at the promoter11:16
mariosquiquell|rover: hence why i wanted to recheck @ rocky to see if that deals with it11:16
quiquell|roversshnaidm: that's why is not updating :-(11:16
sshnaidmquiquell|rover, heh...11:16
sshnaidmquiquell|rover, can you see in logs who did it?11:16
quiquell|rovercentos user :-(11:17
sshnaidmquiquell|rover, ok.. need to block centos user from ssh..11:17
mariosquiquell|rover: sshnaidm why is the upstream check displaying noop11:17
mariosdoes it mean no fails? or probably something wrong?11:18
marioscos our stats are going down :/11:18
mariossshnaidm: quiquell|rover so it was working earlier11:20
quiquell|roversshnaidm: agree11:20
mariosi mean dashboard11:20
quiquell|rovermarios: context11:20
*** abishop has joined #oooq11:21
mariosquiquell|rover: so is ther something wrong with http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 now sshnaidm11:21
*** abishop has quit IRC11:21
marios?11:21
marioslooks like it to me?11:21
mariosi already tried refreshing11:21
quiquell|rovermarios: What's happening ?11:22
marioslike it only shows failing gates from 2018-09-0511:22
mariosnothing from today11:22
mariosand nothing in faling check jobs11:22
quiquell|rovermarios: shoot you are right no upstream gate jobs ...11:24
mariosquiquell|rover: is it working ok for you?11:24
quiquell|rovermarios: let me check11:24
mariosquiquell|rover: it was working earlier11:24
mariossshnaidm: ^11:24
quiquell|roversshnaidm: It make sense that promoter not updating if there is some manual changes ?11:26
quiquell|roversshnaidm: The git command fails, but maybe it's a good thing11:26
sshnaidmquiquell|rover, need to check if it has "force" in git11:26
*** kopecmartin has quit IRC11:27
sshnaidmquiquell|rover, possibly not..11:27
*** ratailor has joined #oooq11:27
quiquell|rovercd ~/ci-config; git pull >/dev/null11:27
quiquell|roversshnaidm: Let's do it right, going to put the manual changes into git11:28
sshnaidmquiquell|rover, yes, it won't update with local changes11:29
quiquell|roversshnaidm: that's good or bad ?11:29
quiquell|roverfrom a workflow perspective11:29
ykarelsshnaidm, can we take https://review.openstack.org/#/c/579406/, seems it's affecting phase 1 as well11:29
sshnaidmquiquell|rover, let's just change it to git reset --hard origin/master && git pull11:29
quiquell|roversshnaidm: chicken/egg thing, we need to do it first manually11:30
sshnaidmquiquell|rover, yeah, np11:30
sshnaidmykarel, do you know why rdo jobs failed there?11:31
ykarelsshnaidm, u mean phase111:31
sshnaidmykarel, seems like infra again..11:31
sshnaidmykarel, no, I mean in the patch11:31
ykarelhmm those are infra ones11:32
sshnaidmmarios, quiquell|rover seems like data is stuck in grafana, opened it just now11:32
ykarelrecenlty ops fixed some infra issues related to metadata, so jobs should be good now11:32
sshnaidmykarel, great11:32
quiquell|roversshnaidm: https://review.rdoproject.org/r/#/c/16166/11:35
*** hubbot1 has joined #oooq11:36
sshnaidmhubbot1, thanks god you're here11:36
hubbot1sshnaidm: Error: "thanks" is not a valid command.11:36
sshnaidm%gatestatus11:37
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7 (3 more messages)11:37
quiquell|rovergood11:37
quiquell|roverour old friend11:37
quiquell|roversshnaidm++11:37
hubbot1quiquell|rover: sshnaidm's karma is now 511:37
sshnaidmhubbot1++11:37
hubbot1sshnaidm: hubbot1's karma is now 111:37
quiquell|rover%config11:38
hubbot1quiquell|rover: (config <name> [<value>]) -- If <value> is given, sets the value of <name> to <value>. Otherwise, returns the current value of <name>. You may omit the leading "supybot." in the name if you so choose.11:38
quiquell|rover%config plugins.GateStatus.changeIDs11:38
hubbot1quiquell|rover: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab If12c8fe9bd0bea98a4842f279399285344f2224611:38
quiquell|rover%config plugins.GateStatus.changeIDs I9c40e97a9fbc850962605b2a46420e6c645d5c23 I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b1 I214272a6f25feb75496e44eb0a16269c6ee4cfe2 I4c5bdf00ce8cf7eabf669b248b99cb8443e82fab If12c8fe9bd0bea98a4842f279399285344f2224611:39
hubbot1quiquell|rover: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.11:39
quiquell|roversshnaidm: ^11:39
quiquell|roversshnaidm: /etc/telegraf directory has disappear from rrcockpit infra server11:49
quiquell|rover:-/11:49
mariosquiquell|rover: woo looks like rocky gate check scen 1 is green on recheck11:49
mariosssbarnea|bkp: ^11:49
*** chem has quit IRC11:49
quiquell|rovermarios: what's the CI Seals ?11:50
quiquell|roverchkumar|ruck: Do we have to assis the CI Seals Daily ?11:50
mariosquiquell|rover: am watching zuul but undercloud-containers fails11:50
mariosquiquell|rover: added comment at https://review.openstack.org/#/c/599216/2 zuul didn't finish yet11:50
mariosquiquell|rover: oh11:50
mariosquiquell|rover: its a call 4 times/week11:50
mariosquiquell|rover: they use https://trello.com/b/j4IcIomh/production-chain-escalation11:51
mariosquiquell|rover: yes ruck/rover both attend and might have to talk about things. there should be nothing in there that you dn't already know about (i mean because there was a lp bug for it) or it is d/stream only so not your issue11:52
quiquell|rovermarios, ssbarnea|bkp: Anything we have to say there ?11:52
mariosquiquell|rover: see previous message ^T11:52
quiquell|rovermarios: ack11:52
mariosquiquell|rover: might want to check out the cards which have LP on them already11:52
mariosquiquell|rover: like open the bug see what status is/what merged didn't etc11:52
mariosquiquell|rover: so basically you should know about anything there that has 'LP' on it !11:53
quiquell|roverDamn forgot to check CIX :-/11:53
marios:)11:53
sshnaidmquiquell|rover, I'll configure them in hubbot.conf11:53
mariosquiquell|rover: rhwy fo critical first11:53
mariosthey go11:53
*** rfolco has joined #oooq11:53
sshnaidmquiquell|rover, but telegraf runs from container, does it need /etc/telegraf? how could it disappear..11:53
quiquell|roversshnaidm: Ahh silly me11:54
chkumar|ruckquiquell|rover: yes11:54
weshayquiquell|rover, chkumar|ruck hey guys :)11:54
sshnaidmquiquell|rover, what is change ID for rocky?11:55
weshayis rdo cloud still down?11:55
quiquell|roversshnaidm: I0cbf9ffb8552411e4dd891c38702ff8d1f6db5b111:55
quiquell|roverweshay: is all green https://semaphore.cloud.upshift.engineering.redhat.com/11:55
sshnaidmquiquell|rover, it's queens11:56
quiquell|roversshnaidm: wait11:56
sshnaidmquiquell|rover, found - If12c8fe9bd0bea98a4842f279399285344f2224611:56
quiquell|roverI9c40e97a9fbc850962605b2a46420e6c645d5c2311:56
quiquell|roverAhh ssbarnea|bkp you already had it :-)11:57
sshnaidmquiquell|rover, yeah, and it's already in config..11:57
quiquell|roverAhh cool cool11:58
quiquell|roverWill update the dashboard-ci later11:58
sshnaidmquiquell|rover, no, it's not good11:59
sshnaidmssbarnea|bkp, you created patch with same change id11:59
weshaysshnaidm, did you get ara working?12:00
sshnaidmweshay, not yet12:00
sshnaidmquiquell|rover, I'll use your patch12:00
quiquell|roversshnaidm: ack12:00
*** trown|outtypewww is now known as trown12:00
mariossshnaidm: quiquell|rover so do we know what the problem with dashboard is ?12:01
sshnaidmquiquell|rover, and we need now to remove both ssbarnea|bkp patch and create different pike patch :/12:01
quiquell|roversshnaidm: Then let just create a new one for pike and use ssbarnea|bkp for rocky12:01
sshnaidmquiquell|rover, we can't, it has same change ID as myoungs patch for pike12:02
sshnaidmquiquell|rover, hubot doesn't work with patches with same change ID :(12:02
quiquell|roverThat's why we do a new one for pike12:02
quiquell|roverwith new change ID12:02
quiquell|roverthat's all12:02
quiquell|roverthat will work ?12:03
sshnaidmquiquell|rover, yea, but need also new patch for rocky12:03
quiquell|roverok take rocky12:03
sshnaidm%gatestatus12:03
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7 (3 more messages)12:03
ssbarnea|bkpsshnaidm not sure what was wrong, I only did a cherry pick of the gate-check CR to another branch where I didn't see it.12:06
quiquell|rovermarios: Any news on https://trello.com/c/MbcQXQ5g/717-cixlp1789416tripleociproa-stable-queens-no-image-with-the-name-bm-deploy-kernel-found-make-sure-you-have-uploaded-boot-images ?12:07
sshnaidmssbarnea|bkp, yeah, it's what we shouldn't do12:08
mariosquiquell|rover: not from me ssbarnea|bkp filed that one12:09
mariosquiquell|rover: but lemme check (reviews merged i think?)12:09
mariosquiquell|rover: i don't think we are seeing this issue anymore though? afaics12:10
mariosquiquell|rover: one of the 2 reviews merged12:10
mariosssbarnea|bkp: thisis about https://bugs.launchpad.net/tripleo/+bug/178941612:10
openstackLaunchpad bug 1789416 in tripleo "stable/queens: No image with the name 'bm-deploy-kernel' found - make sure you have uploaded boot images."" [Critical,Fix released] - Assigned to wes hayutin (weshayutin)12:10
quiquell|rovermarios: ack12:10
quiquell|roverFor next seal session12:11
mariosquiquell|rover: cool so you survived then ;) ?12:11
quiquell|roverDon't know12:12
*** ratailor_ has joined #oooq12:13
*** ratailor__ has joined #oooq12:14
*** ratailor has quit IRC12:15
mariosquiquell|rover: if you get stuck just plead the 5th12:17
mariosalways works in the movies12:17
*** ratailor_ has quit IRC12:17
quiquell|rovermarios: and this ? https://trello.com/c/4UoWap3v/725-cixlp1790199tripleociproa-fs035-openstack-overcloud-deploy-returns-error-code-without-any-signs-or-error12:22
quiquell|roverssbarnea|bkp: ^ ?12:23
*** kopecmartin has joined #oooq12:23
ssbarnea|bkpi need to recheck the fs035 one, to see what happens now.12:24
*** chem has joined #oooq12:24
quiquell|roverssbarnea|bkp: So in progress12:24
ssbarnea|bkpin fact close it, unable to reproduce.12:24
*** thrash|g0ne is now known as thrash12:26
quiquell|roverfewww...12:30
quiquell|roverWas not prepare for this new seal stuff12:30
quiquell|roverthanks ssbarnea|bkp12:30
ssbarnea|bkpi keep seeing the ""The conditional check 'update_containers and update_containers_append_tag is undefined' failed.", can someone give me a hint regarding why? https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/6902/console12:31
ssbarnea|bkpis this caused by lack of promotion on masteR?12:32
chkumar|ruckquiquell|rover: I will be there from monday12:34
quiquell|roverchkumar|ruck: no worries12:34
mariosssbarnea|bkp: sounds like https://launchpad.net/bugs/1789881 ?12:35
openstackLaunchpad bug 1789881 in tripleo "tripleo-quickstart fails to generate /home/stack/containers-default-parameters.yaml" [Medium,In progress] - Assigned to yatin (yatinkarel)12:35
mariosssbarnea|bkp: oh no12:35
mariosssbarnea|bkp: not the same actually12:35
quiquell|roveris different12:36
weshaychkumar|ruck, you have a link to the log that has this error?12:36
weshayhttps://bugs.launchpad.net/tripleo/+bug/1790857/comments/112:36
openstackLaunchpad bug 1790857 in tripleo "Queens tripleo-ci-centos-7-scenario004-multinode-oooq-container failing tempest manila_tempest_tests.tests.api.test_shares.SharesCephFSTest " [Critical,In progress] - Assigned to chandan kumar (chkumar246)12:36
weshaychkumar|ruck, also.. when opening promotion blockers, please only use logs from promotion / periodic jobs.12:36
chkumar|ruckweshay: http://logs.openstack.org/24/567224/99/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5fdb110/logs/subnode-2/var/log/containers/manila/manila-scheduler.log.txt.gz#_2018-09-05_02_04_56_11512:36
weshayyou have a check job listed in that bug.. which raises doubts whether it's a promotion job or simply a bad review12:37
weshaychkumar|ruck, thanks12:37
chkumar|ruckweshay: the above job was broken from last week i think, when it made to tempest, it failed on mistral12:37
*** ccamacho has joined #oooq12:38
*** ccamacho has quit IRC12:38
*** rlandy has joined #oooq12:38
weshaychkumar|ruck, k k12:38
weshaychkumar|ruck, I don't see the same error https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/subnode-2/var/log/containers/manila/manila-scheduler.log.txt.gz12:38
*** ccamacho has joined #oooq12:38
weshaychkumar|ruck, https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/tempest.html.gz12:38
holser_marios Can you reply on my PM?12:39
chkumar|ruckweshay: the above job failed in queens noop job12:39
weshaychkumar|ruck, ah.. I see why you would use that :)12:40
weshaychkumar|ruck, still .. when creating a promotion blocker, it's best to use a periodic job12:40
weshaychkumar|ruck, if that is still confusing.. let me know12:40
chkumar|ruckweshay: yes , noted12:40
*** ykarel is now known as ykarel|away12:41
weshaychkumar|ruck, thanks for running down the fix12:41
mariosholser_: sorry missed it12:41
weshaychkumar|ruck, well done12:41
weshaychkumar|ruck++12:41
hubbot1weshay: chkumar|ruck's karma is now 112:41
rlandymarios: hello ...12:44
rlandymarios: so it looks like the parameters were passed12:45
holser_marios np12:45
holser_just wanted to sort out before Retro12:45
weshaychkumar|ruck, is arx pto?12:45
quiquell|roverweshay: Now we have the CIX thing everyday ?12:45
weshaykopecmartin, we're going to have tempest planning w/ ci this morning12:45
weshayquiquell|rover, I'm doing my best to kill that12:46
weshayquiquell|rover, not having much luck12:46
quiquell|roverweshay: ack, no problem12:46
chkumar|ruckweshay: nope,12:46
rlandymarios: if you look at the results from http://logs.openstack.org/22/596422/35/check/tripleo-ci-centos-7-scenario009-multinode-oooq/80c5d95/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-06_04_34_5112:46
*** ykarel|away has quit IRC12:47
chkumar|ruckweshay: I think he is not in this channel12:47
kopecmartinweshay, sorry, i can't attend, wifi in bus is not fast enough for a call :/12:47
rlandymarios: I agree that the top level params are not needed though12:48
*** arxcruz has joined #oooq12:48
rlandylooking at how bridge_name was passed12:48
arxcruzweshay: hi12:50
weshayarxcruz, there you are12:50
weshay:)12:50
arxcruzi left the channel, no idea why... :/12:50
weshaychkumar|ruck, https://etherpad.openstack.org/p/tripleo-ci-squad-meeting line 6312:58
weshayarxcruz, sleep walking again?12:58
arxcruzno, for some reason i left the channel automatically, dunno why... i'm still in the other channels :/12:59
weshaymyoung, please make me the owner on all the mtgs you have.. or shared owner13:01
*** ratailor__ has quit IRC13:01
weshayrlandy, mtg13:02
weshaychkumar|ruck, ^13:02
weshayhttps://bluejeans.com/7050859455/13:02
toskyI lost track... what is it still missing to fix the scenario004 (aka manila) tempest test in queens?13:08
toskyI thought that the relevant packages were backported13:08
myoungweshay: ack, will do13:10
*** skramaja has quit IRC13:10
myoungweshay: they are (nearly) all on the RHOS calendar, so all changes should be made to the events on that calendar and the individuals' calendar events are then automagically updated.13:10
tbarrontosky: chkumar|ruck - yeah, I thought updating the manila-tempest-plugin to include vkmc's default share type creation would do the trick13:13
vkmctbarron, chkumar|ruck, tosky, I can work on the backport13:14
tbarrontosky: chkumar|ruck vkmc - but we're still seeing emails re. scenario004 and the manila-api logs on subnode2 still show tracebacks for lack of default share type13:14
quiquell|rovermarios, sshnaidm: Looking at dashboard-ci issue... running it locally works fine13:15
toskyvkmc: but the version with the fix is already marked to be used in queens13:15
toskyvkmc: so no need to backport anything (and where would you backport it? There were no branches)13:16
weshaytbarron, tosky https://review.rdoproject.org/r/#/c/16148/213:16
toskyweshay: yes, and we still get failures13:16
tbarronvkmc: ^^^13:17
weshayyou have the right package installed?13:17
weshaymaybe it's not through the system13:17
toskyI don't know, it's marked to be used on queens, so the CI should do the magic, right? :)13:17
tbarronweshay: yeah, I'm thinking the job is still running with the old package, where does it say which it uses?13:17
toskyyou can check the packages installed from the logs13:17
* tosky checks13:18
vkmcyeah, in the environment logs13:18
tbarronETOOMANY different log formats in my life13:18
dtrainorI'm using reproducer to verify a bug (http://logs.openstack.org/14/590214/4/check/tripleo-ci-centos-7-undercloud-containers/84d7e84/logs/reproducer-quickstart.sh) and though it appears to deploy successfully and my undercloud gets created in rdocloud, I see no reference to the files mentioned in the change (e.g. validations/library/ssl_certificate.py)13:18
toskytbarron: in logs/undercloud/var/log/extra there is a rpm-list.txt.gz13:19
toskystill the old version apparently13:19
toskyuhm13:19
tbarrontosky: ty, and that makes sense of the data13:19
toskylet's move to #rdo13:19
tbarroneven if it doesn't make sense that the old pkg is being used13:19
*** kopecmartin has quit IRC13:20
*** vinaykns has joined #oooq13:24
mariosrlandy: ack sorry i was afk before the call will check/we can discuss after13:27
*** dtrainor has quit IRC13:28
*** dtrainor has joined #oooq13:31
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike- (3 more messages)13:36
dtrainorif i run reproducer from what's part of a check for a changeset, then i should expect to see all the bits of the changeset in what reproducer deploys, right?13:38
quiquell|roversshnaidm: Found the issue my end_time from gerrit is breaking it... going to revert13:39
sshnaidmquiquell|rover, ok13:39
toskyoh, quiquell|rover and chkumar|ruck - are you the one in charge of the CI issues right now?13:39
quiquell|rovertosky: Damn you discover us :-)13:40
quiquell|roveryes we are ruck-> find and report, rover-> fix broken stuff13:40
toskyquiquell|rover: ah!13:40
toskyquiquell|rover: so, did you read the discussion here ~20 minutes ago and on #rdo ?13:41
quiquell|rovertosky: will do13:41
toskybasically a certain job on stable/queens is failing and it's apparently using the result of the promotion13:41
toskythe questions are a) is it the case (it's using current-passed-ci) and b) what is it missing for a promotion on queens?13:42
quiquell|rovertosky: talking about tripleo-ci-testing -> current-tripleo ?13:43
quiquell|roverto promoter to current-tripleo you have to pass the following kind of jobs at periodic13:43
toskyquiquell|rover: to be honest I lost track about the names of the tagged releases, so I don't know :)13:43
quiquell|rovertosky: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/queens.ini#L16-L2513:43
toskymy only goal is: have manila-tempest-plugin 0.1.0 used for queens jobs13:44
toskythat's it13:44
quiquell|roverif one of those toci_jobtypes does not pass we don't have a promotion13:44
*** chkumar|ruck is now known as chkumar|out13:44
quiquell|rovertosky: you mean periodics ?13:44
toskyquiquell|rover: one of the job that reported failures is http://logs.openstack.org/15/562215/5/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/574dcb1/job-output.txt.gz#_2018-09-06_13_02_07_12131713:45
toskythat does not seem the periodic queue13:45
quiquell|rovertosky: what featureset includes scenario004 ? let's check13:45
quiquell|rovertosky: so this job has this toci_jobtypemultinode-1ctlr-featureset01913:46
quiquell|rovertosky: and this is included in the promotion criteria13:46
quiquell|rovertosky: for sure we have a counterpart job in periodics13:47
quiquell|rovertosky: this is the periodic job failing at queens execution scenario004 https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/213c32d13:47
*** sshnaidm is now known as sshnaidm|mtg13:49
toskyquiquell|rover: last run was one week ago13:49
rlandymarios: pls - just another 5 mins to check back- we got some working results yesterday13:50
quiquell|rovertosky: Could not be let me check13:50
toskyat least from the log timestamp13:50
quiquell|rovertosky: maybe we have a problem in the promoter13:51
rlandydtrainor: yes - you should see your change in the reproducer undercloud env - we are in meeting atm so response may be slow13:51
quiquell|rovertosky: this is the last http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/13:53
toskyquiquell|rover: and it's failing because the package that should have been there is not there13:55
dtrainorrlandy, all good, just wanted to get the dialogue going.13:56
toskyquiquell|rover: uhm, wait a minute, http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/undercloud/var/log/extra/rpm-list.txt.gz says that it used python2-manila-tests-tempest-0.1.0-0.20180905135747.f6bc6fa.el7.noarch13:56
toskyso that's the correct package13:57
quiquell|roverSo it's failing for other reason ?13:57
toskyvkmc: can you please take a look at the logs of that job? It should use the correct manila tempest plugin13:57
myounghttps://trello.com/c/2y2BVXgO/726-ideas-brainstorm-resulting-from-testing-in-sprint-1214:03
*** d0ugal has quit IRC14:05
*** d0ugal has joined #oooq14:10
quiquell|rovertosky: so the version is not the problem ?14:18
quiquell|rovertosky: so the manila tempest plugin version is not the issue ?14:21
toskyquiquell|rover: it seems not, but vkmc is the export14:22
toskyso I leave it to her :)14:22
quiquell|rovertosky: this is tempest talking with ceph ?14:24
mariosquiquell|rover: dashboard looks to be working again now14:26
mariosquiquell|rover: at least shows fails from 6th14:26
toskyquiquell|rover: it's a manila test14:28
quiquell|rovermarios: I revert some "improvement" I did :-/14:28
quiquell|rovertosky: this is the error http://logs.openstack.org/24/567224/99/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5fdb110/logs/subnode-2/var/log/containers/manila/manila-scheduler.log.txt.gz#_2018-09-05_02_04_56_11514:29
mariosquiquell|rover: cool it is now much improved!14:29
tbarronquiquell|rover: tosky vkmc I still see python2-manila-tests-tempest-0.0.1 r/t 0.1.0 in14:35
tbarronhttp://logs.openstack.org/24/567224/99/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5fdb110/logs/undercloud/var/log/extra/rpm-list.txt.gz14:35
quiquell|rovertbarron: we are looking at the periodic job, that fails in the same point14:35
quiquell|rovertbarron: it has the correct one14:35
quiquell|rovertbarron: http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/undercloud/var/log/extra/rpm-list.txt.gz14:35
tbarronquiquell|rover: i just looked from your last link above, looking at this one now14:36
quiquell|rovertbarron: Have talk to soo, the error does not appear at manila manager in periodic, pero the tempest fail is the same14:38
quiquell|rovertbarron: manila-scheduler in periodic is ok http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/subnode-2/var/log/containers/manila/manila-scheduler.log.txt.gz14:39
tbarronquiquell|rover: tosky vkmc yeah, the14:40
tbarronquiquell|rover: tosky vkmc yeah the 'no default share type' error is gone14:40
tbarronquiquell|rover: vkmc tosky so there's a new reason for failure to investigate ...14:40
quiquell|rovertbarron: yep... http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens/f70676c/logs/undercloud/home/zuul/tempest/tempest.html.gz14:40
quiquell|rovertbarron: where can we look ?14:41
ssbarnea|bkpquiquell|rover : shoudn't this have a an alert or promotion-blocker label to it? https://bugs.launchpad.net/tripleo/+bug/178610614:42
openstackLaunchpad bug 1786106 in tripleo "containers-prep fails w/ update_containers is undefined due to lack of default" [High,In progress] - Assigned to Sorin Sbarnea (ssbarnea)14:42
quiquell|roverssbarnea|bkp: promotion-blocker at least14:44
ssbarnea|bkpthanks.14:44
quiquell|roverssbarnea|bkp: Do you want me to continue with it ?14:45
tbarronquiquell|rover: vkmc we'll lok at the other logs under /var/log/containers/manila on subnod2 for clues14:45
ssbarnea|bkpquiquell|rover it would be great if you do, i got few other changes hitting it.14:45
quiquell|roverssbarnea|bkp: Will re-assign it then14:45
*** d0ugal has quit IRC14:50
*** d0ugal has joined #oooq14:53
rlandymarios: hi - pls ping me when you are ready  to go over the THT review - thanks14:55
*** ratailor has joined #oooq14:57
quiquell|roverchkumar|out, weshay: Looks like RDO jobs are broken here https://logs.rdoproject.org/16/563616/12/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-queens/b9cbc5f/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz14:58
weshayhrm14:58
weshayk14:58
* weshay looks14:58
quiquell|roverI think this is new14:58
* quiquell|rover opens a lp14:59
weshayquiquell|rover, hrm.. openvswitch 2.014:59
weshayquiquell|rover, I think I have a bug on it14:59
weshayno?14:59
weshayquiquell|rover, just check, I opened two last night.. I think one may have this trace14:59
chkumar|outquiquell|rover: weshay fs20 queens periodic job is also broken https://review.openstack.org/59935515:00
chkumar|outi need some help in jinja templates15:00
weshayarxcruz, btw.. txt me when you land in denver15:00
weshaychkumar|out, k.. how can I help15:00
chkumar|outweshay: https://review.openstack.org/#/c/599355/4/roles/validate-tempest/defaults/main.yml@7015:01
chkumar|outcheck ssbarnea|bkp last comment15:01
quiquell|roverchkumar|out, weshay: openned https://bugs.launchpad.net/tripleo/+bug/1791115 you can mark it at duplicate if  necessary, have to leave now15:02
openstackLaunchpad bug 1791115 in tripleo "[ERROR] /opt/stack/new/devstack/functions-common:Detected fatal at RDO jobs" [Critical,New] - Assigned to Quique Llorente (quiquell)15:02
weshaychkumar|out, thanks for sending me that email... I just missed it :)15:02
quiquell|roverjfrancoa: ^15:02
*** quiquell|rover has quit IRC15:02
jfrancoathanks quiquell15:02
weshaychkumar|out, http://jinja.quantprogramming.com/15:03
arxcruzweshay: sunday 6pm15:03
*** hamzy has quit IRC15:04
weshaychkumar|out, you can't do what you are trying to in jinja15:06
weshay{%- else -%}network-feature-enabled.api_extensions: dvr15:07
weshayis not ok15:07
arxcruzchkumar|out: why not create two variables ?15:07
weshaytempest_conf_removal: >-15:09
weshay{% if release is in ['queens'] -%}15:09
weshay{{ tempest_conf_removal_queens }}15:09
weshay{%- else -%}15:09
weshaynetwork-feature-enabled.api_extensions: dvr15:09
weshay{%- endif -%}15:09
weshaychkumar|out, arxcruz that works according to the linter15:09
weshayarxcruz, chkumar|out sent you both an email15:11
arxcruzweshay: cool, what's this tool ?15:12
weshayarxcruz, http://jinja.quantprogramming.com/15:12
arxcruzawesome15:12
weshaynice right15:12
weshayarxcruz, there is src for it.. we could host it too15:13
weshayarxcruz, it goes down from time to time15:13
ssbarnea|bkpquite useful bookmark, thanks.15:14
chkumar|outarxcruz: weshay thanks :-)15:15
weshaychkumar|out, I didn't think defining a var in the jinja if would work15:16
weshayit does there..15:16
weshayhopefully it will work in reality too15:16
weshaychkumar|out, that opens up a lot of potential if it does work15:16
chkumar|outweshay: tempest_whitelist_old tempest_whitelist_new implementation works15:16
chkumar|outweshay: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/general_config/featureset005.yml#n6515:18
*** sshnaidm|mtg is now known as sshnaidm15:20
*** d0ugal has quit IRC15:21
chkumar|outweshay: arxcruz we need discussion on tempest deployer input conf file at ptg https://bugzilla.redhat.com/show_bug.cgi?id=1619247 bugs/RFE started poping up15:24
openstackbugzilla.redhat.com bug 1619247 in python-tripleoclient "Support adding entries based on heat template parameters to tempest-deployer-input.conf" [High,New] - Assigned to rhos-maint15:24
weshaychkumar|out, ah yes15:25
weshaychkumar|out, so...... alex wants to kill that file15:25
arxcruzweshay: tosky already talked with alex about it15:25
chkumar|outweshay: pull tosky in that he will help you15:25
arxcruzwe just didn't have the time to implement15:25
toskyweshay: and again, the idea is that there should be more details, not less15:26
toskymy fault is that I didn't find the time to implement a way to get the data from tripleo and write them down15:26
mariosrlandy: sure gimme 4 mins15:26
toskyweshay: also, last time I spoke with alex he told me that the way I was told originally (mistral workflow) was not the correct one15:27
*** hamzy has joined #oooq15:27
weshaytosky, chkumar|out this is not a CI thing15:28
weshaythis is a project thing15:28
chkumar|outweshay: it needs calloboration from tempest side15:28
toskyweshay: sure, it's a project (tripleo) thing15:28
toskyweshay: but again, its removal is not a CI thing either15:29
weshaychkumar|out, tempest is a consumer of that file right? So the tempest crew could give requirements15:29
weshaytosky, right.. I have no dog in this fight15:29
weshaytosky, chkumar|out I just see Alex wanted it gone15:29
toskyweshay: you could land some resources :D15:29
chkumar|outweshay: this file is consumed by tempest but totally depending on deployment tool15:30
weshaytosky, why is it important to me to lend resources15:30
weshaynot saying no15:30
weshayjust asking15:30
weshaychkumar|out, I know what it is15:30
chkumar|outat the end tests are failing coming back on tempest15:30
toskyweshay: because for the past history I suspect that no one in tripleo "proper" is interested in that part of testing15:30
sshnaidmweshay, rlandy can you review please another fix for reproducer? https://review.openstack.org/#/c/599460/15:30
rlandylooking15:30
toskyweshay: while people here have the interest for doing it properly, in a way that benefit CI greatly15:31
weshaytosky, chkumar|out I think we're all interested in quality tests working well15:31
weshaytosky, I recall these discussions w/ david15:31
rlandysshnaidm: I looked at this before... we discussed pasting results from a reproducer ...15:32
rlandyPatch Set 1: Code-Review+115:32
rlandywill +2 when there are reproducer results to review15:32
weshaytosky, chkumar|out let me ask you this..15:32
*** ssbarnea|bkp has quit IRC15:32
rlandydo you have those results, if not will need to run that to verify15:32
weshayis the storage deployment info you are seeking available in the hiera config? or ansible config15:32
toskyweshay: if tripleo deployed it, it knows about it15:33
weshaytosky, right.. which means it's in hiera15:33
toskyyep15:33
weshaytosky, so why not just parse that15:33
weshaydoc that15:33
toskyyes15:33
toskybut15:33
weshaywhy have something in tripleo common?15:33
toskyI was told to do it with a mistral workflow15:33
toskylast time Alex told me it was not the correct way15:33
weshaytosky, or generalize this...15:33
weshaytake all the heira data.. and dump it to a known location15:34
toskybut you don't want that, you may have more information than needed15:34
toskyextra local settings15:34
weshaytosky, are you at ptg?15:34
toskyyou want someone in the middle, on the tripleo side, that gives you only what you need15:34
toskyyes15:34
*** d0ugal has joined #oooq15:34
toskyI am15:34
sshnaidmrlandy, yep, tested it and this works15:35
weshayk..  chkumar|out I'm inclined to close the bug as written15:35
rlandysshnaidm: pls paste any output and I'll +215:35
weshayand open an RFE15:35
*** udesale has quit IRC15:35
chkumar|outweshay: sure15:35
* tosky needs to leave, bbl15:35
weshaytosky, k15:35
weshaychkumar|out, tosky let's figure this out on monday15:35
hubbot1FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb- (3 more messages)15:36
toskyI will be in the sahara room, in case ping me15:36
*** tosky has quit IRC15:36
sshnaidmrlandy, http://paste.openstack.org/show/729617/15:36
*** ccamacho has quit IRC15:38
rlandymarios: also - I'll need to backport those chnages to pike queens ocata rocky16:03
*** sshnaidm is now known as sshnaidm|off16:10
rlandymarios: this should cover your suggested changes: https://review.openstack.org/#/c/599739/16:11
rlandywill modify again when I get Alex's response16:11
*** ccamacho has joined #oooq16:12
*** verdurin has quit IRC16:13
*** ssbarnea|bkp has joined #oooq16:14
*** verdurin has joined #oooq16:18
*** gouthamr has joined #oooq16:19
*** ccamacho has quit IRC16:22
*** ratailor has quit IRC16:23
*** jfrancoa has quit IRC16:32
myoungrlandy: are you around / free?16:37
*** ykarel|away has joined #oooq16:42
rlandymyoung: ack16:46
*** trown is now known as trown|lunch16:47
dtrainorhey there rlandy, do you have a moment to revisit why my reproducer is not working as expected?16:48
*** dtantsur is now known as dtantsur|afk16:49
*** apetrich has joined #oooq16:55
*** holser_ has quit IRC16:56
*** jaosorior has quit IRC17:00
*** jaosorior has joined #oooq17:01
*** ccamacho has joined #oooq17:05
*** gkadam__ has quit IRC17:09
ssbarnea|bkprevoting after rebase: https://review.openstack.org/#/c/588587/ arxcruz marios17:14
*** vinaykns has quit IRC17:16
*** ChanServ sets mode: -rf 17:18
sshnaidm|offweshay, can you take a look pls: https://review.openstack.org/#/c/59946017:22
*** ykarel|away has quit IRC17:30
rfolcosshnaidm|off, if you around, could you please point me to the log where a standalone job applied testenv/multinode.yml config at https://review.openstack.org/#/c/589448/ ?17:31
rfolcohow to ensure the symlinks work? https://review.openstack.org/#/c/589448/31/toci-quickstart/config/testenv/standalone.yml17:32
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike- (3 more messages)17:36
*** links has joined #oooq17:38
*** links has quit IRC17:38
ssbarnea|bkpIt seems that my UID fix passed https://review.openstack.org/#/c/600407/ -- ready to review17:43
*** saneax has quit IRC17:58
*** vinaykns has joined #oooq17:59
weshayrlandy, k.. back18:02
weshayrlandy, looking at https://bluejeans.com/u/whayutin/18:02
*** links has joined #oooq18:02
rlandyweshay: sec - just putting in a review to address alex's comments -18:02
rlandybe there in a few18:03
*** links has quit IRC18:11
rfolcoweshay, how do I ensure in this standalone run http://logs.openstack.org/48/589448/31/check/tripleo-ci-centos-7-standalone/f42ab97/ that this has been used as extra-vars https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/multinode.yml ?18:13
*** brault has joined #oooq18:23
weshayrfolco, not sure what you mean18:24
rfolcoweshay, the change uses a symlink to the real config multinode.yml for standalone singlenode and multinode jobs18:26
rfolcohttp://logs.openstack.org/48/589448/31/check/tripleo-ci-centos-7-standalone/f42ab97/job-output.txt.gz#_2018-09-06_10_03_48_37824118:26
rfolcohttps://review.openstack.org/#/c/589448/31/toci-quickstart/config/testenv/standalone.yml18:26
*** jaosorior has quit IRC18:27
rfolcopls see my last comment on 58944818:27
*** brault has quit IRC18:27
*** ccamacho has quit IRC18:28
rfolcoapparently jobs are working fine, but the symlink magic doesn't look right to me18:28
weshayrfolco, oci-quickstart/config/testenv/standalone.yml is a symlink to multinode?18:29
rfolcoexactly18:29
rfolcosinglenode too18:29
weshayrfolco, ya.. I suspect the file was not created properly.. I would have to mess w/ git18:30
weshayrfolco, I need to focus on rlandy's patches atm18:30
rfolcoweshay, np18:30
*** trown|lunch is now known as trown18:57
*** jtomasek has quit IRC19:20
*** hamzy has quit IRC19:34
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, (3 more messages)19:36
*** tosky has joined #oooq20:19
*** rfolco has quit IRC21:00
*** rfolco has joined #oooq21:00
*** trown is now known as trown|outtypewww21:12
rlandyweshay: https://github.com/search?q=org%3Aopenstack+%2Fetc%2Fnodepool%2Fprimary_node_private&type=Code21:17
rlandy^^ check of where else /etc/nodepool is referenced21:17
weshayrlandy, ya.. you sent that earlier21:19
rlandyweshay: yeah I did :)21:19
rlandybut it got lost in conversation....21:19
rlandydo we care about nova etc.?21:20
weshayrlandy, help me understand the point you are trying to make21:20
weshaywe don't run nova jobs21:20
rlandykolla-ansible?21:20
rlandyif not, then ignore me21:20
rlandythat is all21:20
weshaynope21:24
weshaywe just use the same tools to build containers atm21:24
rlandyfine21:24
weshayrlandy, :)21:33
rlandyarxcruz: still around?21:33
*** sshnaidm|off has quit IRC21:36
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, (3 more messages)21:36
rlandynvm - kicking again to get logs21:39
myoungweshay, rascasoft: regarding https://trello.com/c/HXhx4FEA/921-fix-rdo-phase-2-executors-storage-cleanup, 5 of the executors are back online.  that virthost is extremely storage overprovisioned and the storage .raw files needed shrinking.21:56
myoung^^ details in card.21:57
myoungI'll sort the remaining 2 executors tomorrow21:57
*** myoung is now known as myoung|off21:57
myoung|offweshay, rascasoft, rlandy: kicked pipelines to look at results tomorrow: rocky, master, 13, 1422:01
*** vinaykns has quit IRC22:01
*** myoung|off has quit IRC22:01
rlandyI have one of the bm machine running rhos-13 right now22:02
weshayrlandy, rfolco https://review.rdoproject.org/r/1617422:17
rlandyweshay: the instances not associated with stacks were nodepool or te-broker instances?22:41
rlandythe ones I saw were nodepool22:41
rlandyand we can't delete those22:41
weshayrlandy, aye22:42
rlandyso there were te-broker?22:43
rlandyugh - that change did not work :(22:43
rlandyThe Parameter (ZuulSubnodeIndex) was not provided.22:44
rlandyit worked before :(22:47
rlandyweshay: ^^ not good22:50
rlandyundercloud deploy hitting this22:50
rlandywas not a problem when we did define it actual file22:50
rlandyopenstack undercloud install22:51
rlandywe don;t pass anything in particular there22:51
*** tosky has quit IRC22:59
*** dsneddon has quit IRC23:14
rlandyweshay: ok - I think I accept the mock solution now.23:18
rlandywe remove all references in out own code and the leave the mock for outside references23:18
weshayrlandy, the mock?23:21
weshayw/ etc/nodepool?23:22
weshayrlandy, based on the results?23:22
* weshay looks23:22
rlandyweshay: still have not gotten tht to work - you can follow the fun on #tripleo23:22
rlandywaiting on any more direction23:22
rlandybut this is complex23:23
weshayrlandy, ok23:25
weshayrlandy, you want to chat on irc or blue23:25
*** thrash is now known as thrash|g0ne23:29
rlandyweshay: ok - let's bj for a few - probably quicker23:29
*** brault has joined #oooq23:29
*** brault has quit IRC23:34
weshayrlandy, ah k23:36
hubbot1FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, (3 more messages)23:36
weshayrlandy, https://bluejeans.com/411356779823:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!