Friday, 2021-02-05

*** jmasud has quit IRC00:14
*** tosky has quit IRC00:27
*** jmasud has joined #oooq00:33
*** jmasud has quit IRC00:49
*** apetrich has quit IRC02:29
*** udesale has joined #oooq03:46
*** ysandeep|away is now known as ysandeep04:55
*** ysandeep is now known as ysandeep|ruck04:55
*** ratailor has joined #oooq04:57
*** ykarel has joined #oooq05:02
*** jmasud has joined #oooq05:02
*** skramaja has joined #oooq05:26
*** ykarel has quit IRC05:50
*** ykarel has joined #oooq05:51
ysandeep|ruckchandankumar, hello I have a tempest query, tempest timed out on this job but i dont see anything in log https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/46a8100/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz06:10
chandankumarysandeep|ruck: actually job got timeout06:17
ysandeep|ruckyes, job timeout while running tempest tests - may be because they were talking long..  its happening a lot https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master06:19
chandankumarat tempest run06:19
chandankumarysandeep|ruck: we can optimize this scenario06:19
chandankumarysandeep|ruck: we might have two jobs for octavia na?06:19
chandankumarone is ovn and another is plain sc10 ?06:20
ysandeep|ruckFor network component only one is running.. periodic-tripleo-ci-centos-8-scenario010-standalone-network-master .. from name looks like its plain sc1006:21
ysandeep|ruckin my opinion running ovn ocatavia scenario instead in network component make more sense06:23
*** jmasud has quit IRC06:23
*** udesale has quit IRC06:27
*** jmasud has joined #oooq06:35
chandankumarysandeep|ruck: yes, make sense06:35
ysandeep|ruckchandankumar, thanks o/ i will add that and may be latter we can remove plain sc10 from network component, but plain sc10 need optimization.. its timing out even in integration line06:37
ykarelysandeep|ruck, i too noticed a thing yesterday: that some jobs running in component pipeline which doesn't seem to have any extra coverage related to that component06:42
ykarellike in octavia component, plane standalone is running06:42
ykarelshould be true for other components too, mainly the ones which are not enabled by default06:43
ykarelnoticed while looking a downstream octavia tempest issue06:43
ysandeep|ruckykarel, true plain standalone don't make sense.. if its not deploying octavia at all06:43
ykarelthere can be more such examples, but i didn't digged much, may be you can note it down in your optimization plannings06:44
ysandeep|ruckykarel, i will do that, Our initial plan is to optimize check/gate jobs this sprint and yes eventually optimizing periodic jobs will be great.06:46
ykarelyeap, it's just about noting it down, it can be prioritized when needed06:47
ykareltoo many unnecessary jobs runs in check/gate06:47
*** jpodivin has joined #oooq06:49
*** udesale has joined #oooq06:56
*** marios has joined #oooq07:02
*** slaweq has joined #oooq07:02
*** udesale has quit IRC07:08
*** udesale has joined #oooq07:09
*** jmasud has quit IRC07:52
*** gchamoul has quit IRC07:56
*** gchamoul has joined #oooq08:02
*** ysandeep|ruck is now known as ysandeep|lunch08:09
*** amoralej has joined #oooq08:12
*** apetrich has joined #oooq08:29
*** udesale has quit IRC08:30
*** ysandeep|lunch is now known as ysandeep08:33
*** ykarel is now known as ykarel|away08:39
*** tosky has joined #oooq08:41
arxcruz|ruckysandeep: good morning08:44
ysandeeparxcruz|ruck, thanks!08:46
*** ykarel|away has quit IRC08:48
*** udesale has joined #oooq08:49
*** jpena|off is now known as jpena08:57
*** ysandeep is now known as ysandeep|rover09:07
arxcruz|ruckysandeep|rover: i see 3 failures on promotion master, scen 10, 039 and 02009:23
arxcruz|rucktwo of them failed on image upload, i think a recheck worth it09:23
*** jfrancoa has quit IRC09:23
arxcruz|rucks/recheck/re-run09:23
ysandeep|roverarxcruz|ruck, https://bugs.launchpad.net/tripleo/+bug/1914585 i am working on them09:23
openstackLaunchpad bug 1914585 in tripleo "Content provider jobs failed after failing to connect to mirrors, Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host" [High,Triaged]09:23
*** jfrancoa has joined #oooq09:23
ysandeep|roversry wrong bug09:23
ysandeep|roverhttps://bugs.launchpad.net/tripleo/+bug/191473509:23
openstackLaunchpad bug 1914735 in tripleo "Master featureset020/fs039 failed while trying to upload image, HttpException: 500: Server Error for url: https://192.168.24.2:13292/v2/images/b898eee8-66c3-4a27-bbad-e6ca19d1f8d8/file, Internal Server Error" [Critical,Triaged] - Assigned to Sandeep Yadav (sandeepyadav93)09:23
ysandeep|rover^^ this is the one09:24
ysandeep|roverarxcruz|ruck, ^^09:24
arxcruz|ruckchandankumar: ysandeep|rover marios https://review.opendev.org/c/openstack/tripleo-quickstart/+/773648 if you may please09:25
ysandeep|roverarxcruz|ruck, basically swift is disabled by default now, but for some featureset that still use nova - we need to explicitily enable swift.. for fs039 we will enable swift, and for fs020 we will disable nova (to be in sync with fsoo1)09:26
arxcruz|ruckysandeep|rover: cool thanks! sorry i'm late today, my daughter did not sleep this night, i'm just the dust here09:26
ysandeep|roverarxcruz|ruck, fs020 didn't ran against https://review.opendev.org/c/openstack/tripleo-quickstart/+/773648 , testproject is currently running right now09:27
ysandeep|roveri added -w till it ran successfully09:28
ysandeep|rovertestproject https://review.rdoproject.org/r/#/c/28446/09:28
ysandeep|roverrerunning, last one failed on tempest..09:28
ysandeep|roverarxcruz|ruck, no worries, i hope your daughter feels better now09:29
arxcruz|ruckysandeep|rover: she's not sick, she just decide she won't sleep lol09:31
ysandeep|rover:)09:32
*** derekh has joined #oooq09:32
ysandeep|roverarxcruz|ruck, appreciate if you can look at sc010 timeout issue.. i will take care of fs020/39 issues09:34
ysandeep|roverfyi.. victoria(no log)/ussuri also failed on TrafficOperationsScenarioTest https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/9773373/logs/undercloud/var/log/tempest/stestr_results.html.gz and neutron/octaiva team member had some comments on our tempest-skiplist patch09:38
*** zbr is now known as zbr|pto09:54
*** dtantsur|afk is now known as dtantsur10:01
arxcruz|ruckysandeep|rover: basically, they complaing that the issue is related to ovn and not the other tests, i skip it only the job10:20
arxcruz|ruckright now the best thing we can do is skip this test10:20
arxcruz|ruckysandeep|rover: i'll add the standalone-ussuri job on the skiplist as well10:20
arxcruz|ruckand talk with the person responsible to ovn to check it10:20
ysandeep|roverarxcruz|ruck, victoria failed on tempest too, but logs don't have details https://logserver.rdoproject.org/49/31849/1/check/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-network-victoria/812015e/job-output.txt10:22
ysandeep|roverlogs missing here, but maybe same issue.. we may testproject it to confirm10:23
ysandeep|roverarxcruz|ruck, the fix which brian mentioned is stuck in network component, because periodic-tripleo-ci-centos-8-scenario010-standalone-network-master timeouts.. looks like same issue as we are facing in integration line10:24
ysandeep|roverarxcruz|ruck, also I am of opinion that we should run sc10-ovn in network component instead of plain sc10.. i have posted a change https://review.rdoproject.org/r/#/c/31849/ .. awaiting testproject run10:26
*** Tengu has quit IRC10:30
arxcruz|ruckysandeep|rover: /periodic-tripleo-ci-centos-8-scenario010-standalone-master has nothing much to do, it time out, not because of tempest, was the run too slow, i'll create a testproject to rerun it10:38
ysandeep|roverarxcruz|ruck, true no failures looks like caused by slowness, but frequency is high https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master (network component version of same job)last 4 run timed out.. chandankumar suggested in morning that we may need to optimize this scenario... But how that still a question.. may be we reduce number of tests we run here..10:42
arxcruz|ruckysandeep|rover: no idea, the highest test takes 6 minutes10:43
arxcruz|rucknot sure if that's the problem on tempest side10:43
*** Tengu has joined #oooq11:08
*** udesale_ has joined #oooq11:10
*** udesale has quit IRC11:13
chandankumarstepping out for a bit will back in an hour11:16
*** ysandeep|rover is now known as ysandeep|afk11:35
*** udesale_ has quit IRC12:07
*** dmellado has quit IRC12:21
*** dmellado has joined #oooq12:23
*** skramaja has quit IRC12:29
*** rlandy has joined #oooq12:30
weshay|ruckchandankumar, 1 min12:30
*** ysandeep|afk is now known as ysandeep|rover12:32
*** jpena is now known as jpena|lunch12:32
arxcruz|ruckmarios: weshay|ruck chandankumar https://review.opendev.org/c/openstack/tripleo-quickstart/+/77364812:34
arxcruz|ruckthis will unblock fs020 and fs03912:35
arxcruz|ruckthen we only have scen10 to promote master12:35
arxcruz|ruckysandeep|rover: u da man!12:35
arxcruz|ruckysandeep|rover++12:35
ysandeep|roverfor fs39 we need https://review.opendev.org/c/openstack/tripleo-quickstart/+/773560 , failed due to content provider job, rerunning12:36
ysandeep|roverarxcruz|ruck, :)12:36
*** amoralej is now known as amoralej|lunch12:40
rlandychandankumar: marios: ysandeep|rover: do you guys still want to keep the dependency pipeline meetings?12:40
rlandyotherwise we can cancel12:40
rlandytoday's one?12:40
ysandeep|roverrlandy, I am focusing on Ruck/rovering.. may not be able to join today12:41
rlandyysandeep|rover: np12:41
mariossure we can cancel rlandy i just need some reviews now12:49
rlandymarios: k - pls  post - will look at them12:50
mariosrlandy: we can sync on monday scrum if there aren't any blockers to discuss12:50
rlandymarios; sure - I was thinking that the stream will wil really take over here12:50
mariosrlandy: https://projects.engineering.redhat.com/browse/TRIPLEOCI-294 rhel & https://projects.engineering.redhat.com/browse/TRIPLEOCI-197 centos reviews are listed please thanks12:51
mariosrlandy: not urgent just add to your reviews rotation please12:51
rlandymarios: sure12:52
*** ratailor has quit IRC13:06
ysandeep|roverweshay|ruck, ovb jobs for 17 line https://code.engineering.redhat.com/gerrit/#/c/226758/ ,  https://code.engineering.redhat.com/gerrit/#/c/226754/13:09
ysandeep|roverDo you want to wait for first green run before merge or want to handle issues later?13:09
ysandeep|roverrlandy, ^^ fyi..13:10
rlandyysandeep|rover: weshay|ruck: perhaps we could all discuss the timeline for 17 in general13:10
rlandywhen we want to get all those jobs working13:10
rlandywe will need them for rhel 913:11
ysandeep|roverweshay|ruck, rlandy, agree, another thing is we need more quota for ovb in downstream.. we are hitting quota issue on daily basis13:12
rlandyysandeep|rover: I +1'ed it13:12
rlandypersonally, I would add the definitions first13:12
rlandyand then merge the triggering13:12
rlandyysandeep|rover: but again - 17 is a bit of a mess in general now13:13
rlandyI am personally in favor of getting those jobs working again - ahead of rhel913:13
rlandyand then merging more jpbs to run13:13
rlandybut weshay|ruck has the say here on 1713:13
ysandeep|rovermaster is in good shape.. to save us lot of debugs for 17.. we may promote all the components once if they are stuck and deal with issue later.. in that case we know we are targetting right issues13:15
ysandeep|roverrlandy, could you please also review https://review.opendev.org/c/openstack/tripleo-quickstart/+/773648 , we need it to clear https://bugs.launchpad.net/tripleo/+bug/191473513:18
openstackLaunchpad bug 1914735 in tripleo "Master featureset020/fs039 failed while trying to upload image, HttpException: 500: Server Error for url: https://192.168.24.2:13292/v2/images/b898eee8-66c3-4a27-bbad-e6ca19d1f8d8/file, Internal Server Error" [Critical,Triaged] - Assigned to Sandeep Yadav (sandeepyadav93)13:18
ysandeep|roverto solve bug - We have 2 options, i.e disable nova / explicitly enable swift on featureset where we use nova13:19
chandankumarrlandy: hello, please have a look https://review.opendev.org/c/openstack/tripleo-quickstart/+/773285 , https://review.rdoproject.org/r/#/q/topic:containertools_depspipeline+(status:open+OR+status:merged) it will complete rdo container tools deps pipeline13:20
rlandysure13:21
rlandylooking13:22
weshay|ruckarxcruz|ruck, can we start 10min late so I can chat w/ bhagyashris ?13:23
*** jpena|lunch is now known as jpena13:31
arxcruz|ruckweshay|ruck: sure13:31
rlandyysandeep|rover: done13:35
ysandeep|rover++13:35
rlandychandankumar: looking13:37
rlandyysandeep|rover: we're also having issues with ovb 16.213:37
rlandylooking into those13:37
ysandeep|roverrlandy, ack o/ i saw wes and your chhater from last night, but was not sure where you left. I will look as well.13:41
*** amoralej|lunch is now known as amoralej13:43
weshay|ruckarxcruz|ruck, I'm in13:44
*** amoralej is now known as amoralej|dojo13:47
*** jpena is now known as jpena|dojo13:51
weshay|ruckping marios 0/ we should chat for 513:52
mariosweshay|ruck: o/13:56
weshay|ruckk.. finishing w/ arx.. few min13:58
weshay|rucksorry13:58
mariosweshay|ruck: np13:58
weshay|ruckactually  /me just reading ur email resonse13:58
weshay|ruckprobably fine.. no need to sync13:59
*** chandankumar is now known as raukadah13:59
weshay|ruckmarios++14:01
weshay|ruckno need to sync14:01
weshay|ruckthank you14:02
mariosweshay|ruck: ack cool thanks14:02
*** holser_ has joined #oooq14:07
*** holser has quit IRC14:08
ysandeep|roverrlandy, i think we can create a bug for this.. i haven't seen this error before - FATAL | Check Keystone user assignment to roles status | undercloud |14:27
ysandeep|roveralso infra is not stable, we also hitting no valid host 500 for some runs14:27
ysandeep|roverchecking if we have noticed similiar in train14:28
*** udesale has joined #oooq14:33
mariosplease comment if you care about this: http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020274.html * [TripleO] Moving stable/rocky for *-tripleo-* repos to End of Life OK? https://review.opendev.org/c/openstack/releases/+/77424414:41
ysandeep|roverrlandy, the only success we had in a while was on 1st feb, and 3rd sept run failed with "Check Keystone user assignment to roles status" , But they both were using same hash 47da4d8a93dbf8e945e4de23d78ed632, To me looks like issue is intermittent.15:00
ysandeep|rover1st feb - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2/e202eb6/job-output.txt15:00
ysandeep|rover4th feb - https://sf.hosted.upshift.rdu2.redhat.com/logs/10/207510/99/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2/6e3dfe0/job-output.txt15:00
ysandeep|roverLets see how today's run goes, if it is reproducible lets open a bug15:01
rlandyysandeep|rover: ok - I'll watch it in my afternoon hours15:02
ysandeep|roverthanks!15:02
* ysandeep|rover leaves for the day, see you guys on Monday :)15:02
*** ysandeep|rover is now known as ysandeep|away15:03
matbuweshay|ruck: not sure about the dependencies: https://review.rdoproject.org/r/#/c/31857/15:05
weshay|ruckTHANK YOU!15:05
matbui mean the job dependency15:05
matbu:)15:06
weshay|ruckmatbu, add the dep15:07
weshay|rucklook at the other jobs15:07
matbuack15:08
matbuok done15:10
rlandyysandeep|away: sorry was in meeting - seen it upstream or only downstream?15:13
rlandyraukadah: https://review.opendev.org/c/openstack/tripleo-quickstart/+/773285 +2 on that 0 can merge15:14
rlandymarios: ^^ want to give a second vote15:14
weshay|ruckrlandy, non-criteria atm.. but fyi https://review.rdoproject.org/r/#/c/31857/2/zuul.d/project-templates-components.yaml15:14
rlandyhttps://logserver.rdoproject.org/59/31759/1/check/periodic-tripleo-ci-centos-8-standalone-container-tools-master/cc45851/logs/undercloud/home/zuul/control_test_diff_table.log.txt.gz15:14
rlandyyay - right table - raukadah: ^^ thank you15:15
rlandyweshay|ruck: ok - non-criteria - I'm +2 on it15:15
mariosrlandy: sure, checking15:15
rlandymarios: also https://review.rdoproject.org/r/#/q/topic:containertools_depspipeline+(status:open+OR+status:merged) - so we can merge for raukadah15:17
rlandychecking your reviews now15:17
rlandymarios: ^^15:17
rlandylet's get this stuff in15:17
mariosraukadah: i merged your stuff ^^^ please replay by adding to your reviews queue: https://review.opendev.org/c/openstack/tripleo-quickstart/+/770620 https://review.rdoproject.org/r/31539 https://review.rdoproject.org/r/#/c/31841/ openvswitch15:22
mariosreplay repay ;)15:22
mariosraukadah: also standing charge of mojito/review applies15:22
weshay|ruckrlandy, need wf15:23
rlandymarios: +2 on https://review.opendev.org/c/openstack/tripleo-quickstart/+/77062015:23
rlandyweshay|ruck: done15:24
weshay|ruckthanks15:24
rlandyyou don;t have wf on rdo-jobs??15:24
rlandyhow did that happen?15:24
rlandyweshay|ruck: how did you get +2 on infrared abut no +2 on rdo-jobs?15:24
weshay|rucksec.. talking to migi15:25
mariosrlandy: thanks15:26
matbua merged a friday at 4:30 pm ... if the pipeline explode i'm away from kb :d15:27
rlandymarios: small fix required on https://review.rdoproject.org/r/#/c/31539/15:28
mariosrlandy: ack thanks for any comments i will review & update15:28
rlandymarios: comment on trigger days on https://review.rdoproject.org/r/#/c/31841/15:30
rlandyfine as it is - but doesn't run on the days the comment says it does15:31
rlandyzuul has it's  own day definitions - not the same as cron - for some stupid reason15:31
*** gchamoul has quit IRC15:33
mariosrlandy: oh i see so 0 != sunday? checking patch thanks15:36
raukadahmarios: once we meet in person someday, I will open the refrigerator filled with mozitaoo15:46
mariosraukadah: ;)15:47
*** jpodivin has quit IRC15:47
mariosraukadah: we met before in pune man you kept trying to feed me tings15:47
mariosthings15:47
raukadahmarios: hehe15:48
raukadahmarios: next time no feeding, just drinking15:48
*** tosky has quit IRC16:29
*** jmasud has joined #oooq16:46
*** marios is now known as marios|out16:49
*** marios|out has quit IRC16:56
*** jmasud has quit IRC17:14
*** udesale has quit IRC17:33
*** amoralej|dojo is now known as amoralej|off17:35
*** jpena|dojo is now known as jpena|off17:48
*** derekh has quit IRC18:05
*** dtantsur is now known as dtantsur|afk18:06
weshay|ruckrlandy, ruh roh18:28
rlandywhat now?18:28
rlandyovb?18:31
*** tosky has joined #oooq18:37
weshay|ruckrlandy, fyi https://code.engineering.redhat.com/gerrit/22678618:54
rlandyvoted18:56
rlandyrevote?18:58
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2/0b3e56e/logs/undercloud/var/log/tempest/stestr_results.html.gz18:59
rlandyhmm ...18:59
rlandycomparing with train18:59
*** jmasud has joined #oooq19:15
weshay|ruckrlandy, ok.. 5 min chat or are you done?20:21
rlandyweshay|ruck: let's chat20:21
weshay|ruckmeet.google.com/coa-accx-gfv20:21
*** slaweq has quit IRC20:37
*** jmasud has quit IRC20:57
*** jmasud has joined #oooq21:00
*** slaweq has joined #oooq21:16
*** slaweq has quit IRC21:22
*** rlandy has quit IRC22:00
*** jmasud has quit IRC22:06
*** jmasud has joined #oooq22:08
*** jmasud has quit IRC22:08
*** jmasud has joined #oooq22:10

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!