Tuesday, 2021-10-19

« Monday, 2021-10-18 Index Wednesday, 2021-10-20 »
*** dviroel|rover|afk is now known as dviroel|out00:50
*** pojadhav|out is now known as pojadhav|ruck03:11
*** ysandeep|out is now known as ysandeep05:02
*** ykarel_ is now known as ykarel05:20
*** ysandeep is now known as ysandeep|afk06:30
*** ysandeep|afk is now known as ysandeep|trng06:58
chandankumarysandeep|trng: for tempest cs9, I am proposing a patch in few mins07:12
*** jpena|off is now known as jpena07:32
*** amoralej|off is now known as amoralej07:36
ykarelpojadhav|ruck, is the issue with RDO third party jobs known? 07:50
ykarelfor promoted container images not found07:51
ykarelex. failure https://logserver.rdoproject.org/07/814407/5/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/fb85cf7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz07:51
pojadhav|ruckykarel, is this the same you are talking about : https://bugs.launchpad.net/tripleo/+bug/194665907:52
pojadhav|ruckykarel, i think issue is the same but jobs that are failing are different.07:54
ykarelpojadhav|ruck, that issue is different07:57
ykarelthat specific to upgrade jobs and when there is new promotion07:57
ykareland that last for some time07:57
pojadhav|ruckykarel, https://bugs.launchpad.net/tripleo/+bug/1946659/comments/507:58
ykarelpojadhav|ruck, yes both are seperate, i read that comment07:58
pojadhav|ruckykarel, I will report seperate bug then for this one.07:59
ykarelokk, also check if periodics also hitting it and if only master affected or it's generic08:00
ykarelseems generic but good to check08:00
pojadhav|ruckykarel, I seen this error in component line as well. but i was under impression like that issue already reported.08:01
ykarelpojadhav|ruck, component are similar to third party jobs so should be impacted08:01
ykarelas both use promoted containers from trunk rdo registry08:01
ykarelyou seein in master only, or non master as well?08:02
pojadhav|ruckykarel, ack.. will report a bug.08:02
pojadhav|ruckmaster08:02
pojadhav|ruckykarel, https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-common-master/3f6de84/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz08:02
pojadhav|ruckmaster common component08:03
ysandeep|trngchandankumar: ack, yes the last failure was in running tempest https://logserver.rdoproject.org/29/35229/20/check/periodic-tripleo-ci-centos-9-standalone-master/8b18cd0/job-output.txt , I see you have uploaded the testpatch with depends-on the fix.. thanks!08:17
*** ykarel is now known as ykarel|lunch08:48
Tengufolks, any new issues with the collect-logs and sealert?08:54
Tengujust wanting to double-check.08:55
Tenguerf...09:08
Tengu"cannot run as noservice with other option09:08
Tengu"09:09
Tengucan't add the -S option to sealert.09:09
pojadhav|ruckTengu, not came across the issue you are mentioning, could you provide log url if any..09:12
Tengupojadhav|ruck: https://28fe65829e35c5f04be9-f0a11af2fda48208ba4880b30ffd186c.ssl.cf2.rackcdn.com/814047/8/check/tripleo-ci-centos-8-containers-multinode/7f652d7/logs/undercloud/var/log/extra/selinux_denials_detail.txt09:12
Tengupojadhav|ruck: that "-S" option was introduced lately. Is there anything checking for that sealert result?09:13
Tengupojadhav|ruck: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/81453309:13
Tenguthat's the correction patch.09:13
TenguI mostly ensured my consolidation script was working back then, and didn't check the sealert output -.-. Of course I should have checked when I "imported" that change from the time-limiting patch :/09:14
pojadhav|ruckTengu, ack09:28
*** pojadhav|ruck is now known as pojadhav|lunch09:32
mariosplease add to your review queue https://review.opendev.org/c/openstack/tripleo-repos/+/813975 09:57
*** pojadhav|lunch is now known as pojadhav09:59
chandankumarbhagyashris: please have a look at this https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36300 thanks!10:08
bhagyashrischandankumar, ack10:15
chandankumarsshnaidm: please add cs9 hashtag to this review https://review.opendev.org/c/openstack/diskimage-builder/+/806819 when free, thanks :-)10:28
sshnaidmchandankumar, done10:29
chandankumarthanks :-)10:30
chandankumarsshnaidm: marios please add this to review queue https://review.opendev.org/c/openstack/tripleo-heat-templates/+/813941 when free, thanks!10:32
*** rlandy is now known as rlandy|ruck10:34
*** ykarel|lunch is now known as ykarel10:34
mariosack chandankumar 10:35
rlandy|ruckpojadhav: hi10:39
rlandy|ruckpojadhav: how are things?10:39
*** pojadhav is now known as pojadhav|ruck10:40
pojadhav|ruckrlandy|ruck, https://bugs.launchpad.net/tripleo/+bug/194767910:40
pojadhav|ruckwe were came across above issue ^^10:40
pojadhav|ruckwallaby promoted 3 hrs ago :)10:41
pojadhav|ruckneed to look at master promotions atm. 10:42
rlandy|ruckpojadhav|ruck: let's sync in a few10:44
pojadhav|ruckrlandy|ruck, yup.. will take short break then will sync.10:45
* pojadhav|ruck brb10:45
rlandy|ruckpojadhav|ruck: k - let's sync for a few10:59
rlandy|ruckakahat: let's sync in a bit re: scenario 01011:00
rlandy|ruckbhagyashris: hey - are we all set for the new sprint board?11:01
rlandy|ruckbhagyashris: need to be sure I can do this when you are out :)11:01
* pojadhav|ruck back11:03
pojadhav|ruckrlandy|ruck, should we wait for dviroel|out ?11:04
bhagyashrisrlandy|ruck, yes today chandan and me updated board 11:05
*** amoralej is now known as amoralej|lunch11:05
bhagyashrisall c9 related work11:05
rlandy|ruckbhagyashris: awesome  - thank you ... we need one more - monitoring for c9 - cockpit work11:06
*** dviroel|out is now known as dviroel|rover11:06
dviroel|roverpojadhav|ruck: rlandy|ruck o/11:06
pojadhav|ruckdviroel|rover, \o11:07
chandankumarrlandy|ruck: cards are there for cockpit on jira board11:07
rlandy|ruckpojadhav|ruck: dviroel|rover: hey   - need to meet with chandankumar in a three minutes - let's meet at half passed the hour11:08
rlandy|ruckchandankumar: oh - you added it in the mini line11:08
rlandy|ruckhmmm - all else is separated11:08
rlandy|ruckok - we'll work it out11:08
pojadhav|ruckrlandy|ruck, ack11:08
dviroel|roverok11:10
bhagyashrisrlandy|ruck, ack11:14
chandankumarbhagyashris: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36300 address your comments11:20
rlandy|ruckmarios: pls will you vote on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813654/11:25
*** jpena is now known as jpena|lunch11:26
rlandy|ruckysandeep|trng: pls vote on https://review.opendev.org/c/openstack/tripleo-ci/+/813619/ - consider downstream impact11:28
ysandeep|trngrlandy|ruck: ack, will check in sometime11:29
mariosrlandy|ruck: looking11:32
mariosrlandy|ruck: i voted on that earlier was there an update? 11:32
mariosrlandy|ruck: done 11:33
rlandy|ruckmarios: thanks11:33
rlandy|rucksshnaidm: pls review and vote on https://review.opendev.org/c/openstack/tripleo-ci/+/814428/11:35
chandankumarsshnaidm: https://review.opendev.org/c/openstack/tripleo-repos/+/813975 this one also11:35
rlandy|ruckarxcruz: is https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809979 ready to merge?11:36
arxcruzrlandy|ruck: yes 11:36
rlandy|ruckdviroel|rover: pojadhav|ruck: sorry - https://meet.google.com/zpk-ssbn-vwz?pli=1&authuser=011:36
pojadhav|ruckrlandy|ruck, dviroel|rover : https://bugs.launchpad.net/tripleo/+bug/194767911:39
pojadhav|ruckhttps://bugs.launchpad.net/tripleo/+bug/194665911:40
pojadhav|ruckrlandy|ruck, dviroel|rover :11:50
pojadhav|ruckhttps://review.rdoproject.org/r/c/testproject/+/36274/211:50
pojadhav|ruckrlandy|ruck, dviroel|rover : https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-common-ussuri/45243e0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz11:51
ysandeep|trngrlandy|ruck: fyi.. Testing downstream here for "exclude list" patch: https://code.engineering.redhat.com/gerrit/c/testproject/+/27301811:54
dviroel|roverrlandy|ruck: pojadhav|ruck https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81448811:58
* pojadhav|ruck brb12:00
akahatrlandy|ruck, hey.. can we sync now re: sc10 jobs.12:06
rlandy|ruckakahat: sure - let's meet at 12:15 UTC - like in 8 minutes12:08
akahatrlandy|ruck, okay. cool12:08
rlandy|rucksoniya29: arxcruz: what happened with the port tracking on tempest in upstream12:09
arxcruzrlandy|ruck: which port tracking?12:10
rlandy|rucksoniya29's patch12:10
soniya29arxcruz, https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/81399012:10
rlandy|rucknetstat12:10
arxcruzsoniya29: rlandy|ruck I'm sorry, for some reason, my gmail is moving all review notifications to spam, let me check 12:11
soniya29arxcruz, chandankumar, as per our last discussion main.yml is the right place to add the netstat logging thing right?12:13
arxcruzsoniya29: just post my comments 12:14
arxcruzsoniya29: you need to add it on the tempest_post_install.yml to start the netstat, and on tempest_cleanup.yml a task to stop the netstat 12:14
arxcruzalso, these should have a tempest_netstat (or whatever) variable default to false and then we can set this variable to true on our tempest.yml playbook 12:15
*** jpena|lunch is now known as jpena12:15
ysandeep|trngakahat, rlandy|ruck downstream is not happy with exclude list patches: https://sf.hosted.upshift.rdu2.redhat.com/logs/18/273018/9/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/179fd20/job-output.txt 12:15
chandankumarysandeep|trng: checking12:17
ysandeep|trngrlandy|ruck, I see you already +wed https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813654 , could we please remove that till we fix this issue.. (we may need abandon/restore)12:17
rlandy|ruckysandeep|trng: I un w+'ed it12:18
rlandy|ruckit was not yet in gates12:18
rlandy|ruckso it should be fine12:18
rlandy|ruckysandeep|trng: feel free to w-1 it12:18
soniya29arxcruz, i believe we want to log the netstat data because we want to track which ports are open and other network related stuffs before the deployment of tempest12:18
soniya29arxcruz, here is the infrared patch - https://review.gerrithub.io/c/redhat-openstack/infrared/+/524690/6/plugins/tempest/main.yml12:19
rlandy|ruckakahat: https://meet.google.com/uct-prgg-fwk?pli=1&authuser=012:20
soniya29arxcruz, if we move this to tempest_post_install, i am just concerned about the earlier motivation behind the netstat logging since we wanted to log the data before tempest run12:20
soniya29chandankumar, ^^12:21
arxcruzsoniya29: but tempest_post is executed before tempest_run 12:23
rlandy|ruckmarios: hey - chatting with akahat about log rotate - can you help there?12:23
rlandy|ruckwe need some more eyes on that12:23
rlandy|ruckthanks12:23
chandankumarakahat: https://review.opendev.org/c/openstack/tripleo-ci/+/813619/8#message-77d400ddb73b5b1e40fc35e75720837c08a3cffe12:24
chandankumarit is rhel vs redhat12:24
mariosrlandy|ruck: sure i don't know much about it but i can try and help... i think there is a reveiw already right? will keep an eye on it and akahat we can sync up if you like maybe tomorrow our morning for example? 12:24
rlandy|ruckmarios: yea - akahat will send it to you and what's already on the promoter12:25
mariosrlandy|ruck: k 12:25
akahatmarios, hey.. we can sync tomorrow morning, i'll sent you invite. 12:41
mariosakahat: ack sounds good you bring breakfast12:42
akahatchandankumar, okay.. fixing it.12:42
akahatmarios, sure.. and you coffee. :)12:42
mariosakahat: ack 12:42
soniya29arxcruz, okay12:49
*** amoralej|lunch is now known as amoralej12:53
*** ysandeep|trng is now known as ysandeep12:56
rlandy|ruckPTG times13:01
rlandy|ruckhttps://www.openstack.org/ptg/rooms/folsom13:03
rlandy|ruckarxcruz: can you join #rhos-ops internal13:07
rlandy|ruckdviroel|rover: pojadhav|ruck: lol - we're still out on cloudops 13:47
rlandy|ruck2021-10-19 13:31:01.144221 | fa163e04-66c6-9e8b-9b20-00000000301b |      FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | error={"changed": false, "msg": "Service multipathd has not started yet"}13:47
rlandy|ruckwill try that again after change merges in gate13:47
rlandy|ruckscnario007 on wallaby13:48
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/2499513:51
rlandy|rucktrying all scenario001 again13:51
pojadhav|ruckrlandy|ruck, ack13:52
ysandeepchandankumar, rlandy|ruck downstream container build passing with latest patchset: https://code.engineering.redhat.com/gerrit/c/testproject/+/273018 14:07
chandankumarysandeep: cool ++14:07
chandankumarysandeep: please convert -2 to +214:08
ysandeepchandankumar, already done 14:08
chandankumarvery good14:08
rlandy|ruckysandeep: thanks for that update14:08
chandankumarrlandy|ruck: updated this one https://etherpad.opendev.org/p/centos-stream-9-upstream and anything more to add?14:11
rlandy|ruckchandankumar: looking - btw: https://etherpad.opendev.org/p/branched-unbranched-dependency-clashes14:11
rlandy|ruck^^ for dependencies topic we discussed14:12
rlandy|ruckchandankumar: anything in https://etherpad.opendev.org/p/centos-stream-9-upstream that is not in presentation?14:12
rlandy|ruckpresentation looks great14:12
rlandy|ruckwe just need to raise updates/upgrades14:12
rlandy|ruckchandankumar: ok if I add a slide for that?14:13
chandankumarrlandy|ruck: yes sure14:14
*** lbragstad_ is now known as lbragstad14:15
rlandy|ruckchandankumar: added slide 'Open questions - update/upgrades'14:16
rlandy|ruckpls review14:16
rlandy|ruckand add what you want14:16
rlandy|ruckchandankumar: the rest is great14:16
rlandy|ruckthanks for this work14:16
chandankumarrlandy|ruck: moving risks and questions in the last14:19
rlandy|ruckchandankumar: sure14:20
chandankumarrlandy|ruck: looks good now :-) Thank you for the push and all the help :-)14:20
rlandy|ruckchandankumar: sur e- pls see if you want to add anything to https://etherpad.opendev.org/p/branched-unbranched-dependency-clashes14:21
rlandy|rucktox dep issues can go there14:21
rlandy|ruckwe have the last session on wed for the dependencies topic14:21
chandankumarrlandy|ruck: I am finding one more review on which we worked14:22
rlandy|ruckmarios: sshnaidm: akahat: ^^ fyi - if you have anything to add re: pip install, dependencies, constraints etc.14:22
rlandy|ruckreleases etc.14:22
rlandy|ruckjpodivin added most of the good stuff there14:22
*** ykarel_ is now known as ykarel14:22
rlandy|ruckdviroel|rover: pojadhav|ruck: do either if you want to present at the program call?14:23
mariosrlandy|ruck: ack will check thanks14:23
rlandy|ruckI will update the doc before my EoD14:23
pojadhav|ruckrlandy|ruck, I will do14:24
rlandy|ruckpojadhav|ruck++ thanks14:24
pojadhav|ruck:)14:24
chandankumarrlandy|ruck: one good exmaple of dance https://review.opendev.org/q/topic:%22log_timeout%22+(status:open%20OR%20status:merged)14:25
rlandy|ruckadding14:26
chandankumaradded 14:27
rlandy|ruckoh I see you added it14:27
* dviroel|rover lunch, biab14:44
chandankumaramoralej: rlandy|ruck: ysandeep we have standalone deployment on cs9 with 4 failing tests https://logserver.rdoproject.org/29/35229/21/check/periodic-tripleo-ci-centos-9-standalone-master/f2ee6cd/logs/undercloud/var/log/tempest/tempest_run.log15:00
rlandy|ruckchandankumar++ ysandeep++15:00
rlandy|ruckgreat news15:00
* arxcruz is having connection issues, there's an outage on my area right now15:08
jpodivinAre there recommendations for the nodeset/featureset combinations? 15:25
jpodivinbecause I think 010 doesn't play well with 1ctlr_1comp15:25
ysandeepfs010 goes with "nodes: 1ctlr" 15:27
ysandeephttps://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset010.yml#L29-L40 - only controller role, no compute role in fs01015:28
jpodivinhm15:31
jpodivinI guess the only way to make sure is to manually check the definitions then, either that or pick one fs which already uses that node set. 15:34
jpodivinThanks ysandeep++15:34
rlandy|ruckchandankumar: sshnaidm: marios: we got a w+ on https://review.opendev.org/c/openstack/diskimage-builder/+/81408116:20
rlandy|ruckamoralej: ^^16:20
chandankumarsweet16:20
chandankumarsweet16:20
amoralejcool16:20
rlandy|ruckgetting there :)16:21
amoraleji'm updating my dib patch to clean hacks16:21
rlandy|rucksshnaidm's patch may need a recheck when this merges16:21
amoralejand make it ready for commit16:21
sshnaidmrlandy|ruck, nice16:21
rlandy|rucktwo depends-on merged16:21
rlandy|ruckone in zuul-jobs is merged as well16:21
chandankumarrlandy|ruck: get these merged https://review.opendev.org/c/openstack/tripleo-ci/+/808177 and https://review.opendev.org/c/openstack/diskimage-builder/+/806819 and recheck this https://review.rdoproject.org/r/c/config/+/3623416:22
chandankumaron your day time16:22
chandankumarwill add overcloud jobs in pipeline tomorrow16:22
chandankumarall hail to marios sshnaidm amoralej \o/16:22
chandankumarsee ya!16:23
rlandy|ruckchandankumar: have a good night16:23
*** marios is now known as marios|out16:25
*** pojadhav|ruck is now known as pojadhav|out16:26
*** jpena is now known as jpena|off16:27
*** ysandeep is now known as ysandeep|out16:29
-opendevstatus- NOTICE: Both Gerrit and Zuul services are being restarted briefly for minor updates, and should return to service momentarily; all previously running builds will be reenqueued once Zuul is fully started again16:59
rlandy|rucklunch - brb17:22
rlandy|ruckdviroel|rover: want to ruck/rover sync now?18:01
rlandy|ruckintrospection and whatever else18:02
dviroel|roverrlandy|ruck: yes18:02
rlandy|ruckdviroel|rover: https://meet.google.com/khe-hvnt-ghd?pli=1&authuser=018:02
dviroel|roverrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-common/+/809148/2#message-0feb9c22078df028151f277b3ea01af8a607fca019:12
rlandy|ruckdviroel|rover: interesting find19:13
rlandy|ruckand on the right date19:13
rlandy|ruckhttps://review.opendev.org/q/b02ad2618639ef39bd03c58d845ac8091d9bb90a19:13
rlandy|ruckit's cherry picked right back19:13
rlandy|ruckdviroel|rover: we could try a revert on that and see if introspection speeds up19:14
rlandy|ruckhttps://bugzilla.redhat.com/show_bug.cgi?id=199953919:15
rlandy|ruckdviroel|rover: that would certainly account for the longer time19:18
rlandy|ruckinteresting that impacts master differently19:19
rlandy|ruckmaybe we can try cut the time and or the delay19:19
dviroel|roveryeah, merged in 10/0919:23
dviroel|roveralready merged in downstream too19:25
rlandy|ruckwe should check 16.2 :)19:26
rlandy|ruckonly way to know is t try play with that19:27
rlandy|ruckdviroel|rover++ nice catch19:27
dviroel|roverwill create the revert and test with fs03519:27
rlandy|ruckthank you, sir19:27
* rlandy|ruck needs to create vm for interview19:28
rlandy|ruckping if needed19:28
dviroel|roverack, tks19:28
dviroel|roverrlandy|ruck: so, will continue with the revert test19:53
dviroel|roverand we test both approaches19:53
ade_leedviroel|rover, let me know if you need help in putting together the changes for multinode stuff19:54
dviroel|roverade_lee: ack, tks19:55
ade_leedviroel|rover, right now, the dependency is on the rdocloud job, but we could just add it to the underlying patch19:55
ade_leeas it'll need to be there in any case19:55
rlandy|ruckdviroel|rover: sure - submitted the concurrency change 19:58
rlandy|rucktesting it out19:58
dviroel|rovertesting it here: https://review.rdoproject.org/r/c/testproject/+/3513220:00
dviroel|roverade_lee: for the multinode-fips, i will let the fips enable/disable flag, but let 'enabled' as default20:14
ade_leeyeah20:15
rlandy|ruckupdating program doc20:16
dviroel|roverade_lee: https://review.opendev.org/c/zuul/zuul-jobs/+/81325320:22
ade_leedviroel|rover, looks good - lets try it out20:29
dviroel|roveryeah, we need to change 'tripleo-ci-base-required-projects-centos-8' parent to 'multinode-fips'20:31
dviroel|roverand disable fips on content providers20:31
dviroel|roverI will need to create another 'tripleo-ci' change20:32
dviroel|roverade_lee: https://review.opendev.org/c/openstack/tripleo-ci/+/81464820:52
dviroel|rovers/multinode/multinode-fips20:52
dviroel|roverade_lee: https://review.rdoproject.org/r/c/testproject/+/3523520:53
dviroel|rovernot sure if I missed some details20:53
ade_leedviroel|rover, ack -- looking20:53
ade_leedviroel|rover, you can remove the dependency on https://review.opendev.org/c/openstack/tripleo-ansible/+/810783  its already merged20:56
dviroel|roveroh cool, will remove in the next ps20:57
ade_leedviroel|rover, other than that , I think it looks fine20:57
ade_leedviroel|rover, lets see what happens :)20:58
dviroel|roveryep :)20:58
*** dviroel|rover is now known as dviroel|rover|afk20:59
« Monday, 2021-10-18 Index Wednesday, 2021-10-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!