Friday, 2022-03-04

« Thursday, 2022-03-03 Index Monday, 2022-03-07 »
*** frenzy_friday is now known as frenzyfriday|ruck04:38
*** ysandeep|out is now known as ysandeep04:47
ysandeepfrenzyfriday|ruck, fyi.. i had to kill 16.2 line run to get some nodes for debugging (it was taking very long to get a node) - only 1 ovb job need pass to promote 16.2 - I will run it using testproject05:23
ysandeepfrenzyfriday|ruck, https://code.engineering.redhat.com/gerrit/c/testproject/+/276230 retriggered fs035 job here.. 05:25
frenzyfriday|ruckysandeep, ack, thanks05:31
*** soniya29 is now known as soniya29|ruck05:33
frenzyfriday|ruckhey, does anyone know if the upstream cockpit is down? http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1 shows 40405:37
*** frenzyfriday|ruck is now known as frenzyfriday|rover05:40
frenzyfriday|roversoniya29|ruck, Hey 0/ good morning. Are you able to access the upstream cockpit?05:41
pojadhavfrenzyfriday|rover, not working... getting 404 not found06:02
frenzyfriday|roverpojadhav, ack06:02
marioschandankumar: akahat: what is next to review for the reparenting? can we start to remove some of the unused patches (abandond them)06:55
marioschandankumar: is this the right topic? https://review.opendev.org/q/topic:cs-parenting06:55
mariosi think there are many more patches than this06:55
chandankumarmarios: https://review.opendev.org/q/topic:cs-parenting-multinode06:56
chandankumarand then https://review.opendev.org/q/topic:cs-parenting-build-containers06:57
chandankumarmarios: on rdo side https://review.rdoproject.org/r/q/topic:cs-parenting-multinode06:57
chandankumarfor downstream container multinode we just merged the patches06:58
chandankumarhttps://review.rdoproject.org/r/q/topic:cs-parenting-containers-build06:58
chandankumarlet me put the order in review hackmd06:59
marioschandankumar: ok. i think it would be better to use a single topic but... up to you07:00
mariosfor me this is confusing th epoint of the topic is to give you a single place to check 07:01
chandankumarmarios: ok I will check the topic to cs-parenting07:01
marioschandankumar: the merge order should just be enforced within the repo (rebase changes into the right order) and use depends-on for rdo<->tripleo-ci07:01
mariosthen we don't have to explain/change the merge order each time07:01
*** ysandeep is now known as ysandeep|mtg07:13
chandankumarysandeep|mtg: pojadhav quick merge https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/315845 and https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/31584607:41
soniya29|ruckfrenzyfriday|rover, i can't access it07:43
frenzyfriday|roversoniya29|ruck, ack. Pooja found out that the containers have exited 8 hrs ago. Might be someone was working on it.07:43
frenzyfriday|roverI'll check with Jakob/Ronelle when they are online07:44
soniya29|ruckfrenzyfriday|rover, sure07:44
ysandeep|mtgchandankumar, left comment on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315846 07:47
* ysandeep|mtg in a debug session, will be slow to respond07:47
chandankumarysandeep|mtg: sure, no problem!07:51
*** amoralej|off is now known as amoralej07:52
chandankumarmarios: moved all reviews to hashtags08:02
chandankumarhttps://review.rdoproject.org/r/q/hashtag:cs-parenting+08:02
chandankumarhttps://code.engineering.redhat.com/gerrit/q/hashtag:%2522cs-parenting%2522+08:02
chandankumarhttps://review.opendev.org/q/hashtag:cs-parenting+08:02
ysandeep|mtgchandankumar, +wed, lets hit the testproject again08:09
chandankumaraye sir08:09
*** jpena|off is now known as jpena08:10
chandankumarpojadhav: ysandeep|mtg you might see container-multinode downstream job to error state08:15
chandankumarthat is expected08:15
ysandeep|mtgchandankumar: ack.. fyi.. soniya29|ruck frenzyfriday|rover ^^ too as upstream ruck/rover monitors downstream lines now.08:16
mariosthank you chandankumar 08:17
chandankumarrdo jobs does not have nay impact08:17
chandankumar*any08:17
frenzyfriday|roverack08:22
pojadhavchandankumar, ack08:26
marioschandankumar: is that because you are moving the base definitino out of the config repo you mean? 08:27
marioschandankumar: is it related to https://review.rdoproject.org/r/c/rdo-jobs/+/39343 ? but we should merge this now? 08:27
chandankumarmarios: this one is good to go08:28
chandankumarmarios: yes08:28
chandankumarmarios: we saw some breakage in downstream https://code.engineering.redhat.com/gerrit/c/testproject/+/301974/3#message-45afa3b18d3e8b6264206c326cedf7a7e776345708:28
chandankumarbut will be fixed now, just rechecking the patches08:29
chandankumarmarios: rlandy|out merged multinode rdo parenting patches, they saw some zuul job freeze and they reverted it08:30
chandankumarin the morning, I self merged the reverted RDO config patch and tested via test project08:30
chandankumarthings are fine now08:30
marioschandankumar: k merged that //review.rdoproject.org/r/c/rdo-jobs/+/3934308:31
*** ysandeep|mtg is now known as ysandeep|lunch08:45
mariosrelocate biab10:03
frenzyfriday|roverhey bhagyashris|ruck, do we already have a bug on kerberos issue in tripleo-ci-centos-9-standalone-on-multinode-ipa? I see only https://bugs.launchpad.net/tripleo/+bug/1963545 for fs3910:54
bhagyashris|ruckfrenzyfriday|rover, for kerberos we dont have10:56
bhagyashris|ruckfs064 is also failing with same https://bugs.launchpad.net/tripleo/+bug/196354510:56
frenzyfriday|roverbhagyashris|ruck, ack, thanks10:57
*** rlandy_ is now known as rlandy|ruck11:12
chandankumarrlandy|ruck: hello11:14
chandankumarrlandy|ruck: I have re reverted the parenting patch and merged 11:15
chandankumarwe are currently testing it, we have not seen any breakage and no job freeze issue11:15
rlandy|ruckchandankumar: we're on review time11:16
rlandy|ruckfrenzyfriday|rover: soniya29|ruck: you around?11:19
rlandy|ruckfrenzyfriday|rover: soniya29|ruck: pls join review time11:19
frenzyfriday|roverrlandy|ruck, 0/11:19
chandankumarrlandy|ruck: ysandeep can we set this pipeline debug https://zuul-ci.org/docs/zuul/latest/config/project.html#attr-project.%3Cpipeline%3E.debug11:28
chandankumarto true11:28
chandankumarysandeep: passed me the link11:28
ysandeepfrenzyfriday|rover, fyi.. https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-glance-master/8e387e7/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz11:35
ysandeephttp://pastebin.test.redhat.com/103447711:35
* frenzyfriday|rover checking the component failures11:35
*** bhagyashris|ruck is now known as bhagyashris11:36
ysandeepfrenzyfriday|rover, component lines failing at container update ^^ logs above11:36
chandankumarrlandy|ruck: marios frenzyfriday|rover removing ceph-ansible line https://review.rdoproject.org/r/c/config/+/4008111:59
frenzyfriday|roverack11:59
marioschandankumar: k 12:03
rlandy|ruckjm1: http://tripleo-cockpit.lab4.eng.bos.redhat.com/ down12:12
soniya29|ruckrlandy|ruck, frenzyfriday|rover, https://review.rdoproject.org/r/c/testproject/+/40083/12:21
frenzyfriday|roversoniya29|ruck, I think it should be the other way round for the dependencies, no?12:23
soniya29|ruckfrenzyfriday|rover, for all the dependencies you mean or just fs01?12:24
rlandy|rucksoniya29|ruck: let's go through this12:25
rlandy|ruckhttps://meet.google.com/znd-cepr-ofb?pli=1&authuser=012:26
rlandy|rucksoniya29|ruck: ^^12:26
frenzyfriday|roversoniya29|ruck, http://pastebin.test.redhat.com/103451012:27
soniya29|ruckfrenzyfriday|rover, thanks12:30
soniya29|ruckrlandy|ruck, i have updated the testproject patch12:32
soniya29|ruckrlandy|ruck, moving forward with 17 line issues12:32
rlandy|rucksoniya29|ruck: dependenies is not spaced correctly12:33
rlandy|rucksoniya29|ruck: will edit after 1-1-12:33
chandankumarmarios: ysandeep I can confirm multinode jobs after reparenting working fine12:34
chandankumaron rdo side12:35
chandankumarmarios: ysandeep good to merge upstream also 12:35
chandankumarhttps://review.rdoproject.org/zuul/status#4006812:35
marioschandankumar: ack will look in bit 12:39
pojadhavfolks, need reviews on component line rhel9 osp17 : https://code.engineering.redhat.com/gerrit/q/topic:%22rhos17_rhel9%22+(status:open)12:49
frenzyfriday|roverrlandy|ruck, a lot of the component jobs failed with: nerror running container: write containercreatepipe: broken pipe\nerror while running runtime: exit status 1", "stderr_lines": ["time=\"2022-03-04T07:29:34Z\" level=error msg=\"did not get container create message from subprocess - The history doesnt look red. I have rechecked the jobs12:51
rlandy|ruckpojadhav: hey - will be a few minites late13:00
pojadhavrlandy|ruck, sure13:00
*** pojadhav is now known as pojadhav|brb13:09
rlandy|ruckpojadhav|brb: joined13:10
rlandy|rucksoniya29|ruck: pls see updated patch13:12
soniya29|ruckrlandy|ruck, thanks :)13:15
rlandy|ruckfrenzyfriday|rover: ok - so let's concentrate on the integration line failures for now13:40
rlandy|ruckussuri is running13:40
rlandy|ruckwallaby c9 started13:40
frenzyfriday|roverrlandy|ruck, ack, I testprojed master13:41
frenzyfriday|rovertoo many component job failures13:41
*** amoralej is now known as amoralej|lunch13:41
rlandy|ruckfrenzyfriday|rover: and there are a couple more failures in wallaby c813:41
rlandy|ruckfrenzyfriday|rover: ack - put them aside for now13:41
rlandy|ruckfrenzyfriday|rover: do you have a patch for non-voting ipa job?13:42
frenzyfriday|rovernot  yet, creting13:42
rlandy|ruckfrenzyfriday|rover: ok - thanks13:42
rlandy|ruckonce we have the gate cleared - we have time13:42
rlandy|ruckwill work with ade_lee_ later on this13:42
rlandy|ruckwe started looking at it yesterday13:42
rlandy|ruckbut vexx was too crazy13:42
rlandy|ruckdasm|off: ^^ you can join this  - security debug13:43
jm1rlandy|ruck: downstream cockpit up again13:48
rlandy|rucksoniya29|ruck: pls look into the rest of the failures on the wallaby 8 line13:48
rlandy|ruckjm1: thank you13:50
frenzyfriday|roverhey rlandy|ruck https://review.opendev.org/c/openstack/tripleo-ci/+/831942 - makes multinode ipa NV13:54
*** pojadhav|brb is now known as pojadhav|afk14:09
dasm|offo/14:20
*** dasm|off is now known as dasm14:20
chandankumarrlandy|ruck: pojadhav|afk ysandeep please have a look at build container series https://code.engineering.redhat.com/gerrit/q/topic:%22cs-parenting-build-containers%22+(status:open%20OR%20status:merged)14:41
chandankumarwill merge build containers on monday india morning with proper testing14:42
chandankumarsee ya, Happy weekend :-)14:46
frenzyfriday|roverhey rlandy|ruck, periodic-tripleo-ci-build-containers-centos-9-push-master this one timed out last time right? Now it is green, hope the rest starts soon14:47
rlandy|ruckfrenzyfriday|rover: good news14:48
*** amoralej|lunch is now known as amoralej14:50
*** dviroel is now known as dviroel|lunch14:56
rlandy|ruckfrenzyfriday|rover: soniya29|ruck: let's sync after happy friday15:00
rlandy|ruckand see where you are15:00
frenzyfriday|roverrlandy|ruck, ack15:00
soniya29|ruckrlandy|ruck, hackmd for wallaby centos-8 failures15:01
soniya29|ruckrlandy|ruck, https://hackmd.io/uA4i9wDhQfiolHeVOIHbdg15:01
frenzyfriday|roverrlandy|ruck, soniya29|ruck do you want to sync now?15:22
soniya29|ruckfrenzyfriday|rover, sure15:22
rlandy|ruckfrenzyfriday|rover: soniya29|ruck: https://meet.google.com/zzb-xqdi-rik?pli=1&authuser=015:25
ysandeeparxcruz, sry was occupied in other stuff in background, but wanted to see your happy faces - so I have joined the call.15:29
arxcruzysandeep my happy face? sorry for disappoint you this time :)15:30
ysandeeparxcruz, yeah, that's why I didn't speak.. :D15:30
ysandeeprlandy|ruck, chandankumar fyi.. bug for buildah issue: https://bugzilla.redhat.com/show_bug.cgi?id=206093216:02
rlandy|ruckrcastillo: will need to push out your 1-1 to this afternoon16:02
rlandy|ruckdigging out of vexxhost mess16:02
ysandeepykarel++ for helping narrowing down the issue.16:02
rcastillorlandy|ruck: ack16:03
ykarelysandeep++16:10
rlandy|ruckysandeep: hey - can you join https://meet.google.com/zzb-xqdi-rik?pli=1&authuser=016:14
ysandeeprlandy|ruck, http://pastebin.test.redhat.com/103447716:18
frenzyfriday|roverpodman-3.4.5-0.7.el9.x86_64, podman-catatonit-3.4.5-0.7.el9.x86_64 - passing job16:20
*** dviroel|lunch is now known as dviroel16:20
frenzyfriday|roverpodman-4.0.0-6.el9.x86_64,podman-catatonit-4.0.0-6.el9.x86_64 - failing16:20
frenzyfriday|roverhttps://logserver.rdoproject.org/82/40082/2/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/79eae19/logs/undercloud/var/log/extra/rpm-list.txt.gz16:20
frenzyfriday|roverrlandy|ruck, ysandeep ^16:20
ysandeeprlandy|ruck, https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-wallaby/c5c5319/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz16:23
frenzyfriday|roverhttps://bugs.launchpad.net/tripleo/+bug/1963702 rlandy|ruck soniya29|ruck 16:29
*** ysandeep is now known as ysandeep|out16:31
* ysandeep|out out,for the above bug:= I would say trigger sc010 component jobs downstream for a particular component, if that passes and rest which triggered in rdo fails - that's definetely infra.16:33
ysandeep|outrdo(i mean on vexx)16:33
rlandy|ruckpodman.x86_64                               3.0.1-6.module_el8.5.0+736+58cc1a5a       @appstream                         16:39
rlandy|ruckpodman-catatonit.x86_64                     3.0.1-6.module_el8.5.0+736+58cc1a5a       @appstream     16:39
frenzyfriday|roverSK | Check containers status16:46
frenzyfriday|rover2022-03-04 09:51:28 |  [ERROR]: Container(s) which failed to be created by podman_container module:16:46
frenzyfriday|rover2022-03-04 09:51:28 | ['nova_db_sync']16:46
frenzyfriday|rover2022-03-04 09:51:28 |  [ERROR]: Container(s) which did not finish after 300 minutes: ['nova_db_sync']16:46
frenzyfriday|rover2022-03-04 09:51:28 | 2022-03-04 09:51:28.999785 | fa163e1f-30ca-4089-242a-00000000a05b |      FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['nova_db_sync'], check logs in /var/log/containers/stdouts/"}16:46
frenzyfriday|rover2022-03-04 09:51:28 | 2022-03-04 09:51:29.000873 | fa163e1f-30ca-4089-242a-00000000a05b |     TIMING | tripleo_container_manage : Check containers status | overcloud-controller-0 | 1:09:05.953041 | 461.26s16:46
frenzyfriday|roverrlandy|ruck, soniya29|ruck ^16:46
frenzyfriday|roverfor SK | Check containers status16:46
frenzyfriday|rover2022-03-04 09:51:28 |  [ERROR]: Container(s) which failed to be created by podman_container module:16:46
frenzyfriday|rover2022-03-04 09:51:28 | ['nova_db_sync']16:46
frenzyfriday|rover2022-03-04 09:51:28 |  [ERROR]: Container(s) which did not finish after 300 minutes: ['nova_db_sync']16:46
frenzyfriday|rover2022-03-04 09:51:28 | 2022-03-04 09:51:28.999785 | fa163e1f-30ca-4089-242a-00000000a05b |      FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['nova_db_sync'], check logs in /var/log/containers/stdouts/"}16:46
frenzyfriday|rover2022-03-04 09:51:28 | 2022-03-04 09:51:29.000873 | fa163e1f-30ca-4089-242a-00000000a05b |     TIMING | tripleo_container_manage : Check containers status | overcloud-controller-0 | 1:09:05.953041 | 461.26s16:46
frenzyfriday|roverperiodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri 16:46
frenzyfriday|roverrlandy|ruck, https://review.opendev.org/c/openstack/tripleo-ci/+/831942 NV16:50
*** marios is now known as marios|out16:52
rlandy|ruckdasm: ^^ security follow up for :)16:55
frenzyfriday|roverhet bot, need reviews on https://review.opendev.org/c/openstack/tripleo-ci/+/83194216:56
reviewbotI have added your review the to Review list16:56
rlandy|ruckfrenzyfriday|rover noted it was simliar to the issue you logged on fd03916:56
rlandy|ruckdasm: gate blocker now16:56
dasmchecking16:56
rlandy|ruckade_lee_: dasm: ^^ can we pick this up this afternoon?16:56
rlandy|ruckfrenzyfriday|rover is adding non-voting16:56
dasmsure16:57
rlandy|ruckoh I'm sorry - discussion going on on sec-interop16:57
rlandy|ruckdasm: ^^16:57
dasmjoined16:57
rlandy|ruckdasm: dwilde is on it - pls check with hi,16:58
rlandy|ruckhim16:58
* dviroel found a issue in compose-pinning - it is not enabling those repos on subnodes16:59
*** jpena is now known as jpena|off17:06
dasmdviroel: are you around? i need your help with custom build (convo on #sec-interop )17:07
rlandy|ruckvexxhost is hosed until weekend17:07
dviroeldasm: reading now17:08
dasm:/17:08
dasmthx17:08
rlandy|ruckrcastillo: sent you a new time for 1-1 - 6:30 UTC17:12
rcastillorlandy|ruck: just saw, thanks17:13
frenzyfriday|roverrlandy|ruck, soI am leaving for the day17:28
frenzyfriday|roverhttps://review.opendev.org/c/openstack/tripleo-ci/+/831942 is still running check jobs17:28
* frenzyfriday|rover afk17:31
dasmdviroel: does it look good? https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/83211617:35
dviroellooking17:39
rlandy|ruckfrenzyfriday|rover: thanks - may not be needed if dasm's patch works17:40
*** amoralej is now known as amoralej|off17:41
dviroeldasm: lgtm17:41
dasmsweet. testproject is already running17:44
rlandy|ruckdviroel: you had a question on fips on secondary node?19:28
dviroelrlandy|ruck: i am testing a change atm, i figure it out that subnode-1 is actually a nodepool node, right? might be the secondary19:29
dviroelthis https://github.com/openstack/tripleo-quickstart/blob/master/roles/tripleo-inventory/tasks/inventory.yml#L45-L4919:30
rlandy|ruckdviroel: correct19:30
rlandy|ruckmultinode gets you two nodepool nodes19:30
rlandy|ruckunlike ovb19:30
dviroelack - now i am testing zuul host-vars to disable fips in secondary node - lets see if will work19:30
rlandy|ruckso if you only want fips enabled on one node, you need to specify the host on those tasks to primary only19:30
rlandy|ruckdasm: how is your testing going on ipa?19:31
rlandy|ruckif it's looking positive, please -1 vote on ananya's patch to make it non-voting19:31
dasmstill running19:34
dasmchecking it's status19:34
dasmhttps://review.rdoproject.org/zuul/stream/d1dbc5ae45ff495a905f1f830c9541e9?logfile=console.log19:36
dasmit barely started19:36
dasmit's on > TASK [ipa-multinode : Temporarily downgrade openssl]19:47
dasmseems like it worked > echo 'Playbook run of multinode-standalone-ipa.yml passed successfully'20:08
dasmchecking for official test run results20:08
*** dviroel is now known as dviroel|out20:43
dasmrlandy|ruck: i just got test result. My change failed even before it could test kinit. I'm rerunning that, but it's gonna take over 1h for result.20:45
dasmMaybe we should merge "non-voting" patch?20:46
rlandy|ruckdasm: sure - let's clear the gate and revert when you have the better solution20:46
dasm+120:46
rlandy|ruckdasm: pls remove your -120:47
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/83194220:47
dasmdone20:47
rcastillogoing out to get my third covid shot21:44
rcastillohave a good weekend everyone!21:45
rlandy|ruckbye21:45
rlandy|rucksee you next week21:45
dasmo/21:47
* dasm leaves22:06
*** dasm is now known as dasm|off22:07
« Thursday, 2022-03-03 Index Monday, 2022-03-07 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!