Tuesday, 2022-09-06

*** ysandeep|out is now known as ysandeep00:51
ysandeepdviroel|out, rlandy|out hmmm, ideally we should have merged https://github.com/openstack-k8s-operators/cinder-operator/pull/10 before https://github.com/openshift/release/pull/3199700:52
ysandeepdviroel|out, what is broken?00:52
ysandeeprlandy|out, fyi.. I have merged https://github.com/openstack-k8s-operators/cinder-operator/pull/10 01:06
ysandeepchkumar|ruck, when you are in want to check if this is known? https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/872e186/job-output.txt 01:15
ysandeep~~~01:15
ysandeep2022-09-05 13:28:06.295452 | primary | TASK [modify-image : Mount image] **********************************************01:15
ysandeep2022-09-05 13:28:06.295689 | primary | Monday 05 September 2022  13:28:06 -0400 (0:00:04.745)       0:06:15.475 ******01:15
ysandeep2022-09-05 21:10:57.612258 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]01:15
ysandeep~~~01:15
ysandeepTengu, ^^ fyi.. fs035 still hitting other issues, I couldn't test your fix for ipv6.01:42
*** ysandeep is now known as ysandeep|afk03:36
chkumar|ruckysandeep|afk: thank you, we saw that yesterday arxcruz|rover is on it05:02
*** ysandeep|afk is now known as ysandeep05:29
ysandeepchkumar|ruck: thanks!05:29
akahatmarios, o/ https://review.rdoproject.org/r/c/testproject/+/44499 testproject is green for both jobs. 06:18
akahatrunning it again with force_periodic06:18
mariosakahat: thanks will check and then we can sync later06:20
mariosakahat: one request before we sync can you plese also post a test (e.g. tripleo-heat-templates stable/wallaby) with t06:23
mariosakahat: depends-on your patch to make sure we don'[t break the gate job06:23
mariosakahat: since you are tweaking the gate release files please06:23
akahatmarios, sure.06:29
mariosthank you akahat 06:30
Tenguysandeep: heya! wokay... anything I can help with in order to stabilize it?06:38
ysandeepTengu: thanks! we will let you know if we need help with it.06:40
Tengunp06:40
* Tengu goes back to his ansible things for httpd06:41
Tenguuho... chkumar|ruck heya! anything to do on my side for this error? https://zuul.opendev.org/t/openstack/build/b8710e23996048eab6386265e5229cf706:41
Tengumolecule fails to bootstrap due to missing gpg key or something, for the ubi9 image.06:41
chkumar|ruckTengu: I have seen this issue06:44
chkumar|ruckTengu: I need to workaround that, give me few mins, patch incoming06:45
Tenguchkumar|ruck: np - thanks! lemme know if I can help in anything.06:45
jm1o/06:58
mariosakahat: please see pvt when you have a sec07:28
mariosjm1: \o07:28
chkumar|ruckTengu: ysandeep: fyi https://bugs.launchpad.net/tripleo/+bug/198880807:31
chkumar|ruckarxcruz|rover++07:31
chkumar|ruckdpawlik: we are currently hitting this https://bugs.launchpad.net/tripleo/+bug/198880807:36
chkumar|ruckibm stress testing might impacted via this07:36
*** jpena|off is now known as jpena07:37
dpawlikindeed chkumar|ruck07:37
dpawlikthanks for info07:37
akahatmarios, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85602707:47
mariosthanks akahat 08:05
jm1frenzy_friday, dasm: downstream cockpit is running in regular mode since yesterday evening without issues09:19
frenzy_fridayjm1++ awesome!09:19
jm1frenzy_friday, dasm: or lets say without issues which are not expected ;)09:19
mariosgreat jm1 - was it all really because of the |default thing or also something else? 09:30
mariosdid you get to the bottom of the issue(s?)09:30
jm1marios: the default thing was unrelated. i just discovered that while running playbooks manually09:32
mariosjm1: so was the problem the new added dashboards from cre? something else?09:33
mariosand/or 09:33
marios;D09:33
mariosxor09:34
jm1marios: the code to update the cre dashboard was not even put inside the container hence it was not responsible09:34
jm1marios: the vm was in a broken state already with ansible-pull being blocked etc. since this issue started on 2022-08-03 it wasnt possible to debug the root cause as logs were already rotated etc09:35
jm1marios: frenzy_friday and i have some ideas what could have happened but its speculation. we applied some changes to ensure its not going to happen again09:36
jm1marios: for example, our ansible_pull.sh script will now aquire a lock before trying to run ansible-pull. this does not prevent us from manually running ansible-pull but one cannot prevent every operator error09:38
jm1marios: ^ btw this is the reason why putting some info about where our vms are, into bitwarden, could do more harm then good: one has to know how the system works before applying changes09:39
jm1marios: putting the info into the readme or other docs of our infra-setup code at least ensures that people are aware of the code before doing stuff manually09:41
mariosjm1: 12:39 < jm1> marios: ^ btw this is the reason why putting some info about where our vms are, into bitwarden, could do more harm then good: one has to know how the system works before applying changes09:41
mariosjm1: need both then. bitwarden as that is what we agreed to host our 'infra info' doc and in there you can point to the 'proper docs'09:42
mariosjm1: for example yesterday we relied on you and frenzy_friday to get onto the box09:42
jm1marios: when did we agree on bitwarden being our infra overview doc? imho it is meant as a replacement for our google doc, aka contains our passwords etc. it being an overview is a sideefffect of it hosting all our passwords and links to the systems09:44
frenzy_fridayyeah, there is a notes section on bitwarden. We can put stuff like Warning, pls check readme before you do anything on the server etc09:44
mariosjm1: not infra overview - like 'this ip address for promoter username foo password bar' this information09:44
mariosjm1: this is what we had in the google doc ^09:45
mariosjm1: and you can use the notes field to say "go read this before working here _link_" 09:45
jm1marios: ack, so we actually want the same ;) bitwarden as a pointer to our systems (and our code). looks like bitwarden needs some love09:47
mariosjm1: yes 12:42 < marios> jm1: need both then. bitwarden as that is what we agreed to host our 'infra info' doc and in there you can point to the 'proper docs'09:47
mariosjm1: no docs on bitwarden just the info i need to go and login to the cockpit vm and try see whatsup 09:48
jm1marios: yep, just wanted to confirm09:48
mariosjm1: and point to extra docs if we need them for that particular thing09:48
mariosk 09:48
jm1marios: dasm wants to talk about infra stuff in our community meeting. maybe you want to bring this topic (jira card to update bitwarden links and notes) up there?09:50
mariossure lets try remember to bring it together ;)09:50
jm1marios: will put a note into ci community call doc09:50
mariosack sounds good jm1 09:50
jm1marios: https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?view#Agenda ;)09:53
mariosarxcruz|rover: chkumar|ruck: why is there no more pcci section in the program call doc? was it removed/moved? 09:57
mariosknow anything about that ?09:58
mariosah looks like someone merged some cells probably 09:58
chkumar|ruckmarios: rlandy said our status is merged with prod chain update09:58
mariosack i see chkumar|ruck 09:59
mariosmakes sense 09:59
chkumar|ruckrlandy will share the template 09:59
mariosnow that i understand what i'm seeing 09:59
mariosthanks 09:59
jm1marios, chkumar|ruck, arxcruz|rover: mburns gave the overview about our prod chain in program call last week, as an experiment. whether he will do the same again this week?10:00
*** ysandeep is now known as ysandeep|afk10:03
mariosjm1: thanks 10:04
mariosjm1: i don't see why we would object to that seems fine by me10:04
marioswe still have to update that section and mburns can talk about it. we are available to jump in for clarifications as needed10:05
jm1marios: no objections from my side, i actually appreciate it. i was just wondering whether you have some updates on our process this week10:07
*** rlandy|out is now known as rlandy10:30
rlandyysandeep|afk: thanks - fixed now?10:32
rlandymarios: "why is there no more pcci section in the program call doc? was it removed/moved?"- we report up now with program call10:32
frenzy_fridayrlandy, hey, we have component reviews from the cre team now. I have added them to review list10:33
rlandyfrenzy_friday: great10:34
rlandyfrenzy_friday: we should probably look through them10:34
rlandywill ping you in a few10:34
frenzy_fridayonly 1 review 10:34
rlandychkumar|ruck: arxcruz|rover: hi10:34
frenzy_fridayI have commented 10:34
rlandyhow are things?10:35
arxcruz|rovernot so good on upstream 10:35
rlandychkumar|ruck: arxcruz|rover: time to sync quickly?10:35
arxcruz|roversure 10:35
chkumar|ruckrlandy: sure10:35
rlandyhttps://meet.google.com/jrm-jsza-xuu?pli=1&authuser=010:35
rlandyarxcruz|rover: chkumar|ruck: ^^10:35
mariosrlandy: yes there is now only production chain section - discussed with chkumar|ruck and jm1 already thanks 10:37
mariosrlandy: i was away when this was done10:37
*** ysandeep|afk is now known as ysandeep10:40
ysandeeprlandy, I didn't get what was broken.10:41
rlandyysandeep: jobs looked like they were hanging10:41
rlandywill show you in a bit10:42
ysandeepack10:42
ysandeepfyi.. If you post first PR, you need ok-to-test label, maybe that's why jobs were hanging.10:43
chkumar|ruckarxcruz|rover: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby.yaml#L4310:44
bhagyashrisrlandy, marios add into your review list https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427343 https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/42734511:02
bhagyashris when you have time 11:02
reviewbotI could not add the review to Review List11:02
reviewbotI could not add the review to Review List11:02
arxcruz|roverrlandy chkumar|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/4478811:04
chkumar|ruckarxcruz|rover: tp on https://review.rdoproject.org/r/c/testproject/+/44388 ibm cloud for cs8 wallaby passed11:05
chkumar|ruckarxcruz|rover: left a comment11:06
rlandybhagyashris++ - will review in a bit11:08
rlandymarios: hi - have time to touch base now?11:08
bhagyashrisack np11:08
rlandyfrenzy_friday: we are reporting on 17.1 on rhel-9 component jobs in http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline?orgId=111:09
rlandyso there must be some, right?11:09
frenzy_fridaylooking11:10
rlandylooks like jenkins jobs and vanilla standalone11:10
rlandyis that accurate?11:10
mariosrlandy: o/ sure11:13
rlandymarios: https://meet.google.com/yzh-aqjk-sjc?pli=1&authuser=011:13
frenzy_fridayrlandy, I think the jenkins jobs are running in pipelines and the standlones are from testprojs from CRE team11:14
frenzy_fridayexample: https://code.engineering.redhat.com/gerrit/c/testproject/+/42733911:14
*** dviroel|out is now known as dviroel11:19
ysandeepbhagyashris, I have left comments on the bug you opened for 17.1/811:25
ysandeepbhagyashris, We need to modify this condition to include 17.1 as well - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/817435/5/roles/standalone/templates/standalone_config.yaml.j2#611:25
bhagyashrisysandeep, ack thanks 11:29
bhagyashrisi will fix11:29
* bhagyashris stepping out for bit11:51
mariosthank you rlandy 11:55
rlandyfrenzy_friday: ok to meet in 15?11:56
rlandyactually 10 :)11:56
rlandyysandeep: dviroel: will miss podified meeting today - have clashing other meeting11:57
frenzy_fridayrlandy, sure11:57
rlandyysandeep: dviroel: would like to start taking a cut at kuttl tests11:57
rlandyysandeep: will touch base with you afterwards on that11:58
rlandyactually I can make the weekly sync11:58
ysandeepI don't have experience with kuttl tests yet, but we can discuss in our friday's call on how to progress there.12:01
rlandyok12:07
rlandywe have jon coming to friday's meeting12:08
rlandyfrenzy_friday: hi ... https://meet.google.com/wih-vorh-yia?pli=1&authuser=012:09
dviroelrlandy: it seems that we need to merge this before testing it - https://review.rdoproject.org/r/c/rdo-jobs/+/4473412:38
dviroelrlandy: pls review when free12:38
rlandydviroel: w+'ed12:41
dviroeltks12:41
rlandydviroel: I thought that should work with depends on13:06
rlandybut ok13:06
dviroelyeah13:08
pojadhavcommunity call in 5 mins : arxcruz|rover , rlandy, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chkumar|ruck , frenzy_friday, anbanerj,  dviroel, rcastillo, dasm, jm1, marios13:25
pojadhavhttps://meet.google.com/igc-nxwj-gws13:27
pojadhavplease update agenda : https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg from line @3113:29
*** dasm is now known as Guest211513:31
frenzy_fridayI will add ER and reviewbot to bitwarden14:02
*** Guest2115 is now known as dasm14:02
dasmo/14:02
rlandychkumar|ruck: arxcruz|rover: https://docs.engineering.redhat.com/display/PRODCHAIN/2022-09-05+Meeting+Notes14:05
rlandypls see if you can access that14:05
rlandyfor program call tomorrow14:05
chkumar|ruckrlandy: I can access it14:07
*** pojadhav is now known as pojadhav|afk14:10
rlandychkumar|ruck: ok - so we need that filled tomorrow morning14:10
rlandyour sections14:10
rlandy16.214:10
rlandy17 17.114:10
rlandywallaby train14:10
chkumar|ruckrlandy: will take care of that, thank you :-)14:11
rlandygreat14:11
chkumar|ruckrlandy: there is no section for wallaby and train14:13
rlandychkumar|ruck; hmm ... checking last week's doc14:16
rlandychkumar|ruck: pls slot those in where they were last week14:17
rlandywallaby in the 17.1 section14:17
chkumar|ruckrlandy: just below 16 and 17 we can put the status14:17
rlandyand train in 16.2 section14:17
rlandyyou'll have 17.1 on rhel8 and rhel914:18
chkumar|ruckack thanks !14:19
rlandyarxcruz|rover: ha - this passed: https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/81132dd/job-output.txt14:19
rlandypojadhav|afk++ on picking up the TC role so well14:25
arxcruz|roverrlandy 14:28
arxcruz|roverrlandy so the culprit was ibm node 14:28
rlandyarxcruz|rover: did you fix something?14:30
rlandythat is passed now?14:30
arxcruz|roverrlandy cs8 is running now on vexx instead of ibm 14:31
*** ysandeep is now known as ysandeep|afk14:32
chkumar|ruckarxcruz|rover: time for happy dance fs01 master passed https://review.rdoproject.org/zuul/stream/5c4e3dc11e964460838cdea45dfd79fd?logfile=console.log14:33
arxcruz|roverrlandy chkumar|ruck please https://review.opendev.org/c/openstack/tripleo-common/+/855564 14:33
chkumar|ruckonce logs will be available and get it merged14:33
chkumar|ruckarxcruz|rover++ for hunting it down :-)14:34
arxcruz|roverchkumar|ruck come on man, you did mostly of the finding :)14:34
chkumar|ruckarxcruz|rover: man team work :-)14:34
chkumar|ruckwe did it14:34
rlandychkumar|ruck: arxcruz|rover: should we get harold to vote on that?14:34
chkumar|ruckrlandy: yes14:35
arxcruz|roveryup14:35
rlandyarxcruz|rover++ chkumar|ruck++ for work on that14:35
rlandyo - I'll ping him to vote14:35
rlandydasm: hey - sent you invite for this afternoon14:36
dasmrlandy: accepted that. thanks14:36
rlandydasm: k - get ready to wrangle with bitwarden14:36
rlandyarxcruz|rover: chkumar|ruck: will merge once harold votes or later if he has not responded yet14:37
rlandyarxcruz|rover: pls remind me by your EoD14:38
arxcruz|roverok14:38
arxcruz|rovermy day never ends though :)14:38
arxcruz|roveri'm always around 14:38
dasmarxcruz|rover: do you know there is something called "not at work"?14:38
arxcruz|roverdasm i'm a computer addicted, so i just switch to my personal laptop and play, but still around if someone needs me 14:39
dasmarxcruz|rover: i've been like that few years back. now i'm more and more disconnecting in the afternoons. It's a nice thing to sit in the backyard or in a forest.14:40
arxcruz|roverdasm i'm not a nature man, but i do have my steam deck ready to play afterworks :)14:41
dasmarxcruz|rover: i wasn't a nature man too :D Just the life told me, it's so precious to not spend it glued to the screen14:41
arxcruz|roverdasm nah, i don't like to go out x(14:42
dasmarxcruz|rover: i feel you :) but believe me. It's fun there too :)14:43
chkumar|ruckarxcruz|rover: rlandy results posted on patch14:57
* rlandy looks14:57
chkumar|ruckrlandy: keep an eye on master component line tp https://review.rdoproject.org/r/c/testproject/+/4480114:59
rlandyok - will rerun what is needed15:01
jm1rlandy, marios: is it okay to discuss the ownership issue here in chat?15:05
rlandyjm1: probably better on internal15:05
rlandy#openstack-pcci15:06
rlandyas likely reference service access15:06
mariosjm1: middle of some stuff so might not be able to participate now 15:07
jm1rlandy, marios: my concern is not to accidentially expose credentials. i am worried because this is team internal orga stuff.. anyway i will write a mail15:07
mariosjm1: but go ahead obviously if you have folks to discuss with 15:07
chkumar|rucksee ya!15:08
chkumar|ruckarxcruz|rover: rlandy leave notes :-)15:08
jm1marios: i wanted to address what you said in our community call, so the target audience is you ;)15:08
jm1chkumar|ruck: have a nice evening!15:08
chkumar|ruckjm1: you too sir :-)15:08
rlandyjm1: sure - and thursday we can pick up the discussion as well15:09
rlandysorry of I cut you off15:09
rlandyjust wanted to give dasm time15:09
jm1rlandy: no worry, you were right, we wanted to talk about dasm's plans, not orga15:10
mariosjm1: perhaps better to discuss when there are more folks around i.e thu scrum 15:11
mariosjm1: but if you really want to talk to me ;) then we can sync tomorrow our EU morning? 15:11
mariosping me 15:11
mariosor better send me an invite jm1 15:11
rlandydviroel: fips job is still not running15:13
dasmjm1: email thread sounds good. it allows for async communication15:13
rlandy45 mins15:13
jm1marios: we are always running out of time in scrum, hence i write a mail ;)15:13
rlandyI don't think this has anything to do with merge15:13
rlandylooks like the node build is hanging15:13
rlandywe will need infra to see what is going on15:13
dviroelrlandy: yeah, raising that on rhos-ops15:14
rlandyarxcruz|rover: so https://review.opendev.org/c/openstack/tripleo-common/+/855564 is blocking master, right15:24
rlandywe could get another tripleo core merge later if needed15:24
arxcruz|roverrlandy promotion yeah 15:27
rlandywoohoo15:28
*** dviroel is now known as dviroel|lunch15:29
*** ysandeep|afk is now known as ysandeep15:43
mariosrlandy: as discussed earlier will re-send timesheet in few mins before leaving for the day thanks again15:48
rlandymarios: ack - have a good night15:48
* jm1 bbl15:48
mariospojadhav|afk: o/ when you are next at work please see email from me re meeting please add me thanks15:48
mariosthanks you too rlandy 15:48
rlandyjm1: thanks for sending your thoughts - will read through15:49
*** ysandeep is now known as ysandeep|out15:49
*** marios is now known as marios|out15:54
rlandylunch - brb16:02
pojadhav|afkmarios|out, done16:07
*** pojadhav|afk is now known as pojadhav16:07
*** dviroel|lunch is now known as dviroel16:26
*** jpena is now known as jpena|off16:31
rlandybhagyashris: +2'ed both https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/427343 and https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/42734517:23
rlandywil let marios have one last look17:23
rlandylet's merge that tomorrow17:23
rlandyat review time if not before17:23
rlandythanks for following this through17:24
rlandyarxcruz|rover: 17.1 reruns: https://code.engineering.redhat.com/gerrit/c/testproject/+/39871817:54
rlandychkumar|ruck: rerunning the failures on https://review.rdoproject.org/r/c/testproject/+/4480117:55
rlandyyou got three passing jobs :)17:55
* jm1 out for today, have a nice evening!19:01
dasmjm1[m]: o/19:01
rlandydasm: dviroel: review time?19:02
dasmwill join in a bit. on a phone call19:02
dviroelyes19:02
rlandyarxcruz|rover: still around?19:24
* dasm brb19:28
arxcruz|roverrlandy sorry, yes 19:40
rlandynp - answered on tripleo20:01
rlandywe got thr merge20:01
* dasm is back20:20
rlandyarxcruz|rover: late for you20:49
rlandyif you want to skip and promote, you and chkumar|ruck can decide on that tomorrow20:50
rlandychecking on periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-rhos-17.120:50
rlandychkumar|ruck: arxcruz|rover: left notes in rr hackmd21:02
*** dviroel is now known as dviroel|biab21:19
* dasm => offline22:56
dasmtake care team, i'm going afk22:56
*** dasm is now known as dasm|off22:56
*** dviroel|biab is now known as dviroel23:17
*** rlandy is now known as rlandy|out23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!