Wednesday, 2022-12-07

*** yadnesh|away is now known as yadnesh04:43
ysandeepgood morning o/04:51
akahatGood Morning o/04:52
akahatrlandy|out, ack. Will take a look.04:53
*** ysandeep is now known as ysandeep|brb05:07
yadneshhello, has anyone come across this issue https://paste.openstack.org/show/bHq74lA2bY68tZKYDjqJ/ ? or have i executed incorrect command ?05:42
ysandeep__chandankumar, hi o/ znc bouncer issues again05:49
*** ysandeep__ is now known as ysandeep05:51
ysandeepyadnesh, hi o/ I haven't tried manual deployment in while but I remember some breakages when we moved to whole_disk_image05:51
ysandeepCan you try adding --extra-vars whole_disk_images=False in the command05:51
yadneshysandeep, ack, will retry05:54
yadneshysandeep, still same06:01
ysandeepyadnesh, could you please share the complete quickstart console logs06:02
yadneshysandeep, i am deploying it locally06:02
ysandeepyes, I mean if you can copy the console log in a pastebin06:03
ysandeepwant to check from where it copies /home/stack/overcloud-full.qcow2 in your case06:04
ysandeepyadnesh, or i can jump in a tmux/tmate06:05
yadneshincluded: /home/deployer/.quickstart/tripleo-quickstart/roles/fetch-images/tasks/fetch.yml for 127.0.0.2 => (item={'name': 'overcloud-full', 'url': 'https://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/overcloud-full.tar', 'type': 'tar'})                                                               06:06
yadneshincluded: /home/deployer/.quickstart/tripleo-quickstart/roles/fetch-images/tasks/fetch.yml for 127.0.0.2 => (item={'name': 'ipa_images', 'url': 'https://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/ironic-python-agent.tar', 'type': 'tar'})  06:06
ysandeephttps://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/overcloud-full.tar location is present06:07
ysandeepyadnesh, before this try, when was the last time you tried quickstart successfully?06:09
yadneshysandeep, dont remember, trying this after a long time06:09
ysandeepyadnesh, ack, I will try to reproduce the issue06:09
ykarel_yadnesh, what is your virthost OS?06:16
ykarel_it could also happen if it's not centos 906:16
yadneshykarel_, rhel806:16
ykarel_ok then it's possible06:17
ykarel_as virt-customize do not work on rhel8 for rhel9 guest images06:17
yadneshykarel_, so i need centos9 or rhel9 on host ?06:17
ykarel_yadnesh, you either need rhel9 virthost or can use custom appliance to get virt-customize work06:18
*** marios is now known as marios|ruck06:19
ykarel_yadnesh, you can try out https://github.com/karelyatin/tripleo-quickstart, with that i was able to deploy it06:21
marios|ruck\o06:21
ysandeepykarel_, need tqe changes also, right? https://github.com/karelyatin/tripleo-quickstart-extras/commit/5aae668f16081180bcac15785464736e90b28d6c06:23
ykarel_ysandeep, yes that will be auto pulled06:23
ysandeepahh, okay because of git+https://github.com/karelyatin/tripleo-quickstart-extras/#egg=tripleo-quickstart-extras in quickstart-extras-requirements.txt06:24
ykarel_yes06:24
ysandeepykarel_, your these two patches just need more testing, right? 06:26
ysandeepIt would be good to propose and merge these patches.06:27
ykarel_ysandeep, mainly need to make that work without LIBGUESTFS_PATH and setup appliance automatically only if required06:27
ykarel_would need to download https://download.libguestfs.org/binaries/appliance/appliance-1.46.0.tar.xz and untar at /tmp and export LIBGUESTFS_PATH=/tmp/appliance before running quickstart06:28
ykarel_iirc you said someone will work on it06:29
ykarel_i will try to work on it in coming days06:30
ykarel_if it's not yet started06:30
ysandeepykarel_, yes we created a card and rcastillo was supposed to pick that at that time, but i think he got occupied ansible colleciton work due to priorties06:30
ysandeepwith* ansible06:31
ykarel_ack06:32
yadneshykarel_, where are oooq patches reviewed, i wanted to cherry pick that commit06:33
ykarel_yadnesh, that's my local fork, you can add remote and cherry-pick06:35
yadneshykarel_, ok06:35
ykarel_oooq patches are in gerrit but it's not pushed there yet06:35
yadneshykarel++ ysandeep++06:44
*** yadnesh is now known as yadnesh|afk07:46
jm1marios|ruck, chandankumar, ysandeep: o/ Do you know how to install python libraries which are used by modules in Zuul pre-run: playbooks?08:29
frenzy_fridaypojadhav, bhagyashris 0/ sorry, I still have some changes to do on the nodeset patch. I'll ping you when done08:31
pojadhavfrenzy_friday, yep08:33
marios|rucknot something i recall doing/looking at jm1 but how about making the python install the first pre-run play08:38
marios|ruckjm1: but probably you are hitting limitation on zuul ansible eg something similar to not being able to install and use a collection natively in a job (are you installing the libraries but they aren't being found?)08:40
*** jpena|off is now known as jpena08:41
bhagyashrisfrenzy_friday, ack08:41
jm1marios|ruck: Zuul uses the full ansible package which also includes ansible openstack collection. but Zuul does not install openstacksdk. So I can theoretically use aoc modules in Zuul pre-run playbooks but they require openstacksdk..08:46
jm1marios|ruck: so now i am wondering whether i can install openstacksdk somehow in a zuul pre-run playbook08:46
marios|ruckjm1: i was referring specifically to installing and using collection, during a job08:46
marios|ruckjm1: which we cant do natively (it was beign worked on but i haven't followed any updated there)08:47
marios|ruckjm1: 'natively' means with zuul/ansible tasks in the zuul plays directly, vs what we do which is inserting bash invocation in the middle where we install our collections08:47
jm1marios|ruck: yeah, zuul's ansible which runs all those zuul playbooks has aoc installed already, so its only openstacksdk which is missing08:48
*** yadnesh|afk is now known as yadnesh08:49
marios|rucksorry cant help more but sounds like you may be hitting a similar thing (like installing and configuring something for ansible, but during the zuul/ansible execution?) found that https://lists.zuul-ci.org/pipermail/zuul-discuss/2021-November/001752.html just fyi/incase useful09:03
marios|ruckjm1: ^^ 09:03
jm1marios|ruck: thank you! I will have a look :)09:04
jm1marios|ruck: oh wow, that was already one year ago 😳09:04
marios|ruckjm1: there was an update after that about a newer version of zuul allowing a new trusted execution environment (don't recall details, can try find if interested later)09:05
jm1marios|ruck: lucky me, the module is already available in ansible, so zuul can find the module. atm i am trying to install openstacksdk immediately before calling that module but it fails because pip cannot be found. so i have to install pip before installing openstacksdk before using the aoc module .....09:09
frenzy_fridaymarios|ruck, rlandy|out I am working on the upstream cockpit. It may go down10:58
marios|ruckfrenzy_friday: thx - fixing/adding something? 11:00
frenzy_fridaymarios|ruck, yeah, the puppet dashboard that kkula added has not been getting data since its addition. 11:01
frenzy_fridaydebugging that11:01
frenzy_fridayfolks, pls add to your review lists https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/436040 11:05
reviewbotI have added your review to the Review list11:05
*** dviroel|biab is now known as dviroel11:14
*** rlandy|out is now known as rlandy|rover11:14
rlandy|roverreview time?11:15
rlandy|roverakahat: review time pls11:17
*** yadnesh is now known as yadnesh|afk11:50
rlandy|roverpojadhav: frenzy_friday: does the 9.1 node seem to be working as expected?12:20
rlandy|roverpojadhav: frenzy_friday: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435796 is still running on a 9 node12:25
pojadhavrlandy|rover, checking12:26
rlandy|roverpojadhav: you'll also need the release file12:28
rlandy|roverpojadhav: maybe let's chat to get you started12:28
pojadhavrlandy|rover, sure12:28
rlandy|roverhttps://meet.google.com/cqa-icrv-kae?pli=1&authuser=012:29
rlandy|roverpojadhav: ^^12:29
rlandy|roverfrenzy_friday: ^^ you can join if around12:29
*** yadnesh|afk is now known as yadnesh12:34
frenzy_fridayrlandy|rover, sorry , was in a call with kkula12:40
rlandy|roverfrenzy_friday: np12:41
frenzy_fridaypojadhav, rlandy|rover releasefile: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/43590412:41
rlandy|roverjust chatted with pojadhav to start container builds12:41
frenzy_fridayforgot to put it on review list12:41
rlandy|roverfrenzy_friday: yeah  np - we found it12:41
rlandy|roveralso better to start on the check job - not periodic12:42
rlandy|roverbhagyashris: hey - you still blocked on 18?12:42
rlandy|roverbhagyashris: need any help there?12:42
frenzy_fridaydasm|off, jm1 when you are here need some help with the dashboard -  https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46047/5/ci-scripts/infra-setup/roles/rrcockpit/files/grafana/puppet-promotion-pipelines.dashboard.json12:43
pojadhavrlandy|rover, now it is taking correct label : upstream-rhel-9-113:01
rlandy|roverpojadhav: great13:02
rlandy|rovergood start13:02
jm1frenzy_friday: o/ whats the issue?13:06
frenzy_fridayjm1, http://dashboard-ci.tripleo.org/d/wplmDxdVz/puppet-promotion-pipelines?orgId=1&from=now-30d&to=now we do not see any data here: http://dashboard-ci.tripleo.org/d/wplmDxdVz/puppet-promotion-pipelines?orgId=1&from=now-30d&to=now 13:08
frenzy_fridayThe json looks correct, but when I enter edit mode in the panel the query is missing and datasource is wrong13:08
frenzy_fridayIf I manually edit the panel and add the correct query it works - then if I export the json it is the same as what we had already13:09
*** frenzy_friday is now known as frenzy_friday|food13:09
jm1frenzy_friday: can you get it working on our prod cockpit?13:09
jm1frenzy_friday|food: the issue i have is, that i dont know how the datasource is supposed to look like13:09
jm1frenzy_friday|food: or the whole dashboard in general. so what we should do is ask kkula to jump on a call with us and debug it live13:14
frenzy_friday|foodjm1, kkula and I were doing that this morning. We are able to edit the panels on the prod and get the data, but then if I export the json it is the same as what we already had. If I delete the locally made changes after that and get ansible pull to pull the dashboard from the repo again we dont see any data. 13:16
frenzy_friday|foodI will set up the dashboard on the prod and we can debug again13:16
frenzy_friday|food^ in some time :D13:17
bhagyashrisrlandy|rover, hi, i was using 9 nodeset not 9.1 let me change that and i not sure what to do with trunk13:21
chandankumarrlandy|rover: ysandeep dviroel checking this url13:21
bhagyashrisbecause we have rhel9 trunk https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp18/13:21
chandankumarclick on details of "continuous-integration/jenkins/pr-merge This commit looks good "13:21
chandankumarit points to downstream url13:21
chandankumarif we downstream zuul can vote back to upstream with url13:22
rlandy|roverbhagyashris: checking13:22
chandankumarthen we will have a better solution13:22
chandankumarwe donot need to worry about leaking secrets or content13:22
chandankumarfor third party person, url will give 40413:23
ysandeepchandankumar, sry which url? I missed the context13:23
chandankumarrlandy|rover: yadnesh dviroel url https://github.com/openstack-k8s-operators/osp-director-operator/pull/710#event-793890669413:23
chandankumaryadnesh: sorry13:23
chandankumarysandeep: ^^13:23
bhagyashrisrlandy|rover, and as discussed with alan he said dont run job until further notice13:23
pojadhavrlandy|rover, build container is failing now on "error creating build container: loading registries configuration "/etc/containers/registries.conf": mixing sysregistry v1/v2 is not supported".. is anything still am i missing to add ??13:32
rlandy|roverbhagyashris: so you're basically blocked?13:32
rlandy|roveruntil dlrn has 9.1?13:32
ysandeepchandankumar, yeah looks good, if we are not planning to build upstream community around operators.13:32
ysandeepwhich is the case right now.13:33
bhagyashrisrlandy|rover, yeah 13:33
bhagyashrisand waiting for Alan notice13:33
rlandy|roverchandankumar: so when I click on https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-ospk8s-osp-director-operator-multibranch/job/PR-710/1/display/redirect. I get 40413:34
rlandy|rovercorrect?13:34
ysandeepand I see the missing jenkins :) 13:35
chandankumarrlandy|rover: yes it is giving 404 as pr merged long time ago13:35
rlandy|roverbhagyashris: sorry - pls catch me up - did he give you an expected date?13:35
rlandy|roverchandankumar: where would that link go? logs inside the firewall?13:36
ysandeepbut is it okay to give internal link? we are giving ideas about hostname13:37
ysandeepimho.. it would be good to cross check those links are well protected behind firewalls13:38
bhagyashrisrlandy|rover, nope. Can we meet now?13:38
rlandy|roverbhagyashris: ok13:38
chandankumarrlandy|rover: the logs will open inside RH internal network13:38
bhagyashrisrlandy|rover, https://meet.google.com/vzm-nrah-qqf?authuser=013:38
rlandy|roverchandankumar: idk about internal links on public PR ... let's check on the sec of that13:39
chandankumarysandeep:  if we get confirmation about exposing internal links outside, then we can try that also13:39
rlandy|roverjason>13:39
jm1frenzy_friday|food: ok it would be great if we could debug together :) just ping me when you have set your env up :)13:46
*** pojadhav is now known as pojadhav|afk13:48
bhagyashrisamoralej|off, rlandy|rover https://sf.hosted.upshift.rdu2.redhat.com/logs/51/436651/6/check/periodic-tripleo-rhel-9-rhos-18-component-baremetal-promote-consistent-to-component-ci-testing/2e253b2/job-output.txt13:48
rlandy|roverpojadhav|afk: there is a BZ with details on how to fix that13:49
* ysandeep out, see everyone tomorrow o/13:58
*** ysandeep is now known as ysandeep|out13:59
dviroelpojadhav|afk: rlandy|rover: are we going to have retro today? It is on my calendar.14:09
rlandy|roverdviroel: ack ...14:24
rlandy|roverdviroel: 3pm UTC14:25
rlandy|roverTripleO CI Retrospective14:25
rlandy|roverakahat is running year end retro14:25
dviroelok14:36
* dviroel need to get some food14:36
dasm|offo/14:43
*** dasm|off is now known as dasm14:43
dasmpojadhav|afk: ping14:43
dasmfrenzy_friday|food: i see timeline with "A-series". I'm assuming it is not right.14:43
dasmfrenzy_friday|food: have you tried editing this view and checking how it is configured?14:44
rlandy|rovermarios|ruck: have the wallaby c9 failures in rerun14:46
*** pojadhav|afk is now known as pojadhav14:46
marios|ruckrlandy|rover: thanks14:46
pojadhavrlandy|rover, ack14:46
pojadhavdasm, pong14:46
pojadhavdasm, https://meet.google.com/xct-nazz-hyn14:46
pojadhavfolks: retro in 2 mins arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj,  dviroel, rcastillo, dasm, jm1 14:58
pojadhavrlandy|rover, marios|ruck ^^ 14:59
pojadhavmiro board : https://miro.com/app/board/uXjVP8YIpl0=/14:59
pojadhavdasm, ysandeep|out : retro15:02
pojadhavrcastillo, ^15:02
frenzy_friday|foodjm1, dasm - dashboard looks like this after changing directly on the prod: http://dashboard-ci.tripleo.org/d/wplmDxdVz/puppet-promotion-pipelines?orgId=1&from=now-30d&to=now This is the diff it generated (doesnt look like there are major syntax changes): https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4637015:31
frenzy_friday|foodI will re enable the ansible pull now and see if my changes disappear15:32
dasmfrenzy_friday|food: ack15:33
*** frenzy_friday|food is now known as frenzy_friday15:34
* frenzy_friday always forgets to remove |food15:34
dasmfrenzy_friday: don't remove your food :)15:38
jm1frenzy_friday: ack, cool!15:52
rlandy|roverakahat: marios|ruck: hmmm ...15:56
rlandy|rovercontainer login fail15:57
rlandy|roverhttps://review.rdoproject.org/r/c/testproject/+/3625415:57
dpawlikfolks, sorry for disturbing. I got a question. if you are checking CI job logs from our infra, did someone see any tracebacks in dmesg / messages  on Centos 9 stream?15:57
rlandy|rovernothing running on https://review.rdoproject.org/zuul/status now15:58
akahatwe need to revert patch.. and we cna't test it without merge :(15:58
rlandy|rover2022-12-07 15:57:08.958204 | 15:58
rlandy|rover2022-12-07 15:57:08.958340 | TASK [Set vars for quay login]15:58
rlandy|rover2022-12-07 15:57:09.026002 | primary | Output suppressed because no_log was given15:58
rlandy|rover2022-12-07 15:57:09.040212 | 15:58
rlandy|rover2022-12-07 15:57:09.040351 | PLAY RECAP15:58
rlandy|rover2022-12-07 15:57:09.040411 | primary | ok: 2 changed: 1 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 015:59
rlandy|roverakahat: yeah - pls put in the reverts15:59
rlandy|rovermarios|ruck: ^^15:59
akahatrlandy|rover, marios|ruck https://review.rdoproject.org/r/c/config/+/4621815:59
rlandy|roverakahat: do we need reverts for the others?16:00
akahatrlandy|rover, no. We don't need other reverts.16:00
marios|ruckrlandy|rover: akahat: looking16:00
rlandy|rovermarios|ruck: akahat: hmmm ... no way to know other than merge revrt16:01
marios|ruckakahat: when we're ready to try again we can do it in the morning 16:01
marios|ruckakahat: rlandy|rover: ack lets revert16:01
rlandy|rovermarios|ruck: akahat: ok - w+'ed16:02
rlandy|roverlet's see if it helps16:02
rlandy|roverfrenzy_friday: 1-1?16:02
akahatrlandy|rover, marios|ruck ack. Will take a look.16:02
akahatrlandy|rover, thanks !!16:02
marios|ruckakahat: but what about those then we need revert there too no? https://review.rdoproject.org/r/c/rdo-jobs/+/46181 16:04
marios|ruckakahat: rlandy|rover: and criteria https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46345 ? ^ 16:04
akahatmarios|ruck, rlandy|rover not needed to revert other patches16:05
akahattestporject is passed: https://review.rdoproject.org/r/c/testproject/+/4591016:06
akahatand it's pushing to quay.rdoproject.org.16:06
marios|ruckakahat: yes, but those patches depends-on the one we are going to revert no? so they will break now?16:06
akahatmarios|ruck, Those are job related changes. reverted patches is the only one which modifies in the playbook.16:07
akahatwe have cross fingers but let's see what the zuul says.16:08
*** dviroel is now known as dviroel|lunch16:15
rlandy|roverfrenzy_friday: pojadhav: https://github.com/containers/image/issues/105416:16
rlandy|roverfrenzy_friday: pojadhav: https://bugs.launchpad.net/tripleo/+bug/190965816:16
rlandy|roverdviroel|lunch: ^^ pls remind me of the var to avoid v1 registry16:20
rlandy|roverpojadhav: frenzy_friday: dviroel|lunch; found it ...16:28
rlandy|rover        - when:16:28
rlandy|rover            - build_container_images|default(false)|bool16:28
rlandy|rover            - configure_registries_v1|default(true)|bool16:28
rlandy|roverconfigure_registries_v1 - set that to false16:28
rlandy|roverin job def16:29
marios|ruckrlandy|rover: need sthing? don't have something to handoff today this end16:31
rlandy|rovermarios|ruck: should be ok16:37
rlandy|roverthink jobs are running again now16:37
rlandy|roverthat the revert has merged16:37
rlandy|roverwill try the wallaby c9 rerun again16:38
marios|ruckk will pickup in the morning rlandy|rover o/16:40
*** marios|ruck is now known as marios|out16:40
frenzy_fridayrlandy|rover, thanks16:43
rlandy|roverpojadhav: dasm: pls cancel scrum tomorrow16:47
rlandy|roverwe'll just have planning16:47
dasmpojadhav: can you do that? i don't have superpowers for that.16:48
*** yadnesh is now known as yadnesh|away17:14
*** dviroel|lunch is now known as dviroel17:16
*** jpena is now known as jpena|off17:31
rlandy|roverdviroel: https://issues.redhat.com/browse/OSP-20021 - is that still under investigation?17:56
dviroelrlandy|rover: oh, i have the answers for that, I will add my comments and we can close18:06
dviroelrlandy|rover: I think we can close https://issues.redhat.com/browse/OSP-2002118:31
rlandy|roverdviroel: thanks - sorry - was on meeting19:11
rlandy|roverFYI - all tox jobs are broken - see openstack-discuss "Tox 4.0.0 breaking changes"19:45
dviroelrlandy|rover: ack - will zuul-jobs fix work for all our jobs?19:56
rlandy|roverdviroel: patch is not yet merge20:03
rlandy|roverwe will have to see the damage after that20:03
rlandy|roverfollowing thread20:03
rlandy|roverthey are still working on it - on #opendev20:04
dviroelhum, fix is on ensure-tox role20:06
dviroelhttps://review.opendev.org/c/zuul/zuul-jobs/+/866928/2/roles/ensure-tox/tasks/main.yaml20:06
*** dviroel is now known as dviroel|brb20:14
*** dviroel|brb is now known as dviroel|afk20:47
* jm1 out for today20:59
dasmo/22:11
*** dasm is now known as dasm|off22:11
*** rlandy|rover is now known as rlandy|out22:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!