| *** rlandy is now known as rlandy|out | 02:47 | |
| *** rcastillo|rover_ is now known as rcastillo | 03:56 | |
| *** chandankumar is now known as chkumar|ruck | 05:02 | |
| dpawlik | dasm|off: hey, I'm starting to like influx, when I see such error :) | 05:06 |
|---|---|---|
| *** ysandeep|out is now known as ysandeep | 05:22 | |
| ysandeep | happy friday tripleo-ci o/ | 05:34 |
| ykarel | Error: invalid policy in \"/etc/containers/policy.json\": Unknown key \"keyPaths\" | 05:35 |
| ykarel | ok it's known | 05:35 |
| ykarel | https://bugs.launchpad.net/tripleo/+bug/1988500 | 05:36 |
| sshnaidm | ykarel, huh, just was talking about it yesterday :) | 06:59 |
| ykarel | sshnaidm, so you noticed it faster then upstream CI :) | 06:59 |
| sshnaidm | ykarel, yeah, because it broke before on RDO: https://review.rdoproject.org/zuul/builds?pipeline=github-check&skip=0 | 07:01 |
| sshnaidm | not sure how that happened | 07:01 |
| sshnaidm | probably image building delay | 07:01 |
| ykarel | mm but images i don't think have containers rpms installed | 07:02 |
| ykarel | okk it's because these jobs using centos mirrors | 07:03 |
| ykarel | https://logserver.rdoproject.org/68/468/36e6c58626941415ee7e715cdd836b99ac16fbfe/github-check/tripleo-ci-centos-9-standalone/2bf30d2/logs/undercloud/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz | 07:03 |
| ykarel | and opendev mirrors refreshed last night, so they hit the issue late | 07:03 |
| sshnaidm | ykarel, that explains | 07:07 |
| sshnaidm | but why do these jobs not use rdo mirros..? (afs?) | 07:07 |
| dpawlik | you can check the date when it was rebuild: https://nb01.opendev.org/images/ | 07:08 |
| dpawlik | for c9 seems that it was done today | 07:08 |
| dpawlik | the rdo mirror is "binding" to opendev mirror. If issue is on opendev we also have | 07:09 |
| ykarel | sshnaidm, using mirrors relies on /etc/ci directory created by mirror-info-fork role in rdo | 07:11 |
| ykarel | and mirror-info role in upstream | 07:11 |
| sshnaidm | ah, probably it has only mirror-info | 07:12 |
| sshnaidm | because it's same job exactly as in upstream | 07:12 |
| *** arxcruz is now known as arxcruz|rover | 07:12 | |
| ykarel | i don't see that role is executed in those jobs | 07:12 |
| arxcruz|rover | chkumar|ruck hello | 07:12 |
| arxcruz|rover | good morning | 07:12 |
| chkumar|ruck | arxcruz|rover: hello good morning | 07:12 |
| arxcruz|rover | chkumar|ruck want to sync? or wait for ronelle? | 07:13 |
| *** jpena|off is now known as jpena | 07:36 | |
| *** ysandeep is now known as ysandeep|afk | 09:49 | |
| *** rlandy|out is now known as rlandy | 10:37 | |
| rlandy | arxcruz|rover: chkumar|ruck: hi - anything we need to sync about? | 10:38 |
| chkumar|ruck | rlandy: o/ good morning | 10:39 |
| chkumar|ruck | rlandy: waiting on this https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 to clear our gate and master line | 10:39 |
| arxcruz|rover | rlandy not from my side, rerunning wallaby c8 and train | 10:39 |
| rlandy | arxcruz|rover: yesterday - only fs001 was out on train | 10:39 |
| arxcruz|rover | rlandy yup | 10:39 |
| arxcruz|rover | rerunning it | 10:39 |
| rlandy | if we had two diff set of tempest failures we can skip promote there | 10:40 |
| rlandy | we should do that before your EoD | 10:40 |
| rlandy | arxcruz|rover: got one other CIX for you ... | 10:40 |
| chkumar|ruck | rlandy: https://bugs.launchpad.net/tripleo/+bug/1988514 | 10:41 |
| chkumar|ruck | gate blocker | 10:41 |
| chkumar|ruck | https://bugs.launchpad.net/tripleo/+bug/1988500 | 10:41 |
| rlandy | arxcruz|rover: https://trello.com/c/3p8i2YdZ/2639-cixlp1982874tripleociproa-testcreateobjectwithtransferencoding-is-failing-on-tripleo-jobs | 10:41 |
| rlandy | chkumar|ruck: do we have a workaround? | 10:42 |
| chkumar|ruck | rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 will clear everything | 10:42 |
| chkumar|ruck | it is as usual centos stream mess | 10:42 |
| chkumar|ruck | they updated the config but forgot to put the file | 10:43 |
| chkumar|ruck | and shipped the code | 10:43 |
| chkumar|ruck | it broke our stuff | 10:43 |
| rlandy | chkumar|ruck: ok - thanks - voted there - pls merge when ready | 10:43 |
| chkumar|ruck | waiting for zuul +1 | 10:43 |
| rlandy | arxcruz|rover: hi - pls see christian's comments on that card | 10:43 |
| rlandy | arxcruz|rover: pls review and let's decide how to go on this one | 10:45 |
| rlandy | ysandeep|afk: chkumar|ruck: forwarding you eoghan's response | 10:46 |
| rlandy | we are a no on vienna | 10:46 |
| rlandy | bhagyashris: hi - you around? | 10:48 |
| rlandy | let's touch base on 17.1 on 8 | 10:48 |
| bhagyashris | rlandy, yes | 10:48 |
| bhagyashris | standalone is passing now | 10:48 |
| bhagyashris | running other jobs | 10:48 |
| rlandy | bhagyashris: k ... https://meet.google.com/zgp-qyas-dxv?pli=1&authuser=0 | 10:48 |
| arxcruz|rover | rlandy sorry, i was lunching, i reply there, i'll check with gman what is the best action, maybe we should not update the urllib3 that might break other things | 10:50 |
| arxcruz|rover | so better patch tempest, i'll create the patch | 10:50 |
| chkumar|ruck | rlandy: thank you :-) | 10:52 |
| rlandy | arxcruz|rover++ thanks - let's close that out | 10:56 |
| rlandy | chkumar|ruck: arxcruz|rover: downstream promos needs some love - as well as the tripleo component on rhos-17 | 10:57 |
| rlandy | since ovb is functional again | 10:57 |
| rlandy | chkumar|ruck: arxcruz|rover: the rest of the downstream components are cleaned up now | 10:57 |
| rlandy | which is good | 10:57 |
| rlandy | chkumar|ruck: arxcruz|rover: I'm kicking a rerun on 16.2 failed jobs now | 10:58 |
| chkumar|ruck | ok | 10:59 |
| arxcruz|rover | ok | 10:59 |
| rlandy | we should stagger the 17 and 17.1 reruns for ovb | 10:59 |
| rlandy | since bhagyashris is also running ovb jobs now for 17.1 on 8 | 11:00 |
| arxcruz|rover | periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train passes | 11:03 |
| rlandy | arxcruz|rover: woohoo | 11:03 |
| rlandy | so you'll be clean on upstream promos | 11:03 |
| *** ysandeep|afk is now known as ysandeep | 11:08 | |
| dviroel | o/ | 11:22 |
| *** carloss is now known as carloss|afk | 11:28 | |
| rlandy | ysandeep; chkumar|ruck: I have a few minutes now if you want to discuss earlier? | 11:40 |
| ysandeep | rlandy, sure | 11:40 |
| ysandeep | chkumar|ruck, you around? | 11:41 |
| rlandy | forwarded one last email | 11:43 |
| ysandeep | thanks, received :) lets meet when chkumar|ruck is back | 11:46 |
| rlandy | yep | 11:46 |
| chkumar|ruck | ysandeep: back | 11:49 |
| ysandeep | chkumar|ruck, rlandy meet.google.com/pbx-jpyt-uht | 11:50 |
| arxcruz|rover | brb in 45 min | 11:54 |
| chkumar|ruck | rlandy: https://zuul.opendev.org/t/openstack/status#855587, | 12:21 |
| chkumar|ruck | rlandy: tripleo-ci-centos-9-undercloud-containers seems to be stuck https://zuul.opendev.org/t/openstack/stream/52ffc5d44ebd456e94bbbab7b740a1c0?logfile=console.log | 12:21 |
| chkumar|ruck | 2022-09-02 10:19:19.112661 | TASK [Update all installed packages after new repos are setup] | 12:21 |
| chkumar|ruck | rlandy: we need to kill that job | 12:22 |
| chkumar|ruck | and or recheck it | 12:22 |
| chkumar|ruck | rlandy: will I update the review? | 12:22 |
| chkumar|ruck | or ask infra to do a force merge | 12:22 |
| chkumar|ruck | on update I is going to take 3+ hr to complete | 12:23 |
| rlandy | ok | 12:25 |
| *** dasm|off is now known as dasm | 13:00 | |
| dasm | dpawlik | dasm|off: hey, I'm starting to like influx, when I see such error :) | 13:00 |
| dasm | dpawlik: that was unexpected :) but entertaining | 13:01 |
| dasm | jm1[m]: monday doesn't work for me. Labor Day|Day off. I'm online now if you have some time to chat. | 13:01 |
| *** carloss|afk is now known as carloss | 13:03 | |
| dasm | jm1 ^ | 13:03 |
| dasm | jm1 i might look through current tasks to group them, and what can and cannot be done in 2 weeks. After that, we can discuss what's gonna be the top priority. | 13:06 |
| chkumar|ruck | ysandeep: can you add https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 +w and +2 so that infra can move it at the op of the queue | 13:11 |
| ysandeep | chkumar|ruck, looking | 13:11 |
| ysandeep | chkumar|ruck, done | 13:13 |
| jm1 | dasm: in meetings, will ping you later | 13:29 |
| chkumar|ruck | rlandy: arxcruz|rover please keep an eye on this patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 to clear gate | 14:05 |
| chkumar|ruck | once merges please reply to the gate blocker email | 14:05 |
| arxcruz|rover | yes sir | 14:05 |
| chkumar|ruck | see ya! | 14:05 |
| chkumar|ruck | have a nice weekend ! | 14:06 |
| rlandy | chkumar|ruck: thanks | 14:07 |
| rlandy | have a great weekend | 14:07 |
| *** jpena is now known as jpena|off | 14:07 | |
| rlandy | arxcruz|rover: pls ping when you are EoD | 14:10 |
| rlandy | will take over from there | 14:10 |
| arxcruz|rover | ok | 14:10 |
| rlandy | if patch has not merged | 14:10 |
| rlandy | yet | 14:10 |
| rlandy | sorry - rotating meetings | 14:10 |
| rlandy | until 5:30 utc | 14:10 |
| *** ysandeep is now known as ysandeep|out | 14:34 | |
| jm1 | dasm: brief sync? | 14:56 |
| dasm | jm1: yup | 15:04 |
| dasm | jm1: https://meet.google.com/epy-ofdn-uin | 15:04 |
| *** dviroel is now known as dviroel|lunch | 15:13 | |
| jm1 | dasm: really appreciate that you will focus on our infra :) | 15:36 |
| * jm1 out for today, have a nice (long) weekend 🥂 | 15:38 | |
| rlandy | arxcruz|rover: hey - need help with anything? | 16:04 |
| arxcruz|rover | rlandy no, i'm just waiting for chandan's patch get merged | 16:08 |
| rlandy | k | 16:08 |
| arxcruz|rover | rlandy patch merged, i'm sending an email in response | 16:16 |
| arxcruz|rover | done | 16:17 |
| rlandy | arxcruz|rover: perfect - thanks | 16:27 |
| rlandy | downstream reruns are in progress | 16:27 |
| rlandy | arxcruz|rover: so we need to rerun master/wallaby jobs? | 16:28 |
| arxcruz|rover | rlandy yes, i'll put it to run | 16:29 |
| rlandy | arxcruz|rover: can you look at what's going wrong with https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1-cs8&skip=0 | 16:29 |
| rlandy | container builds | 16:29 |
| rlandy | timed out last two days | 16:29 |
| arxcruz|rover | ok | 16:29 |
| *** dviroel|lunch is now known as dviroel | 16:41 | |
| arxcruz|rover | rlandy i'll continue to check on monday regarding the push jobs | 17:01 |
| arxcruz|rover | i think it might be a problem with disk space | 17:01 |
| arxcruz|rover | since i'm seeing failing because it can't open the log file | 17:01 |
| arxcruz|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-8-quay-push-wallaby/21798d5/logs/build.log | 17:01 |
| rlandy | arxcruz|rover: thanks - I;m out then | 17:01 |
| rlandy | but we need to get this sorted | 17:01 |
| rlandy | thank you | 17:01 |
| arxcruz|rover | 2022-09-02 14:12:03 | 2022-09-02 14:12:03.485 39806 ERROR tripleo_common.image.builder.buildah.BuildahBuilder FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/container-builds/76cdb7b4-6477-4343-86c2-e420ccc6a236/base/os/nova-base/nova-compute/nova-compute-build.log' | 17:01 |
| rlandy | can you add a note to rr hackmd so chkumar|ruck knows you're looking at it? | 17:02 |
| arxcruz|rover | but rlandy it seems chandan saw the same on c9 and have a patch for it https://review.opendev.org/c/openstack/tripleo-quickstart/+/855552 | 17:06 |
| arxcruz|rover | need to add on his patch c8 as well | 17:06 |
| arxcruz|rover | at least the failure seems similar | 17:06 |
| rlandy | k | 17:08 |
| rlandy | dasm; hey | 17:08 |
| rlandy | putting in a sync meeting for infra | 17:08 |
| rlandy | 6pm utc ok? | 17:09 |
| rlandy | lunch brb | 17:13 |
| dasm | rlandy: 6pm works for me | 17:47 |
| rlandy | k | 17:50 |
| rlandy | dasm: https://meet.google.com/utv-xfkz-hot?pli=1&authuser=0 | 18:01 |
| rlandy | 16.2 promoted | 18:43 |
| dviroel | yey | 19:00 |
| dviroel | rlandy: so, wrt fips image on rdo | 19:01 |
| dviroel | rlandy: patch merged, nodepool built the image and uploaded the image | 19:01 |
| rlandy | sounds good | 19:01 |
| dviroel | I created a patch with nodesets | 19:01 |
| rlandy | dviroel: great - need it merged? | 19:02 |
| dviroel | https://review.rdoproject.org/r/c/rdo-jobs/+/44734 | 19:02 |
| dviroel | no | 19:02 |
| dviroel | it should work with depends-on right? | 19:02 |
| rlandy | correct | 19:02 |
| rlandy | you can test with depends | 19:02 |
| dviroel | https://review.rdoproject.org/r/c/testproject/+/44652/5/.zuul.yaml | 19:02 |
| dviroel | NODE_FAILURE | 19:02 |
| dviroel | am I missing a step? | 19:02 |
| dviroel | in this process | 19:03 |
| rlandy | one sec | 19:03 |
| dviroel | will double check if image was uploaded | 19:04 |
| rlandy | sorry - back | 19:11 |
| * rlandy looks | 19:11 | |
| rlandy | hmmm ... no logs | 19:13 |
| rlandy | dviroel: can I rekick | 19:14 |
| dviroel | yes | 19:17 |
| rlandy | job is still queued | 19:18 |
| dviroel | ack | 19:21 |
| rlandy | dviroel: is that what happened last time? | 19:23 |
| rlandy | or is zuul thinking about it more now? | 19:23 |
| dviroel | rlandy: yes, same thing | 19:23 |
| dviroel | then I stop watching, and it failed :) | 19:24 |
| rlandy | you'd have to ping rhos-ops | 19:24 |
| rlandy | maybe they can see the error | 19:24 |
| rlandy | I don't see it on our warnings | 19:24 |
| dviroel | ack, will do | 19:24 |
| rlandy | looks like it's trying to provision the node and failing | 19:24 |
| rlandy | but I can't see why | 19:24 |
| rlandy | we don't have that access | 19:25 |
| rlandy | dviroel: looks like node hang to me | 19:27 |
| rlandy | but I can't see why | 19:28 |
| dviroel | how is still avaible now? | 19:31 |
| dviroel | who* | 19:31 |
| * dviroel needs coffee | 19:31 | |
| dasm | rlandy: dviroel seems like yesterday's rr patch to mitigate issues with querying zuul is not working. dpawlik sent me some message last night. | 19:31 |
| dasm | i just checked my internal irc | 19:31 |
| dasm | i should've done that earlier | 19:32 |
| dviroel | dasm: the workaroung worked? but doesn't solve the issue? | 19:33 |
| dasm | dviroel: yes | 19:33 |
| dviroel | ackl | 19:33 |
| dviroel | needs further investigation, maybe we need to debug on a dev environment | 19:33 |
| dasm | the issue: we're hitting zuul very hard, every 30mins, in parallel | 19:34 |
| dasm | do we have a zuul dev env? | 19:34 |
| dviroel | cockpit dev env | 19:34 |
| dviroel | but you can create a zuul dev env too | 19:34 |
| dasm | we're killing zuul, so only rhos-ops can tell us if that's affecting them | 19:34 |
| dasm | we might need to do so | 19:34 |
| dviroel | the problem will be to populate it | 19:34 |
| dviroel | or not, you can create script to trigger a bunch of noop jobs | 19:35 |
| dviroel | dasm: you can used zuul quickstart tutorial | 19:35 |
| dasm | actually i don't need a zuul | 19:35 |
| dasm | i can even have something else, just being queried. | 19:36 |
| rlandy | dviroel: hmm ... now we are at 21 mins | 19:36 |
| dviroel | +1 | 19:36 |
| dasm | like simple http service should be enough | 19:36 |
| dviroel | rlandy: who is around in rhos-ops? | 19:36 |
| rlandy | dasm: can we stop our collection | 19:36 |
| rlandy | dviroel: nhicher | 19:36 |
| dviroel | ack | 19:37 |
| rlandy | tristan | 19:37 |
| rlandy | both in canada | 19:37 |
| dasm | rlandy: yes, we can remove it from being collected. it will render cockpit unusable | 19:37 |
| dasm | rlandy: although, there is one more thing we might do. we might introduce random delay in the ruck_rover script itself, to avoind querying zuul | 19:38 |
| rlandy | dasm:" we're hitting zuul very hard, every 30mins, in parallel" | 19:39 |
| rlandy | can we split that ^^? | 19:39 |
| dasm | rlandy: technically yes. by separating array of commands into different tasks. | 19:40 |
| rlandy | arxcruz|rover: ok - all downstream lines promoted | 19:40 |
| rlandy | dasm: how hard is that? workable? | 19:42 |
| dasm | rlandy: lemme try doing that | 19:43 |
| dviroel | dasm: dev cockpit seems to be working | 19:44 |
| dviroel | are you using it to test? | 19:44 |
| dasm | dviroel: i got a message: "seems that patch is not working as expected ;/" nothihg less, nothihg more. | 19:45 |
| dasm | it's a separate thing from cockpit | 19:45 |
| dasm | telegraf agent has something like | 19:46 |
| dasm | >collection_offset: Collection offset is used to shift the collection by the given interval. This can be be used to avoid many plugins querying constraint devices at the same time by manually scheduling them in time. | 19:46 |
| dasm | but i'm not sure yet if that applies to input.exec array | 19:46 |
| rlandy | hmm - not very expressive | 19:48 |
| dasm | not really | 19:48 |
| dviroel | btw, how many instances are running at the same time? | 19:48 |
| dasm | dviroel: what do you mean? | 19:48 |
| dviroel | downstream, upstream | 19:49 |
| dasm | i believe just these two | 19:49 |
| dviroel | and the dev one that I just saw | 19:49 |
| dasm | if there is one, yes. then it's 3 | 19:49 |
| dviroel | yeah, make zuul life worst | 19:50 |
| dasm | that actually might be it | 20:00 |
| dasm | > collection_jitter: Overrides the collection_jitter setting of the agent for the plugin. Collection jitter is used to jitter the collection by a random interval. | 20:00 |
| dviroel | ok, beer time. Have a great weekend team o/ | 20:55 |
| dasm | dviroel: o/ have a good one | 20:55 |
| *** dviroel is now known as dviroel|out | 20:56 | |
| dasm | rlandy: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44736 cc dviroel|out | 21:03 |
| dasm | that's the ultimate solution for telegraf. cc dpawlik | 21:03 |
| dasm | in the future, I'm gonna rework it, to make it single rr command, to avoid bunch of duplication. | 21:03 |
| dasm | But for now, that's something which is gonna introduce required jitter in the command execution. | 21:04 |
| * rlandy looks | 21:16 | |
| rlandy | dasm: going to vote there | 21:17 |
| rlandy | but will wait in dpawlik before merge | 21:17 |
| rlandy | dasm; maybe ping him and chkumar|ruck to merge on monday since us is out | 21:17 |
| dasm | indeed | 21:17 |
| rlandy | looks like a good start | 21:18 |
| dasm | dpawlik: chkumar|ruck can you check that one above? to me it's logical. the same way like 2 prior changes ;) | 21:18 |
| rlandy | maybe we rethink in the longer term | 21:18 |
| dasm | definitely | 21:18 |
| dasm | "one small step for TripleO CI team, one giant leap for rhos ops (zuul) team" | 21:19 |
| dasm | ;) | 21:19 |
| * rlandy out | 21:22 | |
| * dasm signing off for today | 21:32 | |
| *** dasm is now known as dasm|off | 21:32 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!