| *** pojadhav|afk is now known as pojadhav | 03:12 | |
| *** yadnesh|away is now known as yadnesh | 04:03 | |
| *** ysandeep is now known as ysandeep|ruck | 04:29 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|brb | 05:34 | |
| *** ysandeep|ruck|brb is now known as ysandeep|ruck | 05:53 | |
| marios | o/ | 06:16 |
|---|---|---|
| ysandeep|ruck | good morning o/ | 06:17 |
| ysandeep|ruck | folks o/ Anyone know if container build job is expected to honor depends-on? I | 06:20 |
| ysandeep|ruck | We were doing a test with depends-on here: https://review.rdoproject.org/r/c/testproject/+/46132 and in the result we don't see build-test-packages triggered at all. | 06:21 |
| ysandeep|ruck | https://logserver.rdoproject.org/32/46132/4/check/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/3896461/job-output.txt | 06:21 |
| marios | ysandeep|ruck: o/ i had this same doubt recently ... doesn't seem to be a branch issue (master @ https://review.opendev.org/c/openstack/tripleo-common/+/865116) | 06:31 |
| marios | ysandeep|ruck: digging a bit will let you know if find something... worst case can try hold the node and see/check | 06:31 |
| marios | ysandeep|ruck: if it isn't then we missed something and it should anyway | 06:32 |
| ysandeep|ruck | build-test-package role was not trigger at all, I guess we need to follow where we trigger that role | 06:32 |
| marios | ysandeep|ruck: maybe we need to add it like we have there (for undercloud for example) https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-setup/tasks/main.yml#L33 | 06:33 |
| ysandeep|ruck | yes we need to add it | 06:36 |
| ysandeep|ruck | Looks like is in container build job we do everything in build-container role from tripleo-ci repo: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/main.yaml | 06:44 |
| Tengu | ysandeep|ruck: heya! don't we have repositories while running the pre.yaml? https://zuul.opendev.org/t/openstack/build/424efb17591e4190af5ed98c4316f535 sounds weird. | 07:40 |
| ysandeep|ruck | we may need to include our after repo-setup role | 07:41 |
| ysandeep|ruck | Tengu, I think local resolver will not help based on my tests so far, please check: https://bugs.launchpad.net/tripleo/+bug/1997202/comments/6 | 07:41 |
| ysandeep|ruck | added one more observation: https://bugs.launchpad.net/tripleo/+bug/1997202/comments/7 podman/buildah/netvark are bumped recently - I wonder if they affected us | 07:42 |
| Tengu | hmm. and of course it's not easy to reproduce? | 07:43 |
| Tengu | would be interesting to be able to jump on the node | 07:43 |
| ysandeep|ruck | I am trying excluding the podman/buildah/netvark .. this was the rpm diff: https://www.diffchecker.com/NTUKFbuV/ | 07:43 |
| ysandeep|ruck | Tengu, not easy to reproduce in upstream but since friday our container build job is taking 20 mins more than usual | 07:43 |
| Tengu | wow... | 07:44 |
| ysandeep|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-9-quay-push-master&result=SUCCESS&skip=0 | 07:44 |
| marios | akahat: o/ lets catch up on the quay push/decommission stuff? sent invite -decline if time doesnt' work for you will send again | 07:44 |
| Tengu | ysandeep|ruck: is the patch switching the build to "network=host" merged? | 07:44 |
| ysandeep|ruck | Tengu, above url.. earlier it was ~35 - 40 mins | 07:44 |
| Tengu | iirc Rabi proposed it | 07:44 |
| marios | ysandeep|ruck: Tengu: is this for the blocked unpin ansible? (content provider failing there) | 07:44 |
| ysandeep|ruck | Tengu, not yet.. its in gate | 07:44 |
| Tengu | marios: nope | 07:45 |
| Tengu | ysandeep|ruck: dang.... that one will help | 07:45 |
| Tengu | hopefully. | 07:45 |
| marios | Tengu: k... there ysandeep|ruck fyi will try and dig in a bit https://review.opendev.org/c/openstack/tripleo-quickstart/+/864690/6#message-557095f65b4404a300bbe159570a2261a544caab | 07:45 |
| ysandeep|ruck | I guess that will help | 07:45 |
| Tengu | so we're back to the good old time when podman failed for unknown reasons - but this time with their new netavark product. | 07:45 |
| Tengu | "great". | 07:45 |
| Tengu | (or, that is, probably) | 07:45 |
| akahat | marios, o/ sure. Accepted invite. | 07:46 |
| marios | seems like the unpin still ... aaahh wallaby and zed is failing there | 07:46 |
| Tengu | marios: ah, MAYBE linked to the patch proposed by Rabi, switching the TCIB network to "host" ? | 07:46 |
| marios | no but zed ... :/ i mean that one should have passed | 07:47 |
| Tengu | why so? | 07:47 |
| marios | Tengu: k will try and look again after | 07:47 |
| ysandeep|ruck | https://review.opendev.org/c/openstack/tripleo-common/+/865116 | 07:47 |
| marios | Tengu: i think zed is meant to be fixed it was unpinned in that review (and see some of the zed cherrypick like https://review.opendev.org/c/openstack/tripleo-ansible/+/864696 | 07:48 |
| Tengu | marios: 2022-11-22 09:32:44.919 68953 ERROR tripleo_common.image.builder.buildah.BuildahBuilder [-] Unexpected error while running command. so helpful. | 07:48 |
| marios | :/ | 07:48 |
| * marios back to reviews for a bit | 07:48 | |
| Tengu | related to package installation.... maybe something here. | 07:48 |
| Tengu | marios: is the Zed content-provider in a master job using Zed code, or is it just running the tripleo-repos with proper parameters, but using master code? | 07:49 |
| Tengu | .... not sure I'm clear.. | 07:49 |
| *** amoralej|off is now known as amoralej | 07:52 | |
| marios | Tengu: checks out stable/zed for all required-projects that zuul fetches https://opendev.org/openstack/tripleo-ci/src/commit/2066a2baa2021598b71c7fb5f41d5b45118a7de8/zuul.d/content-provider.yaml#L45-L47 | 07:53 |
| Tengu | hmm ok. | 07:54 |
| ysandeep|ruck | exclude patch up: https://review.opendev.org/c/openstack/tripleo-quickstart/+/865375/1/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml .. let's see how it goes with test https://review.rdoproject.org/r/c/testproject/+/42657 | 07:54 |
| jm1 | o/ | 08:29 |
| *** jpena|off is now known as jpena | 08:29 | |
| Tengu | ysandeep|ruck: weird... seems dnsmasq won't get installed.... | 08:32 |
| Tengu | ysandeep|ruck: in the meantime, do you happen to know where's that "Copy old /opt" task is defined? | 08:32 |
| Tengu | since it's taking a huge amount of time, I'd love to understand the whys and whats. | 08:32 |
| ysandeep|ruck | looking | 08:33 |
| marios | i think content provider generally a bit unstable today? https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider | 08:35 |
| Tengu | marios: the tripleo-ci "RETRY" are mine mostly | 08:36 |
| Tengu | marios: but I also think Rabi's patch will help https://review.opendev.org/c/openstack/tripleo-common/+/865116 | 08:37 |
| *** yadnesh is now known as yadnesh|afk | 08:47 | |
| akahat | marios, https://review.rdoproject.org/r/c/config/+/45901 | 09:03 |
| marios | thx | 09:03 |
| akahat | marios, https://review.rdoproject.org/r/c/rdo-jobs/+/45921/ | 09:03 |
| marios | thanks Tengu | 09:12 |
| Tengu | ok, still 2 jobs to finish for that patch to merge - should be in ~15 minutes. | 09:19 |
| Tengu | then we'll be able to re-test without excluding netavark from the system update. | 09:19 |
| Tengu | marios: btw, I found another way to get the containers in a local registry, therefore avoiding hitting thresholds and all: use docker-registry as a proxy. It caches the layers locally, and serves them. | 09:20 |
| Tengu | exactly like a standard squid configured as proxy-cache. | 09:20 |
| ysandeep|ruck | exclude podman and friends patch is not merged yet(its wip), hit one deps issue already with just netavark.. Problem: package aardvark-dns-2:1.3.0-1.el9.x86_64 conflicts with netavark < 2:1.3.0 provided by netavark-2:1.1.0-6.el9.x86_64.. re-added aardvark-dns in exclude | 09:22 |
| Tengu | ysandeep|ruck: yeah, was pretty sure it would require some new things, since rpm is dumb. | 09:22 |
| *** yadnesh|afk is now known as yadnesh | 09:22 | |
| arxcruz | marios: ysandeep|ruck chandankumar can you guys take a look at this https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/859401 | 09:30 |
| ysandeep|ruck | Tengu, fyi.. https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/ephemeral.yaml#L108 , missed to share earlier | 09:31 |
| arxcruz | when you have time | 09:31 |
| Tengu | ah, thanks | 09:31 |
| marios | arxcruz: adding to my review queue | 09:31 |
| Tengu | ysandeep|ruck: at that point, what's the size of /opt ? | 09:31 |
| arxcruz | ty | 09:32 |
| Tengu | ysandeep|ruck: also.... I don't get the use of "find" there | 09:32 |
| Tengu | simple `mv' would do fine imho. or rsync, with proper options. | 09:33 |
| ysandeep|ruck | Tengu, these are infra level pre-playbooks and I am looking at them for the first time as well :) | 09:34 |
| Tengu | heh. | 09:34 |
| Tengu | and the comment about the "mount" module dated 2017... | 09:34 |
| Tengu | guess I'll provide some change request. | 09:34 |
| ysandeep|ruck | arxcruz, ack o/ | 09:35 |
| kkula | hello oooq, can you take a look at new dashboard http://dashboard-ci.tripleo.org/d/wplmDxdVz/puppet-promotion-pipelines?orgId=1 ? it was recentely merged, but it's empty. when inspecting dashboard, the data is loading infinitely | 09:36 |
| ysandeep|ruck | jm1, ^^ | 09:39 |
| Tengu | ah, I think I get the reason of the "find" here. it's probably for hidden directories and files. | 09:40 |
| Tengu | would be interesting to know if we have access to rsync at this point. | 09:40 |
| jm1 | kkula: maybe frenzy_friday or pojadhav have some idea whats happening? | 09:40 |
| Tengu | still....... would also be good to know the amount of data in /opt. | 09:40 |
| frenzy_friday | jm1, kkula: /me checking in some time | 09:41 |
| ysandeep|ruck | marios: hey o/ I haven't updated the program doc after we clubbed the notes(doing rr after a while), when you have time can we meet for few to update our status. | 09:41 |
| marios | ysandeep|ruck: sure np lets do now? brb 2 mins and can join | 09:42 |
| ysandeep|ruck | yes I am available to chat now. | 09:42 |
| marios | ysandeep|ruck: 11:41 < ysandeep|ruck> marios: hey o/ I haven't updated the program doc after we clubbed the notes(doing rr after a while), when you have time can we meet for few to update our status. | 09:42 |
| marios | 11:42 < marios> ysandeep|ruck: sure np lets do now? brb 2 mins and can join | 09:42 |
| marios | sorry | 09:42 |
| marios | ysandeep|ruck: https://meet.google.com/fjp-vpgt-eqg | 09:42 |
| ysandeep|ruck | marios++ thanks! | 09:49 |
| marios | thanks ysandeep|ruck | 09:57 |
| Tengu | ysandeep|ruck: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/865383 fyi | 10:04 |
| Tengu | adding some debug/info to that /opt thing, it may help understanding the actual content and all. | 10:04 |
| ysandeep|ruck | nice, Tengu++ | 10:05 |
| Tengu | not sure you can depends-on that one, can you? :) | 10:05 |
| ysandeep|ruck | We don't need depends-on now as its merged, Just for tripleo content we use the latest merged code | 10:08 |
| marios | ysandeep|ruck: we haven't yet, but probably we want to add zed into that pipeline status wdyt? (maybe when we get 18 we add it ?) | 10:08 |
| ysandeep|ruck | delorean-current.repo - https://trunk.rdoproject.org/centos9-master/component/tripleo/4f/c1/4fc1c813e1a490f2bf51c35326b53f88bfd8c1c5_cf306389/ | 10:08 |
| Tengu | ysandeep|ruck: I was more talking about the openstack-zuul-jobs proposal :) | 10:09 |
| ysandeep|ruck | oh sorry, too many thing in parallel | 10:09 |
| Tengu | ;) | 10:09 |
| ysandeep|ruck | let me check if its trusted/untrusted zuul repo | 10:09 |
| Tengu | I bet it's trusted. | 10:10 |
| Tengu | (or any value preventing to depends-on change) | 10:10 |
| ysandeep|ruck | PRE-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/openstack-zuul-jobs/legacy/pre.yaml@master] | 10:11 |
| ysandeep|ruck | ^^ looks like untrusted repo.. | 10:11 |
| ysandeep|ruck | Tengu, depends-on should work | 10:11 |
| Tengu | oh, good | 10:11 |
| Tengu | care to get a content-provider (those are the ones taking long on that location, right) with this patch? | 10:12 |
| Tengu | maybe we can avoid merging a debug thing - Infra would probably appreciate. | 10:12 |
| ysandeep|ruck | marios, yes sounds good to report once we have mini integration line up for 18 | 10:13 |
| ysandeep|ruck | Tengu, ack.. I will propose a test patch | 10:14 |
| ysandeep|ruck | Tengu, we can just reuse https://review.opendev.org/c/openstack/tripleo-ci/+/865290 .. Let me add a depends-on here | 10:14 |
| marios | arxcruz: i think you should update .subunit* to .subunit.gz unless there was a reason to change it | 10:14 |
| marios | arxcruz: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/859401/9#message-0102653df8cee3dd378784dc5e3be8c6ec1ed966 | 10:14 |
| marios | ysandeep|ruck: k | 10:14 |
| ysandeep|ruck | Tengu, https://zuul.openstack.org/status#865290 | 10:16 |
| Tengu | ysandeep++ thanks ! | 10:17 |
| marios | rlandy|out: dviroel|afk: ysandeep|ruck: chandankumar: please add to your reviews list: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/435847 https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435848 | 10:18 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 10:18 |
| Tengu | ysandeep|ruck: hmmmmm weird it doesn't seem to hit the swap thingy... it's dirctly in the build-containers.. ? | 10:20 |
| ysandeep|ruck | It failed else.. Let me remove my older changes so that we know we are testing on sane node. | 10:21 |
| Tengu | ah, the validate-host | 10:21 |
| Tengu | right? | 10:21 |
| ysandeep|ruck | yes | 10:21 |
| Tengu | 'k | 10:21 |
| ysandeep|ruck | Tengu, updated just a content-provider job with depends-on with your other patch: https://review.opendev.org/c/openstack/tripleo-ci/+/865290 | 10:23 |
| Tengu | yup | 10:23 |
| Tengu | oh. omg... /opt is full of git repositories apparently. | 10:29 |
| Tengu | no wonder it takes so long to move X( | 10:29 |
| Tengu | let's wait to actually hit the task | 10:30 |
| Tengu | but I'm pretty sure it's the issue. | 10:30 |
| Tengu | bingo. | 10:31 |
| Tengu | ysandeep|ruck: tripleo-ci/roles/prepare-node/tasks/main.yaml: name: configure-swap <- that's coming so late in the game /opt is full of git. | 10:32 |
| Tengu | ysandeep|ruck: that said, the job doesn't seem to have an ephemeral volume, meaning it doesn't create the /opt partition. | 10:34 |
| Tengu | but I think I just got enough data. | 10:34 |
| *** ysandeep__ is now known as ysandeep|ruck | 10:39 | |
| ysandeep|ruck | Tengu, sry irc bouncer is having some issues | 10:41 |
| ysandeep|ruck | looks like that playbook only includes when there is an ephemeral device attached | 10:41 |
| ysandeep|ruck | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L33-L36 | 10:41 |
| ysandeep|ruck | may be on some providers we get ephemeral disk and on some we don't | 10:42 |
| Tengu | ysandeep|ruck: yep - still, I know what's going on now. | 10:44 |
| ysandeep|ruck | ack, great | 10:44 |
| Tengu | ysandeep|ruck: there's a ton of git repositories in /opt, in fact the creation of that dedicated volume comes far too late. | 10:44 |
| Tengu | so there's no wonder it takes so long to move them around. | 10:45 |
| Tengu | ysandeep|ruck: basically: 2022-11-23 10:26:39.041488 | LOOP [prepare-workspace-git : Set initial repo states in workspace] | 10:46 |
| Tengu | it comes right after the "validate-host". | 10:46 |
| Tengu | and that one comes from... errr... | 10:47 |
| Tengu | no idea. yet another repository apparently. | 10:47 |
| Tengu | ah, zuul-jobs. | 10:48 |
| Tengu | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git/tasks/main.yaml | 10:49 |
| Tengu | imho.... tripleo-ci shouldn't have to include the configure-swap, it should be done far before. | 10:50 |
| ysandeep|ruck | rlandy|out, marios ^^ fyi.. | 10:51 |
| akahat | hello team, please take a look when you are free thanks! https://review.rdoproject.org/r/c/config/+/46180, https://review.rdoproject.org/r/c/rdo-jobs/+/46181 | 11:01 |
| pojadhav | is code.engineering.redhat.com/gerrit/login/ is down ?? | 11:07 |
| *** rlandy|out is now known as rlandy|rover | 11:10 | |
| ysandeep|ruck | rlandy|rover, good morning o/ All good news today :) | 11:14 |
| *** dviroel|afk is now known as dviroel | 11:15 | |
| dviroel | o/ | 11:16 |
| ysandeep|ruck | review time if folks want to join | 11:18 |
| rlandy|rover | ysandeep|ruck: let's meet after review | 11:20 |
| ysandeep|ruck | ack o/ | 11:20 |
| rlandy|rover | Tengu: wr tracking the DNS work, how's it looking for size? | 11:20 |
| rlandy|rover | Tengu: is there a LP bug yet? | 11:21 |
| *** soniya29 is now known as soniya29|afk | 11:23 | |
| dviroel | rlandy|rover: https://review.rdoproject.org/r/c/testproject/+/46046/10#message-0e188416c58f55d5e2a4a631a53e891d042a086f | 11:52 |
| dviroel | \o/ | 11:52 |
| dviroel | there is a patch from hjensas | 11:53 |
| dviroel | on ovb | 11:53 |
| *** ysandeep|ruck is now known as ysandeep|out | 11:57 | |
| ysandeep|out | rlandy|rover, this will need a +w once ci passes: https://review.opendev.org/c/openstack/tripleo-common/+/865391 to clear content-provider issue for zed.. I have added you/ marios in reviewlist. | 11:59 |
| reviewbot | I have added your review to the Review list | 11:59 |
| rlandy|rover | ysandeep|out: ack | 12:02 |
| *** amoralej is now known as amoralej|out | 12:06 | |
| Tengu | rlandy|rover: was side-tracked by something else I found in the way jobs are built... | 12:10 |
| rlandy|rover | Tengu: ack - spoke with ysandeep|out | 12:11 |
| rlandy|rover | Tengu: I asked him to work on merging his patch on network manager | 12:11 |
| rlandy|rover | so it doesn;y hurt us next time | 12:11 |
| Tengu | rlandy|rover, marios, ysandeep|out just nudget the net=host backport - it's clean and needed to stabilize things in CI | 12:11 |
| rlandy|rover | https://review.opendev.org/c/openstack/tripleo-common/+/865391 | 12:12 |
| rlandy|rover | yep - waiting for CI to pass | 12:12 |
| rlandy|rover | then will ask for merge | 12:12 |
| Tengu | rlandy|rover: sounds good. on my side, I found out the /opt thing being crazy as hell - but I'm missing some pieces | 12:12 |
| Tengu | rlandy|rover: ah, yeah, didn't see zuul wasn't in already. bah. | 12:12 |
| Tengu | backport's clean anyway. | 12:13 |
| rlandy|rover | Tengu: are you still interested in doing the dnsmasq move - if so it's not urgent and we can create a card | 12:13 |
| rlandy|rover | if not, we are ok as is - if we clean up the networkmanager piece | 12:13 |
| Tengu | rlandy|rover: I still think dnsmasq is the correct way to handle DNS. | 12:13 |
| rlandy|rover | Tengu: ok - adding card for that | 12:14 |
| Tengu | especially since it would allow to use local DNS servers instead of some fancy remotes. | 12:14 |
| Tengu | but I may go to Infra for it, in order to set things up as soon as possible. | 12:14 |
| rlandy|rover | sounds good | 12:14 |
| Tengu | until now, my tests at getting dnsmasq installed are failing for weird reasons... | 12:14 |
| Tengu | i.e. it sees the package, but can't get it. | 12:14 |
| rlandy|rover | let me know when you are ready - and we'll get you a CI partner | 12:14 |
| Tengu | sure | 12:15 |
| Tengu | that work will help me understanding some more of the internals :). | 12:15 |
| Tengu | rlandy|rover: though I may want to focus first on the "ephemeral.yaml" task file located in the configure-swap role. | 12:16 |
| Tengu | rlandy|rover: here, when there's some ephemeral storage, it has to move git repositories from the /opt to the new partition - meaning ordering is kind of wrong. | 12:17 |
| Tengu | though it SEEMS the /opt content is present since the very begining, maybe directly in the CI base image or something. | 12:17 |
| rlandy|rover | Tengu: https://issues.redhat.com/browse/TRIPLEOCI-1320 in our backlog - can bring into the sprint when ready | 12:18 |
| rlandy|rover | hmmm ... in opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/pre.yaml@master playbook | 12:19 |
| rlandy|rover | checking role | 12:19 |
| Tengu | rlandy|rover: took the card | 12:20 |
| rlandy|rover | Tengu: so the opendev an drdo images are built differently but I'd expect it's on both | 12:24 |
| rlandy|rover | and yes configure-swap is run everywhere - role not defined in tripleo-ci | 12:25 |
| rlandy|rover | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap | 12:25 |
| rlandy|rover | looks like ianw had a hand in that | 12:26 |
| rlandy|rover | Tengu: ^^ | 12:26 |
| Tengu | rlandy|rover: yeah - the thing is, it moves content from /opt to /mnt under some circustances - namely if there's an ephemeral storage. | 12:27 |
| Tengu | rlandy|rover: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/ephemeral.yaml#L108 | 12:27 |
| Tengu | issue here is, that /opt is apparently pre-populated with git repositories | 12:27 |
| Tengu | this one comes far, far earlier in the job bootstrap: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git/tasks/main.yaml | 12:27 |
| rlandy|rover | zuul puts them there? | 12:27 |
| Tengu | and here, we can see /opt already has some content when it's called. | 12:28 |
| Tengu | rlandy|rover: I have the feeling it's in the image itself... ? | 12:28 |
| Tengu | though I'll go to the infra team in order to get some more data about that. | 12:28 |
| Tengu | namely: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git/tasks/main.yaml#L7 | 12:28 |
| Tengu | the "cached_repos_root" is pointing to /opt | 12:28 |
| Tengu | (or some location therein) | 12:28 |
| Tengu | for instance, if you check this (closed) stream: https://zuul.openstack.org/stream/a02b9485ee3e4fb9b8dec559bea87ec9?logfile=console.log | 12:29 |
| Tengu | you can search for /opt/git | 12:29 |
| Tengu | and you'll find that some content are already in - such as ansible-role-collect-logs, ansible-config, ansible-role-python and som more - all from openstack | 12:30 |
| rlandy|rover | 2022-11-23 09:11:32.456050 | primary | + git clone --bare /opt/git/opendev.org/openstack/ansible-config_template /home/zuul/src/opendev.org/openstack/ansible-config_template/.git | 12:30 |
| rlandy|rover | 2022-11-23 09:11:32.456157 | primary | Cloning into bare repository '/home/zuul/src/opendev.org/openstack/ansible-config_template/.git'... | 12:30 |
| rlandy|rover | 2022-11-23 09:11:32.456248 | primary | done. | 12:30 |
| rlandy|rover | yeah | 12:30 |
| Tengu | yeah, that one is fine, because it didn't get the cache. | 12:30 |
| Tengu | or - yes, right, that's from the cache | 12:30 |
| Tengu | sorry | 12:30 |
| Tengu | and that cache is more than probably *in the image* itself. like, built-in | 12:31 |
| Tengu | and I'm really, really unsure it's really useful, especially reading the comment about the "why --bare". | 12:31 |
| Tengu | since it will be modified/overridden later on | 12:31 |
| rlandy|rover | idk - good question | 12:31 |
| Tengu | rlandy|rover: I'm discussing with fungi right now | 12:41 |
| rlandy|rover | k - checking channel | 12:42 |
| Tengu | so indeed, the git repos are in the image. /opt has to be freed on some provider and switched to the ephemeral storage because the VM only has 20G - tempest and other things are hitting in /opt, and /home is still sitting on the main disk.. | 12:42 |
| rlandy|rover | reading back the initial conversation | 12:45 |
| * akahat will be back after few hours. | 13:24 | |
| Tengu | ysandeep: after discussions with Infra, we'll stick to unbound, but I'll propose a patch in configure-unbound role to configure networkmanager properly. | 13:24 |
| Tengu | ysandeep: there's also a will to NOT use providers' DNS, apparently there's a long story of broken dns and all - hence we'll stick to google/cloudflare. | 13:25 |
| rlandy|rover | Tengu: ysandeep has a patch | 13:25 |
| Tengu | it will be useless with the one I'm about to propose :) | 13:26 |
| marios | rlandy|rover: chandankumar: dviroel: bhagyashris: please add to your review queue https://review.opendev.org/c/openstack/tripleo-quickstart/+/864726 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/46148 thank you | 13:29 |
| reviewbot | I have added your review to the Review list | 13:29 |
| reviewbot | I have added your review to the Review list | 13:29 |
| rlandy|rover | dviroel; any news on ibm3? | 13:29 |
| dviroel | rlandy|rover: yes - i sent you a message earlier, but you were busy :) | 13:30 |
| dviroel | rlandy|rover: job is passing with a patch on ovb, proposed by hjensas | 13:30 |
| rlandy|rover | dviroel: sorry - 100 pings on other channels - reading | 13:30 |
| rlandy|rover | dviroel++ awesomeness | 13:31 |
| rlandy|rover | dviroel: will that setting be ok on all clouds? | 13:31 |
| dviroel | rlandy|rover: testing now with fs001 on vexx | 13:32 |
| rlandy|rover | dviroel: ack- pls check downstream | 13:32 |
| dviroel | rlandy|rover: ok | 13:32 |
| rlandy|rover | dviroel: really, really nice work here | 13:32 |
| rlandy|rover | thank you | 13:32 |
| rlandy|rover | frenzy_friday: ok if I move your 1-1 30 mins earlier? | 13:40 |
| rlandy|rover | have new clashing meeting | 13:40 |
| rlandy|rover | if that doesn;t work, I can meet later | 13:41 |
| frenzy_friday | rlandy|rover, yep sure | 13:41 |
| rlandy|rover | thanks | 13:41 |
| rlandy|rover | arxcruz: hey - what should we do with https://trello.com/c/3p8i2YdZ/2639-cixlp1982874tripleociproa-testcreateobjectwithtransferencoding-is-failing-on-tripleo-jobs? | 13:43 |
| rlandy|rover | master and zed are fixed | 13:43 |
| rlandy|rover | close out and keep on skiplist for rest? | 13:43 |
| arxcruz | rlandy|rover: yes | 13:43 |
| rlandy|rover | arxcruz: ok -pls leave that comment on the card and we will close out during CIX call today | 13:44 |
| arxcruz | ok | 13:44 |
| *** amoralej|out is now known as amoralej | 13:54 | |
| * pojadhav stepping out | 13:56 | |
| dviroel | rlandy|rover: ibm bm3 is faster: https://review.rdoproject.org/zuul/build/be1858d65d824d25aae6eb3bb2f82275 | 13:57 |
| dviroel | rlandy|rover: i don't understand why it is running just a subset of tests | 13:58 |
| dviroel | i thought that was due to job name, but don't seems to be | 13:58 |
| rlandy|rover | dviroel; looks like it's running check not periodic tests | 13:59 |
| *** dasm|off is now known as dasm | 13:59 | |
| dasm | o/ | 13:59 |
| rlandy|rover | that would be related to allow list | 13:59 |
| dviroel | hum | 13:59 |
| dviroel | let me check | 14:00 |
| afuscoar | Hello. Sorry, rlandy|rover frenzy_friday can we merge this? https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/431802 thx | 14:01 |
| rlandy|rover | arxcruz: ^^ would you agree on tempest tests? | 14:02 |
| rlandy|rover | for fs001 on ibm cloud | 14:02 |
| arxcruz | checking | 14:02 |
| rlandy|rover | dasm: pls see afuscoar request above | 14:02 |
| frenzy_friday | afuscoar, ++ from me | 14:02 |
| rlandy|rover | jm1: dasm: ^^ ok to merge that? | 14:02 |
| rlandy|rover | afuscoar: want to get dasm to ok this - he's been doing some work in that area | 14:03 |
| afuscoar | that's perfect, thank you very much | 14:04 |
| rlandy|rover | dviroel: no force_periodic: true | 14:05 |
| rlandy|rover | in https://review.rdoproject.org/r/c/testproject/+/46046/12/.zuul.yaml | 14:05 |
| rlandy|rover | so it's running check | 14:05 |
| rlandy|rover | tempest | 14:05 |
| rlandy|rover | arxcruz: ^^ ok - worked it out | 14:05 |
| dviroel | rlandy|rover: arxcruz: yes, missing force_periodic | 14:06 |
| arxcruz | rlandy|rover: ack | 14:06 |
| marios | rlandy|rover: eg https://issues.redhat.com/browse/OSP-19411 is in our backlog, how do i get it into our sprint board? sorry cant find it quickly | 14:06 |
| rlandy|rover | dviroel: here is the conditional: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml#L154 | 14:06 |
| rlandy|rover | marios: drag it up | 14:06 |
| arxcruz | but i think we had an if 'periodic' in zuul_job conditional | 14:06 |
| rlandy|rover | from the backlog tab | 14:06 |
| arxcruz | oh, it's the pipeline not the job name | 14:06 |
| rlandy|rover | correct | 14:07 |
| marios | rlandy|rover: oh | 14:07 |
| marios | thanks :) | 14:07 |
| dasm | rlandy|rover: looks good to me. i can't merge, lacking +W but it's great change. thanks afuscoar | 14:07 |
| rlandy|rover | dasm: will w+ with your vote | 14:08 |
| rlandy|rover | merged | 14:08 |
| dasm | ++ | 14:14 |
| Tengu | ysandeep, rlandy|rover: https://review.opendev.org/c/opendev/base-jobs/+/865428 + https://review.opendev.org/c/openstack/project-config/+/865433 | 14:16 |
| Tengu | so that networkmanager won't mess the resolvers anymore. | 14:16 |
| Tengu | like, for good. | 14:16 |
| Tengu | back in a bit, need some break. | 14:17 |
| rlandy|rover | Tengu: thanks | 14:21 |
| Tengu | sorry if it cuts the grass under your feet, ysandeep - but it's probably better to get this config at the root of the things. | 14:22 |
| rlandy|rover | we just need to test these across all three zuul envs | 14:23 |
| Tengu | "just"... :) | 14:23 |
| Tengu | so ysandeep patch will allow to get a faster way of checking. | 14:24 |
| rlandy|rover | also rdo images are vir-customize | 14:24 |
| Tengu | also, Infra will check my proposals, of course. Maybe there's a reason for not having set that config. | 14:24 |
| rlandy|rover | need to check there are no extra lines there doing config | 14:24 |
| Tengu | though until now I didn't have any info in that direction. | 14:24 |
| rlandy|rover | ok - I see they are WIP | 14:24 |
| Tengu | yeah | 14:24 |
| rlandy|rover | let's see how you progress | 14:25 |
| Tengu | sure | 14:25 |
| rlandy|rover | if infra is ok with it, we'll do the necessary checks | 14:25 |
| rlandy|rover | no point in making you crazy with failures if infra will ditch it | 14:25 |
| Tengu | sorry for the new load ^^' | 14:25 |
| rlandy|rover | Tengu++ for following this up | 14:25 |
| rlandy|rover | not at all | 14:25 |
| rlandy|rover | good to get things more stable | 14:25 |
| Tengu | at least I could learn some interesting things : | 14:25 |
| Tengu | :) | 14:25 |
| rlandy|rover | yep - always nice whe devs get more involved in improving CI | 14:26 |
| Tengu | I'll try to get ianw tomorrow morning for the /opt work as well. | 14:26 |
| rlandy|rover | cool | 14:26 |
| Tengu | if we can make things cleaner.... | 14:26 |
| Tengu | and FASTER | 14:26 |
| Tengu | :)) | 14:26 |
| Tengu | need some break - back in a bit. | 14:26 |
| rlandy|rover | Tengu: sure - when you are back .. we'll probably close https://trello.com/c/qnUhXq1y/2790-cixlp1997202tripleociproa-master-zed-wallaby-container-build-jobs-are-failing-with-error-could-not-resolve-host-mirrorregiononev when the cherry picks are merged unless you want to keep that bug open until all your work is done | 14:28 |
| rlandy|rover | dasm: pls ping me this afternoon to review your rights on the infra-related repos | 14:30 |
| rlandy|rover | you need merge rights there | 14:30 |
| dasm | rlandy|rover++ | 14:30 |
| rlandy|rover | ysandeep: lol - you could have skipped | 14:31 |
| rlandy|rover | dasm: moved 1-1 to today so calendar invite should be sent to to the updates | 14:34 |
| dasm | rlandy|rover: confirmed | 14:35 |
| *** ysandeep__ is now known as ysandeep|out | 14:50 | |
| rlandy|rover | arxcruz: akahat: bhagyashris: marios: chandankumar: frenzy_friday: dviroel: dasm: ysandeep|out: rcastillo: jm1: soniya29|afk: pojadhav: anyone still waiting on any vacation/expense approval from me? | 14:52 |
| arxcruz | not from my side | 14:52 |
| ysandeep|out | I need to apply for 1 week dec vacation, let me do it right away | 14:52 |
| rlandy|rover | pls note I am (somewhat) out tomorrow and fri - so would like to get all approvals in | 14:52 |
| frenzy_friday | forgot to add my wfh stuff, lemme add by tomorrow | 14:52 |
| frenzy_friday | ah, ok, i'll add now | 14:53 |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| *** soniya29|afk is now known as soniya29 | 15:48 | |
| soniya29 | rlandy|rover, i will add it now | 15:49 |
| marios | rlandy|rover: fyi osp mixed rhel story/subtasks on the board now. still todo/pending close out the tripleo-ci ones so we have both for now | 15:52 |
| marios | rlandy|rover: but its progress ;) | 15:52 |
| marios | we need to reconcile our states/jira columns with the ones the OSP tasks expect though | 15:53 |
| rlandy|rover | marios: great news | 15:55 |
| marios | :) | 15:55 |
| rlandy|rover | marios: the columns map like this: | 15:55 |
| rlandy|rover | parked - not started - etc. first column | 15:55 |
| rlandy|rover | then the rest will map to some in progress | 15:56 |
| rlandy|rover | until done | 15:56 |
| rlandy|rover | the rest can't map exactly | 15:56 |
| marios | yeah like ready for review is one obvious one | 15:57 |
| Tengu | rlandy|rover: I propose to close the dnsmasq card https://issues.redhat.com/browse/TRIPLEOCI-1320 as WONTDO | 16:11 |
| Tengu | I just pushed a comment as to "why". | 16:11 |
| dasm | Tengu: checking your comment | 16:11 |
| Tengu | dasm: sure | 16:12 |
| dasm | Tengu: i see you have 2 WIP patches mentioned in the comment. Would you prefer to keep them under this card, or is it better to have it separate? | 16:13 |
| Tengu | dasm: I keep them in check via the gerrit dashboard. So for me, the card is mostly useless. Where it can be good, it's for you, CI, to get a trace of it. | 16:14 |
| *** dviroel_ is now known as dviroel | 16:15 | |
| dasm | Tengu: so, let me ask it other way around: do you need anything from CI team to help you with? | 16:16 |
| * marios off in few | 16:26 | |
| *** marios is now known as marios|out | 16:31 | |
| *** yadnesh is now known as yadnesh|away | 16:41 | |
| Tengu | dasm: (sorry, was on another topic) CI will be needed at some point to ensure this change doesn't break anything - though I really, really doubt it would. | 16:55 |
| dasm | Tengu: mhm... so actually there is no work from CI team, except assist you when you're done? | 16:56 |
| Tengu | dasm: yeah. thing is, with the patches in, it will directly be pushed to "production". Note that ysandeep has a patch doing the same work in tripleo-ci, so it's a way to pre-test things already I think. | 16:58 |
| dasm | mhm | 16:58 |
| akahat | rlandy|rover, testproject jobs pushing to old registry.. we need this patch to use quay.rdoproject.org registry. https://review.rdoproject.org/r/c/config/+/45901 | 16:59 |
| akahat | https://review.rdoproject.org/r/c/testproject/+/36756 | 16:59 |
| rlandy|rover | akahat, will look in a few | 17:18 |
| rlandy|rover | need a few for lunch | 17:18 |
| rlandy|rover | brb | 17:18 |
| *** jpena is now known as jpena|off | 17:30 | |
| *** amoralej is now known as amoralej|off | 18:34 | |
| *** eliadcohen_ is now known as eliadcohen | 19:07 | |
| dviroel | rlandy|rover: hi, so, testing hjensas patch: passing on IBM3, passing on internal. On vexx: 2 failures, both on tempest, but different tests | 19:37 |
| dviroel | vexx: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='10.0.0.5', port=13696): Read timed out. (read timeout=60) | 19:37 |
| rlandy|rover | dviroel: hmm - unrelated? | 19:38 |
| dviroel | yeah, I think so | 19:39 |
| rlandy|rover | ok - so we go with it | 19:41 |
| dviroel | rlandy|rover: we need to also backport it to stable/2.0 - for train | 19:46 |
| dviroel | added my vote | 19:46 |
| dviroel | with comment | 19:46 |
| rlandy|rover | dviroel: you can pass it to steve on the hardware prov chat | 19:46 |
| rlandy|rover | I pinged you there earlier | 19:47 |
| dviroel | rlandy|rover: will do | 19:56 |
| *** dviroel is now known as dviroel|afk | 21:25 | |
| * dviroel|afk afk | 21:25 | |
| *** rlandy|rover is now known as rlandy|out | 22:25 | |
| * dasm => offline | 23:48 | |
| dasm | see you Monday! | 23:49 |
| *** dasm is now known as dasm|off | 23:49 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!