| *** jmasud has joined #oooq | 00:29 | |
| *** jmasud has quit IRC | 00:45 | |
| *** jmasud has joined #oooq | 00:47 | |
| *** jmasud has quit IRC | 01:07 | |
| *** zbr has quit IRC | 01:30 | |
| *** zbr has joined #oooq | 01:43 | |
| *** apetrich has quit IRC | 02:09 | |
| *** jmasud has joined #oooq | 02:10 | |
| *** jmasud has quit IRC | 03:52 | |
| *** ykarel has joined #oooq | 04:15 | |
| *** jmasud has joined #oooq | 04:22 | |
| *** ratailor has joined #oooq | 04:45 | |
| *** ratailor has quit IRC | 04:55 | |
| *** ratailor has joined #oooq | 04:59 | |
| *** ysandeep|away is now known as ysandeep|ruck | 05:05 | |
| *** jfrancoa has joined #oooq | 05:11 | |
| *** marios has joined #oooq | 05:11 | |
| *** ratailor_ has joined #oooq | 05:16 | |
| *** ratailor has quit IRC | 05:19 | |
| *** udesale has joined #oooq | 05:34 | |
| *** udesale has quit IRC | 05:36 | |
| *** udesale has joined #oooq | 05:38 | |
| *** jpodivin has joined #oooq | 06:01 | |
| akahat | bhagyashris, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33855 | 06:10 |
|---|---|---|
| *** marios has quit IRC | 06:12 | |
| *** marios has joined #oooq | 06:13 | |
| *** marios has quit IRC | 06:21 | |
| *** marios has joined #oooq | 06:27 | |
| *** apetrich has joined #oooq | 06:46 | |
| *** ykarel has quit IRC | 06:49 | |
| *** ykarel has joined #oooq | 06:50 | |
| *** jpena|off is now known as jpena | 06:55 | |
| *** ratailor__ has joined #oooq | 07:06 | |
| *** ratailor_ has quit IRC | 07:09 | |
| *** tosky has joined #oooq | 07:18 | |
| zbr | quick review on https://review.rdoproject.org/r/c/config/+/33870 needed please | 07:43 |
| zbr | is just a linter bumping | 07:43 |
| *** jmasud has quit IRC | 07:54 | |
| zbr | marios: bhagyashris ? | 08:29 |
| marios | zbr: make me | 08:32 |
| *** bogdando has joined #oooq | 08:32 | |
| bogdando | hi folks. The zuul reproducer on centos stream on libvirt behaves oddly for me... There is "Generate nodepool main configuration" failing for some subnode with AnsibleError: An unhandled exception occurred while running the lookup plugin 'pipe'. Error was a <class 'ansible.errors.AnsibleError'>, original message: lookup_plugin.pipe(ssh-keyscan -t ed25519 <ip>) returned 1 | 08:33 |
| bogdando | anyone have had that as well? | 08:33 |
| bogdando | that's ansible 2.9.22 | 08:33 |
| bogdando | while ssh-keyscan -t ed25519 <ip> works on console... | 08:36 |
| bogdando | lesigh | 08:36 |
| bogdando | sshnaidm: ^^ perchance | 08:36 |
| bogdando | ok, nvm, the issue is that after a subnode reboot, it doesn't always take its ip via dhcp... logging in and poking dhclient eth0 fixes that. wtf really... | 08:49 |
| *** ykarel is now known as ykarel|lunch | 08:57 | |
| bogdando | PTAL https://review.rdoproject.org/r/c/rdo-infra/ansible-role-tripleo-ci-reproducer/+/33876 | 09:08 |
| *** bogdando has left #oooq | 09:34 | |
| sshnaidm | ysandeep|ruck, I'll do a patch, let's try to catch this ovb error and hold the environment | 09:47 |
| ysandeep|ruck | sshnaidm, ack o/ thanks! | 09:47 |
| *** arxcruz|rover has joined #oooq | 09:54 | |
| arxcruz|rover | ysandeep|ruck: can you see me now ? | 09:55 |
| ysandeep|ruck | arxcruz|rover, yes o/ | 09:55 |
| *** jmasud has joined #oooq | 10:00 | |
| sshnaidm | ysandeep|ruck, https://review.rdoproject.org/r/c/testproject/+/28004/12/zuul.yaml | 10:00 |
| sshnaidm | let's run jobs together to increase the chance | 10:01 |
| *** ykarel|lunch has quit IRC | 10:02 | |
| ysandeep|ruck | sshnaidm: ack o/ do you have privileges to hold these nodes if required? | 10:02 |
| sshnaidm | ysandeep|ruck, yep, I set them for hold in case of failure | 10:03 |
| sshnaidm | ysandeep|ruck, ask from fbo in rdo to get same | 10:04 |
| ysandeep|ruck | sshnaidm, yes i already requested for token today from fbo and mhu. | 10:05 |
| *** jmasud has quit IRC | 10:10 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 10:20 | |
| *** ysandeep|brb is now known as ysandeep|ruck | 10:35 | |
| sshnaidm | ysandeep|ruck, I see multiple attempts for ovb also, it might be a problem with creating ovb stacks. Worth to look how many stack failed to create today. | 10:37 |
| ysandeep|ruck | let me check if we have a easy way to tell that.. http://tripleo-cockpit.usersys.redhat.com/d/Z3PbeiuWz/vexxhost?viewPanel=8&orgId=1&from=now-24h&to=now only give instantious report.. | 10:41 |
| ysandeep|ruck | sshnaidm, based on estimate from ^^ around ~13 stacks failed to create today. (I calculated the difference whenever there is change in number of stacks which are in CREATE_FAILED) | 10:51 |
| ysandeep|ruck | in last 24 hours. | 10:51 |
| sshnaidm | ysandeep|ruck, yeah, not good.. | 10:54 |
| sshnaidm | ysandeep|ruck, I think dpawlik had this stats somewhere in kibana also | 10:55 |
| sshnaidm | interesting to have logs from one of them.. | 10:55 |
| ysandeep|ruck | sshnaidm, ack i will request dpawlik | 10:56 |
| * dpawlik reading | 10:57 | |
| ysandeep|ruck | dpawlik, hello, basically is there any kibana report, which can tell us how many ovb stacks failed to create today? | 10:57 |
| dpawlik | ysandeep|ruck: there was some, let me find it | 10:58 |
| *** sshnaidm is now known as sshnaidm|afk | 10:58 | |
| dpawlik | hmm, can find. Let me do new one | 11:00 |
| dpawlik | ysandeep|ruck: you can try to check https://review.rdoproject.org/analytics/goto/0d4595903e954d30d756e467200248db dashboard | 11:01 |
| dpawlik | probably I got a backup of ovb stack creation fail | 11:01 |
| dpawlik | ysandeep|ruck: try https://review.rdoproject.org/analytics/goto/f966e62bdda69bf511421b6014497d82 | 11:07 |
| *** ykarel has joined #oooq | 11:10 | |
| ysandeep|ruck | dpawlik: thanks! as per report, no stack failed today.. | 11:15 |
| ysandeep|ruck | sshnaidm|afk, https://review.rdoproject.org/analytics/goto/f966e62bdda69bf511421b6014497d82 not much as per log stash for today | 11:17 |
| dpawlik | let me finish one thing and I will check if metrics are fine | 11:20 |
| dpawlik | it can be a situation that comparing to old Kibana, some information are missing on restoring in new Kibana | 11:20 |
| dpawlik | give me few min | 11:21 |
| ysandeep|ruck | sure | 11:21 |
| *** jlarriba has joined #oooq | 11:27 | |
| zbr | i need a review on https://review.rdoproject.org/r/c/config/+/33869 - and i am happy to explain to others the benefits of pinning test deps using pip-compile. | 11:28 |
| zbr | in fact we already use pip-compile in tripleo health project, but i think is time to extend it use, mainly to reduce pypi-injected surprises | 11:29 |
| dpawlik | ysandeep|ruck: do you have some example log? | 11:36 |
| zbr | bhagyashris: any chance to switch you rdo config to "Set new changes to "work in progress" by default"? https://review.rdoproject.org/r/settings/#Profile | 11:36 |
| *** udesale has quit IRC | 11:37 | |
| zbr | that is very useful on rdo in particular as otherwise rdo will add people to the CR as soon you create it, even if you do not want. | 11:37 |
| *** udesale has joined #oooq | 11:38 | |
| *** rlandy has joined #oooq | 11:38 | |
| ysandeep|ruck | dpawlik: we have a logging of number of heat stack in different after every 15 mins: http://tripleo-cockpit.usersys.redhat.com/d/Z3PbeiuWz/vexxhost?viewPanel=8&orgId=1&from=now-24h&to=now | 11:41 |
| bhagyashris | zbr, not sure let me check | 11:42 |
| ysandeep|ruck | dpawlik: and we run a cleanup script after every 4 hours, that contains names of stack that i.e failed to create/delete http://paste.openstack.org/show/805791/ | 11:42 |
| ysandeep|ruck | s/different/ different states | 11:43 |
| dpawlik | ysandeep|ruck: I was filtering only by message: "RUN END RESULT_TIMED_OUT" or " "\"attempts\": 18" | 11:44 |
| ysandeep|ruck | dpawlik, I guess we need to tune the search query which can capture all the reasons an ovb heat stack creation can fail with. | 11:49 |
| dpawlik | ysandeep|ruck: yup + set uniq build_uuid to get correct results | 11:51 |
| dpawlik | let me fix gerrit user creation issue, than I will check the visualization | 11:51 |
| ysandeep|ruck | dpawlik: sure thanks for help! | 11:52 |
| rlandy | pojadhav: do you want to meet? | 12:02 |
| rlandy | otherwise we can cancel | 12:02 |
| pojadhav | we can | 12:02 |
| rlandy | marios: hey - is there work around updates/upgrades or is that mostly on hold? | 12:10 |
| weshay|ruck | ysandeep|ruck, UGH.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145 | 12:10 |
| weshay|ruck | this is going to be slow patch to merge | 12:10 |
| rlandy | marios: chatting with pojadhav about taking on a task | 12:10 |
| ysandeep|ruck | weshay|ruck: yeah i have been doing recheck dance \o/ since morning to merge these patches. | 12:11 |
| marios | rlandy: o/ | 12:12 |
| marios | rlandy: what do you mean work around updates/upgrades? | 12:12 |
| marios | rlandy: for our sprint you mean? | 12:12 |
| marios | rlandy: i have started working on https://projects.engineering.redhat.com/browse/TRIPLEOCI-496 & https://review.opendev.org/q/topic:wallaby-upgrade-jobs | 12:13 |
| rlandy | marios: ^^ yep that task | 12:13 |
| marios | rlandy: mainly looking at re-adding master undercloud upgrade voting and also wallaby undercloud/minor update | 12:13 |
| rlandy | marios: si there work for another person there? | 12:14 |
| marios | rlandy: pojadhav: feel free to pick up any of the other tasks? | 12:14 |
| marios | rlandy: there are a whole lot of tasks | 12:14 |
| weshay|ruck | ysandeep|ruck, arxcruz|rover let's sync in 15min | 12:14 |
| marios | rlandy: pojadhav: e.g. 389 | 12:14 |
| ysandeep|ruck | weshay|ruck, ack | 12:14 |
| weshay|ruck | ysandeep|ruck, arxcruz|rover want to show you something cool | 12:14 |
| rlandy | marios: can you join https://meet.google.com/hfu-icdo-kew?authuser=0 for a few? | 12:14 |
| marios | rlandy: pojadhav: or 390 | 12:14 |
| marios | rlandy: sure | 12:15 |
| weshay|ruck | arxcruz|rover, you avail? | 12:18 |
| arxcruz|rover | weshay|ruck: yes, sorry | 12:18 |
| weshay|ruck | cool.. sync in 12 min.. you'll like this | 12:18 |
| arxcruz|rover | weshay|ruck: ysandeep|ruck let's sync whenever you guys want | 12:18 |
| ysandeep|ruck | arxcruz|rover, in 12 mins, weshay|ruck sent an invite | 12:19 |
| ykarel | ysandeep|ruck, is the cause for overcloud deploy already known https://bugs.launchpad.net/tripleo/+bug/1929745 ? | 12:25 |
| openstack | Launchpad bug 1929745 in tripleo "Unchecked MSR access error - overcloud deploy "timed out waiting for ping module test" [Critical,Triaged] | 12:25 |
| ykarel | noticed in a test patch and reached to the bug | 12:26 |
| *** sshnaidm|afk is now known as sshnaidm | 12:28 | |
| ysandeep|ruck | ykarel: nope, so far we noticed failing node is using a different BIOS version probably compute node in vexx cloud have different version of libvirt/kvm , sshnaidm|afk is running a testproject with autohold to debug that. | 12:29 |
| ysandeep|ruck | ykarel, more discussion happened on #tripleo around this http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2021-05-27.log.html#t2021-05-27T09:54:20 | 12:29 |
| ykarel | ok /me check logs | 12:29 |
| ykarel | have we already checked in #rhos-ops? | 12:30 |
| ykarel | i seen discussion related to upgrade, may be it was for this env only | 12:30 |
| ysandeep|ruck | ykarel, not yet | 12:30 |
| ysandeep|ruck | arxcruz|rover, meet.google.com/dfz-xszy-pdo | 12:31 |
| ykarel | ack | 12:31 |
| rlandy | ysandeep|ruck: arxcruz|rover: weshay|ruck: ok to merge https://review.rdoproject.org/r/c/rdo-jobs/+/33861 Re-enable image builds for OVB check jobs? | 12:34 |
| ysandeep|ruck | rlandy, +1 | 12:35 |
| weshay|ruck | ysandeep|ruck, arxcruz|rover https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/defaults/main.yml#L368 | 12:42 |
| weshay|ruck | http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_040/793154/3/gate/tripleo-ci-centos-8-content-provider/0406160/logs/undercloud/var/log/extra/logstash.txt | 12:43 |
| dpawlik | ysandeep|ruck: checking logs, would add some new filters | 12:48 |
| marios | scrum cancelled today? | 13:01 |
| marios | bhagyashris: ? | 13:01 |
| *** jbadiapa|away is now known as jbadiapa | 13:02 | |
| *** jfrancoa has quit IRC | 13:03 | |
| *** jfrancoa has joined #oooq | 13:04 | |
| dviroel | i'll need to restart here, screen is not responding | 13:05 |
| *** pojadhav is now known as pojadhav|brb | 13:05 | |
| *** jpodivin has quit IRC | 13:07 | |
| *** jpodivin has joined #oooq | 13:08 | |
| *** jfrancoa has quit IRC | 13:09 | |
| *** jfrancoa has joined #oooq | 13:11 | |
| weshay|ruck | arxcruz|rover++ | 13:12 |
| weshay|ruck | ysandeep|ruck++ | 13:12 |
| *** ratailor__ has quit IRC | 13:15 | |
| weshay|ruck | ysandeep|ruck, keep an eye on the tripleo component for 16.2 | 14:04 |
| weshay|ruck | we have some folks getting itchy for promotions | 14:04 |
| * weshay|ruck looking | 14:04 | |
| ysandeep|ruck | weshay|ruck, it was blocked on something, let me recall | 14:05 |
| ysandeep|ruck | weshay|ruck, yeah infrared job was failing with bug: https://bugzilla.redhat.com/show_bug.cgi?id=1962617 | 14:06 |
| openstack | bugzilla.redhat.com bug 1962617 in distribution "16.2 Couldn't install tripleo-ansible-0.7.1-2.20210513023001.3c614f9.el8ost.noarch: nothing provides ansible-collection-ansible-netcommon /-ansible-posix/ -community-general / -containers-podman." [Urgent,Closed: notabug] - Assigned to rhos-maint | 14:06 |
| ysandeep|ruck | i think sagi reverted those patches.. ^^ i think we might be in better position. | 14:07 |
| pojadhav|brb | weshay|ruck, ysandeep|ruck : https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/243388 | 14:07 |
| pojadhav|brb | weshay|ruck, rlandy : pls approve https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33868 | 14:08 |
| *** pojadhav|brb is now known as pojadhav | 14:08 | |
| weshay|ruck | rlandy, https://review.rdoproject.org/r/c/config/+/33869 | 14:11 |
| weshay|ruck | pojadhav, thanks.. please start including a link to a screenshot | 14:30 |
| weshay|ruck | pojadhav, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33868 comment | 14:34 |
| weshay|ruck | ysandeep|ruck, dang it.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145 is not going into the gate.. due to https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098/ | 14:41 |
| weshay|ruck | those did not need to be rebased on top of each other | 14:42 |
| ysandeep|ruck | weshay|ruck, yes they need i.e rebase or depends-On , https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145 - containers-multinode branched jobs will fail otherwise. | 14:49 |
| *** jmasud has joined #oooq | 14:53 | |
| weshay|ruck | oh.. because of openvswitch | 14:56 |
| weshay|ruck | k | 14:56 |
| weshay|ruck | fair enough.. I'll watch the patches through | 14:56 |
| zbr | FYI: https://github.com/containers/podman/issues/10469 -- occasion to provide some feedback if you want to improve podman UX | 14:58 |
| zbr | weshay|ruck: does https://review.rdoproject.org/r/c/config/+/33869/6/README.md explain it? | 14:59 |
| weshay|ruck | zbr, thanks.. I think we can merge that | 15:04 |
| * weshay|ruck adjusting irc/znc | 15:05 | |
| weshay|ruck | may bounce around | 15:05 |
| *** weshay|ruck has quit IRC | 15:05 | |
| zbr | i will try to get few more eyes on it especially as this would be used as a model for other repos where we have the same need, i have the molecule ones in mind in particular. | 15:07 |
| rlandy | ysandeep|ruck: are we still blocked on content providers? | 16:01 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/772571 | 16:02 |
| ysandeep|ruck | rlandy, yes | 16:02 |
| rlandy | trying to get that review through | 16:02 |
| rlandy | ysandeep|ruck: pls can you post the review to fix so I can depends-on or rebase | 16:02 |
| *** marios has quit IRC | 16:02 | |
| ysandeep|ruck | rlandy, sure sec | 16:02 |
| rlandy | thank you | 16:03 |
| ysandeep|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098/ | 16:04 |
| *** ykarel has quit IRC | 16:09 | |
| *** jmasud has quit IRC | 16:10 | |
| rlandy | cool | 16:13 |
| *** jpodivin has quit IRC | 16:19 | |
| *** jmasud has joined #oooq | 16:20 | |
| *** udesale has quit IRC | 16:20 | |
| *** jpena is now known as jpena|off | 16:24 | |
| *** jmasud has quit IRC | 16:31 | |
| *** jmasud has joined #oooq | 16:32 | |
| *** amoralej is now known as amoralej|off | 16:32 | |
| *** ysandeep|ruck is now known as ysandeep|away | 16:36 | |
| ysandeep|away | arxcruz|rover, see you on monday o/ | 16:37 |
| *** jmasud has quit IRC | 16:57 | |
| *** Tengu has quit IRC | 17:00 | |
| *** jmasud has joined #oooq | 17:21 | |
| *** jmasud has quit IRC | 17:29 | |
| *** jmasud has joined #oooq | 18:09 | |
| *** jmasud_ has joined #oooq | 18:13 | |
| *** jmasud has quit IRC | 18:16 | |
| *** jmasud_ has quit IRC | 18:29 | |
| *** jmasud has joined #oooq | 18:52 | |
| *** jmasud has quit IRC | 18:58 | |
| *** jmasud has joined #oooq | 19:01 | |
| *** jfrancoa has quit IRC | 19:03 | |
| *** slaweq has quit IRC | 20:32 | |
| *** jbadiapa has quit IRC | 20:45 | |
| *** dsneddon has quit IRC | 20:58 | |
| *** jmasud has quit IRC | 21:04 | |
| *** jmasud has joined #oooq | 21:04 | |
| *** jmasud has quit IRC | 21:06 | |
| *** jmasud has joined #oooq | 21:29 | |
| *** jmasud has quit IRC | 21:35 | |
| *** slaweq has joined #oooq | 21:48 | |
| *** jmasud has joined #oooq | 21:52 | |
| *** jmasud has quit IRC | 22:09 | |
| *** slaweq has quit IRC | 22:24 | |
| *** dsneddon has joined #oooq | 22:49 | |
| *** tosky has quit IRC | 23:12 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!