Tuesday, 2022-08-30

dasmok, here we go again: https://review.rdoproject.org/r/c/testproject/+/4231900:37
*** rlandy|bbl is now known as rlandy00:54
*** rlandy is now known as rlandy|out01:09
*** dasm is now known as dasm|off02:00
*** ysandeep|out is now known as ysandeep04:52
*** pojadhav|out is now known as pojadhav|ruck04:59
ysandeeprlandy|out, good to see patch to fix bm worked :)05:14
*** ysandeep is now known as ysandeep|afk05:41
*** ysandeep|afk is now known as ysandeep06:02
*** jm1|ruck is now known as jm1|rover06:30
jm1o/ #oooq06:30
akahato/06:33
ysandeephello everyone and good morning/afternoon folks o/06:44
*** ysandeep is now known as ysandeep|afk07:36
*** jpena|off is now known as jpena07:36
jm1jpodivin: found your bug in another non-validation job as well https://bugs.launchpad.net/tripleo/+bug/1987092/comments/1808:02
jpodivinjm1: I see08:12
jpodivinSo it's not just us. 08:12
jpodivinThanks.08:12
jm1pojadhav|ruck: hello :) did you manage do check downstream component jobs or shall i have a look?08:21
pojadhav|ruckjm1, I am checking components as well for d/stream08:23
jm1pojadhav|ruck: ok cool! upstream is done :)08:26
jm1pojadhav|ruck: lets hope our ci keeps calm for another day, then reporting to os program call will be easy: all green ;)08:36
pojadhav|ruckjm1, yep :)09:06
jm1pojadhav|ruck: do you need help with anything?09:08
pojadhav|ruckjm1, nope thank you 09:09
pojadhav|ruckwill let you if anything09:09
jm1pojadhav|ruck: ack09:10
*** ysandeep|afk is now known as ysandeep09:12
jpodivinjm1: about renaming the bug ... I'd do it myself but I don't want to create unnecessary confusion. 09:46
jpodivinwouldn't automation pick it up again? 09:46
jm1jpodivin: no idea, we could discuss that on cix call09:48
jpodivinjm1: ok. 09:49
jm1jpodivin: thanks for bringing up this issue and debugging it. we have a lot of intermittent failures which force us to rerun jobs. this one could be one of the root causes09:50
jpodivinjm1: np. It's a common pipeline/asset, we should all take care of it in our way. 09:53
* soniya29 stepping out for 1 hr10:26
*** soniya29 is now known as soniya29|afk10:26
* jm1 lunch10:32
*** rlandy|out is now known as rlandy10:38
rlandypojadhav|ruck: jm1: hi  - anything we need to sync about today?10:39
rlandypojadhav|ruck; how are the internal components?10:40
rlandyjm1: thank you for the notes on upstream components10:41
rlandyalso - we should see what CIXs we can close out with master and wallaby promotions10:41
rlandypojadhav|ruck; are we still seeing the node provision error?10:41
rlandywhat happened with contacting steve?10:41
rlandyarxcruz: soniya29|afk: pojadhav|ruck: chandankumar: moving the tempst allow list meeting to thurs10:42
chandankumarrlandy: ack10:42
arxcruzok10:42
soniya29|afkrlandy, ack10:47
pojadhav|ruckrlandy, this is having node provision periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 but not promotion blocker10:48
pojadhav|ruck^^ job is failing since long time, I cant see any green run 10:48
pojadhav|ruckrlandy, ok for thurs allow list meeting10:49
pojadhav|ruckrlandy, will ping steve now, havent talked yet10:49
rlandypojadhav|ruck: that job has been failing for a while and we need to investigate what the failure is and why it is still failing10:55
rlandypojadhav|ruck: pls collect some data on that job10:55
rlandywill touch base with you in a bit10:55
pojadhav|ruckrlandy, ack10:55
pojadhav|ruckrlandy, pinged steve about node_provision issue for fs00111:02
rlandychecking if that issue is still going on11:03
rlandyis rdo zuul slow to resolve for anyone?11:03
pojadhav|ruckrlandy, yes11:04
pojadhav|ruckits alow11:04
pojadhav|ruckslow*11:04
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001+&skip=011:04
rlandypojadhav|ruck: ^^ looking better11:04
rlandywe had no passes before11:04
pojadhav|ruckrlandy, yeah..11:05
jm1rlandy, pojadhav|ruck: wanna sync? need to sync?11:08
pojadhav|ruckrlandy, sure11:09
rlandyjm1: pojadhav|ruck: we can quickly  https://meet.google.com/mou-aptt-jxp?pli=1&authuser=011:09
* bhagyashris stepping out for hr11:16
*** ysandeep is now known as ysandeep|break11:27
*** dviroel|out is now known as dviroel11:30
*** soniya29|afk is now known as soniya2911:31
dviroelo;11:32
jm1dviroel: o/11:32
dviroelo/11:32
rlandysoniya29: hello - sorry I need to move your 1-111:38
rlandyI have a manager meeting11:38
rlandysoniya29: is 2pm UTC too late?11:38
rlandyotherwise we can meet on friday11:38
rlandywe have the allow list meeting on thurs11:39
rlandyjm1: pojadhav|ruck: so ... I opened https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft fortuitously 11:45
rlandyit's probably something we should look at11:45
rlandypretty basic test we are skipping11:45
rlandyjm1: ^^ if you have a quiet day tomorrow - pls try unskip this and see if it's still an issue11:46
rlandyif you don;t get to it, np - let me know11:46
rlandysoniya29: moved your 1-1 to next week tuesday11:49
soniya29rlandy, nope11:50
soniya29rlandy, ack11:50
pojadhav|ruckrlandy, skiplist patch : https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/85519112:05
pojadhav|ruckrlandy, criteria change patch : https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/42653212:05
pojadhav|ruckrlandy, testing fs035 with skiplist depends-on : https://code.engineering.redhat.com/gerrit/c/testproject/+/28630712:06
rlandypojadhav|ruck: thanks - pls see comment in line https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/85519112:07
rlandyysandeep|break: dviroel: going to miss both podified meetings - manager stuff12:09
dviroelrlandy: ok12:09
pojadhav|ruckrlandy, change done12:09
rlandypojadhav|ruck: thanks - merging12:10
* pojadhav|ruck taking short break12:18
*** ysandeep|break is now known as ysandeep12:20
chandankumarrlandy: https://review.opendev.org/c/openstack/tripleo-quickstart/+/855115 good to go12:39
chandankumarthanks!12:39
rlandybhagyashris: hi - can you check 17.1 on 8 line12:45
rlandylast promo was 08/2612:45
rlandyhttps://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17-1/12:45
bhagyashrisrlandy, ok 12:45
rlandywe should be promoting daily12:45
rlandyrel delivery asked for it12:46
bhagyashrisrlandy, let me check with component jobs12:46
rlandybhagyashris: k - let me know - the line atm should just be containers and images, right?12:47
bhagyashrisrlandy, yes12:47
rlandypojadhav|ruck: merged criteria patch12:47
rlandycan you rerun fs035 on 17.112:47
rlandylet's see the damage there12:47
rlandyjm1++ pojadhav|ruck++ for rocking out as ruck/rovers12:48
rlandyall: pls note all hands meeting at 3pm UTC12:49
pojadhav|ruckrlandy, \o/12:50
pojadhav|ruckrlandy, sure 12:50
pojadhav|ruckrlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#28630712:52
rlandypojadhav|ruck; actually - my mistake 17.1 line just kicked12:54
rlandyso you can kill your job12:54
rlandyI'll run it if fs035 fails12:54
pojadhav|ruckrlandy, ack12:54
rlandysorry about that12:55
pojadhav|rucknp12:56
jm1rlandy, pojadhav|ruck: after community call and aoc meeting i will need a break. dunno how many lines i wrote in rr notes today 🤯12:59
pojadhav|ruckjm1, hey yeah sure but just try to have updates in short, dont need in detail everything. if the bug is there then logs are already present in the bug.13:01
rlandyjm1: np - things are fine today13:02
jm1pojadhav|ruck: i need the notes to be able to quickly check whether this is a known (intermittent) bug or something new13:11
jm1pojadhav|ruck: ..and to know how often i rechecked a job etc13:12
pojadhav|ruckjm1, ack np 13:12
pojadhav|ruckcommunity call in 6 mins : arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios13:24
pojadhav|ruckkindly add agenda if any : https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg13:25
arxcruzpojadhav|ruck i might be late, need to get my daugther at school 13:25
pojadhav|ruckarxcruz, ack13:26
*** dasm|off is now known as dasm13:46
dasmo/13:46
* pojadhav|ruck afk for few mins13:59
jm1rlandy, pojadhav|ruck: we might have a blocker on c9 master and c9 wallaby. tempest is failing three times on same errors on periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-{master,wallaby}13:59
ysandeeprlandy++ thanks for the local holiday reminder, I forgot about that :D13:59
rlandyysandeep: thank pojadhav|ruck  - she reminded me14:01
rlandyjm1: is that the same as the card dviroel is working on?14:01
rlandynested virt issue?14:01
rlandyfailing octavia tests?14:02
rlandypojadhav|ruck: jm1: got some manager- type stuff that is near due I need to take care of - pls ping if you need help and when you are EoD with anything to watch14:16
dviroelrlandy: jm1: yes, same issue14:28
dviroellooking into periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/e31bce2/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log14:28
dviroelKVM: entry failed, hardware error 0x814:28
jm1dviroel: oh wow, thanks for figuring that out!14:35
jm1rlandy: will update rr notes about failing jobs later, need a break first14:36
rlandyjm1: sure - no burning emergency14:37
jm1rlandy: dviroel's bug is in bugzilla and for osp while we are facing that on c9 master/wallaby. i guess we dont need a new lp bug since its failing on internal job only?14:39
rlandyjm1: it's the same problem14:39
rlandythe fix is related to the cloud itself14:39
*** ysandeep is now known as ysandeep|dinner14:39
rlandythey all run nested virt14:39
jm1rlandy: yeah, ok we do not create a lp bug since bz bug is enough to track both osp and rdo jobs14:43
*** dviroel is now known as dviroel|mtg14:43
rlandyjm1: I think it depends where the fix needs to go14:43
* dviroel|mtg sorry, fips meeting14:43
rlandyin this case, it would land with rhel/virt14:43
rlandyso BZ is probably the right address14:44
rlandydepends on on what dviroel|mtg concludes is the real issue and fix/workaround14:44
jm1rlandy: oh we actually have a bug report for this bug on c8 train https://bugs.launchpad.net/tripleo/+bug/197954614:47
jm1rlandy: updated rr notes and now i am having a longer break14:48
rlandythanks14:58
*** ysandeep|dinner is now known as ysandeep15:00
pojadhav|ruckjm1, rlandy : ack15:00
*** dviroel|mtg is now known as dviroel15:18
ysandeeprlandy, we need some karmas for dviroel++ , first successfull OCP deploy: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/30976/rehearse-30976-pull-ci-openstack-k8s-operators-mariadb-operator-master-e2e-openstack/156460624003255910415:35
dviroel•‿•15:37
* dviroel lunch15:40
*** dviroel is now known as dviroel|lunch15:40
chandankumardviroel|lunch: rlandy w00t, great :-)15:49
chandankumarsee ya!15:49
*** ysandeep is now known as ysandeep|out15:56
rlandydviroel|lunch: awesomeness!!!!!16:00
rlandydviroel++++++++++++++++++++16:00
rlandythis is so cool16:00
rlandylunch - brb16:01
*** dviroel|lunch is now known as dviroel16:33
*** jpena is now known as jpena|off16:42
dviroelreviewbot pls add to the review list https://softwarefactory-project.io/r/c/config/+/2597316:45
reviewbotI could not add the review to Review List16:45
dviroeloh :(16:45
dviroelreviewbot: but you just added :)16:46
rlandydviroel: you're arguing with a bot????16:59
pojadhav|ruck:D17:00
dviroelrlandy: yeah, it is ignoring me now, but at least, did its part of the job17:03
pojadhav|ruckrlandy, I have updated hackmd with notes.. please watch https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#209874 and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#27623017:52
pojadhav|ruckrlandy, leaving for the day17:52
* pojadhav|ruck see you on thursday. bye good night..!17:53
*** pojadhav|ruck is now known as pojadhav|out17:53
rlandydviroel: want to join review time?19:02
rlandyfor your review19:03
dviroelrlandy: on #rhos-dev, talking with artom, maybe you can help us with previous configuration on ocativa tests19:04
dviroelwill join in a min19:04
rlandyon review call19:04
dviroelrlandy: btw, have to go through some manila patches, since it is feature freezer week =/19:20
* dasm => doc's appt19:21
*** dasm is now known as dasm|off19:21
rlandydviroel: you need to or you want us to?19:23
rlandyif it's just you, sure - go4it19:23
dviroelrlandy: just me, review time reminded me that 19:27
jm1rlandy: retriggered c9 wallaby promotion jobs (except for internal) to check whether we have intermittent errors here. will be eod now19:58
* jm1 have a nice evening folks :)19:59
rlandyjm1[m]: thanks20:00
rlandyjm1[m]: updated new prog call doc - will run through with you in the morning20:44
*** dviroel is now known as dviroel|out22:37
*** rlandy is now known as rlandy|out23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!