Monday, 2021-01-04

*** tosky has quit IRC00:12
*** holser has joined #oooq00:18
*** holser has quit IRC01:14
*** raukadah is now known as chandankumar03:04
*** skramaja has joined #oooq03:28
*** jmasud has joined #oooq03:42
*** ratailor has joined #oooq03:56
*** bhagyashris has joined #oooq04:19
*** udesale has joined #oooq04:32
bhagyashrisysandeep, can you please check this link https://review.rdoproject.org/zuul/status is it working for you ?04:58
ysandeepbhagyashris, no i faced same issue.. some proxy error04:59
bhagyashrisysandeep, yeah me too facing same issue :(04:59
ysandeepbhagyashris, i have pinged on internal ops channel..05:01
bhagyashrisyeah i see05:01
bhagyashristhank you :)05:01
*** ykarel has joined #oooq05:13
*** ykarel_ has joined #oooq05:32
*** ykarel has quit IRC05:34
*** ykarel_ is now known as ykarel05:35
*** marios has joined #oooq06:12
*** jmasud has quit IRC06:38
*** matbu has joined #oooq06:48
*** tosky has joined #oooq08:35
*** frenzy_friday has joined #oooq08:36
*** arxcruz|2021 is now known as arxcruz08:58
arxcruzand i'm back :D08:58
marios\o/09:00
mariosme too09:00
arxcruzmarios: happy new year!09:07
arxcruzbhagyashris: ChanServ dmellado soniya29 sshnaidm ysandeep ysandeep kopecmartin happy new year!09:08
mariosarxcruz: you 209:08
soniya29arxcruz: Happy new year09:08
ysandeephey folks o/ Happy new year :D09:08
bhagyashrisHappy new year all ! :)09:09
zbrHappy new year!09:21
arxcruzzbr: happy new year09:22
arxcruzspotz: happy new year :)09:35
*** derekh has joined #oooq09:50
*** ysandeep is now known as ysandeep|afk10:16
*** saneax has joined #oooq10:26
*** sanjayu_ has joined #oooq10:28
*** saneax has quit IRC10:31
*** slaweq has joined #oooq10:36
slaweqmarios: hi, can You take a look at https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764095 ? All other backports of this patch are already merged or at least approved (Train)10:37
mariosslaweq: ack10:37
slaweqmarios++ thx10:38
*** matbu has quit IRC10:40
*** rfolco has joined #oooq10:41
*** udesale_ has joined #oooq10:41
*** matbu has joined #oooq10:42
mariosnp slaweq10:43
*** udesale has quit IRC10:44
rfolcohappy new year folks10:48
rfolcochandankumar, marios thanks for reviewing https://review.opendev.org/c/openstack/tripleo-ci/+/76759110:48
rfolco\o/10:48
*** sanjayu_ has quit IRC10:49
mariosrfolco: o/10:50
*** matbu has quit IRC11:18
*** matbu has joined #oooq11:18
*** ysandeep|afk is now known as ysandeep11:38
*** holser has joined #oooq11:59
chandankumarrfolco: marios ysandeep bhagyashris soniya29 arxcruz kopecmartin zbr frenzy_friday Happy New year !12:06
ysandeepHappy new year to all :D12:06
rfolcohappy 2021 everyone12:09
frenzy_fridayHappy new year everyone!12:14
*** slaweq has quit IRC12:18
*** chem has joined #oooq12:19
chandankumararxcruz: marios quick merge https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/76913712:32
*** ykarel has quit IRC12:33
*** ykarel has joined #oooq12:33
*** ykarel_ has joined #oooq12:37
*** rlandy has joined #oooq12:37
*** ykarel has quit IRC12:37
rlandyhappy new year everyone!!12:39
marioschandankumar: ack12:40
mariosrlandy: o/12:40
chandankumarrlandy: \o, welcome back after an year :-)12:41
rlandygood to back!12:41
rlandyto be12:41
weshay|ruckhellooooooooooooooooo :) happy new year folks12:45
rlandyweshay|ruck: hey - how was the grand canyon?12:47
weshay|rucklolz.. will tell you later12:47
*** holser has quit IRC12:51
weshay|rucksshnaidm, ysandeep can you jump on a call real quick and do a ruck/rover handover?12:51
sshnaidmweshay|ruck, sure12:51
ysandeepweshay|ruck, ack12:51
weshay|ruckk..12:51
weshay|ruckhttps://meet.google.com/psj-ymrs-jqp12:52
ysandeepweshay|ruck, https://code.engineering.redhat.com/gerrit/login/#/c/222438/12:59
bhagyashrisarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar, frenzy_friday12:59
sshnaidmrlandy, hi, can we take a look at 16.1 today? I think there is a problem with openvswitch that breaks overcloud networking.. maybe need a different version12:59
bhagyashrisscrum in one min12:59
rlandysshnaidm: sure12:59
sshnaidmrlandy, thanks12:59
rlandywill check out the versions and job logs13:00
sshnaidmsoniya29, meet.google.com/oiv-geho-mai13:02
weshay|ruckarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar, frenzy_friday  scrum13:03
soniya29sshnaidm, in scrum meeting13:03
rlandyweshay|ruck: we're on scrum13:03
bhagyashrisweshay|ruck, https://meet.google.com/oiv-geho-mai13:04
sshnaidmsoniya29, the scrum is here ^13:04
sshnaidmbhagyashris, need to fix scrum meeting, it leads me to meet.google.com/xnf-tvdh-pmk13:04
*** pojadhav has joined #oooq13:04
bhagyashrissshnaidm, ohh i will check that13:05
ysandeepsshnaidm, hey o/ have we decided to cix https://bugs.launchpad.net/tripleo/+bug/1909574 this one?13:10
openstackLaunchpad bug 1909574 in tripleo "periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-octavia-master/victoria/ussuri failing on tempest tests with SSHTimeout issue with error message "socket.timeout: timed out"" [High,Triaged]13:10
ysandeepwhere we noticed segfault in ovn.13:11
sshnaidmysandeep, well, it's not blocking prodchain, so maybe better to have it in bugzilla13:11
sshnaidmweshay|ruck, wdyt? ^13:11
ysandeepyeah otherwise I will try seek help from neutron guys, they are generally very helpful :D13:12
*** matbu has quit IRC13:16
*** matbu has joined #oooq13:16
ysandeepsshnaidm, i will do that.. thanks!13:18
sshnaidmysandeep++13:18
sshnaidmweshay|ruck, need to look at https://bugzilla.redhat.com/show_bug.cgi?id=191189113:21
openstackbugzilla.redhat.com bug 1911891 in ansible-openstack-modules "[RFE] Improvement in ansible-playbook" [Medium,New] - Assigned to sshnaidm13:21
weshay|rucksshnaidm, ysandeep can one of you folks join me in the cix call in 9 min?13:21
ysandeepweshay|ruck, i will join13:21
*** ratailor has quit IRC13:23
*** tosky has quit IRC13:30
*** tosky has joined #oooq13:30
spotzHappy New Years arxcruz13:37
sshnaidmweshay|ruck, have 2 min?13:58
weshay|ruckaye13:58
sshnaidmhttps://meet.google.com/xod-cdvi-bfb13:58
rlandysshnaidm: ok - so I think I know where the openvswitch is coming in - just not sure how we fix it yet - looking14:11
rlandybefore the job was throwing an error deleting the openvswitch that was installed from zuul14:12
rlandywe need a lower version initially14:12
*** TrevorV has joined #oooq14:18
mariossshnaidm: can you please review when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/761188 - I split the original, this one just adds the new template. plan is to wire it all up, then at the end we can merge https://review.opendev.org/c/openstack/tripleo-ci/+/769131/ to remove excess14:28
mariossshnaidm: topic branch there i will add more reviews https://review.opendev.org/q/topic:reduce-content-providers14:29
mariosrlandy: sshnaidm: weshay|ruck: arxcruz: zbr: ysandeep: rfolco: chandankumar: please review when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/761188 thank you14:36
ysandeepweshay|ruck, hey looks like Attila fixed the infrared job to trigger with extended hash, infrared job triggered today.. if all jobs passes we will promote today.. In today's morning run all tripleo-ci jobs passed..14:36
weshay|ruckmarios, thanks for your doc work.. https://f267574198a517c003d1-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf5.rackcdn.com/767375/7/check/openstack-tox-docs/8252ffe/docs/ci/stages-overview.html#the-component-promotion-pipeline14:36
weshay|ruck:)14:36
mariosweshay|ruck: ack i think its almost ready to merge thanks for checking14:36
weshay|ruckrlandy, rfolco you two should read through this https://f267574198a517c003d1-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf5.rackcdn.com/767375/7/check/openstack-tox-docs/8252ffe/docs/ci/stages-overview.html#the-integration-promotion-pipeline14:37
* rlandy looks14:37
mariosrlandy: https://review.opendev.org/c/openstack/tripleo-docs/+/767375 thanks14:37
rlandymarios: ack - shortly14:38
mariosrlandy: thx14:38
rlandyjust fixing ansible repo link14:38
mariosrlandy: no rush in your next reviews time thank you14:38
weshay|ruckmarios, re: https://review.opendev.org/q/topic:reduce-content-providers  is this something you'd like to present to the team in planning or work through ad-hoc14:46
mariosweshay|ruck: sure yes i think it is worth showing the order/thinking behind that. i have some notes at https://projects.engineering.redhat.com/browse/TRIPLEOCI-24914:47
weshay|ruckk.. thanks.. will get it on the agenda14:47
mariosweshay|ruck: but for now we need to agree and merge the first patch which is https://review.opendev.org/76118814:47
mariosweshay|ruck: thanks14:48
weshay|ruckchandankumar, sshnaidm as you have time.. ^ your review is important here14:50
mariosweshay|ruck: wrt your comment there ^^^ about test/dnm jobs with the patches needed see http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019522.html [tripleo] next meeting Tuesday Dec 22 @ 1400 UTC in #tripleo Please consider adding your news/blockers/plans in https://etherpad.opendev.org/p/tripleo-meeting-items even if you cannot attend.15:10
weshay|ruckaye.. thanks15:10
mariosweshay|ruck: sorry wrong paste... :/ see see comment there https://review.opendev.org/c/openstack/tripleo-ci/+/761188/7/zuul.d/upgrades-jobs-templates.yaml#9715:11
*** jmasud has joined #oooq15:22
*** ykarel_ has quit IRC15:26
mariosweshay|ruck: 1-1 time? we can push if you busy np15:30
weshay|ruckya.. let's chat15:30
*** rfolco has quit IRC15:31
ysandeepweshay|ruck, fyi.. periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2 in current run failed (Looks like random tempest failure) I reran it with https://code.engineering.redhat.com/gerrit/#/c/211643/ so that we can hopefully promote 16.2 today.15:31
*** jmasud has quit IRC15:40
rlandysshnaidm: weshay|ruck: so I think we need http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/playbooks/base/pre.yaml#n37 for 16.1 as well15:52
rlandyit's a config change15:52
rlandyso merge and pray15:52
rlandyhttp://git.host.prod.eng.bos.redhat.com/git/openstack/sf-config.git/tree/roles/openvswitch/vars/RedHat.yaml?id=60a1dbdb60e685cddd2202793f642f1ee12b10c315:52
*** ykarel_ has joined #oooq15:55
*** ykarel_ has quit IRC16:03
weshay|ruckanyone know how ens3 got back into the workflow? https://bugs.launchpad.net/tripleo/+bug/191010716:11
openstackLaunchpad bug 1910107 in tripleo "master/wallaby undercloud install failure unable to start network service" [Critical,Triaged]16:11
rlandyweshay|ruck: yep - 8.316:18
weshay|ruckya.. figures16:18
rlandysshnaidm: ping re: openvswitch versions16:18
*** rfolco has joined #oooq16:19
rlandyweshay|ruck: john fulton was talking about it before break16:19
rlandywe have 2.13 which should be correct according to rhos-delivery16:20
rlandyonly thing I can think of is the bridge is installed with diff version16:20
weshay|ruckk.. I'm not sure.. but the issue I'm seeing in centos ci.. is not related to ovs..16:21
weshay|ruckens3 is a native interface16:21
rlandyweshay|ruck: sshnaidm: not sure if this make sense to do :) ... but https://code.engineering.redhat.com/gerrit/222498 Use the openvswitch role for 16.116:21
rlandyweshay|ruck: looking at error16:22
*** udesale_ has quit IRC16:24
sshnaidmrlandy, thanks, looking16:27
rlandysshnaidm: we can merge that and try16:28
sshnaidmrlandy, ack, merging16:28
rlandysshnaidm: if it does nothing, we have a legit issue16:28
sshnaidmrlandy, ok, will rerun 16.1 job16:28
rlandyweshay|ruck: so .. ens3 ...16:30
rlandywe did a lot of work there when we started working with rhel 8.316:30
rlandyor I guess rhel 8 in general16:30
rlandyon baremetal16:30
rlandydepends what the ci.centso nodes look like16:31
rlandyI had that role included had to remove it as it was failing elsewhere16:31
* rlandy traces back16:32
rlandy<fultonj> Dec 10 13:19:42 undercloud.mydomain.tld network[45261]: Bringing up interface ens3-1:  Error: Connection activation failed: No suitable device found for this connection (device eth0 not ava>16:32
weshay|ruckya..16:33
weshay|rucknot sure why the undercloud is trying that now16:33
rlandyweshay|ruck: pvt sent you chat conversation16:34
weshay|ruckrlandy, k.. I'll check to make sure we're nuking it16:35
rlandyweshay|ruck: just be careful where you do that work - I got yelled at for adding it in a general place16:36
rlandyrevert fiasco16:36
sshnaidmchandankumar, arxcruz maybe you know, do we have somewhere ansible playbook that prepares devstack for tempest? Like creating users, roles, networks etc etc?16:42
chandankumarsshnaidm: there is a script which takes care of that https://opendev.org/openstack/devstack/src/branch/master/lib/tempest16:44
chandankumarembedded with in devstack itself16:45
sshnaidmchandankumar, yeah, I hoped there is ansible stuff..16:45
*** marios is now known as marios|out17:24
*** marios|out has quit IRC17:54
*** derekh has quit IRC18:02
rlandyweshay|ruck: can I merge this? https://code.engineering.redhat.com/gerrit/#/c/221454/ rhel8 container tools18:12
rlandylast successful run was dec 2318:12
rlandythe I can test rhel 8.4 over that18:12
weshay|ruckrlandy, can we chat about what you found w/ 8.4?18:13
rlandyweshay|ruck: nothing yet ... we can only really test 8.4 once https://code.engineering.redhat.com/gerrit/#/c/222354/ merges18:13
rlandyand we switch rhos-release params18:13
rlandyeither way container-tools should be unlocked18:14
weshay|ruckrlandy, k.. are we adding 8.4 to internal dep pipeline or prod jobs?18:14
rlandyweshay|ruck: going to ask fuzzball81 to merge https://code.engineering.redhat.com/gerrit/#/c/222354/ then I was going to runs some tests18:15
rlandyif it was seemless, I would switch prod18:15
rlandyif not, add a dep pipeline18:15
rlandya dep pipeline is just another thing to watch18:15
rlandyif the real deployment is going out on 8.418:16
rlandywe already ran the container build job there18:16
rlandyfor rhos-release18:16
weshay|ruckrlandy, can we chat? 8.4 is not prod18:16
weshay|ruckhttps://access.redhat.com/articles/307818:16
rlandyweshay|ruck: sure18:16
rlandyeither way we need the rhos-release stuff merged up18:17
weshay|ruckaye..18:17
weshay|ruckhttps://meet.google.com/axq-trne-kjy?authuser=118:18
rlandysshnaidm: did you use an ansible rpm to test ansible 2.10?18:42
weshay|ruckrlandy, https://anonbadger.wordpress.com/2020/08/25/why-upstream-ansible-stopped-shipping-rpms/18:50
rlandyhttps://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#upgrading-ansible-from-version-2-9-and-older-to-version-2-10-or-later18:52
*** jmasud has joined #oooq18:56
rlandyweshay|ruck: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit/files/compare_rpms/diff_tripleo_builds/rpm_skip_list18:59
sshnaidmrlandy, no, from pip19:00
rlandyyeah - seems to be the consensus19:00
rlandysshnaidm: looks like we still have the openvswitch tempest fail19:11
sshnaidmrlandy, yeah, though I'm not 100% sure it's openvswitch problem..19:13
sshnaidmI see some selinux errors as well..19:13
rlandysshnaidm: we can try set permissive on the deploy19:13
rlandyand see if that helps19:13
sshnaidmrlandy, yeah19:14
sshnaidmrlandy, how do we do it?19:14
rlandysshnaidm: standalone params, I think  - let me check19:14
rlandyI'll add that change19:14
sshnaidmrlandy, thanks19:15
sshnaidmrlandy, if it doesn't work, seems like broken..19:15
rlandysshnaidm: yeah - then we can BZ with confidence19:15
weshay|ruckrlandy, fyi https://opendev.org/openstack/tripleo-image-elements/src/branch/master/elements/interface-names/install.d/71-clean-stale-interface#L819:29
rlandyweshay|ruck: yeah - that's been there since we first started working with rhel8/centos819:35
rlandynot sure what went wrong with 8.3 images19:35
weshay|ruckaye19:35
rlandyI know the stream images are old19:35
weshay|ruckrlandy, seems like w/ 8.3 at boot the interface is recreated19:35
weshay|ruckit's our overcloud images19:35
rlandyweshay|ruck: way to backslide!19:36
rlandyours?19:36
weshay|ruckrlandy, check out the last two comments in https://bugs.launchpad.net/tripleo/+bug/191010719:36
openstackLaunchpad bug 1910107 in tripleo "master/wallaby undercloud install failure unable to start network service" [Critical,Triaged]19:36
rlandythen we should see it on OVB, no?19:36
weshay|ruckyes19:37
weshay|ruckrlandy, well.. sorry.. it's created at run time19:37
weshay|ruckhrm19:38
rlandysshnaidm: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/templates/overcloud-selinux-config.yaml.j219:45
rlandythink we just set overcloud_selinux_enabled to permissive19:46
rlandyhttps://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/defaults/main.yml#L19219:46
rlandyoverride that19:46
* rlandy tries19:46
*** rlandy is now known as rlandy|drappt20:18
*** holser has joined #oooq20:39
*** holser has quit IRC21:53
*** jmasud has quit IRC21:56
*** holser has joined #oooq22:12
*** jmasud has joined #oooq22:18
*** TrevorV has quit IRC22:37
*** rlandy|drappt is now known as rlandy22:46
rlandysshnaidm: hmmm ... maybe we need to run this by the tempest team ... https://sf.hosted.upshift.rdu2.redhat.com/logs/34/221234/7/check/tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.1/de5b5e7/logs/undercloud/home/zuul/overcloud-selinux-config.yaml22:51
rlandyshould be permissive now22:52
sshnaidmrlandy, heh.. I think maybe network team22:52
sshnaidmthis doesn't look good https://sf.hosted.upshift.rdu2.redhat.com/logs/34/221234/7/check/tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.1/de5b5e7/logs/undercloud/var/log/openvswitch/ovsdb-server.log22:53
sshnaidmI'll hold a node22:56
rlandysshnaidm: yep - possibly legit?22:56
sshnaidmseems like that22:56
*** jmasud has quit IRC23:46
*** jmasud has joined #oooq23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!