Tuesday, 2020-05-12

*** jmasud has quit IRC00:30
*** jmasud has joined #oooq00:34
*** Goneri has quit IRC00:47
*** jmasud has quit IRC01:00
*** rlandy has quit IRC01:17
*** sanjayu__ has joined #oooq01:18
*** sanjayu_ has quit IRC01:20
*** ysandeep|away is now known as ysandeep01:36
*** apetrich has quit IRC02:10
*** jmasud has joined #oooq03:58
*** ratailor has joined #oooq04:06
*** holser has joined #oooq04:35
*** jmasud has quit IRC04:36
*** ykarel|away is now known as ykarel04:41
*** skramaja has joined #oooq04:43
*** marios has joined #oooq05:15
*** udesale has joined #oooq05:35
*** jfrancoa has joined #oooq05:36
*** bogdando has joined #oooq05:44
*** dpawlik has joined #oooq06:05
*** jmasud has joined #oooq06:11
*** jmasud has quit IRC06:17
*** jmasud has joined #oooq06:18
*** ccamacho has joined #oooq06:56
*** ccamacho has quit IRC06:56
*** jbadiapa has joined #oooq07:05
bhagyashris|ruckchandankumar, hi good noon ! have you submitted any patch to avoid the failure of "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master"07:06
*** ccamacho has joined #oooq07:13
*** sanjayu__ has quit IRC07:16
*** jmasud has quit IRC07:17
ysandeepbhagyashris|ruck, may be this is what you are looking for - https://review.opendev.org/#/c/722677/07:17
*** saneax has joined #oooq07:20
*** apetrich has joined #oooq07:23
*** tosky has joined #oooq07:35
bhagyashris|ruckysandeep, yes thank you :)07:44
*** dtantsur|afk is now known as dtantsur07:54
*** jpena|off is now known as jpena07:56
*** holser has quit IRC07:59
*** holser has joined #oooq07:59
chandankumardpawlik, Hello08:06
chandankumardpawlik, somehow rsycn mkdir is not working for creating dir on image server08:07
chandankumardpawlik, https://logserver.rdoproject.org/20/26920/21/check/periodic-tripleo-centos-8-buildimage-overcloud-full-train/b550d93/job-output.txt08:07
chandankumardpawlik, rsync: mkdir "/var/www/html/images/centos8/train/rdo_trunk" failed: No such file or directory (2)08:07
chandankumardpawlik, we need train dir here https://images.rdoproject.org/centos8/08:08
chandankumardpawlik, please have a look thanks, it is blocking c8 train pipeline08:09
dpawlikchandankumar, hi08:09
dpawlikchandankumar, /me checking08:09
chandankumarhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/upload-cloud-images.sh is used for uploading images08:10
dpawlikchandankumar, [uploader@images centos8]$ mkdir -p /var/www/html/images/centos8/train/rdo_trunk08:11
dpawlikwithout any problem08:11
dpawlikits beacause train dir was not there. Could you add please "-p" param to the mkdir?08:11
chandankumardpawlik, yup08:12
dpawlikchandankumar, strange, that rsync doesn't want to create the dir... but the same user is able to create directory via command08:13
dpawlikchandankumar, maybe something is missing in rsync command when I was working on patching the permission problem08:13
chandankumarmarios, dpawlik quick fix https://review.rdoproject.org/r/27427 for above issue ^^08:15
dpawlikchandankumar, there was also another script that can have same issue, or?08:16
dpawlikafair08:16
*** ykarel is now known as ykarel|lunch08:22
marioschandankumar: ack08:24
bhagyashris|ruckchandankumar, can vote for this https://review.opendev.org/#/c/726993/08:27
bhagyashris|ruck+w08:27
chandankumardpawlik, need to check08:28
chandankumardpawlik, first let me update the c8 patch and see whether it is working or not08:28
dpawlikokey08:28
*** ysandeep is now known as ysandeep|lunch08:31
pandamarios: Ψ time08:32
pandabhagyashris|ruck: I had to fix https://review.opendev.org/#/c/722677/, apparently the configuration option needs a description (why can't they get it from the driver itself, I don't know)08:34
bhagyashris|ruckpanda, ok08:36
pandamarios: :(08:38
mariospanda: sorry !08:38
mariospanda: i was staring at get_hash08:38
marioscoming08:38
kopecmartinchandankumar: hi where in the logs of featureset20 can i find, what tempest version was used to run tests? because all tests passed, just 1 neutron plugin tests failed, which is unrelated08:41
chandankumarkopecmartin, review link?08:41
kopecmartinchandankumar: so if the new tempest version was tested, we can bump as the tests passed https://review.rdoproject.org/r/#/c/26769/08:41
chandankumarkopecmartin, https://logserver.rdoproject.org/69/26769/4/check/distgit-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset020-stein/63b1232/logs/undercloud/var/log/extra/rpm-list.txt.gz08:42
kopecmartinchandankumar: there's no tempest, that was the first place i looked08:43
*** derekh has joined #oooq08:43
bhagyashris|ruckpanda, can you point me where exactly you are facing issue about description08:44
chandankumarkopecmartin, https://logserver.rdoproject.org/69/26769/4/check/distgit-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset020-stein/63b1232/logs/undercloud/home/zuul/tempest.log.txt.gz08:44
chandankumarpython2-tempest-23.0.0-0.20200511122550.d432237.el7.noarch08:45
chandankumar2020-05-11 16:00:52 | python2-neutron-tests-tempest-0.9.0-0.20200330151622.15c85f6.el7.noarch08:45
chandankumarkopecmartin, tempest is running from container so the info is available in tempest.log08:45
toskychandankumar: out of curiosity, is it running in a centos8/rhel8 container?08:46
chandankumarbhagyashris|ruck, do you have link to tempest failure bug on stein ?08:46
chandankumartosky, nope, it is on stein, so it is running c7 container08:46
toskyoh, oki08:46
toskychandankumar: but08:46
chandankumartosky, from train, we are using packages installed via os_tempest08:46
toskyok, right, that's the last version which supports python 208:48
bhagyashris|ruckchandankumar, https://bugs.launchpad.net/tripleo/+bug/187608708:48
openstackLaunchpad bug 1876087 in tripleo "Queens, Rocky, Stein, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout" [Critical,Fix released] - Assigned to amolkahat (amolkahat)08:48
pandabhagyashris|ruck: I can, but I won't, because I'm evil08:48
chandankumarkopecmartin, https://bugs.launchpad.net/tripleo/+bug/1876087 is failing only08:48
chandankumarif it is passing there, we can go ahead and update the bump08:48
pandabhagyashris|ruck: https://64d1a84195be846e8a85-cb762a113d4d5e3960c90c07c5780a8b.ssl.cf2.rackcdn.com/722677/2/check/tripleo-ci-centos-8-scenario010-ovn-provider-standalone/b3f2294/logs/undercloud/home/zuul/tempest.log08:48
pandabhagyashris|ruck: oslo_config.cfg.ConfigFileValueError: Value for option enabled_provider_drivers from LocationInfo(location=<Locations.user: (4, True)>, detail='/home/zuul/tempest/etc/tempest.conf') is not valid: Value should be NAME:VALUE pairs separated by ","08:49
pandabhagyashris|ruck: https://docs.openstack.org/octavia/latest/configuration/configref.html in section enabled_provider_drivers08:49
kopecmartinchandankumar: i see, thanks, so i confirm that the tempest 23 has been tested08:50
pandabhagyashris|ruck: it tells it needs a list of dictionaries.08:50
kopecmartinchandankumar: the test mentioned in https://bugs.launchpad.net/tripleo/+bug/1876087 is passing08:50
openstackLaunchpad bug 1876087 in tripleo "Queens, Rocky, Stein, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout" [Critical,Fix released] - Assigned to amolkahat (amolkahat)08:50
pandanever heard of a configuration that embeds description for a driver that is part of the code08:50
kopecmartinchandankumar: only failing test is from neutron plugin - neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_multiple_ports_portrange_remote08:50
*** ykarel|lunch is now known as ykarel08:51
chandankumarkopecmartin, does it was passing earlier?08:52
*** sshnaidm|afk is now known as sshnaidm08:52
kopecmartinchandankumar: yes, when you check older gate runs the tests passed there .. it's a timeout issue so it doesn't have to occur all the time?08:53
chandankumarkopecmartin, ok, let's just open a bug and go with bump08:54
kopecmartinchandankumar: ok, these two reviews are needed for the bump  https://review.rdoproject.org/r/#/c/26777/ https://review.rdoproject.org/r/#/c/26769/08:54
chandankumarmarios, Hello09:12
chandankumarmarios, do we have a patch up for fs039 with to_build flag?09:12
marioschandankumar: o/ no i didn't post something there09:14
pandamarios: do you know what's the public net to use to assign floating IPs in  Ψ  ?09:14
chandankumarmarios, let me put a patch up and see how it goes09:14
chandankumarpanda, what is  Ψ ?09:14
chandankumarit appears to shiva's trident09:15
pandachandankumar: https://en.wikipedia.org/wiki/Psi_(Greek)09:15
mariospanda: yeah let me check my instance sec09:16
mariospanda: pvt09:16
chandankumarpanda, ah great, thanks :-)09:17
chandankumardpawlik, image issue fixed now thanks!09:19
*** ysandeep|lunch is now known as ysandeep09:25
*** skramaja has quit IRC09:37
*** skramaja_ has joined #oooq09:37
chandankumarmarios, panda zbr sshnaidm weshay|ruck rdo-cloud will be staying till may end, may 6 date got postponed, we will get new dates in June09:49
marioschandankumar: ack09:49
pandachandankumar: yey!09:49
sshnaidmchandankumar, cool09:50
sshnaidmwe have 2 clouds in parallel09:50
sshnaidmlet's go nuts09:50
chandankumarsshnaidm, yes09:51
chandankumarsshnaidm, regarding this task https://tree.taiga.io/project/tripleo-ci-board/task/1716 jpena will create a user story on this board to track and mhu has some plans on that09:53
chandankumar*their09:53
sshnaidmchandankumar, great, hope they will find out somehting09:54
bhagyashris|ruckpanda, i see in one place they have mentioned enabled_providerd_driver like this https://opendev.org/openstack/ovn-octavia-provider/src/branch/master/zuul.d/base.yaml10:06
bhagyashris|ruckpanda, i think whatever you defined seems correct to me10:07
bhagyashris|ruckpanda, not sure is this causing failure or not10:08
*** ysandeep is now known as ysandeep|brb10:09
*** udesale_ has joined #oooq10:18
*** dpawlik has quit IRC10:21
*** udesale has quit IRC10:21
*** dpawlik has joined #oooq10:21
chandankumarmarios, running fs039 here https://review.rdoproject.org/r/#/c/27431/10:22
marioschandankumar: ack10:27
*** bogdando has left #oooq11:00
*** ysandeep|brb is now known as ysandeep11:11
*** rfolco has joined #oooq11:16
*** jpena is now known as jpena|lunch11:33
*** chem has quit IRC11:48
*** chem has joined #oooq11:53
*** amoralej is now known as amoralej|lunch11:58
*** skramaja_ is now known as skramaja12:06
chandankumarbhagyashris|ruck, does on c7 train promotion, sc4 and sc7 passing?12:06
*** ratailor has quit IRC12:07
bhagyashris|ruckchandankumar, c7 sc7 is non voting but it's passing https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-scenario007-standalone-train12:08
bhagyashris|ruckchandankumar, sc4 is also passing https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-scenario004-standalone-train12:09
chandankumarbhagyashris|ruck, ok thanks it is failing on c8 looking into it12:09
bhagyashris|ruckchandankumar, ok12:09
pandamarios: are you still using your instance mariossprint24task1597 ?12:13
mariospanda: not right now am not ... do you need it?12:13
chandankumarweshay|ruck, Hola around?12:13
pandamarios: just as a gateway to access the internal promoter, I can't allocate more fips12:14
mariospanda: sure np go ahead i think you should be able to get onto it12:14
pandamarios: I would maybe need to install ansible a copy a few files12:14
mariospanda: do whatever you want12:14
pandamarios: ok, rm -rf /12:14
mariospanda: yes, would it help if i deleted it? do we have a quota issue i mean?12:14
mariospanda: would you be able to create a new one?12:15
pandamarios: no, we have 5 IP and a quota of 10012:15
pandamarios: I opened a ticket.12:15
weshay|ruckchandankumar, 0/12:15
mariospanda: k12:15
chandankumarweshay|ruck, on train c8, on standalone side sc7 and sc4 is failing12:15
chandankumarfs01 is running12:15
weshay|ruckk.. ya.. I was wondering if we needed to change ceph at all for sc412:16
pandamarios: I'm in, thanks12:16
chandankumarhttps://logserver.rdoproject.org/20/26920/22/check/periodic-tripleo-ci-centos-8-scenario004-standalone-train/344fbcf/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz12:16
mariospanda: ack12:16
weshay|ruckchandankumar,         "<13>May 12 10:40:54 puppet-user: Error: Facter: error while resolving custom fact \"rabbitmq_nodename\": undefined method `[]' for nil:NilClass",12:17
*** rlandy has joined #oooq12:18
*** ratailor has joined #oooq12:20
chandankumarweshay|ruck, yes it is for sc4 and sc7 router is not able to pingable12:21
rlandymarios: hey12:22
rlandymarios: see we got a fix: https://review.opendev.org/#/c/727087/12:22
rlandymarios: I merged the pipeline and it ran12:23
chandankumarweshay|ruck, once the job finishes, I will add those c8 train jobs to pipeline12:24
mariosrlandy: o/ hey12:26
*** rfolco is now known as rfolco|rover12:27
mariosrlandy: yeah so the job ran in the pipeline but was faining on reposetup get hash ... so i spent time debugging that (long story short wasted time) cos i posted https://review.opendev.org/727146 but then found your abandoned one https://review.opendev.org/725974 and finally the proper one at https://review.opendev.org/#/c/720776 :/12:27
mariosrlandy: running in test now at https://code.engineering.redhat.com/gerrit/#/c/200050/ (with https://review.opendev.org/725974 which ill re-abandon after the run finishes)12:27
mariosrlandy: once it hits the bug we can request a hold cos damien wanted to check something on there for the mysql fix12:27
mariosrlandy: want to get on phone in half hour?12:28
*** TrevorV has joined #oooq12:28
*** Goneri has joined #oooq12:29
weshay|ruckzbr, https://meet.google.com/kdr-uzwf-yss?authuser=112:31
bhagyashris|ruckpanda, can you rebase this https://review.rdoproject.org/r/#/c/26694/ as it's in merge conflict12:33
*** ratailor has quit IRC12:33
*** zbr_ has joined #oooq12:35
*** zbr_ has quit IRC12:36
rlandymarios: sure - thanks12:38
rlandypojadhav: weshay|ruck; the final tls patch merged last night :)12:38
*** jpena|lunch is now known as jpena12:40
rlandymarios: mind w+'ing https://review.opendev.org/#/c/725253?12:41
*** Goneri has quit IRC12:41
mariosrlandy: ack12:42
rlandythank you12:42
rlandypojadhav: ^^ both your trigger patches are w+'ed12:44
*** Goneri has joined #oooq12:47
rlandymarios; it looks like the deploy past the bug12:47
rlandyugh12:47
rlandyit didn't12:48
mariosrlandy: yeah just failed the deploy waiting for log (but i also need to change the depends-on to use the 'proper fix'12:48
mariosrlandy: i mean for the get hash12:48
rlandymarios: ok - when you start  the job, I'll get that node held12:49
mariosrlandy: i'll also add damien fix there12:49
rlandycool12:49
mariosbrb12:49
mariosrlandy: oh wait... i think we need it to fail for the hold? i.e. if we add the fix there it wont fail and we can't hold it? i think that's the case12:52
pojadhavrlandy, awsome :)12:53
chandankumarrlandy, marios https://review.rdoproject.org/r/#/c/26909/ and then this https://review.rdoproject.org/r/2690812:54
rlandymarios: no need for it to fail12:54
rlandywe can hold it as soon as the job starts12:54
chandankumarplease have a look12:54
mariosrlandy: ok cool then i'll add the change (i recall this from rdo zuul but maybe am confusing different things)12:55
rlandychandankumar: need to change the name of your image jobs from master -> rhos-1712:55
rlandyalso, where are we with the image server?12:55
rlandychandankumar: worth it to ping apevec again?12:55
mariosrlandy: not sure if fbo is in btw going to rhos-ops12:56
chandankumarrlandy, sure pinging him12:56
rlandymarios: nhicher can do that for us12:56
rlandychandankumar: thanks - I'll change the job names later12:57
mariosrlandy: k asked him12:58
rlandygreat12:58
marioschandankumar: ack noting for check later12:58
sshnaidmweshay|ruck, mtg?13:02
*** TrevorV has quit IRC13:02
*** amoralej|lunch is now known as amoralej13:02
*** TrevorV has joined #oooq13:03
*** TrevorV has quit IRC13:05
rlandychandankumar: marios: hey - could you check the current-tripleo release file? - I merged it to try a component job - but can fix in a follow up patch ... http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-17.yml13:05
ykareli noticed rhel8 build containers still running, shouldn't these be cleaned considering it's broken and centos8 train is being prepared13:06
ykarelweshay|ruck, bhagyashris|ruck ^^13:06
ykarelhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-build-containers-rhel-8-train&job_name=tripleo-build-containers-rhel-8-train-branch#13:06
*** TrevorV has joined #oooq13:06
*** sanjayu_ has joined #oooq13:10
*** saneax has quit IRC13:12
chandankumarrlandy, rhos-17 release file looks good, tempest_container_namespace: openstack-tempest13:14
chandankumartempest_container_tag: "{{ container_build_id }}" not needed, as tempest container not used for running tempest13:14
rlandychandankumar: thanks13:14
*** chem has quit IRC13:15
*** chem has joined #oooq13:17
rlandymarsio; I wonder of that needs to be on the skip list13:19
rlandymarios: ^^13:20
rlandychandankumar: build-test-packages ... failing on a depends-on change from puppet-tripleo13:20
rlandyhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L6213:20
rlandyshould that repo be included in ^^?13:20
bhagyashris|ruckykarel, periodic version of those jobs ^ are not present thats why may be those are not notice but i think we can remove that13:21
chandankumarrlandy, I think yes13:21
rlandychandankumar: stupid question ...13:21
chandankumaror rdo-jobs should take care of that13:21
rlandyhow do we get the depends-on change then?13:21
chandankumarsorry build test packages will take care of that via rdopkg13:21
ykarelbhagyashris|ruck, yes please remove, periodic ones were removed earlier, so these are likely missed13:22
bhagyashris|ruckykarel, ack13:22
rlandychandankumar: ok - we get http://pastebin.test.redhat.com/86408313:22
rlandymarios: ok - we need another patch for that job13:22
rlandymarios: to add puppet-tripleo to skip list13:23
rlandylet's run once to see if it works13:23
rlandythen we can run again and ask to hold node13:23
chandankumar-info-repo needs to be changed to ospinfo13:23
chandankumarnot rdoinfo13:23
weshay|ruckysandeep, sshnaidm is going to contact you for a usability review of the reproducer13:24
rfolco|roverpanda, did you run a testproject for ovn job fix ?13:24
rfolco|roverhttps://review.rdoproject.org/r/#/c/26694/13:24
rfolco|roverpanda, ^13:24
mariosrlandy: ack are you posting it?13:24
weshay|ruckysandeep, ping sshnaidm w/ questions or concerns13:24
rlandyoh gee13:24
pandarfolco|rover: no13:24
rlandymarios: we're testing an upstream change with a downstream job13:24
rlandyhow can that work?13:25
ysandeepweshay|ruck, sure o/13:25
rfolco|roverpanda, ok, thanks13:25
pandarfolco|rover: this was supposet to be aquick fix :(13:25
chandankumarrlandy, https://code.engineering.redhat.com/gerrit/gitweb?p=ospinfo.git;a=tree;h=refs/heads/master;hb=refs/heads/master13:25
pandarfolco|rover: and it has become a nightmare.13:25
rfolco|roverpanda, why13:25
rfolco|roverdon't see any objections or concerns in the patch13:25
mariosrlandy: ah... i mean if it was a case of applying it it *could* work i.e. they should be pretty up to date but might not be if there are rebase issues13:26
rfolco|roverit just didn't get merged, isn't it ?13:26
rfolco|roverpanda, ^13:26
pandarfolco|rover: because an obscure conf option was missing in the config option itself13:26
pandaand now it's taking a lot for the jobs13:26
rlandymarios: let's rethink this ...13:27
rlandymarios: pupper-tripleo - is there a downstream equivalent?13:27
mariosrlandy: dlrn urls will be different too for buid-test-packages ?13:27
mariosrlandy: i think there must be13:27
mariosrlandy: checking13:27
rlandythen we need to apply that fix to the downstream repo13:28
chandankumarrlandy, puppet-tripleo https://code.engineering.redhat.com/gerrit/gitweb?p=ospinfo.git;a=blob;f=packages.yml;h=be9ab79f75f11b31bf85689b686bfa988b575ea2;hb=refs/heads/master#l580513:29
mariosrlandy: yeah there is13:29
rfolco|roverarxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:30
rfolco|rovercommunity mtg13:30
mariosrlandy: lets carry on after calls13:30
rfolco|roverping arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar13:30
rfolco|rovercommunity mtg13:30
chandankumarweshay|ruck, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train https://review.rdoproject.org/zuul/build/2b2f7ccd17e54eeb9c208197a18d84fb : SUCCESS in 3h 14m 04s13:37
weshay|ruckoh fuck ya!13:38
zbram i wrong or google is forcing me to user chrome in order to use meet?13:41
pandazbr: I am connected from firefox13:42
zbrbut i wanted a standalone app, not a browser.13:42
chandankumarweshay|ruck, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/defaults/main.yml#L613:43
zbrso I can easily kill it, not having a hidden tab somewhere streaming how I sleep during the day13:43
zbrand fun, apparently there is no option to signout from a single account, only from all.13:45
*** openstackstatus has quit IRC13:53
*** openstackstatus has joined #oooq13:53
*** ChanServ sets mode: +v openstackstatus13:53
pandamarios: what's the starting data of your script? just repos ?14:00
mariospanda: yeah it checks out the repos and uses grep basically14:02
sshnaidmrlandy, can you take a look please in your time: https://review.opendev.org/#/c/726847/14:02
sshnaidmrlandy, if possible, would be nice to have dw job on htat14:02
sshnaidmI mean bm14:02
rlandysshnaidm: ok - will do a run on that this afternoon14:06
mariosrlandy: ykarel: re-added your keys now to the new node @ zuul@10.0.109.12514:06
weshay|ruckchandankumar, when's a good time to sync about rdopkg/train.. don't want to overload you w/ stupid mtgs14:19
ykarelmarios, ack i can access14:24
*** ysandeep is now known as ysandeep|afk14:26
chandankumarweshay|ruck, we can sync right now14:36
weshay|ruckchandankumar, k.. https://meet.google.com/xda-grzq-izz14:36
chandankumarweshay|ruck, https://review.opendev.org/#/q/topic:buildtestpackages+(status:open+OR+status:merged)14:37
chandankumarweshay|ruck, https://hackmd.io/LuRWE2oESKiAUQQw6OYh6w14:39
*** dtantsur is now known as dtantsur|afk15:08
*** ysandeep|afk is now known as ysandeep15:11
weshay|rucksshnaidm, https://review.opendev.org/#/c/727227/15:23
sshnaidmweshay|ruck, broken?15:24
weshay|rucksshnaidm, https://bugs.launchpad.net/tripleo/+bug/187810115:24
openstackLaunchpad bug 1878101 in tripleo "ping br-ctlplane is failing too often, "Trying to ping default gateway"" [Critical,In progress] - Assigned to wes hayutin (weshayutin)15:24
arxcruz rlandy around?15:28
arxcruzrlandy: I'm getting    Unknown project opendev.org/openstack/openstack-tempest-skiplist15:28
arxcruzrlandy: in https://review.opendev.org/#/c/726843/15:28
rlandyarxcruz: where are you getting that error?15:28
rlandylooking15:28
arxcruzrlandy: on rdo side15:28
rlandyarxcruz: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul/rdo.yaml#L108215:30
rlandyneeds to be added to projects that can be included15:30
arxcruzrlandy: ok, thanks15:30
*** jmasud has joined #oooq15:35
*** skramaja has quit IRC15:52
*** marios is now known as marios|out16:07
weshay|ruckrfolco|rover, only thing a bit whacky atm.. is the upstream gate..16:26
rfolco|roveryup16:26
weshay|ruckrfolco|rover, going to look at the criteria for releases and make sure they are right..16:26
weshay|ruckrfolco|rover, hot issues to track https://trello.com/c/E7gL6d4b/1490-cixlp1878101tripleociproa-ping-br-ctlplane-is-failing-too-often-trying-to-ping-default-gateway16:27
rfolco|roverweshay|ruck, besides the jobs we moved to nv, is there anything else we can do to make gate great again?16:27
weshay|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades16:28
weshay|ruckrfolco|rover, ya.. try and drive a fix via cix16:28
weshay|ruckthe gate is mostly random atm.. very unstable compared to early last week and several weeks prior16:29
*** marios|out has quit IRC16:29
*** ccamacho has quit IRC16:31
*** holser has quit IRC16:31
*** holser has joined #oooq16:33
weshay|ruckzbr++ thanks for https://review.opendev.org/#/c/727113/16:38
*** dpawlik has quit IRC16:40
*** udesale_ has quit IRC16:41
*** amoralej is now known as amoralej|off16:42
zbrweshay|ruck: funny part that the correct fix was removal in that case.16:46
zbrmainly it failed, but it was not linting anything there.16:46
zbrping me if there are other places where we see failures.16:46
weshay|ruckk.. thanks16:47
*** ykarel is now known as ykarel|away16:55
weshay|rucksshnaidm, just confirming what I see16:58
weshay|ruckhttps://review.rdoproject.org/r/#/q/owner:%22Sagi+Shnaidman+%253Csshnaidm%2540redhat.com%253E%2216:58
weshay|ruckeveyrthing is on vexx now?16:58
sshnaidmweshay|ruck, promotion?17:00
sshnaidmweshay|ruck, all promotion jobs are there, yes17:00
weshay|ruckcool17:00
weshay|rucklooks like component as well?17:00
sshnaidmweshay|ruck, yep17:00
weshay|rucksshnaidm++17:01
sshnaidmweshay|ruck, didn't move yet github jobs, reproducer jobs, etc17:01
weshay|ruckrfolco|rover, fyi.. NetworkSecGroupTest is failing across stein and train e.g. https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/68f40d9/logs/tempest.html.gz17:01
weshay|ruckworth lp, skip, cix17:01
weshay|rucksshnaidm, aye17:01
weshay|rucksshnaidm, how about the other 3rd party ci.. 35, 3917:01
sshnaidmweshay|ruck, 4th party ci? :)\17:02
weshay|rucklol17:02
rfolco|roverweshay|ruck, ack, adding to skip list17:03
weshay|ruckrfolco|rover, need a lp first :)17:03
*** derekh has quit IRC17:03
rfolco|roverweshay|ruck, I am just checking if fs020 is the same as reported by bhagyashris|ruck17:03
rfolco|roverTestNetworkAdvancedServerOps17:05
rfolco|roverlooks diff17:05
rfolco|roverok17:05
bhagyashris|ruckrfolco|rover, this is diff17:05
*** jpena is now known as jpena|off17:14
bhagyashris|ruckrfolco|rover, this job is consistently failing on stein only17:15
rfolco|roversometimes on train17:15
bhagyashris|ruckrfolco|rover, but very few  times and error is also diff17:18
bhagyashris|ruckon train17:18
*** ysandeep is now known as ysandeep|away17:19
*** apetrich has quit IRC17:20
bhagyashris|ruckrfolco|rover, ahh i see in more detail both are same17:21
bhagyashris|ruckmy bad17:21
*** yolanda has quit IRC17:35
rfolco|roverweshay|ruck, bhagyashris|ruck  https://review.opendev.org/727287 [skiplist] NetworkSecGroupTest timeout17:47
rlandyweshay|ruck: so I read through the new container generation tools spec - after today's meeting - with interest in the component pipeline testing piece17:58
rlandypls ping me if/when you have a moment to discuss17:58
rlandyie: if we should start with this approach downstream17:59
rlandyirrespective of when reldel picks it up17:59
*** jmasud has quit IRC18:16
chandankumarrlandy, I am back now, Anything I can help to update this piece https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/tasks/main.yml#L206-L211 for downstream bits18:18
*** jmasud has joined #oooq18:18
rlandychandankumar: hey18:19
rlandyso a couple of thins18:19
rlandythings18:19
rlandy1, I think that project is not added to projects that can be included in zuul18:20
rlandychecking that18:20
rlandychandankumar: also, only changes to branch master are picked up18:21
rlandychandankumar: build-test-packages would have to be able to pick up changes on diff branches18:21
rlandychandankumar: <jpena> rlandy: I'd try to use the zuul dict, instead of the legacy env vars18:22
rlandychandankumar: this will require some work to build-test-packages18:22
rlandywhich we may or may not want to put in18:22
rlandyI don't see how else we check downstream changes18:22
rlandy        - puppet-tripleo:18:23
rlandy            zuul/include: []18:23
rlandy            repoxplorer/skip: true18:23
rlandyactually it is there18:23
rlandyso then we need the branch thing18:24
rlandychandankumar: ^^18:24
chandankumarrlandy, I am checking this https://sf.hosted.upshift.rdu2.redhat.com/logs/84/200084/3/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/3322e5b/logs/zuul-variables.yaml18:25
chandankumarrlandy, branch: rhos-17.0-trunk-patches18:25
rlandychandankumar: correct - zuul knows about the right branch18:25
rlandybut build-test-packages only picks up master changes18:26
rlandy 2020-05-12 10:55:23.475381 | primary | ZUUL_CHANGES=openstack/tripleo-quickstart:master:refs/changes/76/720776/18^openstack/tripleo-quickstart-extras:master:refs/changes/00/727200/2^openstack/tripleo-ci:master:refs/changes/96/727196/1^openstack/tripleo-ci-internal-jobs:master:refs/changes/84/200084/118:26
rlandychandankumar: ^^ that is the missing piece18:26
rlandychandankumar: hence jpena's suggestion ...18:27
rlandy<jpena> and it looks like it is trying to use that env var to build the list of changes18:27
rlandy<rlandy> jpena: zuul/dlrn is doing fine - our role needs work18:27
rlandy<jpena> rlandy: I'd try to use the zuul dict, instead of the legacy env vars18:27
chandankumarrlandy, let me check how to use zuul dict18:33
rlandychandankumar: see zuul.items18:34
rlandybranch change etc.18:35
*** jmasud has quit IRC18:42
chandankumarrlandy, how to get this one refs/changes/84/200084/1 ?18:47
rlandylooking18:48
*** jmasud has joined #oooq18:50
rlandychandankumar:  refs/changes/zuul.items[i].change{last two digits}/zuul.items[i].change/zuul.items[i].patchset18:51
rlandy ref: refs/changes/84/200084/318:52
rlandychandankumar: ^^ the patch itself is actually there18:52
chandankumarrlandy, not sure it will work https://review.opendev.org/#/c/727200/3/roles/build-test-packages/tasks/main.yml18:57
rlandychandankumar: let's try it18:57
chandankumarrlandy, rebased the downstream patch18:57
rlandychandankumar++18:58
chandankumarnot sure about loop dict part18:58
rlandywe'll see18:58
rlandywe can fix it of need be18:58
rlandychandankumar: it's late for you - I can pick this up19:15
rlandyor we can continue tomorrow19:15
* rlandy just merges pipeline changes19:16
*** apetrich has joined #oooq19:17
weshay|ruckugh.. 1.5 hour debug on undercloud install19:17
weshay|ruckrlandy, os-net-config ugh19:18
rlandycharming19:18
rlandyweshay|ruck: no rush - we can chat at 1-on-1 tomorrow19:18
*** apetrich has quit IRC19:18
weshay|ruckrfolco|rover, fyi.. that undercloud-containers bug sucks19:19
rfolco|roverweshay|ruck, why19:19
*** apetrich has joined #oooq19:19
chandankumarrlandy, I got an idea19:19
chandankumarrlandy, we donot need this script at all https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/library/zuul_deps.py#L6019:20
chandankumarrlandy, we just need a list19:20
chandankumarrlandy, we can construct that using anible19:20
rlandychandankumar: ok19:20
rfolco|roverweshay|ruck, all I could find is that ctlplane is weird https://trello.com/c/E7gL6d4b/1490-cixlp1878101tripleociproa-ping-br-ctlplane-is-failing-too-often-trying-to-ping-default-gateway19:22
weshay|ruckrfolco|rover, comments https://1ee15b38e5321bd48ce9-8aefe49572c9ad91eb217e6fb5236bb6.ssl.cf2.rackcdn.com/712464/20/gate/tripleo-ci-centos-8-undercloud-containers/0639b3e/logs/undercloud/var/lib/tripleo-config/scripts/19:22
*** dsneddon has joined #oooq19:32
chandankumarrlandy, now see https://review.opendev.org/#/c/727200/4/roles/build-test-packages/tasks/main.yml feel free to update it19:36
*** jmasud has quit IRC19:50
weshay|ruckrfolco|rover, put comments on the skip list review19:54
*** TrevorV has quit IRC19:54
rfolco|roverweshay|ruck, thx, will remove train19:56
rfolco|roverweshay|ruck, help on prog call report, can we say we are green in the middle of the storm ?19:58
rlandychandankumar: thanks19:59
weshay|ruckrfolco|rover, status is for train / osp-1619:59
weshay|ruckrfolco|rover, the status you report are the dates for tripleo and rdo promotion based on http://rhos-release.virt.bos.redhat.com:3030/rhosp20:00
rfolco|roverweshay|ruck, green. period.20:00
weshay|ruckrfolco|rover, you can add a note about upstream gate issues, but they won't care about that20:00
rfolco|roverno thanks :)20:00
weshay|ruckrfolco|rover, re: the undercloud-containers bug..20:02
weshay|ruckthe code is what's so nuts.. old old tripleo bits20:02
weshay|ruckfollow https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/net-config-undercloud.j2.yaml -> https://opendev.org/openstack/tripleo-heat-templates/blame/branch/master/network/scripts/run-os-net-config.sh -> https://github.com/openstack/os-net-config20:03
weshay|ruckrlandy, stamp https://review.opendev.org/#/c/727287/20:04
rlandyweshay|ruck: stamped20:04
weshay|ruckthanks20:05
rfolco|roverweshay|ruck, for some reason ctlplane is misconfigured with physical dev ip while it should be the bridge one20:08
rfolco|roverweshay|ruck, looking at the code20:08
weshay|ruckya.. you get a dupe ip20:09
weshay|ruckbut some nodes have two network devices and some have just one20:09
rfolco|roverthis is not the problem. I compared good runs and bad ones w/ single nic20:11
*** dpawlik has joined #oooq20:17
*** sshnaidm is now known as sshnaidm|afk20:18
weshay|ruckrfolco|rover, have you seen any undercloud-containers centos-8 job fail that had two nics?20:18
weshay|ruckjust checking  because I have not20:19
weshay|ruckrfolco|rover, good news.. is the gate seems to be stablizing20:28
rfolco|roverweshay|ruck, I think I saw one job w/ 2 nics that failed... let me find it20:33
rfolco|roverweshay|ruck, 2 nic failure https://6c0545a7fde0c843fe58-c94b7832c8857c8bbfdc284b48fd4077.ssl.cf2.rackcdn.com/722275/3/check/tripleo-ci-centos-8-undercloud-containers/3665c4e/logs/undercloud/var/log/extra/network.txt20:41
weshay|ruckah.. look at that20:42
*** jmasud has joined #oooq20:42
weshay|ruckthanks20:42
* weshay|ruck adds to bug..20:42
weshay|ruckrfolco|rover, keep the technical bits in the bug.. generally.. use the trello board for nags, handoff, team communication etc.20:43
*** sanjayu_ has quit IRC20:43
rfolco|roverweshay|ruck, ok20:43
weshay|ruckrfolco|rover, add your find to https://bugs.launchpad.net/tripleo/+bug/187810120:44
openstackLaunchpad bug 1878101 in tripleo "ping br-ctlplane is failing too often, "Trying to ping default gateway"" [Critical,In progress] - Assigned to wes hayutin (weshayutin)20:44
weshay|ruckrfolco|rover, tomorrow can we build a test project job that launches like 5 undercloud-containers deployments w/ kevin's patch?20:44
rfolco|roverweshay|ruck, I can prepare one now if you link me to it20:51
weshay|ruckrfolco|rover, in the bug20:51
rfolco|roverah of course20:51
weshay|ruckrfolco|rover, k.. just don't want to keep you too late.. so just looking for 5 dupe jobs slightly diff name I guess.. but just to blast away w/ that patch20:52
weshay|rucksince we're hitting something that is inconsistent20:52
rfolco|roverwhy 5 diff ?20:53
rfolco|roverweshay|ruck, rationale ?20:53
weshay|ruckrfolco|rover, will zuul run w/ dupe names? if so fine20:53
weshay|ruckjust wasn't sure20:53
rfolco|roverah you want 5 runs of same20:54
weshay|ruckaye20:54
rfolco|roverk20:54
weshay|ruckcentos-8-undercloud-containers.. blasted.. at least 520:54
weshay|ruckrlandy, https://review.opendev.org/#/c/727438/20:58
rlandyreplacement change?20:58
weshay|ruckhttps://trello.com/c/dxk6rZ61/1489-cixbz1834348componentstandalonepuppet-tripleoosp17rhel8rhel8-osp17-standalone-job-failing-downstream-unknown-function-mysqlpassw20:58
rlandyweshay|ruck: ok - let's see if we can test that downstream20:59
rfolco|roverweshay|ruck, https://review.rdoproject.org/r/#/c/27453/21:05
rfolco|roverttyl21:06
*** rfolco|rover is now known as rfolco|rover|off21:06
weshay|ruckrfolco|rover|off, perfect :)21:06
weshay|ruckthanks21:06
*** jfrancoa has left #oooq21:08
*** jfrancoa has quit IRC21:08
*** jmasud has quit IRC21:09
*** jbadiapa has quit IRC21:17
*** jmasud has joined #oooq21:23
rlandyweshay|ruck: fyi - waiting on a passing standalone to merge: https://code.engineering.redhat.com/gerrit/#/q/topic:component-pipeline-templates+(status:open+OR+status:merged)21:24
rlandypipelines ready to go21:25
weshay|ruckrlandy, why wait?21:41
*** panda has quit IRC21:51
*** panda has joined #oooq21:53
rlandywe'll get tons of failures22:09
*** holser has quit IRC22:11
*** jmasud has quit IRC22:17
*** jmasud has joined #oooq22:19
*** sanjayu_ has joined #oooq23:02
rlandyweshay|ruck: rethinking where we should put the internal criteria file23:13
rlandyweshay|ruck: originally we said in tripleo-environments23:14
rlandya lot of switching code around23:14
*** tosky has quit IRC23:14
*** sanjayu_ has quit IRC23:19
*** jmasud has quit IRC23:19
*** jmasud has joined #oooq23:21
weshay|ruckrlandy, I still think it should be tripleo-environments23:41
rlandyweshay|ruck: ok - mering all pipeline other than the promote one23:41
rlandywe have work to do to get that code working for downstream23:41
rlandyweshay|ruck: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/check-promotion-criteria.yaml#L1823:42
weshay|ruckrlandy, k.. was working w/ panda on that as well..23:42
weshay|ruckso.. tripleo-environments/ci-scripts/dlrnapi_promoter/config23:43
rlandyweshay|ruck: alright will see what damage the component pipelines do23:43
rlandyweshay|ruck: ack23:43
rlandyrelease ... will have to be osp_release23:43
rlandybut that;s a minor change in config23:43
rlandywill have that change up for review23:43
weshay|ruckhttps://code.engineering.redhat.com/gerrit/20019423:44
weshay|ruckaye23:44

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!