Thursday, 2019-12-12

*** dsneddon has quit IRC00:02
*** dsneddon has joined #oooq00:06
*** holser has quit IRC00:06
*** holser has joined #oooq00:10
*** dsneddon has quit IRC00:12
*** dsneddon has joined #oooq00:12
*** brault has quit IRC00:29
*** brault has joined #oooq00:30
*** holser has quit IRC00:38
*** ysandeep has quit IRC01:24
*** epoojad1 has quit IRC01:46
*** dsneddon has quit IRC02:06
*** dsneddon has joined #oooq02:29
*** rfolco has quit IRC02:32
*** apetrich has quit IRC03:09
*** rlandy|bbl has quit IRC03:44
*** udesale has joined #oooq03:45
*** ykarel|away has joined #oooq03:49
*** bhagyashris has joined #oooq04:22
*** ykarel|away has quit IRC04:27
*** ykarel|away has joined #oooq04:40
*** ykarel|away is now known as ykarel04:41
*** ysandeep has joined #oooq04:53
*** ysandeep has quit IRC04:54
*** bhagyashris has quit IRC05:06
*** brault has quit IRC05:13
*** bhagyashris has joined #oooq05:13
*** brault has joined #oooq05:14
*** epoojad1 has joined #oooq05:14
*** surpatil has joined #oooq05:24
*** SurajPatil has joined #oooq05:39
*** surpatil has quit IRC05:42
*** surpatil has joined #oooq05:57
*** dsneddon has quit IRC05:58
*** SurajPatil has quit IRC05:59
*** dsneddon has joined #oooq06:02
*** raukadah is now known as chkumar|ruck06:05
*** dsneddon has quit IRC06:07
*** soniya29 has joined #oooq06:11
*** marios|ruck has joined #oooq06:26
*** SurajPatil has joined #oooq06:26
*** surpatil has quit IRC06:29
*** surpatil has joined #oooq06:31
*** SurajPatil has quit IRC06:34
*** jtomasek has joined #oooq06:41
*** soniya29 has quit IRC06:47
*** jfrancoa has joined #oooq06:48
*** dsneddon has joined #oooq06:49
*** ykarel is now known as ykarel|lunch06:57
*** jbadiapa has joined #oooq07:06
*** gkadam has joined #oooq07:13
*** gkadam has quit IRC07:13
marios|ruckchkumar|ruck: \o/07:25
marios|ruckanother day in paradise07:26
*** chkumar|ruck is now known as chkumar|rover07:32
chkumar|rovermarios|ruck: \o/07:32
*** bhagyashris has quit IRC07:43
*** tesseract has joined #oooq07:59
*** jpena|off is now known as jpena08:01
*** bhagyashris has joined #oooq08:12
*** amoralej|off is now known as amoralej08:26
*** tosky has joined #oooq08:27
*** akahat has joined #oooq08:30
*** dsneddon has quit IRC08:32
*** bhagyashris has quit IRC08:36
marios|ruckchkumar|rover: can you check         * https://bugs.launchpad.net/tripleo/+bug/1855985/comments/9 when you have a chance please and do we want to combine them         * https://bugs.launchpad.net/tripleo/+bug/1856016/comments/308:47
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]08:47
openstackLaunchpad bug 1856016 in tripleo "Tempest basic ops test_cross_tenant_traffic tests failed on fs020 train with Timed out waiting for 10.0.0.102 to become reachable from 10.0.0.117" [Critical,Confirmed]08:47
*** ykarel|lunch is now known as ykarel08:48
chkumar|rovermarios|ruck: yes08:57
*** SurajPatil has joined #oooq08:58
*** dsneddon has joined #oooq08:58
*** surpatil has quit IRC09:00
*** surpatil has joined #oooq09:01
*** SurajPatil has quit IRC09:03
*** dsneddon has quit IRC09:05
*** bhagyashris has joined #oooq09:12
*** soniya29 has joined #oooq09:13
*** holser has joined #oooq09:18
marios|ruckchkumar|rover: we lost the rocky promotion but i can fix it it happened before noting on etherpad for now (i think that was you writing above me you have have seen :D)09:19
*** derekh has joined #oooq09:19
marios|ruckchkumar|rover: will have a look at that today anyway not highest priority but shame since the promotion is ready09:19
marios|ruckbrb09:19
*** ccamacho has joined #oooq09:35
*** dsneddon has joined #oooq09:40
*** ccamacho is now known as ccamacho|pto09:48
*** saneax has joined #oooq09:48
chkumar|rovermarios|ruck: closed this one https://bugs.launchpad.net/tripleo/+bug/1856016 as a won't fix and merged it in the https://bugs.launchpad.net/tripleo/+bug/1855985/comments/1009:50
openstackLaunchpad bug 1856016 in tripleo "Tempest basic ops test_cross_tenant_traffic tests failed on fs020 train with Timed out waiting for 10.0.0.102 to become reachable from 10.0.0.117" [Critical,Won't fix]09:50
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]09:50
chkumar|roverlet me decrease the concurrency to 2 and see what happens09:50
marios|ruckchkumar|rover: ack thanks09:51
*** dtantsur|afk is now known as dtantsur09:52
chkumar|rovermarios|ruck: rdo cloud is back now09:54
*** surpatil has quit IRC09:56
chkumar|rovermarios|ruck: https://review.opendev.org/#/c/698664/09:58
chkumar|rovermarios|ruck: rerrunning fs020 master and train with above patch https://review.rdoproject.org/r/#/c/24051/10:03
marios|ruckthanks chkumar|rover10:03
marios|ruckchkumar|rover: i hope that will fix rocky if green fyi https://review.rdoproject.org/r/#/c/24086/1/.zuul.yaml10:03
chkumar|roveryes10:04
marios|ruckchkumar|rover: (seems the images are removed from https://images.rdoproject.org/centos7/rocky/rdo_trunk/ cos of a cleanup script ... rlandy mentioned it last time but... we need to tweak that for rocky and queens i think10:04
marios|ruckchkumar|rover: https://trunk.rdoproject.org/centos7-rocky/tripleo-ci-testing/delorean.repo10:04
marios|ruckchkumar|rover: it is the missing hash ^ so will upload the right image ERROR: The promoted hash was not found: https://images.rdoproject.org/centos7/rocky/rdo_trunk/5beb8164ecf2f8448727bc3d9773b21a2ec22168_c495453e10:04
marios|ruckchkumar|rover: but only if the job passes :)10:05
chkumar|rovermarios|ruck: ok10:05
chkumar|rovermarios|ruck: so fs002 will run, then the image will be available na?10:06
marios|ruckchkumar|rover: exactly10:07
marios|ruckchkumar|rover: so if it passes rocky will promote today10:08
chkumar|roverdoes train promoted yesterday?10:08
marios|ruckchkumar|rover: problem is it already uploaded the containers10:08
marios|ruckchkumar|rover: ah no that is not a problem10:08
marios|ruckchkumar|rover: cos its container then images then links the link is the one that matters most10:08
chkumar|rovermarios|ruck: what about re-run the container and fs002 as a dependencies10:09
marios|ruckchkumar|rover: yes train yesterday10:09
chkumar|roverawesome10:09
marios|ruckchkumar|rover:  no just the image upload we don't need anything else10:09
marios|ruckchkumar|rover: like see http://38.145.34.55/centos7_rocky.log10:09
marios|ruckchkumar|rover: its promoting10:09
marios|ruckchkumar|rover: uploads containers10:09
marios|ruckchkumar|rover: tries image but fails10:09
chkumar|roverah ok10:12
*** dsneddon has quit IRC10:24
marios|ruckchkumar|rover: no cix call today? can't see in calendar10:32
chkumar|rovermarios|ruck: yes no cix today, it is thursday10:33
chkumar|rovermon and wed only10:33
marios|ruckchkumar|rover: cool thx10:33
*** dsneddon has joined #oooq10:40
*** dsneddon has quit IRC10:46
*** surpatil has joined #oooq10:50
chkumar|rovermarios|ruck: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/b7e6593/build.log10:50
chkumar|rovermarios|ruck: I think know the error [' 7999188 -lt 14680064 ']' check the ram part10:50
chkumar|roverif it is equal then it will pass i think10:51
marios|ruckchkumar|rover: interesting.. i also see the network errors there though maybe that is a root cause 2019-12-12 00:30:29.267 | > failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory10:52
chkumar|rover Running scriptlet: network-scripts-10.00.4-1.el8.x86_6410:53
chkumar|roverit might be related to network-scripts package itself10:53
marios|ruckchkumar|rover: we have https://bugs.launchpad.net/tripleo/+bug/185302810:53
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]10:53
chkumar|roverlet me check the source of the package itself10:54
*** ykarel is now known as ykarel|afk10:55
marios|ruckchkumar|rover: any idea why https://review.rdoproject.org/r/#/c/23919/ still failing on https://bugs.launchpad.net/tripleo/+bug/1855826 http://logs.rdoproject.org/19/23919/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/af3d1c9/job-output.txt         * 2019-12-12 03:17:50.695675 | primary | TASK [fetch-images : Check if we have a latest image] **************************10:59
openstackLaunchpad bug 1855826 in tripleo "RHEL 8 master fs01 fails while getting image expected checksum after multiple retry" [Critical,Fix released]10:59
marios|ruck2019-12-12 03:17:50.706674 | primary | Thursday 12 December 2019  03:17:50 -0500 (0:00:00.079)       0:06:14.401 *****10:59
marios|ruck2019-12-12 03:17:51.099041 | primary | fatal: [undercloud]: FAILED! => {10:59
chkumar|rovermarios|ruck: from current view https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master it got skipped11:00
chkumar|roverin two times, current one is fine and other one is weired11:01
marios|ruckchkumar|rover: yeah but yesterday was green there https://bugs.launchpad.net/tripleo/+bug/1855826/comments/411:01
openstackLaunchpad bug 1855826 in tripleo "RHEL 8 master fs01 fails while getting image expected checksum after multiple retry" [Critical,Fix released]11:01
marios|ruckchkumar|rover: the one i pointed to above with the error is check job at 12:59 < marios|ruck> chkumar|rover: any idea why https://review.rdoproject.org/r/#/c/23919/11:02
marios|ruckwell not green.. but it passed that step i mean 13:01 < marios|ruck> chkumar|rover: yeah but yesterday was green there https://bugs.launchpad.net/tripleo/+bug/1855826/comments/411:03
openstackLaunchpad bug 1855826 in tripleo "RHEL 8 master fs01 fails while getting image expected checksum after multiple retry" [Critical,Fix released]11:03
chkumar|rovermarios|ruck: it is expected as from current run dlrn hash used in fs01 is trying to pull that image which does not exists11:03
chkumar|roveras in that dlrn hash image build got failed11:03
marios|ruckchkumar|rover: i thought the dependencies patch would fix it11:03
chkumar|rovermarios|ruck: in check job it does not work11:03
chkumar|roverwe need to define the job explicitly11:04
marios|ruckchkumar|rover: oh you mean today run failed cos the image build failed11:04
marios|ruckchkumar|rover: but for check i need to add also the image builds into there to run too11:04
marios|ruckwell skipped not failed 13:04 < marios|ruck> chkumar|rover: oh you mean today run failed11:05
marios|ruckwhere is everyoen11:05
marios|ruckare we alone chkumar|rover ?11:05
marios|ruck:D11:05
marios|ruckhello oooq \o11:05
chkumar|rovermarios|ruck: yes today run got failed11:06
chkumar|rovermarios|ruck: let get it fs01 working for you in check11:06
marios|ruckchkumar|rover: failed on image checksum?11:06
chkumar|rovermarios|ruck: image build got failed in last run11:06
marios|ruckchkumar|rover: ack i will update the fs1 i'll add the dependencies11:06
chkumar|rovermarios|ruck: wait11:06
marios|ruckchkumar|rover: right so fs1 skipped ? /me checks11:06
marios|ruckchkumar|rover: 13:06 < chkumar|rover> marios|ruck: wait11:06
marios|ruck?11:06
chkumar|rovermarios|ruck: fs01 skipped11:06
marios|ruckchkumar|rover: ack ok then makes sense i.e. the dependencies are working11:07
chkumar|rovermarios|ruck: I am trying to findout the dlrn hash from passed image build and we can pass it to the check job it will work11:07
*** dtantsur is now known as dtantsur|brb11:09
marios|ruckchkumar|rover: k11:10
chkumar|rovermarios|ruck: put dlrn_hash: 'c999c4ce111149ebe55347f783da5268a30a7058_2bafa07c' in job var11:12
chkumar|roverit is from pass one11:12
chkumar|roverit will work11:12
marios|ruckchkumar|rover: ack trying11:12
marios|ruckchkumar|rover: thanks, if it doesn't i'll just do that https://review.rdoproject.org/r/#/c/24058/3/zuul.d/tripleo.yaml11:12
chkumar|rovermarios|ruck: it will work, but build image job needs to pass11:13
chkumar|roverthen it work, otherwise if any deps fails fs01 will be get skipped11:13
marios|ruckchkumar|rover: right indeed and we have that network race thing11:13
*** dsneddon has joined #oooq11:13
marios|ruckchkumar|rover: lets hope the hash works then ;)11:13
*** udesale has quit IRC11:14
marios|ruckchkumar|rover: thanks https://review.rdoproject.org/r/#/c/23919/3/.zuul.yaml11:14
chkumar|roverperfet11:14
chkumar|roverperfect11:14
marios|ruckchkumar|rover: last time we weren't sure11:14
marios|ruckchkumar|rover: and we used both11:14
marios|ruckchkumar|rover: dlrn_hash and dlrn_hash_tag11:15
marios|ruckchkumar|rover: do you remember off hand if we need the _tag one too or you tried dlrn_hash recently and was OK11:15
chkumar|roverdlrn_hash_tag is default value for containers11:15
chkumar|rovertaggs11:15
marios|ruckchkumar|rover: right so we need it too no?11:15
marios|rucki guess we find out (hoping it defaults to using dlrn_hash for containers tag )11:16
chkumar|rovermarios|ruck: docker_image_tag: "{{ dlrn_hash|default(dlrn_hash_tag) }}"11:16
marios|ruckchkumar|rover: cool thanks11:16
chkumar|roverit will work11:16
*** ykarel|afk is now known as ykarel11:20
*** sshnaidm|afk is now known as sshnaidm11:32
marios|ruckweshay: can you please check that when you have some time https://bugs.launchpad.net/tripleo/+bug/1728886/comments/2 (and cix see pvt)11:34
openstackLaunchpad bug 1728886 in tripleo "test_port_security_macspoofing_port doesn't work when l2pop and arpresponding are enabled" [Critical,Triaged]11:34
* marios|ruck food hunting biab11:36
chkumar|rovermarios|ruck: I am taking care of that11:56
*** SurajPatil has joined #oooq12:00
*** rfolco has joined #oooq12:01
*** surpatil has quit IRC12:02
*** jpena is now known as jpena|lunch12:04
weshaybhagyashris, 1-112:24
*** dsneddon has quit IRC12:28
rfolcoscrum time in 2 min at https://meet.google.com/oiv-geho-mai12:28
weshayrfolco, really?12:29
rfolcoooops12:30
rfolco39 min from now12:30
rfolco3012:30
rfolco3012:30
rfolcosory weshay12:30
rfolcofalse alarm12:30
*** Goneri has quit IRC12:31
marios|ruckchkumar|rover: thanks12:32
*** epoojad1 is now known as epoojad1|afk12:33
chkumar|rovermarios|ruck: I was trying to reproduce this bug https://bugs.launchpad.net/tripleo/+bug/1853028 on rdocloud in a normal rhel8 vm12:33
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]12:33
chkumar|roverno luck12:33
chkumar|roverthe same rpm installation of network-scripts is working fine12:33
chkumar|roverweshay: does the rcm image exists somewhere else?12:34
chkumar|roveri mean rhel8 one exists there12:34
*** akahat has quit IRC12:35
*** sshnaidm has quit IRC12:37
*** akahat has joined #oooq12:37
*** epoojad1|afk has quit IRC12:38
chkumar|rovermarios|ruck: I found one more interesting thing today, in periodic job why we are installing dlrn and rdopkg12:38
chkumar|rovertime to fix it12:38
*** sshnaidm has joined #oooq12:38
marios|ruckchkumar|rover: scen7 tempest fail current master checking it now12:39
weshaychkumar|rover, is amol there?12:39
chkumar|rovermarios|ruck: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario007-standalone-master12:40
chkumar|roverweshay: yes12:40
chkumar|roverlet me find him12:40
marios|ruckchkumar|rover: yeah that one12:40
marios|ruckchkumar|rover: but i don't think its in criteria yet anyway12:40
marios|ruckchkumar|rover: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario007-standalone-master/69fb5c9/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz12:41
marios|ruckchkumar|rover: didn't report yet but can find in zuul status ^12:41
marios|ruckchkumar|rover: i mean the pipeline didn't report yet but that job finished/failed12:41
chkumar|rovermarios|ruck: ok12:42
chkumar|rovermarios|ruck: sorry it failed again overcloud image buid12:43
chkumar|roverno luck there12:43
chkumar|roverbrb12:43
marios|ruckchkumar|rover: https://review.rdoproject.org/r/#/c/23919/ also failed again (console) fails on fetchgin checksum thing12:45
*** soniya29 has quit IRC12:55
*** rlandy has joined #oooq12:56
*** dtantsur|brb is now known as dtantsur12:58
rfolcoscrum starting in 2 min12:59
*** dsneddon has joined #oooq13:00
rfolcorlandy, zbr scrum13:01
rfolcosshnaidm, ^13:01
rfolcoif storm did not catch you sshnaidm13:01
weshayssbarnea, ping13:01
weshaysshnaidm, you able to join?13:02
*** amoralej is now known as amoralej|lunch13:02
sshnaidmno power, will try from phone13:03
migirlandy: hey whenever you will have few min I would like to show what I did and check how this can be plugged in pipeline13:06
rlandymigi - in sprint meeting now - will ping you13:06
migirlandy: ack13:06
*** bhagyashris has quit IRC13:08
*** surpatil has joined #oooq13:15
*** jpena|lunch is now known as jpena13:15
*** SurajPatil has quit IRC13:17
*** sshnaidm has quit IRC13:18
*** sshnaidm has joined #oooq13:19
weshaysshnaidm, no worries.. don't join by phone13:27
chkumar|rovermarios|ruck: for train and master tempest results, we can check this file http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario007-standalone-master/69fb5c9/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz it has results in plain text less messy13:29
chkumar|rovervar/log/tempest/tempest_run_log.txt.gz13:30
*** amoralej|lunch is now known as amroalej13:32
*** amroalej is now known as amoralej13:32
*** fmount has quit IRC13:37
*** fmount has joined #oooq13:38
chkumar|roverkopecmartin: do you have tempestconf train review link handy?13:43
kopecmartinchkumar|rover: sure,  https://review.rdoproject.org/r/#/c/24018/13:44
*** Goneri has joined #oooq13:45
rlandyrfolco: pls review  https://review.rdoproject.org/r/24107 Use consistent 'component' var for component pipeline jobs13:47
rfolcodone :)13:48
chkumar|roverweshay: want to start now?13:49
weshaychkumar|rover, https://meet.google.com/jfx-pwrz-meb?authuser=113:49
weshayaye13:49
*** ykarel is now known as ykarel|afk13:49
chkumar|roverweshay: https://bugs.launchpad.net/tripleo/+bug/185302813:50
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]13:50
*** sshnaidm has quit IRC14:04
chkumar|roverrlandy: left a comment there https://code.engineering.redhat.com/gerrit/#/c/186258/314:09
chkumar|roverrlandy: How to extract those info?14:10
rlandylooking14:12
*** dsneddon has quit IRC14:13
rlandychkumar|rover: not sure what you mean. We have info about the physical switches in use14:13
rlandyand the networks14:13
rlandyand the gateways14:13
*** ykarel|afk has quit IRC14:13
rlandyfrom there you can work out the values14:13
rlandychkumar|rover: master and train are passing again14:14
rlandymigi: ping - whenever you are ready14:14
migirlandy: bluejeans.com/mpryc14:15
*** epoojad1 has joined #oooq14:16
rlandyjoined14:16
*** akahat has quit IRC14:20
migirlandy: https://code.engineering.redhat.com/gerrit/#/c/187618/114:24
weshayrlandy, migi can I pop in?14:30
*** dsneddon has joined #oooq14:30
rlandyweshay: we just dropped14:30
migiweshay: you can, but we covered all points I can explain on the same link14:30
marios|ruckchkumar|rover: https://review.rdoproject.org/r/24086 didn't report yet but it uploaded already https://images.rdoproject.org/centos7/rocky/rdo_trunk/5beb8164ecf2f8448727bc3d9773b21a2ec22168_c495453e/ so rocky should now promote14:30
weshayk.. np14:30
rlandymigi is dependent on the job I abandoned14:30
weshaymigi, did you want to chat about dashboards?14:31
rlandyso the error is my fault14:31
migirlandy: yeah here was the error14:31
migiweshay: sure14:31
rlandywe need to go down the role route14:31
migiweshay: bj/mpryc14:31
rlandyshould I rejoin?14:31
*** dsneddon has quit IRC14:35
*** TrevorV has joined #oooq14:43
*** sshnaidm has joined #oooq14:49
*** ccamacho|pto has quit IRC14:54
weshaymigi, http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=114:59
*** epoojad1 has quit IRC15:03
*** dsneddon has joined #oooq15:08
*** dtantsur is now known as dtantsur|brb15:10
chkumar|roverweshay: https://review.opendev.org/#/c/696387/ this one looks nice15:22
chkumar|roverkopecmartin: regarding train fs017 job I need to make some change there, may be tomorrow will get it done15:23
chkumar|roverkopecmartin: I need one more help https://review.opendev.org/698610 can you add the same on os_tempest side and enable in tripleo job please15:24
chkumar|roverrlandy: awesome, thanks for watching :-)15:25
rlandy?15:25
chkumar|rovermaster and train promotion jobs15:25
kopecmartinchkumar|rover: cool, thanks .. regarding the tempest-cleanup role, sure, when it gets merged in tempest we can definitely reuse it elsewhere15:26
weshaychkumar|rover, if you are cool w/ it as ruck rover.. we can just merge15:26
weshayif fs20 starts to fail on a test.. we just add it back?15:26
chkumar|roverweshay: yes15:26
chkumar|rovermarios|ruck: what do you think https://review.opendev.org/#/c/696387/ ?15:26
chkumar|roverbro!15:26
chkumar|roverkopecmartin: the long term goal is to convince devstack people to move to os_tempest15:27
weshayooh15:28
marios|ruckchkumar|rover: hmmm not sure15:28
weshaythat would be cool15:28
marios|ruckchkumar|rover: i think we should discover and deal with the race first15:28
marios|ruckchkumar|rover: then enable more tests15:28
marios|ruckfs20 is dying15:28
weshaykopecmartin, when are you out for the year15:28
marios|ruckand lots of thigns failing on tempest15:28
weshaymarios|ruck, which branch? train?15:29
chkumar|rovermarios|ruck: https://review.opendev.org/#/c/698664/15:29
chkumar|rovermarios|ruck: we might need some cleanup there15:29
marios|ruckweshay: all the branches fs20 fail15:29
chkumar|roveron test side15:29
marios|ruckchkumar|rover: ack ok but we don't know if it helps yet15:29
kopecmartinweshay: I'll be around the whole next week and maybe part of Dec 23rd, that's all15:29
weshaykopecmartin, k.. I'll need to sync up w/ you regarding the collect_logs infrared work..15:30
marios|ruckweshay: i mean fs20 transient tempest issues are seen very often in all branches which are clear on rerun. today chandan is trying that https://review.opendev.org/#/c/698664/ based on comments https://bugs.launchpad.net/tripleo/+bug/1855985 #7/#815:30
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]15:30
weshaykopecmartin, my hope is to bring it to a combined sprint planning meeting ( QE CI / tripleo ci ) Jan 315:30
chkumar|roverkopecmartin: tosky fyi, I will be going offline from 20th Dec to 05th Jan, 202015:30
toskyack - enjoy!15:30
*** saneax has quit IRC15:31
weshaymarios|ruck, chkumar|rover previously we discussed breaking out fs20 tests into two jobs15:31
* weshay looks at results15:31
weshaythat was for timeout issues.. breaking things out15:31
weshayhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/d7e332e/logs/stestr_results.html15:32
weshaylatest results are pretty good15:32
weshaybut IMHO.. tests that are inconsistent.. need to be bugged and skipped15:32
*** dsneddon has quit IRC15:32
chkumar|roverweshay: waiting for https://tree.taiga.io/project/tripleo-ci-board/task/1383?kanban-status=1447274 sshnaidm to give insight on how to proceed15:32
marios|ruckweshay: trouble is it isn't 'some specific tests' that are inconsistent afaik15:32
weshaytest_cross_tenant_traffic[compute,id-e79f879e-debb-440c-a7e4-efeda05b6848,network]15:32
marios|ruckweshay: chkumar|rover: ^ right? it is some kind of race in running in tree neutron vs tempest tests15:32
weshaythere are a number of them I know15:32
chkumar|rovermarios|ruck: yes15:33
weshayit's a tempest thang15:33
weshayno offense15:33
kopecmartinweshay: sure, i saw the sorin's review, I'll edit it hopefully soon, so we can definitely create some infrared jobs running the collect-logs, jan 3 sounds reasonable, but if we are talking about replacing downstream's collect logs by this one, well, it will take longer (the unification part of those 2 collect-logs role)15:33
*** surpatil has quit IRC15:33
*** saneax has joined #oooq15:33
*** bhagyashris has joined #oooq15:33
weshaykopecmartin, right.. so I'd like to hear your thoughts on how much work is there15:33
weshayto fully switch over15:33
zbrkopecmartin: in fact you don't even have to create new jobs, you can just add an extra functional test that involves using it with infrared.15:34
kopecmartinweshay: ok, we can discuss that tomorrow?15:34
chkumar|roverkopecmartin: please have a look https://bugs.launchpad.net/tripleo/+bug/1855985/comments/8 might be useful15:34
openstackLaunchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed]15:34
zbr(which should be easier)15:35
weshaykopecmartin, yes please15:35
weshayzbr, if you have thoughts about that as well.. I'll pull you in15:35
chkumar|rovermarios|ruck: weshay https://bugs.launchpad.net/tripleo/+bug/1855985/comments/8 may be this needs to be fixed15:35
zbrweshay: i do.15:36
rfolcozbr, quick ansible question, I am trying to read an ini file in ansible, this is not working as I expected https://docs.ansible.com/ansible/latest/plugins/lookup/ini.html, any other suggestion?15:36
marios|ruckchkumar|rover: that's what i was referring to15:37
marios|ruckchkumar|rover: 17:32 < marios|ruck> weshay: chkumar|rover: ^ right? it is some kind of race in running in tree neutron vs tempest tests15:37
zbrread or write? i never had to read from ini, always edited the files.15:37
chkumar|rovermarios|ruck: time to propose fix may be tomorrow15:37
marios|ruckchkumar|rover: exactly from terry comments15:37
zbrlookup runs on localhost! always.15:37
weshayzbr, ok.. added you to the mtg15:37
rfolcozbr, read a config file just like our promotion criteria15:38
marios|ruckchkumar|rover: so i don't think good idea to add more tests until we fix that15:38
zbrrfolco: keep in mind that lookup always run on ansible controller, is never on remote.15:38
chkumar|roverkopecmartin: do we want to move to a new tag for tempest for train?15:38
rfolcozbr, back to original question, how one can easily read a config file w/ sections ?15:39
rfolcowithout much code to parse it15:39
kopecmartinchkumar|rover: i don't know, do we? i'd say yes, we can try15:39
zbrfrom where? what file?  what do you need?15:39
chkumar|roverkopecmartin: yes15:40
chkumar|roversshnaidm: can we add more info on this https://tree.taiga.io/project/tripleo-ci-board/task/1383?kanban-status=1447274 how we want to proceed?15:40
rfolcozbr,I just need to retrieve this list of jobs from this file for example https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini#L16-L4215:40
zbransible is not very good at loading file content, but it rocks at editing them (ini, line_in_file)15:40
chkumar|roversshnaidm: we can get it done from tempest side?15:41
zbrwell, this is not even a valid ini file!15:41
zbrbefore doing this, i would ask myself: what prevents us from converting these files to yaml ?15:41
rfolcoby ini file I mean a user settings file split in sections15:41
rfolcozbr, apparently we want the same format we use in promotions today. If i could convince weshay in a fresh new format, yes, yaml15:42
rfolcoweshay, permission to switch gears to yaml format for criteria15:43
zbri personally like ini file simplicity but it has limitation, and this is a good example of where ini is not good.15:43
rfolcosudo switch_to_yaml15:43
weshayrfolco, zbr if we change the format.. it needs to change for both component and the promotion pipeline15:43
zbrdo you see the file is using ":" instead of "=" ?15:43
zbrparsing INI files is a nightmare.15:43
chkumar|roverzbr: what about moving to toml?15:43
zbrchkumar|rover: well,.... don't get me started before PTO15:44
chkumar|rovermuch better than ini15:44
chkumar|rovereven setup.cfg is getting replaced with pyproject.toml with poetry and fint15:45
rfolcoweshay, the parameters in the file are not used at all, all we need is the list of jobs15:45
chkumar|roversetup.py15:45
weshayrfolco, they are used by the promoter15:45
rfolcoit makes sense to me moving to yaml15:45
weshayrfolco, consistency is key15:45
weshayif you change it for one.. you change it for the other15:45
zbrwe have YAML, with good loaders. why to use something new and questionable?15:45
weshayrfolco, zbr show me15:46
chkumar|roverzbr: ok, yaml is good, as raumel.yaml not pyyaml15:46
rfolcono because we cannot move away from promoter code for the integration promotion15:46
rfolcoonly for component one15:46
weshayhttps://meet.google.com/kry-uxbj-fwq?authuser=115:46
weshayrfolco, zbr ^15:46
rfolcojoinging15:46
rlandychkumar|rover: marios|ruck: weshay: rfolco: migi: possible incoming promotions damage ... https://review.rdoproject.org/r/2411115:47
*** ykarel|afk has joined #oooq15:48
marios|ruckpanda: to test the queens we can't use standalone just realised cos http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone&branch=stable%2Fqueens15:48
marios|ruckrlandy: why do you hate us15:48
marios|ruckrlandy: (ack)15:48
chkumar|roverrlandy: if break, I will revert it15:48
rlandymarios|ruck: it's not personal15:49
chkumar|rovertime to bring more r15:49
*** chkumar|rover is now known as raukadah|r15:49
rlandywe need it for reuse downstream15:49
rlandyif I break, I own it, ok??15:49
raukadah|rokok15:49
sshnaidmraukadah|r, sorry, was in mtg15:50
raukadah|rtime to hunt for food15:50
*** raukadah|r is now known as raukadah15:50
raukadahrlandy: marios|ruck weshay see ya tomorrow, hope to fix tempest train fs02015:50
raukadahsshnaidm: np, take your time15:50
marios|ruckrlandy: o/ have a good 115:51
sshnaidmraukadah, I'm not sure how to split it, maybe just half of tests to hardcode in one job and in other "all" with negative regexp of tests from job115:51
rlandypeople - use more than 1 letter15:51
raukadahone15:51
sshnaidmrlandy++15:51
sshnaidmraukadah, so new tests will fall into job215:52
raukadahsshnaidm: I will check with infra people upstream how they scale tempest15:52
*** dtantsur|brb is now known as dtantsur15:52
sshnaidmraukadah, or maybe all api tests to move to standalone?15:53
raukadahsshnaidm: you mean all positive tempest in fs020 then all negative in other standalone including scenario?15:53
sshnaidmraukadah, no, I mean tests 1-100 in job1, and in job2 - all but not 1-10015:53
raukadahjob1 and job2 will be fs020?15:54
rlandymarios|ruck: how bad will it be if I interrupt promotions for one cycle?15:54
*** bhagyashris_ has joined #oooq15:54
*** bhagyashris has quit IRC15:55
raukadahsshnaidm: what about studying the job and rethink what operation we can perfrom then divide the test there instead of running full tempest?15:55
raukadahwe have now well defined scenario we can ask each dfg to run unique test there15:57
raukadahwithout overlapping15:57
marios|ruckrlandy: let me check cos train/master were looking pretty green15:57
marios|ruckrlandy: (and rocky is currentlypromoting as we speak)15:57
sshnaidmraukadah, maybe, worth to check15:57
marios|ruckrlandy: but when do you want to do it15:58
raukadahfull tempest with all plugins is a myth as not all deployment support all the features15:58
marios|ruckrlandy: and what do you mean by interrupt? stop the promoter?15:58
raukadahso better looking at what we deploy and what we can test, make sense?15:58
raukadahweshay: what do you say?15:58
rlandymarios|ruck: no - will impact promote-to-tripleo-ci-testing15:58
rlandywill test on a component job and revert if it breaks15:58
rlandynvm - fixing15:59
*** bhagyashris_ has quit IRC15:59
marios|ruckraukadah: looks like master should promote centos pretty green https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master15:59
rlandyshocker - failed linters15:59
raukadahmarios|ruck: w00t :-)15:59
marios|ruckrlandy: ack ok then ... today?15:59
rlandymarios|ruck: aso soon as linters pass - probably this afternoon16:00
marios|ruckraukadah: train also good but surprise suprise fs 20 https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released16:00
raukadahmarios|ruck: today is a good day!16:01
marios|ruckraukadah: as we said . living the dream !16:01
marios|ruck;)16:01
raukadahmarios|ruck: did you stocked the mozito in freeze?16:03
marios|ruckraukadah: of course! i had to drink some of the vodka to make space, but i got there in the end!16:04
raukadahmarios|ruck: lovely, I need to stock beer and vodka16:04
marios|ruckraukadah: you can just charge the shadowman card weshay said it was OK16:04
raukadahmarios|ruck: my home state is a dry state, so drinking there16:04
raukadah*no drinking there after 2016:05
migirlandy: maybe it's just myself, but why not to create proper role rather tasks_from ?16:06
marios|ruckpanda: made some updates there https://review.opendev.org/#/c/687267/ so we can run somethign other than standalone but not sure if right yet, and just posted a depends on in queens wdyt? https://review.opendev.org/#/c/698747/ we can discuss tomorrow if you around16:06
rlandymigi: what do you mean by a proper role?16:06
migirlandy: tasks_from is to use tasks from antoher role16:06
migirlandy: in case of this: https://review.rdoproject.org/r/#/c/24111/1/playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/post.yaml@616:07
raukadahtime to go16:07
rlandymigi: I could import the whole rol16:07
raukadahsee ya16:07
raukadah!16:07
rlandyand rum main.yaml16:07
migirlandy: simply have role/promote-consistent-to-tripleo-ci-testing16:07
marios|ruckraukadah: you already said that we don't believe you any more16:07
migiyeah16:07
marios|ruckraukadah: you will ping me in a bit to check a job16:07
rlandybut this keeps the flow of the playbook as it is16:07
raukadahmarios|ruck: all happened because of sshnaidm he returned from meeting that time16:07
*** jpena is now known as jpena|off16:07
rlandyie: for some playbooks - like component,16:07
rlandywe will not use the secondary distro tasks16:08
rlandymigi: I am open to suggestions16:08
rlandyright now need to get past linters /o\16:08
migirlandy: sure, will comment on review16:09
*** irclogbot_3 has quit IRC16:14
*** irclogbot_0 has joined #oooq16:14
*** dsneddon has joined #oooq16:19
rlandyweshay: rfolco: pls give https://review.rdoproject.org/r/#/c/24111/ a quick review. it's a merge a pray situation16:23
weshayrlandy, done16:31
pandamarios|ruck: queens manifests are pushed16:34
marios|ruckpanda: ack tomorrow /me out16:34
pandamarios|ruck: ok16:34
*** marios|ruck has quit IRC16:45
*** tesseract has quit IRC16:48
rlandythank you16:50
*** holser has quit IRC16:51
*** holser has joined #oooq16:52
*** bhagyashris has joined #oooq16:55
rlandymigi: responded to your comments ... we have a mixed history with tags16:55
rlandythey are used in different jobs in different ways16:56
rlandywould require edits in more places than this to use tags effectively here16:56
rlandyhonestly, this may fail horribly16:56
rlandygoing to try merge now while there is no promotion running to check it16:57
*** ykarel|afk has quit IRC17:02
rlandyhttp://logs.rdoproject.org/53/21853/16/check/periodic-tripleo-centos-7-master-component-keystone-promote-consistent-to-tripleo-ci-testing/42d8314/17:04
rlandythat passed so promising that I didn;t hose config17:04
migirlandy: sure, I am fine with that17:06
rlandyweshay: will I do any damage if I run promote-consistent-to-tripleo-ci-testing for train? the pipeline is not running now17:06
rlandyso I assume it will just rerun when the pipeline kicks17:07
rlandyor maybe rocky17:08
*** jtomasek has quit IRC17:08
*** migi is now known as migi_afk17:14
*** migi_afk is now known as migi17:14
rlandyhttp://logs.rdoproject.org/58/23558/9/check/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/2a02ca3/job-output.txt17:17
rlandyok - I don't think I broke anything - we can go downstream now with the roles17:17
rlandymigi: ^^17:17
rlandylunch - will put in downstream review in a bit17:17
*** sshnaidm has quit IRC17:18
*** jpena|off is now known as jpena17:27
weshayrfolco, avail?17:34
rfolcoweshay, coding, yes.17:34
weshayrfolco, want to pick it up now or later?17:34
rfolcoweshay, I am working on the poc, unless if you have anything, can wait until I submit something17:36
weshayk17:36
*** dtantsur is now known as dtantsur|afk17:37
*** bhagyashris has quit IRC17:43
*** derekh is now known as derekh_afk18:00
*** jpena is now known as jpena|off18:36
*** saneax has quit IRC18:37
*** amoralej is now known as amoralej|off18:43
*** sshnaidm has joined #oooq19:09
*** tosky has quit IRC19:21
baharfolco: Our job is green! So it's time to use the revert patch =) https://review.rdoproject.org/r/#/c/24048/19:29
rfolcobaha, nice job19:31
bahathanks =)19:41
*** jfrancoa has quit IRC20:27
*** holser has quit IRC20:32
*** derekh_afk has quit IRC20:48
*** TrevorV has quit IRC21:02
weshaybaha, rfolco very nice :)21:07
*** EmilienM is now known as EmilienM|mtg21:10
rfolcoweshay, rlandy21:12
rfolcoreview.rdoproject.org/r/24135 Move promote-component-to-current-tripleo to a role21:13
rfolcoreview.rdoproject.org/r/24080 Create a criteria file for component promotions21:13
rfolcorlandy, pretty much close to your previous patch21:13
weshayk.. will look in a sec.. :)21:13
rlandylooking what happened to dlrn reporting21:14
*** EmilienM|mtg is now known as EmilienM21:15
rlandywhy do we have multiple roles/playbooks21:18
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/pre.yaml21:18
rlandyand21:18
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/dlrn-report/tasks/dlrn-vars-setup.yml21:18
rlandydownstream calls ^^21:19
rlandygit consistent node failures downstream21:20
*** rfolco has quit IRC21:21
*** Goneri has quit IRC21:27
rlandyweshay: rfolco: more duplication :( ... https://review.rdoproject.org/r/#/c/24137/21:42
rlandybut can't fix everything now21:42
weshayrlandy, aye21:42
weshayk.. still helping in #tripleo  sec21:42
rlandyno rush - can just merge that21:43
rlandynode_failures - whatever21:57
rlandytry again later21:58
weshayrlandy, don't you have a very similiar patch? https://review.rdoproject.org/r/#/c/24135/22:10
weshayhttps://review.rdoproject.org/r/#/c/24111/22:10
weshaywe need two different roles for this? or is one an experiment22:11
*** aakarsh has joined #oooq22:33
*** aakarsh has quit IRC22:46
*** rfolco has joined #oooq22:46
*** rfolco has quit IRC22:50
*** rfolco has joined #oooq23:01
*** rfolco has quit IRC23:06
*** dmellado has quit IRC23:09
*** irclogbot_0 has quit IRC23:11
*** sshnaidm is now known as sshnaidm|off23:11
*** irclogbot_2 has joined #oooq23:11
*** dmellado has joined #oooq23:11
*** rfolco has joined #oooq23:15
rlandyweshay: I think there is a lot of duplication23:17
weshayrlandy, ok.. so we'll sync up tomorrow about it23:17
rlandyweshay: there is so much duplication everywhere here23:17
rlandyand downstream is throwing node failures23:18
rlandycan't test anything right now23:18
rlandymerged the code for dlrn reporting and promotion23:18
rlandybut can't run it23:18
*** holser has joined #oooq23:21
*** saneax has joined #oooq23:22
*** jbadiapa has quit IRC23:37
*** rfolco has quit IRC23:47
*** holser has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!