Monday, 2020-06-01

*** tosky has quit IRC00:20
*** jmasud has quit IRC00:49
*** jmasud has joined #oooq00:57
*** jmasud has quit IRC01:12
*** jmasud has joined #oooq01:16
*** saneax has quit IRC01:24
*** jmasud has quit IRC01:25
*** jmasud has joined #oooq01:26
*** saneax has joined #oooq01:35
*** jmasud has quit IRC02:04
*** jmasud has joined #oooq02:06
*** ysandeep|afk is now known as ysandeep02:08
*** jmasud has quit IRC02:10
*** jmasud has joined #oooq02:11
*** jmasud has quit IRC02:20
*** jmasud has joined #oooq02:26
*** jmasud_ has joined #oooq02:37
*** jmasud_ has quit IRC02:38
*** jmasud has quit IRC02:40
*** saneax has quit IRC02:45
*** apetrich has quit IRC03:51
*** jmasud has joined #oooq04:06
*** ykarel|away is now known as ykarel04:07
*** skramaja has joined #oooq04:18
*** jmasud has quit IRC04:24
*** jmasud has joined #oooq04:36
*** jmasud has quit IRC04:47
*** jmasud has joined #oooq04:57
*** ratailor has joined #oooq05:08
*** marios has joined #oooq05:09
*** jmasud has quit IRC05:16
*** dpawlik has joined #oooq05:18
*** jmasud has joined #oooq05:20
*** jmasud has quit IRC05:35
*** saneax has joined #oooq05:37
ysandeeppojadhav|ruck, Hey o/ If you have some context on why we choose multinode for tripleo-ipa and not singlenode?  Could you please add here https://review.rdoproject.org/r/#/c/27863/1//COMMIT_MSG , Reviewers requesting for some additional context when I was adding that job in integration pipeline - if we had some challenges for running ipa job on a singlenode?05:41
ysandeeppojadhav|ruck, If you are not sure, no worries i will check with ronelle tonight.05:41
pojadhav|ruck ysandeep , I am not pretty sure, I think ronelle can give better exposure about this :)05:43
*** saneax has quit IRC05:44
ysandeeppojadhav|ruck, okay, i will check with her. Thanks!05:44
*** jmasud has joined #oooq05:46
*** saneax has joined #oooq05:55
*** dpawlik has quit IRC05:57
*** dpawlik has joined #oooq05:57
*** dpawlik has quit IRC05:58
*** jfrancoa has joined #oooq06:39
*** dpawlik has joined #oooq07:00
*** matbu has quit IRC07:19
*** yolanda has joined #oooq07:19
*** matbu has joined #oooq07:19
*** tosky has joined #oooq07:28
*** amoralej|off is now known as amoralej07:42
*** ysandeep is now known as ysandeep|lunch07:50
*** jpena|off is now known as jpena07:56
*** jmasud has quit IRC08:12
*** ccamacho has joined #oooq08:19
*** ykarel is now known as ykarel|lunch08:23
*** jmasud has joined #oooq08:25
*** ysandeep|lunch is now known as ysandeep08:45
*** jmasud has quit IRC08:49
ysandeepmarios, chandankumar, fyi.. downstream internal zuul - All jobs seems stuck in queued state - I have pinged on rhos-ops internal channel08:51
mariosthanks ysandeep08:54
ysandeep:)08:54
*** jmasud has joined #oooq08:59
*** ysandeep is now known as ysandeep|brb09:01
*** dpawlik has quit IRC09:08
*** ykarel|lunch is now known as ykarel09:10
chandankumarcgoncalves, Hello09:22
chandankumarcgoncalves, based on fs038 testing https://review.rdoproject.org/r/#/c/27587/ and https://logserver.rdoproject.org/87/27587/12/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset038-queens/2dc1332/logs/undercloud/home/zuul/tempest.log.txt.gz it is passing here09:22
chandankumarI am going to add it to the promotion pipeline09:23
*** jmasud has quit IRC09:23
*** jbadiapa has joined #oooq09:23
mariosysandeep|brb: seems to be OK now fyi (d/stream)09:39
*** ysandeep|brb is now known as ysandeep09:46
ysandeepmarios, yes09:46
*** jmasud has joined #oooq10:07
*** holser has quit IRC10:13
*** holser has joined #oooq10:14
*** matbu has quit IRC10:39
*** matbu has joined #oooq10:41
*** jaosorior has joined #oooq10:49
*** jmasud has quit IRC10:55
*** ratailor has quit IRC11:09
*** ysandeep is now known as ysandeep|afk11:12
*** dpawlik has joined #oooq11:13
weshay|ruckysandeep|afk, you can't run standalone and the ipa server and client all on the same node11:29
weshay|ruckeven with the ipa server as a container11:30
*** jpena is now known as jpena|lunch11:34
*** pojadhav|ruck is now known as pojadhav|away11:34
weshay|ruckysandeep|afk, we tried.. not enough cpu/memm/space on a nodepool node11:35
*** ysandeep|afk is now known as ysandeep11:36
ysandeepweshay|ruck, thank you, ykarel ^^11:36
*** amoralej is now known as amoralej|lunch11:41
weshay|ruckpojadhav|away, 0/11:54
*** pojadhav|away is now known as pojadhav|ruck11:56
pojadhav|ruckweshay|ruck, hey11:56
pojadhav|ruckweshay|ruck, joined sync call :)11:59
weshay|ruckpojadhav|ruck, joinging12:00
weshay|ruckpojadhav|ruck, http://dashboard-ci.tripleo.org/d/QrkFP-zGz/upstream-and-rdo-promotions?orgId=112:01
*** dmellado has quit IRC12:02
*** dmellado has joined #oooq12:06
*** rfolco has joined #oooq12:06
*** rfolco is now known as rfolco|rover12:06
rfolco|roversshnaidm, hey do we have a logstash query handy for this bug ? https://bugs.launchpad.net/tripleo/+bug/187947212:07
openstackLaunchpad bug 1879472 in tripleo "OVB overcloud deploy fails sporadically with "not enough free physical ports" error" [Critical,Triaged]12:07
rfolco|roversshnaidm, need to know if this is still an issue for cix call12:07
*** rlandy has joined #oooq12:10
rlandypojadhav|ruck: ysandee: hey - pls will you vote on https://review.opendev.org/#/c/731264/ so I have other votes before w+12:13
rlandyysandeep: ^^12:13
rlandymarios: hi - want to meet about the new container workflow before ptg?12:14
mariosrlandy: o/12:14
mariosrlandy: sure12:14
rlandymarios: k - at half passed the hour? like 15 mins?12:15
mariosrlandy: ack12:15
rlandywill also need to chat with dpawlik about space on the log/images server12:15
mariosrlandy: oh, did we run out of space?12:15
mariosrlandy: am trying to see what we can do about emit releases but i don't know/think it can work upstream (playing there https://review.opendev.org/732170 )12:16
ysandeeprlandy, lgtm .. voted there12:17
* dpawlik reading12:19
*** amoralej|lunch is now known as amoralej12:19
dpawliksome problems with keeping logs or images?12:19
*** skramaja has quit IRC12:20
dpawlikImage server is little over 50% https://prometheus.monitoring.softwarefactory-project.io/grafana/d/EtRMbV1Zk/node-exporter-full?orgId=1&var-job=node&var-name=images.rdoproject.org&var-node=images.rdoproject.org&var-port=9100&from=1590927633710&to=1591014033710&panelId=152&fullscreen and logserver: https://prometheus.monitoring.softwarefactory-project.io/grafana/d/EtRMbV1Zk/node-exporter-full?orgId=1&var-job=node&var-name=logserver.12:21
dpawlikrdoproject.org&var-node=logserver.rdoproject.org&var-port=9100&from=1590927667709&to=1591014067709&panelId=152&fullscreen12:21
dpawlikis also ok12:21
rlandyysandeep; thank you12:22
rlandydpawlik: sorry - there's plenty of space atm ...12:22
rlandydpawlik: we will need to start uploading docker files as well12:22
rlandywe will be building docker files and uploading them for the reldel team12:23
rlandydpawlik: will we have enough space for a parallel tree of dockerfiles/hash with symlinks for tripleo-ci-testing and current-tripleo12:24
ysandeeprlandy, fyi.. posted tripleo-ipa for master/ussuri integration pipeline - https://review.rdoproject.org/r/#/c/27863/12:24
rlandyysandeep: perfect - thanks12:26
weshay|ruckpojadhav|ruck, http://dashboard-ci.tripleo.org/d/QrkFP-zGz/upstream-and-rdo-promotions?orgId=1&fullscreen&panelId=1912:26
rlandyif it passes well enough, we can add to criteria as well12:26
rlandyweshay|ruck: pojadhav|ruck: rfolco|rover: fyi - I'd like to merge ysandeep's job to add multinode-ipa to integration line: https://review.rdoproject.org/r/#/c/27863/12:27
ysandeeprlandy, ack will keep an eye on job for few days.. if stable will add to criteria as well.12:30
rlandyysandeep: cool - merged12:31
rlandymarios: https://meet.google.com/zbv-vcrv-oqw12:31
mariosrlandy: joining12:31
ysandeeprlandy++ chandankumar++ awesome12:31
*** pojadhav|ruck is now known as pojadhav|away12:32
*** jpena|lunch is now known as jpena12:35
weshay|rucksshnaidm, potential fix for port and resource issues in ovb https://review.opendev.org/#/c/727677/12:42
weshay|rucksshnaidm, can we pop this into a testproject?12:42
sshnaidmrfolco|rover, see my comment in the bug12:51
*** ykarel is now known as ykarel|afk12:51
sshnaidmweshay|ruck, we can, although it's transient issue, idk if we can see it solves it12:52
rlandymarios: https://review.opendev.org/#/c/729465/12:52
sshnaidmweshay|ruck, seems like it requires additional ironic config as well, so need a patch somewhere with it - tht?12:54
*** TrevorV has joined #oooq13:06
*** ratailor has joined #oooq13:11
*** arxcruz is now known as arxcruz|qa_room13:14
rlandyzbr: hi ... back with openstack-tox-linters - do you know where this repo list comes from? "  LOOP [prepare-workspace-git : Clone cached repo to workspace]" example log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_482/730995/6/check/openstack-tox-linters/4826822/job-output.txt14:31
*** ratailor has quit IRC14:34
zbrwhat repo list? i see failure to run because podman_container module is missing.14:35
zbralso link me to change that produced this error14:37
*** ratailor has joined #oooq14:38
mariosrlandy: is d/stream working for you now? my job has been waiting for ~ 20 mins (maybe its just busy i do see some stuff running) https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status14:41
rlandymarios: my jobs kick about an hour ago - the last time I triggered something14:41
mariosweshay|ruck: you're sharing your irc14:42
marios(not any more)14:42
*** ratailor has quit IRC14:42
rlandymarios: kicking another job soon14:42
rlandywill let you know14:42
rlandyinfra should be able to check14:42
mariosrlandy: finally started running (linters) now so maybe just a queue14:43
*** dtantsur|afk is now known as dtantsur14:48
*** ykarel|afk is now known as ykarel14:58
*** matbu has quit IRC15:03
*** Goneri has joined #oooq15:03
*** matbu has joined #oooq15:04
weshay|ruckFOLKS.. anything to add to https://etherpad.opendev.org/p/tripleo-ptg-victoria-ci-updates ?15:10
weshay|ruckrlandy, marios chandankumar rfolco|rover zbr ^15:11
mariosweshay|ruck: can't think of sthing15:12
marioschandankumar: maybe am wrong but that is for 'things we plan to do' not what we did in ussuri (i.e. like the build images unification)15:16
chandankumarmarios, ah, got it15:17
marioschandankumar: unfortunately a lot of the work is d/stream so we can't include it there :/15:18
mariosrlandy: well, the queue is pretty big, or there is a problem (my job finished linters but didn't start anything else yet ~ 1 hour now)15:21
mariosfyi15:21
rlandymarios: let's ping infra15:21
rlandybefore mhu is EOD15:21
*** gthiemonge has quit IRC15:21
mariosrlandy: don't see anything at https://one.redhat.com/tools-and-services/details/psi-openstack-cloud-d15:22
rlandymarios: looks like the promote jobs are kicking now15:23
mariosrlandy: k maybe just a long queue15:28
rlandymarios: there is a job running a bunch of tempest tests15:29
rlandythat is taking some time15:29
rlandymy ovb job also has not yet kicked15:29
rlandymarios: how's your virt-customize knowledge?15:30
mariosrlandy: "i've used it before, but not for a while"15:31
mariosrlandy: what you need15:31
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/undercloud/tasks/main.yml#L18615:31
rlandymarios: ^^ for rhel image ...15:31
rlandy[zuul@rdoci-hp-07 ~]$ virt-filesystems -a undercloud.qcow215:32
rlandydev/sda215:32
rlandydev/sda315:32
rlandyidk why undercloud_partitions.stdout=='/dev/sda1'?15:32
mariosrlandy: don't know sorry :/15:33
mariosrlandy: i think that's not to do with virt-filesystems but more about determining there are partitions? but i don't know for sure what the story is there15:35
*** pojadhav|away is now known as pojadhav|ruck15:39
mariosrlandy: job eventually ran fyi so it was just a long queue15:44
*** ykarel is now known as ykarel|away15:45
rlandymarios: ysandeep: could you guys take a quick look at https://code.engineering.redhat.com/gerrit/#/c/201122/ before I merge15:52
rlandywill add a job for the integration line after this merges15:52
rlandyfyi - I know the parenting is a little different15:53
rlandymarios: ysandeep: ^^15:53
rlandyadding the additional repo threw off the pres if added in that job15:54
mariosrlandy: looking (brain a little fried but will do my best while kevin is talking ;) )15:54
rlandynever the less the pattern remains the same after that15:54
mariosrlandy: threw off the pres what is "pres"15:55
rlandymarios: pre roles15:55
mariosrlandy: ah ack15:56
rlandyFYI - scenario010 is failing15:56
rlandy       - name: set current_md5 fact from glance if image already exists there15:56
rlandyysandeep: ^^ was that failing before? something new?15:57
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-17/9174d73/15:57
ysandeeprlandy, lgtm ^^ ipa15:57
ysandeeprlandy, nah we hit something new .. earlier everything was working15:58
rlandyysandeep: ok - I'll look into it15:58
rfolco|roverarxcruz|qa_room, around?16:16
*** jmasud has joined #oooq16:18
rlandyykarel|away: hmmm ... we killed stable branches somehow16:20
rlandyrfolco|rover: weshay|ruck: ^^ fyi16:20
ykarel|awayrlandy, https://review.opendev.org/#/c/732091/16:21
rlandyykarel|away: thanks - will watch that16:22
rfolco|roverrlandy, ykarel|away ack thx16:22
weshay|ruckCHIME IN16:28
*** dpawlik has quit IRC16:29
*** dpawlik has joined #oooq16:30
*** dpawlik has quit IRC16:41
*** holser has quit IRC16:42
*** holser has joined #oooq16:42
*** dpawlik has joined #oooq16:42
*** jmasud has quit IRC16:43
*** holser has quit IRC16:50
*** holser has joined #oooq16:50
*** jmasud has joined #oooq16:55
*** jaosorior has quit IRC16:57
*** dtantsur is now known as dtantsur|afk17:01
*** jpena is now known as jpena|off17:11
*** marios has quit IRC17:12
*** jmasud has quit IRC17:15
*** jmasud has joined #oooq17:16
*** amoralej is now known as amoralej|off17:18
*** tosky has quit IRC17:19
*** TrevorV has quit IRC17:23
*** TrevorV has joined #oooq17:28
ysandeeprlandy, Please add to your review list - https://code.engineering.redhat.com/gerrit/#/c/201976/ - whenever time permits :)17:31
ysandeepLeaving for the day, Have a good day ahead!17:31
*** ysandeep is now known as ysandeep|away17:35
*** dpawlik has quit IRC17:37
*** jmasud has quit IRC17:57
cgoncalveschandankumar, thanks!18:00
rlandyrfolco|rover: you seeing issues installing  "https://tarballs.opendev.org/openstack/stackviz/dist/stackviz-latest.tar.gz"?18:18
rfolco|roverrlandy, haven't seen this one18:19
rlandyugh18:19
rfolco|roverrlandy, where18:19
rlandyrfolco|rover: downstream18:19
* rlandy is rucking donwstream18:20
*** sshnaidm is now known as sshnaidm|afk18:20
*** dsneddon has joined #oooq18:47
*** jmasud has joined #oooq18:54
*** jmasud has quit IRC18:59
*** TrevorV has quit IRC19:03
*** arxcruz|qa_room is now known as arxcruz19:07
rfolco|roverchandankumar, arxcruz should we replace octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops w/ octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest on scen 10 ovn provider job ?19:09
arxcruzrfolco|rover: i don't know, why ?  what's the context?19:10
weshay|ruckrlandy, ping.. have 5 to chat about CI ptg time for tomorrow?19:16
rfolco|roverarxcruz, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L43819:16
rlandyweshay|ruck: ack19:16
weshay|ruckhttps://meet.google.com/uzz-turx-zyi19:17
*** TrevorV has joined #oooq19:18
rlandyweshay|ruck: https://review.opendev.org/#/c/732091/19:23
rlandyweshay|ruck: do we have a fix for ERROR: No matching distribution found for gitdb>=4.0.1 (from gitdb2>=2.0.0->GitPython<2.1.12,>=1.0.1->tripleo-common>=11.3.1->tripleo-ansible->-r quickstart-extras-requirements.txt (line 11))19:29
*** jmasud has joined #oooq20:05
weshay|ruckrfolco|rover, ussuri is trying to promote but failing20:10
weshay|ruck2020-06-01 19:55:30,350 18142 ERROR    promoter Candidate hash 'aggregate: 3736708f082548d6f0908249e59c416a, commit: f2c8f36ed9949b9cc0ef6bebf24e71db9da66274, distro: 988cf75358582d6b51db498a511a1e2629ce9d7c, component: ui, timestamp: 1591024482': client dlrn_client FAILED promotion attempt to current-tripleo20:10
weshay|ruck2020-06-01 19:55:30,350 18142 ERROR    promoter API returned different promoted hash20:10
weshay|ruckrfolco|rover, train looks good here.. just the scenario010 issue https://trunk.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=e6f08c7e37116f87c5b860a866991bfac3b982a4&distro_hash=ffc4ec1bbbb97b07f94423fa86586efeaf9455d620:13
weshay|ruckrfolco|rover, I see ur email about scenario010-master same issue w/ train?20:13
*** jfrancoa has quit IRC20:14
rlandyweshay|ruck: tripleo-ci-centos-8-scenario010-standalone just failed again on 732091/20:21
rlandyweshay|ruck: possibly just reverting my patch may be enough20:21
rlandyhttps://github.com/openstack/tripleo-quickstart/commit/aeea81aa8837c6f5b6d946365ca974343a034cb420:22
rfolco|roverweshay|ruck, probably20:25
rfolco|roverbrb20:28
*** owalsh has quit IRC20:38
weshay|ruckrfolco|rover, ipv6 and cirros?20:42
weshay|ruckrfolco|rover, master build https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=0a44b8ec4f1960145e300dc8407d888920:50
weshay|ruckrfolco|rover, ussuri is promoting now... on it's own.. but will fail.. due to promoter issues20:52
weshay|ruckmaster will attempt next20:52
*** jmasud has quit IRC20:55
*** owalsh has joined #oooq20:56
rlandyhttps://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs.yaml#L42820:58
rlandyhmmm ....20:58
rlandyI wonder20:58
*** jmasud has joined #oooq20:59
rlandyweshay|ruck: fyi ... https://review.rdoproject.org/r/27881 WIP: Testing to_build true for component jobs21:01
rlandywe must have explicitly set that21:02
rfolco|roverweshay|ruck, ipv6 is buggy according to https://github.com/cirros-dev/cirros/issues/5821:04
rfolco|roverweshay|ruck, but i've seen on fs035 and full-tempest21:05
rfolco|roverweshay|ruck, scen10 master is ok, scen10 ovn is failing21:05
*** jmasud has quit IRC21:06
*** jmasud has joined #oooq21:08
*** ccamacho has quit IRC21:10
rfolco|roverweshay|ruck, weird, scen10 c8 master/ussuri/train is good, only c7 train is failing21:11
rfolco|roverhttps://bugs.launchpad.net/tripleo/+bug/188158421:13
openstackLaunchpad bug 1881584 in tripleo "SC10 train periodic job failing with ConflictException: 409 consistently" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)21:13
rfolco|rovergotta go21:14
rfolco|roverttyl21:14
*** rfolco|rover is now known as rfolco|rover|off21:14
weshay|ruckrlandy, I think we need to mark 010 non-voting in master21:48
weshay|ruckoof and 004 failed again21:49
rlandyweshay|ruck: ugh - the revert is still an option21:49
rlandy004 may be legt - although idk how21:49
weshay|ruck        "ERROR! Invalid callback for stdout specified: tripleo_dense"21:51
rlandywhat is tripelo_dense?21:54
rlandyweshay|ruck: what can I do to help here?21:57
weshay|ruckrlandy, I think we need to mark sce010 non-voting until https://github.com/cirros-dev/cirros/issues/58 resolves if rfolco|rover|off and arxcruz are on to something there22:03
* weshay|ruck looks at sce010 fail22:03
rlandyweshay|ruck: ack - I see it failing downstream as well22:03
rlandyas if this morning22:04
rlandyadding patch to make scenario010 non-voting22:07
rlandyand bug22:09
rlandyoh ... https://bugs.launchpad.net/tripleo/+bug/188158422:10
openstackLaunchpad bug 1881584 in tripleo "SC10 train periodic job failing with ConflictException: 409 consistently" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)22:10
rlandyweshay|ruck: https://review.opendev.org/732464 Move scenario010 jobs to non-voting until bug/1881584 is resolved22:18
rlandynot sure if that's the right bug22:18
* rlandy makes https://review.opendev.org/#/c/732091/ rebased on https://review.opendev.org/#/c/732464/ and dependent on revert22:19
rlandyweshay|ruck: ^^ - agree?22:19
*** tosky has joined #oooq22:32
*** TrevorV has quit IRC22:37
*** jmasud has quit IRC22:45
*** jmasud has joined #oooq23:11

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!