Monday, 2019-12-09

*** irclogbot_1 has joined #oooq00:22
*** irclogbot_1 has quit IRC00:40
*** ysandeep has joined #oooq01:10
*** irclogbot_1 has joined #oooq01:26
*** irclogbot_1 has quit IRC01:48
*** irclogbot_3 has joined #oooq02:46
*** ykarel|away has joined #oooq03:03
*** brault has quit IRC03:16
*** brault has joined #oooq03:22
*** epoojad1 has joined #oooq04:09
*** ykarel|away has quit IRC04:12
*** udesale has joined #oooq04:25
*** ykarel|away has joined #oooq04:27
*** bhagyashris has joined #oooq04:29
*** skramaja has joined #oooq04:33
*** ykarel|away is now known as ykarel04:39
*** surpatil has joined #oooq05:15
*** epoojad1 has quit IRC05:20
*** soniya29 has joined #oooq05:33
*** raukadah is now known as chkumar|ruck06:04
*** dtrainor has quit IRC06:24
*** marios|rover has joined #oooq06:28
chkumar|ruckmarios|rover: Good morning06:46
chkumar|ruckmarios|rover: the jobs which hit time out in master periodic pipeline is hitting this bug https://bugs.launchpad.net/tripleo/+bug/185565506:46
openstackLaunchpad bug 1855655 in tripleo "FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent" [Critical,Confirmed]06:46
chkumar|ruckmarios|rover: fix is in progress https://review.rdoproject.org/r/#/c/24022/06:47
chkumar|ruckthanks!06:47
*** dtrainor has joined #oooq06:47
*** akahat has joined #oooq06:50
marios|rovero/ chkumar|ruck06:52
marios|roverchkumar|ruck: sure gimme few thanks for info06:53
marios|roverchkumar|ruck: nice looks like we got train back up and running07:02
marios|roverchkumar|ruck: promotions fri/sat/sun \o/07:02
chkumar|ruckyup07:03
marios|roverchkumar|ruck: we were chasing it friday cos of rdo issues RETRY_LIMIT etc07:03
*** apetrich has joined #oooq07:03
chkumar|ruckmarios|rover: thank you for fixing that, thanks to rlandy and sshnaidm|off alsp :-)07:03
marios|roverchkumar|ruck: (is on the etherpad)07:03
marios|roverchkumar|ruck: didn't fix something rlandy reverted https://review.rdoproject.org/r/#/c/23994/07:04
chkumar|ruckah, looking07:04
marios|roverchkumar|ruck: we even hit it in the testproject looks like https://review.rdoproject.org/r/#/c/23995/07:05
marios|roverchkumar|ruck: problem was growing number of undeleted stack07:05
chkumar|ruckmarios|rover: Does the post cleanup script is not deleting stack after revert the change https://review.rdoproject.org/r/#/c/23994/ ?07:08
marios|roverchkumar|ruck: i think its fine now the revert was th fix07:08
chkumar|ruckok, cool :-)07:08
*** apetrich has quit IRC07:08
marios|roverhttp://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=23107:09
marios|roverchkumar|ruck: ^^ looks calm07:09
chkumar|ruckyes07:09
marios|roverchkumar|ruck: cool your kolla patch merged07:22
chkumar|ruckyes07:22
chkumar|ruckmarios|rover: I have rerunned fs01 train anf centos7 full tempest master job in test project07:23
marios|roverchkumar|ruck: ack07:25
marios|roverchkumar|ruck: we have a bug for that failed to glob pattern thing07:30
marios|roverchkumar|ruck: finding07:30
chkumar|ruckok07:30
marios|roverhttps://bugs.launchpad.net/tripleo/+bug/185302807:31
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]07:31
marios|roverchkumar|ruck: ^07:31
marios|roverchkumar|ruck: its not consistent so testproject might pass07:31
chkumar|ruckmarios|rover: But it is seen in fs01 train07:32
marios|roverchkumar|ruck: ack add info into bug then07:32
chkumar|ruckmarios|rover: done07:34
chkumar|ruckmarios|rover: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/f16cc6f/build.log07:34
chkumar|ruckmarios|rover: I am not sure what is the error in this one07:34
chkumar|ruckmarios|rover: I am not sure ERROR:root:overcloud-base has no valid mapping for package openssl-perl is related or other message to lvm207:34
marios|roverchkumar|ruck: ack thanks i think there was some fix posted to ensure network was started check cix board there is more info there07:34
marios|roverchkumar|ruck: no i think its red herring unrelated07:35
*** surpatil is now known as surpatil|lunch07:35
*** ykarel is now known as ykarel|lunch07:38
*** fmount has joined #oooq08:00
*** ysandeep has quit IRC08:02
*** ysandeep has joined #oooq08:08
*** sshnaidm|off is now known as sshnaidm08:11
*** ykarel|lunch is now known as ykarel08:11
*** jtomasek has joined #oooq08:15
*** d0ugal has joined #oooq08:19
marios|roverchkumar|ruck: can you check https://bugs.launchpad.net/tripleo/+bug/1853978/comments/16 i vaguely remember you pinging me about that. do we need to file a new bug for it? fs1 is still borked and blocks rhel808:19
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]08:19
*** tosky has joined #oooq08:25
*** tesseract has joined #oooq08:30
*** surpatil|lunch is now known as surpatil08:31
*** tesseract has quit IRC08:31
*** tesseract has joined #oooq08:31
marios|roverchkumar|ruck: ah maybe its that one then https://review.rdoproject.org/r/#/c/24022/ https://bugs.launchpad.net/tripleo/+bug/1853978/comments/1608:37
openstackLaunchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged]08:37
marios|roverchkumar|ruck: no sorry wrong bug08:37
marios|roverchkumar|ruck: its not that08:37
marios|roverchkumar|ruck: yeah added comment/16 to etherpad08:38
marios|roverchkumar|ruck: so we likely need new on there08:38
chkumar|ruckmarios|rover: I have re-runned the fs01 train rhel8 job, it is still running08:39
marios|roverchkumar|ruck: ack08:40
chkumar|ruckmarios|rover: will update the above bug once the run finishes08:40
marios|roverchkumar|ruck: sounds good08:41
marios|roverchkumar|ruck: you running fs1 master somwhere? me scrolls up08:44
chkumar|ruckmarios|rover: fs01 rhel8 train https://review.rdoproject.org/r/2402708:45
chkumar|ruckmaster full tempest standlone https://review.rdoproject.org/r/#/c/24026/08:45
chkumar|ruckwill kick out all timeout job once https://review.rdoproject.org/r/#/c/24025/ gets merged08:45
marios|roverchkumar|ruck: no master fs1 like for https://bugs.launchpad.net/tripleo/+bug/1855655/comments/408:46
openstackLaunchpad bug 1855655 in tripleo "FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent" [Critical,Confirmed]08:46
chkumar|ruckmarios|rover: for this one, I have not runned any testproject job, proposing right now08:47
marios|roverchkumar|ruck: i see also standalone 1 and scen7  & tempest job08:47
marios|roverchkumar|ruck: i mean in https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master08:47
chkumar|ruckmarios|rover: checking08:48
marios|roverchkumar|ruck: its best to log onto promoter and grab the latest centos master criteria there to know which to include08:48
*** saneax has joined #oooq08:51
chkumar|ruckyes sure08:51
chkumar|ruckmarios|rover: I am taking a look at scenario7 centos standalone failure08:53
marios|roverchkumar|ruck: ack08:56
*** udesale has quit IRC09:01
*** ykarel_ has joined #oooq09:03
*** ykarel has quit IRC09:06
*** yolanda has joined #oooq09:09
*** ysandeep has quit IRC09:24
*** ysandeep_ has joined #oooq09:24
*** ysandeep_ has quit IRC09:26
*** kopecmartin|off is now known as kopecmartin09:27
marios|roverchkumar|ruck: did you post something already? me starting to review pipelines should I check centos master?09:35
marios|roverchkumar|ruck: actually scratch that need to do some reviews first will do that next09:35
marios|roverchkumar|ruck: but if you posted/about to let me know i'll start at train09:36
marios|roveror rhel09:36
*** d0ugal has quit IRC09:36
*** derekh has joined #oooq09:38
*** d0ugal has joined #oooq09:38
*** apetrich has joined #oooq09:42
*** holser has joined #oooq09:46
chkumar|ruckmarios|rover: not posted got into meeting09:52
*** epoojad1 has joined #oooq09:54
*** ykarel__ has joined #oooq09:54
*** ykarel__ is now known as ykarel09:54
*** ykarel_ has quit IRC09:56
chkumar|ruckmarios|rover: we need to wait till the rdo current becomes consistent, in another 2 hours promotion jobs will run09:57
chkumar|ruckso sending test project review for master does not help09:58
chkumar|ruckmarios|rover: currently openstack-periodic-latest-released train promotion pipeline is running10:00
chkumar|ruckhttps://review.rdoproject.org/zuul/status10:00
*** dtantsur|afk is now known as dtantsur10:02
marios|roverchkumar|ruck: ack10:03
*** udesale has joined #oooq10:08
*** ssbarnea has quit IRC10:16
*** epoojad1 has quit IRC10:26
*** saneax has quit IRC10:30
*** saneax has joined #oooq10:30
*** udesale has quit IRC10:31
chkumar|ruckmarios|rover: standalone centos 7 full tempest was transient it passed in test project http://logs.rdoproject.org/26/24026/1/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-master/d1574a5/logs/stestr_results.html10:33
marios|roverchkumar|ruck: cool10:39
marios|roverchkumar|ruck: so did you already post the master testproject or should i go ahead?10:43
chkumar|ruckmarios|rover: it will not help currently as rdo current is not yet consistent10:50
chkumar|ruckwe need to wait for 1 hr 10 mins more to wait for promotion job to run10:50
marios|roverchkumar|ruck: ah right10:50
marios|roverchkumar|ruck: rechecking https://review.opendev.org/#/c/696871/2 not sure if that undercloud-upgrades thing is a race seen it before10:51
chkumar|ruckmarios|rover: ack10:51
marios|roverchkumar|ruck: and same there https://review.opendev.org/#/c/696872/110:52
*** matbu has quit IRC10:53
*** matbu has joined #oooq10:53
*** jtomasek has quit IRC10:54
*** jtomasek has joined #oooq10:57
*** ykarel is now known as ykarel|afk10:58
zbrchkumar|ruck: arxcruz: any reasons for not triggering github-check on pull events? see https://review.rdoproject.org/r/#/c/24032/11:00
chkumar|ruckzbr: commented11:05
*** ssbarnea has joined #oooq11:38
marios|roverpanda: can we access logs from queens  promoter?11:57
marios|roverpanda: still on new promoter or did you restore queens?11:57
chkumar|ruckmarios|rover: filed the new bug https://bugs.launchpad.net/tripleo/+bug/185570611:59
openstackLaunchpad bug 1855706 in tripleo "FS 01 train rhel8 overcloud deploy failed with failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory" [Critical,Triaged]11:59
marios|roverpanda: looks like it still isn't on old promoter... is the new one running in a service or are you running queens manually?12:01
marios|roverpanda: also are you using upstream criteria for that?12:01
marios|roverchkumar|ruck: ack12:01
marios|roverchkumar|ruck: are you sure it isn't duplicate for https://bugs.launchpad.net/tripleo/+bug/185302812:03
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]12:03
chkumar|ruckmarios|rover: in RHEL-8 fs01 we are consuming the built overcloud image so it is a different issue all together12:04
pandamarios|rover: manually, but I don't think I need to control the promotions there, the new promoter succeeded, with the modifications12:05
pandamarios|rover: and the criteria were the one in the latest commit, which probaby is not updated.12:06
marios|roverpanda: so once your patch merges ? https://review.rdoproject.org/r/#/c/23931/12:06
marios|roverpanda: we can run it as a service there?12:06
marios|roverpanda: more like a discussion we should have on the call in 1 hour12:06
marios|roverpanda: can you try re-running a promotion later ... i think we are just missing fs20 there (posted https://review.rdoproject.org/r/#/c/24033/ i can ping you later once it reports but we can discuss first on the phone12:08
pandamarios|rover: sure12:09
chkumar|ruckmarios|rover: Do we want to bug networking people about the same failure?12:09
marios|roverchkumar|ruck: not sure... find the trello there's some fix and discussion in there. not clear if that is networking or df12:10
chkumar|ruckok12:10
*** ykarel|afk is now known as ykarel12:11
arxcruzall: I'm in the new office, and it's a little bit messed here still12:23
rfolcosame @home here12:26
*** udesale has joined #oooq12:33
chkumar|ruckmarios|rover: please have a look at this card https://trello.com/c/WTFR2Z8E/1235-cixlp1853652tripleociproa-openstack-overcloud-node-provide-all-manageable-timing-out-and-failing-in-both-centos7-and-rhel8-jobs12:36
chkumar|ruckmarios|rover: I have closed this bug https://bugs.launchpad.net/tripleo/+bug/1855706 in favor of https://bugs.launchpad.net/tripleo/+bug/185302812:37
openstackLaunchpad bug 1853028 in tripleo "duplicate for #1855706 Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]12:37
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]12:37
marios|roverchkumar|ruck: ack12:38
marios|roverchkumar|ruck: noted a couple tempest issues queens/rocky fs20 on ether12:39
chkumar|ruckwill take a look, thanks!12:43
zbrchkumar|ruck: marios|rover : activate periodic for podman package: https://review.rdoproject.org/r/#/c/24036/12:51
*** rlandy has joined #oooq12:59
rfolcoscrum time zbr sshnaidm panda arxcruz chkumar|ruck13:00
rfolcopanda, joining?13:01
*** apetrich has quit IRC13:21
chkumar|ruckpanda: can you take a look why build is failing http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/cd32825/build.log ?13:22
*** skramaja has quit IRC13:26
*** zbr has quit IRC13:27
chkumar|ruckmarios|rover: sorry, we missed the current promotion run, it is not yet conistent https://trunk.rdoproject.org/centos7-master/status_report.html13:28
*** zbr has joined #oooq13:32
*** epoojad1 has joined #oooq13:32
pandachkumar|ruck: 2019-12-09 12:29:02.033 | ERROR:root:overcloud-base has no valid mapping for package openssl-perl13:35
pandachkumar|ruck: this is what I see for now, investigating13:35
rlandymarios|rover: chkumar|ruck: anything you want done/watched after you log off/my afternoon hours?13:35
*** Goneri has joined #oooq13:35
marios|roverrlandy: ack thanks will ping you later if so ?13:35
rlandymarios|rover: ack13:35
*** bhagyashris has quit IRC13:36
chkumar|ruckpanda: we are seeing the same error for lvm2 also13:57
pandachkumar|ruck: yep13:58
ykarelchkumar|ruck, centos7 master repo is consistent now14:01
ykarelu can ask ops to get the current periodic run aborted14:01
ykareland get a rerun14:01
ykarelto have results sooner14:02
chkumar|ruckykarel: marios|rover I am heading home, asked fbo to abort that currently14:03
chkumar|rucksee ya14:03
ykarelchkumar|ruck, ack14:03
marios|roverchkumar|ruck: k14:05
*** epoojad1 has quit IRC14:19
*** soniya29 has quit IRC14:22
migirlandy: hey have 5min over bluejeans ?14:22
migirlandy: re: extra component job14:23
rlandymigi: ack14:23
migirlandy: if you have then bluejenas.com/mpryc14:24
rfolcozbr, need help o/14:29
rfolcozbr, get a min ?14:29
zbrin 5min ok?14:29
sshnaidmchkumar|ruck, hi, can you please take a look at scenario012 in your time? It fails on tempest networks, not sure what is the problem there, maybe you'll know better14:29
rfolcozbr, ok thanks14:31
rfolcodoesn't have to be in live chat, pls take a look at this http://paste.openstack.org/show/787315/14:32
rfolcozbr, ^14:32
chkumar|rucksshnaidm: periodic or upstream?14:33
zbrrfolco: what is th eproble,? the imp module use?14:33
zbrwas dlrn_api tested with both py2/py3?14:34
rfolcozbr, good point maybe not14:34
rfolcomaybe its not ready for consumption on py314:34
migirlandy: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/14:35
rfolcozbr, I think I'll move forward without the module, I am spinning wheels there14:36
*** ykarel is now known as ykarel|afk14:36
zbrrfolco: that imp issue is very easy to fix, i am looking now for the codebase of the module14:36
zbrrfolco: give me ~1h...14:37
chkumar|ruckarxcruz: please merge this https://review.rdoproject.org/r/#/c/22862/14:37
zbralready started14:37
rfolcozbr,  if you think its something quick to fix, we can try it.14:37
rfolcozbr, thx14:38
zbrrfolco: is this what you are using https://github.com/softwarefactory-project/dlrnapi_client ?14:38
rfolcoyes14:39
rfolcogot an example from https://github.com/softwarefactory-project/dlrnapi_client/blob/master/dlrnapi_client/ansible/example_playbook.yaml14:39
*** TrevorV has joined #oooq14:39
rfolcozbr, and then switched to py3 on ansible.cfg14:40
sshnaidmchkumar|ruck, check job, it doesn't run on periodic14:42
chkumar|rucksshnaidm: check checking14:42
chkumar|ruckis it the ironic one?14:42
chkumar|rucksshnaidm: it is failing from very long time14:43
sshnaidmchkumar|ruck, yeah, and Ironic part should be solved now14:43
chkumar|ruckchecking now14:44
*** soniya29 has joined #oooq14:44
*** akahat has quit IRC14:50
*** akahat has joined #oooq14:51
*** soniya29 has quit IRC14:52
*** surpatil has quit IRC14:54
chkumar|ruckmarios|rover: we have killed master periodic pipeline14:59
rfolcozbr, didn't work on py3 :(15:01
marios|roverchkumar|ruck: yeah you said before... there is one more run today so lets see15:02
marios|roverin like 3 hours15:02
chkumar|ruckmarios|rover: we killed it right now15:03
marios|roverchkumar|ruck: oh you mean 16:03 #oooq: < chkumar|ruck> ykarel: marios|rover I am heading home, asked fbo to abort that currently15:03
marios|roverchkumar|ruck: didn't work 1 hour ago?15:03
chkumar|ruckmarios|rover: nope, he killed right now15:03
marios|roverchkumar|ruck: ack ok. same though there is one more in ~ 3 hours15:04
chkumar|ruckmarios|rover: let's watch them tomorrow15:04
marios|roverchkumar|ruck: yeah we'll see results of tonights run in our morning15:04
chkumar|ruck:-)15:04
chkumar|ruckarxcruz: still around?15:04
chkumar|ruckarxcruz: I am going to remove smoke tests from herhttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L23 and running it only in periodic as it takes too much time15:05
chkumar|ruckas we are depreacting os_tempest job15:05
chkumar|ruckrfolco: zbr https://github.com/rdo-infra/weirdo/blob/master/playbooks/dlrn-api-report.yml#L31 I am not sure it will be useful for you related to dlrn api client15:08
rfolcochkumar|ruck, does it work on py3 ?15:09
chkumar|ruckrfolco: need to check with dmsiard15:09
rfolcochkumar|ruck, thats what zbr is testing/fixing15:09
rfolcothx for the pointer chkumar|ruck15:10
chkumar|ruckrfolco: poked dms on #rdo channel15:10
rlandymigi: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml15:12
chkumar|ruckmarios|rover: sshnaidm rfolco rlandy zbr see ya tomorrow, will fix stuff tomorrow15:18
*** chkumar|ruck is now known as raukadah15:19
marios|roverhave a good one raukadah15:19
*** ykarel|afk is now known as ykarel|away15:25
zbrrfolco: create a card for dlrn_api py3 support and assign it to me, i know what needs to be done.15:26
*** ykarel|away has quit IRC15:31
*** ykarel|away has joined #oooq15:31
*** ykarel|away has quit IRC15:32
*** udesale has quit IRC15:36
rfolcozbr, you rock man15:37
pandaanyone care to give a last review and +2 ? https://review.rdoproject.org/r/2393115:38
rfolcozbr, https://tree.taiga.io/project/tripleo-ci-board/task/1429?kanban-status=144727515:38
rfolcozbr, sshnaidm rlandy arxcruz https://review.rdoproject.org/r/2393115:39
rfolcopls review panda's patch asap15:40
rlandythere's a lot there15:43
*** saneax has quit IRC15:53
mjturekasked on friday but figure I'd ask again since there are more people here16:11
mjturekhas anyone seen this error in the container build job before "2019-12-09 12:26:40 | ERROR:kolla.common.utils.base:The command '/bin/sh -c curl -L http://172.17.0.1/delorean.repo -o /etc/yum.repos.d/delorean.repo' returned a non-zero code: 7"16:12
mjturekseeing it here https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1820/logs/logs/build.log16:12
mjturekas far as I can tell the docker bridge is up and pingable, but haven't tried from a container16:12
mjturekalso the curl works locally, again didn't try from a container16:13
mjturekbaha ^16:14
*** akahat has quit IRC16:16
*** soniya29 has joined #oooq16:20
marios|roverpanda: fyi https://review.rdoproject.org/r/#/c/24033/ green if you have time even tomorrow to run the queens again thanks16:26
rfolcomarios|rover, raukadah panda have you seen this error before ? see mjturek message above16:28
marios|roverrfolco: no also replied on friday not seen that one in jobs16:29
rfolcothx marios|rover16:29
* marios|rover shutdown sequence16:29
*** d0ugal has quit IRC16:29
mjturekyep thanks marios|rover and rfolco :)16:29
rfolcomjturek, perhaps your theory about silent error is correct, now need to investigate it16:30
*** d0ugal has joined #oooq16:31
raukadahrfolco: is it possible to hold the node? where failure is coming?16:35
raukadahmay be we can jump an debug16:36
raukadahjpena|off will be coming tomorrow, I will ask for help for sure16:36
mjturekraukadah well need to adjust the job to hold on failure but i think e can do that16:37
mjturekyou'll need access to cico nodes16:37
raukadahmjturek: I will poke tomorrow a little early tomorrow then16:37
mjturekbaha ^ lets do tghe thing we discussed thurs16:37
mjturekadjust the job to hold node on fail16:38
*** marios|rover is now known as marios|out16:39
bahaOK! We're going to need to make the job run less frequently in the meantime to not eat all of cico's nodes16:39
*** soniya29 has quit IRC16:47
*** marios|out has quit IRC16:47
*** holser has quit IRC17:13
zbrrfolco: re dlrn, https://softwarefactory-project.io/r/#/c/17161/ and its dependency are ready.17:18
rfolcozbr, wow17:18
rfolcozbr, looking17:18
zbrrfolco: fwi, i does not fix that specific issue, for the moment only increasing the test matrix17:19
*** ykarel has joined #oooq17:19
rfolcozbr, this puts it in py3 game17:19
rfolcook17:19
zbrmainly i had to add centos-8 testing, we didn't even had the nodeset definition.17:19
*** ykarel is now known as ykarel|away17:19
zbronce i get this merged, i go to next step. which is to replicate that bug, and to fix it.17:20
zbrbut w/o job running on py3 i cannot fix it correctly.17:20
zbrit would be more of a guessing game17:20
rfolcozbr, nice, lets get jpena on it17:20
zbrrfolco: where is your call to "dlrn_api" module?17:25
rfolcozbr, I am trying it manually in a local test....17:26
rfolcozbr, let me get the paste again17:26
zbri have the paste but is not enough17:26
zbri need context17:26
zbrthe real error is that it fails to find the dlrn_client module, because you did not install it.17:26
rfolcozbr, so I created a virtualenv17:27
rfolcozbr, installed dlrapi_client w/ pip17:27
rfolcozbr, then I created the ansible.cfg on it and the playbook...17:27
zbrit needs to be inside venv used by ansible, ansible task will try to run on remove machine17:27
zbrshare the code17:27
rfolcohttp://paste.openstack.org/show/787315/17:27
zbrthis is not the code, this is the output17:28
rfolcosee17:28
rfolcodlrn_client.yaml17:28
rfolcoI did a cat with the ansible play there17:28
rfolco:)17:28
zbri do not see any code insall dlrn_api module17:28
rfolcoah17:28
rfolcoso17:28
zbrsorry, dlnr_client module, python module17:28
rfolcoI did it manually sorry17:28
rfolcoinside the venv17:28
rfolcoinstalled dlrnapi_client17:29
zbryou fixed ansible ability to find its own module, but this ansible module needs the python module inside the same env as the one used by ansible17:29
zbrwhere?17:29
rfolcoline #4117:29
rfolcoparsing your comment, maybe I did something wrong with venv17:30
zbradd this line:  command: {{ ansible_python_interpreter }} -c "import dlrn_client"17:30
zbrand you will find the issue17:30
rfolcozbr, as an additional task?17:30
zbryeah, for debuggin. btw, that pastebin sucks you need to do a lot of horizontal scrolling to see real error17:31
zbrno wrapping17:31
zbrModuleNotFoundError: No module named 'dlrnapi_client'17:31
zbrif you run ansible with -vvv you will also see the python import paths.17:32
zbryou may be able to address the issue by adding "connection: local"17:33
*** dtantsur is now known as dtantsur|afk17:33
zbrputting localhost does not mean it will connect directly, it will still ssh and not use your venv.17:33
zbrin fact is very easy to test using ad-hoc:  just run:  ansible  -m dlnr_api17:34
zbrprobably you will get the same exception17:34
rfolcohmm17:34
rfolcosec17:34
rfolcoyes ansible -m dlrn_api localhost retrurned me : No module named 'dlrnapi_client'17:36
rfolcozbr, ^17:36
zbrexactly what I said17:36
zbrpython fails to find the module in its path17:36
zbransible library path != python path17:37
zbrbut what I succesfully did in the past was to use another ansible task to install that module17:38
zbrbecause is published on pypa is quite easy17:38
rfolcowas hoping this to help17:41
rfolcolibrary = /usr/lib/python3.7/site-packages/dlrnapi_client/ansible:$VIRTUAL_ENV/lib/python3.7/site-packages/dlrnapi_client/ansible17:41
rfolcofinding the library17:41
zbrrfolco: see https://github.com/ssbarnea/harem/commit/d76226403c9b555e3c1f877d9dd1f20cf39047e6#diff-7aba2956bf0cba5d967cef55dfc34d73R34-R3817:41
zbrlibrary is for *ansible* modules, not python modules.17:41
rfolcoisn't what we are using here ?17:42
zbrwe have both17:42
rfolcoso the import inside the module is failing17:42
rfolcois it?17:42
zbrdlnr_api ansible module which needs dnr_client pythob module17:42
zbryep17:42
rfolcok17:42
zbrit finds the ansible module, runs it and faist to import python module at the begening.17:42
rfolcozbr, so i add a pip module before, and then re-use the same venv17:43
rfolcoin the next step17:43
zbrmy example may confuse you because the two modules happen to have the same name, but for dlnr is not the same.17:43
rfolcois that what you suggest?17:43
zbri prefer to use pip from ansible17:43
zbrsee if it works17:43
rfolcok17:44
zbri really do not understand why ansible team refused to make pure python modules, it would so easy to install them.17:45
zbri asked them and they didn't want it17:45
rfolcozbr, so I added and it still complains. Maybe I need to inform the virtualenv previously used in the pip step ?17:46
zbrlet me try to make a test playbook17:47
*** dsneddon has joined #oooq17:48
zbrrfolco: worked for me just fine17:55
zbrhttps://seashells.io/v/3v5FbYRg17:56
zbrrfolco: but what you get may be different, if ansible is installed as system package you may get a very different experience17:57
rfolcozbr, pls share code not result :)17:59
zbransible -m dlrn_api localhost -vv -a "action=repo-status host=foo"18:00
*** derekh has quit IRC18:01
zbrrfolco: another question on which OS are you running and with with version of ansible, i would not be surprised to discover that you have a mix18:03
rfolcozbr, still getting the imp import issue18:04
zbrthat is only a warning that you can ignore, scroll for the real error18:05
rfolcono module named 'dlrnapi_client'18:05
rfolco:(18:05
rfolcoI am sick of this18:05
rfolcowill stick with dlrnapi_client and leave this for future improvement18:05
zbrwhen it will packed as a collection it should be easier.18:06
rfolcozbr, ok cool, thanks for helping, and lets try to get it working for v2 next sprint18:06
zbrsure, time to go18:07
rfolcozbr, cool thx18:15
zbrrlandy: please put a +2 on https://review.rdoproject.org/r/#/c/24036/ to merge it.18:22
rlandydone18:24
*** d0ugal has quit IRC18:33
rlandyrfolco: any issue with my merging https://review.rdoproject.org/r/#/q/topic:keystone-pipeline+(status:open+OR+status:merged)?18:52
rlandyshould not impact anything else18:52
rfolcorlandy, should be fine18:53
rlandyrfolco: can you vote for some legitimacy?18:54
*** ykarel|away has quit IRC18:54
rfolcorlandy, sure18:54
rfolcorlandy, done, +1, one minor question, rebase. Merge.18:59
rlandyrfolco: that was the original plan18:59
rlandyI'm fine to redo it when we redesign in next sprint19:00
rlandyas weshay_ said this is POC19:00
rlandywe will redo it all19:00
rfolcook, no big deal, just need to think about scaling it19:00
rfolcothx19:00
rlandyrfolco: yep - reasonable question19:02
*** dsneddon has quit IRC19:16
*** dsneddon has joined #oooq19:19
*** dsneddon has quit IRC19:20
*** dsneddon has joined #oooq19:20
*** tesseract has quit IRC19:37
mjturekrfolco: here's a patch that will hold two nodes a day from baha https://review.rdoproject.org/r/#/c/24046/19:58
rfolcowill check it asap, hands tied right now20:01
baharfolco: Hold off, we found out cico's CLI doesn't actually expose the fail state so we need to write an API call20:03
rfolcobaha, mjturek ok I don't know how cico works btw :(20:03
rlandyrfolco: pls check me ... https://review.rdoproject.org/r/24047 Use tripleo-ci-testing for the component pipeline20:04
mjturekrfolco thanks, we're looking into how to handle it with curl :( https://wiki.centos.org/QaWiki/CI/Duffy20:04
rlandyzbr: https://review.rdoproject.org/r/#/c/23924/22/zuul.d/projects.yaml20:10
rlandypls add files to the podman tests20:11
rlandythey are running on all rdo-jobs changes20:11
rlandynot necessary20:11
baharfolco: https://review.rdoproject.org/r/#/c/24046/ updated w/ manual API call, should be good20:26
mjturekrfolco basically this will let us take a look at the node after the failure for a couple hours. Should be useful for tomorrow ^21:04
rfolcobaha, mjturek +2, if there is a way to test or show that it works please indicate in the patch. Also, get a revert prepared.21:12
rfolcothx21:13
rfolcootherwise lets just +w and watch21:13
mjturekbaha is submitting a revert patch quick21:20
mjturekwe don't have any testing to show it works other than a curl to the cico admin server21:20
mjtureka test curl that worked21:20
baharfolco: Revert's up at https://review.rdoproject.org/r/2404821:25
mjturekrfolco willing to wf it?21:34
rfolcoyes sorry21:35
rfolcowas finishing up a patch21:35
rfolcodoing now21:35
rfolcomjturek, baha sone21:35
rfolcodone21:35
mjturekthanks!21:35
rfolcoshutdown sequence in a few21:35
mjtureklolol21:35
rfolcolet me know if any others to review21:35
rfolco:)21:35
mjturekwill do!21:36
bahathank you rfolco21:37
rfolcoyw21:42
*** jtomasek has quit IRC21:45
*** Goneri has quit IRC21:56
*** brault has quit IRC21:59
*** dtrainor has quit IRC22:06
*** ysandeep has joined #oooq22:22
*** ysandeep has quit IRC22:50
*** TrevorV has quit IRC23:00
*** rlandy has quit IRC23:43
*** tosky has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!