Friday, 2018-08-17

*** agopi|out has joined #oooq02:22
*** agopi|out is now known as agopi02:24
*** matbu has quit IRC02:37
*** rlandy|bbl is now known as rlandy02:41
*** rlandy has quit IRC02:42
*** gkadam_ has joined #oooq03:24
*** ssbarnea has joined #oooq03:28
*** udesale has joined #oooq03:47
*** ssbarnea has quit IRC04:12
*** ssbarnea has joined #oooq04:21
*** skramaja has joined #oooq04:22
*** gkadam_ has quit IRC04:30
*** apetrich has joined #oooq05:05
*** ratailor has joined #oooq05:16
*** saneax has joined #oooq06:06
*** jfrancoa has joined #oooq06:22
*** bogdando has joined #oooq07:09
*** amoralej|off is now known as amoralej07:11
*** ykarel has joined #oooq07:31
ykarelis this issue known:- resources.ServiceChain: Property error: resources[68].properties: Property DockerKeepalivedConfigImage not assigned07:33
ykarelseeing here:- https://logs.rdoproject.org/82/591982/2/openstack-experimental/legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/2bf5069/logs/undercloud/home/zuul/failed_deployment_list.log.txt.gz07:34
marioshttps://bugs.launchpad.net/tripleo/+bug/1787268 ykarel maybe this07:37
openstackLaunchpad bug 1787268 in tripleo "Property DockerCinderConfigImage not assigned" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)07:37
ykarelmarios, ok so same fix for both cinder and keepalived ?07:42
mariosykarel: yeah i suspect so because the fix in https://review.openstack.org/#/c/592275/ is about all the container image params07:44
ykarelmarios, yes , Thanks07:45
*** ykarel is now known as ykarel|lunch07:46
marioswelcome ykarel|lunch07:46
*** jaosorior has quit IRC07:47
jfrancoaamoralej: ykarel|lunch: marios|rover: hey, could I get any reviews for https://review.rdoproject.org/r/#/c/15557/ ,please? it's supposed to fix all the container-to-container legacy upgrades jobs07:55
mariosack jfrancoa08:00
jfrancoamarios: thanks08:00
*** ykarel|lunch is now known as ykarel08:56
*** panda|off is now known as panda09:08
*** udesale has quit IRC09:31
bogdandoo/ folks, I forgot why we don't want that https://review.openstack.org/#/c/576746 ?09:50
bogdandostill hitting a lot of issues with deploying on my box w/o ssh access to itself09:50
*** udesale has joined #oooq10:31
bogdando^^ PTAL updated. It makes oooq better supporting non-disposable devboxes10:43
*** udesale has quit IRC11:26
weshay|ruckoh man.. where is rlandy12:21
weshay|ruckhttps://bugzilla.redhat.com/show_bug.cgi?id=161634712:21
openstackbugzilla.redhat.com bug 1616347 in openstack-glance "Undercloud deployment gets stuck after starting glance_api_db_sync" [Unspecified,Closed: notabug] - Assigned to cyril12:21
*** ratailor has quit IRC12:23
weshay|ruckbogdando, I'm not against it12:27
bogdandoweshay|ruck: thanks, that's already something! :)12:35
*** trown|outtypewww is now known as trown12:35
*** rlandy has joined #oooq12:39
rlandyweshay|ruck: hi - ticket says ci-rhos should be back - trying again12:41
weshay|ruckrlandy, I kicked a job12:44
rlandyyep - I see12:44
weshay|ruckhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/135/console12:44
rlandyI removed stack delete from that job12:45
rlandyso we could debug12:45
rlandyso we need to manually clean up12:45
rlandyjust fyi12:45
*** amoralej is now known as amoralej|lunch12:46
*** skramaja has quit IRC12:53
rlandyrfolco: hi13:08
rlandyrfolco: need to chat with you re: vars in zuul jobs13:09
rfolcorlandy, bj ?13:10
rlandyrfolco: yep - 5 mins - just checking something - will ping you13:11
rfolcorlandy, ok, its will be good to discuss different approaches to solve the problem13:12
rfolcorlandy, feel free to chat here if you prefer13:14
rlandyrfolco: k - let's bj13:16
rfolcograbbing coffee13:17
rlandyrfolco: https://bluejeans.com/u/rlandy/ - when you are ready13:17
apetrichrlandy, I'm working on https://bugs.launchpad.net/tripleo/+bug/1736950 but I'm unable to make it fail on my env. Do you have a reproducer script that always fails (or fails a lot at least>)13:23
openstackLaunchpad bug 1736950 in tripleo "CI: mistral testmistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_not_in_list_filter fails in gate scenario003 containers" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)13:23
rlandyapetrich: sec in meeting13:30
apetrichno worries13:30
*** amoralej|lunch is now known as amoralej13:43
*** jfrancoa has quit IRC13:46
rlandyapetrich: you tried this reproducer on your personal tenant? http://logs.openstack.org/16/592216/2/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/fc62c9d/logs/reproducer-quickstart.sh13:50
*** jaganathan has quit IRC13:52
apetrichrlandy, That's the one linked in the bug. I runned yesterday but it didn't finish for me.13:53
apetrichlet me run again to see what was wrong with it13:54
*** jfrancoa has joined #oooq13:58
rfolcorlandy, hmm actually ovb legacy has its own hierarchy so if we remove vars from environment_type/ovb.yaml, we have to add to rdo-jobs zuul config13:59
rfolcoovb jobs parent to --> legacy-tripleo-ci-periodic-multinode --> legacy-tripleo-ci-periodic-base13:59
rlandyapetrich: would be good to know why it did not finish13:59
rfolcorlandy, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/jobs.yaml#L9713:59
apetrichit was late, so probably my fault13:59
rlandyrfolco: those jobs never parented off tripleo-ci-base13:59
rlandyI am leaving legacy as they are14:00
rlandyonly concerned about the one job fe053 we are reparenting to tripleo-ci-base-ovb14:00
rfolcorlandy, ok it depends on how you are changing the env_type files to load those vars14:00
rlandyrfolco: yeah - the old type never touched those vars14:01
rlandyrfolco: what about pre-run? https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L3214:11
rlandy^^ only needed for multinode14:11
rlandyworth moving?14:11
rlandynot sure about ordering then14:11
rfolcorlandy, I would keep this for now as I am quite sure if this needs to run before the other pre-runs14:12
rfolcorlandy, worth adding a NOTE :)14:12
rlandyrfolco: k - will do - we can discuss14:12
rlandyrfolco: is this correct - https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/vars/environment_type/singlenode.yaml#L314:19
rlandyshould playbooks be in []?14:19
rlandylike tags14:19
rfolcoyep I got the same question14:20
rfolcorlandy, let me do a quick test14:21
rfolcorlandy, it works14:23
rfolcoalthough not consistent :)14:23
rfolcorlandy, https://review.rdoproject.org/paste/show/133/14:25
rlandyrfolco: https://review.openstack.org/#/c/593063 - note commit message - incomplete item to do14:27
rlandypanda: ^^14:27
rlandyjust want a quick review on this and then will work on incomplete items14:28
weshay|ruck2018-08-17 10:12:21 | 2018-08-17 10:12:21,378 INFO: Notice: /Stage[main]/Glance::Db::Metadefs/Exec[glance-manage db_load_metadefs]: Triggered 'refresh' from 3 events                                  │·············2018-08-17 10:27:21 | 2018-08-17 10:27:21,388 INFO: Error: Command exceeded timeout                                                                                                                    │······14:28
weshay|ruck·······2018-08-17 10:27:21 | 2018-08-17 10:27:21,390 INFO: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout14:28
weshay|ruckrlandy, ^14:28
rlandyweshay|ruck: https://bugs.launchpad.net/tripleo/+bug/166139614:29
openstackLaunchpad bug 1661396 in OpenStack Compute (nova) "undercloud install fails (nova-db-sync timeout) on VM on an SATA disk hypervisor" [Undecided,Incomplete]14:29
rlandy]was looking at that14:29
rlandyosp 13 does include the 900 sec change14:29
rlandybut maybe we should increase even that14:29
rlandyweshay|ruck, https://bugzilla.redhat.com/show_bug.cgi?id=1418782#c714:32
openstackbugzilla.redhat.com bug 1418782 in puppet-tripleo "rhosp-director: Undercloud deployment fails with: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout" [High,Closed: currentrelease] - Assigned to athomas14:32
rlandycould ping alex on his opinion14:32
rlandyweshay|ruck: also ...14:34
rlandy<jschlueter> puddle announcement for OSP 13 z2 just sent out ... finally passed phase 2 ci for 2018-08-16.114:34
rlandymaybe we should switch to z214:34
weshay|ruckrlandy, ok14:46
weshay|ruckrlandy, one more shot on z2 ovb14:46
rlandyweshay|ruck: I think the nova dbsync timeout may still exist14:46
rlandyweshay|ruck: what do you think about increasing that?14:47
* rlandy doesn;t know how to commit a change to osp13 to test14:47
rlandyqueens branch?14:47
rlandywill that take?14:47
rlandyhmmm ... why didn't we get offered a RDO rubber ducky?  https://twitter.com/RDOcommunity/status/103012076877085491314:50
weshay|ruckrlandy, you cool w/ https://review.openstack.org/#/c/587880/14:52
rlandyweshay|ruck: yeah - should not be needed but no harm14:53
rlandyweshay|ruck: pls see question above (re: nova-db-sync timeout)14:54
weshay|ruckrlandy, is that something we can pass as a param?14:55
weshay|ruckthe timeout?14:56
rlandyweshay|ruck: I don;t think so - can ask on #tripleo14:56
rlandyhence question about how to add a change osp will take14:56
weshay|ruckrlandy, maybe in hiera14:56
rlandyweshay|ruck: I think we will see the same problem on z2 - I could be wrong though14:57
rlandybut I'm guessing the hw is not good enough14:57
weshay|ruckhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j214:57
ykarelweshay|ruck, need to push https://review.openstack.org/#/c/592365/ as well to clear rocky promotions, if it's already being tracked somewhere then it's ok14:58
rlandyweshay|ruck: I can try add it to that template14:59
rlandylet's see15:00
* rlandy edits job15:01
ykarelweshay|ruck, current failure are real one, the job is looking for centos7-rocky/current-tripleo, so recheck will not work15:01
ykarelit seems a chicken egg problem15:01
weshay|ruckykarel, w/ https://review.openstack.org/#/c/592365/ ?15:02
ykarelweshay|ruck, yes15:02
ykarelweshay|ruck, see http://logs.openstack.org/65/592365/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/c8d2d56/job-output.txt.gz#_2018-08-17_11_13_05_89327215:02
ykarelso it looks for stable_release -1, that is rocky after that patch. and since there is no promotion centos7-rocky/current-tripleo do not exist15:03
weshay|ruckykarel, ya.. it should not be using rocky15:03
weshay|ruckk15:03
weshay|ruckykarel, we'll fix it15:04
ykarelweshay|ruck, ack Thanks15:04
*** ykarel is now known as ykarel|away15:04
rlandyweshay|ruck: I am deleting that stack15:08
*** ykarel|away has quit IRC15:12
*** jfrancoa has quit IRC15:16
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/147394/15:25
weshay|ruckrlandy, merged15:34
rlandyweshay|ruck: thank - here we go again ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/136/console15:36
rlandyweshay|ruck: https://review.openstack.org/#/c/593077/1/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j215:36
rlandy^^ tried that15:36
weshay|ruckrfolco, http://codesearch.openstack.org/15:43
*** saneax has quit IRC15:45
weshay|ruckhttps://github.com/openstack/tripleo-heat-templates/tree/master/zuul.d16:07
weshay|ruckrfolco, ^16:07
weshay|ruckrfolco, https://github.com/openstack/tripleo-heat-templates/blob/master/zuul.d/layout.yaml16:07
weshay|ruckhttps://github.com/openstack/tripleo-heat-templates/blob/stable/pike/zuul.d/layout.yaml16:08
weshay|ruckrfolco, ^16:08
*** ykarel has joined #oooq16:21
*** ykarel has quit IRC16:32
*** amoralej is now known as amoralej|off16:44
weshay|ruckrfolco, https://docs.openstack.org/tripleo-docs/latest/contributor/index.html16:46
*** bogdando has quit IRC16:52
*** bogdando has joined #oooq16:53
rlandyweshay|ruck: hmm ... did I make the change in the right file? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j2 vs https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/hieradata-overrides-classic-undercloud.yaml.j2?16:55
rlandyonly the second file is included in /home/stack16:55
weshay|ruckrlandy, ha.. tbh.. I would have to check and try17:05
weshay|ruckthat is VERY confusing17:05
weshay|ruckit was originally named undercloud-blah.. but was hierra config for the overcloud17:05
weshay|ruckso when the ability came and we could do the same on the undercloud17:06
weshay|ruckwhat do you name it????17:06
*** myoung is now known as myoung|bbl17:08
apetrichrlandy, I see why it failed to error that bug. the skip list was merged by the time it ran17:11
rlandyweshay|ruck: I think I changed the wrong file but if it fails will run again with the other file changed17:23
*** trown is now known as trown|lunch17:24
rlandyapetrich: cool17:29
rlandypanda: thanks for the review - updated  - now will add the missing pieces17:30
*** bogdando has quit IRC17:48
rlandyweshay|ruck: hmmm ... for better or worse we have a different error this time ... https://paste.fedoraproject.org/paste/dnVoNTGlVuMV6JX36uYddg18:08
rlandyopening undercloud on the old tmate18:09
rlandy  return glareclient.Client18:13
rlandy2018-08-17 14:02:06.440 31775 ERROR mistral.actions.openstack.action_generator.base     return glareclient.Client18:17
rlandy2018-08-17 14:02:06.440 31775 ERROR mistral.actions.openstack.action_generator.base AttributeError: 'NoneType' object has no attribute 'Client'18:17
rlandyrfolco: I think we are doing duplicate work18:29
rlandyhttps://review.openstack.org/#/c/593129/18:29
rlandyvs18:29
rlandyhttps://review.openstack.org/#/c/593063/18:29
rfolcorlandy, I know. This is about getting up to speed more than what patch will go further18:29
rlandyrfolco:ok then18:30
rfolcorlandy, I want to study possible solutions for the problem18:30
rlandyrfolco: np18:30
rfolcorlandy, I tried to add as much as i could to the patch so I can work on any task this sprint18:31
rfolcoand be a good qe for others as i did my own poc :)18:32
rlandyrfolco: that's fine18:40
rfolcorlandy, I guess nobody is working on reproducer side of it. How we load these vars...18:40
rfolcoworing --> poc18:40
rfolcoworking18:40
rlandyrfolco: when we assign tasks, I can do that18:41
rfolcohmm perhaps you have this in your mind already18:41
rfolcoI have no clues18:41
rlandyneed to step out for a but dr appointment. van look at it when I return18:41
rlandycan18:41
rfolcorlandy, cool, thank you18:42
*** rlandy is now known as rlandy|afk18:48
*** trown|lunch is now known as trown18:55
*** apetrich has quit IRC19:26
*** apetrich has joined #oooq20:33
*** rlandy|afk is now known as rlandy20:36
*** trown is now known as trown|outtypewww21:00
*** apetrich has quit IRC21:10
rlandyweshay|ruck: around?21:45
rlandyha - not bad ... https://review.openstack.org/#/c/593063/21:49
rlandyweshay|ruck: 2018-08-17 19:16:00 | Undercloud install complete.23:18
rlandyparty ... we go through undercloud install ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/137/console23:19
*** rlandy has quit IRC23:27

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!