Tuesday, 2016-06-28

*** [1]cdearborn has joined #tripleo00:06
*** bfournie has joined #tripleo00:12
*** julim has joined #tripleo00:26
*** tzumainn has quit IRC00:34
*** dmacpher has joined #tripleo00:35
*** xinwu has joined #tripleo00:38
*** rwsu has joined #tripleo00:39
*** noslzzp has quit IRC00:40
*** noslzzp has joined #tripleo00:45
*** julim has quit IRC00:52
*** xinwu has quit IRC00:53
*** limao has joined #tripleo01:01
*** xinwu has joined #tripleo01:12
*** lblanchard has joined #tripleo01:13
*** stendulker has joined #tripleo01:17
*** akshai has joined #tripleo01:32
*** akshai_ has joined #tripleo01:33
*** noslzzp has quit IRC01:37
*** akshai has quit IRC01:37
*** noslzzp has joined #tripleo01:40
*** akshai_ has quit IRC01:54
*** lblanchard1 has joined #tripleo02:06
*** lblanchard2 has joined #tripleo02:09
*** lblanchard has quit IRC02:10
*** lblanchard1 has quit IRC02:11
*** fragatina has joined #tripleo02:18
*** fragatina has quit IRC02:18
*** fragatina has joined #tripleo02:19
*** lblanchard2 has quit IRC02:19
*** akshai has joined #tripleo02:23
*** bfournie has quit IRC02:24
*** stendulker has quit IRC02:28
EmilienMok so nonha job is fixed but ha & upgrade look broken02:31
*** chlong has quit IRC02:37
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/159675802:47
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]02:47
*** chlong has joined #tripleo02:49
*** yamahata has quit IRC02:52
*** akshai has quit IRC02:56
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675803:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]03:10
*** limao has quit IRC03:10
*** [1]cdearborn has quit IRC03:11
*** tserong has quit IRC03:33
*** limao_ has joined #tripleo03:36
*** yamahata has joined #tripleo03:44
*** links has joined #tripleo03:50
*** numans has quit IRC03:53
*** numans has joined #tripleo03:56
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675804:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]04:10
*** ramishra_ has joined #tripleo04:18
*** numans has quit IRC04:21
*** ramishra has quit IRC04:21
*** jtomasek_ has joined #tripleo04:32
*** fragatina has quit IRC04:39
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Validations new workflow  https://review.openstack.org/33477004:39
*** numans has joined #tripleo04:54
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675805:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]05:10
*** jtomasek_ has quit IRC05:12
*** itamarl has joined #tripleo05:13
*** saneax_AFK is now known as saneax05:13
*** ayoung has quit IRC05:14
*** hanchao has quit IRC05:16
*** fragatina has joined #tripleo05:20
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Temporarily disable the cached images  https://review.openstack.org/33463405:23
*** paramite has quit IRC05:25
*** hanchao has joined #tripleo05:28
*** skramaja has joined #tripleo05:42
*** rajinir has quit IRC05:44
*** oshvartz has joined #tripleo05:52
*** jtomasek has quit IRC05:55
openstackgerritIan Wienand proposed openstack/diskimage-builder: Making element overriding explicit  https://review.openstack.org/33478506:00
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML  https://review.openstack.org/33478606:00
openstackgerritIan Wienand proposed openstack/diskimage-builder: Clear up "already provided" message  https://review.openstack.org/29096806:04
openstackgerritIan Wienand proposed openstack/diskimage-builder: Convert element_dependencies to logging  https://review.openstack.org/32807206:04
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML  https://review.openstack.org/33478606:04
openstackgerritIan Wienand proposed openstack/diskimage-builder: Making element overriding explicit  https://review.openstack.org/33478506:04
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478706:05
*** tbonds has quit IRC06:07
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478706:08
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675806:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]06:10
*** bootsha has joined #tripleo06:21
*** tremble has joined #tripleo06:23
openstackgerritSanjay Upadhyay proposed openstack/tripleo-specs: tripleo-sriov  https://review.openstack.org/31387206:27
*** liverpooler has joined #tripleo06:29
*** davidlenwell has quit IRC06:30
*** rasca has joined #tripleo06:33
*** derekh has joined #tripleo06:35
derekhI'm thinking of just merging this, as its holding up CI on rh2, only 3 days left to get it working https://review.openstack.org/#/c/295243/06:36
derekhanybody want to pull the trigger ^^ , if not I will myself in a bit06:36
*** davidlenwell has joined #tripleo06:40
*** ChanServ sets mode: +v davidlenwell06:40
openstackgerritMerged openstack-infra/tripleo-ci: Add infrstucture scripts to prepare rh2  https://review.openstack.org/29524306:43
derekhdone06:43
*** derekh has quit IRC06:43
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci  https://review.openstack.org/32477706:46
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Fix openstack-tripleo-common package name  https://review.openstack.org/33446706:46
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478706:46
*** pcaruana has joined #tripleo06:46
*** jprovazn has joined #tripleo06:48
*** noslzzp has quit IRC06:52
*** oneswig has joined #tripleo06:57
*** tesseract- has joined #tripleo06:59
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478707:00
*** ebalduf has quit IRC07:00
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML  https://review.openstack.org/33478607:02
openstackgerritIan Wienand proposed openstack/diskimage-builder: Making element overriding explicit  https://review.openstack.org/33478507:02
*** olap has joined #tripleo07:03
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675807:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]07:10
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add a method to wait for messages on the websocket  https://review.openstack.org/32261107:13
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal registration  https://review.openstack.org/32261207:13
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal introspection  https://review.openstack.org/32778007:13
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Don't cache and re-use the websocket  https://review.openstack.org/33198507:13
*** oneswig has quit IRC07:14
*** amoralej|off is now known as amoralej07:15
*** liverpoo1er has joined #tripleo07:16
*** liverpooler has quit IRC07:17
*** florianf has joined #tripleo07:18
*** oneswig has joined #tripleo07:20
*** jpich has joined #tripleo07:20
*** oneswig has quit IRC07:21
*** fzdarsky has joined #tripleo07:23
*** mcornea has joined #tripleo07:24
*** athomas has joined #tripleo07:30
*** ifarkas has joined #tripleo07:31
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478707:31
*** zoli_gone-proxy is now known as zoliXXL07:32
*** ebarrera has joined #tripleo07:35
openstackgerritMartin André proposed openstack/instack-undercloud: Add tripleo deployment config file paths to hiera  https://review.openstack.org/32901607:36
openstackgerritMartin André proposed openstack/instack-undercloud: Introduce 'enable_validations' option  https://review.openstack.org/32289307:36
*** dsariel has joined #tripleo07:37
openstackgerritMartin André proposed openstack/instack-undercloud: Add tripleo deployment config file paths to hiera  https://review.openstack.org/32901607:47
openstackgerritMartin André proposed openstack/instack-undercloud: Introduce 'enable_validations' option  https://review.openstack.org/32289307:47
*** chem has joined #tripleo07:48
*** dmacpher has quit IRC07:50
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: Move nova constraints to tripleo-puppet.  https://review.openstack.org/33207107:53
*** shardy_afk is now known as shardy07:55
*** aufi has joined #tripleo08:00
*** olap has quit IRC08:01
*** oneswig has joined #tripleo08:02
shardyMorning all08:03
*** olap has joined #tripleo08:03
shardyDoes anyone have an update on where we are wrt getting CI working again?08:03
shardycistatus still looks very red :(08:03
shardyanything that needs reviews or other help?08:04
*** dmk0202 has joined #tripleo08:06
ccamachomorning!08:07
*** andreas-f has quit IRC08:08
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675808:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1596758 in tripleo "pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]08:10
*** _milan_ has joined #tripleo08:14
*** gfidente has joined #tripleo08:15
*** dtantsur|afk is now known as dtantsur08:15
*** milan has quit IRC08:17
*** oneswig has quit IRC08:26
*** jtomasek_ has joined #tripleo08:33
*** dsariel has quit IRC08:33
openstackgerritMerged openstack/diskimage-builder: Fix sphinx-build to not depend on diskimage-builder  https://review.openstack.org/33191108:34
*** fzdarsky has quit IRC08:40
*** sshnaidm|afk has quit IRC08:41
*** mbound has joined #tripleo08:42
*** bootsha has quit IRC08:43
shardyjistr: Hi, thanks for the comment on https://bugs.launchpad.net/tripleo/+bug/159675808:49
openstackLaunchpad bug 1596758 in tripleo "[CI down] RabbitMQ fails to form a cluster" [Critical,Confirmed]08:49
shardyDo you have any further analysis on how we might fix it, and/or are you working on it?08:50
jistrshardy: morning :)08:50
jistrdoes anyone hit the problem locally?08:50
jistront their env?08:50
shardyjistr: I was about to try, but didn't want to duplicate effort if you're already fixing it ;)08:50
jistri'm not sure how to fix it...08:51
jistrit's very similar to a thing we've hit and fixed downstream recently08:51
jistrthe fix was this https://github.com/ClusterLabs/resource-agents/commit/bbbc52e335967dae3e548fe8c17bb162b001147c.patch08:51
jistrit's not in any released packages yet AFAIK08:52
jistri'm not sure how to best try to apply this as a workaround in tripleo08:52
*** abehl has joined #tripleo08:52
jistrprobably should happend during image build phase, but we use cached images...08:52
*** andreas-f has joined #tripleo08:53
shardyhmm08:53
* shardy looks around for derekh08:53
shardyjistr: Ok, thanks for the info08:53
shardyI guess we can get that into the image somehow until the fix is released08:53
shardyany idea what package that ends up in?08:54
jistrshardy: i'm not sure if it will fix it, but it might. The issue downstream was that rabbitmq failed to rejoin the cluster, but here it looks like it fails on the first try, so it might be different. But the error message itself looks practically the same.08:54
jistrshardy: the package is resource-agents, we get it from CentOS 7 updates repo08:55
shardyjistr: Ok, thanks08:56
*** jistr is now known as jistr|mtg09:04
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675809:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] RabbitMQ fails to form a cluster" [Critical,Confirmed]09:10
*** bootsha has joined #tripleo09:14
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478709:20
*** limao_ has quit IRC09:24
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles  https://review.openstack.org/31552709:25
*** limao has joined #tripleo09:28
*** olap has quit IRC09:29
*** olap has joined #tripleo09:31
*** jistr|mtg is now known as jistr09:31
openstackgerritMerged openstack/diskimage-builder: Run RHEL system unregister element earlier  https://review.openstack.org/33286609:32
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478709:33
*** derekh has joined #tripleo09:33
*** fzdarsky has joined #tripleo09:36
*** bootsha has quit IRC09:38
*** dsariel has joined #tripleo09:39
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Workaround for resource-agents bug  https://review.openstack.org/33488509:48
jistrshardy, derekh ^^ we'll see09:48
*** bootsha has joined #tripleo09:49
shardyjistr: ah, thanks09:49
*** limao has quit IRC09:50
shardyjistr: I was about to do similar, but by customizing the overcloud-full image ;)09:50
jistrshardy: that might be a better approach. E.g. if we do it in tripleo.sh we'd get it in dev envs too09:50
shardyjistr: Yeah, either is fine to prove if the patch helps tho09:51
*** sshnaidm|afk has joined #tripleo09:51
shardyjistr: do we have an ETA on the release of that fix?09:51
jistrshardy: yea, i wanted something relatively quick to verify if this is fixing the problem and worth pursuing further. I don't think there's an ETA but if it proves helpful for upstream too, i think we can query that from clusterlabs folks.09:54
* jistr -> lunch09:54
*** jpena|off is now known as jpena09:55
*** tosky has joined #tripleo09:57
*** akrivoka has joined #tripleo09:57
*** hanchao has quit IRC09:58
openstackgerritSaravanan KR proposed openstack/python-tripleoclient: NTP arg should be optional when specified in an environment  https://review.openstack.org/33269410:03
cmyster7/10:04
*** bfournie has joined #tripleo10:07
*** bfournie has quit IRC10:07
*** bfournie has joined #tripleo10:07
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675810:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] RabbitMQ fails to form a cluster" [Critical,Confirmed]10:10
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478710:13
*** sambetts|afk is now known as sambetts10:14
*** _milan_ has quit IRC10:22
*** zoliXXL is now known as zoli|afk10:25
*** liverpoo1er has quit IRC10:25
*** liverpooler has joined #tripleo10:25
openstackgerritSaravanan KR proposed openstack/python-tripleoclient: NTP arg should be optional when specified in an environment  https://review.openstack.org/33269410:25
*** weshay has quit IRC10:27
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478710:35
*** fzdarsky is now known as fzdarsky|lunch10:44
*** dtantsur is now known as dtantsur|lunch10:48
*** fzdarsky|lunch has quit IRC10:50
sshnaidm|afkderekh, can you please look at this in your time? https://review.openstack.org/#/c/33430510:50
derekhsshnaidm|afk: yup, looking now10:51
*** sshnaidm|afk is now known as sshnaidm10:52
*** hewbrocca-afk is now known as hewbrocca10:54
*** jcoufal has joined #tripleo10:54
derekhsshnaidm: do you know where JOB_NAME went too?10:56
sshnaidmderekh, sorry?10:57
hewbroccaMorning folks10:57
derekhsshnaidm: the patch is setting JOB_NAME, do you know why it needs to be set? I curious to know what has changed10:58
*** zoli|afk is now known as zoli|wfh10:58
hewbroccaCI green?10:58
*** zoli|wfh is now known as zoliXXL10:58
sshnaidmderekh, https://github.com/openstack-infra/tripleo-ci/blob/master/toci_instack.sh#L32411:00
sshnaidmderekh, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-nonha/095c4ed/console.html#_2016-06-25_08_10_01_22092111:00
derekhsshnaidm: yes, I know why we need it, but that line used to work, I'm just wondering what has changed11:01
sshnaidmderekh, ah, no, I don't know yet11:01
derekhhewbrocca: nonha is green, jistr has a patch to fix/workaround a problem with the other 2 jobs11:02
jistrderekh, hewbrocca: it's not certain though. The error looked the same as what we've tackled downstream with minor updates so i tried to submit the same fix. I don't have an opportunity to test it locally ATM so we'll see what CI says.11:04
sshnaidmderekh, maybe it worked with jenkins, was in global parameters11:04
jistrif someone hits the issue locally and can test if https://review.openstack.org/334885 fixes it, that'd be cool11:05
derekhsshnaidm: ok11:05
*** tbonds has joined #tripleo11:06
sshnaidmderekh, yeah, I guessed right, JOB_NAME is one of jenkins parameters :)11:06
*** weshay has joined #tripleo11:06
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101111:06
derekhsshnaidm: ok +2, although it would be nice if there was a similar variable,11:07
derekhsshnaidm: that patch ^^ is running env to try and find one11:08
sshnaidmderekh, you mean from zuul? I searched, nothing found11:08
sshnaidmderekh, yeah, I also ran env in job11:08
sshnaidmderekh, http://logs.openstack.org/88/334288/1/check-tripleo/gate-tripleo-ci-centos-7-ha/6317489/console.html#_2016-06-27_05_40_56_84089811:09
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675811:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] RabbitMQ fails to form a cluster" [Critical,Confirmed]11:10
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud node provide' command  https://review.openstack.org/33441111:16
*** fragatin_ has joined #tripleo11:22
*** fragatina has quit IRC11:22
*** thrash|g0ne is now known as thrash11:25
*** bfournie has quit IRC11:27
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478711:29
*** fragatin_ has quit IRC11:30
hewbroccaDid we get the swift OOM issue solved, BTW?11:31
*** rodrigods has quit IRC11:32
*** rodrigods has joined #tripleo11:32
*** panda has quit IRC11:34
*** milan has joined #tripleo11:35
*** panda has joined #tripleo11:35
derekhhewbrocca: more avoided then resolved, our glance is temporarily not using swift https://review.openstack.org/#/c/334555/11:37
weshaysshnaidm, can you investigate the last four gate failures on centos-7-nonha.. there are two python-tripleoclient failures and two ci failures11:38
*** rhallisey has joined #tripleo11:38
derekhtrown|outtypewww: you gonna want to change this in tripleo-quickstart http://66.187.229.99/builds/current-tripleo/ .....11:39
derekhtrown|outtypewww: that IP address wont exist when the rack goes down next week11:39
yolandahi, can i get some reviews on a change for tripleo-quickstart? https://review.openstack.org/32955611:39
derekhtrown|outtypewww: we should make it a DNS entry or something that we can change11:39
yolandait allows to clone and specify custom templates11:39
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Allow to specify templates path on overcloud deployment  https://review.openstack.org/32955611:40
*** lucasagomes is now known as lucas-hungry11:42
*** wfoster is now known as wfoster_afk11:43
*** wfoster_afk is now known as wfoster11:43
*** jpich has quit IRC11:46
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates: Dump and restore galera db during major upgrades  https://review.openstack.org/32520511:46
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles  https://review.openstack.org/31552711:47
EmilienMhello11:51
*** fragatina has joined #tripleo11:54
*** jpena is now known as jpena|lunch11:54
ccamachoo/11:54
ccamachoHey EmilienM, man found lot of nits in the gnocchi submission... One of them i.e. service_manage -> manage_service .... I'm fixing and testing one by one.. as that's the issue...11:56
*** jayg|g0n3 is now known as jayg11:56
EmilienMccamacho: ok but our CI is still broken isn't?11:56
ccamachothe nonha is working11:56
EmilienMyeah but the ha & upgrade jobs are broken11:57
EmilienMI won't review the gnocchi thing & continue composable work until CI is back11:57
shardyEmilienM: we're waiting to see if https://review.openstack.org/#/c/334885/ helps with the HA failures11:57
*** paramite has joined #tripleo11:57
ccamachoand jistr submitted some patches for fixing it the other jobs.11:57
ccamachoEmilienM ^11:58
EmilienMjistr: thanks for looking at it11:58
shardyhttps://github.com/ClusterLabs/resource-agents/commit/bbbc52e335967dae3e548fe8c17bb162b001147c.patch is the patch in question11:58
ccamachojistr ++11:58
jistrshardy, EmilienM, ccamacho: sure thing. The fix didn't work though :(11:58
shardydoh! :(11:58
dciabrinmarios, hey hi, I'm updating https://review.openstack.org/#/c/325205 according to your comments11:58
EmilienMshardy: https://github.com/ClusterLabs/resource-agents/commit/bbbc52e335967dae3e548fe8c17bb162b001147c.patch is the patch that caused this failure?11:59
shardyEmilienM: No, it's a fix jistr made which hasn't yet been released11:59
jistrEmilienM: no that's the one that was supposed to fix it :)11:59
shardywe were hoping it might resolve the HA cluster related failures11:59
jistrbut we must be hitting something else i guess11:59
chemjistr: shardy EmilienM last time I looked this morning I was hitting Error: Could not prefetch keystone_tenant provider 'openstack': Command: 'openstack ["project", "list", "--quiet", "--format", "csv", "--long"]' has been running for more then 20 seconds!12:00
*** bfournie has joined #tripleo12:01
ccamachojistr, Spain lost against Italy and now you are telling me that, you are breaking my heart.. :P12:01
EmilienMchem: it might be related to the cluster12:01
jistrhehe12:01
EmilienMchem: usually, this thing happens when VIP or service is not ready12:01
EmilienMchem: and service is apache, managed by pacemaker12:01
chemEmilienM: well, I discussed this withe last week, and it may related to poor performance on the cluster side.  So I though I started something to be able to tweek timeout values for keystone provider : https://review.openstack.org/#/c/334011/12:02
EmilienM20s is not enough?12:03
chemEmilienM: well, it timeouts so it's not :)12:03
*** zoliXXL is now known as zoli|lunch12:03
chemEmilienM: or we are hitting something else, but at least pure timeout would be excluded (the cluster has some low performance node in it I was told)12:04
*** bfournie has quit IRC12:07
slagleEmilienM: any chance one of the recent puppet-keystone commits might be causing it?12:08
slaglei don't see anything obvious12:08
*** numans has quit IRC12:08
slaglebut maybe someone more familiar with it could check12:08
*** amoralej is now known as amoralej|lunch12:09
*** jpich has joined #tripleo12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675812:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] RabbitMQ fails to form a cluster" [Critical,Confirmed]12:10
hewbroccaworth grabbing a Rabbit expert to have a look ^^^12:10
hewbrocca?12:10
EmilienMslagle: yes maybe12:10
EmilienMslagle: I'm currently investigating12:10
slaglehttps://review.openstack.org/#/c/329807/ maybe?12:10
jistrhmm interesting... looking at another CI job, rabbit formed a cluster fine there, and all pacemaker resources are started12:10
jistrhttp://logs.openstack.org/86/334486/2/check-tripleo/gate-tripleo-ci-centos-7-ha/b79b36f/logs/overcloud-controller-0/var/log/host_info.txt.gz12:11
*** bfournie has joined #tripleo12:11
chemslagle: if you have the timeout error, then I dig into the log and they point to a simple transitional timeout, as later on some keystone provider achieve to create their resource.12:11
jistrbut still pingtest fails12:11
*** bfournie has quit IRC12:11
*** bfournie has joined #tripleo12:11
sshnaidmweshay, the last failures I think are the patch problems, not CI, as I looked at it briefly12:11
EmilienMhttps://review.openstack.org/#/q/project:openstack/puppet-keystone+status:merged++branch:master12:12
*** dtantsur|lunch is now known as dtantsur12:12
slaglethis is the tb in keystone.log i'm seeing: http://paste.openstack.org/show/523769/12:12
EmilienMchem: recent commits didn't pass tripleo CI (for external reasons, so we might have injected a regression)12:12
slaglefrom when puppet is running the "openstack token issue ..."12:13
*** lblanchard has joined #tripleo12:13
chemslagle: do you have the associated puppet run log ?12:14
chemEmilienM: yep, but this timeout error is not a conection nor an identification problem, as they produce different type of message12:15
*** trown|outtypewww is now known as trown12:15
EmilienMright12:17
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478712:18
*** athomas has quit IRC12:18
*** [1]cdearborn has joined #tripleo12:20
slaglechem: yes, the puppet logs are in the controller journal12:21
slaglechem: you can download, http://logs.openstack.org/93/322893/12/check-tripleo/gate-tripleo-ci-centos-7-ha/0b6a39a/logs/overcloud-controller-0.tar.xz12:21
chemslagle: you should check this file for puppet error : http://logs.openstack.org/93/322893/12/check-tripleo/gate-tripleo-ci-centos-7-ha/0b6a39a/logs/postci.txt.gz  a bit easier on the eyes12:21
slagleuncompress that, then do journalctl --file /var/log/journal/(some-uuid)/system.journal12:21
mariosdciabrin: ack thanks12:22
*** tosc_fiberdata has joined #tripleo12:22
EmilienMslagle: I first see Error: unable to connect to nodes ['rabbit@overcloud-controller-1']: nodedown\n\nDIAGNOSTICS\n===========\n\nattempted to contact12:22
*** athomas has joined #tripleo12:23
EmilienMand then: Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Command: 'openstack ["token", "issue", ["--format", "value"]]' has been running for more ...12:23
EmilienMbut my theory is that it's because pacemaker didn't start httpd12:23
EmilienMlet me check if it did12:23
tosc_fiberdatahello, tripleO and webgui? there was Tuskar-UI earlier, is there anything similar?12:25
EmilienMtosc_fiberdata: https://github.com/openstack/tripleo-ui - ping jrist or akrivoka about it12:25
shardytosc_fiberdata: Hi!  Yes, there is a new UI currently under development called tripleo-ui12:25
tosc_fiberdatashardy, is it ready for production environments?12:26
*** aufi has quit IRC12:26
EmilienMchem, slagle: mhh, looking at httpd logs & pacemaker logs, httpd seems started12:26
shardytosc_fiberdata: Not yet, the CLI provided via python-tripleoclient is still the primary interface while we develop the UI for the newton cycle12:26
chemslagle: EmilienM : httpd is started as latter on you have some resource created : Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: created12:27
tosc_fiberdataso there's no UI for mitaka?12:27
shardytosc_fiberdata: No, the CLI is the only supported/documented interface for mitaka12:27
tosc_fiberdatawe do not have a choice than to build the impossible heat-templates? :<12:27
tosc_fiberdataah okey12:28
shardytosc_fiberdata: you just do "openstack overcloud deploy --templates"12:28
d0ugalMore at http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html12:28
shardythere's some interaction with heat environment files to enable advanced features, but you don't have to hack on templates unless you wish to customize things or integrate stuff we don't yet support12:29
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying RGW on controllers. (Composable)  https://review.openstack.org/28902712:29
EmilienMchem: http://paste.openstack.org/show/PV4E2UY2gEbtKHbFZrBQ/12:29
EmilienMit's in Notice, not in Error o_O12:30
chemEmilienM: "notice" those are the non created resource, the error is there.  If you look L10 of the paste, then you see that Keystone_service[keystone::identity]/ensure: created12:30
EmilienMright, even the endpoint12:31
EmilienMbut look12:31
chemEmilienM: they are not "skipped" because they don't depend on user admin12:31
EmilienMit tries to create Keystone_user_role[admin@admin] before Keystone_user[admin]12:32
EmilienMthat's why I did https://review.openstack.org/33244012:32
tosc_fiberdatashardy, it's not alot of "just do". but we'll try to figure it out12:32
EmilienMchem: you see Keystone_user in catalog? I don't12:33
chemEmilienM: It's not true : /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Keystone_user[admin] has failures: true12:33
EmilienMsomething is skipping Keystone_user12:33
shardytosc_fiberdata: ack, well let us know if you have questions and we'll be happy to help12:33
EmilienMso we don't have Keystone_user[admin]12:33
EmilienMand everything related to roles & tenant fails, obviousl12:34
chemEmilienM: it's in the stderr (they are unfortunatly splitted)12:34
EmilienMobviously*12:34
EmilienMchem: Keystone_user?12:34
chemEmilienM: Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Command: 'openstack ["token", "issue", ["--format", "value"]]' has been running for more then 20 seconds!12:34
EmilienMok so Keysne_user[admin] is timeouting and it causes serious damage in the catalog12:35
chemEmilienM: I really think it's only a timeout on a very buzy undersized vm12:35
EmilienMbut keystone is running because we create endpoints & services12:35
chemEmilienM: yeap, everything that depends on user admin is skipped12:35
EmilienMchem: why wouldn't we have the timeout issue on services/endpoints?12:35
EmilienMmaybe the admin creation operation takes longer12:35
chemEmilienM: because the load decreases12:35
chemEmilienM: well ... that's my theory :)12:36
EmilienMit's a good theory12:36
EmilienMdid you have something that works in puppet-keystone, so we can test it on ha job in OOO?12:36
EmilienMotherwise I have an other idea12:37
EmilienMwe can just hardcode a bigger timeout in puppet/provider/openstack.rb12:37
EmilienMand use Depends-On, for testing purpose12:37
chemEmilienM: I'm going to trigger a test with a openstackcli timeout set to 60 seconds (hardcoded) and if useful, then maybe make one where the user can config timeout12:37
EmilienMok12:38
yolandahi, i have a question about tripleo-quickstart... i see some of the logs are kept on files inside undercloud and overcloud vms. Is there some way inside the script, to make it copy to a local path, so they can be published for example to a log server? if not, is that a feature that would be accepted?12:38
EmilienMchem: I continue to read logs just a bit, to make sure it's not an orchestration issue12:38
*** cdearborn has joined #tripleo12:38
*** fzdarsky|lunch has joined #tripleo12:38
chemEmilienM: ack, meanwhile I test my theory ...12:38
shardyyolanda: Hi, I'd chat with trown and larsks about it - sounds like a useful option to me12:39
yolandaour use case is that we run it on jenkins, and i'd like to have these logs available on the slave, so i can publish to a log server12:39
trownyolanda: we have a log collection role external to tripleo-quickstart, just putting out some fires atm, but I could show you how to use it later12:39
*** links has quit IRC12:40
yolandatrown, thanks12:40
trownyolanda: but there is a collect logs script in the ci-scripts dir that shows how we use it12:40
yolandai'll look at it12:40
EmilienMchem: ok I found something12:41
EmilienMchem: let me paste it12:41
*** lblanchard1 has joined #tripleo12:42
*** bootsha has quit IRC12:43
EmilienMchem: http://paste.openstack.org/show/523773/12:43
EmilienMchem: pacemaker starts httpd AFTER Keysne_user12:43
EmilienMso it makes sense to have this timeout, keystone is not ready12:43
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: DEBUG: testing increase timeout for tripleo CI.  https://review.openstack.org/33499712:43
EmilienMwe have this: https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/keystone.pp#L5512:44
*** limao has joined #tripleo12:44
EmilienMbut I wonder if it's enough12:44
*** lblanchard has quit IRC12:44
chemEmilienM: hum ... it's in the same minute.  The socket is there as we would have a connection error in puppet provider, not a timeout.12:46
*** lblanchard has joined #tripleo12:46
EmilienMchem: and I just figured that the admin resource is created in a previous puppet run12:47
*** amoralej|lunch is now known as amoralej12:47
*** limao_ has joined #tripleo12:48
*** lblanchard1 has quit IRC12:48
chemEmilienM: yeap, in the error I've had, it was a description *modification* of the admin user, so it was there before12:48
EmilienMchem: Pacemaker::Resource::Service[$::apache::params::service_name] -> Class['::keystone::roles::admin']12:49
EmilienMso we need to include ::apache::params before?12:49
*** xinwu has quit IRC12:49
EmilienMchem: yes because 2 things try to manage the admin user: bootstrap (using keystone-manage, doing mysql operation) and then puppet provider12:50
EmilienMit's supposed to be idempotent because bootstrap is only triggered once12:50
jistri think we're dealing with 2 separate issues12:51
EmilienMjistr: yeah, rabbitmq is still down12:51
chemEmilienM: I don't think that  ::apache::params:service_name is empty there, that would fails at compilation time I think12:51
jistrthis log does fail pingtest but it succeeds when clustering rabbitmq http://logs.openstack.org/86/334486/2/check-tripleo/gate-tripleo-ci-centos-7-ha/b79b36f/12:51
jistrEmilienM: ^12:51
*** limao has quit IRC12:52
*** links has joined #tripleo12:52
EmilienMjistr: ok, so we have the keystone and the rabbit thing12:52
*** olap has quit IRC12:53
jistrthe interesting thing is that pingtest fails when trying to connect to TLS endpoint ports, but the HA and and upgrade jobs are supposed to be non-TLS12:53
*** [1]cdearborn has quit IRC12:53
EmilienMI thought everything was TLS in our CI12:53
EmilienMan upgrade job was the only ipv6 job12:54
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L10112:54
shardyEmilienM: No, only the nonha job enables tls12:54
EmilienMok cool12:54
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Adds missing line continuation and eradown option  https://review.openstack.org/33462412:54
jistri think what might have happened is that this broke the endpoint init https://review.openstack.org/#/c/330096/12:55
jistrfunny thing is that the CI is green on it12:55
jistrso should i guess further, we may have an issue that changes to python-tripleoclient aren't tested properly?12:55
EmilienMoh nice catch12:56
jistrnot sure what could collide with that patch12:56
shardyjistr: can you reproduce locally with that patch applied?12:56
jistrshardy: i don't have a master env right now at all :( working on downstream 8/912:56
*** akshai has joined #tripleo12:57
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates: Dump and restore galera db during major upgrades  https://review.openstack.org/32520512:58
EmilienMjistr: can we try to revert it?12:58
EmilienMjust to see how CI works12:58
openstackgerritJiri Stransky proposed openstack/python-tripleoclient: Revert "Run post deploy config on force"  https://review.openstack.org/33500812:58
EmilienMcool12:59
shardysure, I'm about to try it locally too12:59
*** jprovazn has quit IRC13:00
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Adds missing line continuation and teardown option  https://review.openstack.org/33462413:00
EmilienMjistr: https://review.openstack.org/#/c/334486/213:00
EmilienMeven the backport is not passing CI13:01
EmilienMhttp://logs.openstack.org/86/334486/2/check-tripleo/gate-tripleo-ci-centos-7-ha/b79b36f/console.html#_2016-06-27_17_19_54_53218313:01
EmilienMfor the same reason13:01
EmilienMso there is an high chance that this patch is the reason of the failures.13:01
EmilienMand the thing that chem and I are investigating about keystone, looks like a random issue that our CI also hit sometimes13:02
jistrshardy: cool, thanks. If there's an issue with master CI not testing tripleoclient changes for some reason, the CI on the revert will probably stay red.13:02
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: DEBUG: testing increase timeout for tripleo CI.  https://review.openstack.org/33499713:03
shardyINFO:dlrn:Processing python-tripleoclient a97dc6b1588b6cffc12ed778c787c50865290def13:04
shardythere was a dlrn issue last week wasnt there, where it built from master, not the commit from the local branch?13:04
shardythat would explain it13:04
chemEmilienM: for https://review.openstack.org/#/c/334997/  I set command timeout to 90s!13:04
EmilienMchem: ok13:04
EmilienMshardy: yes13:05
EmilienMshardy: we had similar issue last week with a patch we land13:05
shardyOk, so we tested the wrong commit - at least that explains why CI didn't catch it13:05
*** egafford has joined #tripleo13:07
openstackgerritJiri Stransky proposed openstack/python-tripleoclient: [NO MERGE] test tripleoclient CI  https://review.openstack.org/33501313:08
EmilienMshardy: that's why we had to revert the ceilometer composable services13:08
EmilienMshardy: it seems like we injected a regression13:08
*** jpena|lunch is now known as jpena13:08
*** lucas-hungry is now known as lucasagomes13:09
shardyEmilienM: ack - well that's unfortunate, but at least we know why13:09
jistrshardy: ah ok, thanks for the info13:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675813:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]13:10
sshnaidmplease, TripleO cores, look a the patch in your time, it fixes some of periodic jobs: https://review.openstack.org/#/c/334305/13:12
*** xinwu has joined #tripleo13:13
*** xinwu has quit IRC13:15
*** tzumainn has joined #tripleo13:17
sshnaidmshardy, hi13:21
EmilienMsshnaidm: right now, we're trying to bring master jobs back alive13:23
EmilienMsshnaidm: but once it's fixed, we'll probably merge your patch is it's good & passing ci13:23
sshnaidmEmilienM, sure, ok13:23
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478713:23
shardy+1 getting master jobs working is the priority atm13:24
*** saneax has quit IRC13:25
openstackgerritMatt Young proposed openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior.  https://review.openstack.org/33460013:26
*** snecklifter has quit IRC13:28
*** fzdarsky|lunch has quit IRC13:30
*** jprovazn has joined #tripleo13:32
*** bfournie1 has joined #tripleo13:35
*** bfournie has quit IRC13:35
*** oshvartz has quit IRC13:36
jistri like that zuul hands out telnet connection info for monitoring pending CI jobs nowadays. very cool13:40
*** sanjayu has joined #tripleo13:40
openstackgerritMartin André proposed openstack/tripleo-common: Allow running validation against different plans  https://review.openstack.org/31819413:40
openstackgerritMartin André proposed openstack/tripleo-common: Disable retry files for ansible validations  https://review.openstack.org/32903913:40
openstackgerritMartin André proposed openstack/tripleo-common: Validations actions and workbook  https://review.openstack.org/31363213:40
openstackgerritMartin André proposed openstack/tripleo-common: Deployment workflows for remote execution, etc.  https://review.openstack.org/31395313:40
hewbroccaUgh, so am I reading right that we think this issue may also be related to the delorean bug of last week?13:41
EmilienMjistr: yeah, did you see the work that pabelanger did to open the link directly from browser?13:41
shardyhewbrocca: yup looks that way13:41
EmilienMhewbrocca: yes, we might have failed to catch a broken patch in our CI because of the dlrn bug (fixed now)13:41
hewbroccaAin't that a kick in the head13:42
jistrEmilienM: hmm not yet :) i opened it in terminal13:42
EmilienMjistr: let me find the procedure again13:42
hewbroccaI guess there is a reasonable chance there are other bugs we might have missed this way?13:43
shardyEmilienM: FYI I'm hitting a problem related to disabling the swift backend for glance13:43
shardyeven when a backend doesn't exist, glance won't let you delete images it can no longer access13:44
shardyso I think I'll have to drop the DB and re-upload everything13:44
slaglejistr: nice tip on the telnet access13:44
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior.  https://review.openstack.org/33460013:44
slaglethat is hawt13:44
shardysomething we'll have to document if we leave that disabled13:44
EmilienMslagle: https://etherpad.openstack.org/p/google-telnet13:44
*** zoli|lunch is now known as zoli|wfh13:45
*** zoli|wfh is now known as zoliXXL13:45
EmilienMslagle: so you can open telnet directly from chrome13:45
EmilienMslagle: https://etherpad.openstack.org/p/firefox-telnet if you use firefox13:45
weshayhitting issues w/ newton master and bare metal https://bugs.launchpad.net/tripleo/+bug/159489713:47
openstackLaunchpad bug 1594897 in tripleo "Keystone authentication dies during introspection or deploy on baremetal hardware" [Undecided,New]13:47
*** r-mibu has quit IRC13:48
*** r-mibu has joined #tripleo13:48
*** pradk has joined #tripleo13:50
EmilienMtripleo meeting in 10 min, https://wiki.openstack.org/wiki/Meetings/TripleO -- please add One-off items if needed13:50
*** ayoung has joined #tripleo13:52
*** rhallisey has quit IRC13:53
*** rhallisey has joined #tripleo13:54
sshnaidmEmilienM, can we have etherpad for one-off items? The wiki doesn't accept new users anymore, it's impossible to add something there..13:54
*** ebalduf has joined #tripleo13:54
EmilienMsshnaidm: sure, feel free to create one13:55
shardysshnaidm: yes we could create an etherpad and link it from the wiki13:55
sshnaidmEmilienM, shardy how about this? https://etherpad.openstack.org/p/tripleo-meeting-items13:55
EmilienMthat's good (in puppet project, we have one etherpad per meeting, so we can easily keep track)13:56
EmilienMsee http://docs.openstack.org/developer/puppet-openstack-guide/meetings.html13:56
shardyI think for now one etherpad should be OK, we can just add the latest meeting at the top and keep history from old meetings in the etherpad13:57
shardywe generally have quite few one-off items13:58
*** bfournie has joined #tripleo13:58
*** bfournie1 has quit IRC13:58
ccamachoHey pradk, about the gnocchi submission. The db sync was executed before the service was configured causing the deployment issue, also several nits in the tht manifest, among those, issues like service_manage -> manage_service. All fixed, just finishing the HA deployment and I'll push the last change.13:59
thrash /join #openstack-meeting-alt14:04
thrashlol14:04
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675814:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]14:10
*** tobias_fiberdata has joined #tripleo14:12
*** tosc_fiberdata has quit IRC14:15
*** rook_ has quit IRC14:18
*** rook_ has joined #tripleo14:20
*** rook_ has quit IRC14:21
*** limao_ has quit IRC14:21
*** limao has joined #tripleo14:21
*** rajinir has joined #tripleo14:25
*** itamarl has quit IRC14:26
*** rhallisey has quit IRC14:27
*** rhallisey has joined #tripleo14:28
*** limao has quit IRC14:29
*** rook_ has joined #tripleo14:30
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478714:31
*** noslzzp has joined #tripleo14:35
beaglesshardy: it's actually pretty decent14:37
*** tremble has quit IRC14:37
EmilienMjistr: I'm telneting the revert and it fails on ha job14:38
*** links has quit IRC14:38
*** rook_ has quit IRC14:39
*** Larion has joined #tripleo14:40
*** rook_ has joined #tripleo14:41
*** saneax_AFK has joined #tripleo14:42
*** saneax_AFK is now known as saneax14:42
jristbnemec: is this available for general consumption? http://ucw-bnemec.rhcloud.com/14:42
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles  https://review.openstack.org/31552714:43
*** tobias_fiberdata has quit IRC14:43
*** aufi has joined #tripleo14:44
derekhtrown:  while I think of it, we're going to need to add a DNS entry for the IP you use when downloading the undercloud images for quickstack14:44
derekhtrown: that IP will be vanishing next week for a while14:44
jistrEmilienM: yea it failed even before it got to pingtest... i wonder how the upgrades job will go14:45
bnemecjrist: Yes, although I would like to find a more permanent home for it than my personal OpenShift account.14:46
trownderekh: k but the mirror server will still be there with a different IP?14:46
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles  https://review.openstack.org/31841314:47
EmilienMccamacho: what is http://mosquito-ccamacho.rhcloud.com/_plugin/tripleo-nit-finder/ ?14:47
EmilienMwhat does it do? where is source code?14:47
derekhtrown: I can move the images onto rh2 to host them, while rh1 is down14:47
*** krotscheck_dcm is now known as krotscheck14:48
*** mgould is now known as mgould|brb14:49
ccamachoEmilienM Hey is a plugin for elastic search to search in a nit database stored https://github.com/ccamacho/tripleo-nit-db is just a prototype14:50
d0ugaljpeeler, rbrady: re: 329163, wont the cache only be invalidated when the mistral API is restarted?14:51
jpeelerd0ugal: is there a invalidation comment somewhere else i'm missing? would it matter if the cache is lost when restarting?14:53
d0ugaljpeeler, rbrady: sorry, got distracted by a phone call.14:55
hewbroccaIt was Nicola Sturgeon calling14:55
d0ugaljpeeler, rbrady: I am more concerned that the clients will stop working after 4 hours when the token times out14:55
jpeeleroh i see...14:56
d0ugaljpeeler, rbrady: sorry, not times out, but expires.14:56
d0ugalhewbrocca: aye, we will sort things out yet.14:56
d0ugaljpeeler: I think 4 hours is the default timeout.14:56
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud node provide' command  https://review.openstack.org/33441114:56
jpeeleryeah that's a fair concern14:56
rbradyjpeeler, dmatthews: this is where we should extract the same code from heat that deals with this exact issue14:57
sanjayuto a noob, what is oooq?14:57
hewbroccaLOL14:57
d0ugalrbrady: aha, I didn't know heat done this. That sounds useful :)14:58
rookshadower working on my first set of validation conversions, however I am concerned with the tripleo-ansible-invetory -- it seem pretty limited.14:58
rookshadower, the script works great when you are on the undercloud, but outside of that, no dice.14:58
rookshadower: maybe it is my setup that is causing this issue?14:58
weshayEmilienM, so I'm trying to think of another service we're bringing up15:00
EmilienMweshay: do you mind continuing15:00
weshaywe're only walking the docs w/ quickstart15:00
EmilienMjust to make sure I understand15:00
EmilienMweshay: wait, I have puppet meeting on IRC also now15:00
weshayleifmadsen, rlandy you guys should chat re: https://github.com/redhat-openstack/?utf8=%E2%9C%93&query=baremetal15:01
EmilienMwe have no topic, I'll cancel it15:01
weshaylol15:01
*** fzdarsky has joined #tripleo15:01
trownI think there also needs to be a distinction between tripleo-quickstart and the external roles folks are developing that use the base it sets up15:01
leifmadsenweshay: thanks, checking it out15:01
rookweshay leifmadsen can fill a room with hot air, watch out15:01
leifmadsenbasically, I just have a use case internally for oooq to deploy onto baremetal15:01
rook:)15:01
weshaylol15:01
leifmadsenrook: I will cut you15:01
leifmadsen<315:02
weshayleifmadsen, k.. rlandy is deploying internal jobs w/ it now..15:02
rookback atcha15:02
*** bfournie1 has joined #tripleo15:02
*** bfournie has quit IRC15:02
leifmadsenso this is independent work of oooq?15:03
*** noslzzp has quit IRC15:03
trownleifmadsen: ya15:03
weshayit's outside of the openstack/tripleo-quickstart repo yes.. but it uses quickstart15:03
leifmadsenahhh ok15:03
leifmadsenthat's fine15:03
trownleifmadsen: tripleo-quickstart has been constructed to allow plugging in external ansible roles for added functionality15:03
*** noslzzp has joined #tripleo15:03
leifmadsenahhhh ok that makes sense15:03
weshayrlandy, can you point leifmadsen at one of your internal jobs on an internal channel15:03
*** [1]cdearborn has joined #tripleo15:04
weshayleifmadsen, roles are dropped into the role path w/ python setuptools15:04
leifmadsenrlandy: you can use #nfvpe internal15:04
EmilienMweshay: ok back (no puppet meeting this week)15:04
trownthere has been a bit of an explosion of said roles as more people try to use it for their particular use case15:04
weshayEmilienM, elloo15:04
myoungleifmadsen: additional roles via pip install / egg style mechanism(s)15:04
leifmadsenI guess that means oooq is useful / extensible? :)15:04
leifmadsen+115:04
trownleifmadsen: that is how I would interpret it :)15:04
EmilienMweshay: so I know you don't control IPA role, but I was wondering if oooq would be a new cool place where I can put new roles that can't live on instack-undercloud15:04
openstackgerritMerged openstack/diskimage-builder: Convert element_dependencies to logging  https://review.openstack.org/32807215:04
EmilienMand if yes ^ I don't think it's the right thing, we should rather investigate making instack undercloud multi node, so we have consistency in the way we deploy services (puppet, etc)15:05
weshayEmilienM, I suppose yes.. I don't want offend anyone w/ roles, but one could do that15:05
*** bfournie1 has quit IRC15:05
*** bfournie has joined #tripleo15:05
weshayEmilienM, certainly we could encourage people to do that if that would help15:05
weshaywe could create role policy, what is appropriate, what is not15:05
weshay etc15:05
rlandyleifmadsen: sending links15:06
leifmadsenthx15:06
weshayEmilienM, I think that is what you are getting at right?15:06
trownEmilienM: I think that is a bit putting the cart before the horse in the IPA case though. ie you tell ayoung he has to rework that stuff yet again :P15:06
ayoungtrown, EmilienM I'm flexible15:06
ayoungso long as we have a clear path, I want to do what makes sense long term15:07
*** bfournie has quit IRC15:07
trownEmilienM: for the developer use case, it makes sense if someone creates a role that sets up the extra pieces of their developer environment they need to test against15:07
rbradyjpeeler, dmatthews: this is the code I'm talking about wrt to heat.  I think this is something that needs to be in mistral instead of tripleo-common.  https://github.com/openstack/heat/blob/master/heat/engine/clients/client_plugin.py#L140-L16815:07
EmilienMno I don't tell anyone to rework anything15:07
*** bfournie has joined #tripleo15:07
EmilienMI'm trying to understand the scope of oooq15:07
ayoungEmilienM, it sounds like they want the roles external to OOOqs.  I think I can live with that. The roles I was building were going into Rippowam origianlly15:07
*** fzdarsky has quit IRC15:07
trownEmilienM: that doesn't mean it needs to go in-tree and muddy the basic case, but it allows them to develop their features for tripleo15:08
slagletrown: hey, so how i personally envinsion a potentialy switch to oooq one day in tripleo-ci would be start incrementally, such as maybe replacing tripleo.sh --repo-setup with the oooq equivalent (if there is one)15:08
EmilienMto me, it's to easily setup a tripleo dev env15:08
slagletrown: or tripleo.sh --undercloud15:08
ayoungSo I could see packing up our roles in an egg the same way that the tripleo-inventory role is done, and having quickstart call it15:08
EmilienMbut setting up IPA server is IMHO out of scope15:08
ayoungjmiu is actually looking in to that right now15:08
slagletrown: i dont know if that is in line with other's ideas or not15:08
EmilienMit's like if ayoung would patch tripleo-ci to deploy IPA server15:08
EmilienMit does not make sense ^15:08
EmilienMI would rather use instack-undercloud or THT for that15:08
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478715:09
trownslagle: ya repo setup is not a great example because oooq is image based15:09
EmilienMjistr: Stack overcloud UPDATE_FAILED on the revert15:09
EmilienMjistr: damn15:09
jpeelerrbrady: well, if the code needs additional work AND needs to be in mistral, i can just abandon the review15:09
EmilienMwe might have another issue15:09
trownslagle: but using the undercloud role via tripleo.sh would be possible15:09
ayoungthe IPA server is an optional component.   I was using the undercloud image to start, and installing IPA on top of that15:09
EmilienMjistr: /me analyzing logs now http://logs.openstack.org/08/335008/1/check-tripleo/gate-tripleo-ci-centos-7-ha/5248ea6/15:09
EmilienMayoung: that's my whole point15:09
EmilienMayoung: why not using instack-undercloud then?15:09
EmilienMjistr, chem: I'm hitting the keystone timeout issue :-15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675815:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]15:10
leifmadsenrlandy: where should I be expecting said links?15:10
ayoungEmilienM, does that install the IPA server on the undercloud?  I was just reusing the image15:10
*** sanjayu has quit IRC15:10
EmilienMhttp://logs.openstack.org/08/335008/1/check-tripleo/gate-tripleo-ci-centos-7-ha/5248ea6/logs/postci.txt.gz#_2016-06-28_14_32_52_00015:10
rbradyjpeeler: while it should be in mistral, the answer to the questions of where and when are probably in the result of the changes coming from the custom-actions-api we will be working on.15:10
shardyEmilienM: IMO we should not be using instack-undercloud to deploy additional custom things15:10
slagletrown: or we could even replace the guts of tripleo.sh --undercloud with a call to oooq to do the undercloud role15:10
shardyEmilienM: That's the use-case we expect to solve via custom roles in t-h-t15:10
EmilienMshardy: why that?15:10
slagletrown: then eventually kill all of tripleo.sh15:10
ayoungbut it is in a separate VM.  Swift, Keystone and IPA compete for Web resources, not worth trying to deconflict them15:10
slagleif that's the direction we end up15:10
EmilienMshardy: 100% agree15:10
trownslagle: right that makes sense15:10
EmilienMshardy: I could have not said better15:11
EmilienMshardy: but it seems sometimes some services are required before deploying overcloud15:11
EmilienMshardy: maybe the case of IPA server15:11
shardyEmilienM: That's fine, custom roles will include a way to define dependencies15:11
ayoungEmilienM, I opened this:  https://github.com/redhat-openstack/ansible-role-tripleo-inventory/issues/115:11
socialmarios: btw I have pupet-ironic change that conflicts with instack-undercloud puppet, can I just drop the stuff from instack undercloud or make some by-pass change?15:11
rlandyleifmadsen: just pinged you on internal15:11
shardyso you can say e.g deploy IPAController role before Controller and everything else15:11
leifmadsen+115:11
trownslagle: we could add the repo setup parts too, though I think I would take bnemec's python tool over what is in tripleo.sh for that15:12
chemEmilienM: jistr: well, right this https://review.openstack.org/#/c/334997/ has not landed yet, but I'm kind of confident that increase timeout is the solution15:12
ayounglive this http://docs.ansible.com/ansible/playbooks_roles.html#role-dependencies?15:12
*** noslzzp has quit IRC15:12
mariossocial: comment on the review would be easier (also, on scrum, sorry for slow repsonse)15:12
trownslagle: but right now, repo setup is done at image creation, so tripleo-quickstart doesnt do any of that15:13
mariossocial: easier for discussino i mean15:13
*** jistr is now known as jistr|mtg15:13
chemEmilienM: if it's green I will change my review to hardcore 60 s command timeout, I don't have time currently to make a nice interface in all consumers of puppet-openstackcli (massive change)15:13
slagletrown: ok15:13
EmilienMchem: it's failing HA job15:13
shardyayoung: no, like this: https://review.openstack.org/#/c/315679/6/overcloud.yaml15:13
slagletrown: maybe we could align that somehow15:13
*** milan has quit IRC15:13
EmilienMchem: you need to depends-on the revert patch ;-)15:13
slagletrown: make all the things use dlrn-repo15:14
shardyayoung: you would have a new item added to RoleData that enables building an extra node (or cluster) running whatever you want15:14
EmilienMchem: try again with https://review.openstack.org/#/c/335008/15:14
EmilienMchem: depends-on ^15:14
trownslagle: I would like to complete the blueprint for importing tripleo-quickstart that only has a few fixes needed for the developer workflow, and then posting some docs patches for virt environment in tripleo-docs15:14
shardyit's not about ansible as tripleo doesn't deploy and manage nodes via ansible15:14
*** fzdarsky has joined #tripleo15:14
ayoungshardy, ah we were talking quickstar before15:14
trownslagle: then I could create a new blueprint for replacing tripleo.sh?15:14
trownslagle: +1 to having image building use dlrn-repo15:15
d0ugalrbrady: cool, that code looks good. Linked to it from the review.15:15
ayoungshardy, I've been focused on getting IPA to run in the undercloud.  I see it as an external service15:15
shardyayoung: Yeah, the argument is you shouldn't be using quickstart to deploy anything the deployed overcloud depends on15:15
slagletrown: yea, probably worth a ML thread about tripleo.sh future, etc15:15
shardyayoung: that's fine, but it's something completely outside of tripleo15:15
trownslagle: k, will start with ML15:15
slaglesee if there is agreement on a path forward15:15
ayoungshardy, so, what Tripleo needs is a CA first off15:15
chemEmilienM: can you share you zuulfu so that I can see the log, I'm afraid the depends-on was not correct (I had to change it some time after).  Will add the revert in the mix as well.15:15
rbradyjpeeler, dmatthews: right now we use the mistral.context to get access to an auth_token.  that codes is here: https://github.com/openstack/mistral/blob/master/mistral/context.py15:15
ayoungLDAP is a standard integratin, and so Tripleo needs a way to test-setup LDAP integration15:16
slagletrown: i think tripleo-docs is part of the discussion as well15:16
ayoungits why quickstart..this was starting as a QE/integration effort15:16
EmilienMchem: zuul.openstack.org15:16
chemhehe15:16
slagletrown: somehow get that aligned with what everyone actually uses15:16
ayoungfor the CA...I'm starting to thing the upstream answer is Anchor15:16
shardyayoung: Ok, cool, if it's only for CI then perhaps it makes sense15:16
ayoungwe do anchor as a baseline, FreeIPA is an upgrade15:16
EmilienMchem: and https://etherpad.openstack.org/p/google-telnet if you want to see zuul logs in one click or https://etherpad.openstack.org/p/firefox-telnet15:16
shardyayoung: the concern, I think, is that folks might abuse that method to "integrate" deployment of additional custom roles15:16
chemEmilienM: O_o ... amazed15:17
trownslagle: ya, right now tripleo-docs virt instructions are a mess though, so any patches there that describe something that actually works would be an improvement15:17
chemEmilienM: thanks15:17
weshayhrybacki, ^15:17
ayoungshardy, right now I would settle for a positive thread test of geting IPA and Keycloak integrated with Tripleo15:17
trownslagle: and replacing virt section with tripleo-quickstart was already aproved with the spec15:17
d0ugalrbrady, jpeeler: if we want to improve this in mistral, I would be happy with a naieve cache for now. Even if we only cache for 30 mins or an hour, we are safe and it will be much better than creating clients on each execution.15:17
*** aufi has quit IRC15:17
*** cdearborn has quit IRC15:17
ayoungwe need to sort the TLS/CA Cert story15:17
slagletrown: yea, no issue from me with completing the initial plan around replacing virt-setup15:17
ayoungSelf signed does not work for a multi-node environment15:18
hrybackiweshay: beginning travels15:18
shardyayoung: sure, but that's a broader issue vs using quickstart beyond it's original remit, which was to bootstrap virt tripleo environments15:18
jpeelerd0ugal: so you mean, simply force refresh at a threshold way under the expiration time?15:18
weshayhrybacki, roger.. take care15:18
hrybackiweshay++ will keep an eye on the channel and respond when I can15:18
d0ugaljpeeler: Yeah, I think that would be a reasonable compromise.15:18
d0ugaljpeeler: ... but if it is easy to land a fix in mistral that can help us, we should do that. My guess is that it isn't :)15:19
ayoungshardy, so, if an organziation has an IPA server,they should be able to use it.  If they don't they need a CA for tripleo.  I was going to start by using Quickstart to setup the IPA server to support that first use case15:19
*** hrybacki is now known as hrybacki|afk15:19
ayoungWe can't wait until overcloud deploy as the undercloud would lack a CA15:19
rbradyd0ugal: the cache here would be scoped with the lifetime of the action instance, right?15:20
ayoungI'd love to stick the IPA server on the undercloud, but the only way I can dothat right now is via containers, and we don't do containers15:20
*** mgould|brb is now known as mgould15:20
shardyayoung: what happens if the undercloud goes down when the overcloud is deployed?15:20
*** jtomasek_ has quit IRC15:20
ayoungshardy, for TLS it is not too catastrophic15:20
shardyayoung: we explicitly avoid any hard dependencies between the overcloud and the undercloud, because it's a single point of failure15:21
d0ugalrbrady: Yeah, it will. I missed that. Is a new action instance created for each execution?15:21
ayoungCerts tend to be long lived, and so you don't need to fetch the CA cert or issue new certs on a daily bais15:21
ayoungbasis15:21
shardyayoung: ack, Ok maybe that will work then15:21
shardyayoung: as EmilienM said earlier tho, if it's integrating a new service that will run on the undercloud, the proper integration point is instack-undercloud not tripleo-quickstart15:22
ayoungshardy, so...I think that for Upstream, we look to using Anchor.  And for the cases where, for whatever reason, IPA is too heavy weight15:22
d0ugalrbrady: even if it is, we may hit the issue eventually. We had this problem in the CLI as some deploys took more than 4 hours.15:22
*** myoung is now known as myoung|brb15:22
ayoungAnchor has the short-term cert approach, so it shouldn';t bee allowed to go down for too long15:22
ayoungbut for a "better self signed" I think it is appropriate15:22
ayoungI build one in a COPR15:22
ayounghttps://copr.fedorainfracloud.org/coprs/admiyo/anchor/15:23
ayoungand some HP folks were working on a certmonger plugin15:23
ayounghttps://copr.fedorainfracloud.org/coprs/admiyo/anchor-certmonger-helper/15:23
*** mcornea has quit IRC15:23
ayoungthe goal here is to say "certmonger is our certificate management interface" with Anchor be a placeholder for an organizations own CA processes15:24
ayoungAnd IPA being the "we need something real and don't have one yet" solution15:24
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: DEBUG: testing increase timeout for tripleo CI.  https://review.openstack.org/33499715:24
EmilienMshardy: that was my point, thx for confirming15:24
trownshardy: EmilienM, for bootstrapping said service, what is the difference between a personal ansible role repo, and a personal fork of instack-undercloud?15:25
trownso long as that work is not going into quickstart proper, it feels the same to me15:26
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478715:26
shardytrown: the main difference is it'll have to be rewritten if we ever want to land actual support for it in TripleO15:26
EmilienMtrown: neither of both solutions look clean to me15:26
shardybecause we can't configure the undercloud services with both puppet and ansible15:26
EmilienMif we want the feature in the product, let's make it usable & testable by everyone15:26
shardyright now all undercloud services are configured via puppet, so IMO it doesn't make sense to hack extra stuff in via ansible, unless it's temporary dev only stuff15:27
rbradydmatthews, jpeeler: I think this is where the action lifecycle is: https://github.com/openstack/mistral/blob/master/mistral/engine/default_executor.py#L63-L10615:27
trownshardy: right 100% agree, I just saw it as temporary dev only stuff15:28
shardytrown: Ok, I thought there was discussion around using it in CI etc15:28
EmilienMshardy: right15:28
sshnaidmshardy, can I be added to https://launchpad.net/~tripleo-drivers please ?15:28
rbradydmatthews, jpeeler: If there is an error or action is synchronous, it returns immediately after action runs.  No idea on how long action can run for.  looking for timeout15:28
EmilienMas long as we don't teach people to use OOOQ as a place to put stuffs we could do in THT15:29
EmilienMthat was my whole point ^15:29
shardysshnaidm: yes, what's your launchpad ID?15:29
sshnaidmshardy, sshnaidm15:29
trownEmilienM: I dont think that has happened, at least not in the roles I am aware of15:29
trownroles being the ansible term "roles"15:29
shardysshnaidm: hehe, I should have tried that :)15:29
shardydone15:29
EmilienMtrown: excellent then15:29
sshnaidmshardy, thanks :)15:29
trownEmilienM: I do have to admit to not understanding the IPA work at all though :P15:30
trownI did not understand it needed anything on the undercloud for instance15:31
jokke_me is really happy that this OOOQ discussion happened as that explained the link between ansible roles vs puppet modules vs. tripleo roles15:31
trownI thought it just needed some external servers setup (that would exist as real servers in a production environment015:31
jokke_s/me/\/me/ ;)15:31
trownjokke_: ya, the overloading of the term roles is pretty confusing15:32
trownroles are just the building blocks in ansible equivalent to modules in puppet15:32
*** jistr|mtg is now known as jistr15:32
jokke_trown: specially when I spent past year heavily involved in ansible OpenStack deployer before joining RH and tripleO saga15:32
jistrEmilienM: i rechecked the tripleoclient revert. Neither of the jobs got far enough to actually show if the revert helps or not.15:33
trownjokke_: ah, so it is the reverse that is confusing for you (tripleo use of term roles)15:33
EmilienMjistr: I think it's related to the issue we have with keystone (chem has a patch for it)15:34
jokke_trown: yeah ... so this discussion opened the big confusion I had in the tripleO side ;)15:34
EmilienMwe're currently hitting 2 issues: the tripleoclient and the keystone timeout thing15:34
*** ebarrera has quit IRC15:34
EmilienMtrown: no problem :)15:34
jistrEmilienM: i see, reading what chem posted earlier15:34
jistrthis https://review.openstack.org/#/c/334996/15:34
jistrand the test patch on top. good idea15:34
*** panda has quit IRC15:34
jistrchem++15:34
*** panda has joined #tripleo15:35
trownEmilienM: what is the keystone timeout? is that the same that RDO hits on the slower nodes in centos ci?15:35
chemtrown: EmilienM I think that's the same, yes15:36
EmilienMtrown: yes15:36
trownchem: so we see that in tripleo-ci?15:36
chemtrown: yeap15:36
trownthat is purely resource based, ie I cant reproduce it locally, and it does not happen on the fast nodes in centos ci15:36
chemtrown: we're testing this https://review.openstack.org/#/c/334996/ in https://review.openstack.org/#/c/334997/15:37
*** dsariel has quit IRC15:37
trownso I guess we have once again pushed the limits of tripleo-ci in terms of how many envs we can host per virthost15:37
chemtrown: making the timeout option available for puppet provider is kind of a big change, and I may tackle ... at a later time :) https://review.openstack.org/33401115:38
trownchem: dont we need to bump request_timeout too? it seems like it is command_timeout x retries?15:39
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tests for building *-minimal images  https://review.openstack.org/18116215:39
chemtrown: nope.  request_timeout x retries15:39
trownoh that is retry_sleep anyways not retries15:40
chemtrown: If my memory is right ...15:40
trownjust saw 20,60,3 and seemed like [1] = [0]x[2]15:41
EmilienMchem: maybe we can just increase default to 90 and make it configurable in a next iteration15:41
EmilienMwe might want to bring our CI back asap15:42
chemtrown: command_timeout if for "openstack" command and request timeout is for "request" command, here, each time we had a timeout it was for the openstack command, not the request one15:42
chemEmilienM: that's my thinking as well15:42
*** myoung|brb is now known as myoung15:42
*** dmk0202 has quit IRC15:42
chemEmilienM: I've restarted the test adding the depends-on you pointed to me15:42
trownoh, but actually on line 33 it says "retry the command until the request_timeout," so I think that means we will never retry if command_timeout > request_timeout15:42
trownnot sure if we need retries in that case or not15:43
EmilienMchem: ok if https://review.openstack.org/#/c/334997/ pass CI, I propose we merge both patches in Depends-On15:44
chemEmilienM: ack15:44
EmilienMchem: and in next iteration, you finish your patch on timeout option in puppet-keystone15:44
EmilienMchem: and we figure pradk's patch again15:44
EmilienMchem: thx15:44
chemEmilienM: looks like a good plan.... let's hope it's passing ci as expected now :)15:44
EmilienMyeah15:44
*** rhallisey has quit IRC15:45
chemEmilienM: that's so cool : nc 66.187.229.91 808815:45
EmilienMchem: I clock on zuul.o.o :)15:46
EmilienMchem: with the script pabelanger wrote, you can directly open a terminal from your browser15:46
*** rhallisey has joined #tripleo15:46
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: check the overcloud_deployment_result variable to determine playbook exit  https://review.openstack.org/33399715:46
*** zoliXXL is now known as zoli|gone15:46
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463915:47
*** zoli|gone is now known as zoli_gone-proxy15:47
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478715:53
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior.  https://review.openstack.org/33460015:55
openstackgerritMerged openstack/tripleo-quickstart: Fix gate failures due to change in provisioning behavior.  https://review.openstack.org/33460015:56
*** tesseract- has quit IRC15:56
*** abehl has quit IRC15:57
*** ifarkas has quit IRC15:58
pradkjistr, i think ssl_port might be the cause.. I do the pop before initialize.. perhaps i should do it after16:06
pradkjistr, i'll try a test patch and see16:06
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675816:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]16:10
chemtrown: you're right the timeout exclude any retry.  That's bad, I'm updating the review with saner default.16:11
*** fzdarsky has quit IRC16:15
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient: Fix ssl port removal to happen after keystone init  https://review.openstack.org/33511516:16
pradkjistr, EmilienM, ^^ lets see what this does16:16
pradkwe might not have to revert the whole thing if this works16:17
*** fzdarsky has joined #tripleo16:19
*** sshnaidm has quit IRC16:22
chemtrown: I've adjusted {comand/request}_timeout to have 8 retries within 3 mins, instead of 3 retries within 66s16:22
trownchem: sweet, I guess if it takes more than three minutes to create a tenant in keystone that is a legit problem16:23
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add support for remote element repos  https://review.openstack.org/33139216:23
openstackgerritgreghaynes proposed openstack/diskimage-builder: Move hook generation in to python  https://review.openstack.org/27113916:23
openstackgerritgreghaynes proposed openstack/diskimage-builder: Move hook generation in to python  https://review.openstack.org/27113916:26
chemtrown: hehe.  Actually, it's can be any command really, that comes after all pacemaker resource creation.  I had it with 'openstack ["token", "issue" ....'16:27
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add support for remote element repos  https://review.openstack.org/33139216:28
* EmilienM afk lunch16:28
*** osp has quit IRC16:31
*** derekh has quit IRC16:31
*** jpich has quit IRC16:31
*** Jabadia has joined #tripleo16:36
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Enable IPv4/IPv6 dual-stack Public API endpoints  https://review.openstack.org/28927916:37
rookshadower: happen to be around?16:44
*** fzdarsky has quit IRC16:47
*** noslzzp has joined #tripleo16:49
*** noslzzp has quit IRC16:49
*** noslzzp has joined #tripleo16:49
*** pcaruana has quit IRC16:49
*** yamahata has quit IRC16:54
*** yamahata has joined #tripleo16:55
*** rasca has quit IRC16:55
*** noslzzp has quit IRC16:56
EmilienMchem: you sent new patchsets to puppet-openstacklib, so we need to retrigger the tht patch16:57
EmilienM334997 is not in zuul anymore16:57
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: DEBUG: testing increase timeout for tripleo CI.  https://review.openstack.org/33499716:58
EmilienMdoing it ^16:58
EmilienMpradk: can you depends on https://review.openstack.org/#/c/334996/4 ?16:59
EmilienMpradk: we're tracking down another CI issue16:59
EmilienMso you might need this one to have green jobs16:59
pradkEmilienM, sure should i wait for this to be green ?17:00
EmilienMpradk: no it's ok17:00
EmilienMpradk: go ahead and push with dependson17:00
pradkok17:00
*** yamahata has quit IRC17:00
EmilienMwe'll have early feedback17:01
mariosshardy: sorry about earlier in meeting have been on the phone for far too long today :( and had an overlap at halfway through the upstream meet. wrt https://blueprints.launchpad.net/tripleo/+spec/overcloud-upgrades-workflow-mitaka-to-newton it is actually in progress already, will update it tomorrow (there is a bunch of pointers to bugs+reviews on the bp )... i just moved to good progress... no blockers at17:01
mariosm, am currenlty investigating https://bugs.launchpad.net/tripleo/+bug/159695017:01
openstackLaunchpad bug 1596950 in tripleo "undercloud upgrade from stable/mitaka to latest hangs - possibly for swift services yet tbd" [Medium,Triaged] - Assigned to Marios Andreou (marios-b)17:01
shardymarios: ack, thanks for the update!17:01
mariosshardy: lol, i should hit return more often :)17:01
*** noslzzp has joined #tripleo17:02
* marios eod17:02
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient: Fix ssl port removal to happen after keystone init  https://review.openstack.org/33511517:02
pradkEmilienM, done17:02
EmilienMthx17:03
*** noslzzp has quit IRC17:03
*** noslzzp has joined #tripleo17:04
*** tosky has quit IRC17:06
*** tosky has joined #tripleo17:07
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675817:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]17:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: check the overcloud_deployment_result variable to determine playbook exit  https://review.openstack.org/33399717:11
*** mbound has quit IRC17:17
openstackgerritKeith Schincke proposed openstack/puppet-tripleo: Add support for cinder::volume::rbd hiera parameters  https://review.openstack.org/33513717:18
*** cdearborn has joined #tripleo17:22
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Fixes for the update image script in dev mode  https://review.openstack.org/33514017:23
*** julim has joined #tripleo17:30
*** Jabadia has quit IRC17:31
*** [1]cdearborn has quit IRC17:37
*** d0ugal has quit IRC17:37
*** d0ugal has joined #tripleo17:37
*** d0ugal has quit IRC17:37
*** d0ugal has joined #tripleo17:37
*** tosky has quit IRC17:41
*** d0ugal has quit IRC17:41
*** d0ugal has joined #tripleo17:42
*** d0ugal has quit IRC17:42
*** d0ugal has joined #tripleo17:42
*** yamahata has joined #tripleo17:49
*** ebarrera has joined #tripleo17:49
*** trown is now known as trown|lunch17:50
*** dtantsur is now known as dtantsur|afk17:51
*** cschwede has quit IRC17:52
*** jmiu_ has quit IRC17:53
*** fragatin_ has joined #tripleo17:55
*** fragatina has quit IRC17:55
*** Goneri has joined #tripleo17:57
*** jmiu_ has joined #tripleo17:57
*** ebarrera has quit IRC17:58
*** cschwede has joined #tripleo17:59
*** xinwu has joined #tripleo18:00
*** jpena is now known as jpena|off18:02
*** electrofelix is now known as electrofelix|afk18:03
*** sambetts is now known as sambetts|afk18:06
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675818:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]18:10
*** mbound has joined #tripleo18:18
*** ebarrera has joined #tripleo18:22
*** mbound has quit IRC18:24
*** pcaruana has joined #tripleo18:25
*** noslzzp has quit IRC18:32
*** chem is now known as chem|off18:34
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Fixes for the update image script in dev mode  https://review.openstack.org/33514018:35
EmilienMok pingtest still fails on neutron it seems18:35
*** colonwq has quit IRC18:36
EmilienMResource CREATE failed: NotFound: resources.server1_floating_ip: External network dfc925cc-1bf4-4fa2-90c6-bb43f0e45a37 is not reachable from subnet e4a2e92c-4f04-4d59-aff0-89dde526aef9.  Therefore, cannot associate Port f1446e4a-2021-4ce8-b202-467f8ce73db |18:36
EmilienMbut the revert helped to go further18:36
pradkEmilienM, i think my fix is working.. the pingtest passes 335115 for ha.. lets wait and see18:40
EmilienMlooking18:40
EmilienMpradk: w00t18:41
*** athomas has quit IRC18:41
*** davidlenwell has quit IRC18:42
*** paramite has quit IRC18:42
*** amoralej is now known as amoralej|lunch18:42
*** ebarrera has quit IRC18:43
*** amoralej|lunch is now known as amoralej|off18:43
EmilienMbnemec, slagle: could you review this one please? https://review.openstack.org/335115 it will help to get our CI back alive18:46
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Enable IPv4/IPv6 dual-stack Public API endpoints  https://review.openstack.org/28927918:46
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478718:48
slagleEmilienM: pradk : could you add the "why/how" in the commit msg18:48
slagleit's pretty lacking on details18:48
pradkslagle, will do.. waiting on ci to pass18:49
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/33516918:49
pradkEmilienM, so i added depends on https://review.openstack.org/#/c/334996/4 but that fails ci .. should i remove my depends on and you depend yours on mine?18:50
*** davidlenwell has joined #tripleo18:52
*** ChanServ sets mode: +v davidlenwell18:52
pradkoops i meant https://review.openstack.org/#/c/334997/ ..18:52
EmilienMpradk: don't do anything now18:54
openstackgerritTim Rozet proposed openstack/puppet-tripleo: Adds OpenDaylight  https://review.openstack.org/33517318:54
EmilienMpradk: when CI passed, we'll update commit message and review it again18:54
EmilienMpradk: depends-on is good18:55
EmilienMpradk: the patch in puppet-openstacklib is going to land in the next minutes18:55
EmilienMso we won't have the timeout issue anymore in our CI18:55
pradkEmilienM, cool ok18:55
EmilienMwe'll revert it once we switch to pacemaker lite18:55
EmilienMpradk: ha job passed now waiting on upgrade job18:55
pradkyep18:56
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/33238318:56
EmilienMpradk: it smells good18:56
*** trown|lunch is now known as trown18:56
*** bootsha has joined #tripleo19:01
*** paramite has joined #tripleo19:03
*** rook_ has quit IRC19:03
*** paramite has quit IRC19:09
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/159675819:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,Confirmed]19:10
*** rook_ has joined #tripleo19:11
EmilienMpradk: ok ping test is starting... cross finger19:12
*** jcoufal has quit IRC19:12
bnemecEmilienM: pradk: I left some comments on the change.19:13
*** florianf has quit IRC19:13
bnemecIt looks fine, but I don't think the fix is what the commit message says it is.19:13
pradkbnemec, yea i fixed the commit msg.. will push once ci passes19:14
pradkbnemec, thx for the commenst19:14
trozethi all.  Is there anyway to make os-net-config only run a single time.  I see in /var/log/messages it seems to run periodically19:15
EmilienMtripleo.sh -- Overcloud pingtest, SUCCESS19:15
EmilienMyeah19:15
EmilienMpradk: almost done19:15
bnemectrozet: No, we rely on the fact that os-net-config is supposed to be idempotent.19:16
bnemecIf it's not, then that's a bug.19:16
dsneddonbnemec, trozet: Yeah, it should be idempotent. I am aware of a bug related to mapping NICs by MAC address that isn't idempotent, because the MAC address changes when the NIC joins the bond. https://bugzilla.redhat.com/show_bug.cgi?id=134198919:17
openstackdsneddon: Error: Error getting bugzilla.redhat.com bug #1341989: NotPermitted19:17
trozetbnemec: heh no bug.  I'm trying to "trick" it.19:17
trozetbnemec: after i bind dpdk port, the kernel no longer knows that port, so nic mapping gets screwed up19:17
trozetthen it just ends up tearing apart OVS and the config haha19:17
dsneddontrozet, That sounds like a variant of the above bug, but with another trigger (MAC address changing due to DPDK instead of bonding).19:18
dsneddontrozet, Are you doing NIC mapping by MAC or by NIC number (nic1, nic2)19:18
trozetbnemec, dsneddon: could you maybe point me to the code that triggers os-net-config to run, maybe I can hack it to only run 1 time per node19:18
trozetdsneddon: nic number19:18
trozeti was trying to avoid adding the code to os-net-config to know what a dpdk port is...so i do the bind in puppet pre-config on the compute node, but then hit this19:19
*** athomas has joined #tripleo19:19
dsneddontrozet, OK. Any chance I could take a look at a system where this is happening? I want to see if there is something in /sys/class/net that we can read to get the list of NICs after DPDK is enabled.19:19
trozetdsneddon: it's a LF server so you have to have VPN access.  But there isn't anything there.  The port by its name is effectively gone when you bind to dpdk19:21
trozetdsneddon: the only way at that point to reference it is via pci address19:21
dsneddontrozet, Ugh. I'm not sure how we code around that in os-net-config, we have a similar problem if someone hot-plugs a NIC and it ends up getting names that change the NIC ordering.19:22
openstackgerritMerged openstack/tripleo-quickstart: Adds missing line continuation and teardown option  https://review.openstack.org/33462419:22
dsneddontrozet, You might be able to use the MAC mapping to make your config idempotent.19:23
trozetdsneddon: is the idea of hacking os-collect-config or whatever to only run os-net-config one time out of hte question?19:23
bnemecdsneddon: trozet: In this case, it's possible we could give os-net-config a list of ignored ports.  Presumably we don't want os-net-config messing with a port that's about to be configured by something else anyway.19:24
pradkEmilienM, success19:24
EmilienMpradk: https://review.openstack.org/#/c/335115/ GREEN19:24
EmilienMyeah19:24
EmilienMpradk: ok you can push19:24
*** akuznetsov has joined #tripleo19:24
dsneddontrozet, The problem is, the first time os-net-config is run the /etc/os-net-config/config.json doesn't exist yet19:24
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient: Fix ssl port removal logic in postconfig  https://review.openstack.org/33511519:25
bnemecYou also can't guarantee that os-net-config won't need to be run later.19:25
bnemecAnd if it does, BOOM19:25
trozetbnemec: heh well at first I tried that.  Did not list the nic in the nic template, but I need an ip address.  So I assigned an IP to OVS bridge br-phy, and the deployment failed because there is some ping check for IPs19:25
dsneddontrozet, Why don't you use NIC names instead of NIC numbers?19:26
dsneddontrozet, Although, that won't work for heterogeneous hardware19:26
trozetdsneddon: yeah i could do htat for virtual deployments19:26
trozetdsneddon: but there is still a problem I think19:26
EmilienMpradk: the commit message looks good to me19:26
EmilienMbnemec, slagle: could you revisit https://review.openstack.org/#/c/335115/ please ?19:26
pradkEmilienM, did the depends already land?19:27
pradkah looks like it did.. cool19:27
trozetdsneddon: so what I do now, declare nic2 as a member of OVS bridge br-phy.  os-net-config runs, and then verifies the ping.  Then in pre-deploy puppet I remove nic2 from the bridge, bind to dpdk, restart OVS with dpdk turned on, and add the dpdk0 port back to OVS br-phy19:27
EmilienMok CI should be in a better shape from now19:28
EmilienMI remove the alert on https://bugs.launchpad.net/tripleo/+bug/159675819:28
openstackLaunchpad bug 1596758 in tripleo "[CI down] pingtest fails when accessing to OpenStack APIs" [Critical,In progress] - Assigned to Pradeep Kilambi (pkilambi)19:28
trozetdsneddon: but hte problem is next time os-net-config runs - first it now doesnt have that nic in its nic mapping and thinks nic2 is the next nic on the system - 2nd it hten tries to move it back to the bridge19:28
*** akuznetsov has quit IRC19:28
dsneddontrozet, Hmm, it seems like we should add DPDK ports to os-net-config. I need a list of the commands that you run to add a DPDK port to the bridge. This shouldn't be too hard to make os-net-config run those commands, just like we do when we add an interface to an OVS bridge with ovs-vsctl. I can put that together.19:29
trozetdsneddon: yeah i think someone from the RCIP team filed a request or spec for it19:30
trozetdsneddon: it's this  https://github.com/trozet/opnfv-tht/blob/stable/colorado/puppet/extraconfig/pre_deploy/compute/manifests/ovs_dpdk.pp19:30
openstackgerritMerged openstack/tripleo-quickstart: check the overcloud_deployment_result variable to determine playbook exit  https://review.openstack.org/33399719:31
dsneddontrozet, Is it reasonable to run the dpdk commands from os-net-config without touching selinux enforcing mode? I'd rather that be set outside of os-net-config.19:33
trozetdsneddon: yeah I'm not sure.  I think that was just in the red hat guide I found online.  I'm not sure if it is required19:33
trozetdsneddon: the other issue is the dpdk stuff is pretty fragile.  I was told not to restart OVS after adding the port.  Sometimes it fails19:34
trozetdsneddon: but you have the right idea, there should be a dpdk type added as a member port to OVS in the nic template19:34
*** panda has quit IRC19:34
dsneddontrozet, OK, good to know. I don't think os-net-config will have to restart OVS.19:34
*** ayoung has quit IRC19:35
shadowerrook: hey, sorry, had to leave a little early19:35
shadowerrook: you around now?19:35
*** panda has joined #tripleo19:35
rookshadower: heya19:35
rookshadower , looking for feedback on : https://github.com/jtaleric/clapper/commit/34afbc9749cc028a81e1900bfc157ea915fdf527 -- not sure if it conforms to how you guys like things.19:36
*** fragatin_ has quit IRC19:36
rooki suppose I could do a pr and we discuss it there, or here...19:36
dsneddontrozet, The bridge has to be set as type: netdev, right? I'm thinking we should create a new object class in os-net-config for "dpdk_bridge" similar to the type "ovs_bridge". Make sense? We may or may not need a dpdk_interface type, interface might do.19:36
trozetdsneddon: yeah19:37
*** bootsha has quit IRC19:37
dsneddontrozet, Will it be necessary for os-net-config to configure OVS patches? If so, then this will depend on dprince's patch to add OVS patch support.19:37
trozetdsneddon: so i guess i could do a hack to os-net-config to write a file on its first invoke and then just return if that file exists ;)19:37
shadowerrook: that would be ideal. But yeah, looks good to me. I'll try to run it through our API and see how well it reports the results19:37
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient: Run post deploy config on force  https://review.openstack.org/33448619:38
shadowerrook: regarding the dynamic inventory. We do need to be able to talk to Heat and such, but there's no reason we can't update it to talk to the openstack APIs remotely19:38
dsneddontrozet, Yeah, that'd work. Or just add a simple check to os-net-config to bail if a particular file exists already in /etc/sysconfig/network-scripts19:38
trozetdsneddon: oh patch ports?  Really that should be done in neutron...but I don't think it currently does it19:38
shadowerrook: that should make it useful from outside the undercloud, shouldn't it?19:38
EmilienMthrash, pradk: tomorrow morning, meeting on composable services ok?19:38
openstackgerritMerged openstack/python-tripleoclient: Fix ssl port removal logic in postconfig  https://review.openstack.org/33511519:39
trozetdsneddon: sorry neutorn openvswitch agent I mean..19:39
rookshadower so, with the dynamic inventory tell the client to use the undercloud as a jumphot?19:39
rookhost*19:39
pradkEmilienM, sure I'll try to attend19:39
dsneddontrozet, Just wondering, since the ovs_dpdk.pp has a function to add a patch between br-phy and br-tun.19:39
thrashEmilienM: ack19:39
shadowerrook: not sure I understand19:40
trozetdsneddon: yeah openvswitch agent will create br-tun I think, but I don't think it will create br-phy and patch them19:40
*** egafford has quit IRC19:40
dsneddontrozet, You should be able to use virt-customize or similar tools to edit the os-net-config cli.py in the overcloud-full.qcow2 image and reupload it to Glance.19:40
trozetdsneddon: we already pull a custom os-net-config that has my changes so that we can hiera nic219:40
rookshadower so, from the dynamic inventory, it captures the private/pxe/provision network... a host outside of that network wouldn't have access to that network, so we would need to use the undercloud as a jump host to get on that network.19:40
trozetdsneddon: https://github.com/openstack/os-net-config/compare/master...trozet:hiera_nic_mapping19:41
*** egafford has joined #tripleo19:41
shadowerrook: aaah, I see. Yeah that would be a problem19:41
shadowerrook: so right now we're fine running the validations on the undercloud19:41
rookshadower i don't think it is impossible... but it would be nice to have that flexibility... We currently have that in our script (which isn't a dynamic inventory)...19:42
rookshadower: sure.19:42
shadowerrook: yeah I see that would be nice and I'm not against it19:42
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Fixes for the update image script in dev mode  https://review.openstack.org/33514019:42
*** [2]cdearborn has joined #tripleo19:43
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: First iteration of libvirt and nova-compute as a composable services  https://review.openstack.org/32968619:48
*** fragatina has joined #tripleo19:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: deploy composable neutron service  https://review.openstack.org/33123419:48
EmilienMok let's continue composable roles work now :)19:48
*** colonwq has joined #tripleo19:49
trozetdsneddon: looks like ovs-netdev is written to /sys/class/net/ after ovs is restarted, so i can check for that then bail os-net-config.  Thanks19:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Configure BlockStorage services via resource chains  https://review.openstack.org/32285819:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Composable Timezone service - Controller  https://review.openstack.org/33152819:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Composable Timezone service - Compute  https://review.openstack.org/33153019:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Composable Timezone service - CephStorage  https://review.openstack.org/33153419:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Composable Timezone service - ObjectStorage  https://review.openstack.org/33153619:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Wire NTP and Timezone services into BlockStorage nodes  https://review.openstack.org/33274019:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Switch Ceph Monitor/OSD/Client/External to composable roles  https://review.openstack.org/30469219:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add a method to wait for messages on the websocket  https://review.openstack.org/32261119:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal registration  https://review.openstack.org/32261219:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal introspection  https://review.openstack.org/32778019:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Don't cache and re-use the websocket  https://review.openstack.org/33198519:50
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478719:50
yolandahi, i'm hitting an issue with overcloud-validate and run tempest19:50
yolandait grabs tempest from pacakge, that, has no requirements.txt and test-requirements.txt19:50
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert SNMP configuration to composable services format  https://review.openstack.org/32059719:51
yolandabut then it fails with Could not open requirements file: [Errno 2] No existe el fichero o el directorio: '/home/stack/tempest/requirements.txt'19:51
yolandais that a known issue?19:51
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: moving out iscsi bits, now in puppet-tripleo  https://review.openstack.org/33124719:51
trownEmilienM++ thanks for all the hard work to get CI back working19:51
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Remove NFS bits from overcloud_compute manifest  https://review.openstack.org/33295519:51
EmilienMtrown: it's a team work19:52
trownyolanda: ya there is an RDO BZ about it. 2 actually, because the virtualenv with packages was a workaround for another issue19:52
yolandatrown, is there any fix for it, or what's prefered workaround?19:53
trownyolanda: https://bugzilla.redhat.com/show_bug.cgi?id=134433919:53
openstackbugzilla.redhat.com bug 1344339 in tempest "install_venv script fails to open requirements file" [Medium,New] - Assigned to dmellado19:53
yolandaah cool19:53
yolandathx, i was not sure if that was problem with my config19:54
trownyolanda: right now my workaround is not to run tempest :P, but you could add another workaround on top of the workaround and manually download requirements.txt from upstream tempest19:54
yolandaEmilienM, btw, if you hit some issue with openstack-infra... i've not been very involved recently with all changes, but ping me if you need help at anytime19:54
trownI stopped at my workaround for a workaround being broken19:54
*** akuznetsov has joined #tripleo19:54
yolandatrown, i was experimenting with that tripleo-quickstart for testing... actually i wanted to test tempest and rally19:55
yolandai will ping dmellado tomorrow, to see if we can collaborate fixing that19:55
trownyolanda: and I would accept a tripleo-quickstart patch that fixes it (s/fixes/further works around broken tempest rpm/)19:55
EmilienMyolanda: ack thx a lot19:55
yolandatrown, cool, working on that setup as a priority now, so I could help with that19:56
*** cdearborn has quit IRC19:57
*** gfidente has quit IRC19:57
trownyolanda: hmm I put +2 on https://review.openstack.org/#/c/329556 but then I thought about it... if we had a way to pass in any package we wanted to build from a refspec (or branch) wouldnt that also solve that case?20:00
trownyolanda: we are pretty close to that, which then makes that ^ patch a bit redundant20:01
yolandatrown, but that's not package, is a git repo20:02
trownyolanda: right but we have a tool to build a package from a git repo20:02
*** athomas has quit IRC20:02
yolandatrown, that would involve an extra step then...20:02
yolandafor our direct use case, i think it would be simpler to just consume the git repo20:03
yolandabut i'm quite new to that, so I'd like to hear what people thinks...20:03
trownk, I suppose it doesnt hurt to have both paths for THT20:03
trownit is a bit unique in that it can be consumed directly from git via the deploy command20:04
*** paramite has joined #tripleo20:04
*** ebarrera has joined #tripleo20:08
yolandatrown unique... but it's quite useful20:09
*** jayg is now known as jayg|g0n320:09
trownyolanda: ya, I agree20:09
openstackgerritMerged openstack/diskimage-builder: Release notes for 1.18  https://review.openstack.org/33471120:09
trownyolanda: THT is the easiest tripleo project to test because of that ability20:10
yolandayes... having to create a package is not a blocker, but an extra step20:12
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Composable midonet for neutron  https://review.openstack.org/33338720:12
*** ebalduf has quit IRC20:15
*** jprovazn has quit IRC20:15
*** bootsha has joined #tripleo20:17
*** rook_ has quit IRC20:17
trownyolanda: the package building when integrated into tripleo-quickstart will be very easy, there is already an extra role in RDO that can do it. once we have https://review.gerrithub.io/#/c/277418/ we can actually build any number of packages from open reviews for either packaging or code repos20:18
trownyolanda: but given we dont have that yet, and your patch is ready now, I am still +2 on it20:18
openstackgerritTim Rozet proposed openstack/puppet-tripleo: Adds OpenDaylight  https://review.openstack.org/33517320:19
yolandatrown, that's really amazing, and i think we may use that as well20:19
trownyolanda: that same mechanism can just replace refspec with a branch name, and even the repo used with a private fork20:20
*** jcoufal has joined #tripleo20:21
yolandatrown, indeed, one of our interests is to test new features , changes that not even landed, so thanks for that pointers20:26
yolandaanyway, leaving for today, have a nice day20:26
trownyou as well20:27
*** paramite has quit IRC20:27
dsneddontrozet, You might want to file an RFE BZ for os-net-config writing the NIC mapping to hiera. That wouldn't be too difficult to implement.20:29
*** ayoung has joined #tripleo20:29
trozetdsneddon: i was going to do it when I get time20:29
trozetdsneddon: and write hte dpdk type, but you have volunteered to do that for me ;)20:29
*** ebalduf has joined #tripleo20:29
*** jcoufal has quit IRC20:30
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478720:31
openstackgerritTim Rozet proposed openstack/puppet-tripleo: Adds OpenDaylight  https://review.openstack.org/33517320:37
*** jcoufal has joined #tripleo20:38
*** jtomasek_ has joined #tripleo20:40
*** ebarrera has quit IRC20:50
*** Goneri has quit IRC20:53
*** fragatina has quit IRC20:54
*** fragatina has joined #tripleo20:54
*** oneswig has joined #tripleo20:56
*** weshay has quit IRC21:02
*** akuznetsov has quit IRC21:02
*** lblanchard has quit IRC21:04
* bnemec is having too much fun watching CI graphs: https://fedorapeople.org/~bnemec/CIMon.png21:06
*** trown is now known as trown|outtypewww21:08
*** dmk0202 has joined #tripleo21:11
openstackgerritBen Nemec proposed openstack/tripleo-docs: Group advanced topics by type  https://review.openstack.org/33523221:12
*** akshai has quit IRC21:13
*** bootsha has quit IRC21:13
*** rhallisey has quit IRC21:13
openstackgerritBob Fournier proposed openstack/python-tripleoclient: Remove NeutronEnableTunnelling from python-tripleoclient  https://review.openstack.org/33523421:19
openstackgerritBob Fournier proposed openstack/tripleo-heat-templates: Remove NeutronEnableTunnelling from Heat templates and puppet manifests  https://review.openstack.org/33523521:20
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478721:20
openstackgerritTim Rozet proposed openstack/puppet-tripleo: Adds OpenDaylight  https://review.openstack.org/33517321:26
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Enable IPv4/IPv6 dual-stack Public API endpoints  https://review.openstack.org/28927921:28
openstackgerritBob Fournier proposed openstack/tripleo-heat-templates: Remove NeutronEnableTunnelling from THT and puppet manifests  https://review.openstack.org/33523521:36
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Fixes for dlrn gate repo injection while using devmode  https://review.openstack.org/33469121:38
*** bfournie has quit IRC21:42
*** dmk0202 has quit IRC21:42
*** akrivoka has quit IRC21:55
*** jtomasek_ has quit IRC21:56
*** akshai has joined #tripleo21:57
*** davidlenwell has quit IRC22:02
*** akshai_ has joined #tripleo22:04
*** oneswig has quit IRC22:04
*** akshai has quit IRC22:07
*** davidlenwell has joined #tripleo22:11
*** ChanServ sets mode: +v davidlenwell22:11
*** [2]cdearborn has quit IRC22:27
*** telnoratti has left #tripleo22:31
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478722:32
*** mbound_ has joined #tripleo22:33
*** akshai_ has quit IRC22:36
*** pradk has quit IRC22:53
*** yamahata has quit IRC23:02
*** rlandy has quit IRC23:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Add support for OVB based CI  https://review.openstack.org/33478723:15
*** sanjayu has joined #tripleo23:18
*** bfournie has joined #tripleo23:21
*** [2]cdearborn has joined #tripleo23:25
*** cdearborn has joined #tripleo23:31
*** mbound_ has quit IRC23:44
*** [2]cdearborn has quit IRC23:48
*** numans has joined #tripleo23:51
*** cdearborn has quit IRC23:51
*** akshai has joined #tripleo23:59
*** kbyrne has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!