Thursday, 2016-11-10

*** chlong has joined #tripleo00:23
*** ayoung has quit IRC00:24
*** kjw3 has joined #tripleo00:26
*** limao has joined #tripleo00:35
*** artom has quit IRC00:39
*** artom has joined #tripleo00:40
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles  https://review.openstack.org/33065900:45
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates: WIP use overcloud-full instead of atomic-image  https://review.openstack.org/39588000:45
openstackgerritSteve Baker proposed openstack/tripleo-common: Create new docker command hook  https://review.openstack.org/31272300:46
*** jerrygb has quit IRC00:48
*** jerrygb has joined #tripleo00:49
*** jerrygb has quit IRC00:49
*** jerrygb has joined #tripleo00:49
*** saneax is now known as saneax-_-|AFK00:53
*** afazekas_ has quit IRC01:08
*** kjw3 has quit IRC01:10
*** rajinir has quit IRC01:16
*** kjw3 has joined #tripleo01:21
*** kjw3 has quit IRC01:34
*** yamahata has quit IRC01:40
*** kjw3 has joined #tripleo01:49
ggilliesfultonj: I was able to get my composable roles working fine01:51
ggillieseverything working01:51
*** tzumainn has quit IRC02:03
*** tiswanso has joined #tripleo02:08
*** gbarros has joined #tripleo02:08
gbarrosanyone know what the hiera value for InternalApiNetworkVip is?02:08
gbarrosdsneddon ^02:08
*** fzdarsky|afk has quit IRC02:10
ggilliesgbarros: I've done hacks in the past using something like02:15
ggillies%{hiera('heat::api::bind_host')}02:15
ggillieswhich will resolve to internal api vip02:15
gbarrosty ggillies, will give t a try02:16
fultonjggillies: thanks for the update regarding your composable role. was there anything you had to do that was unexpected? Or was it simply a matter of OS::TripleO::DellCompute::Net::SoftwareConfig: nic-configs/compute.yaml ?02:17
ggilliesfultonj: with network isolation enabled there was a lot I had to add02:17
fultonjggillies: normally for network isolation I use something like this https://github.com/RHsyseng/hci/blob/master/custom-templates/network.yaml02:19
ggilliesI have http://paste.openstack.org/show/588643/02:19
ggilliesbecause I am using static ip assignment02:19
ggilliesyou will need all the ports definitions02:20
ggillieslike I have02:20
ggilliesexcept they need to point to the ones without *_from_pool.yaml02:20
ggilliesif you don't specify port definitions they will just default to ctlplane02:20
ggilliesand you will get no network isolation02:20
ggilliesit's insanely fiddly02:21
fultonjggillies: that's the trick, thanks!02:21
ggilliesI do also recommend you switch to predictable ip assignment if you can02:21
ggillieswe use it for all our environments production or otherwise02:21
ggilliessame with predicatable host assignment02:21
*** artom has quit IRC02:22
*** artom has joined #tripleo02:23
*** cdearborn has quit IRC02:24
*** mrunge has quit IRC02:24
*** mrunge has joined #tripleo02:24
*** dmacpher-afk is now known as dmacpher02:25
*** cwolferh has quit IRC02:25
*** fabbione has quit IRC02:29
*** fabbione has joined #tripleo02:29
fultonjthanks ggillies.02:33
*** fultonj has quit IRC02:33
*** Nagendra has quit IRC02:34
*** thrash is now known as thrash|g0ne02:51
*** kjw3 has quit IRC02:52
*** kjw3 has joined #tripleo02:54
*** lblanchard has quit IRC03:19
*** tiswanso has quit IRC03:25
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup yumdownloader repos  https://review.openstack.org/39592103:26
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544903:30
*** fragatin_ has joined #tripleo03:32
*** jerrygb has quit IRC03:33
*** fragatina has quit IRC03:35
*** fragatina has joined #tripleo03:36
*** fragatin_ has quit IRC03:37
*** fragatina has quit IRC03:40
*** cmyster has quit IRC03:47
*** kjw3 has quit IRC03:50
*** cmyster has joined #tripleo03:50
*** cmyster has quit IRC03:50
*** cmyster has joined #tripleo03:50
*** kjw3 has joined #tripleo03:51
*** skramaja has quit IRC04:00
*** afazekas has joined #tripleo04:04
*** skramaja has joined #tripleo04:07
*** fzdarsky|afk has joined #tripleo04:07
*** chlong has quit IRC04:13
*** coolsvap has joined #tripleo04:18
*** links has joined #tripleo04:23
*** numans has joined #tripleo04:24
*** cwolferh has joined #tripleo04:34
*** jerrygb has joined #tripleo04:34
*** jerrygb has quit IRC04:39
*** chandankumar has joined #tripleo04:42
*** dsariel has joined #tripleo04:54
*** bkreitch has quit IRC05:11
*** skramaja_ has joined #tripleo05:12
*** liverpooler has quit IRC05:36
*** jaosorior has joined #tripleo05:51
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup yumdownloader repos  https://review.openstack.org/39592106:03
*** abregman has joined #tripleo06:05
*** lmiccini has joined #tripleo06:17
*** rcernin has joined #tripleo06:18
*** jaosorior has quit IRC06:34
*** jaosorior has joined #tripleo06:35
*** jerrygb has joined #tripleo06:35
*** iranzo has joined #tripleo06:40
*** jerrygb has quit IRC06:42
*** skramaja has quit IRC06:51
*** pcaruana has joined #tripleo06:51
*** rcernin has quit IRC06:56
*** rcernin has joined #tripleo06:56
*** chlong has joined #tripleo06:58
*** oshvartz has joined #tripleo07:08
*** chlong has quit IRC07:09
*** owalsh has quit IRC07:13
*** tesseract has joined #tripleo07:20
*** tesseract is now known as Guest4021607:20
*** yamahata has joined #tripleo07:21
jaosoriorany reviews for this? https://review.openstack.org/#/c/392519/07:22
*** rasca has joined #tripleo07:25
openstackgerritSagi Shnaidman proposed openstack/instack-undercloud: Remove deprecated network range 192.0.2.0  https://review.openstack.org/39570207:30
*** florianf has joined #tripleo07:31
*** jtomasek has joined #tripleo07:32
*** ealcaniz has joined #tripleo07:37
*** rasca has quit IRC07:37
*** rasca has joined #tripleo07:37
matbuo/07:38
marios\o07:39
chemo_07:40
*** ebarrera has joined #tripleo07:44
panda|zZ_o07:45
*** panda|zZ is now known as panda07:45
rasca_o_07:46
*** openstackgerrit has quit IRC07:48
*** openstackgerrit has joined #tripleo07:48
*** jlinkes has joined #tripleo07:53
*** jprovazn has joined #tripleo07:55
openstackgerritMerged openstack/tripleo-validations: Validation stonith device exists in OpenStack Platform HA cluster  https://review.openstack.org/36010207:57
*** mcornea has joined #tripleo07:58
*** bvandenh has joined #tripleo08:00
*** rasca has quit IRC08:00
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Reset the parameter_defaults between deployments via the CLI  https://review.openstack.org/39441908:00
*** ohamada has joined #tripleo08:02
*** rasca has joined #tripleo08:02
*** ccamacho has joined #tripleo08:04
*** hewbrocca_afk is now known as hewbrocca08:05
openstackgerritMerged openstack/puppet-tripleo: Better way to ensure keepalived before haproxy.  https://review.openstack.org/39563308:08
hewbroccaMorning all08:09
d0ugalMorning!08:12
shardyg'morning all08:13
*** zoli|gone is now known as zoli|wfh08:14
*** jpena|off is now known as jpena08:21
*** b00tcat has quit IRC08:25
*** amoralej|off is now known as amoralej08:26
hewbroccaHave we got ceilometer to stop TRUMPing our CI yet?08:26
*** abehl has joined #tripleo08:27
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Introduce branch-based scenario tests  https://review.openstack.org/39600808:27
*** cylopez has joined #tripleo08:28
*** abregman has quit IRC08:29
jaosoriorshardy: could you give me your opinion on this? ^^08:29
*** abregman has joined #tripleo08:32
ccamachomorning!08:36
jaosoriorccamacho: hey dude, how's it going?08:36
ccamachojaosorior hey man! yesterday was a Holiday in Madrid :) I was going to review your patch in the morning but already landed08:38
jaosoriorno biggie :)08:38
jaosoriorccamacho: this one is still missing reviews :D https://review.openstack.org/#/c/392473/ and it's pretty simple08:38
*** jerrygb has joined #tripleo08:38
*** arxcruz|brb is now known as arxcruz08:40
pandahewbrocca: I don't think that any of the services analyzed on the last session is seriously affecting performance, unless pradk or jd__ confirm that the notify in ceilometermiddleware was causing bottlenecks. The only big thing that remains unsolved (that is until periodic jobs from this morning finish) is the memory allocation error from Cinder. It's still, happening, not very often, but still happening.08:40
pandaI'd like to ask derek what is the memory overcommitment for rh1, when he come online, as it seems happening when the test env is under high load.08:40
dsarielhi is it a way to validate tripleo heat templates before deployment that is very lengthy?08:41
*** jpich has joined #tripleo08:41
pandadsariel: you mean this ? https://github.com/openstack/tripleo-validations08:43
*** apetrich has quit IRC08:43
hewbroccapanda: that's great. So we have mostly fixed the gnocchi vs. redis death match problem?08:43
*** apetrich has joined #tripleo08:44
dsarielpanda, thanks for the reply, I've seen this and have seen also https://github.com/openstack/tripleo-heat-templates/tree/master/validation-scripts08:44
*** jerrygb has quit IRC08:44
dsarielpanda, so the question is which is better?08:44
pandashadower: ^ what you think ? :)08:44
dsarielpanda, maybe I will elaborate what exactly I need08:45
*** aufi has joined #tripleo08:45
dsarielpanda, I have a yaml that overrides default parameters like cinder backups to be ceph instead of swift. I'm specifying to overcloud depoy this yaml file with -e flag. And before actual deployment I need something that validates the overridden parameters08:47
pandahewbrocca: well, gnocchi vs redis, probably. gnocchi vs rest of the world, I don't think so. There was that patch that stopped metricd from retrying frantically for redis, but here for example metricd is retrying frantically because of other problems. This is one of the deployment that fails because of the cinder memory allocation problems.08:48
pandahttp://logs.openstack.org/79/395379/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5cdde75/logs/overcloud-controller-1/var/log/gnocchi/metricd.txt.gz08:48
hewbroccaI see08:48
hewbroccaseems like we need to pay some careful attention to gnocchi failure mode08:49
*** paramite has joined #tripleo08:52
*** paramite has quit IRC08:52
*** abregman has quit IRC08:52
*** paramite has joined #tripleo08:52
openstackgerritMerged openstack/tripleo-heat-templates: Add firewall rules for manila api service  https://review.openstack.org/39576908:53
hewbroccafailure modes08:53
shardydsariel: if you need to validate the heat templates then tripleo-validations won't help - we already do basic validation of the heat syntax when the plan is created (before the deployment starts)08:54
shardythat basically uses the heat template-valdate call under the hood08:54
shardyjaosorior: sorry, opinion on what?08:54
pandamh, that's interesting, I didn't notice it before, cinder allocation error shows on all three controllers. Not at the same time, but sequentially. It's like heat tries with each controller, and they all can't allocate memory. This is another point in favor of memory overcommitment in rh1 causing problems.08:55
*** nyechiel has joined #tripleo08:55
pandaupdating bug.08:55
jaosoriorshardy: this commit https://review.openstack.org/39600808:55
shardydsariel: if you can elaborate on what failure you're trying to validate I can possibly suggest ways to pre-validate, but some errors can only be caught at runtime (e.g when they depend on some value)08:55
*** zoli|wfh is now known as zoli|afk08:56
shardyjaosorior: wouldn't it be easier to just put the $newton_and_below services in some subset of the existing scenarios?08:57
shardyand add new scenarios which are only run on >= $new_release?08:57
shardyI'm not super keen on the duplication implied by copying all-the-tests every release08:57
*** gfidente has joined #tripleo08:58
*** rasca has quit IRC08:59
*** rasca has joined #tripleo09:00
jaosoriorshardy: but then we'll have this same issue for every release as services are added09:02
dsarielshardy, thanks a lot for the help. The script is here; http://paste.openstack.org/show/588669/. I'm trying to run openstack overcloud deploy and if fails because of override.yaml (without override.yaml evrything is ok, but cinder is using swift inctead of ceph backend)09:02
shardyjaosorior: sure, but that could still be handled by having a matrix which grows over time, and only the latest releases run scenarion >= n09:05
shardyand that would avoid duplicating every single scenario for every release09:05
jaosoriorshardy: but then newer releases would have to run more and more scenarios, as some integration of some services will require scenarios that contain pretty much the same services.09:05
shardyjaosorior: so, you're trying to solve the problem that every release, new services are added09:06
shardybut you don't want to run more scenarios on newer releases?09:06
* shardy is confused09:06
*** fabbione has quit IRC09:07
shardyjaosorior: Perhaps there's some other way to make the scenarios more composable09:08
*** abregman has joined #tripleo09:08
shardymy main complaint is the copy/paste of the templates09:08
openstackgerritAdriano Petrich proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544909:08
shardyI had the same complaint when EmilienM started the scenario stuff, and this makes it worse AFAICS09:09
shardylet me think about it and see if I can offer a way around it09:09
jaosoriorshardy: a scenario that only contains barbican wouldn't make any sense. and as features are added in each release, pike will run scenarios for ocata, pike, and below...09:09
jaosoriorsure09:10
jaosoriorshardy: now the issue is that we can't add more services cause that breaks the stable branches. And the current proposal is to stop running the offending scenarios entirely.09:11
pandadam, I will never get https://review.openstack.org/382515 merged. Each recheck, at least one random job red .. :(09:11
shardyjaosorior: Yeah, I'm just saying lets juggle the scenarios so we can say e.g only run scenario 1 & 2 on < newton and 1, 2 3 on > newton09:12
jaosoriorso I having scenarios is not about having a test check-list; but about testing something that makes sense09:12
shardynot that we'd run a scenario per service09:12
shardythere's already a bunch of overlap between scenarios09:12
shardyLike we could add a scenario 409:12
shardyand just disable it for stable branches09:12
shardyit'd run some subset of the existing stuff plus $new_shiny_for_ocata09:13
shardyjaosorior: maybe drop a message to the list, definitely requires some wider discussion I think09:13
jaosoriorso, by "subset" you just mean "a specific set of scenarios" ?09:13
shardyhttps://github.com/openstack-infra/tripleo-ci#service-testing-matrix09:13
*** fabbione has joined #tripleo09:14
shardyeach scenario already tests some subset of services09:14
jaosoriorcan you rephrase what you said about the subset? I'm honestly getting lost here.09:14
* jaosorior needs to brew more coffee09:14
*** b00tcat has joined #tripleo09:15
shardyjaosorior: I'm not sure how to better explain it, if you give me a list of the services you expect to be new for ocata, I'll push a patch illustrating my suggestion to the test matrix09:16
shardyAll I'm saying is rather than copying all the scenarios, lets just not run them all on stable branches09:17
jaosoriorshardy: barbican, mistral & zaqar09:18
jaosoriorI had initially added barbican in scenario 2, since that scenario already does a volume and attaches it to an instance, so I want to eventually be able to test encrypted volumes.09:18
jaosoriorI don't know enough about zaqar and mistral to know how those were meant to be tested in the scenarios though09:19
*** dtantsur|afk is now known as dtantsur09:21
*** lucas-afk is now known as lucasagomes09:23
*** ohamada has quit IRC09:24
*** ohamada has joined #tripleo09:24
*** tremble has joined #tripleo09:27
*** fabbione has quit IRC09:28
*** yamahata has quit IRC09:31
*** yolanda has quit IRC09:31
*** milan has joined #tripleo09:31
*** yolanda has joined #tripleo09:31
d0ugaljaosorior: did you have any luck with your issue yesterday?09:32
jaosoriord0ugal: so, I cannot reproduce it in master. Waiting for confirmation if someone else can reproduce it OSP1009:33
d0ugaljaosorior: k, thanks09:33
jaosoriormcornea: you were running a job to check if its reproduceable in OSP10 right? the issue with updates from yesterday09:34
shardyjaosorior: ack, OK so heat actually supports conditionals now, so we could add coverage to the scenario2 template with those resources conditionally enabled based on the branch name09:34
shardyheat supports resources for all those services already09:34
mcorneajaosorior: yes, let me see the result09:34
*** akrivoka has joined #tripleo09:34
mcorneajaosorior: looks good09:34
jaosoriorshardy: uhm... I like that idea.09:34
jaosoriorshardy: I was already working on adding a scenario004 if that's the way people wanna go09:34
shardyjaosorior: I'll comment on the patch with more details09:35
shardyyeah either would work I think, given that the scenarios are conditionally triggered09:35
jaosoriormcornea: lets wait to see if the dude that reported the issue still has it with the latest OSP puddle09:35
jaosoriormcornea: can you pass a reference to the job definition?09:37
jaosoriorshardy: what should go first for a scenario, the project-config commit or the tripleo-ci commit?09:37
*** fabbione has joined #tripleo09:38
mcorneajaosorior: I'll add it to the bz09:38
shardyjaosorior: I think the tripleo-ci commit has to come first, so make the project-config patch Depends-On it09:44
*** milan has quit IRC09:45
*** milan has joined #tripleo09:46
chemgfidente: hi, I've got this strange problem where using a blockstorage with ceph backend doesn't seem to work at all09:46
chemgfidente: I've documented there https://bugs.launchpad.net/tripleo/+bug/164073009:46
openstackLaunchpad bug 1640730 in tripleo "Installing overcloud with block/object storage node and ceph backend doesn't work." [Undecided,New]09:46
*** zoli|afk is now known as zoli|wfh09:48
*** kaslcrof has joined #tripleo09:48
jaosoriormcornea: right, but can you pass a link to the code that actually executes that job?09:48
*** social has quit IRC09:48
mcorneajaosorior: I manually trigger it09:49
jaosoriorshardy: having gotten more coffee and thought about it. Actually having extra scenarios might be the way to go. Would probably be less of a burden to maintain09:49
*** pcaruana has quit IRC09:49
jaosoriormcornea: I see, and, what about the job definition?09:49
*** limao has quit IRC09:53
jaosoriormcornea: wow, so from what I can see, that job pretty much does the same thing than what the guy that reported the bug is doing09:56
jaosoriormcornea: I think it's just a matter of updating the puddle indeed09:56
*** social has joined #tripleo09:57
gfidentechem, so that is the expected behaviour09:58
gfidenteblockstorage nodes run the cinder/lvm driver only09:58
mcorneajaosorior: yeah, unless there are some environmental things that I miss I think it should work09:58
gfidentein newton it shouldn't have been different09:59
*** katkapilatova has joined #tripleo09:59
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Introduce scenario004  https://review.openstack.org/39605609:59
gfidentechem, so you should have the cephstorage nodes running ceph-osd and the blockstorage nodes running cinder-volume10:01
openstackgerritMerged openstack/tripleo-heat-templates: Defaults kernel.pid_max to 1048576  https://review.openstack.org/39560810:01
chemgfidente: that's corrrect10:01
*** pcaruana has joined #tripleo10:01
gfidentechem, and if you do cinder service-list you should see a single tripleo_ceph backend10:02
gfidenteand multiple tripleo_iscsi agents (one per each blockstorage node)10:02
*** sudipto has joined #tripleo10:04
*** sudipto_ has joined #tripleo10:04
*** athomas has joined #tripleo10:06
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Introduce scenario004  https://review.openstack.org/39605610:09
*** jaosorior is now known as jaosorior_lunch10:10
lucasagomesfolks, can someone take a look at this: https://review.openstack.org/#/c/370773/ It's a tiny patch (1 LOC) and have been hanging around for a long time. Thanks10:10
dtantsurlucasagomes, we should not patch this code any more..10:11
dtantsurwell, if you want to fix it for older releases (pre-newton), then yes10:11
jaosorior_lunchdtantsur, lucasagomes well, if you wanna backport it it needs an LP bug10:12
dtantsurjaosorior_lunch, there no stable branches there10:12
jaosorior_lunchno?10:12
jaosorior_lunchah10:12
jaosorior_lunchok10:12
lucasagomesdtantsur, this patch is from September, I was mostly tyring to get rid of my list of patches... but still, if not used anymore I will just abandon it10:13
dtantsurlucasagomes, mind fixing https://github.com/openstack/tripleo-common/blob/master/tripleo_common/utils/nodes.py#L165 instead please? this is what tripleo really uses. and please create an LP bug, so that we can backport it later (tripleo-common does have stable branches)10:13
lucasagomesdtantsur, on it10:14
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Select bootstrap node by list index not name  https://review.openstack.org/39569910:27
dtantsurshardy, do you by chance have any plans to continue with heat support for ironic?10:28
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Select bootstrap node by list index not name  https://review.openstack.org/39569910:28
shardydtantsur: I don't atm but I think ricolin was planning to revive it10:28
shardydtantsur: for the tripleo use-case, I think defining a mistral workflow which deploys the nodes is somewhat more interesting10:29
shardy(that workflow could then be triggered via heat)10:29
shardyhttps://review.openstack.org/#/c/313048/ was a first pass at that, but I've not had time to get it fully working10:29
shardydtantsur: did you have a reason for wanting those resources?10:34
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Replace ucs_hostname with ucs_address  https://review.openstack.org/39608610:34
lucasagomesjaosorior_lunch, dtantsur ^10:34
dtantsurshardy, creating a scenario job exercising ironic in overcloud, primarily10:39
*** jerrygb has joined #tripleo10:40
shardydtantsur: Ack, Ok we'll have to think how to do that - we could revive the heat resources, but I reached the conclusion that they won't be that useful without the workflow to drive deployment (which is a better fit for mistral than heat I think)10:43
shardyOne option would be to drive the interaction with Ironic with a OS::Mistral::Workflow resource, but that doesn't run the workflow (it's triggered via an alarm or user signal)10:45
*** jerrygb has quit IRC10:45
openstackgerritChris Jones proposed openstack/puppet-tripleo: Improve failed mysql node removal time in HA deploys.  https://review.openstack.org/39609210:45
dtantsurhmm10:45
dtantsurunrelated: could you please review one more small ocata-1/netwon bug fix: https://review.openstack.org/#/c/392181/10:46
shardyThat looked directed at me, but I've already +2'd it so other folks please review :)10:47
*** abregman is now known as abregman|afk10:47
dtantsuryeah, sorry, it was a general request not related to the discussion10:48
dtantsurjust trying to clean up my bug list for newton downstream10:48
openstackgerritAnshul Jain proposed openstack/diskimage-builder: DIB element to support cinder local attach/detach functionality  https://review.openstack.org/38588010:50
*** tosky has joined #tripleo10:50
*** chandankumar has quit IRC10:52
*** chandankumar has joined #tripleo10:58
*** gfidente has quit IRC10:58
*** owalsh has joined #tripleo11:03
openstackgerritChris Jones proposed openstack/puppet-tripleo: Improve failed mysql node removal time in HA deploys.  https://review.openstack.org/39609211:03
*** mgould|afk is now known as mgould11:11
*** gfidente has joined #tripleo11:17
*** hjensas has quit IRC11:20
*** jerrygb has joined #tripleo11:25
*** hjensas has joined #tripleo11:30
*** abregman|afk is now known as abregman11:34
*** thrash|g0ne is now known as thrash11:36
*** jkilpatr has quit IRC11:37
*** sudipto has quit IRC11:38
*** sudipto_ has quit IRC11:38
shardythrash: Hey, good morning - am I right in thinking that you've been doing some work on the overcloud node delete command?11:40
thrashshardy: sort of... rbrady-afk did that one.11:42
shardythrash: ah, OK no worries11:42
thrashshardy: wait...11:42
thrashshardy: he used some ideas of mine from stack update.11:42
shardyI raised https://bugs.launchpad.net/tripleo/+bug/1640770 with a few usability issues I noticed, was looking for a volunteer to fix them ;)11:42
openstackLaunchpad bug 1640770 in tripleo "openstack node delete usability issues" [Medium,Triaged]11:42
thrashshardy: sure. I'll take a look.11:42
shardynot super urgent, I can probably get to it myself in due course if you're busy with other stuff11:43
*** chlong has joined #tripleo11:43
thrashshardy: not at all. It's on a list of things to improve the cli this cycle.11:44
*** chlong has quit IRC11:44
*** cdearborn has joined #tripleo11:47
openstackgerritKaterina Pilatova proposed openstack/tripleo-validations: callback_plugins/validation_output.py: improved output  https://review.openstack.org/39613511:48
d0ugalthrash: I have done something similar to the usability issues11:55
d0ugalthrash: https://review.openstack.org/#/c/395522/11:55
*** bkreitch has joined #tripleo11:55
d0ugalthrash: I have only taken into account the first part11:56
thrashd0ugal: that goes a long way to solve shardy's main complaint11:56
d0ugaland mine too :)11:56
d0ugalthrash: I can update it to default to 'overcloud'11:56
thrashd0ugal: +111:57
thrashrelocating to RHT...11:57
*** thrash is now known as thrash|biab11:57
d0ugalthrash|biab: k, i'll do that, then you just need to do the 2nd part11:57
*** jkilpatr has joined #tripleo11:58
mariosreviews please https://review.openstack.org/#/c/395053/11:59
*** paramite has quit IRC12:03
*** jpena is now known as jpena|lunch12:04
openstackgerritMerged openstack/tripleo-specs: Spec: undercloud-ntp-server - Add NTP services to Undercloud  https://review.openstack.org/35072912:06
*** paramite has joined #tripleo12:07
openstackgerritMerged openstack/tripleo-specs: TripleO Repo Management Tool  https://review.openstack.org/38923712:08
*** fragatina has joined #tripleo12:11
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Default the stack name to 'overcloud  https://review.openstack.org/39552212:11
*** fragatina has quit IRC12:11
*** fragatina has joined #tripleo12:12
marios/win 2912:13
d0ugalso much win12:13
akrivokalol12:14
*** cylopez has quit IRC12:19
*** dprince has joined #tripleo12:19
*** cylopez has joined #tripleo12:20
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Default the stack name to 'overcloud' in overcloud node delete  https://review.openstack.org/39552212:20
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Normalize civetweb binding address if IPv6  https://review.openstack.org/39578512:21
*** arxcruz is now known as arxcruz|brb12:23
*** masco has quit IRC12:24
*** bkopilov has quit IRC12:31
*** dmacpher has quit IRC12:33
*** dmacpher has joined #tripleo12:33
*** jaosorior_lunch is now known as jaosorior12:34
gfidentemarios good call on the chcon12:34
gfidenteI think we need to find an agreement12:35
*** thrash|biab is now known as thrash12:35
*** pkovar has joined #tripleo12:36
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Deployed server: switch to apply-config hook  https://review.openstack.org/39617112:39
jaosorioris there anybody with some extra cycles to help me out debugging a heat deployment over httpd?12:39
*** chandankumar has quit IRC12:41
*** lucasagomes is now known as lucas-hungry12:41
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Handle null role_data in services  https://review.openstack.org/39617512:43
thrashd0ugal: can you tag your patch with Partial-Bug (I think that's the right one)12:45
thervejaosorior, What's up?12:48
*** trown|outtypewww is now known as trown12:48
*** maticue has joined #tripleo12:50
d0ugaltherve: I thought I did. Checking.12:50
d0ugaltherve: I almost did12:50
thervethrash, ^^12:50
therve:)12:50
jaosoriortherve: hey, so, I'm trying to deploy both heat and heat-cfn over httpd in TripleO's undercloud. And I ended up writing the necessary stuff in puppet-heat (which pretty much mimics how other services do it) https://review.openstack.org/#/c/394835/ ; however, when doing overcloud deploy, the deployment times out... In this attempt I'm only deploy heat-api over httpd https://review.openstack.org/#/c/394837/312:50
thrashtherve: haha...12:51
d0ugaltherve: sorry12:51
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Default the stack name to 'overcloud' in overcloud node delete  https://review.openstack.org/39552212:51
*** panda has quit IRC12:51
therved0ugal, No worries12:51
d0ugalthrash: I had used "Parial-Bug" lol12:51
jaosoriortherve: this attempt is failing on the multinode job only; when I was trying to deploy heat-api and heat-cfn also, the ovb jobs were failing as well.12:51
thrashd0ugal: haha12:51
openstackgerritHonza Pokorny proposed openstack/tripleo-specs: Add gui logging spec  https://review.openstack.org/39513812:52
jaosoriortherve: now, I tried it in a local deployment, and it both the heat endpoint and the heat-cfn endpoint seem to work :/ .... so I'm not entirely sure what part is messed up12:52
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Neutron L3 service cleanups for hiera json hook  https://review.openstack.org/39326212:54
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Horizon service cleanups for hiera json hook  https://review.openstack.org/39325812:54
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook  https://review.openstack.org/38475712:54
thervejaosorior, Have you managed to extract anything from the failures?12:54
jaosoriorthat's the issue, no logs indicating failure :/12:55
jaosoriorI'm doing another local install now to reproduce it12:56
therveI just see 'CREATE aborted' on the controller12:56
*** kjw3 has quit IRC12:56
jaosoriortherve: yep, it times out12:56
*** panda has joined #tripleo12:57
dprinceshardy: this seemed to work for me with my role_data issue https://review.openstack.org/#/c/396175/12:57
jaosoriorhaven't figured out why :/12:57
therveOh12:57
thervejaosorior, I was confusing hours and minutes12:58
therveIt's actually sitting there 1h20 doing nothing12:58
jaosoriorany idea what's it waiting for?12:58
jaosoriorpanda:regarding https://review.openstack.org/#/c/396008/ I mention Ocata in the commit message12:59
jaosoriorpanda: anyway, in that scheme, master would be tested without a prefix13:00
*** ealcaniz has quit IRC13:00
jaosoriorpanda: so that's one proposition, the other one https://review.openstack.org/#/c/396056/ is to have more scenarios13:00
jaosoriorpanda: which at the moment I'm starting to favor more13:00
*** zoli|wfh is now known as zoli|lunch13:00
*** ealcaniz has joined #tripleo13:01
jaosoriorpanda: so in this case, scenario004 wouldn't run in newton and below, this way, we can safely add new servicse.13:01
jaosorior*services13:01
*** panda has quit IRC13:02
*** jpena|lunch is now known as jpena13:03
*** jayg|g0n3 is now known as jayg13:03
*** egafford has joined #tripleo13:04
openstackgerritTom Barron proposed openstack/tripleo-heat-templates: Add firewall rules for manila api service  https://review.openstack.org/39618313:04
*** panda has joined #tripleo13:06
openstackgerritJiri Tomasek proposed openstack/tripleo-specs: GUI Deployment Configuration improvements  https://review.openstack.org/39336513:08
jtomasekflorianf: ^ I hope it makes more sense now13:08
thervejaosorior, Oh, that changed the *undercloud* heat?13:09
therveI misunderstood13:09
therveOK so yeah os collect config fails to talk to heat13:10
jaosoriortherve: would that be heat-api or heat-cfn?13:10
thervejaosorior, api13:10
florianfjtomasek: thanks, lgtm13:10
florianfjtomasek: thanks for the changes.13:10
jaosoriorok13:11
jaosoriortherve: and what would that imply? we can see that we can begin the stack creation through heat-api. And I'm not changing any ports for heat either :/13:11
*** bfournie has quit IRC13:11
*** Vijayendra has quit IRC13:11
thervejaosorior, I'm not exactly sure yet, but it makes more sense :D13:12
*** panda is now known as panda|afk13:12
thervejaosorior, See http://logs.openstack.org/37/394837/3/check/gate-tripleo-ci-centos-7-nonha-multinode/6479036/logs/var/log/httpd/heat_api_wsgi_access.txt.gz we're getting 404 on metadata calls, that's the issue13:12
jaosoriortherve: I can discard selinux being an issue cause I'm seeting I'm seeing the same issue in a local deployment running on permissive mode13:13
*** liverpooler has joined #tripleo13:13
*** coolsvap has quit IRC13:13
*** rbrady-afk is now known as rbrady13:13
jaosoriortherve: so... trying to access heat-cfn (which is the metadata service...right?) is returning 404?13:15
*** arxcruz|brb is now known as arxcruz13:15
thervejaosorior, No that's heat-api13:15
jaosoriorOook13:15
thervejaosorior, I believe the issue is with the %2F in there, I don't know what it is doing13:16
shardytherve, jaosorior: we use the request collector by default, so we poll swift not heat-api13:16
shardywe do still use CFN signaling tho13:17
jaosoriortherve: which %2F?13:17
jaosoriorI see13:17
thervejaosorior, "GET /v1/e17833ec2d13432e9ccb74483530aeaf/stacks/overcloud-Controller-nlbcp2mngxpo-0-zpjcit3idr4y-Controller-3lfxsulsbqt3-deployed-server-h72riaxxmln4%2F0bc4f9f6-ba41-4d4c-ad47-0c708049c7f0/resources/deployed-server-config/metadata HTTP/1.1" 40413:17
therveThe %2F in there13:17
Ngjaosorior: would you care to look at the newton backport of the haproxy/mysql patch you +2'd? It's identical :) https://review.openstack.org/#/c/396092/213:17
therveshardy, What's talking to metadata then?13:17
Ngjaosorior: much appreciated, thank you13:18
shardytherve: Oh, I guess this is the multinode job?13:18
therveshardy, Yes13:19
shardy./overcloud-resource-registry-puppet.j2.yaml:  SoftwareConfigTransport: POLL_TEMP_URL13:20
therveOh I think I know13:20
shardyI don't see us set that in tripleo-ci13:20
*** kjw3 has joined #tripleo13:21
thervejaosorior, Where's the puppet code that configure httpd? Or the resulting config?13:21
akrivokarbrady: jtomasek: ping, I'd like to discuss the import and export of plans, when you have a minute13:21
jtomasekakrivoka: yep, ready13:22
jaosoriortherve: /etc/httpd/conf.d/10-heat_api_wsgi.conf13:23
shardyCan I get another review for https://review.openstack.org/#/c/395699/ please?13:23
thervejaosorior, I meant in the gate logs :)13:23
therveBut I think I found it13:23
shardyfixes a problem with node replacement that I want to backport to newton asap13:23
akrivokarbrady: jtomasek: basically I'd like to write blueprints for plan and environment import and export from gui and need to understand what exactly we need on the front and back end13:24
thervejaosorior, It's missing AllowEncodedSlashes13:24
jaosoriortherve: duuuude yes!13:24
shardytherve: nice work spotting that :)13:24
therveshardy, Nothing liking finding a %2F in a 240 chars URL :)13:25
shardyhehe :)13:25
jaosoriortherve: thanks a lot dude! I've been struggling with this13:25
thervejaosorior, No problem!13:26
jtomasekakrivoka: so for import I think all we need it a way to identify the mistral environment. as that would most probably be exported as json, then for import I think a convention of calling the file mistral-environment.json is enough. This is the same approach we use to identify capabilities-map.yaml13:26
*** lucas-hungry is now known as lucasagomes13:27
jtomasekakrivoka: so when creating the plan, tripleo-common would find that the swift container includes that file and updates mistral environment with it's content13:27
jtomasekakrivoka: (same for updating the plan)13:27
*** udesale_ has joined #tripleo13:28
d0ugalakrivoka: I have been looking into this from the mistral workflow side.13:28
d0ugaljtomasek: ^13:29
jtomasekakrivoka: the problem I see here is that tripleo-common would have to take care of removing that file from swift once it updates mistral environment with its contents13:29
shardyCan I also get reviews on https://review.openstack.org/#/q/Id240709e256b057bbdff038aefc7fcacb0d339b8,n,z please13:29
shardyone failed the ha ovb job but that file isn't tested in CI anyway13:29
*** amoralej is now known as amoralej|lunch13:30
jtomasekakrivoka: exporting the plan is a matter of downloading all the files from swift container in tar + include mistral environment contents as mistral-environment.json13:30
d0ugaljtomasek, akrivoka - it is, but a workflow should provide that13:30
jtomasekd0ugal: yep, agree13:30
d0ugaljtomasek, akrivoka - importing is just "create", we have that. It might need to be extended slightly13:31
d0ugaljtomasek, akrivoka - and then an export workflow - I am not sure how it gives you the files yet...13:31
d0ugaljtomasek, akrivoka - but I am just walking out the door to get lunch - back in a bit.13:31
*** redhatkj has joined #tripleo13:32
*** fragatina has quit IRC13:33
akrivokajtomasek: regarding removing the mistral-environement.json after the update - how does are we handing that with capabilities-map.yaml?13:34
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Normalize civetweb binding address if IPv6  https://review.openstack.org/39578513:34
bandinichem: did you manage to conclude your tls/nontls testing on https://review.openstack.org/#/c/395454/ ? if yes, can you throw in the results in the review?13:34
jtomasekakrivoka: regarding GUI for import no changes should be needed and for export, we only need to add button to plans list which triggers an action to run the export workflow13:34
*** fultonj has joined #tripleo13:34
jtomasekakrivoka: capabilities-map.yaml is different in this regards, it is not being changed13:35
chembandini: didn't I ... sorry, doing it.  all is green and good13:35
*** jerrygb has quit IRC13:35
bandinichem: awesome, thanks ;)13:35
*** jerrygb has joined #tripleo13:36
jtomasekakrivoka: mistral-environment.json is just a tool which transports its content to actual mistral environment. then it is not needed13:36
*** rhallisey has joined #tripleo13:36
*** kjw3 has quit IRC13:36
jtomasekakrivoka: if we left it in the swift container and user would update the plan with additional plan, tripleo-common would overide the mistral environment with that file again, which is not what we want. this is why we need to remove it after it is read13:37
akrivokajtomasek: is there any harm in just leaving it in the swift container?13:37
akrivokajtomasek: ah I see :)13:37
jtomasekakrivoka: or we would have to keep it in sync with mistral environment which would be more problematic and totally not necessary13:37
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Deploy heat APIs over httpd  https://review.openstack.org/39483713:38
*** morazi has joined #tripleo13:38
akrivokajtomasek: so there simply needs to be a step (action?) in the workflow to remove this file from the container13:38
chembandini: but on the tls setup, but I had an unfortunate vpn disconnection during convergence.  Replying to the review anyway13:38
jtomasekakrivoka: yes13:38
*** jerrygb has quit IRC13:40
jtomasekakrivoka: pdate the plan -> is there mistral-environment.json? ->  I read it and store it's content in mistral environment -> remove mistral-environment.json from swift container13:40
akrivokajtomasek: got it13:41
*** jprovazn has quit IRC13:42
bandinichem: oh I see. I guess if it was during convergence we *should* be good as I cannot possibly think how our review affects anything but the major-upgrade step13:42
*** rhallisey has quit IRC13:43
*** oshvartz has quit IRC13:43
akrivokajtomasek: d0ugal: ok, I'll try to formalize this in a spec, I think one spec for both import and export would be ok, then we can create blueprints and bugs off of that as needed. sounds good?13:43
chembandini: ack, updated the review accordingly13:43
openstackgerritTomas Sedovic proposed openstack/tripleo-docs: Add tripleo-validations user documentation  https://review.openstack.org/39619713:43
openstackgerritMerged openstack/tripleo-docs: Link to RDO built images  https://review.openstack.org/39460213:44
bandinichem: thank you sir ;)13:44
chembandini: hehe, you're welcome.13:45
openstackgerritMerged openstack/tripleo-docs: Remove the replace controller docs  https://review.openstack.org/38146913:46
*** jerrygb has joined #tripleo13:47
*** bfournie has joined #tripleo13:48
*** bogdando has quit IRC13:50
*** rlandy has quit IRC13:52
openstackgerritDan Prince proposed openstack/tripleo-puppet-elements: Move nova.conf truncation into separate element  https://review.openstack.org/39255913:53
*** rhallisey has joined #tripleo13:56
*** cmyster has quit IRC13:56
*** jcoufal has joined #tripleo13:57
*** cmyster has joined #tripleo13:57
*** cmyster has quit IRC13:57
*** cmyster has joined #tripleo13:57
openstackgerritJoe Talerico proposed openstack/tripleo-heat-templates: Increase reserved memory for computes when enabling DVR  https://review.openstack.org/38259413:57
*** rlandy has joined #tripleo13:57
jaosoriorsshnaidm: for reference, this is the bug about the network range https://bugs.launchpad.net/tripleo/+bug/155322213:58
openstackLaunchpad bug 1553222 in tripleo "Default undercloud control plane network violates rfc5737" [Undecided,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)13:58
openstackgerritKaterina Pilatova proposed openstack/tripleo-validations: undercloud-tokenflush: changed group to pre-introspection  https://review.openstack.org/39621213:59
*** athomas has quit IRC13:59
openstackgerritSteven Hardy proposed openstack/tripleo-docs: Add note with boot config image options  https://review.openstack.org/33607813:59
sshnaidmjaosorior, ok13:59
openstackgerritMerged openstack/tripleo-docs: Fix OpenStack client invocation  https://review.openstack.org/38235314:03
*** Goneri has joined #tripleo14:04
*** athomas has joined #tripleo14:04
afazekasIs a current  tripleo setup able to boot up a vm in an ipv6 setup ?14:04
*** amoralej|lunch is now known as amoralej14:05
sshnaidmjaosorior, trying to find why instack-undercloud tests fail..14:06
*** tzumainn has joined #tripleo14:07
*** jprovazn has joined #tripleo14:07
jaosoriorsshnaidm: what was the patch?14:09
sshnaidmjaosorior,  https://review.openstack.org/#/c/395702/14:09
jaosoriorsshnaidm: alright, trying to reproduce14:11
jaosoriorshardy: here are the patches for the scenario004 that will only run on on newer releases than newton https://review.openstack.org/#/c/396062/  https://review.openstack.org/#/c/396056/14:12
openstackgerritDan Prince proposed openstack/os-collect-config: Add support for Zaqar websockets  https://review.openstack.org/34248614:13
sshnaidmjaosorior, how do you run them? I fail to run tests with tox -v -epy2714:14
*** cdearborn has quit IRC14:14
jaosoriorsshnaidm: have you tried re-creating the tox environment?14:14
jaosoriorsshnaidm: tox -r -e py2714:14
*** lblanchard has joined #tripleo14:14
*** panda|afk is now known as panda14:15
sshnaidmjaosorior, the same, everything fails14:16
jaosoriorsshnaidm: well, I can reproduce it in my local environment, I'll take a look14:16
*** tiswanso has joined #tripleo14:17
sshnaidmjaosorior, do you have FAILED (id=5, failures=15 (+15)) ?14:17
jaosoriorI have 15 failures14:18
jaosoriordifferent id, but that's to be expected14:18
*** jerrygb has quit IRC14:19
jaosoriorthe issue is this: https://review.openstack.org/#/c/395702/3/instack_undercloud/undercloud.py we're setting the sample_default, not the actual default value14:19
*** sudipto_ has joined #tripleo14:19
*** sudipto has joined #tripleo14:19
jaosoriorsshnaidm: fixing14:20
jaosoriorfixed14:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Remove deprecated network range 192.0.2.0  https://review.openstack.org/39570214:20
jaosoriorsshnaidm: ^^14:20
d0ugaljtomasek, akrivoka - back.14:20
*** jkilpatr has quit IRC14:20
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: Test SoftwareConfigTransport: ZAQAR_MESSAGE  https://review.openstack.org/34436514:20
jaosoriorsshnaidm: this is the only thing that changed https://review.openstack.org/#/c/395702/3..4/instack_undercloud/undercloud.py14:20
d0ugaljtomasek, akrivoka - I am trying to get a spec together for "plan management". so I expect i'll touch on this stuff too, but I wont mention the GUI at all14:21
pandabandini: are you a mysql god ?14:21
sshnaidmjaosorior, thanks, not obvious for me :)14:21
*** jkilpatr has joined #tripleo14:22
jtomasekd0ugal: cool14:22
*** jkilpatr has quit IRC14:22
*** zoli|lunch is now known as zoli|wfh14:22
bandinipanda: lol I barely know how to spell it. but thankfully we have dciabrin and mike around ;) what's the problem?14:23
jaosoriorshardy: is it cool if I specify which scenario is run on what branch in another CR?14:24
jtomasekd0ugal: could you please add me as a reviewer when you have the patch ready?14:24
d0ugaljtomasek: sure, hoping for a first draft early tomorrow14:25
jaosoriorsshnaidm: no biggie, glad to help14:25
jtomasekd0ugal: thanks14:25
Ngjaosorior: ccamacho: thanks for the +2s, 396092 now has (sufficiently) green CI :)14:25
openstackgerritDan Prince proposed openstack/tripleo-puppet-elements: Move nova.conf truncation into separate element  https://review.openstack.org/39255914:26
shardyjaosorior: sure14:26
pandabandini: then .. die! (force lightning)14:26
ccamachoNg awesome, I was going to make the review in the morning but was already landed :)14:26
Ngccamacho: :D14:27
bandinipanda: rotfl14:27
sshnaidmpanda, can you please try to recreate your dev env? I wonder if "nova boot" works for you..14:27
*** zoli|wfh is now known as zoli|brb14:27
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Use j2 loops in post.j2.yaml  https://review.openstack.org/39623014:27
b00tcathey y'all - when building the undercloud images I'm getting: "ironic-python-agent already found in services list.  Check ironic-agent for duplicate service name." and cannot continue, does this ring a bell?14:28
b00tcatI could build overcloud-full just fine before14:28
pandasshnaidm: ok14:29
*** jkilpatr has joined #tripleo14:29
b00tcatsorry, overcloud iamges*14:30
*** links has quit IRC14:31
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add note about scenario004 in README  https://review.openstack.org/39623814:32
jaosoriorshardy: what about this? ^^14:32
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook  https://review.openstack.org/38475714:33
pandajaosorior: where exactly are we using scenarios ?14:33
*** jaosorior has quit IRC14:34
openstackgerritJiri Tomasek proposed openstack/tripleo-specs: GUI Deployment Configuration improvements  https://review.openstack.org/39336514:34
*** jaosorior has joined #tripleo14:35
*** b00tcat has quit IRC14:36
jpichd0ugal: I'm talking about https://bugs.launchpad.net/tripleo/+bug/1640815 with florianf but there seems to be a Mistral bug in there as well (in api.log) - might be of interest!14:37
openstackLaunchpad bug 1640815 in tripleo "Selecting too many environments breaks Mistral (?)" [High,Triaged]14:37
d0ugaljpich: oooh, fun14:37
pandasshnaidm: it's taking a lot of time to spawn14:38
d0ugaljpich: TypeError: __init__() takes exactly 4 arguments (2 given)14:38
pandasshnaidm: but worked14:38
d0ugaljpich: that bit?14:38
jpichd0ugal: Yes14:38
sshnaidmpanda, so nova boot worked>14:38
d0ugaljpich: that sounds horrific. I'll take a look14:38
sshnaidmpanda, which version of nova client do you have?14:38
jpichd0ugal: That's what gets returned to the API consumer which is not the bestest :) Sorry for the crappy formatting on the bug, I'll see if I can do better14:38
jpichflorianf: Did you see the same output about the SizeLimitException in the engine.log?14:39
jaosoriorpanda: what do you mean?14:39
d0ugaljpich: lol, it's fine. I'll see what I can find out.14:39
*** jcoufal has quit IRC14:39
pandasshnaidm: using the one in jumphost (6.0.0-1.el7)14:40
florianfjpich: let me see.14:40
sshnaidmpanda, do you run this from jumphost?14:40
pandajaosorior: do we have gate or periodic jobs for those ? are they used somewhere else ?14:40
pandasshnaidm: yes14:40
sshnaidmpanda, and it's rh1, right?14:40
jaosoriorpanda: they run on the gate14:40
jaosoriorpanda: and only run when specific files are modified, depending on the file it's the scenario we run14:41
florianfjpich: yep, got that error too14:41
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Create directories with root  https://review.openstack.org/38489214:41
jaosoriorpanda: here's the definition https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L155314:42
*** zoli|brb is now known as zoli|wfh14:43
*** numans has quit IRC14:43
*** numans has joined #tripleo14:43
pandajaosorior: ah, thanks.14:44
d0ugaljpich: I can't see how that error is possible14:44
florianfjpich: OK, it doesn't seem to have to do with the TLS endpoint env. if I select an environment whose file has a similar size (like network iso), it hits the sizelimit error too14:45
florianfd0ugal: ^14:45
d0ugalflorianf: What do you mean by "select an environment"?14:45
jpichd0ugal: I've got it in both a Newton and Ocata env (<- could give you access to that one?), let me know if I can give you any additional information/logs? Should I file a "FYI" Mistral bug?14:46
jpichd0ugal: "enable" - it's checkboxes from the UI side :)14:46
d0ugaljpich: Yeah, probably a good idea. We should never get tracebacks in the api.log14:46
jpichflorianf: Ok, I shall try adjusting the Mistral conf and see if that helps... just need to figure out which configuration setting that could be14:46
florianfd0ugal: checking the box for an environment in the ui14:47
d0ugaljpich, florianf: is the size limit an issue with the size of the workflow input? or something else?14:47
jpichd0ugal: I'm not sure... seems to be related to task execution from the log extract?14:48
tzumainnjpich, florianf, just a heads up - the project I'm working on got a similar error half a year ago, we had to adjust the max_json_body_size in /etc/heat/heat.conf14:48
florianfjpich: sorry, forget what I said, I didn't save my overall settings, so one of the tls endpoint envs was still checked.14:48
jpichtzumainn: In heat, huh. This one seems to be coming from Mistral but I'll have a look there too14:49
jpichflorianf: Ok!14:49
tzumainnjpich, I'm just wondering if you guys will hit it once you try and deploy an overcloud with a lot of environments (in other words, once the mistral issue is cleared up)14:49
tzumainnif you do, it's probably ^14:50
jpichtzumainn: Bleh, thanks for the heads-up14:50
*** jcoufal has joined #tripleo14:50
florianfd0ugal, jpich: arent these supposed to be valid json strings? https://github.com/openstack/tripleo-heat-templates/blob/master/environments/tls-endpoints-public-ip.yaml14:53
*** derekh has joined #tripleo14:53
jaosoriorAny +As for this? https://review.openstack.org/#/c/392473/ :D14:54
jpichflorianf: I'm not sure, that's a good point14:55
d0ugaltzumainn: we have people deploying with lots of envs via the CLI I believe.14:55
d0ugalflorianf: are they not?14:56
jpichflorianf: As a dictionary it seems a bit broken? Might be worthwhile comparing with other yaml and see?14:56
d0ugalflorianf: well, I mean, it's yaml. /me is confused.14:56
jaosoriorshardy: could you check this out? https://review.openstack.org/#/c/392473/14:56
florianfd0ugal: no, the params in this template look like a python dict notation14:57
florianfd0ugal: but that isn't valid json14:57
shardyflorianf: it's a yaml map, not a json string14:57
d0ugalflorianf: it's valid YAML.14:59
d0ugalyeah14:59
jpichd0ugal: Surprised to see the dictionary keys are not between quotes, but that is valid yaml as well then?14:59
jpichcool14:59
florianfshardy, d0ugal: OK, thanks. It was just a wild guess...14:59
d0ugalIn yaml, almost everything is valid. it is scary14:59
jpichUnderstood!14:59
* jpich goes back to config parameter hunting14:59
jpichflorianf: Did you manage to reproduce the error with other environments in the end?15:00
florianfjpich: nope15:00
jpichflorianf: Interesting... Would you mind adding that information to the bug?15:00
florianfjpich: yep15:01
jpichflorianf: :D15:01
florianfjpich: uh... no, I wouldn't, of course :-)15:01
*** dsneddon has quit IRC15:04
*** noslzzp has quit IRC15:04
jaosoriortherve: is zaqar supported in newton and below?15:04
jaosoriorhuh15:05
jaosoriorsorry15:05
jaosoriorthrash: is the zaqar tripleo profile supported in newton and below?15:05
*** noslzzp has joined #tripleo15:05
thrashjaosorior: no, it hasn't even landed yet.15:05
*** lblanchard has quit IRC15:05
thrashjaosorior: Is that why barbican was taken out?15:06
jaosoriorthrash: so, for the barbican profile I had an issue that... it merged and broke CI... and that was because the scenarios run for all branches, so if the service is not available in an earlier release, it will crash15:06
thrashjaosorior: gotcha15:06
jaosoriorso you might wanna wait until the scenario004 merges15:06
jaosoriorI've set that up so it runs in branches newer than newton15:06
thrashjaosorior: ok. Will have to change layout.yaml...15:07
openstackgerritMerged openstack/tripleo-heat-templates: Select bootstrap node by list index not name  https://review.openstack.org/39569915:07
jaosoriorthrash: this is the commit to project-config https://review.openstack.org/#/c/396062/15:07
thrashjaosorior: can I just move the zaqar and mistral stuff in that patch?15:08
jpichflorianf: Updating the Mistral configuration fixed this, I'll update the bug and possibly propose a patch15:08
thrashjaosorior: or if you want to do it. Would be easier if it was done in one fell swoop.15:09
jpichflorianf: I still think there's something funky if folks never hit anything similar when deploying with a bunch of environments with the CLI15:09
*** lblanchard has joined #tripleo15:09
thrashjaosorior: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1569-L1570 and https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1579-L158015:10
jaosoriorthrash: you can do it if you want to15:10
thrashjaosorior: ok. i'll hijack it. :)15:10
*** numans has quit IRC15:10
*** yolanda has quit IRC15:11
*** rbrady is now known as rbrady-brb15:12
openstackgerritKaterina Pilatova proposed openstack/tripleo-validations: callback_plugins/validation_output.py: improved output  https://review.openstack.org/39613515:12
jaosoriorccamacho: could you check out these two? https://review.openstack.org/#/c/392090/ https://review.openstack.org/#/c/392095/15:13
thrashjaosorior: updated.15:14
jaosoriorthrash: commented15:16
*** yamahata has joined #tripleo15:17
thrashjaosorior: ack15:17
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Normalize civetweb binding address if IPv6  https://review.openstack.org/39578515:17
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Select bootstrap node by list index not name  https://review.openstack.org/39628015:19
florianfjpich: yeah, that's strange. I would have expected it to be reproducable with other files as well. but maybe I just didn't try enough...15:19
jpichflorianf: I might have an idea why it doesn't reproduce from the CLI, it looks like all the environments get bunched up together in one big "user-environment.yaml" file15:20
*** jaosorior has quit IRC15:20
*** skramaja has joined #tripleo15:21
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: Support whole disk images in TripleO  https://review.openstack.org/39442615:22
d0ugaljpich: yeah, that will likely hide it15:22
openstackgerritKaterina Pilatova proposed openstack/tripleo-validations: callback_plugins/validation_output.py: improved output  https://review.openstack.org/39613515:24
*** ealcaniz has quit IRC15:24
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario004  https://review.openstack.org/36502615:24
openstackgerritBrent Eagles proposed openstack/os-net-config: Add support for name replacement in OVS_EXTRA  https://review.openstack.org/39579515:24
florianfjpich, d0ugal: Do we mind that cli and ui do this differently? Just asking because in general we want to keep the differences minimal...15:24
*** yolanda has joined #tripleo15:26
jpichflorianf: Personally I like the UI way better because it's easier to see what envs are included just by peeking into Mistral, but I don't know why the CLI is doing it this way and/or how painful it would be to do it differently. Ideally they'd be more similar though, as you say :)15:26
gbarros#tripleo, having some issues with a deployment after adding a few extraconfig, see this error: http://pastebin.com/b8077rzQ15:26
d0ugalflorianf, jpich - the CLI only does this for backwards compat issues15:27
florianfd0ugal: which means, we shouldn't touch it?15:27
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Fix typo in Keystone Sensu subscription  https://review.openstack.org/39482415:27
paramiteGuys, can we get the following patch merged, so I can backport it to stable/newton (and have that merged also)? ^15:27
d0ugalflorianf: shouldn't touch the CLI? :)15:28
d0ugalflorianf: it means we want to do it the same way as the GUI probably, but can't at the moment15:28
florianfd0ugal: :-D15:28
trowngbarros: that looks like the extraconfig has syntax issues, could you post the environment with that too?15:28
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Mistral to scenario004  https://review.openstack.org/36880515:28
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: WIP: Add fail_mode workaround for connectivity issues on reboot  https://review.openstack.org/39628515:28
florianfd0ugal: OK, that's what I wanted to know15:28
florianf:-)15:28
d0ugalflorianf, jpich - I can share you the nitty gritty details if  you like ;)15:28
florianfd0ugal: Sure!15:29
jpichd0ugal: I'm kinda curious :)15:29
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: WIP: Add fail_mode workaround for connectivity issues on reboot  https://review.openstack.org/39628515:30
gbarrostrown, here is the env. The new section is NovaComputeExtraConfig http://pastebin.com/FquyXRWk15:30
d0ugalflorianf, jpich - haha, okay, give me a few mins15:31
*** jerrygb has joined #tripleo15:31
*** jerrygb has quit IRC15:31
jpichd0ugal, florianf: "Long, long ago, when TripleO was but a little owlet..."15:31
chemany core for validating https://review.openstack.org/#/c/395053/ ?15:31
*** jerrygb has joined #tripleo15:32
*** abregman has quit IRC15:33
trowngbarros: line 61/62 look suspicious there15:34
gbarrostrown, i was thinking the same thing15:35
*** b00tcat has joined #tripleo15:36
pandaany network guys here ?15:36
*** abehl has quit IRC15:37
*** sudipto_ has quit IRC15:37
*** sudipto has quit IRC15:37
pandaI wonder if all the bad VRRP packets here are normal http://logs.openstack.org/92/388592/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/2959df7/logs/overcloud-controller-0/var/log/messages15:37
d0ugaljpich, florianf: Okay, so with the CLI deploy command you can do something like this "openstack overcloud deploy --templates -e myenv.yaml"15:37
d0ugaljpich, florianf: that would use the default templates with a custom env. simple enough.15:37
tzumainnd0ugal, use the default templates without the need for a 'plan'?15:38
d0ugaltzumainn: this is historical information only :) but yeah15:38
tzumainnI like stories too, DAD15:39
*** jerrygb has quit IRC15:39
d0ugaljpich, florianf: However, the custom env can include relative file paths like this: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/net-bond-with-vlans.yaml15:39
openstackgerritJulie Pichon proposed openstack/instack-undercloud: Increase Mistral Task Size limit  https://review.openstack.org/34821615:39
d0ugaljpich, florianf: so one env could reference files anywhere locally - for this to continue to work, we need to use heatclient to discover all the referenced files but then we end up with a merged environment15:39
*** panda has quit IRC15:40
d0ugaljpich, florianf: hopefully that makes sense15:40
d0ugaltzumainn: then go read a book :P15:40
*** panda has joined #tripleo15:40
tzumainnlol15:40
tzumainnd0ugal, wait, just to confirm - openstack overcloud deploy *right now* doesn't require a mistral plan to exist?15:41
d0ugaltzumainn: no, it does15:41
d0ugaltzumainn: every deploy uses plans15:41
tzumainnokay, so that's what you meant by historical15:41
tzumainnI GOT IT15:41
d0ugaltzumainn: but the CLI creates plans on the fly15:41
d0ugaltzumainn: to maintain the existing deploy command15:41
tzumainnd0ugal, ah, so if the plan doesn't exist, it creates it15:41
d0ugaltzumainn: Yeah, or updates it15:41
florianfd0ugal: thanks! it does15:42
*** skramaja has quit IRC15:42
*** ayoung has joined #tripleo15:43
jtomasekd0ugal: question is, do we want to support this or deprecate it and get rid of that functionality?15:44
jpichd0ugal: Thank you! Want to add this as a historical note on that bug too, so folks understand why the behaviour is different? 0=)15:44
jtomasekshardy, d0ugal, akrivoka: see last comment here https://bugs.launchpad.net/tripleo/+bug/1626955 is it crazy idea?15:45
openstackLaunchpad bug 1626955 in tripleo "No way to override roles_data.yaml" [High,In progress] - Assigned to Steven Hardy (shardy)15:45
d0ugaljtomasek: we want to deprecate it15:45
jtomasekok15:45
d0ugaljpich: wich bug?15:45
jpichd0ugal: https://bugs.launchpad.net/tripleo/+bug/1640815 , I note the difference between UI and CLI in comment #315:45
openstackLaunchpad bug 1640815 in tripleo "Selecting too many environments breaks Mistral (?)" [High,In progress] - Assigned to Julie Pichon (jpichon)15:45
d0ugaljpich: sure, I can add something15:46
d0ugaljpich: actually, we need a CLI bug for this, maybe I'll open one and that can be referenced?15:46
d0ugaljtomasek: it doesn't seem that crazy to me :)15:47
jpichd0ugal: Sounds good :)15:47
shardyjtomasek: yes we could store the data in the mistral environment, but IMO we don't need a parameter because there's $RoleServices parameters already for defining the list of services15:47
jpichI might move the bug I just mentioned to Critical though, as it's impossible to deploy anything with more than 2/3 environments without it...15:48
shardyjtomasek: the main advantage of keeping the sample roles_data in tripleo-heat-templates is it's obvious where to update it when there's a new service added15:48
shardyso personally I favor keeping it there, even if we copy the data into the mistral environment during the plan creation15:48
jtomasekshardy: ok, yeah, that answers the services my question15:48
jtomasekshardy: yeah, the proposal is just about renaming that file to plan.yaml and include more information there, not just roles stuff and use this file to populate mistral environment when plan is created15:49
jtomasekshardy: so the file would be part of the repository but not part of plan (stored in swift) since it's content would be in mistral environment15:50
slagleakrivoka: jtomasek jrist: can you comment on https://review.openstack.org/#/c/391799/ ?15:50
*** ayoung has quit IRC15:50
slaglethere are some questions there about whether bumping the patch version for tripleo-ui is appropriate15:50
slagleor if we should bump the minor version instead15:50
jristslagle: looking15:50
jristwell see UI is in a tough position15:51
jristwe had to scramble to get our patches up15:52
trozetdprince, slagle: can you please review https://review.openstack.org/#/c/394640/ ?15:53
dprincetrozet: +2 +A15:54
b00tcatin the latest releases of TripleO, let's assume that I want to add a custom service to a node that is being deployed - how would I do this?15:54
trozetdprince: ty.  Also this one if you have time https://review.openstack.org/#/c/394651/15:54
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart: Add support of deploy on a VM w/o nested virt  https://review.openstack.org/39372015:54
jtomasekslagle: replied, jrist: I cherrypicked/backported only bugfix patches, not all of them have bug referenced, sorry about that15:55
jristthanks jtomasek15:55
shardyjtomasek: that sounds reasonable, what sort of extra data are you expecting to put in there?15:55
*** bogdando has joined #tripleo15:55
shardyjtomasek: I'd suggest maybe we rename the file in t-h-t to default_plan.yaml or something15:56
b00tcatI see that all the services are listed in overcloud-resource-registry-puppet.j2.yaml15:56
shardyjtomasek: the name doesn't really matter provided the mistral action reads the content and passes it to jinja215:56
*** Guest40216 has quit IRC15:56
shardyb00tcat: you can just copy the list of default services from overcloud-resource-registry-puppet.j2.yaml for the role you want to customize, and add your new service15:57
shardye.g:15:57
jtomasekshadower: yep, the name of that file needs to be standardized that is all. So in theory, that file could include roles_data.yaml, capabilities-map.yaml and potentially what mistral environment can include15:58
jtomasekshardy: ^15:58
*** mcornea has quit IRC15:59
shardyb00tcat: http://paste.openstack.org/show/588760/15:59
shardyhttp://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html15:59
shardyalso see that blog post (sorry tripleo-docs still todo)15:59
b00tcatthanks shardy - if I have the time I might even contribute to the docs myself :D16:00
shardyb00tcat: feel free! :)16:00
jtomasekshardy: exporting the plan would then be a matter of providing tar which includes swift container contents + plan.yaml which would include contents of mistral environment16:00
shardyb00tcat: if you do start working on them please let me know then I'll know not to do it myself16:00
shardyI've been meaning to do them for a few weeks16:01
shardyjtomasek: ack, that would be fine I think16:01
shardyyou could just write the roles_data.yaml data into plan.yaml regardless of any renaming16:01
b00tcatsure!16:02
beaglesgeneral heads up... I'm getting a lot of checksum errors with ssh sessions between undercloud and overcloud guests16:02
*** rajinir has joined #tripleo16:02
*** BLOODBATH has joined #tripleo16:02
*** BLOODBATH is now known as Guest1377316:03
beagleslet me know if anybody else runs into this kind of thing .. if for no other reason than to let me know I'm not going mad :)16:03
*** ebarrera has quit IRC16:03
*** jkilpatr_ has joined #tripleo16:03
beaglesI haven't been able to work out the cause yet.... symptoms would include really long time to ssh from one node to the other16:04
*** ayoung has joined #tripleo16:04
jtomasekshardy: on export, yes. having it all in one file in tht would probably simplify tripleo-common as it would just look for single file and populate the mistral environment with it when plan is created. Rather than looking separately for 3 files. Also maybe it would be simpler to the user as he'd know that all 'plan metadata' are in plan.yaml16:04
beagles(particulars, virtual environment with virtio drivers only)16:05
jtomasek(by 3 files I mean roles_data.yaml, capabilities-map.yaml and mistral-environment.yaml - which would be needed to introduce for exporting plan)16:05
jtomasekshardy: ^16:06
shardyjtomasek: ack, but I think we'll need to support roles_data.yaml for backwards compatibility anyway, so we can just look for plan.yaml (or that content in the mistral environment), then fall back to reading roles_data.yaml16:06
jtomasekshardy: oh, right. should I create spec for it or blueprint is enough?16:07
shardyjtomasek: I think a blueprint is enough16:07
jtomasekshardy: ok, I'll create one, thanks16:07
jtomasekd0ugal: actually this blueprint might fit in your plans management spec, right?16:08
*** links has joined #tripleo16:09
gbarrostrown that wasnt it, same error16:09
d0ugaljtomasek: Yeah, maybe. I am probably going to do a blueprint btw16:09
jtomasekd0ugal: ok16:09
*** udesale_ has quit IRC16:11
*** rcernin has quit IRC16:11
*** apetrich has quit IRC16:15
*** iranzo has quit IRC16:15
*** apetrich has joined #tripleo16:16
*** [1]cdearborn has joined #tripleo16:18
*** ramishra has quit IRC16:18
*** ramishra has joined #tripleo16:20
*** chandankumar has joined #tripleo16:21
*** nyechiel has quit IRC16:21
*** tremble has quit IRC16:22
openstackgerritMerged openstack/tripleo-validations: callback_plugins/validation_output.py: improved output  https://review.openstack.org/39613516:24
dhill_hi guys16:25
*** dsneddon has joined #tripleo16:25
*** chandankumar has quit IRC16:31
*** rcernin has joined #tripleo16:32
*** links has quit IRC16:37
ansiwenshardy, mwhahaha, chem: when you have a calm minute, you might look at https://review.openstack.org/#/c/336562/   I'd love to move forward with this neverending story16:37
*** mhenkel has joined #tripleo16:39
chemansiwen: this is working for you rigth ?16:39
*** TSCHAK has joined #tripleo16:41
shardyansiwen: is there a corresponding t-h-t patch which consumes the profile?16:41
*** rbrady-brb is now known as rbrady16:44
ansiwenshardy: well, that will follow16:45
ansiwenchem: I have no idea yet, there is no consumer.16:46
chemgfidente: for the blockstorage stuff, I tried those two environments http://paste.openstack.org/show/588774/ to disable ceph and enable iscsi but nothing get passed down in the hieradata : I just re-run the convergence step with the env added16:46
ansiwenshardy: so, should I make the THT patch as well and make it depending on this?16:46
chemgfidente: if you don't have anything in mind regarding this, I'm just going to remove the blockstorage from my test env for now (as it may not be a requirement)16:47
chemansiwen: ack16:47
gfidentechem you don't see them in hieradata at all?16:47
ansiwenchem, mwhahaha: thanks for the review. I just wanted to make sure you guys don't see an obvious mistake, and that my general approach is correct.16:48
chemgfidente: nope nothing move in this directory16:48
*** chandankumar has joined #tripleo16:48
chemgfidente: the /etc/puppet/hieradata/extraconfig.yaml desperately stays empty16:49
*** ccamacho has quit IRC16:51
chemgfidente: so this doesn't work  BlockStorageExtraConfig: {CinderEnableIscsiBackend: true, CinderEnableRbdBackend: false} (in openstack stack environment show overcloud)16:51
gfidentechem the file should be named16:52
gfidentevolume_extraconfig16:52
chemgfidente: ah, yes, sorry16:52
gfidentebut you can only pass the puppet params via extraconfig16:53
gfidentenot the template params16:53
chemgfidente: oki, I've just got success :) You have to use the puppet parameter directly16:53
*** abehl has joined #tripleo16:53
gfidentechem yep16:54
chemgfidente: in the paste this is the first version.16:54
chemgfidente: oki, parameters are there, waiting for the converge to happen and then I'll add it as a workaround.16:54
chemgfidente: thanks a lot.16:55
gfidentechem cool, ty16:55
chemgfidente: oki, step4 -> enabled_backends = tripleo_iscsi,16:56
*** ohamada has quit IRC16:56
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: POC: Use tripleo-repos in tripleo.sh  https://review.openstack.org/39581316:57
gfidentechem cool16:57
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Always use cached images in updates job  https://review.openstack.org/38671316:59
*** abehl has quit IRC17:02
*** hewbrocca is now known as hewbrocca_afk17:03
*** aufi has quit IRC17:06
*** dhill_ has quit IRC17:06
*** jpich has quit IRC17:06
*** dtantsur is now known as dtantsur|afk17:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Fix inconsistent Manila service naming  https://review.openstack.org/39631717:07
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Always use cached images in updates job  https://review.openstack.org/38671317:08
*** ccamacho has joined #tripleo17:08
*** chandankumar has quit IRC17:10
*** dhill_ has joined #tripleo17:11
bnemecOuch, zuul restart.  rh1 is about to have a bad day.17:11
sshnaidmHave anybody seen error in Ci jobs like: Failed to call refresh: Could not restart Service[httpd] Execution of '/usr/bin/systemctl restart httpd' returned 1: Job for httpd.service failed because a timeout was exceeded17:12
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Normalize civetweb binding address if IPv6  https://review.openstack.org/39578517:12
slaglebnemec: do we need to do anything in response to the restart?17:13
slagleclear out all jobs/envs or anything?17:13
bnemecslagle: I don't think so, but everything that was running will need to be rechecked.17:13
bnemecIIRC anyway.17:13
slagleyea ok, that's what i thought as well17:13
slaglewe'll have to keep an eye on it to just make sure new stuff is getting scheduled17:13
bnemecI'm wondering if this would be a good time to merge https://review.openstack.org/#/c/384731/17:14
shardyansiwen: yes, then you'll have a way to test the profile17:14
shardytesting it without a tripleo patch will be pretty inconvenient I think17:14
trozetmwhahaha: when you have a minute: https://review.openstack.org/#/c/396318/17:14
*** paramite has quit IRC17:15
*** abehl has joined #tripleo17:15
mwhahahatrozet: sure17:15
trozetmwhahaha: thanks man17:15
*** _milan_ has joined #tripleo17:19
pandadsneddon: I wonder if all the bad VRRP packets here are normal http://logs.openstack.org/92/388592/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/2959df7/logs/overcloud-controller-0/var/log/messages17:19
dsneddonpanda, That doesn't look normal to me. But 192.0.2.6 is a strange address to see there. Did you modify the 192.0.2.2/192.0.2.3 defaults in undercloud.conf?17:20
dsneddonpanda, Because 192.0.2.6 conflicts with the normal DHCP range of 192.0.2.5-2417:21
*** milan has quit IRC17:22
dsneddonpanda, Oh wait, I think I'm misreading these logs. These are from the Overcloud controller?17:22
pandadsneddon: yes17:22
dsneddonpanda, Not sure what is going on there, but those aren't normal log messages.17:23
pandadsneddon: I'll see how often we get those17:23
dsneddonpanda, Is 192.0.2.6 the address of the Control Plane VIP?17:24
pandabnemec: this finally passed on all gates https://review.openstack.org/38251517:24
*** jlinkes has quit IRC17:24
dsneddonpanda, You can run "neutron port-list" against the undercloud to figure out which IPs are assigned on that network.17:24
bnemecpanda: Yes, but I still don't want it. :-)17:24
pandabnemec: ...17:24
derekhbnemec: would this be a good time to test drive https://review.openstack.org/#/c/384731/ ?17:25
derekhgiven the restart and all17:25
bnemecderekh: <bnemec> I'm wondering if this would be a good time to merge https://review.openstack.org/#/c/384731/17:25
trozetmwhahaha: fixed the whitespace17:25
bnemecI was thinking the same thing.17:25
pandadsneddon: It's a bit hard to reproduce in a live environment, but I can try17:26
dsneddonpanda, I'm just trying to interpret the log messages, and it would help if I knew what 192.0.2.6 was.17:26
ansiwenshardy: so, this is my first "optional" feature, so I will not hardwire it into the controller node role. can you point me to a minimal as possible example of such an optional service?17:27
dsneddonpanda, If that's an IP of a controller, that would be different than if it was the IP of the VIP.17:27
pandabnemec: can you answer to this then stating you don't want this to be merged ? http://lists.openstack.org/pipermail/openstack-dev/2016-October/105934.html17:28
shardyansiwen: Nothing is hardwired into the controller role since back in Newton17:28
shardyansiwen: https://github.com/openstack/tripleo-heat-templates/tree/master/puppet/services17:28
shardythere are many examples in there17:28
ansiwenshardy: thanks!17:28
shardyall follow basically the same pattern, so you can pick any of them17:28
bnemecpanda: I left an early review on https://review.openstack.org/#/c/382515/ explaining why I don't like it.17:29
shardyansiwen: also see http://docs.openstack.org/developer/tripleo-docs/developer/tht_walkthrough/tht_walkthrough.html17:29
shardyand17:29
*** dsariel has quit IRC17:29
ansiwenshardy: oh, this I know of course... but I have to reference the yaml file somehow, so that it get's tested17:29
shardyhttp://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html17:29
shardyansiwen: yes, you'll add it to overcloud-resource-registry-puppet.j2.yaml17:30
ansiwenshardy: I implemented the composable service for nova-compute already... but as I remember I inserted that into the compute role17:30
ansiwenshardy: ok, so everything there will be picked up for testing?17:30
shardyansiwen: yeah if it's disabled by default you'll add it to overcloud-resource-registry-puppet.j2.yaml and not to the roles_data.yaml17:31
shardywe'll need to add it to a CI scenario17:31
ansiwenshardy: got it, thanks!17:31
shardybut first step it get it working locally17:31
*** _milan_ has quit IRC17:32
pandadsneddon: it's the controller VIP17:32
ansiwenshardy: "locally", well, on my 64GB tarox I guess ;-)17:32
ansiwenshardy: can I test this without a full tripleo deployment?17:32
*** ccamacho has quit IRC17:34
*** katkapilatova has left #tripleo17:35
pandabnemec: yeah, I partially agreed with you, but nobody answered, that's why I created that post, hoping to start a discussion, detailing what I was trying to do. What else I can do now ?17:35
*** akrivoka has quit IRC17:36
*** ccamacho has joined #tripleo17:36
pandaI will probably take another month before planets align correctly and I get green on all the gates.17:36
bnemecpanda: I want to push to get the updates job back in the gate.17:37
bnemecWe need it anyway, and it solves the ipv6 coverage problem too.17:37
bnemecWith only two jobs the best we can do is trade one CI hole for another.17:37
bnemecslagle: Would you be okay merging https://review.openstack.org/#/c/384731/ to keep the flood of rechecks under control?17:38
openstackgerritDan Prince proposed openstack/tripleo-puppet-elements: Disable old hiera.yaml o-a-c script  https://review.openstack.org/39261517:39
slaglebnemec: i hesitate a bit...to make any changes. has it been tested in any live environment?17:40
*** hjensas has quit IRC17:40
*** hjensas has joined #tripleo17:41
*** lucasagomes is now known as lucas-afk17:41
sshnaidmfyi https://bugs.launchpad.net/tripleo/+bug/164087917:42
openstackLaunchpad bug 1640879 in tripleo "CI: apache fails to start on overcloud controller" [Undecided,New]17:42
*** pkovar has quit IRC17:43
bnemecslagle: There's not really any way to test it live without deploying it to all the te workers on rh1.  derekh and I have both tested the semaphore behavior standalone though.17:44
*** trown is now known as trown|lunch17:45
*** owalsh has quit IRC17:46
hrybackihey all. I'm seeing an 'Enviornment not found <stack> ' when calling `openstack overcloud node delete --stack <stack>' Verified that the stack id is correct -- any pointers? https://paste.fedoraproject.org/477444/79992414/17:48
hrybackiNewton ^^17:48
slaglebnemec: so posix_ipc.Semaphore will block indefinitely?17:51
slagleor until we hit the job timeout i guess17:52
*** derekh has quit IRC17:55
bnemecslagle: Yes: "A timeout of None (the default) implies no time limit. The call will not return until its wait condition is satisfied. "17:57
bnemecFrom the acquire docstring: http://semanchuk.com/philip/posix_ipc/#semaphore17:57
*** morazi has quit IRC17:58
*** thrash is now known as thrash|f00dz18:00
openstackgerritMerged openstack-infra/tripleo-ci: Redirect image build stderr to stdout  https://review.openstack.org/38264118:01
openstackgerritMerged openstack/python-tripleoclient: Reset the parameter_defaults between deployments via the CLI  https://review.openstack.org/39441918:04
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Refactor creation of no_proxy list for postconfig  https://review.openstack.org/39119718:04
*** chandankumar has joined #tripleo18:06
openstackgerritMerged openstack/tripleo-heat-templates: Ensure heat-domain hiera is in nodes that contain keystone  https://review.openstack.org/39251918:07
openstackgerritMerged openstack/tripleo-heat-templates: Fixes incorrect reference to OpendaylightApiNetwork  https://review.openstack.org/39464018:08
*** morazi has joined #tripleo18:10
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Add unit tests for bracket_ipv6  https://review.openstack.org/39634518:12
*** chandankumar has quit IRC18:13
*** numans has joined #tripleo18:13
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Force the use of python version 2.7  https://review.openstack.org/36801318:14
*** jpena is now known as jpena|off18:14
cmyster.918:14
*** rasca has quit IRC18:14
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: POC: Use tripleo-repos in tripleo.sh  https://review.openstack.org/39581318:15
shardyansiwen: I'm not sure I understand, you have a 64G box, why not install a tripleo VM environment and test it on that?18:21
shardythat's what most of us are doing18:21
* shardy only has 32G tho ;(18:21
shardyI don't think you can really develop tripleo profiles or composable services without testing them locally on tripleo18:21
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates: Fixes incorrect reference to OpendaylightApiNetwork  https://review.openstack.org/39634818:22
shardywell, you could, but it will take forever18:22
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Add configuration of HA IPv6 deployments  https://review.openstack.org/38035818:22
trozetdprince, shardy: ^^ cherry pick to stable/newton18:23
*** chandankumar has joined #tripleo18:25
*** chandankumar has quit IRC18:26
*** panda_ has joined #tripleo18:26
*** dsariel has joined #tripleo18:27
*** panda has quit IRC18:29
*** chandankumar has joined #tripleo18:30
*** cdearborn has joined #tripleo18:35
openstackgerritMerged openstack/tripleo-heat-templates: Removes deprecated overcloud VIP outputs  https://review.openstack.org/39465118:36
openstackgerritMerged openstack/tripleo-heat-templates: set url_base option in static web middleware  https://review.openstack.org/39565418:36
*** [1]cdearborn has quit IRC18:39
*** shardy has quit IRC18:41
*** tosky has quit IRC18:43
*** chandankumar has quit IRC18:46
*** dprince has quit IRC18:48
*** trown|lunch is now known as trown18:48
*** thrash|f00dz is now known as thrash18:51
*** Guest13773 has quit IRC18:51
*** jkilpatr_ has quit IRC18:51
*** athomas has quit IRC18:52
*** jkilpatr has quit IRC18:52
*** florianf has quit IRC18:54
*** panda_ is now known as panda|bbl18:56
*** chandankumar has joined #tripleo18:58
*** amoralej is now known as amoralej|off18:58
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: [WIP] DO NOT MERGE -- Add tuned check to OOOQ  https://review.openstack.org/39636218:59
*** florianf has joined #tripleo18:59
openstackgerritMerged openstack/tripleo-heat-templates: Fix race during major-upgrade-pacemaker step  https://review.openstack.org/39545419:00
*** zoli|wfh is now known as zoli|brb19:00
*** abehl has quit IRC19:01
*** dprince has joined #tripleo19:03
*** jkilpatr_ has joined #tripleo19:04
*** Guest13773 has joined #tripleo19:05
*** jkilpatr has joined #tripleo19:05
*** fragatina has joined #tripleo19:08
*** yamahata has quit IRC19:16
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Create directories with root  https://review.openstack.org/38489219:17
gbarros#tripleo, im using '%{hiera(''keystone_admin_api_vip'')}' in one of my env's but it does not seem to expand to anything. Any pointers?19:24
*** numans has quit IRC19:24
*** chandankumar has quit IRC19:25
*** ebalduf has quit IRC19:28
*** zoli|brb is now known as zoli|wfh19:30
*** liverpooler has quit IRC19:43
*** jerrygb has joined #tripleo19:43
*** openstackgerrit has quit IRC19:48
*** openstackgerrit has joined #tripleo19:48
openstackgerritMerged openstack/instack-undercloud: Increase Mistral Task Size limit  https://review.openstack.org/34821619:49
*** bana_k has joined #tripleo19:53
*** yamahata has joined #tripleo19:55
*** owalsh has joined #tripleo19:58
openstackgerritHonza Pokorny proposed openstack/tripleo-specs: Add network diagram spec  https://review.openstack.org/39638320:01
*** cylopez has quit IRC20:04
*** zoli|wfh is now known as zoli|gone20:04
*** chem has quit IRC20:12
*** gregwork has quit IRC20:16
*** gregwork has joined #tripleo20:18
*** ccamacho1 has joined #tripleo20:18
*** ccamacho has quit IRC20:20
*** bana_k has quit IRC20:21
*** bana_k has joined #tripleo20:23
*** panda|bbl is now known as panda20:24
jidarcan you fit a HA deployment in 32 gigs?20:25
jidar(I assume yes)20:25
slagleyes, but it's tight20:35
slaglei assume you're asking about for a virt environment20:35
*** rcernin has quit IRC20:36
*** jcoufal has quit IRC20:38
*** jcoufal has joined #tripleo20:40
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Fix race during major-upgrade-pacemaker step  https://review.openstack.org/39546020:41
*** mhenkel has quit IRC20:43
*** dprince has quit IRC20:50
jidaryea, on the one box for testing/development20:50
jidarI keep meaning to put together a box at home for this but I keep having to do prod deployments so it's not as important (*sigh*)20:50
*** Guest13773 has quit IRC20:51
*** jkilpatr has quit IRC20:51
trown32G is enough if you can do KSM to overcommit a bit20:51
*** jkilpatr_ has quit IRC20:51
trownI dont think it actually fits anymore otherwise20:52
bnemecCI uses exactly 32 G worth of memory for an HA deployment right now - 4x6G overcloud nodes + 8G undercloud20:54
mwhahahawhat's the best way to get the puppet logs from a deploy?20:55
*** lblanchard has quit IRC20:55
*** jkilpatr has joined #tripleo20:55
trownya, in order to get 32G of VMs on a 32G host though, overcommit is required :)20:56
*** iranzo has joined #tripleo20:56
*** iranzo has joined #tripleo20:56
trownmwhahaha: from a failed deploy or a successful one?20:56
mwhahahayes?20:56
trownhehe... well a failed one is much eaiser `openstack stack failures list overcloud`20:57
mwhahahawell that only gives me a snippet20:57
mwhahahai want it all20:57
mwhahahawhich is why i asked20:57
mwhahaha(which should be a thing i might add)20:58
trownindeed20:58
trownI dont think it is though... you would need to look up each SoftwareDeployment resource and do resource show on it20:58
slagleit's in the os-collect-config log on each node20:59
trownthat is probably nicer :)20:59
slaglewhich is accessible via journalctl20:59
mwhahahahttps://review.openstack.org/#/c/388210/20:59
thrashmwhahaha: actually, log into the node you are interested in and 'journalctl -u os-collect-config'20:59
mwhahahano i just want the puppet logs20:59
mwhahahathat's really a terrible thing to have to do21:00
slagleyes, but you asked for the best way21:00
thrash:D21:00
mwhahahano i said the best way, not the only way :)21:01
trozetdsneddon: can you please set workflow on this https://review.openstack.org/#/c/396348/ ?21:01
mwhahahait just so happens that it's the only way right now it appears, that's totally not the best way :)21:01
*** jkilpatr has quit IRC21:03
trozetslagle, beagles: ^^^21:07
stevebakermwhahaha: `openstack stack failures list --long overcloud` should give the full puppet output21:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921521:10
mwhahahastevebaker: thanks, that's a bit better :)21:11
sshnaidmweshay, there is weird problem with periodics, they run each time on the same delorean hash and promote it21:11
stevebakermwhahaha: also /var/run/heat-config/deployed will have all puppet logs if you're running a new enough image21:11
sshnaidmweshay, sent a patch to check what is going on there..21:11
*** florianf has quit IRC21:13
weshaysshnaidm, k21:13
*** jayg is now known as jayg|g0n321:15
trownsshnaidm: that actually might be because there was a stalled "consistent" repo in RDO for a few days21:15
trownit is fixed as of today though21:15
*** ayoung has quit IRC21:15
*** jkilpatr has joined #tripleo21:16
sshnaidmtrown, oh, that's it then21:17
sshnaidmtrown, because today it has different hash21:17
*** bana_k has quit IRC21:18
trownsshnaidm: ya, rally added a new dep that needed to be packaged21:18
*** iranzo has quit IRC21:19
*** jkilpatr has quit IRC21:21
pandaso upstream periodic jobs didn't promote anything even if they were all green for the past few days ?21:22
*** Goneri has quit IRC21:22
trownthey promoted the same thing over and over :P21:24
tbarronslagle: when you have a minute, stable/newton backport https://review.openstack.org/#/c/396183/ is ready for you21:25
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Add configuration of HA IPv6 deployments  https://review.openstack.org/38035821:28
dsneddontrozet, That review looked good, I +2, +1'd it.21:29
*** ccamacho1 has quit IRC21:31
*** maticue has quit IRC21:32
*** gfidente has quit IRC21:32
*** bana_k has joined #tripleo21:37
*** zaneb has quit IRC21:39
tbarronslagle: ty21:41
slagletbarron: yw21:48
*** jkilpatr has joined #tripleo21:50
*** jkilpatr has joined #tripleo21:51
*** ccamacho has joined #tripleo21:51
*** owalsh has quit IRC21:53
*** rhallisey has quit IRC21:57
openstackgerritBen Nemec proposed openstack/puppet-tripleo: Fix puppet lint failure  https://review.openstack.org/39641921:59
*** bana_k has quit IRC22:00
*** panda has quit IRC22:01
*** panda_ has joined #tripleo22:01
*** jprovazn has quit IRC22:02
*** lmiccini has quit IRC22:02
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: POC: Use tripleo-repos in tripleo.sh  https://review.openstack.org/39581322:03
*** panda_ has quit IRC22:05
*** panda has joined #tripleo22:12
*** jcoufal has quit IRC22:15
*** trown is now known as trown|outtypewww22:15
*** aufi has joined #tripleo22:17
*** jerrygb_ has joined #tripleo22:21
mwhahahabnemec: the lint failure slipped in because puppet-tripleo was using an older puppet-lint (via puppet-openstack_spec_helper) and we just merged the version increase.22:21
mwhahahabnemec: https://review.openstack.org/#/c/353488/ ccamacho forgot to recheck puppet-tripleo :o22:22
bnemecmwhahaha: Okay, I suspected it was something like that but I don't know enough about how the puppet jobs work. :-)22:22
*** ccamacho has quit IRC22:22
*** colonwq has quit IRC22:22
*** jerrygb has quit IRC22:24
*** jerrygb has joined #tripleo22:25
*** jerrygb_ has quit IRC22:26
*** tiswanso has quit IRC22:26
*** jerrygb_ has joined #tripleo22:27
*** jerrygb has quit IRC22:30
*** jerrygb_ has quit IRC22:31
bnemecHuh, vlan-aware vms merged to neutron in newton?  Cool.22:34
*** fragatin_ has joined #tripleo22:34
* bnemec needs to rebuilt his ovb environment22:34
bnemecOr rebuild even22:34
*** fragatina has quit IRC22:37
openstackgerritMerged openstack/tripleo-heat-templates: Fixes incorrect reference to OpendaylightApiNetwork  https://review.openstack.org/39634822:45
openstackgerritMerged openstack/tripleo-heat-templates: Select bootstrap node by list index not name  https://review.openstack.org/39628022:46
openstackgerritBen Nemec proposed openstack/instack-undercloud: Nothing to see here  https://review.openstack.org/38664822:47
openstackgerritMerged openstack/tripleo-heat-templates: Add firewall rules for manila api service  https://review.openstack.org/39618322:51
*** aufi has quit IRC22:55
*** mhenkel has joined #tripleo22:59
*** limao has joined #tripleo23:01
*** fragatin_ has quit IRC23:04
*** fragatina has joined #tripleo23:05
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Fix barbican server name to not use aodh hiera  https://review.openstack.org/39643423:20
*** Alex___1234 has joined #tripleo23:22
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add panko service support  https://review.openstack.org/39643523:22
*** ayoung has joined #tripleo23:22
*** pmannidi has quit IRC23:23
Alex___1234Hi Everyone. I'm attempting to deploy an overlcoud and getting errors with no available hosts found. I have two nodes that are supposed to be used but I think they are failing introspection. Does anyone have a few minutes to help me dig deeper into this?23:24
*** pmannidi has joined #tripleo23:24
Alex___1234One thing I should mention is that all 3 parts (undercloud, controller node, and compute node) are VMs23:25
*** limao has quit IRC23:28
*** fultonj has quit IRC23:35
*** cmyster has quit IRC23:36
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add panko api support to service templates  https://review.openstack.org/39643923:37

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!