| openstackgerrit | Merged openstack/instack-undercloud: Ensure that the default plan was created successfully https://review.openstack.org/373446 | 00:04 |
|---|---|---|
| *** rajinir has quit IRC | 00:05 | |
| *** gfidente is now known as gfidente|afk | 00:05 | |
| *** yamahata has quit IRC | 00:09 | |
| *** limao has joined #tripleo | 00:23 | |
| *** rbrady has quit IRC | 01:09 | |
| dtrainor | I have a stuck deployment. There's nothing happening on either of my one Control or one Compute Overcloud instances. No obvious errors. It's just... stuck. They're stuck on ComputeNodesPostDeployment and ControllerNodesPostDeployment, respectively. I've exhausted my troubleshooting options. | 01:12 |
| *** dmacpher-afk is now known as dmacpher | 01:21 | |
| *** dprince has joined #tripleo | 01:22 | |
| *** dprince has quit IRC | 01:22 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 01:22 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Add swift proxy for ceilometer middleware https://review.openstack.org/371591 | 01:24 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 01:26 |
| *** bana_k has quit IRC | 01:33 | |
| *** bfournie has joined #tripleo | 01:55 | |
| *** kjw3 has joined #tripleo | 02:04 | |
| *** kjw3 has quit IRC | 02:19 | |
| *** thrash is now known as thrash|g0ne | 02:35 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci https://review.openstack.org/374931 | 02:47 |
| *** rajinir has joined #tripleo | 02:56 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Remove the get_hiera_key function https://review.openstack.org/367367 | 03:02 |
| openstackgerrit | Merged openstack/tripleo-common: Updated from global requirements https://review.openstack.org/373722 | 03:02 |
| openstackgerrit | Merged openstack/instack-undercloud: Update default VM memory https://review.openstack.org/375054 | 03:02 |
| *** david-lyle has quit IRC | 03:04 | |
| *** rbrady has joined #tripleo | 03:05 | |
| *** ayoung has joined #tripleo | 03:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Undercloud install with tripleo-quickstart https://review.openstack.org/358919 | 03:26 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:30 |
| *** fultonj has quit IRC | 03:36 | |
| *** jrist has joined #tripleo | 03:59 | |
| *** pgadiya has joined #tripleo | 04:06 | |
| *** coolsvap has joined #tripleo | 04:08 | |
| *** limao has quit IRC | 04:26 | |
| *** limao has joined #tripleo | 04:28 | |
| *** limao has quit IRC | 04:31 | |
| *** limao has joined #tripleo | 04:34 | |
| *** rajinir has quit IRC | 05:05 | |
| *** jaosorior has joined #tripleo | 05:10 | |
| *** bana_k has joined #tripleo | 05:12 | |
| *** ebalduf has quit IRC | 05:14 | |
| *** limao has quit IRC | 05:32 | |
| *** limao has joined #tripleo | 05:37 | |
| *** absubram has quit IRC | 05:47 | |
| bandini | morning | 05:54 |
| jaosorior | bandini: hey dude! how's it going? | 05:56 |
| bandini | jaosorior: hola! it's friday, nothing can bring me down ;) how about you? | 05:59 |
| jaosorior | haha all good here | 05:59 |
| jaosorior | coffee coffee coffee | 05:59 |
| bandini | yay! | 05:59 |
| matbu | bandini: even another new upgrade blocker bug ? :) | 06:00 |
| bandini | matbu: ahaha I collect those like folks collect pokemons ;) | 06:02 |
| bandini | matbu: https://bugs.launchpad.net/tripleo/+bug/1626628 this is my last one from last night btw | 06:02 |
| openstack | Launchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,New] | 06:02 |
| *** ianw is now known as ianw_pto | 06:02 | |
| matbu | bandini: lol | 06:03 |
| matbu | bandini: hmm progress now you hit pcs issue :) | 06:04 |
| bandini | matbu: i think it is heat. we never get to pcs commands | 06:08 |
| *** yamahata has joined #tripleo | 06:09 | |
| bandini | I can reproduce it 100% it seems | 06:12 |
| *** mbozhenko has joined #tripleo | 06:14 | |
| *** rcernin has joined #tripleo | 06:15 | |
| *** rasca has joined #tripleo | 06:15 | |
| *** saneax-_-|AFK is now known as saneax | 06:25 | |
| *** tobias_fiberdata has joined #tripleo | 06:27 | |
| *** bana_k has quit IRC | 06:31 | |
| marios | matbu: bandini jaosorior o/ | 06:34 |
| marios | we made it to friday, now just keep calm and it will finish soon enough | 06:34 |
| * marios cries | 06:34 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize https://review.openstack.org/370114 | 06:37 |
| *** aufi has joined #tripleo | 06:39 | |
| matbu | marios: o/ | 06:40 |
| *** florianf has joined #tripleo | 06:41 | |
| matbu | marios: don't cries, it's a light week for you next week ? :) | 06:41 |
| jaosorior | hahaha | 06:42 |
| jaosorior | marios: coffee will make it all better dude | 06:42 |
| *** tremble has joined #tripleo | 06:42 | |
| marios | matbu: :( i think that makes it worse. i mean there is so much stuff going on my head hurts | 06:43 |
| *** jprovazn has joined #tripleo | 06:43 | |
| openstackgerrit | Merged openstack/diskimage-builder: Allow ramdisk-create to run without $USER set https://review.openstack.org/357985 | 06:46 |
| *** cylopez has joined #tripleo | 06:47 | |
| *** mcornea has joined #tripleo | 06:52 | |
| *** jlinkes has joined #tripleo | 06:57 | |
| *** tzumainn has quit IRC | 07:05 | |
| *** dsariel has joined #tripleo | 07:09 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 07:12 |
| *** gfidente|afk is now known as gfidente | 07:13 | |
| gfidente | matbu, curiosity, have you attempted upgrade with ceph again? | 07:14 |
| matbu | gfidente: nop not yet | 07:15 |
| matbu | gfidente: i think chem (not online) does | 07:15 |
| *** panda|zZ is now known as panda | 07:16 | |
| gfidente | it looks like the version of the heatclient we have | 07:16 |
| gfidente | we install | 07:16 |
| matbu | gfidente: you mean regarding this issue https://bugs.launchpad.net/tripleo/+bug/1623506 ? | 07:18 |
| openstack | Launchpad bug 1623506 in tripleo "Unknown Property batch_create in SoftwareDeploymentGroup for CephMon upgrade" [Undecided,Fix released] - Assigned to Giulio Fidente (gfidente) | 07:18 |
| *** jbadiapa has joined #tripleo | 07:18 | |
| gfidente | so that should be sorted now | 07:18 |
| gfidente | but now I needed https://github.com/openstack/python-heatclient/commit/da0da4c1afc00dd2732b678b09fc2d826137a44d | 07:19 |
| gfidente | and it's not there | 07:20 |
| marios | gfidente: good morning man. thanks very much for looking at puppet-manila I just couldn't get to it yesterday was planning on digging today (not sure when exactly) | 07:21 |
| gfidente | marios, I think I'll test if that change fixes it or not | 07:21 |
| gfidente | as it might not be the root cause | 07:22 |
| marios | gfidente: sure, i know tbarron was also looking, hopefully he will get a run in today too | 07:22 |
| gfidente | though I saw db sync failing on the controller so I suppose it was just trying to run it before the db was created | 07:22 |
| jaosorior | marios, gfidente if you have time can you check this commit out? https://review.openstack.org/#/c/365585/ | 07:22 |
| marios | jaosorior: ack | 07:22 |
| gfidente | jaosorior, that's cool | 07:24 |
| gfidente | but is it setting the servername to which the virthost responds to? | 07:24 |
| *** flepied has quit IRC | 07:24 | |
| gfidente | I think that if haproxy is calling it by ip we need to add serveralias too? | 07:25 |
| jaosorior | gfidente: so, at the moment haproxy is using the IP to point to that server | 07:25 |
| jaosorior | gfidente: so, right now it will route with the IP and port that the virtualhost is using | 07:26 |
| gfidente | jaosorior, so if I remember correctly httpd will see if string matches the virtualhost name and if it doesn't, it will serve the first virtualhost | 07:27 |
| jaosorior | if there are collisions in the IP, then it will route using the servername | 07:27 |
| gfidente | yeah so don't we need to use the IP as servername/serveralias? | 07:27 |
| jaosorior | gfidente: https://httpd.apache.org/docs/2.4/vhosts/name-based.html | 07:27 |
| jaosorior | "It is important to recognize that the first step in name-based virtual host resolution is IP-based resolution." | 07:28 |
| jaosorior | gfidente: not really. we already have the IP set | 07:28 |
| gfidente | ah the virtualhosts are set as IP:name not *:name | 07:28 |
| jaosorior | indeed | 07:29 |
| gfidente | ack, thanks | 07:29 |
| *** zoli_gone-proxy is now known as zoliXXL | 07:29 | |
| jaosorior | gfidente: it is only when we want to access the server via the hostname that we run into trouble | 07:29 |
| jaosorior | which is what that fix wants to address | 07:29 |
| jaosorior | gfidente: thanks for checking it out dude :D | 07:29 |
| zoliXXL | good morning | 07:30 |
| gfidente | so do you guys have any idea why on a fresh tripleo.sh | 07:30 |
| gfidente | I don't get a change for heatclient dated 25th of aug? | 07:30 |
| jaosorior | whaa | 07:30 |
| jaosorior | gfidente: maybe it's cause heat is not gotten from delorean-current | 07:31 |
| jaosorior | AFAIK | 07:31 |
| gfidente | yeah I was looking into that | 07:31 |
| gfidente | we don't install clients from -current | 07:31 |
| jaosorior | nope | 07:31 |
| jaosorior | should we? | 07:31 |
| gfidente | not sure, we don't get heat from -current either apparently | 07:32 |
| gfidente | but still does that mean delorean did not promote these from a month now? | 07:32 |
| *** jpena|off is now known as jpena | 07:33 | |
| jaosorior | gfidente: I have no idea | 07:33 |
| jaosorior | gfidente: would need to ask in #rdo I guess | 07:33 |
| gfidente | yep going :) | 07:34 |
| *** ebarrera has joined #tripleo | 07:35 | |
| *** jpich has joined #tripleo | 07:37 | |
| gfidente | jaosorior, apparently heatclient is pinned to some old tag | 07:39 |
| gfidente | see #rdo | 07:39 |
| jaosorior | gfidente: oh :/ | 07:43 |
| *** yolanda has quit IRC | 07:44 | |
| *** ohamada has joined #tripleo | 07:47 | |
| *** yolanda has joined #tripleo | 07:47 | |
| *** shardy has joined #tripleo | 07:49 | |
| *** hjensas has joined #tripleo | 07:50 | |
| jpich | All the client patches are failing with a UT failure - is anyone looking into this? I don't see any open bug | 07:53 |
| * jpich looking | 07:53 | |
| *** athomas has quit IRC | 07:55 | |
| * jpich filed https://bugs.launchpad.net/tripleo/+bug/1626885 | 07:56 | |
| openstack | Launchpad bug 1626885 in tripleo "Client unit tests failing: "TypeError: 'Mock' object is not iterable"" [Critical,Triaged] - Assigned to Julie Pichon (jpichon) | 07:56 |
| matbu | jpich: hey thanks | 07:57 |
| matbu | jpich: i looked briefly yesterday | 07:57 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Stacks and Resources data storing in app state https://review.openstack.org/374227 | 07:57 |
| matbu | jpich: cause one of my review was hitting this failure | 07:57 |
| jpich | matbu: Yeah, that's what got me started too :) | 07:58 |
| jpich | matbu: Did you find anything? I can reproduce after recreating my tox env, that's about as far as I got | 07:58 |
| jpich | If you had a fix on the way feel free to reassign! | 07:59 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add metricd workers support in gnocchi https://review.openstack.org/374704 | 07:59 |
| matbu | jpich: no i don't try to fix yet, i was just making sure that it wasn't my review which break it :) | 07:59 |
| matbu | i'll try to see if i can | 08:00 |
| jpich | Probably should have just renamed my tox env to compare versions more easily, blergh | 08:00 |
| jpich | matbu: Cool, cheers :) | 08:00 |
| *** athomas has joined #tripleo | 08:01 | |
| shardy | bandini: Hey, quick sanity check re custom-roles and ha-lite when you get a moment | 08:03 |
| shardy | bandini: the DB and rabbit services are still managed by pacemaker in Newton, and that means they are tied to the controller, right? | 08:04 |
| shardy | I'm being asked if it's possible to define e.g a "database" role, and just deploy the DB services on those nodes | 08:04 |
| shardy | but I think that won't be possible until we've got the composable HA stuff worked out? | 08:05 |
| d0ugal | At the moment if a workflow fails and doesn't post to Zaqar the CLI hangs forever. I have a patch that adds timeouts, it works well enough but does anyone have any idea how I can figure out what a sensible default timeout is for each of the commands? | 08:05 |
| gfidente | shardy, curious what are the expectations, to have two different pacemaker clusters one for rabbit and one for database nodes | 08:05 |
| shardy | d0ugal: can you put a log line with some timing into the client, then look at CI results for a few days to figure out the upper bound? | 08:06 |
| gfidente | shardy, or to deploy the database role withouth pacemaker? | 08:06 |
| *** mbozhenko has quit IRC | 08:06 | |
| shardy | d0ugal: then double it or something? | 08:06 |
| gfidente | shardy, looks like the 2nd scenario could be worked out fixing the non-pacemaker roles for galera | 08:06 |
| d0ugal | shardy: Sure, that seems fine - but say, for example, introspection could take a really long time in the real world if you have enough nodes. | 08:06 |
| d0ugal | shardy: so maybe that one should never timeout? | 08:07 |
| d0ugal | or maybe 4 hours as then we will hit token issues anywya. | 08:07 |
| bandini | shardy: correct yes. until composable ha you cannot really split out pacemaker services out | 08:07 |
| shardy | d0ugal: that makes me think the client should never be making synchronous calls to the workflow? | 08:07 |
| d0ugal | shardy: it doesn't. | 08:07 |
| shardy | e.g perhaps we should poll the queue rather than block on a websocket? | 08:07 |
| d0ugal | shardy: hrm, maybe. | 08:08 |
| shardy | d0ugal: my thinking is you could then provide incremental status to users and just let them decide if it's stuck | 08:08 |
| d0ugal | shardy: Isn't the result the same? How long do we poll for? | 08:08 |
| shardy | vs giving no output for 4 hours | 08:08 |
| d0ugal | shardy: I see - but what output would you give? "Polling for messages... none found" over and over. | 08:09 |
| shardy | d0ugal: isn't the problem that when a workflow fails, we should always get a failure notification? | 08:09 |
| d0ugal | shardy: Yeah, so I guess polling Mistral for the workflow status makes sense. | 08:09 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials and CephClusterFSID needs to be set with the overcloud password https://review.openstack.org/374892 | 08:10 |
| d0ugal | This would all make much more sense if the CLI didn't block - then we could check a few things. | 08:10 |
| shardy | gfidente: I'm not clear on the requirement around HA here, the only request is how do we scale e.g DB nodes beyond 3 nodes | 08:10 |
| shardy | gfidente: I assume they would want a separate pacemaker cluster, but the non-pcmk approach might be a workable interim solution I guess | 08:10 |
| *** masco has joined #tripleo | 08:10 | |
| shardy | d0ugal: Yeah, that's what I was getting at with the synchronous comment | 08:11 |
| shardy | like, show the status of the nodes during introspection vs doing nothing | 08:11 |
| shardy | I do wonder if we could wire that in via the workflow though? | 08:11 |
| d0ugal | shardy: Right, but really the workflow should be sending us all that status information - the only reason it doesn't is due to time constraints. | 08:12 |
| d0ugal | shardy: I'd like to do that for O. | 08:12 |
| bandini | shardy, gfidente: here is an initial wip for composable HA roles https://review.openstack.org/#/c/362389/. my plan is to write a spec and submit it for ocata | 08:12 |
| shardy | d0ugal: you could have a trigger introspection workflow, and a check for introspection complete workflow | 08:12 |
| shardy | the latter could be restarted on timeout | 08:12 |
| shardy | (same pattern for all long running calls) | 08:13 |
| matbu | jpich: few hours ago a commit has fixed the failure | 08:13 |
| shardy | ideally we'll fix the token reauth thing with mistral in ocata tho | 08:13 |
| matbu | jpich: i comment the LP | 08:13 |
| matbu | jpich: all UT are green in my local env | 08:13 |
| shardy | bandini: thanks, will check it out | 08:13 |
| jpich | matbu: Oh? That's great \o/ | 08:13 |
| shardy | bandini: my query is more about what is possible with Newton tho | 08:13 |
| d0ugal | shardy: okay, so rather than blocking and waiting for messages - poll the workflow status and then consume the messages. | 08:13 |
| shardy | d0ugal: Yeah, just throwing ideas around but perhaps that would help? | 08:14 |
| shardy | it'd be much easier to set a sane timeout on a thing that just checks status | 08:14 |
| d0ugal | shardy: Yeah, I think it would. It is less elegant but I can't think of any other ideas. | 08:14 |
| bandini | shardy: yeah so with newton you are basically limited to the number of pacemaker nodes you instal. you install 5 you get five rabbits and five galeras running there | 08:14 |
| d0ugal | shardy: Why would that be easier? | 08:15 |
| d0ugal | shardy: essentially the waiting for queue messages is just checking status too | 08:15 |
| d0ugal | shardy: and that is easy to timeout, I've already done it. | 08:15 |
| shardy | bandini: Ok, I think we only reccomend max 3 controllers but that makes sense | 08:15 |
| matbu | jpich: yep, a nice friday bug ;) | 08:15 |
| *** akuznetsov has joined #tripleo | 08:16 | |
| shardy | d0ugal: we can guestimate a reasonable time to wait for a check_$foo workflow to send a status message | 08:16 |
| bandini | shardy: that is correct yes. mainly due to a) none tested != 3 b) none proved it actually improves performance | 08:16 |
| shardy | because a few GET calls shouldn't take a really long time, unlike creating some resource then waiting for an inteterminate amount of time for it to finish | 08:16 |
| shardy | d0ugal: e.g ironic node-list and nova list | 08:16 |
| shardy | I'd say anything more than, say, a minute for either and things are very broken, no? | 08:17 |
| shardy | bandini: ack, yeah I was under the impression galera scale limitations meant it didn't necesarily improve performance, but have no data to back that up :) | 08:18 |
| shardy | bandini: Ok, thanks for the confirmation! | 08:18 |
| bandini | shardy: yeah we should really gather some data on that eventually :) | 08:18 |
| bandini | np | 08:18 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 08:20 |
| jpich | matbu: Awesome, thanks. Of course d0ugal was already on top of it :) | 08:20 |
| d0ugal | shardy: jpich ? | 08:20 |
| jpich | d0ugal: Please open bugs when you fix big problems ;) | 08:20 |
| d0ugal | shardy: oops, that wasn't meant for you - sorry. | 08:20 |
| d0ugal | jpich: The failing unittest? | 08:20 |
| jpich | d0ugal: Yup :) https://bugs.launchpad.net/tripleo/+bug/1626885 | 08:21 |
| openstack | Launchpad bug 1626885 in tripleo "Client unit tests failing: "TypeError: 'Mock' object is not iterable"" [Low,Triaged] - Assigned to Julie Pichon (jpichon) | 08:21 |
| d0ugal | shardy: Okay, yeah, I think that makes sense. Trying to figure out how to change the CLI to do something like that now. | 08:21 |
| jpich | d0ugal: Just so people don't get too much of a surprise when they wake up to a red gate!! | 08:21 |
| d0ugal | jpich: heh, indeed, that is why I done a rushed fix yesterday | 08:21 |
| d0ugal | jpich: but I failed to realise it would need to be backported. | 08:22 |
| jpich | d0ugal: Tis ok. Wanna click that button or I can follow up with the backport? | 08:22 |
| matbu | yolanda: morning, i saw you merged : https://review.openstack.org/373406, do you know when the change would be apply with jjb ? | 08:23 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test https://review.openstack.org/375311 | 08:23 |
| d0ugal | jpich: ^ | 08:23 |
| yolanda | hi matbu , if that already was merged, it can take over half an hour to be applied | 08:24 |
| jpich | d0ugal: \o/ Thank you for always being on top of this kind of problems! | 08:24 |
| matbu | yolanda: k, thx | 08:25 |
| d0ugal | jpich: np, I just run head first into them quickly it seems | 08:26 |
| *** akrivoka has joined #tripleo | 08:26 | |
| *** flepied has joined #tripleo | 08:28 | |
| *** dsariel has quit IRC | 08:29 | |
| *** absubram has joined #tripleo | 08:30 | |
| shardy | jaosorior: Hey, I had a question re https://review.openstack.org/#/c/370573 | 08:30 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Stop plan creation when container exists https://review.openstack.org/369623 | 08:30 |
| shardy | jaosorior: do you think we still need global_config_settings now we have the stricter service_config_settings interface? | 08:30 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Remove deprecation warning for bulk introspection https://review.openstack.org/374935 | 08:30 |
| shardy | I remember we were going to use global_ for some stuff, but I don't see any actual usage of it in-tree atm | 08:30 |
| *** absubram_ has joined #tripleo | 08:32 | |
| *** absubram has quit IRC | 08:34 | |
| *** absubram_ is now known as absubram | 08:34 | |
| *** absubram has quit IRC | 08:34 | |
| *** hewbrocca-afk is now known as hewbrocca | 08:35 | |
| jaosorior | shardy: I remember we talked about this before. I don't think we need that anymore. The service-specific configs are sufficient | 08:37 |
| jaosorior | shardy: only thing is that for the haproxy use-case we needed a deep merge | 08:38 |
| jaosorior | shardy: https://review.openstack.org/#/c/355366/ | 08:38 |
| jaosorior | so we could get rid of that ^^ | 08:38 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername https://review.openstack.org/365585 | 08:39 |
| *** florianf has quit IRC | 08:41 | |
| shardy | jaosorior: ah, yeah that would be nice to clean up! :) | 08:43 |
| *** chem has joined #tripleo | 08:43 | |
| shardy | Ok, I'll propose a patch removing global_config_settings as if we release with it, I guess we'll have to maintain it | 08:43 |
| jaosorior | shardy: sounds good to me | 08:43 |
| *** zoliXXL is now known as zoli|brb | 08:44 | |
| *** derekh has joined #tripleo | 08:44 | |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 08:45 |
| *** paramite has joined #tripleo | 08:45 | |
| *** chem has quit IRC | 08:46 | |
| *** chem has joined #tripleo | 08:46 | |
| *** florianf has joined #tripleo | 08:47 | |
| jistr | merge? :) https://review.openstack.org/#/c/374791/ | 08:47 |
| *** b00tcat has joined #tripleo | 08:48 | |
| openstackgerrit | Merged openstack/diskimage-builder: elements: opensuse: Add support for openSUSE Leap https://review.openstack.org/371602 | 08:48 |
| shardy | jistr: done! | 08:49 |
| jistr | thanks! | 08:49 |
| shardy | Nice that the PostDeploySteps refactoring made this a little cleaner :) | 08:50 |
| *** zoli|brb is now known as zoli | 08:50 | |
| jistr | yea :) | 08:50 |
| gfidente | shardy, so service_config_settings is like global_ | 08:50 |
| gfidente | but it only distributes to the nodes where service_name is deployed? | 08:51 |
| shardy | gfidente: yes, exactly | 08:51 |
| gfidente | wow :) | 08:51 |
| shardy | :) | 08:51 |
| *** cylopez has quit IRC | 08:56 | |
| *** cylopez1 has joined #tripleo | 08:56 | |
| hewbrocca | Morning folks! | 08:56 |
| shardy | 'morning hewbrocca ! | 08:56 |
| *** pkovar has joined #tripleo | 08:58 | |
| Jokke_ | moin | 08:59 |
| b00tcat | another silly question: if I don't specify a `-e` option to `openstack overcloud deploy`, what's the used env file? | 09:03 |
| b00tcat | or are the default values used? | 09:03 |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** openstackgerrit has joined #tripleo | 09:03 | |
| *** yamahata has quit IRC | 09:03 | |
| shardy | b00tcat: it only includes the "base" environment in that case, which is overcloud-resource-registry-puppet.yaml | 09:04 |
| shardy | b00tcat: the idea is you pass a series of -e options which then either override or add to what's in that base registry | 09:04 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add the workflow name to the starting workflow message https://review.openstack.org/375331 | 09:05 |
| *** pmannidi has quit IRC | 09:06 | |
| b00tcat | got it shardy , thanks again! | 09:06 |
| tbarron | morning | 09:08 |
| * tbarron drinks coffee and reads the backlog | 09:08 | |
| tbarron | gfidente: w.r.t. https://review.openstack.org/374961 did you want to try something yourself on that beaker machine? Otherwise since I don't know fancy artefact injection tricks I'd tear down the overcloud, pull in that patch on top of https://review.openstack.org/358525, and redeploy | 09:14 |
| Jokke_ | morning tbarron | 09:14 |
| tbarron | matbu: Jokke_ ^^ | 09:14 |
| tbarron | Jokke_: morning it is :) | 09:15 |
| gfidente | tbarron, give me a sec and we do artifacts together | 09:15 |
| tbarron | gfidente: k, no rush, thanks | 09:15 |
| * matbu brb | 09:15 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove hard-coded roles in EnabledServices output https://review.openstack.org/375339 | 09:15 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-puppet-elements: Install ceph-selinux in overcloud-{controller,cephstorage} images https://review.openstack.org/373998 | 09:21 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Use kwargs to pass in data and error to Mistral Result https://review.openstack.org/375348 | 09:24 |
| *** limao has quit IRC | 09:24 | |
| *** limao has joined #tripleo | 09:26 | |
| d0ugal | Every time I see "WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils" it makes me a little mad | 09:28 |
| gfidente | tbarron, now? | 09:29 |
| gfidente | Jokke_, ^^? | 09:29 |
| tbarron | gfidente: Jokke_ I'm there | 09:31 |
| tbarron | gfidente: Jokke_ watching | 09:31 |
| *** milan has joined #tripleo | 09:33 | |
| jpich | d0ugal: Shouldn't it all be gone now though? | 09:37 |
| *** akuznetsov has quit IRC | 09:37 | |
| d0ugal | jpich: I feel like it should, but it isn't | 09:40 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/366025 | 09:42 |
| *** panda is now known as panda|afk | 09:43 | |
| jpich | d0ugal: Maybe 'cause it's an old environment | 09:48 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/366025 | 09:49 |
| bandini | any heat pro that can tip me in the right direction on how to debug https://bugs.launchpad.net/tripleo/+bug/1626628? | 09:49 |
| openstack | Launchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,New] | 09:49 |
| d0ugal | jpich: Yeah, maybe. I'm reinstalling anyway. | 09:49 |
| therve | shardy, I'm looking at heat memory issue | 09:50 |
| therve | shardy, Are you still using CFN as a signal transport everywhere? | 09:50 |
| gfidente | bandini, me worried about the cephmonupgrade part | 09:51 |
| bandini | gfidente: it could be that it is simply because it is the first resource there? | 09:52 |
| gfidente | bandini, or the script itself timing out | 09:52 |
| gfidente | do you see any of the echo lines from the shell in the journal log? | 09:52 |
| bandini | gfidente: let me double check | 09:53 |
| *** abehl has joined #tripleo | 09:53 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/366025 | 09:54 |
| *** tosky has joined #tripleo | 09:56 | |
| gfidente | bandini, how many storage nodes you have? | 09:57 |
| gfidente | this loop would run forver https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/major_upgrade_ceph_mon.sh#L29 if OSDs < replica size | 09:57 |
| gfidente | well not forver, 300secs | 09:57 |
| shardy | therve: Hi! | 09:57 |
| *** florianf has quit IRC | 09:58 | |
| bandini | gfidente: I have none | 09:58 |
| shardy | therve: thanks for looking into it - I tried the three patches from zaneb and it's not made a lot of difference | 09:58 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Use kwargs to pass in data and error to Mistral Result https://review.openstack.org/375348 | 09:58 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: No-op Puppet for upgrades/migrations according to composable roles https://review.openstack.org/374791 | 09:58 |
| shardy | therve: yes, we're still using the default CFN transport for signalling | 09:58 |
| therve | OK | 09:58 |
| shardy | therve: but we switched to swift tempurl for polling metadata | 09:58 |
| therve | shardy, Which ones are those patches? | 09:58 |
| therve | Ah, ok | 09:58 |
| shardy | https://review.openstack.org/#/q/project:openstack/heat+branch:master+topic:bug/1626675 | 09:59 |
| shardy | therve: ^^ | 09:59 |
| shardy | I rebased on latest master and applied the two remaining patches | 09:59 |
| therve | Cool, thanks | 09:59 |
| shardy | peak memory usage is slightly reduced, but not by much | 09:59 |
| shardy | therve: I can fairly easily reproduce my test now, do you want me to try with a different signal transport? | 10:01 |
| therve | shardy, No I was wondering if that could a reason for the change | 10:01 |
| shardy | the plan is to switch everything over to zaqar, but we've not quite got there yet | 10:02 |
| skramaja | mcornea: thanks for the ServiceNetMap. now the host name is proper for the new role (compute-dpdk) | 10:03 |
| skramaja | shardy: mcornea: but i am facing another issue at step 3 in the controller.. | 10:04 |
| skramaja | http://paste.openstack.org/show/582712/ | 10:04 |
| jaosorior | shardy: by the way, have you come up with a solution for the remaining haproxy configuration that's outside the service profile? the service_certificate | 10:04 |
| skramaja | pasted the log output. any clue? | 10:04 |
| shardy | skramaja: is that error happening on your new role, e.g the compute-dpdk node? | 10:05 |
| shardy | or the controller? | 10:05 |
| skramaja | its happening on the controller shardy | 10:06 |
| shardy | skramaja: Hmm, and this environment was all working fine before you started working with custom roles? | 10:06 |
| mcornea | skramaja: I haven't seen this but it points that it can't run openstack project list so I'd check how keystone looks on controller nodes | 10:06 |
| skramaja | yes.. | 10:06 |
| shardy | yeah, it looks like keystone is broken | 10:06 |
| shardy | but adding the new compute role shouldn't have caused that (in theory at least..) | 10:06 |
| skramaja | yes.. thats my confusion too. | 10:07 |
| skramaja | anyways still looking.. if you have any pointers it would make easier.. | 10:07 |
| shardy | I'd check the status of keystone and try running it manually to get an idea of why it's not responding | 10:07 |
| *** thrash|g0ne is now known as thrash | 10:07 | |
| *** yolanda has quit IRC | 10:09 | |
| *** florianf has joined #tripleo | 10:10 | |
| *** zoli is now known as zoli|lunch | 10:11 | |
| skramaja | ok shardy | 10:13 |
| *** limao has quit IRC | 10:14 | |
| *** leanderthal|afk is now known as leanderthal | 10:17 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add network gateway validation https://review.openstack.org/329385 | 10:17 |
| *** panda|afk is now known as panda | 10:22 | |
| *** fultonj has joined #tripleo | 10:23 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: explicitly set fluentd service_provider https://review.openstack.org/374850 | 10:23 |
| *** jprovazn has quit IRC | 10:23 | |
| shadower | Hey folks, could I get these two validations reviewed? https://review.openstack.org/#/c/366025/ and https://review.openstack.org/#/c/329385/ | 10:24 |
| shadower | the first one fixes a bug with a couple existing ones and a lot of others depend on the latter | 10:24 |
| jpich | shadower: Hello! When you have a chance sometime, do you think you could add a few thoughts/limitations/possible workarounds/words of warnings about validations in https://blueprints.launchpad.net/tripleo/+spec/tripleo-ui-i18n-support ? | 10:25 |
| shadower | jpich: interesting. Yeah I'll do | 10:27 |
| jpich | shadower: Thank you! | 10:27 |
| shardy | shadower: lgtm, added to rc2 | 10:27 |
| shardy | shadower: do we have any CI coverage of validations yet? | 10:27 |
| shadower | shardy: thanks! And not yet but I'd like to make that happen in a few weeks (though we may want to discuss it at the summit first) | 10:28 |
| *** yolanda has joined #tripleo | 10:30 | |
| *** kbyrne has quit IRC | 10:31 | |
| shardy | shadower: ack, sounds good | 10:32 |
| shardy | I think a discussion on how we get validations generally better integrated (including CI) would be a good thing :) | 10:33 |
| shadower | cool | 10:33 |
| *** dtantsur is now known as dtantsur|bbl | 10:34 | |
| *** kbyrne has joined #tripleo | 10:34 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validate the IP range for the control plane https://review.openstack.org/353950 | 10:34 |
| *** kbyrne has quit IRC | 10:39 | |
| *** kbyrne has joined #tripleo | 10:39 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Validation to verify connectivity to overcloud https://review.openstack.org/353934 | 10:51 |
| *** dprince has joined #tripleo | 10:58 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 11:01 |
| *** rasca has quit IRC | 11:03 | |
| *** rasca has joined #tripleo | 11:04 | |
| *** lucas-afk is now known as lucasagomes | 11:05 | |
| thrash | ccamacho|afk: it's not creating the endpoint, which is why the pingtest is failing. | 11:11 |
| *** ccamacho|afk is now known as ccamacho | 11:11 | |
| ccamacho | thrash ack let me it | 11:12 |
| thrash | ccamacho: http://paste.openstack.org/show/582723/ | 11:12 |
| thrash | or I should say, it's not creating the endpoint correctly. | 11:12 |
| ccamacho | thrash ill deploy locally, as that service its almost good to go then, are you working on it? | 11:13 |
| jaosorior | thrash: ccamacho: by the way, the keystone::auth parameters will change place (hopefully today) you will need to specify them in t-h-t in another section called service_config_settings | 11:13 |
| thrash | ccamacho: yes, I'm working on it. | 11:14 |
| thrash | jaosorior: ack. | 11:14 |
| jaosorior | thrash, ccamacho: if you need help debugging that endpoint issue I can take a look. | 11:14 |
| ccamacho | jaosorior, can you paste that submission like? | 11:14 |
| ccamacho | s/like/link/ | 11:14 |
| jaosorior | ccamacho: https://review.openstack.org/#/c/370573/9 | 11:14 |
| thrash | jaosorior: it's creating the publicurl, but not admin and internal. | 11:15 |
| thrash | see the paste above.. | 11:15 |
| ccamacho | thrash, then I wont push any update, just local tests and just give some feedback | 11:15 |
| thrash | ccamacho: thanks | 11:15 |
| jaosorior | thrash: undercloud or overcloud? | 11:15 |
| thrash | jaosorior: over | 11:15 |
| jaosorior | thrash: pass the CR | 11:15 |
| thrash | https://review.openstack.org/#/c/323436/ | 11:15 |
| jaosorior | thrash: mistral has no network assigned | 11:16 |
| jaosorior | thrash: you need to add it here https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.yaml#L15 | 11:17 |
| thrash | jaosorior: ack | 11:17 |
| jaosorior | that's the issue. the endpoint map tries to fetch the value of MistralApiNetwork, and it fails. Thus, internal and admin are empty. | 11:17 |
| thrash | ahhh | 11:18 |
| jaosorior | public doesn't need the network, it uses the public ~VIP | 11:18 |
| jaosorior | thrash: but it's good that it manifested there, you were gonna have a bunch of other issues too | 11:18 |
| thrash | jaosorior: thanks. that would have taken me forever to figure out. :) | 11:18 |
| thrash | jaosorior: like? | 11:19 |
| thrash | Or just because of that? | 11:19 |
| jaosorior | because of that | 11:19 |
| jaosorior | so, without that network mapping, probably the mistral_network variable wouldn't have been generated, or the nodes, or even the mistral_vip | 11:19 |
| jaosorior | so to start with haproxy would have had a wrong configuration | 11:19 |
| thrash | gotcha | 11:20 |
| jaosorior | (haproxy would have worked anyway, cause it will default to using ctlplane) | 11:20 |
| *** dtantsur|bbl is now known as dtantsur | 11:20 | |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Move inclusion of ::manila::db::mysql in manila/api profile https://review.openstack.org/374961 | 11:20 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Activate StorageMgmtPort on computes in HCI environment https://review.openstack.org/375404 | 11:26 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 11:27 |
| *** hjensas has quit IRC | 11:27 | |
| *** hjensas has joined #tripleo | 11:29 | |
| *** hjensas has joined #tripleo | 11:29 | |
| *** yolanda has quit IRC | 11:33 | |
| *** zoli|lunch is now known as zoli | 11:33 | |
| *** zoli is now known as zoliXXL | 11:33 | |
| *** jeckersb is now known as jeckersb_gone | 11:34 | |
| *** yolanda has joined #tripleo | 11:34 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Tolerate missing keys from role_data in service templates https://review.openstack.org/374237 | 11:35 |
| *** jprovazn has joined #tripleo | 11:35 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 11:38 |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 11:43 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings https://review.openstack.org/370573 | 11:43 |
| *** rhallisey has joined #tripleo | 11:50 | |
| *** rbrady has quit IRC | 11:51 | |
| *** rbrady has joined #tripleo | 11:52 | |
| *** fultonj has quit IRC | 11:52 | |
| jaosorior | dprince: hey man, could you take another look at this? https://review.openstack.org/#/c/347553/ | 11:54 |
| dprince | jaosorior: yep | 11:55 |
| *** jrist has quit IRC | 11:56 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/375416 | 11:56 |
| openstackgerrit | Merged openstack/tripleo-validations: Make the ini-based validations more robust https://review.openstack.org/366025 | 12:01 |
| *** ccamacho is now known as ccamacho|lunch | 12:02 | |
| *** jpena is now known as jpena|lunch | 12:03 | |
| mcornea | EmilienM: shardy is https://bugs.launchpad.net/tripleo/+bug/1620595 being addressed by https://review.openstack.org/#/c/370573/ ? IIUC the patch resolves the situation when keystone runs on a different role but the bug is about the databases not being created on the db cluster | 12:04 |
| openstack | Launchpad bug 1620595 in tripleo "When Heat API services run on a different node than controller the heat-dbsync fails due to incorrect mysql permissions" [High,In progress] - Assigned to Dan Prince (dan-prince) | 12:04 |
| jaosorior | mcornea: uhm... actually I think that's a bigger issue. | 12:05 |
| jaosorior | mcornea: to do the db-sync, for most services, there is a check if it's the "bootstrap node" | 12:05 |
| mcornea | jaosorior: from what I've seen the database gets created on the node where the service is running | 12:06 |
| *** zoliXXL is now known as zoli|brb | 12:06 | |
| jaosorior | mcornea: https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/heat/engine.pp | 12:06 |
| jaosorior | I'm not sure how the bootstrap node part will work on different roles | 12:07 |
| jaosorior | shardy: how does the bootstrap node part work nowadays? | 12:08 |
| *** jrist has joined #tripleo | 12:08 | |
| EmilienM | hello | 12:11 |
| openstackgerrit | Jiri Stransky proposed openstack/diskimage-builder: Fix grub installation for RHEL https://review.openstack.org/375430 | 12:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 12:18 |
| *** pkovar has quit IRC | 12:18 | |
| *** pkovar has joined #tripleo | 12:22 | |
| EmilienM | mcornea: I'll look in a few | 12:23 |
| mcornea | EmilienM: thanks | 12:23 |
| *** hjensas has quit IRC | 12:24 | |
| shardy | jaosorior: we set bootstrap node for all roles now, but we've not yet solved how to nominate a boostrap host if you deploy a service on multiple roles | 12:25 |
| *** masco has quit IRC | 12:25 | |
| jaosorior | shardy: so we now trigger db-sync on all nodes that deploy a service? | 12:26 |
| shardy | jaosorior: No, only on the bootstrap node, exactly as before | 12:26 |
| shardy | but there's one bootstrap node per role (group of nodes) | 12:26 |
| shardy | so you can't deploy a service whidh does bootstrap things on two roles | 12:26 |
| mandre | Seing this in my keystone logs: SQL connection failed. -614 attempts left. | 12:27 |
| shardy | EmilienM: No, I think it's closely related but we may need to move some mysql things into service_config_settings | 12:27 |
| mandre | someone messed his loop :) | 12:27 |
| jaosorior | shardy: ok, that makes sense | 12:27 |
| shardy | jaosorior: I think we can rework it in ocata, to e.g look at $service_node_names and only use the first name in the list | 12:28 |
| jaosorior | shardy: but... why don't we have a bootstrap node per service? | 12:28 |
| shardy | jaosorior: because nobody has written that patch yet ;) | 12:28 |
| jaosorior | shardy: well, seems that would fix the bug mcornea mentioned | 12:28 |
| *** jayg|g0n3 is now known as jayg | 12:29 | |
| jaosorior | I don't have cycles for that :( | 12:29 |
| jaosorior | but alright | 12:29 |
| jaosorior | thanks for the explanation | 12:29 |
| shardy | jaosorior: maybe, isnt' the problem there that it's trying to do the db-sync on the wrong role? | 12:29 |
| shardy | I need to test it tbh | 12:29 |
| jaosorior | shardy: well, if heat is deployed in a role that isn't the controller, how would it even do the db-sync? If that's what we explicitly check for | 12:31 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials and CephClusterFSID needs to be set with the overcloud password https://review.openstack.org/374892 | 12:31 |
| openstackgerrit | Merged openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test https://review.openstack.org/375311 | 12:33 |
| *** zoli|brb is now known as zoli | 12:34 | |
| *** zoli is now known as zoliXXL | 12:34 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Display error message when socket is closed https://review.openstack.org/374669 | 12:34 |
| *** pgadiya has quit IRC | 12:34 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Provide more information when 'node provide' fails https://review.openstack.org/374670 | 12:34 |
| shardy | jaosorior: The idea is it'd do the db-sync on the bootstrap node of $role_not_controller | 12:34 |
| shardy | but evidently something isn't wired in atm to allow that | 12:34 |
| jaosorior | exactly | 12:35 |
| shardy | provided we have the connection string to reach the DB, I think it should work | 12:35 |
| jaosorior | that was my understanding | 12:35 |
| shardy | but something is missing from the configuration atm | 12:35 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/338088 | 12:36 |
| *** rcernin has quit IRC | 12:38 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/338088 | 12:39 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Make the ovb-updates job work again https://review.openstack.org/374406 | 12:39 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:40 |
| gfidente | bnemec, dah rebased https://review.openstack.org/374406 | 12:42 |
| gfidente | wanted to make https://review.openstack.org/338088 depend on it | 12:43 |
| *** pcaruana|afk| has joined #tripleo | 12:44 | |
| *** cylopez1 has left #tripleo | 12:45 | |
| mandre | hello, none of my services on the overcloud can talk to mariadb terminated by haproxy, http://paste.openstack.org/show/582741/ | 12:46 |
| *** fultonj has joined #tripleo | 12:47 | |
| mandre | here's my haproxy.conf: http://paste.openstack.org/show/582743/ | 12:49 |
| *** pgadiya has joined #tripleo | 12:50 | |
| *** david-lyle has joined #tripleo | 12:56 | |
| *** tzumainn has joined #tripleo | 12:57 | |
| *** pcaruana|afk| has quit IRC | 13:00 | |
| EmilienM | shardy: I'll let you +A https://review.openstack.org/#/c/374835/ or not | 13:00 |
| *** pcaruana has quit IRC | 13:01 | |
| *** pcaruana has joined #tripleo | 13:02 | |
| beagles | shardy: what are your thoughts on when we move the l3 ha logic from the tht to puppet-tripleo? Better to do it now, or would it be better to hold onto it until Ocata? I'm good either way. | 13:02 |
| *** rcernin has joined #tripleo | 13:06 | |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add support for rabbit hosts to mistral https://review.openstack.org/375461 | 13:11 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 13:12 |
| EmilienM | thrash: why not doing it via Hiera? ^ | 13:12 |
| thrash | EmilienM: I was just following along with what every other service was doing... Am I doing it wrong then? :) | 13:12 |
| *** jpena|lunch is now known as jpena | 13:13 | |
| EmilienM | yes, we need to define it with hiera | 13:13 |
| EmilienM | in the composable service | 13:13 |
| *** jaosorior has quit IRC | 13:13 | |
| thrash | EmilienM: just curious, but why is literally every other service doing it that way? (I don't care, makes it easier for me) | 13:14 |
| *** jaosorior has joined #tripleo | 13:14 | |
| EmilienM | thrash: let me check but it's a mistake | 13:14 |
| EmilienM | thrash: we use hiera for parameters | 13:14 |
| EmilienM | jaosorior: fyi ^ | 13:14 |
| thrash | EmilienM: https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/neutron.pp#L31 | 13:14 |
| thrash | EmilienM: https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/heat.pp#L47 | 13:14 |
| *** cdearborn has joined #tripleo | 13:14 | |
| EmilienM | yes | 13:15 |
| thrash | Literally every other one. | 13:15 |
| EmilienM | it's bad | 13:15 |
| thrash | :D | 13:15 |
| EmilienM | we need to use hiera in THT | 13:15 |
| EmilienM | that's how we configure services | 13:15 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common https://review.openstack.org/368150 | 13:15 |
| EmilienM | in "puppet way" | 13:15 |
| thrash | EmilienM: shardy did it 26 days ago | 13:15 |
| thrash | :) | 13:15 |
| thrash | https://github.com/openstack/puppet-tripleo/commit/e5f0042e47eb4c2b77348dcec0d3d2730e5d9841 | 13:15 |
| EmilienM | maybe | 13:15 |
| EmilienM | but again that's an error | 13:16 |
| EmilienM | puppet profiles should only do "include ::myclass" | 13:16 |
| EmilienM | and THT should feed data with Hiera | 13:16 |
| thrash | EmilienM: I'm not arguing. I totally agree. | 13:16 |
| thrash | shardy: so, why did you do it that way, and not via hiera? | 13:17 |
| EmilienM | thrash: I'm working on a fix | 13:17 |
| *** zoliXXL is now known as zoli|brb | 13:18 | |
| gfidente | marios, can you add the depends-on https://review.openstack.org/338088 | 13:19 |
| gfidente | for the manila submission? | 13:19 |
| gfidente | seems like we got it passing | 13:19 |
| *** adarazs is now known as adarazs_brb | 13:19 | |
| gfidente | deployment failed on httpd failing to start but the manila part was in place | 13:19 |
| EmilienM | thrash: wait | 13:20 |
| gfidente | while we retry that I think depends-on will speed up things | 13:20 |
| EmilienM | I don't see rabbitmq_node_ips in THT | 13:20 |
| thrash | EmilienM: I was just seeing that myself... | 13:20 |
| thrash | But yet, somehow, all the other services are getting it configured... | 13:20 |
| *** myoung|gone is now known as myoung | 13:21 | |
| EmilienM | thrash: go ahead with mistral | 13:21 |
| EmilienM | ignore my -1 | 13:21 |
| thrash | EmilienM: ack | 13:21 |
| marios | gfidente: ack incoming fantastic to hear | 13:21 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add support for rabbit hosts to mistral https://review.openstack.org/375461 | 13:21 |
| thrash | EmilienM: it looks like mistral was just missed in https://review.openstack.org/#/c/361730/ | 13:23 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add integration with Manila CephFS Native driver https://review.openstack.org/358525 | 13:23 |
| gfidente | marios, wait wrong link | 13:23 |
| gfidente | https://review.openstack.org/#/c/374961/ | 13:24 |
| marios | Jokke_: added the depends on gfidente yes it should be right | 13:24 |
| marios | Jokke_: gfidente: https://review.openstack.org/#/c/358525/ | 13:24 |
| *** pkovar has quit IRC | 13:24 | |
| EmilienM | thrash: so there is some magic in tht | 13:25 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback https://review.openstack.org/375477 | 13:25 |
| *** rcernin has quit IRC | 13:26 | |
| marios | gfidente: ah i see you changed it a bit, removed the rabbit_hosts added bootstrap_node | 13:26 |
| gfidente | marios, that's so scheduler can pick up rabbit hosts from different node | 13:27 |
| gfidente | but the db sync issue was inclusion of ::db::mysql in scheduler role while it's ::api attempting ::sync | 13:27 |
| marios | gfidente: right thanks I +2 - you had a good run with it right? | 13:28 |
| gfidente | yes but I am fixing lint stuff yet | 13:28 |
| gfidente | sorry :( | 13:28 |
| gfidente | thanks! | 13:28 |
| marios | gfidente: kk thanks np ping me when/if wipes the votes | 13:28 |
| marios | gfidente: thank YOU gfidente but stop talking now thanks | 13:28 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback https://review.openstack.org/375477 | 13:29 |
| *** kjw3 has joined #tripleo | 13:29 | |
| trown | panda: do you happen to know where in tripleo-ci we setup the undercloud for network isolation? I see https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/deploy.sh#L81-L98 but have no idea how eth2 is setup on OVB | 13:31 |
| EmilienM | gfidente: can you approve this quick patch? https://review.openstack.org/#/c/374150/ | 13:32 |
| *** jeckersb_gone is now known as jeckersb | 13:32 | |
| gfidente | bnemec, so no reason for adding it back to HA except I wanted it combined with netiso | 13:32 |
| EmilienM | gfidente: same for https://review.openstack.org/#/c/373513/ | 13:32 |
| gfidente | EmilienM, ack | 13:32 |
| *** pkovar has joined #tripleo | 13:33 | |
| EmilienM | gfidente: thanks :) | 13:33 |
| *** Goneri has joined #tripleo | 13:34 | |
| bnemec | gfidente: Ah, I guess that makes sense. Too bad. :-/ | 13:34 |
| gfidente | bnemec, but | 13:34 |
| bnemec | Rather not add more time to the ha job, but it's better than adding it to updates. :-) | 13:34 |
| gfidente | but I agree if we could balance something out on nonha it'd be better | 13:34 |
| gfidente | especially because ceph doesn't care about ha/nonha | 13:34 |
| bnemec | gfidente: Yeah, but I don't know that I want to add net-iso to nonha. It's the only job that tests without it right now. | 13:35 |
| gfidente | bnemec, now, out of the blue | 13:35 |
| gfidente | do we really need the non-netiso scenario in tripleo? :) | 13:36 |
| panda | trown: what is missing from those lines that you're not finding ? | 13:36 |
| bnemec | gfidente: Well, that's a valid question. | 13:36 |
| *** rcrit has joined #tripleo | 13:36 | |
| bnemec | And we do sort of cover it in the multinode jobs. | 13:36 |
| trown | panda: well there is no eth2 on default quickstart undercloud, and if I make just an extra network, that does not work either :) | 13:37 |
| trown | panda: in the RDO CI code we make a vlan for network isolation, so trying to figure out what tripleo-ci needs from undercloud | 13:37 |
| EmilienM | gfidente: why do you tag it rc2 ? https://review.openstack.org/#/c/375404/ | 13:37 |
| EmilienM | does it fix an RC2 bug? | 13:37 |
| panda | trown: I'm stealing bnemec today to talk more or less about this. I'm trying to understand how to setup network environment to work with VM and get the same tripleo-ci behaviour, want to join ? | 13:40 |
| EmilienM | gfidente: i'm find +2 if we're improving coverage | 13:40 |
| trown | panda: yes please, I got nonha job working (with hacks) on top of quickstart undercloud, but ha job is causing me issues because network isolation | 13:40 |
| *** adarazs_brb is now known as adarazs | 13:41 | |
| panda | bnemec: whenever you feel comfortable, I have a fresh provisioned undercloud with some vm waiting to be deployed | 13:42 |
| gfidente | EmilienM, so it's not really a bug | 13:42 |
| gfidente | but if you want to do netiso and hci those two lines should be uncomented | 13:42 |
| gfidente | we thought it would be better to uncomment them by default | 13:42 |
| gfidente | and to land that with the release to not change the default after the release | 13:43 |
| EmilienM | gfidente: ok, fair enough, +2 | 13:43 |
| gfidente | hehe sorry should have been uncommented from the start | 13:43 |
| trown | panda: ah looks like we need more than 1 extra network: https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/network-templates/nic-configs/controller.yaml | 13:43 |
| trown | looks like we need 6 overcloud networks to use the tripleo-ci method | 13:44 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-quickstart: Remove really old cruft from instackenv.json.j2 https://review.openstack.org/375490 | 13:44 |
| dtantsur | trown, ^^ | 13:45 |
| trown | dtantsur: cool, I guess we will find out if it works for old versions of ironic via CI :) | 13:45 |
| dtantsur | now we need the same kind of clean up in instack-virt-setup, but I guess it may be harder... | 13:46 |
| trown | do we? | 13:47 |
| dtantsur | well, or whatever else generates instackenv.json | 13:47 |
| *** akshai has joined #tripleo | 13:47 | |
| trown | I mean if it doesnt break anything, cosmetic cleanup there seems a bit like lipstick on a bull | 13:47 |
| dtantsur | I particularly adore references to nova-baremetal there :) | 13:47 |
| dtantsur | lol | 13:47 |
| dtantsur | oh, that's tripleo-incubator: https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest_testenv.sh#L255-L264 | 13:48 |
| dtantsur | I'm a bit scared to touch this code :D | 13:48 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Midonet element now complies with version 5.2 https://review.openstack.org/373352 | 13:50 |
| openstackgerrit | Merged openstack/tripleo-common: Add FS_TYPE xfs to RHEL and CentOS images https://review.openstack.org/369600 | 13:50 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Always configure ipv6 address with net-iso https://review.openstack.org/374922 | 13:51 |
| panda | trown: how many do we have now ? | 13:51 |
| mandre | quick question about haproxy, do we enable it for all overcloud deployments, even if there is only one control node? | 13:51 |
| trown | panda: by default we just make one overcloud network, and one external network, the RDO CI code uses single-nic-vlans to do netiso | 13:52 |
| hewbrocca | I thought we killed all the crap in incubator | 13:52 |
| trown | panda: it looks like tripleo-ci uses a custom version of multi-nic | 13:52 |
| trown | panda: testing a run now with 6 overcloud networks to see if it magically will "just work" | 13:53 |
| openstackgerrit | Dimitri Savineau proposed openstack/tripleo-heat-templates: Added support for pass-through iSER configuration https://review.openstack.org/324781 | 13:53 |
| *** rcernin has joined #tripleo | 13:53 | |
| *** zoli|brb is now known as zoli | 13:53 | |
| *** zoli is now known as zoliXXL | 13:53 | |
| jaosorior | mandre: yes | 13:53 |
| gfidente | matbu, marios what is going on with https://review.openstack.org/#/c/374892/5 ? | 13:54 |
| gfidente | from what I can tell it *will* regenerate the keys on upgrade now | 13:54 |
| mandre | jaosorior: I do see a haproxy_enabled hieradata, does it mean it's optional? | 13:54 |
| jaosorior | mandre: it is. If you configure the overcloud with an external loadbalancer | 13:54 |
| jaosorior | mandre: why would you want to get rid of it? | 13:55 |
| marios | gfidente: the assumption is it would be set on deploy and stored in the passwords file no? /me trying to finish something before scrum will look more later | 13:55 |
| mandre | jaosorior: just being generally curious :) | 13:55 |
| mandre | jaosorior: also, my overcloud services can't talk to mysql, i think it might be a misconfiguration of haproxy | 13:56 |
| *** jeckersb is now known as jeckersb_gone | 13:56 | |
| jaosorior | mandre: I can work with you on that | 13:56 |
| jaosorior | what's up? | 13:56 |
| matbu | gfidente: the goal was to manage those keys like the other password are managed | 13:57 |
| mandre | jaosorior: haproxy logs reported that mysql service was down, and it didn't allow connection | 13:57 |
| mandre | jaosorior: here's part of my haproxy.conf http://paste.openstack.org/show/582743/ | 13:58 |
| jaosorior | mandre: and I assume that mysql is indeed running, right? | 13:58 |
| jaosorior | mandre: I've seen it go down due to memory issues :/ | 13:58 |
| marios | jistr: bandini ha https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/major_upgrade_controller_pacemaker_2.sh#L72 - i will fix it momentarily as I need to add an exception there for sahara | 13:58 |
| marios | matbu: ^ | 13:58 |
| mandre | jaosorior: yes it is, and I can connect to it when not via haproxy | 13:58 |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback https://review.openstack.org/375477 | 13:58 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-incubator: Remove really old stuff from generated instackenv.json https://review.openstack.org/375503 | 13:59 |
| bandini | marios: ack | 13:59 |
| bandini | matbu: do you ever get to the stage of running the major-upgrade-pacemaker step? | 13:59 |
| dtantsur | I'm ready to risk ^^ | 13:59 |
| jaosorior | mandre: undercloud or overcloud? | 13:59 |
| jaosorior | ah | 13:59 |
| jaosorior | nevermind | 13:59 |
| jaosorior | yes | 14:00 |
| marios | bandini: doing so on 10 packages still tho | 14:00 |
| jaosorior | overcloud | 14:00 |
| bandini | marios: 10 packages? | 14:00 |
| marios | bandini: like we don't have latest things, even the change to postdeploy noop you hit yesterday | 14:00 |
| jaosorior | mandre: so, the configuration I'm seeing (I just deployed) is this: http://paste.openstack.org/show/582754/ | 14:00 |
| trown | dtantsur: there is no CI on that excercises that code :) | 14:00 |
| marios | bandini: i mean osp10 ... upgrading osp9 to osp10 | 14:00 |
| jistr | folks how can i pull this into Newton RC2 for tripleo-common? I don't see a way to do it in the interface, neither to change the importance on it https://bugs.launchpad.net/tripleo-common/+bug/1609020 | 14:00 |
| openstack | Launchpad bug 1609020 in tripleo-common "Servers get rebuilt during a stack update following new image rollout" [Undecided,In progress] | 14:00 |
| mandre | jaosorior: it happens on the overcloud | 14:00 |
| dtantsur | trown, no CI on tripleo-incubator? Oo | 14:00 |
| jistr | EmilienM: would you know? ^^ | 14:00 |
| marios | bandini: so is a little behind upstream | 14:00 |
| gfidente | matbu, marios agreed they are stored in that file and not recreated | 14:01 |
| gfidente | but not on upgrade | 14:01 |
| jaosorior | mandre: quite strange that it's lacking some options for you. Did you modify it manually? | 14:01 |
| trown | dtantsur: CI might run on it, but it doesnt run instack-virt-setup, or use that instackenv.json from there | 14:01 |
| bandini | marios: ah ok sorry I am slower then usual ;) | 14:01 |
| gfidente | because from previous deployment those were not dumped in that file, so they *will* actually be regenerated | 14:01 |
| EmilienM | jistr: do you have a patch? | 14:01 |
| dtantsur | trown, ok, I'll withdraw this patch... | 14:01 |
| trown | dtantsur: but if quickstart CI passes on that change, the incubator one is probably safe | 14:01 |
| mandre | jaosorior: hmmm, I though I deployed using master puppet module | 14:01 |
| dtantsur | one more reason to not use tripleo-quickstart | 14:01 |
| EmilienM | https://review.openstack.org/#/c/350778 | 14:01 |
| mandre | jaosorior: maybe I have an older version? | 14:01 |
| dtantsur | s/not// | 14:01 |
| jaosorior | mandre: | 14:01 |
| trown | lol | 14:01 |
| jaosorior | nevermind | 14:01 |
| jaosorior | it's fine | 14:01 |
| jistr | EmilienM: yea that's the one | 14:02 |
| jaosorior | it seems that it's the standard nonha config | 14:02 |
| jaosorior | I just checked on CI | 14:02 |
| panda | dtantsur: hey! | 14:02 |
| EmilienM | jistr: if you feel like we need it, yes.. | 14:02 |
| jaosorior | and what we're deploying on CI for the nonha configuration is just like that too | 14:02 |
| mandre | jaosorior: right, I only have one control node | 14:02 |
| marios | bandini: heh np bandini ... btw the migration handles the removal of sahara nicely, i mean i went looking for what we had to remove, the constraints and r esource definition | 14:02 |
| marios | bandini: so all we have to do is stop it being started | 14:02 |
| jistr | EmilienM: upgrades are a no go without it, it rebuilds the whole cloud if we don't have it | 14:02 |
| shardy | jistr: We've been trying to track everything in the main tripleo LP project, so you can just mark the bug as affecting tripleo | 14:02 |
| shardy | then target it to rc2 | 14:02 |
| marios | bandini: for the 'remove sahara' case... i mean | 14:02 |
| bandini | marios: nice! | 14:02 |
| jistr | shardy: ah perfect, thanks | 14:02 |
| EmilienM | jistr: upgrade are automatically rc2 | 14:02 |
| matbu | bandini: nop stuck somewhere on upgrade controller | 14:02 |
| EmilienM | don't even ask :) | 14:02 |
| jaosorior | mandre: yeah, I deployed ha... that's why it's different | 14:03 |
| jaosorior | alright then | 14:03 |
| jaosorior | mandre: can you pass the log again? | 14:03 |
| *** jeckersb_gone is now known as jeckersb | 14:03 | |
| mandre | jaosorior: haproxy logs show http://paste.openstack.org/show/582755/ | 14:03 |
| bandini | matbu: is it bug https://bugs.launchpad.net/tripleo/+bug/1626628 ? i.e. after networkdeplymend gives SIGNAL_COMPLETE nothing happens? | 14:03 |
| openstack | Launchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,Triaged] | 14:03 |
| jaosorior | connection refused... what the hell | 14:04 |
| jaosorior | mandre: is your undercloud accessible somehow? | 14:04 |
| matbu | bandini: yep | 14:05 |
| *** ramishra has quit IRC | 14:05 | |
| mandre | jaosorior: I suppose I can make it accessible to you but I need a moment then | 14:05 |
| jaosorior | mandre: sure | 14:05 |
| matbu | bandini: /me looks the LP | 14:05 |
| bandini | matbu: alrighty at least we're on the same issue as well, I am quite stuck on that atm | 14:05 |
| *** jprovazn has quit IRC | 14:06 | |
| matbu | gfidente: if the FSID exist, it won't override, if not it will create one, right ? | 14:06 |
| gfidente | matbu, yes the problem is that on upgrade there is no FSID in the passwords file | 14:06 |
| *** ramishra has joined #tripleo | 14:06 | |
| mandre | jaosorior: the log says connection error when connecting to the mysql backup server, which is supposedly listening on 192.0.2.14 | 14:07 |
| matbu | gfidente: yes, but (stupid question) why is a pb ? | 14:07 |
| matbu | gfidente: i mean, the upgrade will generate a new FSID | 14:08 |
| gfidente | matbu, yeah the ceph osds are bound to an FSID | 14:08 |
| mandre | jaosorior: 192.0.2.14 is just another address for my overcloud control node | 14:08 |
| jaosorior | mandre: I understood what the log says. It was just weird that it got a connection error | 14:08 |
| gfidente | matbu, so the ceph cluster loose all the data if that changes | 14:09 |
| *** yamahata has joined #tripleo | 14:09 | |
| jaosorior | mandre: anyway, lets check out what's up. Let me know when I can log in | 14:09 |
| mandre | jaosorior: according to galera.conf, mysql binds to overcloud-controller-0 | 14:09 |
| mandre | jaosorior: and overcloud-controller-0 resolves to 127.0.0.1, so it's not really surprising that we get a connection error when connecting on 192.0.2.14 | 14:11 |
| jaosorior | mandre: interesting | 14:12 |
| jaosorior | mandre: lets figure out why | 14:12 |
| jaosorior | mandre: I have the same thing. Except that for me it actually resolves to the internal_api network (probably cause of net-iso | 14:12 |
| matbu | gfidente: ok, then, if we don't fsid and the stack is not new, we should not generate an uuid. we only handle this uuid on new deployment, do we want to store the FSID in the password file ? | 14:13 |
| *** pgadiya has quit IRC | 14:14 | |
| jaosorior | mandre: found the issue | 14:14 |
| mandre | jaosorior: already? nice :) | 14:14 |
| gfidente | matbu, right, probably adding it into passwords file is a good idea | 14:15 |
| jaosorior | mandre: this is wrong https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/database/mysql.pp#L69 | 14:15 |
| shardy | $package_manifest_name = join(['/var/lib/tripleo/installed-packages/overcloud_controller ... | 14:16 |
| jaosorior | mandre: lets try to fix it now | 14:16 |
| shardy | can anyone point me to where we build the list of packages? | 14:16 |
| gfidente | matbu, but we need some logic to pick it up from the passwords file (and generate if missing) only if stack is not new | 14:16 |
| mandre | jaosorior: what should it be binding on? | 14:16 |
| matbu | gfidente: yep | 14:17 |
| *** Goneri has quit IRC | 14:17 | |
| jaosorior | mandre: I'm gonna get it to bind, still to a hostname, but one that's according to it's appropriate network | 14:17 |
| jaosorior | mandre: can you do facter -p $fqdn_internal_api ? | 14:17 |
| jaosorior | mandre: what's the result? | 14:18 |
| jaosorior | sorry | 14:18 |
| jaosorior | without the $ | 14:18 |
| jaosorior | facter -p fqdn_internal_api | 14:18 |
| jaosorior | mandre: also, can you file a bug? | 14:18 |
| shardy | nvm, found it | 14:18 |
| mandre | jaosorior: facter -p fqdn_internal_api returns nothing | 14:19 |
| *** ccamacho|lunch is now known as ccamacho | 14:20 | |
| openstackgerrit | Merged openstack/diskimage-builder: Shorten DHCP timeout in dhcp-all-interfaces https://review.openstack.org/375073 | 14:20 |
| shardy | Do we still need controller-config.yaml and controller-config-pacemaker.yaml? | 14:20 |
| shardy | seems like we can combine them into a common template as the only difference is the path for the package_manifest provider | 14:20 |
| jaosorior | mandre: ah... true.. that fact is only available in puppet-tripleo | 14:21 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add Removal of Sahara during upgrade https://review.openstack.org/375517 | 14:21 |
| mandre | jaosorior: but there is a internal_api_virtual_ip hieradata if that helps | 14:22 |
| marios | tosky: ^ cant find you in gerrit to add you on review | 14:22 |
| tosky | marios: uuuh | 14:22 |
| tosky | but I'm there | 14:22 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable https://review.openstack.org/375519 | 14:23 |
| marios | tosky: i am also in mad rush cos have scrum in 5 mins and didn't prep :) | 14:23 |
| marios | tosky: so fyi | 14:23 |
| tosky | marios: I would say that "for $service in " is an unrelated fix, but I would not complain | 14:23 |
| mandre | jaosorior: or even the cloud_name_internal_api hieradata that contains the hostname | 14:25 |
| marios | tosky: ack | 14:25 |
| tosky | no, but it's important to have anyway | 14:25 |
| tosky | otherwise services are not stopped | 14:25 |
| EmilienM | panda: ipv6 job still timeouting at step5 | 14:25 |
| marios | tosky: i can split noproblem, just wanted to get it out | 14:26 |
| marios | tosky: will revisit later add a comment thanks | 14:26 |
| tosky | as I said, leaving it to you, no strong push there | 14:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable https://review.openstack.org/375519 | 14:27 |
| b00tcat | hi - I'm testing some changes I did on the tripleo-heat-templates manifests and had some syntax error | 14:27 |
| b00tcat | so I SSHed into the overcloud controller and did `puppet apply /var/lib/heat-config/heat-config-puppet/*.pp` | 14:27 |
| b00tcat | however I'm getting "Error: Could not find data item step in any Hiera data file and no default supplied at blablabla" | 14:28 |
| tosky | apart from that - argh, array manipulated as strings in bash | 14:28 |
| b00tcat | it is not enough to do `puppet apply` ? | 14:28 |
| shardy | b00tcat: we pass a "step" variable to puppet when the puppet apply is done via heat | 14:28 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network https://review.openstack.org/375523 | 14:29 |
| jaosorior | mandre: that should do the trick ^^ | 14:29 |
| shardy | you can hack the step you want into a file in /etc/puppet/hieradata, then do puppet apply if you want for debugging | 14:29 |
| shardy | remember to remove it afterwards ;) | 14:29 |
| jaosorior | mandre: I need a bug number to be able to get it to newton though | 14:29 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings https://review.openstack.org/370573 | 14:29 |
| shardy | or pass the step variable on the CLI I guess | 14:29 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 14:32 |
| b00tcat | shardy: let me check if I got you - so basically I can hardcode the step "I'm in" on the hieradata, then do puppet apply and everything will go smooth? | 14:33 |
| *** pkovar has quit IRC | 14:33 | |
| shardy | b00tcat: yes, you have to pass the step you want to test to puppet, then it will do the same as if heat applied it | 14:33 |
| trown | panda: so far so good on just making extra networks to get tripleo-ci netiso working on top of quickstart undercloud, deploy made it past the network config phase anyways | 14:34 |
| b00tcat | shardy: got it, thanks! | 14:34 |
| panda | trown: your previously deploys hung on NetworkDeployment step ? | 14:35 |
| panda | EmilienM: any guess on what is haning on step5 ? | 14:36 |
| trown | panda: ya when I only added one extra network so there would be an eth2 on the undercloud it hung there | 14:36 |
| EmilienM | panda: I'm looking | 14:36 |
| trown | panda: but with 6 overcloud networks, it seems happy | 14:36 |
| mandre | jaosorior: sorry, was on the phone, I'm filling the bug right now | 14:37 |
| jaosorior | mandre: awesome | 14:37 |
| panda | bnemec: :( | 14:38 |
| bandini | panda: you get something hung on NetworkDeployment step? | 14:41 |
| panda | bandini: yes, but mainly because I'm trying to apply trieplo-ci test environemtn file to a virt env generated by quickstart | 14:42 |
| *** mah has joined #tripleo | 14:43 | |
| jaosorior | bandini: galera is still using pacemaker, right? | 14:43 |
| mah | Hi all, I am using opnfv-apex which is based on tripleo.. but I got an issue stucked at the end (post install configuration) configuring neutron external network | 14:43 |
| mandre | jaosorior: https://bugs.launchpad.net/tripleo/+bug/1627060 | 14:44 |
| openstack | Launchpad bug 1627060 in tripleo "Can't connect to mariadb in nonha haproxy setup" [Undecided,New] | 14:44 |
| jaosorior | mandre: awesomeness. I'll use that | 14:44 |
| trown | panda: http://chunk.io/f/572a3c53beb34a69bc4482588dafba67 is the config I used to setup quickstart env with 6 overcloud networks | 14:44 |
| mah | and I tried to do openstack user list , but got stucked too .. stucked at making authentication to httP://ip:5000/v2.0/tokens | 14:44 |
| mah | any help please ? | 14:45 |
| mandre | jaosorior: and I'll try your patch | 14:45 |
| *** florianf has quit IRC | 14:45 | |
| jaosorior | mandre: please do. Only thing is that it's incomplete. I'm not taking into account HA | 14:46 |
| jaosorior | mandre: so I gotta update it. But the nonha part should stay the same | 14:46 |
| bandini | jaosorior: yes | 14:46 |
| mandre | jaosorior: ack | 14:47 |
| jaosorior | bandini: thanks | 14:47 |
| bandini | jaosorior: ;) np | 14:47 |
| panda | trown: thanks. I plan o understand a bit more what is expected from networks .. | 14:48 |
| trown | panda: so far it seems like, just having the interfaces connected | 14:49 |
| *** Ryjedo has joined #tripleo | 14:49 | |
| trown | panda: since we are using a version of multi-nic and not single-nic-vlans, there is not really anything extra to setup | 14:49 |
| trown | panda: but I still wouldnt mind sitting in with you and bnemec, networking is my weakest area | 14:50 |
| panda | trown: do you know if undercloud is still acting as default route ? | 14:51 |
| *** pkovar has joined #tripleo | 14:51 | |
| openstackgerrit | Ryan Brady proposed openstack/python-tripleoclient: [WIP] Change tripleoclient to use password generation from workflows https://review.openstack.org/375544 | 14:52 |
| trown | panda: ya, https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/network-templates/nic-configs/controller.yaml#L61-L64 | 14:52 |
| trown | panda: and we set that on the undercloud interface in deploy.sh: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/deploy.sh#L81-L98 | 14:53 |
| jaosorior | mandre: note that the t-h-t patch depends on a puppet-tripleo patch | 14:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable https://review.openstack.org/375519 | 14:53 |
| mandre | jaosorior: ohhh I totally missed that | 14:53 |
| *** florianf has joined #tripleo | 14:54 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network https://review.openstack.org/375523 | 14:55 |
| *** jlinkes has quit IRC | 14:56 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network https://review.openstack.org/375523 | 14:56 |
| gfidente | mwhahaha, not sure if you have a minute but have any clue why 4.5 is failing https://review.openstack.org/#/c/371756/ ? | 14:57 |
| mwhahaha | gfidente: missing facts | 14:58 |
| mwhahaha | gfidente: i'll find the specifics and comment | 14:58 |
| gfidente | so is this something changed in puppet 4.5 | 14:58 |
| gfidente | ? | 14:58 |
| gfidente | thanks btw :) | 14:59 |
| mwhahaha | yea in 4 it doesn't like it when vercmp is given an undef or a '' | 14:59 |
| openstackgerrit | Martin Mágr proposed openstack/instack-undercloud: Revert "Deploy Monitoring on the undercloud with Puppet" https://review.openstack.org/375557 | 14:59 |
| gfidente | mwhahaha, eh I was looking into that but we seem to do versioncmp with operatingsystemmajrelease | 15:00 |
| gfidente | can that ever be '' ? | 15:00 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 15:00 |
| mwhahaha | yea that's a fact so in rspec that has to be provided | 15:00 |
| mwhahaha | not sure why your change is triggering that | 15:00 |
| *** milan has quit IRC | 15:01 | |
| mwhahaha | might have been an apache change that broke stuff | 15:01 |
| gfidente | right doesn't seem related to the change itself | 15:01 |
| mwhahaha | yea let me propose a fix and i'll rebase that change on top of it | 15:01 |
| mwhahaha | cause we'll probably have to backport the fix | 15:02 |
| *** jistr is now known as jistr|call | 15:03 | |
| jaosorior | gfidente, bandini: Do you guys know if galera can take hostnames in the gmcast.listen_addr parameter? | 15:03 |
| bandini | dciabrin: ^ | 15:04 |
| bandini | jaosorior: I'd be surprised if it could not, but maybe Damien knows for sure? | 15:04 |
| bnemec | EmilienM: panda: I just got a successful ping test on an ipv6 overcloud: http://paste.openstack.org/show/582766/ | 15:04 |
| jaosorior | bandini: the documentation is a bit lacking | 15:04 |
| EmilienM | bnemec: woot | 15:04 |
| panda | bnemec: any significant changes from what we're testing ? | 15:05 |
| EmilienM | bnemec: nice :) | 15:05 |
| EmilienM | yeah, why does it timeout in CI? | 15:05 |
| *** abehl has quit IRC | 15:05 | |
| EmilienM | bnemec: are you running the same envs as in https://review.openstack.org/#/c/363674/ ? | 15:05 |
| mah | hi all, openstack commands stucks always here --> Making authentication request to http://192.168.162.13:5000/v2.0/tokens | 15:05 |
| mah | anyhelp ? | 15:05 |
| dciabrin | bandini, jaosorior checking at once | 15:06 |
| bnemec | panda: EmilienM: I had to bump my overcloud memory to 7 GB. I kept OOMing at 6 yesterday. | 15:07 |
| bnemec | That's likely part of why CI keeps failing on thos patches. | 15:07 |
| EmilienM | bnemec: why doesn't it fail on regular ha job? | 15:07 |
| EmilienM | or at least not much | 15:08 |
| EmilienM | ipv6 always timeout | 15:08 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common https://review.openstack.org/368150 | 15:08 |
| openstackgerrit | Martin Mágr proposed openstack/instack-undercloud: Revert "Deploy Monitoring on the undercloud with Puppet" https://review.openstack.org/375557 | 15:08 |
| EmilienM | bnemec: can you double check panda's patch to make sure we use the right environments? | 15:08 |
| bnemec | EmilienM: I've seen quite a few failures on the regular ha job too. | 15:08 |
| bnemec | It's possible I have more cpus on my overcloud vms too, which would make it worse. | 15:08 |
| bnemec | Yeah, I have 2 cpus per overcloud node, CI only has 1. | 15:10 |
| bnemec | But I saw similar hangs when I had 6 GB overcloud nodes too. | 15:10 |
| mwhahaha | gfidente: yea puppet ceph tests are hosed, i'll propose a fix | 15:10 |
| bandini | gfidente: https://paste.fedoraproject.org/433539/47464349/ here is my last finding | 15:11 |
| bandini | matbu: ^ | 15:11 |
| EmilienM | dprince: I see you assigned on https://bugs.launchpad.net/tripleo/+bug/1620595 | 15:12 |
| openstack | Launchpad bug 1620595 in tripleo "When Heat API services run on a different node than controller the heat-dbsync fails due to incorrect mysql permissions" [High,In progress] - Assigned to Dan Prince (dan-prince) | 15:12 |
| EmilienM | dprince: do you have WIP about the db_sync issue mcornea reported? | 15:12 |
| *** leanderthal is now known as leanderthal|afk | 15:12 | |
| dprince | EmilienM: I will look at it. now WIP yet though but it will be similar to the keystone issue I think | 15:15 |
| dprince | no WIP yet | 15:15 |
| fultonj | puppet ceph tests are hosed, mwhahaha to propose fix | 15:16 |
| dprince | EmilienM: we'll have to change the puppet manifests for this I think too | 15:16 |
| dciabrin | jaosorior, it looks like it can. I can bootstrap an ipv4 cluster with hostname in it, galera version 25.3.5-7 | 15:17 |
| matbu | bandini: works, if you don't make the BlockStorageUpgradeDeployment ? | 15:17 |
| *** saneax is now known as saneax-_-|AFK | 15:17 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Separate Template Processing From Create/Update Plan https://review.openstack.org/370868 | 15:17 |
| dciabrin | bandini, jaosorior what's the issue with having ip values? | 15:17 |
| jaosorior | dciabrin: no issue | 15:18 |
| jaosorior | dciabrin: hope there's no issue with using hostnames either :D | 15:18 |
| dciabrin | jaosorior, haha :) | 15:18 |
| *** rcernin has quit IRC | 15:19 | |
| *** lucasagomes is now known as lucas-hungry | 15:20 | |
| *** zaneb has quit IRC | 15:23 | |
| mah | hi all, openstack commands stucks always here --> Making authentication request to http://192.168.162.13:5000/v2.0/tokens | 15:24 |
| *** pkovar has quit IRC | 15:25 | |
| *** aufi has quit IRC | 15:25 | |
| bandini | matbu: ok found a workaround for the issue ;) | 15:25 |
| bandini | I don't *why* it works, but we'll figure that out eventually | 15:26 |
| *** jistr|call is now known as jistr | 15:26 | |
| *** bana_k has joined #tripleo | 15:27 | |
| *** jistr is now known as jistr|biab | 15:28 | |
| matbu | bandini: lol cool | 15:29 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Update minimal create docs payload to use overcloud prep roles https://review.openstack.org/374231 | 15:31 |
| *** ebarrera has quit IRC | 15:32 | |
| *** rajinir has joined #tripleo | 15:32 | |
| *** chem` has joined #tripleo | 15:33 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password https://review.openstack.org/374892 | 15:34 |
| *** chem has quit IRC | 15:34 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: WIP Work around stuck get_param calls during major-pacemaker-upgrade https://review.openstack.org/375576 | 15:34 |
| *** jistr|biab is now known as jistr | 15:36 | |
| *** mcornea has quit IRC | 15:38 | |
| gfidente | bandini++ on https://review.openstack.org/#/c/375576/1 | 15:41 |
| gfidente | :) | 15:41 |
| zoliXXL | have a good weekend | 15:41 |
| gfidente | marios, jistr ^^ sounds like we want to tag tag tripleo/rc2 and merge quickly | 15:42 |
| bandini | gfidente: note that we have many more instances https://paste.fedoraproject.org/433563/45356147/ | 15:42 |
| gfidente | bandini, right, of stuff we don't test in CI indeed :) | 15:43 |
| *** zoliXXL is now known as zoli|gone | 15:43 | |
| gfidente | they all need fixing | 15:43 |
| bandini | gfidente: ack I will submit v2 | 15:43 |
| bandini | I think a heat bug is in order as well | 15:44 |
| *** zoli|gone is now known as zoli_gone-proxy | 15:44 | |
| bandini | it could barf instead of timing out | 15:44 |
| gfidente | you mean syntax validation? | 15:44 |
| jistr | gfidente, bandini: i'm a bit confused maybe, i thought we should be using get_attr for these? https://review.openstack.org/#/c/375576/1/extraconfig/tasks/major_upgrade_pacemaker.yaml | 15:44 |
| bandini | don't you think? | 15:44 |
| gfidente | jistr, no we get list of servers as param there | 15:44 |
| jistr | ah true just looked it up in docs... | 15:45 |
| jistr | thanks | 15:45 |
| *** chem`` has joined #tripleo | 15:45 | |
| bandini | I will make a v2 after dinner. need to tackle the kids now | 15:46 |
| gfidente | bandini, it's friday! | 15:46 |
| jistr | i know get_param is for things passed in, get_resource is for resources, but never sure about get_attr :) | 15:46 |
| bandini | gfidente: :) amen to that | 15:46 |
| *** chem` has quit IRC | 15:46 | |
| jistr | if get_resource behaved similarly as get_param then perhaps we wouldn't need get_attr at all | 15:46 |
| shardy | jistr: get_attr always gets outputs from resources :) | 15:46 |
| shardy | jistr: where it gets confusing I guess is that outputs from nested stacks are referenced as attributes of the owning resource (in the parent stack) | 15:47 |
| gfidente | but the 'resource' is local to the parent in that case though | 15:48 |
| gfidente | local went in quotes , not resource | 15:48 |
| shardy | Yeah, the resource is a reference to the nested stack, so you can see it's outputs (as attributes) | 15:48 |
| shardy | you can't replace get_attr with get_resource, because get_resource always returns the ID of the resource | 15:48 |
| shardy | e.g the nova server ID | 15:48 |
| *** absubram has joined #tripleo | 15:48 | |
| *** yamahata has quit IRC | 15:49 | |
| shardy | I suppose you could extend it to allow {get_resource: [foo, some_attr]} | 15:49 |
| *** tremble has quit IRC | 15:49 | |
| jistr | ^ that's exactly what i meant :) | 15:49 |
| shardy | jistr: aha | 15:49 |
| jistr | and then get_attr is superfluous | 15:49 |
| *** flepied has quit IRC | 15:49 | |
| * shardy is running out of brain power, it's friday ;) | 15:49 | |
| jistr | that's what confuses me always... we have get_param which is powerful enough to fetch the outputs too, but get_resource isn't, and instead of that functionality we have get_attr | 15:50 |
| jistr | and i never remember which one is the more powerful one :) | 15:50 |
| *** dtantsur is now known as dtantsur|afk | 15:50 | |
| shardy | jistr: Yeah, well get_param and get_attr both support path based lookups | 15:51 |
| *** chem`` has quit IRC | 15:51 | |
| shardy | so you have a valid point :) | 15:51 |
| gfidente | but in that yaml we had to use _param | 15:52 |
| marios | gfidente: reading back | 15:52 |
| gfidente | so jistr is complaining on friday | 15:52 |
| marios | gfidente: for which one? manila? | 15:52 |
| gfidente | marios, https://review.openstack.org/#/c/375576/1 | 15:52 |
| panda | shardy: so, ben was able to make a test pass that is failing without apparent reason in CI just increasing the amount of RAM on the overcloud, so the ram problem may affect overcloud nodes too ... | 15:53 |
| gfidente | jistr | 15:53 |
| marios | gfidente: why the rush? i mean yes we want to land it but is upgrades related so has exception. it still has WIP? | 15:53 |
| gfidente | marios, it blocks upgrade attempts | 15:53 |
| jistr | true maybe we could un-wip :D | 15:54 |
| marios | bandini: is this the fix for the heat timetout | 15:54 |
| gfidente | jistr | 15:54 |
| dtrainor | Are any and all resources and parameters changeable using *ExtraConfig using t-h-t? Should that be the expectation when poking and prodding templates? | 15:54 |
| gfidente | don't say you took me seriously there | 15:54 |
| jistr | gfidente: yes i hear you :D | 15:54 |
| gfidente | because I don't expect you to | 15:54 |
| bandini | marios: yes | 15:54 |
| jistr | i'm complaining every day :D | 15:54 |
| gfidente | excatly | 15:54 |
| bandini | jistr: that's the spirit ;) | 15:54 |
| gfidente | that's what I meant | 15:54 |
| jistr | BNaaS bad news as a service | 15:54 |
| gfidente | ahahhaa | 15:55 |
| marios | bandini: this is fantastic cool stuff | 15:55 |
| gfidente | now nobody please | 15:55 |
| gfidente | go talking about conferencing as a service | 15:55 |
| bandini | ahahah | 15:55 |
| *** myoung is now known as myoung|lunch | 15:55 | |
| mwhahaha | gfidente, fultonj: proposed fixes for puppet-ceph and rebased the osd change so it should pass now | 15:56 |
| gfidente | mwhahaha, which one is it? was it a backward compat issue of 4.5 | 15:56 |
| gfidente | or something actually wrong in puppet-ceph we didn't notice before? | 15:56 |
| mwhahaha | gfidente: puppetlabs-apt release 2.3.0 that now uses lsbdistrelease fact which we were not defining in the tests | 15:57 |
| mwhahaha | it's not actually a functional issue just tests | 15:57 |
| EmilienM | gfidente: https://review.openstack.org/#/c/375584/ | 15:57 |
| Jokke_ | any chance to get review eyes on this https://review.openstack.org/#/c/374961/ ? | 15:57 |
| bnemec | Man, if we get BNaaS I can retire early. | 15:57 |
| bnemec | That is Ben Nemec as a Service, right? ;-) | 15:57 |
| *** bnemec is now known as beekneemech | 15:58 | |
| beekneemech | And hey, it's Friday! | 15:58 |
| *** jpena is now known as jpena|off | 15:58 | |
| *** Goneri has joined #tripleo | 15:58 | |
| *** jpena|off is now known as jpena | 15:59 | |
| Jokke_ | beekneemech: indeed :( | 15:59 |
| *** ebarrera has joined #tripleo | 16:04 | |
| *** dprince has quit IRC | 16:05 | |
| gfidente | now guys | 16:06 |
| gfidente | I will miss you during the weekend | 16:07 |
| gfidente | but I think I am leaving for today | 16:07 |
| gfidente | though I migh send you some review requests on a late night saturday | 16:07 |
| gfidente | who knows | 16:07 |
| gfidente | pretend everything is urgent for rc2 | 16:07 |
| gfidente | to add some last minute bugs | 16:07 |
| gfidente | as I am used to | 16:07 |
| hewbrocca | gfidente: yessss moar bugz | 16:09 |
| marios | gfidente: have a good one faidentee | 16:09 |
| *** rasca has quit IRC | 16:12 | |
| *** hewbrocca is now known as hewbrocca-afk | 16:12 | |
| *** bana_k has quit IRC | 16:14 | |
| *** bkopilov has quit IRC | 16:14 | |
| *** ohamada has quit IRC | 16:15 | |
| *** fultonj has quit IRC | 16:15 | |
| jaosorior | mandre: so my commit fails cause the hostnames used in the multinode gate get too long :(. I'll fix it on monday. | 16:16 |
| mandre | jaosorior: right now, I'm having issue with the upload-puppet-module script that doesn't copy anything so I couldn't actually test your patch | 16:17 |
| *** gfidente has left #tripleo | 16:18 | |
| *** gfidente has joined #tripleo | 16:18 | |
| *** jistr is now known as jistr|afk | 16:18 | |
| jaosorior | mandre: so upload-puppet-module script is broken it seems | 16:19 |
| jaosorior | I worked around it by deleting this from upload-swift-artifact: | 16:19 |
| EmilienM | gfidente: lol | 16:19 |
| gfidente | EmilienM, oh you you | 16:19 |
| gfidente | you | 16:19 |
| mandre | jaosorior: yeah, I'm currently looking into it | 16:19 |
| mandre | jaosorior: good to know | 16:19 |
| gfidente | why you hate me for the puppet openstack integration comment | 16:20 |
| jaosorior | mandre: https://github.com/openstack/tripleo-common/blob/master/scripts/upload-swift-artifacts#L98 | 16:20 |
| jaosorior | I deleted that one | 16:20 |
| jaosorior | and only used the second one | 16:20 |
| EmilienM | gfidente: where? | 16:20 |
| jaosorior | getting it off the if | 16:20 |
| gfidente | I get it's different purpose from tripleo-ci | 16:20 |
| gfidente | and politically not nice to share ci | 16:20 |
| jaosorior | mandre: anyway, that should get it working | 16:20 |
| jaosorior | I gotta go now | 16:20 |
| jaosorior | happy weekend everyone! | 16:20 |
| mandre | jaosorior: cool, will try it | 16:20 |
| gfidente | but we'd be repeating lots of similar studd | 16:20 |
| mandre | jaosorior: enjoy | 16:20 |
| gfidente | EmilienM, anyway, we can talk another day | 16:20 |
| *** jaosorior has quit IRC | 16:21 | |
| EmilienM | gfidente: oh I see | 16:21 |
| EmilienM | gfidente: I agree it would be nice to share things but I don't see how here | 16:22 |
| EmilienM | gfidente: the scenario job is still a good start :) | 16:22 |
| gfidente | how is the problem agreed | 16:22 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common https://review.openstack.org/368150 | 16:23 |
| *** gfidente has quit IRC | 16:25 | |
| EmilienM | we have 2 patches to help with puppet tripleo unit testing, please review them https://review.openstack.org/#/c/374357/ and https://review.openstack.org/#/c/374916/ | 16:28 |
| *** yamahata has joined #tripleo | 16:29 | |
| *** lucas-hungry is now known as lucasagomes | 16:29 | |
| openstackgerrit | Merged openstack/puppet-tripleo: certmonger: improve orchestration for puppet4 https://review.openstack.org/373513 | 16:32 |
| EmilienM | ayoung: taking it here since it's about tripleo. If you plan to do the tripleoclient thing, make sure upgrade work. My initial patch was not working for upgrades, I missed it. See https://bugs.launchpad.net/tripleo/+bug/1626422 | 16:33 |
| openstack | Launchpad bug 1626422 in tripleo "Failed to validate nested template: Property error: resources[10].properties: Property KeystoneCredential0 not assigned" [Critical,In progress] - Assigned to mbu (mat-bultel) | 16:33 |
| EmilienM | matbu is working on it: https://review.openstack.org/#/c/374892/ | 16:33 |
| EmilienM | so I guess we'll need this patch before doing fernets | 16:33 |
| ayoung | EmilienM, good to know. I was thinkg more like 1. Make sure we could enable Fernet for a new install and then 2. make it default, to include upgrades. Is that a reasonable path | 16:34 |
| ccamacho | Guys enjoy the weekend!!!! | 16:35 |
| ccamacho | ciao! | 16:35 |
| ayoung | EmilienM, also, limiting it to 2 credentials might be problematic. | 16:35 |
| ayoung | EmilienM, I was thinking through this. Credentials are different from Fernet Keys. Fernet is always 2 (or so) but credentials might be an evergrowing list | 16:36 |
| EmilienM | ayoung: I see | 16:37 |
| ayoung | EmilienM, there is always one active credential that you use to encrypt new passwords, but for old ones, you need to make sure that they are all decryptable. | 16:37 |
| EmilienM | ayoung: again, it was a first iteration | 16:37 |
| ayoung | I'm going to discuss that in upstream Keystone at the tsummit | 16:37 |
| EmilienM | I guess we need to improve it in Ocata | 16:37 |
| ayoung | I think we want a tool in keystone-manage to update all the passwords to the latest keys so we can dump old ones | 16:37 |
| EmilienM | what is worries me is how to transfer the key to all keystone servers/containers | 16:38 |
| EmilienM | it would be cool if we could use swift as a backend storage | 16:39 |
| EmilienM | directly in keystone | 16:39 |
| EmilienM | instead of local file | 16:39 |
| ayoung | EmilienM, I'm discussing that, too. It will be similar for both Fernet and credentials | 16:39 |
| *** myoung|lunch is now known as myoung | 16:41 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 16:41 |
| ayoung | EmilienM, I can see a solution that uses PKI, but it will take some coding. It would also take each of the Keystone instances producing an X509 cert that would be used as part of an encryption process. I want to discuss with the Keystone team, and maybe we can come up with something less heavyweight | 16:41 |
| EmilienM | in any way, I don't think storing certificates on the machine is good | 16:42 |
| EmilienM | we should use external storage to make it accessible from all keystone services | 16:42 |
| EmilienM | swift has proven to be stable and secured, why not using it? | 16:42 |
| *** b00tcat has quit IRC | 16:43 | |
| ayoung | EmilienM, So, the way that PKI works is that each machine has to have its own secure storage for Private Keys. They never leave that machine. But the certificate can and should leave the machine, as it only has public information. | 16:43 |
| ayoung | That is actually the case for far larger uses than Keystone, to include all of the HTTPS/TLS work that jaosorio is doing. | 16:44 |
| *** paramite has quit IRC | 16:44 | |
| ayoung | It makes the workflow really painful, though, and that is why I don't want to do it if we don't have to | 16:45 |
| EmilienM | I'm afk for lunch, we can talk later | 16:45 |
| ayoung | EmilienM, the problem with Swift is that to access it, the overcloud keystone server needs a credential that is accepted in the undercloud | 16:45 |
| ayoung | We don't have that today. So the alternative is insecure | 16:46 |
| *** fultonj has joined #tripleo | 16:47 | |
| *** ccamacho is now known as ccamacho|out | 16:49 | |
| mah | hi all, at deployment opnfv-apex which is based on tripleo ,, the post instal configuration stuck at Configuring Neutron external Network ... any solutions ? | 16:50 |
| trown | panda: beekneemech, should IPv6 in CI be using https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/ipv6-network-templates/network-environment.yaml instead of the packaged net-multiple-nics-v6.yaml https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L112 | 16:50 |
| *** dmacpher is now known as dmacpher-afk | 16:51 | |
| beekneemech | trown: Yes, see https://review.openstack.org/374406 | 16:51 |
| beekneemech | We just never updated the updates job for ovb. | 16:52 |
| trown | beekneemech: awesome giving that a go | 16:53 |
| beekneemech | Unfortunately all of the ipv6-based jobs are failing for some reason right now. :-/ | 16:53 |
| *** jpena is now known as jpena|off | 16:54 | |
| *** derekh has quit IRC | 16:56 | |
| *** abregman has joined #tripleo | 16:56 | |
| trown | beekneemech: k, I am just testing the IPv6 net-iso part (not updates), and will provide feedback on the patch | 16:56 |
| *** bana_k has joined #tripleo | 16:56 | |
| *** jistr|afk is now known as jistr | 16:59 | |
| *** ccamacho|out has quit IRC | 17:05 | |
| *** trown is now known as trown|lunch | 17:06 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common https://review.openstack.org/368150 | 17:16 |
| *** athomas has quit IRC | 17:22 | |
| *** ebarrera has quit IRC | 17:27 | |
| *** jpich has quit IRC | 17:29 | |
| *** tosky has quit IRC | 17:35 | |
| *** akshai has quit IRC | 17:35 | |
| *** florianf has quit IRC | 17:36 | |
| *** akshai has joined #tripleo | 17:37 | |
| jrist | honza: do you have a trick for testing this https://review.openstack.org/#/c/370765/ or have you actually gotten a full deploy to work | 17:37 |
| honza | jrist: i have a full deploy working and just refresh the page when i change the code, sorry | 17:38 |
| honza | jrist: it's using the heat api so i don't think you can really fake it | 17:38 |
| honza | (easily) | 17:38 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: get_param calls with multiple arguments need brackets around them https://review.openstack.org/375576 | 17:39 |
| jrist | honza: I'm jealous. | 17:42 |
| jrist | honza: got a screenshot? | 17:42 |
| honza | jrist: one sec | 17:43 |
| *** jbadiapa has quit IRC | 17:44 | |
| openstackgerrit | Merged openstack/tripleo-common: Remove the old, deprecated Mistral action names https://review.openstack.org/366529 | 17:44 |
| openstackgerrit | Merged openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant https://review.openstack.org/373477 | 17:44 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove hard-coded roles in EnabledServices output https://review.openstack.org/375339 | 17:45 |
| honza | jrist: http://i.imgur.com/zvXwl41.png | 17:46 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function https://review.openstack.org/375648 | 17:47 |
| jrist | good stuff | 17:47 |
| jrist | no validations! | 17:47 |
| jrist | :) | 17:47 |
| *** florianf has joined #tripleo | 17:47 | |
| honza | jrist: *shrug* | 17:48 |
| jrist | wait a minute | 17:48 |
| jrist | how'd you get the deployment to pass | 17:48 |
| jrist | if you have no nodes | 17:48 |
| jrist | lol | 17:48 |
| honza | magic | 17:48 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-common: Remove references to overcloud-without-mergepy https://review.openstack.org/375540 | 17:48 |
| honza | I think it was deployed via CLI | 17:49 |
| * jrist grumbles | 17:49 | |
| *** egafford has joined #tripleo | 17:49 | |
| *** flepied has joined #tripleo | 18:02 | |
| *** trown|lunch is now known as trown | 18:06 | |
| *** rhallisey has quit IRC | 18:08 | |
| EmilienM | can someone review this quick puppet4 fix? https://review.openstack.org/#/c/374339/ | 18:08 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 18:11 |
| *** florianf has quit IRC | 18:12 | |
| openstackgerrit | John Eckersberg proposed openstack/tripleo-puppet-elements: Add puppet-qdr module https://review.openstack.org/373488 | 18:14 |
| openstackgerrit | John Eckersberg proposed openstack/tripleo-puppet-elements: Add qpid-dispatch-router to overcloud-controller element https://review.openstack.org/373489 | 18:14 |
| dtrainor | I've been battling with a deployment that just stalls at "Running /var/lib/heat-config/hooks/puppet < /var/lib/heat-config/deployed/51333d0a-cbc9-4c9a-8f68-9799d4bdf302.json" for both my Compute and my Controller nodes. | 18:27 |
| dtrainor | I found this https://bugs.launchpad.net/heat/+bug/1488366 and this https://bugzilla.redhat.com/show_bug.cgi?id=1262189 but I can't see how they would apply | 18:28 |
| openstack | Launchpad bug 1488366 in tripleo "TripleO Deployments get stuck sometimes until a timeout is reached" [High,Fix released] | 18:28 |
| openstack | bugzilla.redhat.com bug 1262189 in openstack-puppet-modules "Overcloud Deployment fails: either hangs or raises resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]" [High,Closed: errata] - Assigned to ichavero | 18:28 |
| dtrainor | The hanging does in fact happen on the ComputeNodesPostDeployment and ControllerNodesPostDeployment resources. | 18:29 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 18:33 |
| *** rcarrillocruz has quit IRC | 18:40 | |
| *** rcarrillocruz has joined #tripleo | 18:45 | |
| *** jeckersb is now known as jeckersb_gone | 18:49 | |
| beagles | dtrainor: I ran into that once... but I think I had some things out of sync. I couldn't see anything obviously wrong, it just wasn't going anywere. | 18:56 |
| beagles | or anywhere rather | 18:56 |
| dtrainor | i may have stumbled upon something here | 18:56 |
| beagles | dtrainor: it was also with some stuff I was working on. I rebased, updated my images and then it worked... | 18:57 |
| beagles | dtrainor: not at all helpful I know | 18:57 |
| dtrainor | hehe no worries | 18:58 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 18:58 |
| dtrainor | i think it has to do witht he order in which templates are applied, coupled with a (seemingly) infinite timeout | 18:59 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add support for rabbit hosts to mistral https://review.openstack.org/375461 | 19:08 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add FQDN testcase in swift proxy profile rspec tests https://review.openstack.org/374150 | 19:09 |
| *** david-lyle_ has joined #tripleo | 19:10 | |
| EmilienM | beekneemech, slagle: can you guys review https://review.openstack.org/#/c/374339/ and https://review.openstack.org/#/c/374916/ please? easy patches | 19:12 |
| EmilienM | beekneemech: also that one, https://review.openstack.org/#/c/374982/ if the format is ok for you | 19:12 |
| *** david-lyle has quit IRC | 19:13 | |
| beekneemech | EmilienM: I'm curious, why use tee when you're going to throw away stdout anyway? | 19:14 |
| EmilienM | beekneemech: i don't have an answer, I was using sudo in Puppet CI and copy pasted this command. We could also redirect I guess | 19:14 |
| beekneemech | EmilienM: Ah, copy pasta from sudo makes sense. I don't really care, I was just wondering if I missed something. | 19:15 |
| EmilienM | beekneemech: I'm wondering why we had this csv format | 19:15 |
| EmilienM | is it something we can exploit? | 19:16 |
| * EmilienM afk | 19:17 | |
| beekneemech | EmilienM: No idea. I assumed we copied whatever devstack was doing with dstat. | 19:17 |
| *** r-mibu has quit IRC | 19:17 | |
| *** r-mibu has joined #tripleo | 19:17 | |
| *** abregman is now known as abregman|afk | 19:18 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Add in rspec-puppet-facts support https://review.openstack.org/374357 | 19:25 |
| openstackgerrit | Merged openstack/python-tripleoclient: Stop plan creation when container exists https://review.openstack.org/369623 | 19:26 |
| slagle | EmilienM: does the FF / rc bugs only, not apply to puppet-tripleo? | 19:36 |
| slagle | or is puppet-tripleo already branched? | 19:36 |
| *** jeckersb_gone is now known as jeckersb | 19:41 | |
| *** absubram has quit IRC | 19:41 | |
| mwhahaha | no i don't believe it's branched yet | 19:42 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 19:55 |
| *** akrivoka has quit IRC | 19:55 | |
| *** absubram has joined #tripleo | 19:56 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 19:57 |
| *** absubram_ has joined #tripleo | 19:57 | |
| *** absubram has quit IRC | 20:01 | |
| *** absubram_ is now known as absubram | 20:01 | |
| *** david-lyle_ is now known as david-lyle | 20:06 | |
| *** lucasagomes is now known as lucas-afk | 20:09 | |
| *** absubram has quit IRC | 20:10 | |
| *** rajinir has quit IRC | 20:15 | |
| *** akshai has quit IRC | 20:15 | |
| *** akshai has joined #tripleo | 20:20 | |
| *** jayg is now known as jayg|g0n3 | 20:23 | |
| *** mburned is now known as mburned_out | 20:38 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 20:39 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Add FixedIPs parameter to from_service.yaml https://review.openstack.org/375721 | 20:53 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: dstat: improve output to track high cpu process https://review.openstack.org/374982 | 20:55 |
| openstackgerrit | Merged openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper https://review.openstack.org/374916 | 20:55 |
| *** trown is now known as trown|outtypewww | 21:09 | |
| *** kberger has quit IRC | 21:13 | |
| *** kberger has joined #tripleo | 21:13 | |
| *** pradk has quit IRC | 21:14 | |
| *** thrash is now known as thrash|g0ne | 21:21 | |
| *** myoung is now known as myoung|afk | 21:21 | |
| *** coolsvap has quit IRC | 21:22 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo: Add aodh profile rspec testing https://review.openstack.org/374402 | 21:30 |
| *** abregman|afk has quit IRC | 21:31 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Include ceilometer in swift proxy pipeline https://review.openstack.org/371950 | 21:38 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Add swift proxy for ceilometer middleware https://review.openstack.org/371591 | 21:39 |
| *** cdearborn has quit IRC | 21:45 | |
| *** rlandy has quit IRC | 21:49 | |
| *** fultonj has quit IRC | 21:52 | |
| *** yamahata has quit IRC | 21:57 | |
| *** yamahata has joined #tripleo | 21:58 | |
| *** kberger has quit IRC | 22:02 | |
| *** jrist has quit IRC | 22:08 | |
| *** akshai has quit IRC | 22:25 | |
| *** Ryjedo has quit IRC | 22:51 | |
| *** jrist has joined #tripleo | 23:00 | |
| openstackgerrit | Sarath Kumar Sankaran Kutty proposed openstack/os-net-config: Add bonding_options attribute to NFVSwitch https://review.openstack.org/375763 | 23:00 |
| *** akshai has joined #tripleo | 23:03 | |
| *** jrist has quit IRC | 23:03 | |
| *** jrist has joined #tripleo | 23:04 | |
| *** jrist has joined #tripleo | 23:04 | |
| *** saneax-_-|AFK is now known as saneax | 23:18 | |
| *** abehl has joined #tripleo | 23:37 | |
| *** akshai has quit IRC | 23:41 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu https://review.openstack.org/374366 | 23:43 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
