| *** penick has joined #tripleo | 00:01 | |
| *** dmacpher has joined #tripleo | 00:04 | |
| *** rlandy has quit IRC | 00:05 | |
| *** rlandy has joined #tripleo | 00:23 | |
| *** derekh has quit IRC | 00:26 | |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-common: Revert "Pin puppet heat" https://review.openstack.org/272848 | 00:29 |
|---|---|---|
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 00:30 |
| *** jcoufal has quit IRC | 00:34 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Implement str_replace to unify IPv4/IPv6 ports [DO NOT MERGE] https://review.openstack.org/272856 | 00:57 |
| *** eil397 has quit IRC | 01:01 | |
| *** rhallisey has quit IRC | 01:02 | |
| *** egafford has joined #tripleo | 01:07 | |
| *** dsneddon is now known as dsneddon_biab | 01:16 | |
| *** alop has quit IRC | 01:28 | |
| *** ccrouch has quit IRC | 01:29 | |
| *** thrash is now known as thrash|g0ne | 01:31 | |
| *** egafford has quit IRC | 01:31 | |
| *** david-lyle has quit IRC | 01:36 | |
| *** penick has quit IRC | 01:37 | |
| *** dsneddon_biab is now known as dsneddon | 01:41 | |
| *** tiswanso has joined #tripleo | 01:47 | |
| *** tiswanso has quit IRC | 01:47 | |
| *** tiswanso has joined #tripleo | 01:48 | |
| *** trozet has quit IRC | 02:08 | |
| *** egafford has joined #tripleo | 02:12 | |
| *** trozet has joined #tripleo | 02:14 | |
| *** egafford has quit IRC | 02:17 | |
| *** cwolferh has quit IRC | 02:19 | |
| openstackgerrit | ayoung proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi https://review.openstack.org/213175 | 02:20 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Implement str_replace to unify IPv4/IPv6 ports [DO NOT MERGE] https://review.openstack.org/272856 | 02:20 |
| *** egafford has joined #tripleo | 02:35 | |
| *** egafford has quit IRC | 02:40 | |
| *** yamahata has quit IRC | 02:44 | |
| *** Marga_ has quit IRC | 02:53 | |
| *** pradk has quit IRC | 02:59 | |
| *** Marga_ has joined #tripleo | 03:09 | |
| *** pradk has joined #tripleo | 03:12 | |
| *** trozet has quit IRC | 03:13 | |
| *** Marga_ has quit IRC | 03:14 | |
| *** yuanying has quit IRC | 03:21 | |
| *** Marga_ has joined #tripleo | 03:23 | |
| *** yuanying has joined #tripleo | 03:23 | |
| *** cwolferh has joined #tripleo | 03:23 | |
| *** Marga_ has quit IRC | 03:27 | |
| *** tzumainn has quit IRC | 03:27 | |
| *** yuanying has quit IRC | 03:28 | |
| *** yuanying has joined #tripleo | 03:33 | |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: puppet-stack-config: make sure heat use 'rabbit' rpc_backend https://review.openstack.org/272886 | 03:34 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: puppet-stack-config: make sure heat use 'rabbit' rpc_backend https://review.openstack.org/272886 | 03:35 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: controller: make sure heat use 'rabbit' rpc_backend https://review.openstack.org/272887 | 03:36 |
| *** Marga_ has joined #tripleo | 03:36 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common: Revert "Pin puppet heat" https://review.openstack.org/272848 | 03:37 |
| *** yuanying has quit IRC | 03:40 | |
| *** yuanying has joined #tripleo | 03:40 | |
| *** shivrao has quit IRC | 03:41 | |
| *** Marga_ has quit IRC | 03:41 | |
| *** shivrao has joined #tripleo | 03:43 | |
| *** shivrao has quit IRC | 03:44 | |
| *** shivrao has joined #tripleo | 03:47 | |
| *** shivrao has quit IRC | 03:51 | |
| *** Marga_ has joined #tripleo | 03:52 | |
| *** sthillma has quit IRC | 03:54 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements https://review.openstack.org/270597 | 03:54 |
| *** yuanying has quit IRC | 03:56 | |
| *** yuanying has joined #tripleo | 03:57 | |
| *** stendulker has joined #tripleo | 03:58 | |
| *** yuanying_ has joined #tripleo | 03:58 | |
| *** stendulker_ has joined #tripleo | 04:00 | |
| *** yuanying has quit IRC | 04:01 | |
| *** stendulker has quit IRC | 04:03 | |
| *** rlandy has quit IRC | 04:12 | |
| *** stendulker has joined #tripleo | 04:23 | |
| *** stendulker_ has quit IRC | 04:23 | |
| *** david-lyle has joined #tripleo | 04:25 | |
| *** david-lyle has quit IRC | 04:25 | |
| *** coolsvap|away is now known as coolsvap | 04:31 | |
| *** stendulker_ has joined #tripleo | 04:32 | |
| *** stendulker has quit IRC | 04:34 | |
| *** david-lyle has joined #tripleo | 04:38 | |
| *** shivrao has joined #tripleo | 04:39 | |
| *** shivrao has quit IRC | 04:43 | |
| *** cwolferh has quit IRC | 04:45 | |
| *** cwolferh has joined #tripleo | 04:45 | |
| *** shivrao has joined #tripleo | 04:49 | |
| *** masco has joined #tripleo | 04:54 | |
| *** cwolferh has quit IRC | 05:00 | |
| *** yamahata has joined #tripleo | 05:04 | |
| *** dmacpher has quit IRC | 05:08 | |
| *** rbrady has quit IRC | 05:11 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements https://review.openstack.org/270597 | 05:13 |
| *** lazy_prince has joined #tripleo | 05:51 | |
| *** penick has joined #tripleo | 05:52 | |
| *** rbrady has joined #tripleo | 05:55 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements https://review.openstack.org/270597 | 06:03 |
| *** liverpooler has quit IRC | 06:13 | |
| *** jaosorior has joined #tripleo | 06:27 | |
| *** cwolferh has joined #tripleo | 06:38 | |
| *** dshulyak has joined #tripleo | 06:42 | |
| *** larstobi has quit IRC | 06:51 | |
| *** larstobi has joined #tripleo | 06:55 | |
| *** aufi has joined #tripleo | 06:56 | |
| *** shivrao has quit IRC | 06:58 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements https://review.openstack.org/270597 | 07:02 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 07:04 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-common: Revert "Pin puppetlabs-mysql to get CI going" https://review.openstack.org/272119 | 07:15 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-common: Revert "Pin puppet heat" https://review.openstack.org/272926 | 07:15 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 07:17 |
| *** Marga_ has quit IRC | 07:18 | |
| *** stendulker has joined #tripleo | 07:20 | |
| *** stendulker_ has quit IRC | 07:21 | |
| *** AJaeger has joined #tripleo | 07:22 | |
| AJaeger | Hi tripleo cores, for 4 of you repos, I've removed argparse, a useless import since you're not supporting python 2.6 anymore. Could you review, please? https://review.openstack.org/270375 https://review.openstack.org/270376 https://review.openstack.org/270377 https://review.openstack.org/270378 | 07:23 |
| *** oshvartz has joined #tripleo | 07:24 | |
| *** ukalifon1 has joined #tripleo | 07:24 | |
| *** rcernin has joined #tripleo | 07:27 | |
| *** penick has quit IRC | 07:27 | |
| *** penick has joined #tripleo | 07:28 | |
| *** chlong_zzz is now known as chlong | 07:31 | |
| *** jcoufal has joined #tripleo | 07:37 | |
| *** penick has quit IRC | 07:49 | |
| *** bvandenh has joined #tripleo | 07:56 | |
| openstackgerrit | Evgeny Bagdasaryan proposed openstack/tripleo-heat-templates: Add BondInterfaceOvsOptions parameter to net-config-bond.yaml https://review.openstack.org/245086 | 07:59 |
| *** liverpooler has joined #tripleo | 08:02 | |
| *** tzumainn has joined #tripleo | 08:03 | |
| *** fgimenez has joined #tripleo | 08:04 | |
| *** fgimenez has quit IRC | 08:04 | |
| *** fgimenez has joined #tripleo | 08:04 | |
| *** aufi has quit IRC | 08:07 | |
| *** aufi has joined #tripleo | 08:11 | |
| *** hjensas has quit IRC | 08:13 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable. https://review.openstack.org/260413 | 08:14 |
| *** ifarkas has joined #tripleo | 08:18 | |
| *** tzumainn has quit IRC | 08:19 | |
| *** regebro has joined #tripleo | 08:22 | |
| *** jprovazn has joined #tripleo | 08:22 | |
| *** shardy has joined #tripleo | 08:29 | |
| *** bvandenh has quit IRC | 08:30 | |
| *** mkovacik has quit IRC | 08:39 | |
| *** derekh has joined #tripleo | 08:56 | |
| *** lucas-dinner is now known as lucasagomes | 08:58 | |
| *** hjensas has joined #tripleo | 08:59 | |
| *** hjensas has quit IRC | 08:59 | |
| *** hjensas has joined #tripleo | 08:59 | |
| *** mbound has joined #tripleo | 09:00 | |
| *** hjensas has quit IRC | 09:00 | |
| *** hjensas has joined #tripleo | 09:01 | |
| *** hjensas has quit IRC | 09:01 | |
| *** hjensas has joined #tripleo | 09:01 | |
| *** devvesa has joined #tripleo | 09:04 | |
| *** cmyster has quit IRC | 09:07 | |
| *** cmyster has joined #tripleo | 09:07 | |
| *** cmyster has quit IRC | 09:07 | |
| *** cmyster has joined #tripleo | 09:07 | |
| *** gfidente has joined #tripleo | 09:10 | |
| *** gfidente has quit IRC | 09:10 | |
| *** gfidente has joined #tripleo | 09:10 | |
| *** mkovacik has joined #tripleo | 09:16 | |
| marios | :/ gerrit being weird/flaky | 09:17 |
| shardy | Yeah | 09:17 |
| marios | o/ morning thanks see you already commented on the "Make enabling of controller services configurable." https://review.openstack.org/#/c/260413/7 | 09:18 |
| *** fgimenez has quit IRC | 09:18 | |
| shardy | marios: Hey g'morning, yeah I think we need that other backport to land | 09:18 |
| shardy | it failed pingtest on one job, not sure why so I rechecked | 09:18 |
| *** Marga_ has joined #tripleo | 09:19 | |
| *** jaosorior has quit IRC | 09:19 | |
| marios | shardy: am waiting for gerrit to show me that other change you reference. also, wrt the pingtest, i am still not convinced about cirros. i still have it fail sometimes for me locally (virt env always) when i run it. | 09:19 |
| marios | shardy: it may be we want to revisit building fedora-user for example | 09:20 |
| *** jaosorior has joined #tripleo | 09:20 | |
| shardy | marios: Ok, that's weird | 09:20 |
| shardy | I thought nova used cirros for nearly all tests, so assumed it'd be solid | 09:21 |
| shardy | we use it for some heat tests too in the gate | 09:21 |
| shardy | we've always had problems using a fedora image because it takes sooo much longer to boot virt-on-virt | 09:21 |
| marios | shardy: yeah i don't know. i mean it totally makes sense since *that* is precisely what it is for, it is small file size etc etc. just saying when i tested i had issues, but randomly | 09:21 |
| shardy | marios: ack, well it'd be good to figure those out for sure | 09:22 |
| shardy | maybe a local test running in a loop overnight? | 09:22 |
| shardy | then we can at least figure out what the bad state is when it fails | 09:22 |
| shardy | I'm assuming that will be hard to do in the gate | 09:22 |
| *** fgimenez has joined #tripleo | 09:22 | |
| shardy | Is anyone else hitting https://bugs.launchpad.net/tripleo/+bug/1538254 ? | 09:23 |
| openstack | Launchpad bug 1538254 in tripleo "Error: Must pass controller_virtual_ip to Class[Tripleo::Loadbalancer]" [Undecided,New] | 09:23 |
| *** Marga_ has quit IRC | 09:23 | |
| shardy | I guess I can pass ControlFixedIPs to work around it, but I didn't have to previously, and I don't think we do in the gate | 09:23 |
| *** mcornea has joined #tripleo | 09:23 | |
| marios | shardy: yeah i'd like to find out more about why it fails too. Initially i was waiting to see if it was OK once it landed in gate and seemingly it is (seen a couple runs poking at logs) but if it continues to fail then it might point to something in tripleo/env | 09:24 |
| * shardy regrets updating all-the-things yesterday :( | 09:24 | |
| *** jaosorior has quit IRC | 09:25 | |
| gfidente | shardy, so you continue to get empty fixed_ips in the neutron port? | 09:25 |
| *** jaosorior has joined #tripleo | 09:25 | |
| shardy | gfidente: Yeah, I deleted my undercloud, rebuilt everything with tripleo.sh, and rebuilt all my images (again) | 09:25 |
| shardy | same problem | 09:25 |
| jaosorior | anybody know if the stable/liberty gate is working yet? | 09:25 |
| shardy | jaosorior: we need https://review.openstack.org/272194 before the HA job will pass | 09:26 |
| jaosorior | shardy: Well, there's my +1 already | 09:26 |
| gfidente | shardy, and is this only for the controlvirtualip? | 09:27 |
| shardy | gfidente: ControlVirtualIP and RedisVirtualIP both have empty fixed_ips | 09:27 |
| shardy | all the other ports have IPs | 09:27 |
| gfidente | but this makes no sense to me | 09:28 |
| jaosorior | gfidente: What happened? | 09:29 |
| gfidente | shardy, all other ports you mean the node ports or have you deployed using network-isolation ? | 09:30 |
| shardy | gfidente: I'm not deploying with network isolation | 09:30 |
| shardy | I mean if I do neutron port-list | 09:30 |
| shardy | the redis/controller ports have empty fixed_ips | 09:30 |
| gfidente | ok so control_virtual_ip and redis are the only two ports we create in neutron from the templates in this case | 09:31 |
| shardy | the other unnamed ones do | 09:31 |
| gfidente | yeah exactly | 09:31 |
| marios | shardy: i wana +2 this https://review.openstack.org/#/c/272194/2 but i also don't want to land it until master does | 09:31 |
| gfidente | so I'm thinking if any recent change is tricking the neutron port resource into *not* allocating any ip because it gets [] as fixed_ips property? | 09:31 |
| shardy | gfidente: Yeah, that's what it looks like, but I don't understand why the gate isn't broken | 09:32 |
| gfidente | shardy, so I am just randomly guessing, maybe gate is pinning something outside of tripleo.sh> | 09:33 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Make enabling of controller services configurable. https://review.openstack.org/260413 | 09:40 |
| marios | shardy: rebased onto /#/c/272194 | 09:40 |
| shardy | marios: thanks | 09:41 |
| *** nico_auv has joined #tripleo | 09:41 | |
| ramishra_ | shardy/gfidente: hi, passing [] as fixed_ips actually creates port without any ip. This is standard neutron behaviour. Earlier we used clean it up in the neutron port resource in heat, but we fixed that in https://review.openstack.org/#/c/256328/ | 09:41 |
| *** akrivoka has joined #tripleo | 09:41 | |
| gfidente | ramishra_, oh that explains it I think | 09:42 |
| ramishra_ | I assume that has created the issue you guys are discussing here. | 09:42 |
| shardy | ramishra_: Hmm | 09:42 |
| shardy | ramishra_: Isn't that a backwards incompatible change? | 09:42 |
| shardy | (I know, I reviewed it) | 09:42 |
| *** devvesa has quit IRC | 09:42 | |
| shardy | I saw that we removed the default=[] but at the time didn't realize the significance | 09:43 |
| ramishra_ | yeah:), We should not pass anything when we want neutron to allocate the ips. | 09:43 |
| ramishra_ | That's the default behaviour. However, I agree it's not backward compatible from the earlier template pov | 09:43 |
| *** jistr has joined #tripleo | 09:43 | |
| shardy | ramishra_: I think we'll have to fix it | 09:44 |
| shardy | I agree the new behavior is more technically correct, but we can't break existing templates | 09:44 |
| shardy | what do you think? | 09:44 |
| ramishra_ | Because during update, [] means clear the ips | 09:44 |
| ramishra_ | Yeah, If you can raise a bug I'll make the change | 09:45 |
| ramishra_ | Does that sound ok? | 09:45 |
| shardy | ramishra_: Sure, will do | 09:45 |
| shardy | ramishra_: the problem is, we want to have a parameter wired in to fixed_ips, but have it be optional | 09:46 |
| shardy | I can't see any way to do that with this change in place | 09:46 |
| shardy | e.g both an empty list or "" will pass the " | 09:46 |
| shardy | if self.FIXED_IPS in props test | 09:46 |
| ramishra_ | Yeah, but then we'll be masking neutron behaviour for ever:) | 09:47 |
| *** bvandenh has joined #tripleo | 09:47 | |
| shardy | ramishra_: I'll raise a bug and we can discuss it there and in #heat | 09:47 |
| shardy | thanks for pointing it out! | 09:47 |
| ramishra_ | sure, I'll do a quick fix. | 09:48 |
| shardy | ramishra_: | 09:53 |
| shardy | https://bugs.launchpad.net/heat/+bug/1538473 | 09:53 |
| openstack | Launchpad bug 1538473 in heat "Neutron port fixed_ips backwards incompatible change wrt "[]"" [Undecided,New] | 09:53 |
| shardy | Also this explains why we're not seeing it in the gate, we're pinned to an old heat, I'm running master | 09:53 |
| *** paramite has joined #tripleo | 09:56 | |
| *** devvesa has joined #tripleo | 09:57 | |
| *** bvandenh has quit IRC | 09:57 | |
| *** olap has joined #tripleo | 10:07 | |
| *** rebrego has joined #tripleo | 10:22 | |
| *** regebro has quit IRC | 10:22 | |
| *** rebrego is now known as regebro | 10:22 | |
| *** mgould has joined #tripleo | 10:23 | |
| *** killer_prince has joined #tripleo | 10:26 | |
| *** lazy_prince has quit IRC | 10:29 | |
| AJaeger | Hi tripleo cores, for 4 of you repos, I've removed argparse, a useless import since you're not supporting python 2.6 anymore. Could you review, please? https://review.openstack.org/270375 https://review.openstack.org/270376 https://review.openstack.org/270377 https://review.openstack.org/270378 - shardy was so kind to +2 all except 270375 | 10:32 |
| *** tosky has joined #tripleo | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:37 | |
| *** electrofelix has joined #tripleo | 10:38 | |
| *** bvandenh has joined #tripleo | 10:39 | |
| *** devvesa has quit IRC | 10:39 | |
| *** killer_prince has quit IRC | 10:40 | |
| *** lazy_prince has joined #tripleo | 10:41 | |
| openstackgerrit | Merged openstack/os-refresh-config: Remove argparse from requirements https://review.openstack.org/270378 | 10:42 |
| openstackgerrit | Merged openstack/os-cloud-config: Remove argparse from requirements https://review.openstack.org/270376 | 10:43 |
| AJaeger | thanks, derekh | 10:43 |
| derekh | AJaeger: np, thanks | 10:43 |
| openstackgerrit | Merged openstack/os-collect-config: Remove argparse from requirements https://review.openstack.org/270377 | 10:44 |
| *** bvandenh has quit IRC | 10:44 | |
| openstackgerrit | Merged openstack/os-apply-config: Remove argparse from requirements https://review.openstack.org/270375 | 10:49 |
| *** stendulker_ has joined #tripleo | 10:50 | |
| *** gchamoul has left #tripleo | 10:51 | |
| *** stendulker has quit IRC | 10:53 | |
| *** stendulker_ has quit IRC | 10:56 | |
| *** rbrady has quit IRC | 10:56 | |
| *** bvandenh has joined #tripleo | 10:57 | |
| derekh | Ok people, we can move onto a new delorean repository (tests are passing here https://review.openstack.org/#/c/229789/ ), the recheck has passed the ceph and HA job just hasn't reported back yet | 11:05 |
| derekh | we just gotta merge a few patches together and update the current-tripleo link, I'd like to do this now as its taken trown a lot of work to get to this point and we need to try and keep it like that (next step) | 11:06 |
| derekh | these are what needs to merge, I'd say lets ignore the CI, the patch that tests them all together shows them passing | 11:06 |
| derekh | puppet modules reverts - https://review.openstack.org/#/c/272926/ https://review.openstack.org/#/c/272119/2 | 11:06 |
| derekh | Remove empty value for wsrep_notify_cmd https://review.openstack.org/#/c/272149/ | 11:06 |
| derekh | updates for new heatclient https://review.openstack.org/#/c/270890/ https://review.openstack.org/#/c/272479/2 | 11:06 |
| derekh | so how about it, can we go ahead and merge these right now ? | 11:07 |
| derekh | *or as close to now as possible | 11:07 |
| shardy | derekh: ack, looking | 11:08 |
| *** hjensas has quit IRC | 11:10 | |
| shardy | Hrm, so we've made a completely backwards incompatible change to heatclient? ugh :( | 11:13 |
| derekh | shardy: ya, trown|outttypeww knows the details, but iirc something that used to be output alone, is now formatted in a table | 11:15 |
| shardy | derekh: Ok, that seems odd and wrong for -F raw | 11:15 |
| shardy | but +1 on landing all-the-things, we can potentially fix that in heatclient - IMO it's a bug | 11:15 |
| derekh | shardy: yup, now that you mention it, it does seem wrong | 11:16 |
| shardy | e.g we can potentially fix it later | 11:16 |
| derekh | shardy: yup | 11:16 |
| shardy | I'll ask trown|outttypeww to raise a heatclient bug and we'll investigate | 11:16 |
| derekh | shardy: ok | 11:16 |
| *** fgimenez has quit IRC | 11:18 | |
| shardy | derekh: All looks fine modulo the heatclient thing - how long will it take for CI to report on https://review.openstack.org/#/c/229789/ ? | 11:18 |
| shardy | if approving a bunch of stuff with failing CI it'd be nice to reference that as justification in the comments | 11:19 |
| *** trown|outttypeww is now known as trown | 11:19 | |
| derekh | shardy: its waiting on the containers job to timeout/fail , shouldn't be much longer, I'll ping back when its done | 11:20 |
| *** fgimenez has joined #tripleo | 11:20 | |
| *** fgimenez has joined #tripleo | 11:20 | |
| shardy | derekh: ack, I've got all the reviews open ready to approve ;) | 11:20 |
| trown | shardy: ya I looked into it, and I think it is actually coming from cliff | 11:20 |
| trown | shardy: and heatclient just inherits it | 11:21 |
| derekh | shardy: cool, thanks | 11:21 |
| shardy | trown: ouch, I thought we only used that for the new heat oscplugin | 11:21 |
| shardy | trown: would you be able to please raise a heatclient bug explaining the issue? | 11:21 |
| trown | shardy: sure | 11:21 |
| shardy | then we can make a call as to if a fix is possible | 11:22 |
| * derekh is tempted to ssh onto the instance running the containers test and kill the deploy command ;-) | 11:23 | |
| shardy | derekh: That test was passing a while ago, so I guess that's the next challenge ;) | 11:24 |
| derekh | shardy: ya, not sure if anybody is looking at it / caring about it | 11:25 |
| shardy | derekh: we should chat with rhallisey later - he, jpeeler and Slower have been working hard to get that working | 11:25 |
| shardy | So, I think we do need to care about it after the other stuff gets fixed | 11:26 |
| derekh | shardy: yup, we need to sort it out, otherwise its just a waist of resources | 11:26 |
| shardy | Woot! Overcloud create - DONE. | 11:28 |
| shardy | first time in two days | 11:28 |
| derekh | \o/ | 11:28 |
| shardy | ramishra_: thanks, your patch fixed my latest issue :) | 11:28 |
| ramishra_ | shardy: np:) btw what happened to the tripleo gate job for heat? | 11:30 |
| shardy | ramishra_: you can run it via check experimental | 11:30 |
| shardy | it's not been running by default for some time | 11:31 |
| ramishra_ | shardy: ok:) I thought we wanted a voting job;) | 11:31 |
| shardy | ramishra_: ideally we do, but TripleO CI just isn't reliable enough unfortunately | 11:32 |
| ramishra_ | shardy: yeah | 11:32 |
| derekh | shardy: ya, we shouldn't turn it back on until we get to a place were we arn't broken all the time | 11:33 |
| derekh | Just to reword slightly, most of the time tripleo ci is doing its job perfectly, its tripleo itself that not working. | 11:34 |
| *** devvesa has joined #tripleo | 11:34 | |
| derekh | the last 2 breakages on master were because or people ignoring CI results | 11:34 |
| shardy | hehe | 11:34 |
| derekh | *of | 11:34 |
| shardy | derekh: very true :( | 11:34 |
| shardy | I think folks just see the high recheck/false-negative rate and assume it's OK to ignore | 11:35 |
| *** gfidente has quit IRC | 11:36 | |
| derekh | shardy: yup, this is exactly what happens and it also causes part of the problem | 11:36 |
| trown | derekh: I have also seen where we merged something with 2 week old CI results which is equivalent | 11:38 |
| trown | it is a bit frustrating for a downstream consumer :) | 11:38 |
| shardy | trown: It's frustrating for everyone unfortunately | 11:39 |
| derekh | trown: yup, that happens also, its a little more forgivable but we should be careful | 11:39 |
| shardy | folks don't want to recheck after two weeks because it might take another 2 weeks to get a green run | 11:39 |
| shardy | I agree we need to be careful tho | 11:39 |
| trown | at least its not boring | 11:40 |
| shardy | lol | 11:40 |
| derekh | +1 | 11:40 |
| * trown relocating | 11:41 | |
| *** trown is now known as trown|outttypeww | 11:41 | |
| *** gfidente has joined #tripleo | 11:42 | |
| derekh | shardy: 9 minutes to timeout | 11:42 |
| * derekh goes for tes | 11:42 | |
| derekh | *tea | 11:42 |
| *** rbrady has joined #tripleo | 11:43 | |
| *** pcaruana has joined #tripleo | 11:44 | |
| *** AJaeger has left #tripleo | 11:47 | |
| *** jkraj has joined #tripleo | 11:48 | |
| derekh | shardy: https://review.openstack.org/#/c/229789/ | 11:50 |
| *** dprince has joined #tripleo | 11:52 | |
| derekh | shardy: ready to update the link when you are | 11:53 |
| shardy | [heat-api]: Could not evaluate: Cannot allocate memory - fork( | 11:55 |
| shardy | I thought we added swap already? | 11:55 |
| derekh | shardy: we did, where did you see that? | 11:55 |
| shardy | In the nonha job failure for the patch you just linked | 11:56 |
| shardy | It's odd as that should use the least memory | 11:56 |
| derekh | shardy: and it passed before the recheck, could it have been triggered by a retry or something | 11:57 |
| shardy | It's hard to say, but we shouldn't have used 1G of swap | 11:58 |
| derekh | shardy: | 11:58 |
| derekh | + free -h | 11:58 |
| derekh | total used free shared buff/cache available | 11:58 |
| derekh | Mem: 4.8G 4.4G 180M 284K 285M 222M | 11:58 |
| derekh | Swap: 1.0G 452M 571M | 11:58 |
| derekh | shardy: from that same job | 11:59 |
| shardy | Anyway, I'll land the patches so we can get things running again and investigate further | 11:59 |
| derekh | at the end of the job, something may have been killed by then | 11:59 |
| derekh | shardy: ack | 11:59 |
| shardy | weird, although it's not good that we're swapping so much | 11:59 |
| shardy | I guess that slows things down a lot | 11:59 |
| *** tzumainn has joined #tripleo | 12:00 | |
| derekh | shardy: if its memory that not accessed much it mightn't effect things much, hard to know with the details we have | 12:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove empty value for wsrep_notify_cmd https://review.openstack.org/272149 | 12:00 |
| shardy | Yeah it'd be nice to see the vmstat through the run and see how much is getting swapped out | 12:01 |
| *** egafford has joined #tripleo | 12:01 | |
| openstackgerrit | Merged openstack/tripleo-common: Update pingtest for newer heatclient https://review.openstack.org/270890 | 12:02 |
| shardy | derekh: we need another reviewer for the two reverts, or do you want me to just approve? | 12:02 |
| * derekh wonders if this would give us the numbers we want https://review.openstack.org/#/c/271218/ | 12:02 | |
| derekh | shardy: I think we can approve to move the whole things along, I'll do it | 12:03 |
| derekh | shardy: one more https://review.openstack.org/#/c/272479/2 | 12:04 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client https://review.openstack.org/272479 | 12:06 |
| openstackgerrit | Merged openstack/tripleo-common: Revert "Pin puppetlabs-mysql to get CI going" https://review.openstack.org/272119 | 12:09 |
| openstackgerrit | Merged openstack/tripleo-common: Revert "Pin puppet heat" https://review.openstack.org/272926 | 12:09 |
| derekh | ALL we have updated http://trunk.rdoproject.org/centos7/current-tripleo/ to a repo from monday | 12:09 |
| *** hjensas has joined #tripleo | 12:11 | |
| *** mgould has quit IRC | 12:14 | |
| shardy | \o/ | 12:23 |
| shardy | Nice work derekh and trown|outttypeww | 12:23 |
| *** thrash|g0ne is now known as thrash | 12:28 | |
| *** mgould has joined #tripleo | 12:29 | |
| *** pcaruana has quit IRC | 12:37 | |
| *** rhallisey has joined #tripleo | 12:37 | |
| *** Goneri has quit IRC | 12:38 | |
| *** weshay_xchat has joined #tripleo | 12:38 | |
| *** Marga_ has joined #tripleo | 12:38 | |
| *** thrash has quit IRC | 12:40 | |
| *** thrash has joined #tripleo | 12:49 | |
| *** thrash has joined #tripleo | 12:49 | |
| jistr | hey folks, anybody able to give some heat hints? trying to update from kilo to liberty, i get: Stack failed with status: resources.Controller: ValueError: resources[0]: "u'clock.redhat.com'" is not a list | 12:59 |
| *** trown|outttypeww is now known as trown | 12:59 | |
| jistr | i think the trigger here is that we changed NtpServer to be able to process an array https://github.com/openstack/tripleo-heat-templates/commit/16093c3932545b1a8d1f4572c98d6953c277b3d5 | 13:00 |
| jistr | but a string should still be a valid value for that | 13:00 |
| trown | woot, thanks derekh, shardy | 13:01 |
| jistr | i guess this is something about internal heat representation then | 13:01 |
| jistr | now the interesting thing is, even if i set it to something completely different via both parameters and parameter_defaults (an array which doesn't mention clock.redhat.com at all), i still get the error mentioning clock.redhat.com | 13:02 |
| jistr | it's as if it tried to use/validate the old parameter value in the new templates, even though i provided a new different value to the stack-update call | 13:03 |
| jistr | possibly a heat bug worth reporting? | 13:03 |
| jistr | shardy: could you check please if my conclusion sounds correct, when you have a minute? | 13:06 |
| *** david-lyle has quit IRC | 13:08 | |
| *** coolsvap is now known as coolsvap|away | 13:11 | |
| *** Marga_ has quit IRC | 13:11 | |
| *** tiswanso has quit IRC | 13:11 | |
| *** Marga_ has joined #tripleo | 13:12 | |
| *** chlong has quit IRC | 13:15 | |
| *** jayg|g0n3 is now known as jayg | 13:17 | |
| jistr | reported https://bugs.launchpad.net/heat/+bug/1538551 to catch the info before i try to revert the NtpServer patch locally to move forward | 13:22 |
| openstack | Launchpad bug 1538551 in heat "Unable to update a parameter from string to comma_delimited_list" [Undecided,New] | 13:22 |
| *** chlong has joined #tripleo | 13:28 | |
| *** fgimenez has quit IRC | 13:30 | |
| *** fgimenez has joined #tripleo | 13:32 | |
| *** akuznetsov has joined #tripleo | 13:46 | |
| *** julim has joined #tripleo | 13:48 | |
| *** absubram has quit IRC | 13:51 | |
| *** egafford has quit IRC | 13:52 | |
| *** lucasagomes is now known as lucas-hungry | 13:54 | |
| *** jhenner has quit IRC | 13:56 | |
| *** jhenner has joined #tripleo | 13:56 | |
| *** oshvartz has quit IRC | 13:58 | |
| *** jprovazn has quit IRC | 13:59 | |
| *** jkraj has quit IRC | 13:59 | |
| *** julim_ has joined #tripleo | 14:02 | |
| *** tiswanso has joined #tripleo | 14:03 | |
| *** Goneri has joined #tripleo | 14:03 | |
| *** jhenner has quit IRC | 14:05 | |
| *** julim has quit IRC | 14:05 | |
| marios | shardy: /me palmface "2016-01-27 14:04:42.107 | ERROR: <html><body><h1>503 Service Unavailable</h1>" for overcloud heat @ pingtest https://jenkins03.openstack.org/job/gate-tripleo-ci-f22-ha/331/console for ha job of https://review.openstack.org/#/c/260413/9 | 14:07 |
| marios | :/ | 14:07 |
| marios | so it will fail | 14:07 |
| *** regebro has quit IRC | 14:08 | |
| *** rook-desktop has quit IRC | 14:08 | |
| *** regebro has joined #tripleo | 14:08 | |
| *** masco has quit IRC | 14:09 | |
| *** morazi has quit IRC | 14:11 | |
| shardy | jistr: have you tried any minimal templates to reproduce? | 14:11 |
| jistr | shardy: no, just the tripleo ones | 14:12 |
| shardy | It'd be good to confirm the same behavior is observed via heatclient directly | 14:12 |
| *** lblanchard has joined #tripleo | 14:12 | |
| shardy | jistr: also, can you confirm you see the expected NtpServer in the --debug output getting passed to heat from tripleoclient? | 14:12 |
| jistr | shardy: i'm on a call atm, but will do that next | 14:13 |
| shardy | jistr: ack, I'll also try to reproduce later | 14:13 |
| *** rpothier has joined #tripleo | 14:13 | |
| *** morazi has joined #tripleo | 14:13 | |
| *** oshvartz has joined #tripleo | 14:13 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: controller: make sure heat use 'rabbit' rpc_backend https://review.openstack.org/272887 | 14:14 |
| EmilienM | derekh: hey, trying to understand the heat bug with rpc_backend | 14:14 |
| *** rlandy has joined #tripleo | 14:14 | |
| EmilienM | the default value in oslo messaging is 'rabbit' which was the same in puppet-heat before the breakage | 14:15 |
| EmilienM | shardy: ^ | 14:15 |
| *** david-lyle has joined #tripleo | 14:17 | |
| shardy | EmilienM: I think it was due to packaging defaults: | 14:17 |
| shardy | https://github.com/openstack-packages/heat/commit/a2ed21a64ca39596bb94d9af063f02c7dba1cd0f | 14:17 |
| EmilienM | ahah ! | 14:17 |
| EmilienM | nice shot | 14:17 |
| EmilienM | shardy: so I can abandon my tripleo patches | 14:17 |
| EmilienM | https://review.openstack.org/#/c/272886/ and https://review.openstack.org/#/c/272887/ | 14:18 |
| shardy | EmilienM: I think so, now that the current-tripleo pin has been moved | 14:18 |
| EmilienM | shardy: thx for that information | 14:18 |
| EmilienM | I did not know it | 14:18 |
| shardy | EmilienM: np, sorry for the inconvenience | 14:18 |
| *** jhenner has joined #tripleo | 14:21 | |
| EmilienM | shardy: why did you dupplicate the patch 272848 ? | 14:22 |
| EmilienM | I was trying to rebase it and I noticed you did https://review.openstack.org/#/c/272926/ | 14:23 |
| EmilienM | anyway, it's merged now, but I see CI not passing on https://review.openstack.org/#/c/272926/, is it expected? | 14:23 |
| EmilienM | shardy, derekh: please abandon https://review.openstack.org/#/c/272848/ I can't do it | 14:24 |
| shardy | EmilienM: I didn't, derekh pushed that review | 14:25 |
| shardy | we landed a series of patches which were checked via another patches earlier, to unblock CI | 14:26 |
| shardy | EmilienM: they were checked via https://review.openstack.org/#/c/229789 | 14:27 |
| shardy | unfortunately there was no way to get all of the required patches passing CI | 14:27 |
| EmilienM | shardy: ok | 14:27 |
| *** jhenner has quit IRC | 14:31 | |
| derekh | EmilienM: the reason I did a second identical patch doing the revert was so that I could test it without any depends-on in it | 14:32 |
| EmilienM | cool | 14:32 |
| EmilienM | most important thing is that is fixed now | 14:32 |
| derekh | EmilienM: will abandon the other one now | 14:32 |
| EmilienM | cool | 14:32 |
| derekh | EmilienM: yup, we're all good now | 14:32 |
| ayoung | EmilienM, I'm still battling the Keystone/HTTPD issue. Latest failure is /home/jenkins/workspace/gate-tripleo-ci-f22-ha/devstack-gate/functions.sh: line 1088: 15495 Killed timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/functions.sh && $cmd" This is from http://logs.openstack.org/75/213175/18/check-tripleo/gate-tripleo-ci-f22-ha/45fd4a0/console.html | 14:32 |
| jaosorior | ayoung: that timeout is a common hassle. I think shardy was dealing with it at some point | 14:34 |
| *** ccrouch has joined #tripleo | 14:34 | |
| ayoung | jaosorior, so not a direct result of the patch? | 14:35 |
| shardy | well it could mean anything, it just tells you the job timed out and was killed | 14:35 |
| ayoung | shardy, yeah, and I can't find anything that says what the job actually was that was killed | 14:35 |
| * shardy looks at logs | 14:35 | |
| derekh | So now that we have a recent trunk repository working, lets try and get the periodic job working | 14:36 |
| derekh | Here is the fix for the current reason the periodic job is failing https://review.openstack.org/#/c/271559/ | 14:36 |
| *** jprovazn has joined #tripleo | 14:36 | |
| ayoung | shardy, last success reported before the line was tripleo.sh -- Undercloud install - DONE. | 14:37 |
| ayoung | grep for 2016-01-27 04:09:18.296 | 14:37 |
| derekh | And these will give us support for a report that just shows us the results of a periodic job https://review.openstack.org/#/c/271370/ https://review.openstack.org/#/c/235421/3 | 14:37 |
| shardy | ayoung: it looks like it timed out trying to build the images | 14:37 |
| ayoung | shardy, OK, I thought that, but wasn't sure if it was the next task that failed | 14:38 |
| *** jprovazn has quit IRC | 14:38 | |
| *** jprovazn has joined #tripleo | 14:39 | |
| derekh | unfortunately when we get those timouts, the function that collects the logs doesn't run either ;-( | 14:40 |
| shardy | ayoung: the ceph job is failing with a different error | 14:40 |
| *** liverpooler has quit IRC | 14:40 | |
| shardy | complaining about line 179 here: | 14:40 |
| shardy | https://review.openstack.org/#/c/213175/18/puppet/manifests/overcloud_controller.pp | 14:40 |
| jaosorior | marios: I remember you +1ing this patch https://review.openstack.org/#/c/272194/ cause master hadn't merged yet, got some time to check it out again? Now that the one proposed for master is in | 14:40 |
| derekh | I think that would be fixed by collecting the logs outside of the devstack-gate runner | 14:40 |
| shardy | ayoung: so the ha job failure may be spurious, but I think that one is real | 14:41 |
| *** ron___ has joined #tripleo | 14:41 | |
| shardy | http://logs.openstack.org/75/213175/18/check-tripleo/gate-tripleo-ci-f22-nonha/c1a66a4/console.html | 14:41 |
| shardy | nonha has the same issue | 14:41 |
| marios | jaosorior: ack | 14:42 |
| *** david-lyle has quit IRC | 14:44 | |
| *** jhenner has joined #tripleo | 14:47 | |
| *** ron___ has quit IRC | 14:48 | |
| ayoung | shardy, yeah, I noticed. I've not been able to get even master tripleo to install on the Dell workstation I just got, although I did manage to get Director to run once (I think) | 14:49 |
| ayoung | It makes it hard to code | 14:49 |
| shardy | ayoung: Yeah, I'm sorry we couldn't figure out your Nova issues yesterday | 14:50 |
| *** ron___ has joined #tripleo | 14:50 | |
| shardy | FWIW I rebuilt my tripleo environment from scratch hoping to reproduce, and while I hit other problems I didn't see that one | 14:50 |
| ayoung | shardy, I'm going to strip down to baremetal and try that one again todya | 14:51 |
| shardy | ayoung: ack - if using tripleo.sh ensure you've pulled the latest as some changes landed today | 14:51 |
| ayoung | shardy, so the big ticket item is getting Keystone to run in HTTPD | 14:51 |
| *** akuznetsov has quit IRC | 14:51 | |
| ayoung | I don;t care how that happens, If I do it or someone else | 14:51 |
| ayoung | without that, Federation can't happen, and that screws over a lot of people | 14:52 |
| shardy | ayoung: also, I'd suggest not using --all, instead run each step, and in particular ensure after --register-nodes that nova hypervisor-stats is updated | 14:52 |
| ayoung | shardy, ++ I learned that/ | 14:52 |
| ayoung | ok, kernel is upgraded ... time to install instack. | 14:52 |
| *** egafford has joined #tripleo | 14:54 | |
| ayoung | shardy, check me on this, but image building happens before Keystone runs, no? | 14:55 |
| *** pradk_ has joined #tripleo | 14:56 | |
| shardy | ayoung: well if you've run --undercloud then it's running on the undercloud, but yeah there's no overcloud at that point | 14:57 |
| ayoung | shardy, so I can't see how my changes could be screwing that up | 14:58 |
| shardy | ayoung: they probably aren't, but the other two jobs are failing due to the patch | 14:58 |
| ayoung | shardy, OK, so the first thing I guessed at was this https://review.openstack.org/#/c/213175/18/puppet/manifests/overcloud_controller.pp | 14:59 |
| ayoung | https://review.openstack.org/#/c/213175/18/puppet/manifests/overcloud_controller.pp | 15:00 |
| ayoung | I know nothing from Puppet, copied that from other examples | 15:00 |
| ayoung | it was based on a comment in an earlier review | 15:00 |
| ayoung | bnemec, said "I have the unpleasant suspicion that controller_host isn't going to be defined on the overcloud. If that's the case, I'm not sure off the top of my head what the right way to tell it to bind only to the local IP is though. :-/" | 15:00 |
| ayoung | and marios responded with "I believe Ben is right with his comment here on v16 - we do pass 'controller_node_names' - note however that this is a comma delimited list of all controller host names | 15:01 |
| ayoung | looking at https://github.com/openstack/puppet-keystone/blob/4c5c3e0b76e0c0ea5c73b1d76fe0fd9a284fb524/manifests/wsgi/apache.pp#L28 sounds like it expects only one (if so you can split on ',')" | 15:01 |
| ayoung | I really would rather not have gerrit be my debugger, as it has a very slow turn-around | 15:01 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for gentoo to a bunch of elements https://review.openstack.org/270597 | 15:01 |
| *** lucas-hungry is now known as lucasagomes | 15:02 | |
| dprince | jtomasek: zaqar is here http://trunk.rdoproject.org/centos7/current/ and we have a puppet-zaqar module as well... so I will see about getting you some patches to have it available in the undercloud | 15:03 |
| jtomasek | dprince: thanks! | 15:04 |
| *** oshvartz has quit IRC | 15:06 | |
| *** trozet has joined #tripleo | 15:15 | |
| *** oshvartz has joined #tripleo | 15:18 | |
| *** tosky has quit IRC | 15:21 | |
| EmilienM | dprince: puppet-zaqar is empty afik | 15:24 |
| EmilienM | it's noop at this time | 15:24 |
| EmilienM | a guy was working on it but no progress since months | 15:25 |
| EmilienM | let me ask on #openstack-zaqar | 15:25 |
| dprince | EmilienM: oh no | 15:25 |
| dprince | EmilienM: sounds like I may be your man then :) | 15:25 |
| dprince | EmilienM: go easy on me though | 15:25 |
| *** dmacpher has joined #tripleo | 15:25 | |
| EmilienM | dprince: if you could do the same thing you did with puppet-mistral, I'll pay you french wine | 15:26 |
| dprince | EmilienM: not critical, but it sounds like it is becoming of interest for the UI team for things like websockets integration | 15:26 |
| EmilienM | makes sense | 15:26 |
| dprince | EmilienM: sounds like we may have a deal | 15:26 |
| *** yamahata has quit IRC | 15:30 | |
| *** yamahata has joined #tripleo | 15:30 | |
| *** jistr|mobi has joined #tripleo | 15:35 | |
| *** tosky has joined #tripleo | 15:35 | |
| *** dprince has quit IRC | 15:37 | |
| d0ugal | mgould: I am asking about the failure in #openstack-infra | 15:53 |
| d0ugal | mgould: It sounds like new images are needed for the builders, that should be automated but "sometimes fails" | 15:55 |
| d0ugal | :) | 15:55 |
| *** egafford has quit IRC | 15:58 | |
| slagle | trunk.rdoproject.org down? | 15:59 |
| *** egafford has joined #tripleo | 16:00 | |
| mgould | d0ugal, thank goodness, I was starting to lose my faith in determinism :-) | 16:02 |
| *** masco has joined #tripleo | 16:06 | |
| EmilienM | slagle: yes | 16:08 |
| EmilienM | slagle: #rdo is aware | 16:09 |
| *** rcernin has quit IRC | 16:09 | |
| trown | EmilienM: is it self-aware as in the singularity? | 16:10 |
| EmilienM | :) | 16:10 |
| gfidente | shardy, so I do get the error logged for a specific resoucr | 16:13 |
| gfidente | shardy, but it doesn't look like a simple case of the softwaredeployment config being too big | 16:13 |
| gfidente | because I can't reproduce it that way with a simple softwaredeployment pushing a big file on a single server | 16:14 |
| d0ugal | mgould: I guess after a number of retires it is best to ask around :) | 16:14 |
| *** david-lyle has joined #tripleo | 16:14 | |
| *** masco has quit IRC | 16:15 | |
| ayoung | woot! 90bdd865-0b58-4735-add6-554b45cf08f1 | overcloud | CREATE_COMPLETE | 16:15 |
| ayoung | OK, I have a successful overcloud deployment | 16:16 |
| shardy | ayoung: \o/ | 16:16 |
| shardy | Now don't change *anything* ;) | 16:16 |
| trown | lol | 16:16 |
| ayoung | now...I need to do development, I assume, tripleo.sh --overcloud-delete | 16:17 |
| *** shivrao has joined #tripleo | 16:17 | |
| ayoung | and then try the same thing again with git checkout against master | 16:17 |
| shardy | ayoung: you can do that, or just "heat stack-delete" | 16:17 |
| shardy | ayoung: the difference is tripleo.sh polls until the delete is done | 16:17 |
| shardy | vs doing heat stack-list a few times to check | 16:17 |
| *** paramite has quit IRC | 16:17 | |
| shardy | ("heat stack-delete overcloud") | 16:18 |
| prometheanfire | is gate-tripleo-ci-f22-nonha still failing for everyone or just me? | 16:18 |
| ayoung | shardy, actually, to be even more cautious....once I do that, how far back do I need to go to redeploy? | 16:18 |
| ayoung | tripleo-common/scripts/tripleo.sh --register-nodes | 16:18 |
| ayoung | or just | 16:18 |
| shardy | ayoung: you're making changes to tripleo-heat-templates right? | 16:18 |
| ayoung | shardy, eventually, | 16:18 |
| shardy | ayoung: if that's all you're changing, then it's just: | 16:19 |
| shardy | heat stack-delete overcloud (or tripleo.sh --overcloud-delete | 16:19 |
| shardy | then openstack overcloud deploy --templates /path/to/git/tripleo-heat-templates | 16:19 |
| shardy | you can hack on the local tree of t-h-t and just pass the path | 16:20 |
| ayoung | ok, let me make sure I can deploy a second time with no changes | 16:20 |
| mgould | d0ugal, and only now do I notice that check-osc-plugins is non-voting :-( | 16:20 |
| *** mbound has quit IRC | 16:21 | |
| *** oshvartz has quit IRC | 16:23 | |
| EmilienM | slagle: should be back now | 16:24 |
| *** NobodyCa1 has joined #tripleo | 16:24 | |
| *** NobodyCam has quit IRC | 16:25 | |
| *** mcornea has quit IRC | 16:25 | |
| *** NobodyCa1 is now known as NobodyCam | 16:27 | |
| d0ugal | mgould: hah, so it is. However, good to get these things resolved if we can | 16:27 |
| *** aufi has quit IRC | 16:28 | |
| mgould | yeah, definitely | 16:28 |
| d0ugal | mgould: FWIW, that failure is happening everywhere :) | 16:37 |
| d0ugal | mgould: Just noticed it on one of my other reviews | 16:37 |
| *** david-lyle has quit IRC | 16:38 | |
| mgould | d0ugal, bizarre | 16:39 |
| mgould | it looks like a new Jenkins image was cut some time after 1400 UTC | 16:39 |
| d0ugal | mgould: Yeah, they confirmed the image was built correctly - but couldn't confirm it was uploaded :) | 16:40 |
| mgould | aaaaah | 16:40 |
| *** david-lyle has joined #tripleo | 16:40 | |
| *** bnemec has quit IRC | 16:42 | |
| d0ugal | mgould: but anyway, I think we can ignore it since it is non-voting and trust that it will be resolved in time :) | 16:42 |
| d0ugal | I guess being non-voting somebody is working it anyway | 16:43 |
| d0ugal | (to get it in a state to become voting) | 16:43 |
| d0ugal | mgould: but now we need to worry about the other CI failures :( | 16:43 |
| mgould | yeah | 16:44 |
| prometheanfire | ? | 16:44 |
| mgould | prometheanfire, the check-osc-plugins CI has been failing for days despite the fix already being merged | 16:46 |
| gfidente | shardy, so the message we always timed out | 16:46 |
| gfidente | INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : e86088d2266f4d10984abe5b469cb032 | 16:47 |
| gfidente | which I think explains the timeout | 16:47 |
| prometheanfire | mgould: then the fix isn'ta fix? | 16:47 |
| gfidente | it's the only message id printing that | 16:47 |
| mgould | prometheanfire, nope | 16:47 |
| shardy | gfidente: Hmm, that's strange, sounds like either a process got killed or a greenthread handling the request itself died | 16:48 |
| mgould | the fix removes the line from the script that errors | 16:48 |
| shardy | no backtrace before that? | 16:48 |
| mgould | yet it's still being run | 16:48 |
| mgould | so the CI workers are still running the old version of the script | 16:48 |
| mgould | prometheanfire, gate-tripleo-ci-f22-nonha passed for me half an hour ago: https://review.openstack.org/#/c/265336/ | 16:49 |
| mgould | everything else is failing, though :-( | 16:49 |
| *** bnemec has joined #tripleo | 16:49 | |
| prometheanfire | mgould: odd, I got everything else to pass | 16:50 |
| prometheanfire | mgould: https://review.openstack.org/#/c/270597/ | 16:50 |
| mgould | prometheanfire, so if we combine our patches then everything will work? :-) | 16:50 |
| prometheanfire | mgould: you fine with a 1000 line patch? :P | 16:51 |
| prometheanfire | most of that is in growpart though | 16:51 |
| mgould | BTW: I seem to spend an awful lot of time in the "patch failed CI; read logs; determine it's not my fault; recheck; goto 10" loop | 16:51 |
| mgould | am I just doing it wrong? | 16:51 |
| prometheanfire | no, that's my cycle too | 16:51 |
| mgould | :-( | 16:51 |
| mgould | flaky CI is No Fun | 16:51 |
| mgould | do we have any stats on how many transitory CI failures we get? | 16:54 |
| derekh | mgould: lots, but the problem isn't CI, the problem is that everybody ignores the intermittent erros and keeps hitting recheck until they get a pass, the intermittent errors go unfixed and pill up on top of each other | 16:58 |
| derekh | and we eventually get into a state where there are so many errors orrcuring that we cant get anything merged | 16:58 |
| *** sthillma has joined #tripleo | 16:58 | |
| mgould | derekh, sure | 16:58 |
| *** trown is now known as trown|lunch | 16:58 | |
| * mgould was thinking that having the numbers might convince people to throw resources at fixing the problem | 16:59 | |
| derekh | mgould: yup, it would probably help | 16:59 |
| *** jistr has quit IRC | 16:59 | |
| derekh | mgould: this page give you a visual indication of ci jobs that have failed http://tripleo.org/cistatus.html | 17:00 |
| *** sthillma_ has joined #tripleo | 17:01 | |
| mgould | derekh: awesome, thanks! | 17:01 |
| *** dprince has joined #tripleo | 17:01 | |
| derekh | mgould: but to do it properly, somebody needs to go through all the logs and see which failures were false negatives and with were legitimate, I've done this in the past but its very time consuming | 17:01 |
| mgould | wow, that's a lotta red | 17:01 |
| d0ugal | mgould: check-osc-plugins passed! | 17:01 |
| mgould | d0ugal, \o/! | 17:02 |
| d0ugal | so I guess the image finally got where it needed to be. | 17:02 |
| mgould | thank goodness for that | 17:02 |
| mgould | now let's see what happens in gate-tripleo... | 17:02 |
| derekh | mgould: yup, we've had a bad few days, all of yesterday tripleo master was broken | 17:03 |
| mgould | :( | 17:03 |
| mgould | we've had a bad few days in ironic too | 17:03 |
| *** sthillma has quit IRC | 17:04 | |
| *** sthillma_ is now known as sthillma | 17:04 | |
| * derekh is trying now to reproduce some of the intermittent errors | 17:04 | |
| * mgould applauds derekh | 17:04 | |
| prometheanfire | also, lol 2016-01-27 15:08:56.045 | fatal: A branch named 'master' already exists. | 17:05 |
| mgould | oh dear | 17:06 |
| mgould | I thought we tested everything in detached HEAD state? | 17:06 |
| *** yamahata has quit IRC | 17:07 | |
| prometheanfire | from http://logs.openstack.org/97/270597/15/check-tripleo/gate-tripleo-ci-f22-nonha/474ef96/console.html | 17:07 |
| prometheanfire | http://logs.openstack.org/97/270597/15/check-tripleo/gate-tripleo-ci-f22-nonha/474ef96/console.html#_2016-01-27_15_08_56_045 | 17:07 |
| prometheanfire | I'm going to type recheck and see what happens | 17:07 |
| *** devvesa has quit IRC | 17:08 | |
| *** fgimenez has quit IRC | 17:11 | |
| derekh | prometheanfire: it wont pass, the problem with that patch is that it introduces this file https://review.openstack.org/#/c/270597/15/elements/growroot/init-scripts/openrc/growroot | 17:13 |
| derekh | prometheanfire: forget about the master branch error, its a red herring | 17:13 |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| prometheanfire | why is that an issue? | 17:13 |
| derekh | prometheanfire: rpmbuild sees "#!/sbin/runscript" and adds an autorequires for a package that provides that script | 17:14 |
| derekh | prometheanfire: fails to find one | 17:14 |
| derekh | prometheanfire: in the delorean logs you'll see | 17:14 |
| derekh | DEBUG: Error: Package: diskimage-builder-1.8.1-dev7.el7.centos.noarch (/diskimage-builder-1.8.1-dev7.el7.centos.noarch) | 17:14 |
| derekh | DEBUG: Requires: /sbin/runscript | 17:14 |
| prometheanfire | but it's an init script | 17:14 |
| prometheanfire | that's how our init scripts work... | 17:15 |
| prometheanfire | how do I work around it? | 17:16 |
| prometheanfire | https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-cluster/nova/files/nova.initd as an example | 17:17 |
| derekh | prometheanfire: we had a similar issue last week when "/usr/local/bin/dib-python" was added into another script | 17:17 |
| derekh | prometheanfire: if that line is needed in the script then it will need to be excluded from autorequires | 17:17 |
| derekh | prometheanfire: like https://review.gerrithub.io/#/c/260429/3/diskimage-builder.spec | 17:17 |
| mgould | question unrelated to CI: should I abandon https://review.openstack.org/#/c/263831 in favour of https://review.openstack.org/#/c/272206 ? | 17:18 |
| *** mkovacik has quit IRC | 17:18 | |
| mgould | 2683831 is meant to be a hacky version that can be merged without waiting for lots of dependencies | 17:19 |
| prometheanfire | what project is that in? | 17:19 |
| mgould | python-tripleoclient | 17:19 |
| *** jaosorior has quit IRC | 17:20 | |
| mgould | they add support for the new version of the Ironic state machine | 17:20 |
| prometheanfire | sorry, was asking derekh how what project I need to submit a review for | 17:20 |
| *** jaosorior has joined #tripleo | 17:20 | |
| mgould | prometheanfire, oh, sorry | 17:20 |
| *** jaosorior has quit IRC | 17:21 | |
| derekh | prometheanfire: so that is a complicated part, the packaging on a different gerrit (openstack-packages/diskimage-builder on gerrithub), I'd love to see it moved onto our gerrit some time soon but for now the best we can do is line up the two changes and try and merge them together | 17:23 |
| prometheanfire | ya, I'm depending on that change :( | 17:23 |
| derekh | actually it might be possible to do the packaging change first, we could try that | 17:23 |
| derekh | If you submit a change to add it, we can get it merged then your patch to DIB shouldn't fail any longer (atleast for the readon it currently is) | 17:25 |
| derekh | if you prefer I can take a look at the packaging part, | 17:25 |
| prometheanfire | so do I need to clone from gerrithub? | 17:25 |
| prometheanfire | it'd help, it is just that one line | 17:26 |
| *** shivrao has quit IRC | 17:26 | |
| *** shardy has quit IRC | 17:26 | |
| derekh | prometheanfire: on it | 17:27 |
| prometheanfire | thanks | 17:27 |
| *** lazy_prince has quit IRC | 17:27 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Update yaml-validate.py to accept files or directories https://review.openstack.org/269281 | 17:28 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add simple parameter test to yaml-validate.py https://review.openstack.org/269282 | 17:28 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove unused parameters https://review.openstack.org/269283 | 17:28 |
| *** tosky has quit IRC | 17:29 | |
| *** jistr|mobi has quit IRC | 17:30 | |
| *** yamahata has joined #tripleo | 17:31 | |
| derekh | prometheanfire: https://review.gerrithub.io/#/c/261386/1 , gotta run out, I'll follow up with it when I'm back in a bit | 17:33 |
| prometheanfire | derekh: that redefines it I think | 17:35 |
| prometheanfire | derekh: https://fedoraproject.org/wiki/Packaging:AutoProvidesAndRequiresFiltering These macros are not cumulative | 17:35 |
| ayoung | EmilienM, OK, I have a reproducible setup now. I can create and destroy overclouds at whim. Care to help me beat the Keystone HTTPD Review into submission? | 17:40 |
| ayoung | I'm running an overcloud-deploy right now, and I should get a better view into where it fails | 17:41 |
| EmilienM | I was about to get lunch | 17:41 |
| EmilienM | can you run puppet and give me the output? | 17:41 |
| EmilienM | I'll catch-up after my quick lunch break | 17:42 |
| *** dshulyak has quit IRC | 17:42 | |
| *** mbound has joined #tripleo | 17:43 | |
| *** mbound has quit IRC | 17:44 | |
| *** tiswanso has quit IRC | 17:46 | |
| ayoung | EmilienM, aftet lunch is good | 17:46 |
| ayoung | I need to eat too | 17:46 |
| derekh | prometheanfire: doh, I'll have to take a closer look later on, | 17:50 |
| prometheanfire | derekh: https://review.gerrithub.io/#/c/261387/ | 17:50 |
| *** dmsimard has quit IRC | 17:51 | |
| derekh | prometheanfire: thanks, I'll try it out later this evening with your patch and make sure there isn't anything else | 17:51 |
| *** derekh has quit IRC | 17:52 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: DO NOT MERGE: unmount debugging https://review.openstack.org/273171 | 17:53 |
| *** mbound has joined #tripleo | 17:54 | |
| *** jcoufal has quit IRC | 17:54 | |
| *** jdob has quit IRC | 17:57 | |
| *** shivrao has joined #tripleo | 17:58 | |
| *** rcernin has joined #tripleo | 17:59 | |
| *** regebro has quit IRC | 17:59 | |
| *** dshulyak has joined #tripleo | 18:00 | |
| *** lucasagomes is now known as lucas-dinner | 18:04 | |
| dprince | rbrady: nice on the action executions suggestion, I think that helps the UI quite a bit | 18:04 |
| ayoung | EmilienM, http://paste.openstack.org/show/485178/ based on the steps from http://hardysteven.blogspot.com/2015/04/debugging-tripleo-heat-templates.html leads me to think I have a syntax error. Trying to map that to the .pp file | 18:10 |
| ayoung | since it is line 179 I'm guessing bind_host => split(hiera('controller_node_names'), ',')[0], is no good | 18:11 |
| ayoung | according to the reviews, we need an IP address for the Keystone service to listen on. Seems to me it should follow the pattern done for Horizon | 18:12 |
| ayoung | Horizon does not specify anything... | 18:14 |
| ayoung | I think I'm going to yank that line and see what happens. There and in the controller.pp | 18:14 |
| *** olap has quit IRC | 18:15 | |
| *** jcoufal has joined #tripleo | 18:18 | |
| *** mbound has quit IRC | 18:18 | |
| *** athomas has quit IRC | 18:22 | |
| *** athomas has joined #tripleo | 18:22 | |
| *** ukalifon1 has quit IRC | 18:26 | |
| *** sthillma has quit IRC | 18:29 | |
| *** ifarkas has quit IRC | 18:30 | |
| *** electrofelix has quit IRC | 18:32 | |
| *** regebro has joined #tripleo | 18:34 | |
| EmilienM | ayoung: back, looking now | 18:38 |
| ayoung | EmilienM, so, I think it has to do with setting the IP address for HTTPD for Keystone | 18:38 |
| ayoung | my syntax was bad. I am trying right now with nothing in that line, to see if we actually need to set it | 18:38 |
| ayoung | the horizon analogue does not set a host to listne on | 18:39 |
| ayoung | EmilienM, it seems to get further, but I don;t know if that constitutes success | 18:40 |
| ayoung | this one failed on overcloud-ControllerNodesPostDeployment-y2pizpkdnnto-ControllerOvercloudServicesDeployment_Step4-3ypex4d5dutp | 18:41 |
| *** mgould has quit IRC | 18:43 | |
| ayoung | EmilienM, yeah, without that line I get an error message like this | 18:43 |
| ayoung | http://paste.openstack.org/show/485183/ | 18:43 |
| *** tiswanso has joined #tripleo | 18:48 | |
| *** trown|lunch is now known as trown | 18:49 | |
| *** rbrady has quit IRC | 18:54 | |
| *** jdob has joined #tripleo | 18:58 | |
| *** dprince has quit IRC | 18:59 | |
| *** dprince has joined #tripleo | 19:00 | |
| *** trown is now known as trown|brb | 19:02 | |
| *** sthillma has joined #tripleo | 19:04 | |
| *** sthillma has quit IRC | 19:11 | |
| *** trown|brb is now known as trown | 19:13 | |
| *** dshulyak has quit IRC | 19:15 | |
| *** rbrady has joined #tripleo | 19:21 | |
| *** oshvartz has joined #tripleo | 19:21 | |
| *** penick has joined #tripleo | 19:22 | |
| *** nico_auv has quit IRC | 19:22 | |
| prometheanfire | and derekh is gone | 19:27 |
| *** weshay_xchat has quit IRC | 19:27 | |
| EmilienM | slagle, dprince: maybe you guys can review https://review.openstack.org/#/c/272699/ | 19:29 |
| *** weshay_xchat has joined #tripleo | 19:29 | |
| dprince | EmilienM: looks fine, but would it be reasonable to set controller_host via hiera instead? | 19:31 |
| dprince | EmilienM: stylistically then we could just use 'include keystone::wsgi' right? | 19:31 |
| *** sthillma has joined #tripleo | 19:31 | |
| EmilienM | I can do that! | 19:31 |
| EmilienM | let me update | 19:31 |
| dprince | EmilienM: cool. Yeah, We've been gradually moving more and more into heira in the undercloud | 19:32 |
| EmilienM | nice | 19:32 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: keystone/wsgi: bind on local IP https://review.openstack.org/272699 | 19:34 |
| EmilienM | dprince: let's try ^ | 19:34 |
| *** leanderthal has quit IRC | 19:38 | |
| *** Marga_ has quit IRC | 19:40 | |
| dprince | EmilienM: one more thing, lets just use {{LOCAL_IP}} directly in hiera I think | 19:40 |
| slagle | EmilienM: what causes the db_sync's to happen in the puppet modules? | 19:50 |
| slagle | EmilienM: on stable/liberty, i'm seeing that they are failing with tables already created errors | 19:51 |
| slagle | which makes me think a race condition with each controller trying to run them at close to the same time | 19:51 |
| prometheanfire | well, got this working... https://review.gerrithub.io/#/c/261404/ | 19:53 |
| prometheanfire | dunno who gets to push the button now that derekh is gone | 19:53 |
| *** rbrady has quit IRC | 19:54 | |
| EmilienM | dprince: ok will do | 19:55 |
| EmilienM | slagle: package upgrade / service restart does | 19:55 |
| EmilienM | db_sync is (or should be) idempotent | 19:55 |
| slagle | EmilienM: yea, i see what my problem is | 19:56 |
| EmilienM | if it's failing, that's a bug in core projects | 19:56 |
| dprince | bnemec: hey, this would be helpful to some of the tarball -> swift container stuff we'd like to use: https://review.openstack.org/#/c/264931/ | 19:56 |
| slagle | it is idempotent | 19:56 |
| slagle | except when you run it at the same time on 3 controllers at once | 19:56 |
| slagle | but...i forgot to pass the pacemaker environment file | 19:56 |
| EmilienM | slagle: that's a race | 19:56 |
| slagle | overcloud_controller.pp is broken | 19:56 |
| EmilienM | slagle: we fixed it in spinalstack by running it on one controller | 19:56 |
| EmilienM | slagle: let me show you, a sec | 19:57 |
| EmilienM | I'll send a patch after that | 19:57 |
| slagle | but overcloud_controller_pacemaker.pp only runs it on the bootstrap | 19:57 |
| slagle | there is a guard already in the pacemaker manifest, so that works | 19:57 |
| slagle | we should probably retire overcloud_controller.pp since it doesnt actually work anymore | 19:58 |
| slagle | at least for ha it doesn't | 19:58 |
| EmilienM | slagle: wait, overcloud_controller.pp is not used in ha scenario, isn't? | 19:58 |
| *** rbrady has joined #tripleo | 19:59 | |
| slagle | EmilienM: it's not used for ha | 19:59 |
| prometheanfire | who else is in openstack-packages/diskimage-builder project that can +workflow it? | 19:59 |
| slagle | ha requires the pacemaker environment file (which i just forgot to do), but that ought to be encoded somewhere | 20:00 |
| EmilienM | slagle: well our workaround was not really clean anyway | 20:03 |
| *** Marga_ has joined #tripleo | 20:03 | |
| EmilienM | we should run the db_sync only on the first controller node | 20:03 |
| *** barra204 has quit IRC | 20:06 | |
| ayoung | dprince, can I do {{LOCAL_IP}} for the overcloud, too? | 20:06 |
| dprince | ayoung: in hiera, yeah? patch link? | 20:07 |
| *** jcoufal has quit IRC | 20:07 | |
| ayoung | dprince, in https://review.openstack.org/#/c/213175/ | 20:08 |
| ayoung | https://review.openstack.org/#/c/213175/8/puppet/manifests/overcloud_controller.pp | 20:08 |
| ayoung | dprince, I'm not certain what is correct there, but I assume it should mirror the undercloud | 20:08 |
| ayoung | dprince, sorry, that was the old commit. Here was my horrible hack https://review.openstack.org/#/c/213175/18/puppet/manifests/overcloud_controller.pp | 20:09 |
| *** oshvartz has quit IRC | 20:12 | |
| *** julim_ has quit IRC | 20:17 | |
| *** akrivoka has quit IRC | 20:24 | |
| *** mbound has joined #tripleo | 20:31 | |
| *** eggmaster has quit IRC | 20:33 | |
| *** olap has joined #tripleo | 20:35 | |
| *** weshay_xchat has quit IRC | 20:36 | |
| ayoung | dprince, EmilienM what should be after bind_host => in the ::keystone::wsgi::apache' section? It looks like the undercloud is using ::keystone::wsgi::apache so should overcloud do the same? | 20:43 |
| EmilienM | ayoung: yeah | 20:44 |
| ayoung | EmilienM, should I do the same thing you did for undercloud, and put that in the template? | 20:44 |
| ayoung | EmilienM, or does https://review.openstack.org/#/c/272699/5/elements/puppet-stack-config/puppet-stack-config.yaml.template does that for us implicitly? | 20:45 |
| EmilienM | ayoung: it should do it | 20:46 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: keystone/wsgi: bind on local IP https://review.openstack.org/272699 | 20:46 |
| EmilienM | ayoung: see my patch ^ | 20:46 |
| EmilienM | we should rely on hiera when possible | 20:47 |
| ayoung | EmilienM, right, I should follow suite on the overcloud patch | 20:47 |
| EmilienM | cool | 20:47 |
| ayoung | so I change puppet/manifests/overcloud_controller.pp from class { '::keystone::wsgi::apache': | 20:47 |
| ayoung | + ssl => false, | 20:47 |
| ayoung | ... | 20:47 |
| ayoung | to class { '::keystone::wsgi::apache': | 20:47 |
| ayoung | + ssl => false, | 20:47 |
| ayoung | EmilienM, ^^ | 20:48 |
| EmilienM | you can drop it | 20:48 |
| EmilienM | and put it in hiera | 20:48 |
| EmilienM | in controller.yaml | 20:48 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates https://review.openstack.org/273233 | 20:49 |
| ayoung | EmilienM, I'm not quite sure what that means. DO you want to make that change? | 20:49 |
| prometheanfire | derekh :( | 20:49 |
| ayoung | bnemec, did you look in to using certmonger first? | 20:49 |
| bnemec | ayoung: No | 20:50 |
| ayoung | bnemec, much better to make our Cert story around certmonger. It has a selfsigned CA if necessary, and lets us tie in with a real CA if it is available | 20:50 |
| *** weshay_xchat has joined #tripleo | 20:50 | |
| ayoung | does cert reup when they are about to expire as well | 20:50 |
| bnemec | I'll put it on my todo list. :-) | 20:52 |
| * bnemec wants to stop being responsible for SSL anyway | 20:52 | |
| ayoung | bnemec, https://fedorahosted.org/certmonger/ | 20:53 |
| ayoung | ships by default on Fedora Centos | 20:53 |
| ayoung | available on Ubuntu | 20:53 |
| EmilienM | ayoung: I can do it if you want | 20:54 |
| ayoung | I might have a blogpost or two for you. let me look | 20:54 |
| EmilienM | ayoung: or we can hack on it together | 20:54 |
| EmilienM | ayoung: sorry I was kind of busy by our puppet openstack sprint | 20:54 |
| ayoung | EmilienM, I don;t mind learning | 20:54 |
| ayoung | EmilienM, no problem, you are much in demand | 20:54 |
| ayoung | bnemec, I have this one http://adam.younglogic.com/2014/03/certmonger-session/ | 20:55 |
| ayoung | bnemec, Ah here http://adam.younglogic.com/2014/02/certmonger-selfsigned-cms-cert/ | 20:55 |
| ayoung | bnemec, its your call. I don't want to make things tougher, but certmonger is supposed to offload the cert responsibility | 20:56 |
| ayoung | EmilienM, when I run the overcloud deploy, I run openstack overcloud deploy --template /home/stack/tripleo-heat-templates/ | 20:57 |
| ayoung | In order to pick up https://review.openstack.org/#/c/272699/6/elements/puppet-stack-config/puppet-stack-config.pp I need a different repo | 20:57 |
| EmilienM | https://review.openstack.org/#/c/272699/6/elements/puppet-stack-config/puppet-stack-config.pp is undercloud, just fyi | 20:57 |
| EmilienM | so my patch in THT is supposed to work but we can improve it to use 100% Hiera | 20:58 |
| ayoung | EmilienM, so the template used in ^^ is not also used in overcloud? | 20:59 |
| slagle | ok, finally got my local liberty cloud failed the same way CI is failing. now to see why | 21:00 |
| EmilienM | ayoung: no | 21:00 |
| ayoung | EmilienM, OK so do we still need something like | 21:01 |
| ayoung | class { '::keystone::wsgi::apache': | 21:01 |
| ayoung | + ssl => false, | 21:01 |
| ayoung | + bind_host => split(hiera('controller_node_names'), ',')[0], | 21:01 |
| ayoung | + } | 21:01 |
| ayoung | or does that go into the equivalent of the template? | 21:01 |
| *** mbound has quit IRC | 21:01 | |
| *** gfidente has quit IRC | 21:01 | |
| dprince | ayoung: commented on your patch | 21:01 |
| dprince | ayoung: on a call, then got ping for something else | 21:02 |
| dprince | ayoung: want me to help with this? | 21:02 |
| ayoung | dprince, I would love some help | 21:02 |
| ayoung | I can wait until you are off the call | 21:02 |
| dprince | ayoung: I'm free now | 21:03 |
| *** julim has joined #tripleo | 21:03 | |
| *** mbound has joined #tripleo | 21:04 | |
| dprince | ayoung: so there is a potential problem here in that previously we allowed the networks for the public (port 5000) and admin (port 35357) networks to be on totally separate networks | 21:05 |
| dprince | ayoung: they had separate bind IPs | 21:05 |
| dprince | dsneddon: are you around? | 21:05 |
| dprince | ayoung: anyways, with WSGI there is only 1 bind host now so if we land your patch we also need to correct the network isolation settings (merge them I think) for keystone. See here: http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud.yaml#n631 | 21:07 |
| dprince | EmilienM: can you follow this conversation. I wondering if for backwards compat we need keystone WSGI to support 2 bind hosts, right now it just supports the 1 | 21:07 |
| dprince | dsneddon: ^^^ | 21:08 |
| EmilienM | we actually need the 2 params | 21:08 |
| dprince | EmilienM: okay, is this done already (I didn't see it in the module) | 21:08 |
| dsneddon | dprince, I'm following | 21:08 |
| dprince | EmilienM: should I push a patch? | 21:08 |
| dsneddon | dprince, I thought we were having Keystone bind on one IP, then using HAProxy to present it on different networks, but I'd need to double-check. | 21:09 |
| dprince | dsneddon: no, we need to locally have WSGI run on the separate networks | 21:10 |
| ayoung | dsneddon, I don't think tthat will work for old V2 stuff. For V3 we onlty need one port anyway | 21:10 |
| dprince | dsneddon: the HAproxy config for this should support whatever we do | 21:10 |
| ayoung | but V2 has diffferent stuff exposed on 5000 vs 35357 | 21:10 |
| dprince | EmilienM: sec, and I'll push a sample puppet-keystone patch to add the new setting (bind_host) | 21:10 |
| dprince | EmilienM: once we have that I can update ayoung's patch and we can move this forwards... | 21:11 |
| ayoung | dprince, awesome. The driving factor here is Keystone in HTTPD is needed for Federation and SSO. | 21:11 |
| dprince | EmilienM: https://review.openstack.org/273241, still a WIP but once I fix/add tests do you buy this? | 21:13 |
| dprince | EmilienM: this gives us parity with the local bind port settings when running keystone under eventlet... which is something that matters to TripleO | 21:14 |
| *** nkinder has joined #tripleo | 21:14 | |
| EmilienM | dprince: wait | 21:15 |
| EmilienM | where is used bind_host now? | 21:15 |
| EmilienM | oh | 21:15 |
| EmilienM | right | 21:15 |
| dprince | bind_host would be "public" | 21:15 |
| dprince | which I think makes sense | 21:15 |
| EmilienM | we need to add backward compt | 21:15 |
| dprince | admin_bind_host would be the admin network | 21:16 |
| EmilienM | if empty -> take same as public | 21:16 |
| EmilienM | you'll need to patch puppet-openstacklib | 21:16 |
| EmilienM | err no | 21:16 |
| EmilienM | nevermind my last comment | 21:16 |
| dprince | EmilienM: backwards compat is fine | 21:16 |
| dprince | EmilienM: I will fixup the tests and repost | 21:16 |
| EmilienM | we need to feed the param is empty | 21:16 |
| EmilienM | cool | 21:16 |
| EmilienM | +A | 21:16 |
| *** rcernin has quit IRC | 21:16 | |
| ayoung | That is right | 21:17 |
| *** trown is now known as trown|outttypeww | 21:18 | |
| ayoung | BTW, I would be totally cool with everything just listening on port 443, but that is too big a change for this release | 21:21 |
| dsneddon | ayoung, I'm down with that, but we'll need to move to per-service VIPs if we want multiple services on 443 | 21:22 |
| ayoung | dsneddon, nope | 21:22 |
| dsneddon | ayoung, Oh? | 21:22 |
| ayoung | we put em all in HTTP and make the URLS deconflict | 21:22 |
| ayoung | dsneddon, https://wiki.openstack.org/wiki/URLs | 21:22 |
| ayoung | I know that morgan former PTL of Keystone is working on a Proof of concept right now that does that | 21:23 |
| dsneddon | ayoung, Ah, yeah, I used to use a similar method with Pound instead of HAProxy doing the HTTP decode and sending requests to the right backend. | 21:23 |
| ayoung | dsneddon, yeah, putting all the services into HTTPD actually makes it simpler. The different ports were an artifact of running them via different processes | 21:24 |
| ayoung | the ports 5000 and 35357 are both problematic | 21:24 |
| ayoung | 5000 is assigned to a different service (Universal Plug and PLay) and 35357 is in the middle of the ephemeral range | 21:24 |
| dsneddon | ayoung, Yeah, and running HTTP servers inside of the Python processes was the original legacy application. | 21:24 |
| ayoung | right | 21:24 |
| ayoung | Next release, though. For now, I just need Keystone in HTTPD | 21:25 |
| *** jayg is now known as jayg|g0n3 | 21:27 | |
| *** jayg|g0n3 is now known as jayg | 21:27 | |
| *** jayg is now known as jayg|g0n3 | 21:27 | |
| *** weshay_xchat has quit IRC | 21:32 | |
| *** jprovazn has quit IRC | 21:35 | |
| *** eggmaster has joined #tripleo | 21:38 | |
| *** weshay_xchat has joined #tripleo | 21:39 | |
| *** weshay_xchat is now known as weshay | 21:39 | |
| slagle | EmilienM: hey again | 21:40 |
| EmilienM | slagle: hey james how are you today | 21:40 |
| slagle | EmilienM: i think there might be some sort of regression in puppetlabs-mysql for stable/liberty | 21:40 |
| ayoung | dprince, so what then will go in https://review.openstack.org/#/c/213175/ | 21:40 |
| slagle | oh i'm great | 21:40 |
| EmilienM | damn, mysql again | 21:41 |
| EmilienM | slagle: is it the same bug as last time? with wsrep? | 21:41 |
| slagle | EmilienM: so in /etc/my.cnf.d/galera.cnf, i'm seeing just a line of "wsrep_notify_cmd" which actually needs to be "wsrep_notify_cmd =" | 21:41 |
| slagle | we set it to empty string | 21:41 |
| slagle | but the equals is missing | 21:41 |
| ayoung | bind_host => hiera('keystone::admin_bind_host')? | 21:41 |
| slagle | EmilienM: this is causing mysqld to fail to start | 21:41 |
| slagle | EmilienM: I noticed the module was rebased recently on the stable/liberty branch of opm, https://github.com/redhat-openstack/openstack-puppet-modules/commit/889e89763f0a58d944277e14c047c5f48ec73a9b | 21:42 |
| dprince | ayoung: almost finished, I'll update your patch too | 21:44 |
| ayoung | dprince, thanks | 21:44 |
| *** jcoufal has joined #tripleo | 21:52 | |
| *** lifeless has quit IRC | 21:53 | |
| *** lifeless has joined #tripleo | 21:55 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi https://review.openstack.org/213175 | 21:56 |
| dprince | ayoung: boom ^^^ | 21:56 |
| dprince | EmilienM: ^^^, depends on the puppet-keystone patch | 21:57 |
| ayoung | dprince, if I grab that branch, can I test it, or do I need a seperate repo as well. Rigjht now, the only code I have from git is tripleo-heat-templates | 21:58 |
| *** jhenner1 has joined #tripleo | 21:58 | |
| dprince | ayoung: you'd need to get the updated puppet-keystone code into your overcloud-full.qcow image first | 21:59 |
| dprince | ayoung: we've got upstream ways to update puppet modules on-the-fly but sadly that hasn't all landed :/ | 21:59 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: puppet: run keystone in wsgi https://review.openstack.org/213175 | 22:00 |
| EmilienM | dprince: no need of depends | 22:01 |
| *** jhenner has quit IRC | 22:01 | |
| dprince | EmilienM: if we don't do depends on we are missing the 'admin_bind_host' setting | 22:01 |
| dprince | EmilienM: it functionally would be incomplete... | 22:01 |
| EmilienM | ah ok | 22:02 |
| EmilienM | right | 22:02 |
| ayoung | dprince, I can regen the images. | 22:02 |
| dprince | export DIB_REPOREF_puppet_keystone=refs/changes/41/273241/2 | 22:03 |
| ayoung | dprince, that for tripleo.sh? | 22:04 |
| dprince | ayoung: source that first ^^^ | 22:04 |
| dprince | ayoung: that is for diskimage-builder | 22:04 |
| ayoung | dprince, will tripleo-common/scripts/tripleo.sh --overcloud-images honor that or should I call it by hand? | 22:05 |
| dprince | ayoung: setting it should (I think) get it propigated down to the right places | 22:05 |
| dprince | ayoung: you probably aren't using network isolation though | 22:05 |
| dprince | ayoung: just test the t-h-t patch, if that works you'll be happy | 22:05 |
| ayoung | OK | 22:05 |
| dprince | ayoung: trying to save you some time... | 22:05 |
| dprince | ayoung: CI will test it anyways if you don't want to bother, but I'm guessing you want to be hands on with it | 22:06 |
| ayoung | dprince, yeah, plus it is about time I learned the install proces | 22:06 |
| dprince | ayoung: we would value your expertise on having a go at it. But you may want to wait till CI passes it so as not to wast time | 22:06 |
| *** jhenner1 has quit IRC | 22:06 | |
| dprince | ayoung: our CI is a bit intermittent this week, something slipped in :/ | 22:07 |
| ayoung | dprince, so I'm just rerunning overcloud deploy with the template dire pointing at your latest | 22:07 |
| *** marcusvrn_ has quit IRC | 22:07 | |
| dprince | ayoung: the non-ha job may pass sooner than the others http://tripleo.org/cistatus.html | 22:07 |
| ayoung | dprince, there was some concernt that the HA job will get messed up by having Keystone running in HTTPD due to the way things get restarted | 22:08 |
| *** egafford has quit IRC | 22:08 | |
| *** jdob has quit IRC | 22:08 | |
| ayoung | TBH, for now I would be happy with at least a non HA success that I could then test Federation against | 22:08 |
| dprince | ayoung: yeah, I would defer to the pacemaker experts on this. For the non-ha puppet t-h-t integration now I like the patch now | 22:09 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1538761 | 22:10 |
| openstack | Launchpad bug 1538761 in tripleo "stable/liberty HA: mysqld on overcloud failing to start with /usr/libexec/mysqld: option '--wsrep_notify_cmd' requires an argument" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| *** dprince has quit IRC | 22:11 | |
| *** lblanchard has quit IRC | 22:15 | |
| bnemec | slagle: I think we probably need to just merge https://review.openstack.org/#/c/272194/ | 22:15 |
| *** lblanchard has joined #tripleo | 22:15 | |
| *** lblanchard has quit IRC | 22:15 | |
| bnemec | It did pass one CI job, and the other two failed on ping test issues so we could probably just pull the trigger. | 22:17 |
| slagle | bnemec: yea i just saw that | 22:18 |
| slagle | it's had a collective pass of all 3 jobs :) | 22:19 |
| slagle | i merged it | 22:20 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove empty value for wsrep_notify_cmd https://review.openstack.org/272194 | 22:20 |
| *** jhenner has joined #tripleo | 22:21 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates https://review.openstack.org/273233 | 22:21 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Resolve sylinks when unmounting https://review.openstack.org/273260 | 22:22 |
| *** penick has quit IRC | 22:27 | |
| *** thrash is now known as thrash|pto | 22:31 | |
| *** rpothier has quit IRC | 22:36 | |
| *** penick has joined #tripleo | 22:37 | |
| *** Goneri has quit IRC | 22:39 | |
| *** olap has quit IRC | 22:45 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Resolve sylinks when unmounting https://review.openstack.org/273260 | 22:54 |
| *** jcoufal has quit IRC | 22:55 | |
| *** jdob_lt has joined #tripleo | 22:59 | |
| *** jdob_lt has left #tripleo | 23:00 | |
| *** tiswanso has quit IRC | 23:00 | |
| *** tiswanso has joined #tripleo | 23:00 | |
| *** yuanying_ has quit IRC | 23:02 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates https://review.openstack.org/273233 | 23:03 |
| *** tiswanso has quit IRC | 23:05 | |
| *** davidlenwell has quit IRC | 23:06 | |
| *** davidlenwell has joined #tripleo | 23:08 | |
| *** ChanServ sets mode: +v davidlenwell | 23:08 | |
| *** weshay has quit IRC | 23:11 | |
| *** yuanying has joined #tripleo | 23:12 | |
| *** chlong has quit IRC | 23:18 | |
| *** chlong has joined #tripleo | 23:30 | |
| *** dmacpher has quit IRC | 23:43 | |
| *** trozet has quit IRC | 23:44 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates https://review.openstack.org/273233 | 23:55 |
| *** pradk_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!