| *** rhallisey has quit IRC | 00:19 | |
| *** saneax is now known as saneax_AFK | 00:21 | |
| *** lblanchard has joined #tripleo | 00:34 | |
| *** rhallisey has joined #tripleo | 01:01 | |
| *** ccamacho has quit IRC | 01:11 | |
| *** penick has joined #tripleo | 01:19 | |
| *** penick has quit IRC | 01:28 | |
| *** dmacpher-afk is now known as dmacpher | 01:37 | |
| *** shivrao has quit IRC | 01:43 | |
| *** Marga_ has quit IRC | 01:43 | |
| *** openstackgerrit has quit IRC | 01:47 | |
| *** openstackgerrit has joined #tripleo | 01:48 | |
| *** tiswanso has joined #tripleo | 01:48 | |
| *** tiswanso has quit IRC | 01:49 | |
| *** tiswanso has joined #tripleo | 01:50 | |
| *** lblanchard has quit IRC | 02:08 | |
| *** shivrao has joined #tripleo | 02:10 | |
| openstackgerrit | Richard Su proposed openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/287561 | 02:15 |
|---|---|---|
| *** yuanying_ has joined #tripleo | 02:15 | |
| *** Marga_ has joined #tripleo | 02:17 | |
| *** trown has joined #tripleo | 02:18 | |
| *** Marga_ has quit IRC | 02:18 | |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Store events in Undercloud Ceilometer https://review.openstack.org/286734 | 02:18 |
| *** hoobaman has joined #tripleo | 02:18 | |
| *** lynxman_ has joined #tripleo | 02:19 | |
| *** Marga_ has joined #tripleo | 02:19 | |
| *** adarazs_ has joined #tripleo | 02:22 | |
| *** stevebak` has joined #tripleo | 02:22 | |
| *** jrist_ has joined #tripleo | 02:23 | |
| *** stevebaker has quit IRC | 02:23 | |
| *** therve has quit IRC | 02:23 | |
| *** lynxman has quit IRC | 02:23 | |
| *** adarazs has quit IRC | 02:23 | |
| *** hoobaman_ has quit IRC | 02:23 | |
| *** panda has quit IRC | 02:23 | |
| *** yuanying has quit IRC | 02:23 | |
| *** jrist has quit IRC | 02:23 | |
| *** trown|outtypewww has quit IRC | 02:23 | |
| *** lynxman_ is now known as lynxman | 02:23 | |
| *** adarazs_ is now known as adarazs | 02:23 | |
| *** therve has joined #tripleo | 02:23 | |
| *** yamahata has quit IRC | 02:24 | |
| *** panda has joined #tripleo | 02:24 | |
| *** rhallisey has quit IRC | 02:36 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 02:43 |
| *** rhallisey has joined #tripleo | 02:44 | |
| *** jrist_ is now known as jrist | 02:52 | |
| *** jrist has quit IRC | 02:52 | |
| *** jrist has joined #tripleo | 02:52 | |
| *** tzumainn has quit IRC | 02:52 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 03:04 |
| *** yuanying has joined #tripleo | 03:21 | |
| *** michchap_ has joined #tripleo | 03:22 | |
| *** xinwu_ has joined #tripleo | 03:22 | |
| *** nijaba_ has joined #tripleo | 03:24 | |
| *** pino|work_ has joined #tripleo | 03:25 | |
| *** dtantsur has joined #tripleo | 03:26 | |
| *** yamahata has joined #tripleo | 03:26 | |
| *** xinwu_ has quit IRC | 03:26 | |
| *** dtantsur|afk has quit IRC | 03:26 | |
| *** ryansb has quit IRC | 03:26 | |
| *** pino|work has quit IRC | 03:26 | |
| *** nijaba has quit IRC | 03:26 | |
| *** yuanying_ has quit IRC | 03:26 | |
| *** xinwu has quit IRC | 03:26 | |
| *** HenryG has quit IRC | 03:26 | |
| *** Tristitia has quit IRC | 03:26 | |
| *** michchap has quit IRC | 03:26 | |
| *** leifmadsen has quit IRC | 03:26 | |
| *** dmacpher is now known as dmacpher-lunch | 03:26 | |
| *** HenryG has joined #tripleo | 03:26 | |
| *** leifmadsen has joined #tripleo | 03:27 | |
| *** ryansb has joined #tripleo | 03:28 | |
| *** ryansb has joined #tripleo | 03:28 | |
| *** Tristit1a has joined #tripleo | 03:28 | |
| *** links has joined #tripleo | 03:29 | |
| *** bandini has quit IRC | 03:30 | |
| *** bandini has joined #tripleo | 03:32 | |
| *** rasca has quit IRC | 03:36 | |
| *** panda has quit IRC | 03:37 | |
| *** rasca has joined #tripleo | 03:38 | |
| *** panda has joined #tripleo | 03:40 | |
| *** Tristit1a is now known as Tristitia | 03:43 | |
| *** dmacpher-lunch is now known as dmacpher | 04:10 | |
| *** trown is now known as trown|outtypewww | 04:17 | |
| *** Goneri has quit IRC | 04:17 | |
| *** Goneri has joined #tripleo | 04:21 | |
| *** tiswanso has quit IRC | 04:33 | |
| *** masco has joined #tripleo | 04:46 | |
| openstackgerrit | Richard Su proposed openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/287561 | 04:46 |
| *** Marga_ has quit IRC | 04:50 | |
| *** Marga_ has joined #tripleo | 05:16 | |
| *** Marga_ has quit IRC | 05:16 | |
| *** Marga_ has joined #tripleo | 05:16 | |
| *** cmyster has quit IRC | 05:23 | |
| *** shivrao has quit IRC | 05:28 | |
| *** saneax_AFK is now known as saneax | 05:28 | |
| *** xinwu has joined #tripleo | 05:33 | |
| *** mannidi has joined #tripleo | 05:48 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add Management Network For System Administration. https://review.openstack.org/264963 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add all isolated networks to all nodes. https://review.openstack.org/268833 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Support network isolation without external nets https://review.openstack.org/287599 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix yaml validation errors in multiple-nics templates https://review.openstack.org/287601 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Adding ManagementIpSubnet to linux bridge net conf https://review.openstack.org/287602 | 05:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change default bond-mode https://review.openstack.org/287603 | 05:52 |
| openstackgerrit | Merged openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 06:07 |
| *** prometheanfire has left #tripleo | 06:20 | |
| *** jtomasek has joined #tripleo | 06:33 | |
| openstackgerrit | Sergey Gotliv proposed openstack/puppet-tripleo: Sahara integration https://review.openstack.org/271038 | 06:41 |
| *** dshulyak has joined #tripleo | 06:42 | |
| *** shivrao has joined #tripleo | 06:53 | |
| *** rlandy has quit IRC | 07:04 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287620 | 07:05 |
| *** jaosorior has joined #tripleo | 07:15 | |
| *** rdopiera has joined #tripleo | 07:23 | |
| openstackgerrit | Sergey Gotliv proposed openstack/tripleo-heat-templates: Sahara Integration https://review.openstack.org/272614 | 07:24 |
| openstackgerrit | Sergey Gotliv proposed openstack/tripleo-heat-templates: Removing Sahara password default https://review.openstack.org/272627 | 07:26 |
| *** ccamacho has joined #tripleo | 07:27 | |
| *** jprovazn has joined #tripleo | 07:39 | |
| jaosorior | Is anyone getting these errors still? Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password MQ67mevhB4RRReeR6bnz2ujqz returned 1 instead of one of [0] | 07:40 |
| *** jcoufal has joined #tripleo | 07:42 | |
| *** pcaruana has joined #tripleo | 07:45 | |
| *** mikelk has joined #tripleo | 07:59 | |
| *** mkovacik has joined #tripleo | 08:02 | |
| *** mkovacik has quit IRC | 08:11 | |
| *** fgimenez has joined #tripleo | 08:11 | |
| *** aufi has joined #tripleo | 08:11 | |
| *** mkovacik has joined #tripleo | 08:12 | |
| *** mkovacik has quit IRC | 08:12 | |
| hewbrocca | jaosorior: morning | 08:12 |
| jaosorior | hewbrocca: Hey dude | 08:13 |
| hewbrocca | Anyone have any idea if dprince was able to get the testenvs reworked for network isolation last night | 08:14 |
| *** athomas has joined #tripleo | 08:16 | |
| *** ohamada has joined #tripleo | 08:17 | |
| *** hjensas has joined #tripleo | 08:21 | |
| *** shardy has joined #tripleo | 08:37 | |
| *** devvesa has joined #tripleo | 08:39 | |
| jaosorior | marios, shardy: Have you guys seen this error when doing an HA deployment? Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password MQ67mevhB4RRReeR6bnz2ujqz returned 1 instead of one of [0] | 08:42 |
| marios | jaosorior: yeah i think there was a bug for it, lemme check | 08:43 |
| marios | jaosorior: might be this (bug linked in review )https://review.openstack.org/#/c/286352/ "controller/ha: disable keystone-manage bootstrap." | 08:44 |
| jaosorior | marios: Ah, nice, thanks! | 08:45 |
| *** olap has joined #tripleo | 08:46 | |
| *** hjensas has quit IRC | 08:56 | |
| *** ohamada has quit IRC | 09:03 | |
| *** hjensas has joined #tripleo | 09:04 | |
| *** ohamada has joined #tripleo | 09:04 | |
| *** mbound has joined #tripleo | 09:05 | |
| *** ishant has joined #tripleo | 09:08 | |
| *** akrivoka has joined #tripleo | 09:08 | |
| ishant | hi folks, Can someone please review https://review.openstack.org/#/c/284725/ ? | 09:08 |
| *** bvandenh has joined #tripleo | 09:11 | |
| *** cmyster has joined #tripleo | 09:11 | |
| *** liverpooler has joined #tripleo | 09:11 | |
| *** athomas has quit IRC | 09:14 | |
| *** jistr has joined #tripleo | 09:17 | |
| *** lucas-dinner is now known as lucasagomes | 09:17 | |
| *** athomas has joined #tripleo | 09:20 | |
| *** shivrao_ has joined #tripleo | 09:25 | |
| *** shivrao has quit IRC | 09:28 | |
| *** xinwu has quit IRC | 09:28 | |
| *** cmyster has quit IRC | 09:29 | |
| *** shivrao_ has quit IRC | 09:30 | |
| *** olap has quit IRC | 09:31 | |
| *** olap has joined #tripleo | 09:33 | |
| jprovazn | OC fails to deploy with "DiscoveryFailure: Could not determine a suitable URL for the plugin" in neutron logs - http://paste.openstack.org/show/489104/ , anyone hit this? | 09:40 |
| *** cmyster has joined #tripleo | 09:41 | |
| *** cmyster has quit IRC | 09:41 | |
| *** cmyster has joined #tripleo | 09:41 | |
| *** mgould has joined #tripleo | 09:43 | |
| *** panda is now known as loser_ | 09:49 | |
| *** openstackgerrit_ has joined #tripleo | 09:53 | |
| *** leanderthal|afk is now known as leanderthal | 09:54 | |
| openstackgerrit | Merged openstack/tripleo-docs: Recommend using root device hints even in the basic deployment https://review.openstack.org/282298 | 10:00 |
| *** openstackgerrit_ has quit IRC | 10:02 | |
| *** olap has quit IRC | 10:09 | |
| *** olap has joined #tripleo | 10:10 | |
| *** cmyster has quit IRC | 10:12 | |
| *** cmyster has joined #tripleo | 10:12 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 10:15 |
| *** olap has quit IRC | 10:20 | |
| *** olap has joined #tripleo | 10:22 | |
| *** fgimenez has quit IRC | 10:31 | |
| dtantsur | Morning TripleOwl! | 10:33 |
| *** fgimenez has joined #tripleo | 10:35 | |
| *** fgimenez has quit IRC | 10:35 | |
| *** fgimenez has joined #tripleo | 10:35 | |
| shadower | do we have a fix for the str_replace thing? | 10:35 |
| shardy | shadower: yes, sec | 10:35 |
| hewbrocca | LOL yes | 10:36 |
| shardy | https://review.openstack.org/#/c/282403/ | 10:36 |
| shadower | I admit, the question was more about "can someone paste the link here" | 10:36 |
| shardy | shadower: that's the liberty backport, the master commit is referenced in the t-h-t commit: | 10:37 |
| shadower | shardy: thanks | 10:37 |
| shardy | https://review.openstack.org/#/c/275605/ | 10:37 |
| shardy | "Note that this depends on a fix for heat bug #1539737" | 10:37 |
| openstack | bug 1539737 in heat liberty "str_replace params can't be from parameters" [Undecided,Fix committed] https://launchpad.net/bugs/1539737 - Assigned to Steven Hardy (shardy) | 10:37 |
| *** ishant has quit IRC | 10:43 | |
| openstackgerrit | Merged openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector https://review.openstack.org/277974 | 10:43 |
| openstackgerrit | Merged openstack/instack-undercloud: Enable IPA debug logging during introspection when undercloud_debug is True https://review.openstack.org/286096 | 10:44 |
| *** olap has quit IRC | 10:44 | |
| hewbrocca | shardy: hey, do you have any idea how far dprince got with the network isolation setup on the CI rack | 10:48 |
| shardy | hewbrocca: Hey, sorry I'm not sure on the status of that - I think derekh would be the person to ask, but I don't think he's around today | 10:50 |
| hewbrocca | yeah, he's out today, and dprince didn't send any status yesterday | 10:50 |
| hewbrocca | I guess we'll just have to wait until he gets in | 10:50 |
| shardy | yeah, sorry about that | 10:50 |
| hewbrocca | not your fault :) | 10:50 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: quiet yum update https://review.openstack.org/287707 | 10:52 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: install zaqarclient https://review.openstack.org/287708 | 10:52 |
| lucasagomes | shardy, morning. Any updates re UEFI? | 10:58 |
| *** tosky has joined #tripleo | 10:59 | |
| shardy | lucasagomes: Hi, not yet, been working my way through morning code reviews first :) | 11:00 |
| lucasagomes | shardy, oh fair enough, lemme know if something is needed | 11:01 |
| shardy | lucasagomes: sure, will do, thanks for the help! | 11:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector https://review.openstack.org/287713 | 11:02 |
| dtantsur | liberty backport ^^ | 11:03 |
| shardy | dtantsur: Hey, FYI I tested that and it works really nicely :) | 11:03 |
| dtantsur | cool :) | 11:04 |
| openstackgerrit | Merged openstack/instack-undercloud: run keystone in a wsgi process https://review.openstack.org/282535 | 11:04 |
| shardy | dtantsur: question re backport - is it reasonable to expect existing users, e.g RDO, to have images with the python-hardware library? | 11:04 |
| dtantsur | shardy, we've merged the appropriate change quite some time ago... but I can swap the default to False, if it makes you feel safer | 11:04 |
| dtantsur | (then we'll have to make it True again downstream, I guess) | 11:05 |
| shardy | dtantsur: No, it's fine, I just wanted to confirm - we could check with trown, but I'm just wary of anything that might break folks with existing images on update | 11:05 |
| dtantsur | trown|outtypewww, ^^^ | 11:05 |
| shardy | Any idea which commit switched the image build to include the library? | 11:05 |
| dtantsur | lemme find | 11:05 |
| dtantsur | shardy, https://github.com/openstack/python-tripleoclient/commit/d702e9992a23da6e3910367f8e6ca1f638b2a15d merged 21 days ago | 11:06 |
| shardy | Ok, let's chat with trown before landing then, as that's not been present in tripleoclient for a super-long time | 11:07 |
| dtantsur | shardy, on one hand we don't want people to get broken by just updating u-c.. on the other, this change only matters when (re)building undercloud | 11:07 |
| *** masco has quit IRC | 11:08 | |
| shardy | dtantsur: well, I guess that's true - folks might re-run the undercloud install command for other reasons tho | 11:08 |
| *** links has quit IRC | 11:08 | |
| shardy | I guess that's somewhat a corner case which we could just document tho | 11:08 |
| dtantsur | shardy, though rebuilding is a serious concern indeed... I'd err on a safer side, and swap it to False for upstream liberty... any objections to such change in the backport? | 11:09 |
| *** masco has joined #tripleo | 11:09 | |
| shardy | dtantsur: that's fine with me, just need to call it out in the commit message | 11:09 |
| dtantsur | definitely.. then we'll reenable it with a downstream patch | 11:09 |
| shardy | +1 | 11:09 |
| dtantsur | also good morning pic for this channel: https://40.media.tumblr.com/092e1ac4b7572ed173bccc1730e4cd8d/tumblr_o3fn7zLIEC1ryf03io1_540.jpg | 11:10 |
| *** dmsimard has quit IRC | 11:10 | |
| *** links has joined #tripleo | 11:10 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector https://review.openstack.org/287713 | 11:17 |
| dtantsur | shardy, ^^ | 11:17 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 11:19 |
| openstackgerrit | Merged openstack/python-tripleoclient: Use the grub2 element instead of grub2-deprecated https://review.openstack.org/277842 | 11:19 |
| *** chem has quit IRC | 11:21 | |
| *** mannidi has quit IRC | 11:21 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 11:31 |
| hewbrocca | Owlet! | 11:33 |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Fix bad backport for 'openstack overcloud profiles' commands https://review.openstack.org/287734 | 11:33 |
| hewbrocca | very nide | 11:33 |
| shadower | Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::nova::db::mysql_api at /var/lib/heat-config/heat-config-puppet/420b2f5a-cc4a-4b90-bd91-6265421ac0fc.pp:506 on node overcloud-controller-0.localdomain | 11:34 |
| shadower | anyone else seeing that? | 11:34 |
| *** electrofelix has joined #tripleo | 11:37 | |
| *** loser_ has quit IRC | 11:38 | |
| *** loser_ has joined #tripleo | 11:38 | |
| *** miles has joined #tripleo | 11:39 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Fix bad backport for 'openstack overcloud profiles' commands https://review.openstack.org/287734 | 11:39 |
| dtantsur | shardy, ^^ | 11:39 |
| *** yamahata has quit IRC | 11:39 | |
| shardy | thanks! | 11:39 |
| *** mgould has quit IRC | 11:40 | |
| dtantsur | (referencing commit ID as review ID is useless outside of gerrit) | 11:40 |
| shardy | dtantsur: hey, I was looking to revive https://review.openstack.org/#/c/255228 so we can land it for mitaka (allow yaml nodes file) | 11:40 |
| shardy | I'm not super familiar with openstackclient - I assume the --type is wired in as a common option to osc, and we can deprecate the old json/csv options? | 11:40 |
| shardy | I just copied what was there, obviously ;) | 11:40 |
| dtantsur | shardy, --type is not something standard afaik, I'm just suggesting a more OSC-ish way to handle it.. | 11:41 |
| dtantsur | shardy, actually I see much more value in detecting the format by extension | 11:41 |
| shardy | dtantsur: sure, I agree - I was just a bit confused by the "please deprecate old options" remark | 11:42 |
| dtantsur | cause really typing --yaml file.yaml is a bit lame :) | 11:42 |
| shardy | e.g it implies there's a "new" way I should be doing it :) | 11:42 |
| dtantsur | shardy, well... ideally it should be --file-type=yaml --file-type=csv etc | 11:42 |
| shardy | dtantsur: agreed, I can for sure add some auto-detecting stuff | 11:42 |
| dtantsur | what we do now feels a bit wrong.. but I can leave with it, especially if we add autodetection | 11:43 |
| shardy | dtantsur: Ok, thanks - I'll try reworking it later, just wanted to confirm as I couldn't see any osc type args to plug into | 11:43 |
| dtantsur | shardy, yeah, it's not an existing thing, it's just an example how we should have implemented it | 11:44 |
| shardy | ack, I'll see what I can do to improve it, thanks! | 11:44 |
| *** ohamada has quit IRC | 11:45 | |
| *** ohamada has joined #tripleo | 11:48 | |
| slagle | shardy: hi, when you get a chance could you re-review https://review.openstack.org/#/c/266602/, i think i make the improvement you were asking for | 11:53 |
| shardy | slagle: thanks, sorry I thought I'd done that one yesterday | 11:56 |
| *** olap has joined #tripleo | 11:56 | |
| shardy | slagle: you can actually unregister nodes via e.g the spacecmd tool which talks to the xmlrpc API | 11:56 |
| shardy | but I guess we won't have credentials to do that | 11:56 |
| *** cmyster has quit IRC | 11:56 | |
| slagle | shardy: i didnt know about spacecmd, but it doesnt look like it's installed by default on rhel | 11:57 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add Satellite 5 support https://review.openstack.org/266602 | 12:00 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add Satellite 5 support https://review.openstack.org/287745 | 12:00 |
| *** saneax is now known as saneax_AFK | 12:02 | |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** openstackgerrit has joined #tripleo | 12:03 | |
| *** bvandenh has quit IRC | 12:03 | |
| *** rhallisey has joined #tripleo | 12:17 | |
| *** rhallisey has quit IRC | 12:18 | |
| *** rhallisey has joined #tripleo | 12:18 | |
| *** gfidente has joined #tripleo | 12:21 | |
| gfidente | marios, jrist I have something I want to share with you guys | 12:22 |
| hewbrocca | gfidente: I'm dying of suspense | 12:25 |
| marios | sup gfidente | 12:28 |
| *** thrash|g0ne is now known as thrash | 12:29 | |
| *** sshnaidm has quit IRC | 12:30 | |
| *** NobodyCam has quit IRC | 12:31 | |
| *** NobodyCam has joined #tripleo | 12:31 | |
| gfidente | hewbrocca, marios it's not *so* great | 12:32 |
| gfidente | https://fedorapeople.org/~gfidente/pics/tripleo_the_real_thing.jpg | 12:32 |
| gfidente | but I couldn't resist | 12:32 |
| *** sshnaidm has joined #tripleo | 12:35 | |
| openstackgerrit | Merged openstack/diskimage-builder: Replace sfdisk partitioning with parted https://review.openstack.org/258721 | 12:35 |
| slagle | that owl looks really zoned out | 12:36 |
| *** jaosorior has quit IRC | 12:39 | |
| gfidente | slagle, I knew you would say it | 12:39 |
| gfidente | but you didn't see jrist's owl that's why | 12:39 |
| *** jaosorior has joined #tripleo | 12:40 | |
| *** dprince has joined #tripleo | 12:40 | |
| shadower | gfidente: did you come across these deployment errors? http://paste.openstack.org/show/489123/ | 12:41 |
| shadower | I'm getting them when trying to deploy the overcloud (both master THT and with the ipv6 patches) | 12:41 |
| gfidente | shadower, nope, I had an old version of memcached so memcached doesn't bind on the ipv6 address though | 12:42 |
| gfidente | but you don't see that? are you deploying on centos? | 12:42 |
| shadower | gfidente: on rhel | 12:42 |
| *** dprince has quit IRC | 12:46 | |
| *** masco has quit IRC | 12:46 | |
| gfidente | shadower, so I'm going to try that, back in a few | 12:47 |
| shadower | could you point me in the right direction on debugging these? I looked at the https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller_pacemaker.pp#L441 | 12:48 |
| shadower | but it looks like the other ones | 12:48 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Moves the swift start/stop into the common_functions.sh file https://review.openstack.org/287320 | 12:48 |
| shadower | gfidente: ^ | 12:49 |
| shadower | so could that mean that ::nova::db::mysql_api module isn't on the machine? Where/how could I check that? | 12:49 |
| gfidente | shadower, so the error on the compute looks like the version of puppet-nova is not good for the templates | 12:51 |
| gfidente | the module expects neutron_admin_password and we are not passing it | 12:51 |
| gfidente | is this was changed in the puppet module we probably need to figure which patch it is and add into OPM | 12:52 |
| gfidente | and I'd say this is same reason for the error you see on the controller too | 12:52 |
| shadower | yeah I was afraid of that | 12:53 |
| *** dprince has joined #tripleo | 12:53 | |
| gfidente | I think we need to git blame puppet-nova and see what we miss | 12:54 |
| gfidente | shadower, I'd say this is the problem we're trying to fix with the idea of deploying the modules as artifacts | 12:57 |
| gfidente | dprince, ^^ | 12:57 |
| gfidente | (I mean in general) | 12:57 |
| dprince | gfidente: is something wrong w/ puppet-nova today? | 12:59 |
| gfidente | dprince, no not in CI | 13:00 |
| gfidente | it's probably the OPM builds | 13:00 |
| gfidente | was just pointing out we figured this problem existed and had ideas | 13:01 |
| *** miles is now known as mgould | 13:01 | |
| gfidente | *I should say exists | 13:01 |
| dprince | gfidente: the puppet module swift thing finally landed. | 13:02 |
| gfidente | yes exactly | 13:02 |
| gfidente | that | 13:02 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287784 | 13:03 |
| *** morazi has joined #tripleo | 13:05 | |
| jprovazn | gfidente: hi, when deploying OSP8, did you hit a similar issue? <jprovazn> OC fails to deploy with "DiscoveryFailure: Could not determine a suitable URL for the plugin" in neutron logs - http://paste.openstack.org/show/489104/ , anyone hit this? | 13:05 |
| gfidente | jprovazn, this is in the undercloud? | 13:06 |
| jprovazn | gfidente: yes | 13:06 |
| gfidente | it's the notifications thing it seems | 13:06 |
| gfidente | I think slagle or bnemec are your guys | 13:06 |
| jprovazn | gfidente: slagle: bnemec: by making manual changes in /etc/neutron/neutron.conf (commented in project id) I was able to mitiage the exception but I can't still boot an instance with error: RESP BODY: {"itemNotFound": {"message": "No instances found for any event", "code": 404}} | 13:08 |
| slagle | jprovazn: you get that when trying to deploy overcloud nodes? | 13:10 |
| jprovazn | slagle: yes | 13:10 |
| hewbrocca | dprince: ! | 13:10 |
| jprovazn | slagle: err, no | 13:10 |
| jprovazn | slagle: the last "no instances" when trying to boot a single instances | 13:10 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add HostnameMap to allow granular control of hostnames https://review.openstack.org/286739 | 13:10 |
| hewbrocca | dprince: did you have any luck with the net-iso stuff last night | 13:10 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287784 | 13:11 |
| jprovazn | slagle: the first fpaste http://paste.openstack.org/show/489104/ is when trying to deploy OC | 13:11 |
| slagle | jprovazn: a single instance on the overcloud? or the undercloud? | 13:11 |
| jprovazn | slagle: on undercloud | 13:11 |
| dprince | hewbrocca: progress... I'm trying not to break CI so testing very carefully | 13:11 |
| hewbrocca | dprince: excellent | 13:11 |
| hewbrocca | Obviously, there is a reason I keep harassing you about this... I'm anxious to get IPv6 landed | 13:13 |
| *** pradk has joined #tripleo | 13:13 | |
| jprovazn | slagle: I *think* neutron.conf was not configured properly on UC by default - I have project_domain_id=default in neutron.conf which causes then that keystone api endopoint v2.0 is ignored - https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/auth/identity/generic/base.py#L162 | 13:13 |
| *** dmsimard has joined #tripleo | 13:14 | |
| slagle | jprovazn: i also have project_domain_id=default, and deployed a new oc node fine yesterday | 13:16 |
| *** jayg|g0n3 is now known as jayg | 13:17 | |
| jprovazn | slagle: that makes me unhappy :) | 13:17 |
| jayg | gfidente: is your issue with puppet-nova using master of opm/delorean? | 13:19 |
| gfidente | jayg, I think I found the two submissions which blocked shadower | 13:20 |
| gfidente | https://review.openstack.org/#/c/257522/ | 13:20 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Upgrades: quiet yum update https://review.openstack.org/287707 | 13:20 |
| gfidente | https://review.openstack.org/#/c/274280/ | 13:20 |
| gfidente | jayg, I am not sure what is the approach here though | 13:20 |
| jayg | gfidente: ah, cool, was going to say that I just updated opm yesterday or the day before, so I could give you a commit list to puppet-nova | 13:20 |
| gfidente | if we prefer to port those to liberty or to not use the features in tht | 13:20 |
| * jayg looks | 13:20 | |
| gfidente | jayg, I think we need them in liberty eventually because this is for the liberty release | 13:21 |
| gfidente | dprince, what's your take? | 13:21 |
| gfidente | do we port the puppet nova changes to liberty too or rather not use them in tht/liberty? | 13:21 |
| gfidente | cause you own both | 13:21 |
| gfidente | changes | 13:21 |
| jayg | I have no idea if they are needed in liberty, I haven't looked at these before, though I see EmilienM was a reviewer, so he may know | 13:22 |
| jayg | oh, and co-author on one | 13:22 |
| gfidente | jayg, I think porting is good | 13:23 |
| jayg | yeah, if it is not a hardship, and will make less change from liberty -> mitaka, I would go for it | 13:23 |
| gfidente | me too | 13:23 |
| gfidente | let's see if it's hard :) | 13:23 |
| gfidente | jayg, which one you take? | 13:24 |
| jayg | gfidente: haha, you mean liberty backport or changing tht? | 13:24 |
| gfidente | porting the changes to liberty | 13:24 |
| jayg | I can give it a go right now | 13:24 |
| gfidente | yep which one of the two? | 13:25 |
| gfidente | so I do the other | 13:25 |
| jayg | they both have merge conflict, yay | 13:26 |
| *** pino|work_ is now known as pino|work | 13:26 | |
| jayg | gfidente: I don't care, I can take the api_db one if you want | 13:26 |
| gfidente | jayg, no that actually seems to go for mitaka only | 13:27 |
| gfidente | so we need a change to not do that in tht I think | 13:27 |
| gfidente | for tht/liberty | 13:28 |
| *** trown|outtypewww is now known as trown | 13:28 | |
| jayg | ah, tht/liberty is already doing it the mitaka way? | 13:28 |
| *** rlandy has joined #tripleo | 13:28 | |
| * jayg confused | 13:28 | |
| trown | good morning tripleo | 13:30 |
| slagle | mornin | 13:30 |
| trown | shardy: there is a pretty nasty bug in heat on trunk, but I am not really sure what details would be helpful to file it on launchpad | 13:31 |
| trown | shardy: stack-delete fails unconditionally (both deleting overcloud stacks and the ping test cleanup) | 13:32 |
| *** fgimenez has quit IRC | 13:32 | |
| *** links has quit IRC | 13:34 | |
| *** fgimenez has joined #tripleo | 13:34 | |
| shardy | trown: Ok, got any more details? | 13:35 |
| d0ugal | Does anyone know where the Swift rings are setup? tht? puppet? | 13:35 |
| shardy | I saw an issue today where if you delete an in-progress stack, nova can 500 which causes the delete to fail, but this sounds different | 13:35 |
| trown | shardy: this happens with succesful stacks and unsuccesful stacks, and the stack actually gets fully cleaned up except for the heat database | 13:36 |
| trown | shardy: as in all of the nova/neutron stuff gets deleted, but there is still a DELETE_FAILED stack sticking around, and retrying the delete fails every time | 13:37 |
| trown | shardy: rhallisey hit the same thing | 13:37 |
| trown | shardy: it is really easy to reproduce using trunk heat, what other detail would be helpful? | 13:38 |
| *** dmacpher has quit IRC | 13:42 | |
| *** dmacpher has joined #tripleo | 13:42 | |
| shardy | trown: I've been deleting stacks all day and other than the nova thing, I've not seen this at all | 13:42 |
| shardy | trown: if you can raise a launchpad bug with more details that would be good - even better if there's a non-TripleO (e.g devstack) reproducer | 13:42 |
| shardy | we have functional tests for deleting stacks, so it'd be a little surprising if deleting is completely broken | 13:43 |
| shardy | possibly something tripleo specific, but we'll have to figure out what from the logs | 13:43 |
| trown | shardy: have you been using trunk heat? or the repo setup of tripleoci | 13:43 |
| shardy | e.g trace the cause of the DELETE_FAILED back from he heat-engine.log | 13:43 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Function library for major upgrades https://review.openstack.org/285416 | 13:44 |
| trown | shardy: the main evidence of it not being tripleo specific is the simple ping test stack on the overcloud failing in the same way | 13:44 |
| trown | delete failing I should say, which makes the ping test hang | 13:45 |
| dtantsur | liberty backport that passed gate, 1x +2 required: https://review.openstack.org/#/c/287713/ :) | 13:45 |
| shardy | trown: ack - I may have a slightly older (week old) heat actually, it's locally built but I've not updated for a few days due to PTO | 13:45 |
| shardy | trown: let me pull latest and try to reproduce | 13:45 |
| trown | shardy: ah yep, this started yesterday | 13:46 |
| *** apetrich has quit IRC | 13:47 | |
| *** apetrich has joined #tripleo | 13:47 | |
| *** weshay has joined #tripleo | 13:49 | |
| *** hjensas has quit IRC | 13:59 | |
| shardy | trown: Ok, I pulled latest heat and so far can't reproduce, can you please raise a LP bug including the heat-engine traceback associated with the DELETE_FAILED? | 14:04 |
| shardy | there are quite a few possible reasons for a delete to fail, not all heat bugs :) | 14:04 |
| trown | shardy: sure will do | 14:04 |
| trown | shardy: I will try to get a trace from both the pingtest and the overcloud scenarios | 14:05 |
| shardy | trown: great, that should give us a few clues, thanks! | 14:05 |
| jaosorior | shardy: What was the error you where seeing when deleting the overcloud stack? I'm seeing something where deleting the stack just stalls when trying to delete the nova servers | 14:05 |
| jaosorior | happens if I deploy with network isolation | 14:05 |
| shardy | jaosorior: nova was returning a 500 when interruping a deployment and trying to delete the stack when the nodes were still spawning (ironic nodes in wait-call-back) | 14:06 |
| shardy | not yet fully investigated, it may be a nova or ironic bug tho | 14:06 |
| shardy | lucasagomes: hey, so I re-tested with boot_mode:uefi, and got some strange results | 14:07 |
| jaosorior | shardy: That's different in my case then. It just stalls trying to delete the nova servers | 14:07 |
| shardy | jaosorior: I'd check the nova logs I guess | 14:08 |
| shardy | lucasagomes: so, with just boot_option:local (no boot_mode:uefi), the box will pxe boot OK, and the image appears deployed, but then won't boot (assume broken bootloader, just a flashing cursor, no grub) | 14:08 |
| shardy | lucasagomes: however, if you then add boot_mode:uefi, it won't pxe boot anymore, but something happens which fixes the bootloaded, so it will always boot from the local disk | 14:09 |
| shardy | e.g the image deployed without boot_mode:uefi | 14:09 |
| shardy | I can't figure out a way to have both the deploy pxe boot part, and the local disk-image boot part work | 14:09 |
| lucasagomes | shardy, interesting, I think what fixes the local boot is that it will create an EFI partition | 14:10 |
| lucasagomes | shardy, lemme look at the code to see if I can spot something about the PXE boot | 14:10 |
| shardy | lucasagomes: aha | 14:10 |
| lucasagomes | shardy, does it show any error? | 14:10 |
| shardy | lucasagomes: no, it just drops through the pxe boot stage as if the server isn't responding, then boots from the local disk | 14:11 |
| shardy | lucasagomes: if I remove boot_mode:uefi then pxe boot & the deploy ramdisk work fine again | 14:12 |
| lucasagomes | shardy, oh, so there's an option called "uefi_pxe_bootfile_name" and "uefi_pxe_config_template" | 14:12 |
| shardy | aha! | 14:12 |
| lucasagomes | shardy, you may want to set it to the same values as the pxe_bootfile_name and pxe_config_template | 14:12 |
| lucasagomes | since you are using that compatibility mode | 14:12 |
| lucasagomes | by default it iwll try to load the uefi pxe template and the efi ROM | 14:12 |
| lucasagomes | which may not be present in ur system | 14:12 |
| lucasagomes | the options are under the [pxe] section | 14:13 |
| *** links has joined #tripleo | 14:14 | |
| shardy | lucasagomes: I see, got it, changed those and re-testing, thanks! | 14:15 |
| lucasagomes | cool, let's see how it goes | 14:15 |
| lucasagomes | I've never tried this UEFI pxe compat thing, so it's new for me | 14:16 |
| *** tzumainn has joined #tripleo | 14:16 | |
| *** Goneri has quit IRC | 14:17 | |
| EmilienM | can we +A https://review.openstack.org/#/c/263366/ please ? reviewed lot of times, CI passing, etc... | 14:18 |
| trown | EmilienM: CI is not passing on that | 14:19 |
| *** lblanchard has joined #tripleo | 14:19 | |
| EmilienM | trown: HA job is failing | 14:19 |
| EmilienM | but if you look in the CI history, my patch used to pass | 14:19 |
| trown | EmilienM: ya, HA job is voting | 14:19 |
| EmilienM | and it's not related | 14:19 |
| EmilienM | anyway | 14:19 |
| shardy | lucasagomes: cool - it's ipxe loading the deploy ramdisk now :) | 14:20 |
| shardy | thanks! | 14:20 |
| lucasagomes | shardy, w00t, let's see how the deployment goes | 14:20 |
| hewbrocca | indeed, could we land that? | 14:21 |
| dtantsur | UEFI needs a lot of testing.. | 14:21 |
| lucasagomes | dtantsur, yeah... it's a diff case here too cause shardy has some PXE compat layer for UEFI | 14:21 |
| lucasagomes | so it doesn't need to use the EFI ROM for pxe booting it | 14:22 |
| dtantsur | meh, so many combinations | 14:22 |
| lucasagomes | we need to come up with some good documentaiton for such things | 14:22 |
| lucasagomes | dtantsur, exactly, we have so many permutations that it makes it hard to write about it concisely | 14:22 |
| shardy | Yeah, this may be something of a special case, because it's a fairly new desktop machine | 14:23 |
| *** tiswanso has joined #tripleo | 14:23 | |
| lucasagomes | shardy, yeah, I'm happy we can support it with what we have already | 14:23 |
| lucasagomes | but would be good to make it more usable if we have better instrunctions about how to set it up | 14:23 |
| shardy | (tarox box with an asus z170m motherboard if anyone else has same/similar) | 14:23 |
| *** tiswanso has quit IRC | 14:24 | |
| *** tiswanso has joined #tripleo | 14:24 | |
| *** dmsimard has quit IRC | 14:25 | |
| lucasagomes | shardy, cool, I added to my todo list to try to come up with some sane documentation for UEFI | 14:25 |
| *** tosky has quit IRC | 14:25 | |
| trown | jdob: we are just merging things without even all CI passing now? | 14:26 |
| * trown gives up | 14:26 | |
| shardy | lucasagomes: sounds good - I'll probably document what I have from a TripleO perspective too, even if it's just a blog post | 14:26 |
| jdob | trown: that was a special request from EmilienM since the failure wasn't related to his change | 14:26 |
| lucasagomes | shardy, nice, thanks for that! | 14:26 |
| jdob | that was also like three seconds ago, you're on top of shit :) | 14:26 |
| lucasagomes | shardy, did the whole deployment went through now? | 14:27 |
| trown | jdob: ya I was on that review :p | 14:27 |
| *** tosky has joined #tripleo | 14:27 | |
| shardy | lucasagomes: still deploying the image | 14:27 |
| * shardy goes to make tea | 14:27 | |
| lucasagomes | ack | 14:27 |
| trown | it is hard enough unfubaring trunk without us merging stuff that is not even passing our mocked up world CI | 14:28 |
| jdob | ok, i have both EmilienM and trown telling me "i give up"; you two sort this shit out | 14:29 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Ceph settings https://review.openstack.org/287827 | 14:29 |
| EmilienM | i'm fixing bugs, my patch pass CI, got never merged, I spent time to rebase them, not pass CI all the time. | 14:29 |
| EmilienM | and now it's not approved | 14:29 |
| EmilienM | so yeah, take my patch if you want or I can abandon it | 14:30 |
| *** links has quit IRC | 14:30 | |
| jdob | i removed the +A; again, you two sort it out | 14:31 |
| *** rbrady has quit IRC | 14:31 | |
| *** rbrady has joined #tripleo | 14:31 | |
| slagle | bnemec: shardy : can i get a backport +a? https://review.openstack.org/#/c/287745/ | 14:35 |
| slagle | or jdob , you seem to be in the review mood ^ | 14:36 |
| trown | EmilienM: you cant complain about trunk delorean not moving forward and want to merge stuff that did not pass CI | 14:36 |
| trown | EmilienM: those are mutually exclusive views of the world | 14:36 |
| EmilienM | trown: did I complain about trunk delorean? | 14:37 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 14:37 |
| trown | EmilienM: well you have moved puppet off of the current-passed-ci link... | 14:37 |
| trown | EmilienM: I assume because that is not updating frequently enough | 14:38 |
| EmilienM | trown: how is it a problem for you, I'm ahead of current-passed-ci so I can see issues earlier and fix in Puppet when needed | 14:38 |
| *** absubram has joined #tripleo | 14:38 | |
| EmilienM | it's actually a good thing for you that I'm ahead | 14:38 |
| EmilienM | we're gating things earlier so fix faster and reduce your workload on fixing upstream issues | 14:38 |
| jrist | lol gfidente nice owl | 14:38 |
| trown | ok... lets just merge everything... let downstream sort it out | 14:39 |
| EmilienM | and increase the chance of successful weirdo jobs for puppet openstack | 14:39 |
| EmilienM | trown: I think we misunderstand, I never asked for that. I don't understand your concern about "Puppet OpenStack CI not using current-passed-ci" | 14:39 |
| EmilienM | trown: what is the problem with that? | 14:40 |
| *** trozet has joined #tripleo | 14:40 | |
| trown | EmilienM: as long as it is not a response to the RDO promote job always failing, I have no problem | 14:41 |
| trown | EmilienM: I took it as a reaction to that, which I think is a mistake to react that way | 14:41 |
| EmilienM | you misunderstood I think | 14:41 |
| trown | fair enough | 14:42 |
| shardy | lucasagomes: so, it definitely got further, it appears to have written the image, unmounted the filesystem, but then got stuck before signalling ironic and halting the machine | 14:42 |
| shardy | the node is stuck in "deploying", and the deploy ramdisk appears to have hung | 14:42 |
| EmilienM | trown: but anyway: again, puppet openstack CI is using an RDO URL ahead of current-passed-ci because we don't want to wait you to promote and find bugs we need to fix in puppet modules - we want to go ahead and make sure "when you promote, your puppet jobs will pass" - does it make sense? | 14:43 |
| * shardy may have to look for a legacy boot mode option ;) | 14:43 | |
| trown | EmilienM: sure. as much as any of our spaghetti of CI everywhere makes sense. | 14:44 |
| jayg | if anyone has time, have a simple patch that could use a review - https://review.openstack.org/#/c/284775/ | 14:44 |
| EmilienM | trown: weirdo makes sense to me, fwiw | 14:44 |
| lucasagomes | shardy, hmm... and nothing in the logs? Because Ironic should be instrucing the ramdisk to power off now | 14:45 |
| trown | EmilienM: weirdo just runs your jobs :p | 14:45 |
| trown | in a slightly different context | 14:45 |
| trown | it makes the most sense of anything to me too fwiw | 14:45 |
| shardy | lucasagomes: ah, it's not hung, it's stuck in a loop, where it keeps mounting, then unmounting sda3 every few minutes | 14:47 |
| shardy | I see corresponding activity in the conductor log, sec | 14:47 |
| lucasagomes | shardy, lemme try to look at the code | 14:48 |
| lucasagomes | I wonder if it's the local boot code that may be broken | 14:49 |
| *** pradk_ has joined #tripleo | 14:53 | |
| *** pradk has quit IRC | 14:53 | |
| *** pradk_ is now known as pradk | 14:53 | |
| *** dustins has joined #tripleo | 14:55 | |
| shardy | lucasagomes: aha, there is an error attempting to install the bootloader: | 14:56 |
| shardy | http://paste.openstack.org/show/489160/ | 14:56 |
| *** Goneri has joined #tripleo | 14:56 | |
| shardy | It seems to just try again rather than fail after that | 14:57 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add Satellite 5 support https://review.openstack.org/287745 | 14:57 |
| lucasagomes | shardy, oh right, yeah I can see grub is ocmplaining about installing it | 14:57 |
| lucasagomes | shardy, I will try to investigate that erorr when I get some time | 14:58 |
| lucasagomes | shardy, for now you may want to disable local boot for the tests, it will still be fine I guess by booting it from network all the time | 14:58 |
| lucasagomes | at least you can continue the tests on that machines until we get the grub problem sorted | 14:58 |
| shardy | lucasagomes: ack, I'll give that a try, thanks for the help! | 14:59 |
| lucasagomes | shardy, np! | 15:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix endpoint names https://review.openstack.org/275442 | 15:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Generate the endpoint map statically https://review.openstack.org/275448 | 15:01 |
| *** bvandenh has joined #tripleo | 15:01 | |
| *** liverpooler has quit IRC | 15:02 | |
| dprince | Just a heads up there are going to be a few intermittent CI failures from the last 2 hours | 15:04 |
| dprince | anything that ran on testenv9 has failed... and it was my fault | 15:04 |
| dprince | fixing it now... | 15:04 |
| *** rcernin has joined #tripleo | 15:04 | |
| slagle | bnemec: since you merged the add swap patch...if you could also add this to your review queue...https://review.openstack.org/#/c/241408/ | 15:06 |
| *** fgimenez has quit IRC | 15:06 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Enable the ML2 port security extension driver by default https://review.openstack.org/287866 | 15:08 |
| *** fgimenez has joined #tripleo | 15:08 | |
| *** fgimenez has quit IRC | 15:08 | |
| *** fgimenez has joined #tripleo | 15:08 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use set -e for validation-scripts/all-nodes.sh https://review.openstack.org/285857 | 15:12 |
| *** hjensas has joined #tripleo | 15:13 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 15:14 |
| trown | jdob: EmilienM and I worked it out... feel free to put the +A back on https://review.openstack.org/#/c/263366/ | 15:16 |
| jdob | kk | 15:16 |
| jdob | thanks for the heads up | 15:16 |
| trown | I would like folks to be extra careful with the HA job, since that is precisely what I am trying to fix on trunk | 15:16 |
| EmilienM | trown: do you think they should be voting? | 15:17 |
| EmilienM | (voting like, blocking Gerrit merge for real) | 15:17 |
| *** bvandenh has quit IRC | 15:17 | |
| trown | EmilienM: as in on the gate queue? | 15:17 |
| trown | yes... that would be ideal | 15:17 |
| trown | derek has tried to explain to me why that creates problems for tripleo, but I do not fully grok the argument | 15:18 |
| *** hjensas has quit IRC | 15:18 | |
| rhallisey | trown, are you have heat stack-delete success today? | 15:27 |
| trown | rhallisey: nope... I did just manage to get a trace in heat-engine.log... is this what you are seeing http://chunk.io/f/9f5f5ce02cc3461b9f949c8e1d9415ba | 15:29 |
| trown | rhallisey: odd thing is that the ping test also fails to delete the overcloud, but I would assume it is for a different reason... havent got a deployed overcloud yet today to find out | 15:29 |
| hewbrocca | EmilienM, trown you guys got an arrangement worked out? | 15:30 |
| trown | hewbrocca: that makes it sound like we are getting married | 15:30 |
| rhallisey | trown, I did notice that it was able to delete pretty much everything except the controller and compute stacks | 15:30 |
| trown | which my wife would not be pleased about | 15:31 |
| hewbrocca | Maybe we could merge some more of EmilienM patches while we're at it :) | 15:31 |
| EmilienM | trown: lol | 15:31 |
| *** hjensas has joined #tripleo | 15:34 | |
| trown | rhallisey: any chance you want to file a heat bug for the stack-delete issue? I will get to it later today if not | 15:36 |
| rhallisey | sure | 15:36 |
| rhallisey | I like your logs I may steal them | 15:36 |
| trown | for sure | 15:37 |
| trown | I just learned about chunk.io this week | 15:37 |
| trown | pretty neat service | 15:37 |
| slagle | neat, that looks even easier to use than clbin | 15:40 |
| trown | credit to larsks for the suggestion | 15:41 |
| dtantsur | folks, please review a tiny liberty fix, found during internal testing: https://review.openstack.org/#/c/287734/ | 15:42 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrade of Cinder block storage nodes https://review.openstack.org/287929 | 15:48 |
| d0ugal | Anyone around that understands the interaction between t-h-t and puppet-swift? | 15:49 |
| trown | d0ugal: anything in particular? | 15:50 |
| *** jcoufal has quit IRC | 15:51 | |
| d0ugal | trown: Yes! If you deploy with two swift nodes, everything is fine and they are in the same swift ring | 15:51 |
| d0ugal | trown: but then if you add another node it isn't added into the swift ring | 15:52 |
| d0ugal | it seems to get it's own, new swift ring as far as I can tell | 15:52 |
| d0ugal | so it seems like something simple is missing but I am finding it hard to know where I should look | 15:52 |
| trown | d0ugal: hmm seems like we need different behavior for a stack update than a stack create wrt swift nodes | 15:55 |
| trown | d0ugal: I think https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/ringbuilder.pp is what drives creating the rings... but I dont think there is any logic for adding a new node to the existing rings | 15:57 |
| *** aufi has quit IRC | 15:57 | |
| hewbrocca | there wouldn't be, no | 15:57 |
| hewbrocca | it's hard | 15:57 |
| hewbrocca | you actually need something like a resource agent | 15:57 |
| d0ugal | slagle: ^ | 15:57 |
| *** yamahata has joined #tripleo | 15:57 | |
| trown | which then makes scaling a bit of a non-starter | 15:57 |
| hewbrocca | jistr: | 15:58 |
| hewbrocca | ^^^ | 15:58 |
| d0ugal | Fun. | 15:58 |
| *** jprovazn has quit IRC | 16:01 | |
| pradk | Could i please request some more reviews so we can get this in - https://review.openstack.org/#/c/241408/ | 16:05 |
| pradk | trown, chunk.io looks pretty neat, nice find | 16:06 |
| *** rcernin has quit IRC | 16:06 | |
| *** jistr is now known as jistr|call | 16:08 | |
| *** dmsimard has joined #tripleo | 16:09 | |
| trown | pradk: ya, not my find, but pretty neat indeed | 16:09 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests https://review.openstack.org/244162 | 16:11 |
| shardy | Be good if we could land this os-cloud-config change: https://review.openstack.org/#/c/273710/ | 16:13 |
| trown | could someone who understands how the puppet managed keystone init is supposed to work look at that THT patch ^... the pacemaker case is still not working | 16:15 |
| openstackgerrit | Merged openstack/python-tripleoclient: Fix bad backport for 'openstack overcloud profiles' commands https://review.openstack.org/287734 | 16:17 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common: Adds override for the overcloud node user in upgrade-non-controller https://review.openstack.org/287953 | 16:19 |
| openstackgerrit | Merged openstack/os-cloud-config: Switch keystone certs to parameter_defaults https://review.openstack.org/273710 | 16:20 |
| *** fgimenez has quit IRC | 16:22 | |
| *** jcoufal has joined #tripleo | 16:22 | |
| *** chem has joined #tripleo | 16:22 | |
| *** fgimenez has joined #tripleo | 16:24 | |
| *** fgimenez has quit IRC | 16:24 | |
| *** fgimenez has joined #tripleo | 16:24 | |
| *** yamahata has quit IRC | 16:26 | |
| *** jistr|call is now known as jistr | 16:26 | |
| *** hewbrocca has quit IRC | 16:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements https://review.openstack.org/285049 | 16:28 |
| *** xinwu has joined #tripleo | 16:29 | |
| jistr | d0ugal: yea, from what you post it looks like scaling swift might have never been supported | 16:30 |
| d0ugal | jistr: Fun :) | 16:31 |
| *** yamahata has joined #tripleo | 16:32 | |
| d0ugal | jistr: in theory, what would be required? | 16:32 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Moves the swift start/stop into the common_functions.sh file https://review.openstack.org/287960 | 16:34 |
| jaosorior | bnemec: online yet? | 16:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Ceph settings https://review.openstack.org/287827 | 16:35 |
| openstackgerrit | Merged openstack/python-tripleoclient: Refresh node power state at start of loop https://review.openstack.org/283809 | 16:37 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287961 | 16:37 |
| jistr | d0ugal: proper ring-amending logic, i didn't look into this at all but it might not be very easy to achieve unless someone has already implemented something that we can just plug in (even then it would need integration work, testing incl. upgrades). IIRC one cannot just recreate new rings from scratch, as that could result in a reshuffling of what's stored where, which is very undesirable. Not sure if it would mean data loss too. This is just | 16:39 |
| jistr | my non-swift-expert view of things so it's better to check with someone else i think, perhaps someone from swift team directly. | 16:39 |
| jistr | i saw a doc about it somewhere | 16:40 |
| *** olap has quit IRC | 16:40 | |
| *** devvesa has quit IRC | 16:40 | |
| *** yamahata has quit IRC | 16:41 | |
| jistr | i recall reading this, but it's more theory about the rings themselves rather than practical manipulation http://docs.openstack.org/developer/swift/overview_ring.html | 16:41 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 16:42 |
| *** jprovazn has joined #tripleo | 16:45 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Fix password issue with mysql address for ceilometer https://review.openstack.org/287970 | 16:45 |
| d0ugal | jistr: ugh, okay. I think I am convinced that this is way over my head anyway | 16:46 |
| d0ugal | jistr: Thanks for the input. | 16:46 |
| openstackgerrit | Jason Guiditta proposed openstack/puppet-tripleo: loadbalancer: fix Redis timeout HAproxy config https://review.openstack.org/287974 | 16:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove DNS hack. https://review.openstack.org/284979 | 16:47 |
| *** mikelk has quit IRC | 16:53 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Fix password issue with mysql address for ceilometer https://review.openstack.org/287970 | 16:53 |
| *** dmsimard has quit IRC | 16:57 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 16:58 |
| *** yamahata has joined #tripleo | 17:00 | |
| *** dtantsur is now known as dtantsur|afk | 17:01 | |
| *** rdopiera has quit IRC | 17:02 | |
| *** jaosorior has quit IRC | 17:04 | |
| *** mkovacik has joined #tripleo | 17:07 | |
| *** dmsimard has joined #tripleo | 17:08 | |
| trown | dprince: any chance you could help me figure out what is going wrong with the HA part of https://review.openstack.org/#/c/244162 | 17:10 |
| trown | dprince: seems like the puppet keystone bootstrapping is not working | 17:10 |
| trown | http://chunk.io/f/c48f9cd3bdb846be98ebea6aa88fdacd and http://chunk.io/f/5ca73902a4a04e15b0a49edad98a7876 are evidence of that | 17:11 |
| dprince | trown: yes, but give me a few. I'm building new testenv's ATM for the network ISO job... | 17:12 |
| trown | dprince: ah right, forgot about the other fire, no problem | 17:13 |
| jistr | d0ugal: been scanning through code a bit more, it looks like we generate the rings on each node separately, and count on the operation being deterministic. There seems to be some support in puppet-swift to manipulate the rings and we use it, but i don't know if we can count on ring *updates* being deterministic too. Perhaps a more usual way would be to build the rings ahead of time in a single place and feed the same ring files to all nodes | 17:16 |
| jistr | as-is. This might get easier with splitting HW vs. SW stack. Deploy HW -> generate/amend rings from HW stack info -> deploy SW (e.g. fencing config would benefit from split stack exactly the same way). I'd probably have to spend a lot more time on this to be of any real help though, i haven't played with Swift in TripleO at all yet. | 17:16 |
| *** Marga_ has quit IRC | 17:16 | |
| *** mbound has quit IRC | 17:16 | |
| *** Marga_ has joined #tripleo | 17:17 | |
| d0ugal | jistr: heh, I just started to learn about it this week. You have been very helpful, if nothing else we can better evaluate how much work is required. | 17:19 |
| *** fgimenez has quit IRC | 17:21 | |
| gfidente | jistr, d0ugal I was thinking to replace swift with ceph rgw entirely ... | 17:22 |
| gfidente | but hey swift is probably one of the most successful openstack core projects | 17:23 |
| *** jaosorior has joined #tripleo | 17:26 | |
| *** jistr has quit IRC | 17:33 | |
| openstackgerrit | Javier Peña proposed openstack/tripleo-heat-templates: Do not include port in Glance rabbit_hosts https://review.openstack.org/288001 | 17:33 |
| bnemec | jaosorior: Yeah, I'm trying to get a local environment set up where I can test SSL. So far I'm failing miserably. | 17:48 |
| *** shivrao has joined #tripleo | 17:49 | |
| jaosorior | bnemec: I've been failing too. Getting a very strange issue where, if I deploy using network isolation, then if I try to delete the stack it stalls | 17:50 |
| jaosorior | I can however delete a stack that wasn't deployed with net-iso | 17:50 |
| jaosorior | so yeah, testing SSL has gotten a bit difficult | 17:50 |
| bnemec | jaosorior: Yeah, trown mentioned that yesterday, and I'm pretty sure I'm seeing the same thing. :-( | 17:50 |
| jaosorior | it really blocked a lot of stuff for me | 17:50 |
| jaosorior | and I see nothing in the novs logs | 17:50 |
| jaosorior | *nova | 17:51 |
| bnemec | Although I would have to get back to a point where I can deploy a stack to know for sure. | 17:51 |
| bnemec | I think it's probably a Heat problem. | 17:51 |
| bnemec | In my case I don't think it was even getting to where it would talk to Nova. | 17:51 |
| d0ugal | gfidente: Yeah, I think lots of people want to use it | 17:51 |
| jaosorior | now I'm trying to deploy with stable/liberty | 17:51 |
| jaosorior | lets see if that helps | 17:51 |
| gfidente | d0ugal, it's actually easier to manage on our side I think | 17:52 |
| gfidente | and if we really want to use ceph by default for cinder a/a | 17:52 |
| jaosorior | bnemec: If you get some time, could you take a quick look at this CR? https://review.openstack.org/#/c/287961/ I've been trying to test that too, and it fails in the CI with problems with the database connection... which is damn strange, since I don't touch those ports :/ | 17:52 |
| gfidente | then one could scale storage for all glance/swift/cinder by just scaling ceph | 17:53 |
| gfidente | but there are downsides too, like the api isn't really on a 1.1 feature parity | 17:53 |
| gfidente | tty tomorrow guys! | 17:53 |
| jaosorior | gfidente: Have a good one dude | 17:54 |
| gfidente | jaosorior :) | 17:54 |
| *** ccamacho has quit IRC | 17:54 | |
| *** dmsimard has quit IRC | 17:56 | |
| trown | rhallisey: I have more logs to add to the heat stack-delete bug | 17:57 |
| trown | http://chunk.io/f/9d1436dca1c64533a52087c9b5f81a45 | 17:57 |
| trown | that is the simple overcloud ping test stack failing to delete | 17:58 |
| rhallisey | k I'll add them | 17:58 |
| rhallisey | here's the bug for reference: https://bugs.launchpad.net/heat/+bug/1552760 | 17:59 |
| openstack | Launchpad bug 1552760 in heat "heat stack-delete is fails to delete anything then times out" [Undecided,New] | 17:59 |
| trown | rhallisey: thanks... I do not have the delete_in_progress for a few hours symptom... I get to delete_failed pretty quick | 18:00 |
| rhallisey | oh really | 18:00 |
| rhallisey | my just hangs | 18:00 |
| rhallisey | add your comments because we are doing slightly different things | 18:00 |
| trown | do you have some automation around heat stack-delete? is it possible it is failing because of the new confirmation in heat client? | 18:01 |
| trown | will do | 18:01 |
| *** masco has joined #tripleo | 18:02 | |
| jaosorior | rhallisey: I see the same fwiw | 18:03 |
| jaosorior | it just stalls | 18:03 |
| rhallisey | jaosorior, ya it's lame :/ | 18:03 |
| jaosorior | and stays in DELETE_IN_PROGRESS | 18:03 |
| bnemec | jaosorior: I'm seeing this in the hieradata from the failed SSL CI run: controller.yaml:neutron::server::notifications::auth_url: ://:/v3 | 18:04 |
| bnemec | I'm kind of amazed it isn't failing hard than it is. | 18:04 |
| bnemec | *harder | 18:04 |
| *** hjensas has quit IRC | 18:05 | |
| *** ohamada has quit IRC | 18:05 | |
| jaosorior | bnemec: What the hell | 18:05 |
| bnemec | I'm so tired of this notification auth BS. | 18:06 |
| jaosorior | bnemec: Well. I do remember there were some changes recently related to the nova-neutron configuration... just don't remember exactly which | 18:06 |
| jaosorior | bnemec: pretty big headache | 18:06 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/268528 | 18:06 |
| jaosorior | bnemec: So it seems somewhere it's not being set up... or some url is messed up (might be a typo somewhere... or something of the sort) | 18:07 |
| bnemec | jaosorior: Oh, I know what the problem is. | 18:08 |
| jaosorior | bnemec???? | 18:08 |
| openstackgerrit | Gonéri Le Bouder proposed openstack/instack-undercloud: add INTERFACE_MTU parameter https://review.openstack.org/288041 | 18:08 |
| bnemec | FFS, I even thought this would probably fail the SSL job, but I forgot about it from last night to today. | 18:08 |
| bnemec | jaosorior: https://review.openstack.org/#/c/287465/ | 18:08 |
| bnemec | The KeystoneV3 endpoint is missing from the current enable-tls.yaml. | 18:09 |
| bnemec | And it's required now. | 18:09 |
| *** lucasagomes is now known as lucas-afk | 18:09 | |
| *** penick has joined #tripleo | 18:09 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 18:10 |
| jaosorior | well, that makes sense | 18:10 |
| bnemec | I added a depends-on to the ci patch. That will probably get it working. | 18:11 |
| bnemec | Or at least get us past this problem to the next one. :-) | 18:11 |
| jaosorior | bnemec: is stable/liberty switching to keystone v3? or is that only in master? | 18:11 |
| bnemec | jaosorior: Not sure. There have been so many keystone-related changes lately that I can't keep track of what happened where. :-/ | 18:12 |
| jaosorior | ssl used to kinda work... but if the keystone changes went in to stable/liberty, then we need to backport your commit too | 18:12 |
| jaosorior | EmilienM do you know if stable/liberty is also switching to keystonev3? | 18:13 |
| EmilienM | jaosorior: what do you mean by switching to keystone v3 ? v3 API is exposed since Juno I think | 18:14 |
| bnemec | Actually, it should be pretty easy to figure out if this went into stable. | 18:14 |
| jaosorior | EmilienM: even though the API has been exposed, the services where hardcoded to use v2.0 | 18:14 |
| EmilienM | right | 18:15 |
| jaosorior | EmilienM: No worries, just asked in case you had that info from the top of your head. We can figure it out | 18:15 |
| jaosorior | thanks dude | 18:15 |
| bnemec | jaosorior: No keystone v3 endpoints in liberty yet, so we should be okay there. | 18:15 |
| jaosorior | bnemec: alright | 18:16 |
| jaosorior | bnemec: you had tried to backport the overcloud ssl job to stable/liberty, right? | 18:16 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 18:16 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: TEST: Run the ssl ci change against stable https://review.openstack.org/287425 | 18:16 |
| bnemec | jaosorior: ^ | 18:16 |
| bnemec | I didn't retry it again last night because I kept fscking up the rebase of the ci change and didn't want to waste any more CI time on it. :-) | 18:17 |
| *** trown is now known as trown|lunch | 18:17 | |
| * jaosorior is in suspense | 18:18 | |
| *** Marga_ has quit IRC | 18:19 | |
| * bnemec is dropping f-bombs and it's barely noon | 18:19 | |
| bnemec | This bodes well for the rest of the day. ;-) | 18:20 |
| jaosorior | hahaha know the feel | 18:21 |
| bnemec | Fortunately I have an appointment this afternoon so there will be an enforced break from broken OpenStack services. | 18:22 |
| jaosorior | on my side I'm not so lucky. It's 8:23 pm and I'm still here O_O | 18:23 |
| bnemec | Ugh. Go home. Or at least step away from the laptop if you're already home. :-) | 18:24 |
| jaosorior | bnemec: yeah, that's the downside of working from home | 18:24 |
| jaosorior | and I made the mistake of putting my working spot in my bedroom | 18:24 |
| ayoung | EmilienM, so, yeah, no HTTPD in this Keystone, still eventlet, even with the Heat changes. Both underclouid and overcloud | 18:25 |
| bnemec | slagle: Left a review on https://review.openstack.org/#/c/241408 It's going to need a change to work with SSL. We can merge it as-is, but since I assume it needs backporting it's probably easier to do it as one patch. | 18:27 |
| jaosorior | bnemec: I guess that also needs changes in puppet-tripleo | 18:27 |
| openstackgerrit | James Slagle proposed openstack/instack-undercloud: Nova should not sync power state of overcloud nodes https://review.openstack.org/288052 | 18:28 |
| bnemec | jaosorior: I assumed that was already done or it wouldn't work at all. | 18:28 |
| jaosorior | and maybe os-cloud-config? | 18:28 |
| jaosorior | bnemec: Let me double check | 18:28 |
| bnemec | jaosorior: puppet-tripleo is actually in the depends-on: https://review.openstack.org/#/c/241395/ | 18:29 |
| bnemec | So that's done anyway. | 18:29 |
| jaosorior | bnemec: True, found it | 18:29 |
| jaosorior | bnemec: I was looking at stable/liberty initially, that's why I thought it still needed to be done | 18:30 |
| jaosorior | anyway, I gotta go | 18:31 |
| bnemec | jaosorior: Have a good night. | 18:31 |
| jaosorior | bnemec: If at some point in the day you can take a look at https://review.openstack.org/#/c/287199/ to see if it makes sense or I messed up, would really appreciate it | 18:31 |
| jaosorior | have a good one guys! | 18:31 |
| slagle | bnemec: k, thanks. yea, it has to work with ssl | 18:31 |
| bnemec | Oh, pradk, you should also see my comment on https://review.openstack.org/#/c/241408 | 18:31 |
| *** jaosorior has quit IRC | 18:31 | |
| pradk | bnemec, sure, i'll address that now, thx for the review | 18:32 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 18:35 |
| pradk | bnemec, done, thx again | 18:35 |
| *** loser_ is now known as panda | 18:36 | |
| *** gmmaha has joined #tripleo | 18:36 | |
| bnemec | pradk: slagle: +2 | 18:36 |
| ayoung | shardy, if the heat template says to deploy the undercloud Keystone in Apache, why would it still be in Eventlet? | 18:36 |
| shardy | ayoung: because the undercloud isn't deployed via heat? | 18:37 |
| gmmaha | Hi, i have been using bifrost/ironic standalone to deploy some machines and i was wondering if there was a dib element to modify the grub cmdline parameters on the image.. | 18:37 |
| gmmaha | tyring to enable net.ifnames=1 in the ubuntu image thats being deployed on machines | 18:37 |
| ayoung | shardy, ah...good point, I meant overcloud | 18:37 |
| ayoung | shardy, I'm debugging both in this deploy | 18:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Remove unsafe "unset" defaults https://review.openstack.org/288057 | 18:38 |
| ayoung | EmilienM, one big clue on the overcloud: it seems to have failed after Keystone. the other services are running, but not entered into the service catalog; may be related | 18:38 |
| shardy | ayoung: aha :) | 18:38 |
| ayoung | shardy, according to your debugging steps... | 18:39 |
| ayoung | | d7e339be-6d09-4bad-9007-33e62c560734 | overcloud-ControllerNodesPostDeployment-oyoxyg3vuak6-ControllerOvercloudServicesDeployment_Step5-fltpqc6ebthr | CREATE_FAILED | 2016-03-02T16:07:10 | None | ec28bbc2-91f2-4dbc-8c51-6b0cfe143c53 | | 18:39 |
| ayoung | shardy, and then | 18:40 |
| ayoung | | 0 | 82621f7d-e775-4203-9104-44b5f38d6042 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-03-02T16:07:10 | overcloud-ControllerNodesPostDeployment-oyoxyg3vuak6-ControllerOvercloudServicesDeployment_Step5-fltpqc6ebthr | | 18:40 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: updating enable_ceph conditions for controller https://review.openstack.org/288060 | 18:41 |
| *** masco has quit IRC | 18:42 | |
| ayoung | Could not evaluate: Cannot allocate memory | 18:44 |
| ayoung | shardy, so, not related | 18:44 |
| *** trown|lunch has quit IRC | 18:44 | |
| *** rcernin has joined #tripleo | 18:44 | |
| *** rwsu has quit IRC | 18:45 | |
| *** mgould has quit IRC | 18:45 | |
| ayoung | shardy, yeah...thought I had upped the memory to 8092. 4096 is not enough anymore for a control node | 18:46 |
| shardy | ayoung: Yeah, unfortunately several things have been added which increased memory requirements somewhat | 18:48 |
| shardy | ayoung: another option is https://review.openstack.org/#/c/273752/ which adds some swap | 18:48 |
| shardy | maybe useful if you're only just running out of actual memory | 18:48 |
| ayoung | shardy, Its ok, I think I can get away with the large nodes... | 18:48 |
| *** akrivoka has quit IRC | 18:50 | |
| ayoung | shardy, will export NODE_MEM=6144 work? Can we do none powers of 2> | 18:50 |
| ayoung | ? | 18:50 |
| ayoung | well..I guess we'll find out | 18:51 |
| shardy | ayoung: yup should do | 18:51 |
| shardy | IIRC the undercloud defaults to 6G | 18:51 |
| ayoung | I set it to 8 | 18:52 |
| shardy | Yeah, good plan, 6 won't be enough without swap | 18:52 |
| *** leanderthal is now known as leanderthal|afk | 18:53 | |
| *** trown has joined #tripleo | 18:55 | |
| ayoung | EmilienM, so...the failure was due to memory size. I've just rebuilt the VMs, and about to redeploy the undercloud using tripleo.sh | 18:55 |
| ayoung | anything I should try to capture? | 18:56 |
| *** shardy has quit IRC | 18:56 | |
| EmilienM | ayoung: good to know, I was not aware of that | 18:56 |
| *** jcoufal has quit IRC | 18:58 | |
| *** athomas has quit IRC | 18:59 | |
| *** Marga_ has joined #tripleo | 19:01 | |
| *** dshulyak has left #tripleo | 19:01 | |
| *** rwsu has joined #tripleo | 19:03 | |
| *** Marga_ has quit IRC | 19:03 | |
| *** tosky has quit IRC | 19:05 | |
| *** Marga_ has joined #tripleo | 19:06 | |
| ayoung | EmilienM, Notice: /Stage[main]/Keystone::Wsgi::Apache/Apache::Vhost[keystone_wsgi_main]/Concat[10-keystone_wsgi_main.conf]/Exec[concat_10-keystone_wsgi_main.conf]/returns: executed successfully | 19:08 |
| ayoung | that looks good, right? | 19:08 |
| EmilienM | ayoung: yes Sir! | 19:08 |
| ayoung | EmilienM, and yet | 19:08 |
| ayoung | [stack@instack ~]$ ps -ef | grep keystone | 19:08 |
| ayoung | keystone 22582 22580 9 19:08 ? 00:00:00 keystone-admin -DFOREGROUND | 19:08 |
| ayoung | keystone 22583 22580 9 19:08 ? 00:00:00 keystone-main -DFOREGROUND | 19:08 |
| ayoung | stack 22648 22557 0 19:08 pts/1 00:00:00 grep --color=auto keystone | 19:08 |
| ayoung | EmilienM, I bet it is the systemd config...let's look | 19:09 |
| ayoung | EmilienM, yep | 19:10 |
| ayoung | less /usr/lib/systemd/system/openstack-keystone.service | 19:10 |
| ayoung | ExecStart=/usr/bin/keystone-all | 19:10 |
| ayoung | I think that should maybe just get yanked, or chkconfiged off | 19:11 |
| ayoung | $ systemctl status openstack-keystone.service | 19:12 |
| ayoung | ● openstack-keystone.service - OpenStack Identity Service (code-named Keystone) | 19:12 |
| ayoung | Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; disabled; vendor preset: disabled) | 19:12 |
| ayoung | hmmmm | 19:12 |
| *** xinwu has quit IRC | 19:16 | |
| *** ifarkas has quit IRC | 19:17 | |
| ayoung | EmilienM, I'm the one that should be facepalming | 19:17 |
| ayoung | root 22580 1 0 19:08 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND | 19:17 |
| ayoung | keystone 22582 22580 0 19:08 ? 00:00:03 keystone-admin -DFOREGROUND | 19:17 |
| ayoung | Its working fine.... | 19:17 |
| * ayoung bone head | 19:17 | |
| ayoung | I am surprised that there is only one process per for each main and admin, but...that might be a config option to fix later | 19:18 |
| trown | ayoung: do you have any thoughts on how to further troubleshoot https://review.openstack.org/244162 | 19:19 |
| ayoung | trown, is that using keystone-manage bootstrap? | 19:21 |
| trown | or EmilienM ^, it is like the service tenant create part of kesytone::role::admin is not working | 19:21 |
| trown | ayoung: yep | 19:21 |
| ayoung | trown, well, let me see if ithat creates the "project" | 19:21 |
| EmilienM | trown: what says puppet logs? | 19:21 |
| EmilienM | can you connect to keystone api? | 19:21 |
| EmilienM | puppet use osclient to connect to keystone api and manage resources | 19:22 |
| trown | EmilienM: sort of... the service is running, but I cant auth to it | 19:22 |
| ayoung | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/cmd/cli.py#n83 | 19:22 |
| trown | EmilienM: as in I can curl it and get the top level version json, but /tokens not so much | 19:22 |
| ayoung | so...it looks like it only creates one project? Is the admin project created on your system trown ? | 19:23 |
| EmilienM | trown: /me currently dealing with CI issues in puppet | 19:23 |
| *** jtomasek_ has joined #tripleo | 19:23 | |
| trown | ayoung: ya this is what the keystone DB looks like after the stack goes to CREATE_COMPLETE http://chunk.io/f/5ca73902a4a04e15b0a49edad98a7876 | 19:23 |
| ayoung | trown, OK, two domains (one root) one project (admiun) | 19:24 |
| ayoung | one role assignment | 19:24 |
| ayoung | admin user has admin role on admin project | 19:25 |
| ayoung | trown, can you auth as Admin? | 19:25 |
| ayoung | trown, ah...and nothing in the endpoint table | 19:25 |
| ayoung | I think I saw a commit for this... | 19:25 |
| trown | ayoung: ya... | 19:26 |
| ayoung | https://review.openstack.org/#/c/285102/ | 19:26 |
| trown | ayoung: even more odd, is that this only happens for the pacemaker managed keystone... the nonha job does not hit this | 19:26 |
| ayoung | trown, maybe that is still using ADMIN_TOKEN | 19:28 |
| dprince | EmilienM: trown there was a bad testenv this morning. I was testing a new build of the testenv's (and I broke it) | 19:29 |
| rhallisey | trown, ramishra might have a solution for us fyi | 19:29 |
| rhallisey | check the bug | 19:29 |
| trown | ayoung: I guess it would have to be? but it doesnt seem like it: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp#L179 says just give me puppet defaults | 19:29 |
| dprince | EmilienM, trown: on that note, I'd like to deploy a new testenv now. Any issues if I just go on and do that? | 19:29 |
| trown | dprince: go for it | 19:29 |
| EmilienM | go! | 19:30 |
| trown | ayoung: and the puppet defaults are now to run the bootstrap | 19:30 |
| rhallisey | trown, work around would be to use old neutronclient | 19:30 |
| dprince | EmilienM, trown: okay, anything running would fail. We can recheck them later... | 19:30 |
| trown | oh wow... neutronclient is the problem? | 19:30 |
| rhallisey | trown, I haven't confirmed yet, but will test later | 19:30 |
| trown | rhallisey: thanks! | 19:30 |
| rhallisey | trown, https://github.com/openstack/python-neutronclient/commit/65118c09eb08e760590939e57d3b1a485c567f91 | 19:30 |
| ayoung | trown, on a non-HA node, you get a service catalog with an identity endpoint? | 19:31 |
| trown | ayoung: ya, I can fully validate the deployed overcloud with the exact packages/images for nonha | 19:31 |
| trown | ayoung: it is only HA that is not fully configuring keystone... though it still gets to CREATE_COMPLETE | 19:32 |
| ayoung | trown, include ::keystone::endpoint where does that come from? I am not fluent in puppet yet | 19:32 |
| trown | https://github.com/openstack/puppet-keystone/blob/master/manifests/endpoint.pp | 19:32 |
| *** electrofelix has quit IRC | 19:33 | |
| trown | ayoung: no worries, I have been staring at this for 2 days, so I know where all the tripleo/puppet bits come from | 19:33 |
| trown | ayoung: so that endpoint manifest is what is meant to create the identity endpoint | 19:34 |
| ayoung | trown, so, that injects the endpoint in, but I can't see how it authenticates | 19:34 |
| ayoung | user_domain => $user_domain, | 19:35 |
| ayoung | project_domain => $project_domain, seem like they are for a token | 19:35 |
| ayoung | but if SERVICE_TOKEN is set, or whatever the newish term is for that, I think they would get ignored | 19:35 |
| ayoung | keystone_admin_token => hiera('keystone::admin_token'), is specified https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp#L285 | 19:36 |
| trown | ayoung: hmm that is only in the midonet stuff though | 19:38 |
| trown | which is not run in either scenario for me | 19:38 |
| ayoung | trown, yeah, but I wonder if it "pollutes" the auth plugin...somehow? | 19:38 |
| ayoung | its the only thinkg I can see. It needs to be yanked anyway | 19:38 |
| trown | it would only run if this conditional is true https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp#L251 | 19:39 |
| ayoung | trown, so, if we are going to do bootstrap, we should disable admin_token both in the config file and in the paste pipeline | 19:41 |
| ayoung | that latter is tought because we bury it in a non-config location | 19:41 |
| trown | ya, and I think that is unrelated to the issue blocking delorean promotion | 19:41 |
| ayoung | but I think now if we set ADMIN_TOKEN= | 19:41 |
| ayoung | for None | 19:42 |
| trown | since that would be the same in the ha and nonha cases | 19:42 |
| ayoung | we can be assured that ADMIN_TOKEN is not being used, and cartch errors like this | 19:42 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Add capabilities filter for Nova https://review.openstack.org/288087 | 19:51 |
| *** dmacpher has quit IRC | 19:55 | |
| *** ayoung has quit IRC | 19:56 | |
| *** mkovacik has quit IRC | 19:58 | |
| *** jtomasek_ has quit IRC | 19:58 | |
| *** mkovacik has joined #tripleo | 20:04 | |
| *** shardy has joined #tripleo | 20:10 | |
| pradk | hmm any idea why ci is failing with "testenv-client - ERROR - The command hasn't completed but the testenv worker has released the environment. Killing all processes." | 20:14 |
| *** yamahata has quit IRC | 20:14 | |
| slagle | pradk: i think there is some rebuilding of the testenv's going on | 20:15 |
| pradk | ah ok | 20:16 |
| *** penick has quit IRC | 20:16 | |
| *** pradk has quit IRC | 20:20 | |
| *** rwsu has quit IRC | 20:25 | |
| *** rwsu has joined #tripleo | 20:25 | |
| *** dmsimard has joined #tripleo | 20:27 | |
| *** penick has joined #tripleo | 20:28 | |
| *** dmacpher has joined #tripleo | 20:32 | |
| *** ayoung has joined #tripleo | 20:33 | |
| *** dmacpher has quit IRC | 20:33 | |
| *** dmacpher has joined #tripleo | 20:36 | |
| openstackgerrit | Rajini Ram proposed openstack/tripleo-heat-templates: Updated the heat_template_version https://review.openstack.org/288116 | 20:37 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/241408 | 20:40 |
| *** pradk has joined #tripleo | 20:43 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-puppet-elements: Add Aodh component https://review.openstack.org/288119 | 20:48 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Remove Ceilometer Alarm from the overcloud https://review.openstack.org/288120 | 20:48 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: loadbalancer: add Aodh API support https://review.openstack.org/288122 | 20:53 |
| *** stevebak` is now known as stevebaker | 21:00 | |
| *** mbound has joined #tripleo | 21:01 | |
| *** lblanchard has quit IRC | 21:01 | |
| *** jayg is now known as jayg|g0n3 | 21:04 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Remove Ceilometer Alarm from the overcloud https://review.openstack.org/288120 | 21:07 |
| *** penick has quit IRC | 21:18 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/288130 | 21:21 |
| *** ayoung has quit IRC | 21:24 | |
| *** jprovazn has quit IRC | 21:25 | |
| openstackgerrit | Kurt Hey proposed openstack/tripleo-heat-templates: Updated the heat_template_version https://review.openstack.org/288134 | 21:27 |
| *** xinwu has joined #tripleo | 21:35 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/python-tripleoclient: Aodh integration to tripleoclient https://review.openstack.org/283857 | 21:37 |
| EmilienM | pradk: in theory, https://review.openstack.org/#/c/283857/ would be useless if we land https://review.openstack.org/244162 one day | 21:39 |
| pradk | EmilienM, ok is that expected to land in liberty? | 21:40 |
| *** gmmaha has left #tripleo | 21:40 | |
| EmilienM | pradk: I'm not sure | 21:40 |
| EmilienM | I wish :) | 21:40 |
| EmilienM | but IIRC there is a problem during upgrade | 21:40 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: Allow node import via yaml not only csv/json https://review.openstack.org/255228 | 21:42 |
| *** penick has joined #tripleo | 21:42 | |
| *** rcernin has quit IRC | 21:45 | |
| *** r-mibu has quit IRC | 21:46 | |
| *** r-mibu has joined #tripleo | 21:47 | |
| *** Goneri has quit IRC | 21:47 | |
| EmilienM | dprince: I noticed that running tripleo jobs in puppet module CI was good but not enough, sometimes some patches can break HA jobs. Do you think we can switch non-ha to ha? or is it too hardwae expensive? | 21:48 |
| dprince | EmilienM: ha job is slower | 21:49 |
| EmilienM | yeah but not that much | 21:49 |
| dprince | EmilienM: and non-ha is more stable | 21:49 |
| dprince | EmilienM: but sure, the HA job does give extra coverage | 21:50 |
| EmilienM | its 1h35 versus 2h | 21:50 |
| *** jtomasek_ has joined #tripleo | 21:50 | |
| EmilienM | mhh ok | 21:50 |
| EmilienM | let's keep like this | 21:50 |
| EmilienM | dprince: the keystone bootstrap stuff broke ha job last time | 21:50 |
| dprince | EmilienM: for now, or we can bring in up in next weeks tripleO IRC meeting | 21:50 |
| EmilienM | ok | 21:51 |
| dprince | EmilienM: I just rebuilt all the testenv's btw | 21:51 |
| EmilienM | nice! | 21:51 |
| dprince | EmilienM: jobs probably failed over the last 2 hours, that was me being disruptive | 21:51 |
| EmilienM | ok | 21:51 |
| EmilienM | can I run recheck? | 21:52 |
| dprince | EmilienM: anyway, hopefully now things work again | 21:52 |
| EmilienM | ok | 21:52 |
| dprince | EmilienM: before running recheck in mass maybe just do one and see if it is green | 21:52 |
| *** dshulyak has joined #tripleo | 21:52 | |
| EmilienM | ok | 21:52 |
| dprince | EmilienM: I'd like an open queue to test some things | 21:52 |
| *** penick has quit IRC | 21:53 | |
| dprince | EmilienM: if you see failing jobs try and send me the testenv number. Like testenv6 or something... | 21:56 |
| dprince | EmilienM: this puppet-cinder job is on a new worker https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-nonha/1048/ | 21:56 |
| EmilienM | dprince: ok I'll let you know | 21:57 |
| *** trozet has quit IRC | 22:01 | |
| *** gfidente has quit IRC | 22:02 | |
| *** shivrao has quit IRC | 22:06 | |
| *** dshulyak has quit IRC | 22:06 | |
| *** shivrao has joined #tripleo | 22:07 | |
| *** shardy has quit IRC | 22:10 | |
| *** jtomasek_ has quit IRC | 22:11 | |
| openstackgerrit | Richard Su proposed openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/287561 | 22:18 |
| *** jtomasek has quit IRC | 22:19 | |
| *** penick has joined #tripleo | 22:29 | |
| *** trown is now known as trown|outtypewww | 22:30 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-common: Expose TENANT_STACK_DEPLOY_ARGS https://review.openstack.org/288161 | 22:30 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: IPv4 network isolation testing for Ceph https://review.openstack.org/288163 | 22:34 |
| *** ayoung has joined #tripleo | 22:36 | |
| *** dprince has quit IRC | 22:36 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/235423 | 22:40 |
| *** rhallisey has quit IRC | 22:41 | |
| *** absubram has quit IRC | 22:41 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/235423 | 22:44 |
| *** mburned is now known as mburned_out | 22:44 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs https://review.openstack.org/269872 | 22:48 |
| *** penick has quit IRC | 22:52 | |
| *** ayoung has quit IRC | 22:55 | |
| *** dustins has quit IRC | 23:05 | |
| *** marcusvrn_ has quit IRC | 23:07 | |
| *** pradk has quit IRC | 23:11 | |
| *** penick has joined #tripleo | 23:13 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Add capabilities filter for Nova https://review.openstack.org/288087 | 23:17 |
| *** ayoung has joined #tripleo | 23:18 | |
| *** weshay has quit IRC | 23:21 | |
| *** saneax_AFK is now known as saneax | 23:28 | |
| *** ianw has quit IRC | 23:28 | |
| *** mburned_out is now known as mburned | 23:29 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: test all jobs w/ NETISO_V4 https://review.openstack.org/288176 | 23:32 |
| *** morazi has quit IRC | 23:35 | |
| *** yamahata has joined #tripleo | 23:35 | |
| *** penick has quit IRC | 23:37 | |
| *** panda has quit IRC | 23:38 | |
| *** panda has joined #tripleo | 23:38 | |
| *** rhallisey has joined #tripleo | 23:42 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Add capabilities filter for Nova https://review.openstack.org/288087 | 23:52 |
| *** ianw has joined #tripleo | 23:59 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Add capabilities filter for Nova https://review.openstack.org/288087 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
