| *** slagle has joined #tripleo | 00:02 | |
| *** Marga__ has joined #tripleo | 00:04 | |
| *** Marga__ has quit IRC | 00:06 | |
| *** Marga__ has joined #tripleo | 00:07 | |
| *** Marga_ has quit IRC | 00:07 | |
| *** morazi has quit IRC | 00:16 | |
| *** jrist has quit IRC | 00:17 | |
| *** jrist has joined #tripleo | 00:18 | |
| *** saneax is now known as saneax_AFK | 00:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: [WIP] Add image size report https://review.openstack.org/289629 | 00:19 |
|---|---|---|
| *** bnemec has quit IRC | 00:24 | |
| pradk | slagle, hi network isolation is enabled in ci now correct? | 00:41 |
| pradk | slagle, i reverted the revert commit and ci passes on the jobs https://review.openstack.org/#/c/289435/ afaict | 00:42 |
| *** dmacpher-afk has quit IRC | 00:50 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common: Change the private subnet of the overcloud tenant network https://review.openstack.org/289639 | 00:54 |
| *** trown|outtypewww has quit IRC | 00:55 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 00:56 |
| slagle | i'm not going to approve that | 01:13 |
| slagle | ww | 01:14 |
| *** trozet has quit IRC | 01:21 | |
| pradk | slagle, i'm trying to understand why it was reverted to being with | 01:22 |
| pradk | begin* | 01:22 |
| slagle | pradk: that comment wasn't for you, it was the wrong window | 01:22 |
| *** stevebaker has joined #tripleo | 01:22 | |
| slagle | pradk: but anyway, i gather it was reverted because it didnt work with network isolation | 01:22 |
| pradk | slagle, right but isnt ci running with net iso now? and since its passing the ci in all scenarios i'm wondering whats the issue to fix | 01:23 |
| *** lazy_prince has joined #tripleo | 01:23 | |
| slagle | pradk: maybe there's nothing left to fix | 01:24 |
| slagle | it's passed Ci now, that's good | 01:25 |
| slagle | jistr ought to review it, since he's the one working on upgrades | 01:25 |
| slagle | he may or may not have time for that | 01:25 |
| pradk | slagle, ok, I ran the upgrade script past him this morning and he was mostly fine with it.. one thing i was not sure is does removing pcs resource removes the constraint or if i have to do it explicitly like i did | 01:26 |
| pradk | i did it anyway | 01:27 |
| *** lazy_prince has quit IRC | 01:28 | |
| *** killer_prince has joined #tripleo | 01:28 | |
| *** killer_prince has quit IRC | 01:34 | |
| *** lazy_prince has joined #tripleo | 01:35 | |
| *** dmacpher has joined #tripleo | 01:35 | |
| *** trozet has joined #tripleo | 01:37 | |
| *** panda has quit IRC | 01:39 | |
| *** panda has joined #tripleo | 01:40 | |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Zerofree the image if possible https://review.openstack.org/289054 | 01:45 |
| *** trozet has quit IRC | 01:47 | |
| *** killer_prince has joined #tripleo | 01:50 | |
| *** lazy_prince has quit IRC | 01:50 | |
| *** killer_prince has quit IRC | 01:57 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: [WIP] Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 02:04 |
| *** yamahata has joined #tripleo | 02:12 | |
| *** shivrao has quit IRC | 02:26 | |
| EmilienM | dsneddon: can you maybe use my patch in depends-on? so we can actually test it | 02:33 |
| EmilienM | https://review.openstack.org/#/c/286344/ | 02:33 |
| *** Slower has quit IRC | 02:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add image size report https://review.openstack.org/289629 | 02:58 |
| openstackgerrit | xin wu proposed openstack/os-net-config: Enable os_net_config to configure IVS https://review.openstack.org/274492 | 02:59 |
| *** psanchez has joined #tripleo | 03:04 | |
| *** stevebaker has quit IRC | 03:06 | |
| *** stevebaker has joined #tripleo | 03:06 | |
| *** yuanying has quit IRC | 03:16 | |
| *** masco has joined #tripleo | 03:29 | |
| *** yuanying has joined #tripleo | 03:38 | |
| *** yuanying has quit IRC | 03:40 | |
| *** yuanying has joined #tripleo | 03:41 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: clear our rpm/dnf/yum data in chroot https://review.openstack.org/281596 | 03:54 |
| *** yuanying has quit IRC | 04:00 | |
| *** yuanying has joined #tripleo | 04:01 | |
| *** yuanying has quit IRC | 04:05 | |
| *** yuanying has joined #tripleo | 04:07 | |
| *** lazy_prince has joined #tripleo | 04:25 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: [WIP] Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 04:27 |
| *** links has joined #tripleo | 04:33 | |
| *** Marga__ has quit IRC | 04:34 | |
| *** xinwu has quit IRC | 04:36 | |
| *** jaosorior has joined #tripleo | 04:46 | |
| *** Marga_ has joined #tripleo | 04:51 | |
| *** saneax_AFK is now known as saneax | 04:52 | |
| openstackgerrit | Merged openstack/diskimage-builder: Zerofree the image if possible https://review.openstack.org/289054 | 04:53 |
| *** Marga_ has quit IRC | 04:57 | |
| *** Marga_ has joined #tripleo | 04:57 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/tripleo-common: Fix typos in code https://review.openstack.org/289707 | 05:00 |
| *** Marga_ has quit IRC | 05:01 | |
| *** Marga_ has joined #tripleo | 05:01 | |
| *** xinwu has joined #tripleo | 05:08 | |
| *** shivrao has joined #tripleo | 05:08 | |
| *** stendulker has joined #tripleo | 05:09 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add an environment to use a swap partition https://review.openstack.org/289084 | 05:12 |
| *** rlandy has quit IRC | 05:24 | |
| *** akuznetsov has joined #tripleo | 05:26 | |
| *** shivrao has quit IRC | 05:29 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: [WIP] Add support for building images capable of UEFI https://review.openstack.org/287784 | 05:34 |
| *** zaneb has quit IRC | 05:35 | |
| *** zaneb has joined #tripleo | 05:38 | |
| *** dmacpher has quit IRC | 05:47 | |
| *** dmacpher has joined #tripleo | 05:47 | |
| *** stendulker has quit IRC | 05:54 | |
| *** shivrao has joined #tripleo | 05:56 | |
| *** stendulker has joined #tripleo | 05:59 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/tripleo-docs: Fix some typos in docs https://review.openstack.org/289720 | 06:01 |
| *** shakamunyi has quit IRC | 06:02 | |
| *** rcernin has joined #tripleo | 06:02 | |
| *** shivrao has quit IRC | 06:29 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: [WIP] Add support for building images capable of UEFI https://review.openstack.org/287784 | 06:32 |
| *** jcoufal has joined #tripleo | 06:34 | |
| *** jcoufal has quit IRC | 06:39 | |
| *** admin0 has joined #tripleo | 06:44 | |
| *** tzumainn has quit IRC | 06:44 | |
| *** david-lyle has quit IRC | 06:44 | |
| *** david-lyle_ has joined #tripleo | 06:44 | |
| *** admin0 has quit IRC | 06:48 | |
| *** jtomasek has joined #tripleo | 06:49 | |
| *** Marga_ has quit IRC | 06:56 | |
| *** pcaruana has quit IRC | 06:57 | |
| *** leanderthal|afk is now known as leanderthal | 07:00 | |
| *** shivrao has joined #tripleo | 07:03 | |
| marios | looks like review bot might be down/not working fyi | 07:16 |
| *** hjensas has quit IRC | 07:17 | |
| *** akuznetsov has quit IRC | 07:17 | |
| *** liverpooler has joined #tripleo | 07:19 | |
| *** shivrao has quit IRC | 07:19 | |
| *** akuznetsov has joined #tripleo | 07:21 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/289758 | 07:22 |
| marios | hmm looks working again | 07:22 |
| *** ccamacho has joined #tripleo | 07:24 | |
| *** sshnaidm has quit IRC | 07:26 | |
| *** gfidente has joined #tripleo | 07:26 | |
| *** dshulyak has joined #tripleo | 07:27 | |
| *** shivrao has joined #tripleo | 07:32 | |
| *** oshvartz has joined #tripleo | 07:35 | |
| *** akuznetsov has quit IRC | 07:38 | |
| *** rdopiera has joined #tripleo | 07:45 | |
| *** akuznetsov has joined #tripleo | 07:45 | |
| *** athomas has joined #tripleo | 07:46 | |
| *** dshulyak has quit IRC | 07:50 | |
| *** ohamada has joined #tripleo | 07:54 | |
| *** rwsu has joined #tripleo | 07:56 | |
| *** cmyster has joined #tripleo | 07:59 | |
| *** jprovazn has joined #tripleo | 08:01 | |
| openstackgerrit | Richard Su proposed openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/287561 | 08:02 |
| *** fgimenez has joined #tripleo | 08:03 | |
| *** tzumainn has joined #tripleo | 08:03 | |
| *** fgimenez has quit IRC | 08:03 | |
| *** fgimenez has joined #tripleo | 08:03 | |
| *** admin0 has joined #tripleo | 08:05 | |
| *** akuznetsov has quit IRC | 08:07 | |
| openstackgerrit | Merged openstack/instack-undercloud: Store events in Undercloud Ceilometer https://review.openstack.org/286734 | 08:09 |
| *** shivrao has quit IRC | 08:11 | |
| *** aufi has joined #tripleo | 08:14 | |
| *** shardy has joined #tripleo | 08:18 | |
| *** paramite has joined #tripleo | 08:23 | |
| *** paramite_ has joined #tripleo | 08:24 | |
| *** paramite_ has quit IRC | 08:24 | |
| *** paramite_ has joined #tripleo | 08:26 | |
| *** paramite has quit IRC | 08:28 | |
| *** mikelk has joined #tripleo | 08:30 | |
| *** chem has joined #tripleo | 08:31 | |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Merge "Store events in Undercloud Ceilometer" (cherry picked from commit e4789782cdf5cb8373ab318bb2c5d39421eb3259) https://review.openstack.org/289788 | 08:38 |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Store events in Undercloud Ceilometer https://review.openstack.org/289788 | 08:42 |
| jprovazn | gfidente: good morning | 08:45 |
| gfidente | jprovazn, morning :) | 08:45 |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Store events in Undercloud Ceilometer https://review.openstack.org/289789 | 08:45 |
| jprovazn | gfidente: do you have a sec? I've made it to another level. Although this might be the final one, it seems that difficulty is set to "hard" or "godlike" | 08:47 |
| jprovazn | this isn't a symptom of unreachable keystone tomas was hitting yesterday, is it? http://paste.openstack.org/show/489641/ | 08:49 |
| jprovazn | gfidente: ? | 08:49 |
| gfidente | don't think so because you apparently get 500 | 08:49 |
| jprovazn | yep :) | 08:50 |
| gfidente | do the keystone logs in the undercloud tell anything useful? | 08:50 |
| *** jcoufal has joined #tripleo | 08:50 | |
| jprovazn | gfidente: it's quite interesting that the failed request is not present in keystone.log at all | 08:51 |
| jprovazn | and same for httpd | 08:51 |
| gfidente | adn you can source overcloudrc and work with keystone? | 08:52 |
| jprovazn | gfidente: yes | 08:53 |
| gfidente | but I think endpoint-list | 08:54 |
| gfidente | will give you only the keystone endpoint, it didn't create the other endpoints right? | 08:54 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 08:54 |
| *** dmacpher has quit IRC | 08:56 | |
| *** admin0 has quit IRC | 08:57 | |
| *** admin0 has joined #tripleo | 08:57 | |
| *** olap has joined #tripleo | 08:58 | |
| openstackgerrit | Richard Su proposed openstack/instack-undercloud: Rename store_events to undercloud_ceilometer_store_events https://review.openstack.org/289795 | 08:59 |
| *** hjensas has joined #tripleo | 09:00 | |
| jprovazn | gfidente: you mean endpoint-list on UC? | 09:00 |
| gfidente | on OC | 09:00 |
| jprovazn | gfidente: thare are many endpoints | 09:01 |
| jprovazn | the list looks complete | 09:01 |
| gfidente | uhm | 09:05 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 09:06 |
| *** admin0 has quit IRC | 09:07 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/289758 | 09:08 |
| *** admin0 has joined #tripleo | 09:08 | |
| *** mbound has joined #tripleo | 09:08 | |
| *** admin0 has quit IRC | 09:09 | |
| *** admin0 has joined #tripleo | 09:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/270110 | 09:11 |
| *** jistr has joined #tripleo | 09:11 | |
| *** ifarkas has joined #tripleo | 09:13 | |
| *** lucas-dinner is now known as lucasagomes | 09:19 | |
| *** akrivoka has joined #tripleo | 09:27 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Remove GlanceRegistry from EndpointMap https://review.openstack.org/289812 | 09:30 |
| *** aufi_ has joined #tripleo | 09:30 | |
| *** aufi has quit IRC | 09:31 | |
| *** nico_auv has joined #tripleo | 09:34 | |
| *** shadower has joined #tripleo | 09:37 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 09:37 |
| *** mkovacik has quit IRC | 09:44 | |
| *** derekh has joined #tripleo | 09:44 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/tripleo-heat-templates: Fix typos https://review.openstack.org/265126 | 09:46 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Upgrades: object storage node upgrade fix https://review.openstack.org/289826 | 09:56 |
| gfidente | d0ugal, you around? | 09:57 |
| d0ugal | gfidente: Yup! | 09:57 |
| gfidente | d0ugal, hey with jprovazn I think we hit a problem where novaclient.v2.client.Client shouldn't be called directly, http://paste.openstack.org/show/489646/ | 09:58 |
| gfidente | d0ugal, I was wondering why we don't remove the lines which create the flavor in the OC entirely https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L510 | 09:58 |
| gfidente | ? | 09:58 |
| d0ugal | gfidente: Makes sense, I don't think they are needed now | 09:59 |
| gfidente | d0ugal, yeah there are some default flavors from nova anyway | 10:00 |
| *** fgimenez has quit IRC | 10:02 | |
| *** fgimenez has joined #tripleo | 10:02 | |
| *** fgimenez has joined #tripleo | 10:02 | |
| d0ugal | gfidente: Why do we import from os_cloud_config.utils :/ that seems wrong. | 10:05 |
| d0ugal | Anyway, deleting :D | 10:05 |
| *** tosky has joined #tripleo | 10:06 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup for unbound variable in swift upgrade script heredoc https://review.openstack.org/289831 | 10:07 |
| marios | jistr: i think https://review.openstack.org/#/c/289826/1/extraconfig/tasks/major_upgrade_object_storage.sh is better going to abandon ^^^ | 10:09 |
| marios | (didn't look in time) | 10:09 |
| *** dtantsur|afk is now known as dtantsur | 10:11 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the demo flavour creation https://review.openstack.org/289838 | 10:11 |
| d0ugal | gfidente: ^ | 10:11 |
| dtantsur | morning folks | 10:12 |
| dtantsur | we have a crazy snow today, so here's your daily owl: https://dl.dropboxusercontent.com/u/1730743/owls/vM54ZvmrdAE.jpg | 10:13 |
| dtantsur | EmilienM, hey! please take a look at https://review.openstack.org/#/c/285333/ I got it finally passing the gate | 10:14 |
| *** xinwu has quit IRC | 10:15 | |
| shadower | d0ugal: does python-tripleoclient have a CI? I'm not seeing anything in https://review.openstack.org/289838 | 10:16 |
| d0ugal | shadower: It does, probably just being a bit slow. | 10:17 |
| dtantsur | it's slow like usual ;) | 10:17 |
| shadower | right, good :-) | 10:17 |
| * shadower was worried there for a moment | 10:17 | |
| d0ugal | CI always worries me. | 10:17 |
| shadower | lol | 10:17 |
| gfidente | guys I only started realizing how complex that is when getting my hands dirty with it | 10:19 |
| *** mikelk has quit IRC | 10:20 | |
| gfidente | d0ugal, believe it or not, our pingtest in CI was using exactly m1.demo | 10:21 |
| d0ugal | gfidente: lol | 10:22 |
| d0ugal | gfidente: oh dear :) | 10:22 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common: Use m1.tiny instead of m1.demo for the pingtest VM https://review.openstack.org/289845 | 10:24 |
| gfidente | d0ugal, so the other change probably needs a depends-on | 10:24 |
| d0ugal | gfidente: I'll add that. | 10:28 |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** openstackgerrit has joined #tripleo | 10:33 | |
| shardy | gfidente, jistr: Hey, question about pacemaker_resource_restart.sh | 10:36 |
| * jistr listening | 10:36 | |
| shardy | it seems like we run that via OS::TripleO::Tasks::ControllerPostPuppet every update as it's defined in ./environments/puppet-pacemaker.yaml | 10:36 |
| jistr | yes | 10:36 |
| shardy | is that expected, e.g that we run that *every* update, even when it's not an update or upgrade? | 10:36 |
| shardy | because it takes services down, which can interrupt things if e.g you're just doing a scale-out, not changing anything on the controllers? | 10:37 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/pacemaker_resource_restart.sh#L21 | 10:37 |
| jistr | yes, to make sure that config changes get applied. Ideally we'd have puppet output a list of services that need restart, and only restart those. enotimplemented | 10:37 |
| shardy | jistr: but doesn't this mean we take the whole cloud down every time we add a compute node? | 10:38 |
| jistr | yea it does, every time stack-update runs, services are bumped | 10:38 |
| shardy | :( | 10:38 |
| shardy | I think we have to fix that | 10:38 |
| jistr | yea. Though making it run only on update/upgrade isn't the right fix here... | 10:39 |
| shardy | rcernin: ^^ sounds like it's a known issue which we need to address | 10:40 |
| rcernin | shardy, jistr: thanks, I will open BZ for this. | 10:41 |
| jistr | making it run only on update/upgrade would result in "lurking" config file changes. In the best case someone would go and restart the service properly via pacemaker manually, but in worse case they might restart it on only one controller, and then services run different configs on different nodes, and that can lead to issues which are hard to debug. | 10:42 |
| jprovazn | Overcloud Endpoint: http://[2001:db8:fd00:1000::10]:5000/v2.0 | 10:42 |
| jprovazn | Overcloud Deployed | 10:42 |
| jprovazn | clean_up DeployOvercloud: | 10:42 |
| jprovazn | END return value: 0 | 10:42 |
| jprovazn | magic! | 10:42 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 10:42 |
| jistr | so what we probably should do here is to make puppet output a list of what needs restarting, and read that list from that shell script | 10:42 |
| shardy | jistr: ack, I appreciate it's a hard problem, but I guess we have to come up with a better solution than always bouncing everything | 10:43 |
| *** mgould has joined #tripleo | 10:43 | |
| jistr | yea. I think what i mentioned above is doable. I think social said they did something similar in his previous company. | 10:44 |
| gfidente | jprovazn, finally :) | 10:44 |
| jistr | just needs someone with bandwidth allocation to this issue | 10:44 |
| jistr | would need puppet-pacemaker changes as well as t-h-t | 10:44 |
| gfidente | I think social started working on this already | 10:45 |
| gfidente | EmilienM, knows more about status | 10:45 |
| *** admin0 has quit IRC | 10:55 | |
| *** stendulker has quit IRC | 10:56 | |
| *** pblaho has joined #tripleo | 10:58 | |
| shadower | jprovazn: congrats! | 10:58 |
| *** rwsu has quit IRC | 11:01 | |
| jprovazn | gfidente: shadower: yea, I think my hope was rushed :) - [stack@instack ~]$ nova list | 11:03 |
| jprovazn | ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5947dfcb-4ae2-411a-82de-838dc5b10292) | 11:03 |
| *** hjensas has quit IRC | 11:03 | |
| jprovazn | gfidente: ^ this is probably the same 500 error I was hitting before | 11:04 |
| * jprovazn looks closer | 11:04 | |
| *** rwsu has joined #tripleo | 11:04 | |
| *** oshvartz has quit IRC | 11:04 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 11:06 |
| *** fgimenez has quit IRC | 11:08 | |
| jprovazn | gfidente: it seems to be an ipv6 error - http://paste.openstack.org/show/489663/ | 11:09 |
| *** fgimenez has joined #tripleo | 11:10 | |
| shadower | :-( | 11:12 |
| gfidente | jprovazn, that's the version of python-memcached | 11:12 |
| jprovazn | gfidente: ummm, I used your images | 11:13 |
| gfidente | shadower, ^^ I had to update the image with that too | 11:13 |
| *** trown has joined #tripleo | 11:13 | |
| gfidente | jprovazn, yes but in there I updated memcached only, not the python client | 11:13 |
| jprovazn | aha, ack | 11:13 |
| gfidente | jprovazn, we can hack the image and redeploy and it should work | 11:13 |
| gfidente | or you can update-in-place and restart nova | 11:14 |
| jprovazn | gfidente: yep | 11:14 |
| *** admin0 has joined #tripleo | 11:14 | |
| *** trown is now known as trown|outtypewww | 11:14 | |
| shadower | ah ok | 11:14 |
| shadower | gfidente: do you have a link to the python-memcached rpm? | 11:15 |
| *** Marga_ has joined #tripleo | 11:16 | |
| *** oshvartz has joined #tripleo | 11:17 | |
| gfidente | shadower, jprovazn so the fix was this https://bugs.launchpad.net/python-memcached/+bug/1028412 | 11:19 |
| openstack | Launchpad bug 1028412 in Python Memcached "IPv6 not supported" [Medium,Fix committed] - Assigned to Sean Reifschneider (jafo) | 11:19 |
| gfidente | we need to track back if there is a build which includes that | 11:19 |
| jprovazn | ack | 11:19 |
| openstackgerrit | Karim Boumedhel proposed openstack/puppet-pacemaker: puppet-pacemaker rhevm stonith fails https://review.openstack.org/288527 | 11:20 |
| gfidente | from the commit log/tags it looks like it was firstly shipped in 1.49 | 11:21 |
| gfidente | derekh, see my last comment in https://review.openstack.org/#/c/289445/1 | 11:27 |
| gfidente | I noticed in rhel there are backports of the needed patches into slightly older packages | 11:28 |
| gfidente | do you know how those get to centos? | 11:28 |
| derekh | gfidente: do you know when they were backported in rhel? I kindof assumed it would just appear in centos if its in RHEL | 11:30 |
| social | gfidente: jayg|g0n3 might be doing something like that | 11:34 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common: Adds override for the overcloud node user in upgrade-non-controller https://review.openstack.org/289871 | 11:34 |
| social | jistr: yes we had service and exec override and we just created list of things to restart but that was on downtimeless updates where we preffered manual restart on critical services, normally we would just reload/restart | 11:35 |
| jistr | social: yea, unfortunately letting just puppet restart pacemaker-managed services isn't possible, as puppet only knows about a single node, while restart of pacemaker services needs to be orchestrated cluster-wide (e.g. we need to ensure that all controllers have already written their conf changes into files before we restart the service) | 11:37 |
| * social would prefer less pacemaker | 11:38 | |
| *** aufi_ has quit IRC | 11:38 | |
| social | I don't see much point for it if we have haproxy on openstack services | 11:38 |
| *** pblaho has quit IRC | 11:39 | |
| jistr | perhaps for many services it's not that important, but for the several which require a resource agent (e.g. galera, rabbit etc.) it's kina helpful | 11:40 |
| jistr | *kinda | 11:41 |
| *** mikelk has joined #tripleo | 11:42 | |
| gfidente | marios, netiso on liberty/ci is still broken, we need to land https://review.openstack.org/#/c/289489/ to liberty for it to work | 11:42 |
| social | jistr: another way is not to restart just mark as tainted and restart in another run? | 11:48 |
| social | jistr: or restart always from puppet with pacemaker disabled and again after puppet run restart with pacemaker | 11:49 |
| *** ansiwen has joined #tripleo | 11:49 | |
| *** hjensas has joined #tripleo | 11:50 | |
| marios | gfidente: ok thanks for info | 11:53 |
| jistr | social: tainted and restart in another run might work i think. Restart from puppet with pcmk in maintenance might not work well always, as we'd get services running different configs on different nodes. For openstack services this might not be a problem, but for the backend services it might i think. Also currently Puppet doesn't know how to restart pacemaker services (which is lucky because without proper orchestration it could break things). | 11:53 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Remove the demo flavour creation https://review.openstack.org/289838 | 11:57 |
| *** sshnaidm has joined #tripleo | 11:57 | |
| *** rhallisey has joined #tripleo | 11:59 | |
| *** jaosorior has quit IRC | 12:02 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 12:02 |
| *** jaosorior has joined #tripleo | 12:03 | |
| *** olap has quit IRC | 12:04 | |
| *** olap has joined #tripleo | 12:06 | |
| rhallisey | derekh, https://review.openstack.org/#/c/288915/ | 12:08 |
| openstackgerrit | Merged openstack/tripleo-docs: Update README files for TripleO documentation https://review.openstack.org/267199 | 12:09 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 12:10 |
| gfidente | slagle, the results were good :) | 12:12 |
| slagle | should we just merge it? | 12:13 |
| slagle | it had actually passed ha on the last run | 12:13 |
| slagle | so that means it passed ceph and ha on 2 different runs | 12:13 |
| gfidente | yeah I'd also land the other two ci/common patches | 12:13 |
| gfidente | and your port for -common into liberty | 12:13 |
| slagle | gfidente: the tripleo-common one failed on master | 12:14 |
| slagle | with a puppet keystone problem | 12:14 |
| slagle | http://logs.openstack.org/89/289489/1/check-tripleo/gate-tripleo-ci-f22-ceph/00c4ae8/console.html | 12:15 |
| openstackgerrit | Merged openstack/tripleo-docs: Quote control plane mask length https://review.openstack.org/281516 | 12:15 |
| *** weshay has joined #tripleo | 12:15 | |
| gfidente | slagle, yeah but it's unrelated, it even happened before the tenantvm was actually launched | 12:16 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 12:17 |
| *** ccamacho has quit IRC | 12:18 | |
| *** ccamacho has joined #tripleo | 12:18 | |
| *** lucasagomes is now known as lucas-hungry | 12:19 | |
| derekh | rhallisey: cool, is there anything else to finsih it or is it failing on something on something non container specific ? | 12:19 |
| *** jcoufal has quit IRC | 12:22 | |
| *** jcoufal has joined #tripleo | 12:22 | |
| rhallisey | derekh, the patch isn't pulling in all my deps | 12:22 |
| shardy | derekh: Hey, looking here http://logs.openstack.org/15/288915/4/check-tripleo/gate-tripleo-ci-f22-containers/95dc917/console.html | 12:22 |
| shardy | ZUUL_CHANGES='openstack/tripleo-heat-templates:master:refs/changes/22/288822/4 openstack/tripleo-heat-templates:master:refs/changes/18/287918/4 openstack-infra/tripleo-ci:master:refs/changes/15/288915/4' | 12:22 |
| shardy | It's missing the Depends-On pointing at https://review.openstack.org/#/c/289565 | 12:23 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 12:23 |
| derekh | rhallisey: shardy typo "Deponds" | 12:24 |
| shardy | doh! | 12:24 |
| rhallisey | lol.. | 12:24 |
| shardy | derekh: thanks! :) | 12:24 |
| derekh | shardy: np | 12:24 |
| openstackgerrit | Merged openstack/tripleo-common: Change the private subnet of the overcloud tenant network https://review.openstack.org/289489 | 12:25 |
| slagle | folks, i'm going to merge some patches to unblock liberty ci | 12:25 |
| slagle | they aren't 100% green, but afaict, they have passed enough to say it fixes the problem | 12:25 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 12:26 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Collect the common logic in tripleo-common https://review.openstack.org/228991 | 12:27 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Collect constants in one file https://review.openstack.org/235977 | 12:27 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Generate a unique DeployIdentifier on updates https://review.openstack.org/268126 | 12:27 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 12:28 |
| derekh | Hey, I've been asked recently 2 or 3 times by people joining tripleo what kind of dev env is needed, and I started listing options, I think may of us have a different setup | 12:28 |
| derekh | andbody want to describe there setup here so we can point newcommers at it ? https://etherpad.openstack.org/p/tripleo-dev-env-census | 12:29 |
| gfidente | ccamacho, ^^ | 12:29 |
| *** saneax is now known as saneax_AFK | 12:29 | |
| openstackgerrit | Merged openstack/tripleo-common: Change the private subnet of the overcloud tenant network https://review.openstack.org/289639 | 12:30 |
| shardy | derekh: I've got a couple of nodes I've been deploying via fake_pxe, but I'm considering switching to ovb on them instead | 12:31 |
| shardy | derekh: do your nodes have 2 nics, or can you get away with one? | 12:32 |
| *** mkovacik has joined #tripleo | 12:34 | |
| derekh | shardy: I've gotten away with one nic on each | 12:36 |
| *** Goneri has quit IRC | 12:36 | |
| shardy | derekh: cool, I've not yet got a vlan capable switch either so was wondering if I can still get things running | 12:36 |
| shardy | thanks | 12:36 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add a ceph-storage node upgrade script for the upgrade workflow https://review.openstack.org/289896 | 12:36 |
| derekh | shardy: yup it should be possible, I've not done anything special with vlans | 12:37 |
| *** mannidi has joined #tripleo | 12:37 | |
| shardy | derekh: good to know - I'll give it a go and make some notes - would probably be a nice setup to document in addition to single node virt | 12:38 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make AllNodesExtraConfig depend on the validation deployments https://review.openstack.org/289568 | 12:38 |
| derekh | shardy: I've setup my OVB cloud with 172.24.4.0/24 floating IP's then put a rout on my laptop to get to them via the controller IP "ip route add 172.24.4.0/24 dev wlp4s0 via 192.168.1.51" | 12:39 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 12:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 12:43 |
| shardy | derekh: nice - I was actually considering using another (laptop) box as the controller node, then using the wireless as the public nic and wired connected to a seperate switch as the private lan | 12:44 |
| slagle | gfidente: all merged. let's hope we see some green | 12:44 |
| shardy | that works OK using the laptop as the undercloud, but I'm not sure with a full controller configured via packstack due to the bridges etc | 12:44 |
| *** admin0 has quit IRC | 12:44 | |
| shardy | I'm guessing that won't work so well on the wireless device | 12:44 |
| * shardy should just buy some more nics really ;) | 12:45 | |
| *** admin0 has joined #tripleo | 12:46 | |
| *** trown|outtypewww has quit IRC | 12:48 | |
| * gfidente gets popcorns | 12:48 | |
| slagle | gfidente: i'm fine with actually just rechecking the tip of your patch series in liberty | 12:49 |
| slagle | then we can merge everything before it all at once | 12:49 |
| gfidente | slagle, ack, it still misses stuff which isn't landed in master yet | 12:49 |
| slagle | assuming the tip passes :) | 12:49 |
| gfidente | but we were testing the remaining patches from master with shadower and jprovazn and I think they're not too bad | 12:50 |
| jprovazn | gfidente: slagle: yep, although I didn't make it working from e2e I think we are close :) | 12:51 |
| *** lazy_prince has quit IRC | 12:51 | |
| gfidente | slagle, liberty tip is https://review.openstack.org/#/c/289758/ | 12:52 |
| gfidente | slagle, master tip is https://review.openstack.org/#/c/272089/ | 12:52 |
| gfidente | plus if anyone wants to merge https://review.openstack.org/#/c/277601/ that'd be useful too | 12:52 |
| *** masco has quit IRC | 12:52 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Increase default netdev_max_backlog to 10x https://review.openstack.org/277601 | 12:57 |
| gfidente | thanks guys, the liberty port ^ :) | 12:57 |
| marios | gfidente: err... you just uploaded v5 | 12:58 |
| gfidente | marios, shit :( | 12:58 |
| marios | gfidente: wai | 12:58 |
| gfidente | because I was in wrong branch | 12:58 |
| *** dprince has joined #tripleo | 13:00 | |
| adarazs | gfidente: hi, can you tell me again where this /tmp/net-iso.yaml file comes from in the OVERCLOUD_DEPLOY_ARGS? | 13:00 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Increase default netdev_max_backlog to 10x https://review.openstack.org/277601 | 13:01 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Increase default netdev_max_backlog to 10x https://review.openstack.org/289907 | 13:02 |
| *** aufi_ has joined #tripleo | 13:03 | |
| *** jayg|g0n3 is now known as jayg | 13:03 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Add an IPv6 gate job https://review.openstack.org/289445 | 13:04 |
| gfidente | adarazs, from tripleo-ci/test-environments | 13:06 |
| gfidente | tripleo-ci the actual github repo I mean | 13:06 |
| jprovazn | gfidente: shadower: the new python-memcached solved the issue, my OC is happy now | 13:06 |
| gfidente | jprovazn :) | 13:06 |
| adarazs | gfidente: hm, if I understand correctly these IPs will remain the same for the V6 deployment too. | 13:07 |
| *** trown has joined #tripleo | 13:07 | |
| shadower | jprovazn: awesome! My undercloud keeps runinng oom again with 8 bloody GB | 13:08 |
| jprovazn | shadower: that's really weird, which process is so greedy? | 13:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Increase default netdev_max_backlog to 10x https://review.openstack.org/277601 | 13:10 |
| derekh | Hey, I've been asked recently 2 or 3 times by people joining tripleo what kind of dev env is needed, and I started listing options, I think may of us have a different setup anybody want to describe their setup here so we can point newcommers at it ? https://etherpad.openstack.org/p/tripleo-dev-env-census | 13:10 |
| *** oshvartz has quit IRC | 13:12 | |
| *** olap has quit IRC | 13:12 | |
| shadower | jprovazn: going to observe it now during redeployment. But I've restarted the undercloud & hope that helps | 13:13 |
| *** olap has joined #tripleo | 13:14 | |
| jprovazn | shadower: fwiw, OOM killer usually prints current state into logs when killing a process | 13:14 |
| *** pblaho has joined #tripleo | 13:15 | |
| shadower | jprovazn: ooh, thanks I'll have a look | 13:17 |
| shadower | jprovazn: fwiw swift-proxy-server just came out of nowhere and started occupying 40% of RAM | 13:17 |
| *** admin0 has quit IRC | 13:20 | |
| jprovazn | shadower: ha, weird | 13:20 |
| slagle | is it the image upload? | 13:22 |
| shadower | slagle: possibly, not sure how to check | 13:23 |
| derekh | shadower: we need to find out why, tag your it ;-) | 13:23 |
| *** lucas-hungry is now known as lucasagomes | 13:23 | |
| shadower | but it ends up killing the ovs agent which in turn prevents nova servers from booting | 13:23 |
| slagle | gfidente: can you sync with dprince about his comments on https://review.openstack.org/#/c/269058 | 13:23 |
| shadower | so I guess image upload is a plausible timeline | 13:24 |
| shadower | derekh: lol, fair enough | 13:24 |
| *** admin0 has joined #tripleo | 13:24 | |
| slagle | gfidente: i think we need to come to agreement on that. whether it's something we ought to fix now, could fix later, etc | 13:24 |
| *** oshvartz has joined #tripleo | 13:24 | |
| *** palexster has quit IRC | 13:25 | |
| pradk | jprovazn, hi i recall you reporting yesterday that aodh port was conflicting with haproxy with net iso? we revived the reverted patch and its passing ci with net iso https://review.openstack.org/#/c/289435/ | 13:26 |
| pradk | jprovazn, so i dont see anything that needs to be fixed afaict.. you think you can confirm in your env to make sure you dont see it again with this patch? | 13:27 |
| jprovazn | pradk: hi, yes - that's correct - yesterday I was hitting an issue when httpd service was failing to start during puppet run because the port was already occupied | 13:27 |
| jprovazn | pradk: I could give it a shot later today | 13:29 |
| pradk | jprovazn, that would be super helpful, thx! | 13:29 |
| jprovazn | pradk: I wonder if it might be a race issue - e.g. can you check on your deployment that if you start haproxy first and *then* httpd, that it works? | 13:29 |
| gfidente | adarazs, though I think those params can stay the same because the ctlplane network remains on IPv4 | 13:30 |
| jistr | marios: i'm thinking of picking up the channel switching next. I thought about various approaches, e.g. another script in tripleo common, or a nested stack switchable via resource-registry (defaulting to no-op and allowing a switch to accomodate for downstream needs). But in the end i think the snippet for this should be so short that we could just pass it as a generic heat param, and we'd get away from resource registry switches etc. I | 13:30 |
| jistr | realized i was thinking of "using a canon to hit a mosquito" (as the Czech proverb goes :) ) with resource registry. It could be really simple like this: http://fpaste.org/335711/44375314/raw/ | 13:30 |
| pradk | jprovazn, yea i'll play around, my default run worked fine | 13:30 |
| adarazs | gfidente: yep, exactly. | 13:30 |
| gfidente | adarazs, and those are the IPs the undercloud has on the ctlplane so it should work | 13:30 |
| gfidente | adarazs, ack | 13:30 |
| jprovazn | back in ~3 hours | 13:30 |
| adarazs | gfidente: I think so... waiting for the change to get through the zuul queue :) | 13:30 |
| *** jprovazn has quit IRC | 13:30 | |
| adarazs | going to take forever. | 13:30 |
| gfidente | like 2h yes :( | 13:31 |
| *** adarazs is now known as adarazs_afk | 13:32 | |
| jistr | marios: also if someone has a good reason to use a different upgrade init command, they can do so very easily (setting one param), as opposed to having to create their own nested stack, or edit t-h-t in place | 13:33 |
| jistr | i mean different than default | 13:33 |
| marios | jistr: ack looking | 13:34 |
| marios | jistr: sounds good talk some more on the call in a while? | 13:34 |
| jistr | marios: yup, thx | 13:34 |
| *** pblaho has quit IRC | 13:35 | |
| *** rlandy has joined #tripleo | 13:37 | |
| *** panda has quit IRC | 13:39 | |
| *** panda has joined #tripleo | 13:40 | |
| *** pradk_ has joined #tripleo | 13:45 | |
| *** morazi has joined #tripleo | 13:51 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 13:52 |
| *** saneax_AFK is now known as saneax | 13:56 | |
| *** saneax is now known as saneax_AFK | 13:56 | |
| *** rpothier has joined #tripleo | 13:56 | |
| *** saneax_AFK is now known as saneax | 13:57 | |
| *** gfidente^2nd has joined #tripleo | 13:57 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change default bond-mode https://review.openstack.org/287603 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add Management Network For System Administration. https://review.openstack.org/264963 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Adding ManagementIpSubnet to linux bridge net conf https://review.openstack.org/287602 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs https://review.openstack.org/269883 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix rabbit_hosts list for glance-api for IPv6 https://review.openstack.org/289432 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Set /64 cidr_netmask for pcmk VIPs when IPv6 https://review.openstack.org/289461 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Update typos https://review.openstack.org/289305 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync https://review.openstack.org/289422 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add all isolated networks to all nodes. https://review.openstack.org/268833 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/289758 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Make the Neutron subnet ipv6_{ra,address}_mode configurable https://review.openstack.org/289417 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/289355 | 13:58 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow for usage of pre-allocated IPs for the management network https://review.openstack.org/289289 | 13:58 |
| jistr | gfidente: congrats | 13:58 |
| gfidente^2nd | :( | 13:58 |
| jistr | that is the longest patch chain i've seen i think :D | 13:58 |
| *** gfidente has quit IRC | 13:58 | |
| adarazs_afk | :) | 13:59 |
| *** adarazs_afk is now known as adarazs | 13:59 | |
| gfidente^2nd | I am just cherry-picking from master as they merge and adding on-top to avoid conflicts beforehand | 13:59 |
| derekh | everybody go home, gfidente^2nd has just taken all the CI capacity ;-) | 13:59 |
| *** gfidente^2nd is now known as gfidente | 13:59 | |
| shardy | lol | 13:59 |
| *** jistr is now known as jistr|call | 13:59 | |
| gfidente | wait, I didn't rebase merge/conflict the series in master yet! | 14:00 |
| dprince | meeting time | 14:00 |
| *** gfidente has quit IRC | 14:00 | |
| *** gfidente has joined #tripleo | 14:00 | |
| *** eggmaste` has joined #tripleo | 14:05 | |
| *** Goneri has joined #tripleo | 14:06 | |
| *** liverpooler has quit IRC | 14:10 | |
| *** jcoufal has quit IRC | 14:10 | |
| *** jcoufal has joined #tripleo | 14:11 | |
| *** lblanchard has joined #tripleo | 14:21 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add Management Network For System Administration. https://review.openstack.org/264963 | 14:27 |
| *** links has quit IRC | 14:27 | |
| *** bnemec has joined #tripleo | 14:29 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Adding ManagementIpSubnet to linux bridge net conf https://review.openstack.org/287602 | 14:30 |
| *** miles has joined #tripleo | 14:32 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Make upload with --old-deploy-image only look for the old image https://review.openstack.org/289946 | 14:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add all isolated networks to all nodes. https://review.openstack.org/268833 | 14:33 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Change default bond-mode https://review.openstack.org/287603 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow for usage of pre-allocated IPs for the management network https://review.openstack.org/289289 | 14:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Update typos https://review.openstack.org/289305 | 14:34 |
| *** mgould has quit IRC | 14:34 | |
| thrash | dtantsur: trown d0ugal please have a look at the above patch: https://review.openstack.org/289946 | 14:35 |
| *** pblaho has joined #tripleo | 14:35 | |
| d0ugal | thrash: will do | 14:36 |
| thrash | d0ugal: thanks | 14:36 |
| thrash | d0ugal: it's in stable/liberty on purpose... I will update the commit message as to why. | 14:36 |
| thrash | (I forgot to put that) | 14:36 |
| dtantsur | thrash, please don't do that. introspection must use IPA | 14:37 |
| thrash | dtantsur: this fixes a bug. | 14:37 |
| dtantsur | which bug? | 14:37 |
| thrash | dtantsur: the --old-deploy-image will fail if the new images are not present. | 14:37 |
| dtantsur | thrash, that's not a bug | 14:37 |
| thrash | which is a very valid case. | 14:37 |
| dtantsur | do you want to disable introspection completely? | 14:37 |
| thrash | dtantsur: nope. | 14:38 |
| trown | ya it should fail if IPA is not available | 14:38 |
| dtantsur | thrash, but you're trying to ignore its ramdisk | 14:38 |
| trown | there is no other ramdisk for inspection | 14:38 |
| dtantsur | +1 | 14:38 |
| thrash | then why are we even bothering with this --old-deploy-image???? | 14:38 |
| thrash | dtantsur: trown https://bugzilla.redhat.com/show_bug.cgi?id=1295912 | 14:38 |
| openstack | bugzilla.redhat.com bug 1295912 in python-tripleoclient "rhel-osp-director: As part of "deploy oc 7.2 from uc 8.0", loading only 7.2 glance images results in:"Required file "./ironic-python-agent.initramfs" does not exist."" [Unspecified,New] - Assigned to brad | 14:38 |
| *** mannidi has quit IRC | 14:38 | |
| trown | because IPA does not support LIO on liberty | 14:38 |
| trown | ...yet... maybe | 14:39 |
| trown | I cant get those backports to pass CI | 14:39 |
| thrash | trown: dtantsur help text for '--old-deploy-image': help="Whether to use the deprecated deploy image instead of agent" | 14:39 |
| thrash | Note... "instead of" | 14:39 |
| trown | thrash: yep **deploy** | 14:39 |
| dtantsur | thrash, Note... "deploy" ;) | 14:39 |
| *** jistr|call is now known as jistr | 14:40 | |
| thrash | so, we should skip the copy to httpboot if old-deploy-image is specified? | 14:40 |
| thrash | If you only have the old images, this fails. Period. | 14:41 |
| thrash | I am fine with removing that part. | 14:41 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Make upload with --old-deploy-image only look for the old image https://review.openstack.org/289946 | 14:43 |
| thrash | dtantsur: trown that better? | 14:43 |
| thrash | crap | 14:44 |
| thrash | :P | 14:44 |
| dtantsur | thrash, no, sorry :) the old deploy ramdisk does not work for introspection | 14:44 |
| dtantsur | there is no bug, really. we're using a new ramdisk for introspection now | 14:44 |
| trown | thrash: ironic-python-agent is a requirement for liberty | 14:45 |
| trown | thrash: just not for **deploy** on libery | 14:45 |
| thrash | trown: dtantsur I get that it's a requirement... | 14:45 |
| thrash | trown: dtantsur if that's the case, then can one of you respond to the bz? | 14:46 |
| dtantsur | yep | 14:46 |
| thrash | and explain why you think it's not a bug | 14:46 |
| thrash | https://bugzilla.redhat.com/show_bug.cgi?id=1295912 | 14:46 |
| openstack | bugzilla.redhat.com bug 1295912 in python-tripleoclient "rhel-osp-director: As part of "deploy oc 7.2 from uc 8.0", loading only 7.2 glance images results in:"Required file "./ironic-python-agent.initramfs" does not exist."" [Unspecified,New] - Assigned to brad | 14:46 |
| *** dustins has joined #tripleo | 14:46 | |
| thrash | trown: dtantsur My thought on this is that if you specify --old-deploy-image, then it should not even attempt to upload the agent image | 14:47 |
| thrash | nor should it copy the agent image to /httpboot | 14:48 |
| trown | thrash: you are missing the fact that the agent image is used for inspection | 14:48 |
| thrash | trown: no, I am not. | 14:48 |
| pradk | jistr, hi | 14:48 |
| trown | not sure how else to explain it | 14:48 |
| thrash | that is not lost on me, I promise. | 14:48 |
| dtantsur | then how do you expect it to work? :) | 14:48 |
| thrash | trown: if you want your system to work, you run without '--old-deploy-image' | 14:48 |
| pradk | jistr, so for aodh upgrades, was discussing with Jarda, we could have the upgrades fallback to ceilo-alarm and only do aodh for fresh installs.. if that makes things less complicated for upgrades | 14:49 |
| thrash | trown: dtantsur I don't see the point of the --old-deploy-image then... | 14:49 |
| pradk | jistr, so the upgrade procedure will default to ceilo alarms in liberty (we just keep the code intact with some flag to flip) | 14:49 |
| thrash | if we aren't going to skip the NEW deploy image. | 14:50 |
| dtantsur | thrash, it was by trown's request due to the fact that liberty IPA image does not *deploy* with RHEL and no EPEL | 14:50 |
| thrash | what's the point of even HAVING the old stuff? | 14:50 |
| trown | thrash: IPA does not support LIO for deploy on liberty... and tgt is only shipped in EPEL | 14:51 |
| thrash | dtantsur: trown you two obviously have a better grasp on this than i do. I'm just trying to fix what appears to be a bug to me. However, if you feel it is not, then please take the bug and do as you see fit. | 14:51 |
| *** paramite_ is now known as paramite | 14:51 | |
| trown | thrash: so we needed a way to use the bash ramdisk in the meantime.... but only for deploy | 14:51 |
| thrash | trown: dtantsur I have abandoned the change for now., | 14:51 |
| dtantsur | thanks | 14:52 |
| thrash | if y'all decide you want it back, let me know. | 14:52 |
| *** lazy_prince has joined #tripleo | 14:52 | |
| trown | thanks thrash | 14:52 |
| openstackgerrit | Devananda van der Veen proposed openstack/diskimage-builder: [WIP] switch enable-serial-console element to ttyS2 https://review.openstack.org/289953 | 14:52 |
| jistr | pradk: hmm... then the patch itself would have to be reshuffled quite a bit though. The new puppet manifests will be run on upgraded deployments too as the last part of the upgrade, and also whenever an operation on overcloud is done via heat (e.g. scaling up computes). So essentially what that would require is to support switching between *both* options (alarm vs. AODH) in the puppet manifests, to make it work for existing deployments. imho | 14:53 |
| jistr | that's not too much easier than implementing a full upgrade, but i might be mistaken. /cc jcoufal | 14:53 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 14:55 |
| pradk | jistr, ok jcoufal's point was that you were concerned about upgrades in general and getting aodh this late into upgrade procedure was risky | 14:55 |
| pradk | jistr, so perhaps we could alleviate that concern by keeping aodh out of upgrade procedure | 14:55 |
| pradk | jistr, so from what i see we just need a flag to turn on ceilo-alarm in the puppet/manifest/overcloud*.yaml ? and we turn it on during upgrades but default to aodh in fresh installs? | 14:56 |
| *** dmsimard has joined #tripleo | 14:56 | |
| pradk | jistr, yea this would be a reshuffled patch for liberty only | 14:56 |
| *** trozet has joined #tripleo | 14:57 | |
| *** hjensas has quit IRC | 14:57 | |
| thrash | trown: dtantsur for what it's worth, I think there is still a bug there. :P | 14:59 |
| thrash | I just may have been over-reaching. | 14:59 |
| thrash | trown: dtantsur if the --old-deploy-image is specified, it tries to send both the old and the new to glance. That doesn't make sense, does it? | 15:00 |
| dtantsur | thrash, to glance - maybe. but we still must copy IPA to /httpboot | 15:00 |
| thrash | dtantsur: agreed. | 15:00 |
| thrash | dtantsur: so I think my initial patch was a bit over-reaching. | 15:00 |
| thrash | Let me update it, and see what you think? | 15:01 |
| trown | does it matter to send both to glance though? | 15:01 |
| thrash | I believe there is a bug there. | 15:01 |
| gfidente | dprince, the issue is that if we don't land that patch, upgrades are broken | 15:01 |
| jistr | jcoufal, pradk: yea, a flag in hiera, and if/else branching in puppet modules (incl. pcmk resources and constraints), and then supporting and testing essentially two different setups wrt ceilometer in Liberty (exploding the test matrix by another factor of 2). Tbh that proposal is a bit concerning. | 15:01 |
| dtantsur | please do, though I don't think we actually put 2 images to glance.. | 15:01 |
| thrash | trown: i think so, since they have the same name. | 15:01 |
| gfidente | dprince, for example, cli args don't work anymore because they stick to the value initially passed via parameters: | 15:01 |
| dtantsur | thrash, however, the bugzilla is definitely NOTABUG. even with your potential fix we must not use 7.2 deploy images (IPA or not) for 8.0 | 15:02 |
| gfidente | so I think it's worth landing that in liberty | 15:02 |
| thrash | dtantsur: ok. But there is *a* bug there. :) | 15:02 |
| dtantsur | maybe :) lets see how your patch shapes | 15:02 |
| dprince | gfidente: in those cases for upgrades shouldn't we be passing in new parameter values then? | 15:03 |
| dprince | gfidente: is there a specific bug you can link that better explains this? | 15:03 |
| dprince | gfidente: the patch seems a bit evil I think | 15:03 |
| *** miles is now known as mgould | 15:03 | |
| gfidente | dprince, yeah that's exactly the problem, we do pass new values but the new client passes them as parameter_defaults, so the values don't override what *was* passed as parameters: | 15:03 |
| gfidente | d0ugal, ^^ do you know if we had a LP bug for this? | 15:04 |
| shardy | dprince: we've moved towards using parameter_defaults for everything, as parameters only work when wired in to the top-level template | 15:04 |
| shardy | the side effect of this, combined with PATCH updates means the old parameters are sticky in a really opaque way | 15:05 |
| dprince | Yeah, I get that. I'm just worried there are going to be other side effects from this | 15:05 |
| slagle | derekh: just continuing to talk about testenv's...i think cpu is also killing us | 15:05 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Make upload with --old-deploy-image only look for the old image https://review.openstack.org/289946 | 15:06 |
| derekh | slagle: Ya, its not low, but there are 24 threads so, not as bad as that 15 load would suggest, installing dstat on one of the machines now to get some more info | 15:06 |
| thrash | dtantsur: trown ^^^ | 15:06 |
| thrash | dtantsur: trown I think you will see it's subtle, but will make more sense. | 15:06 |
| derekh | *15 minute load | 15:06 |
| slagle | derekh: yep | 15:07 |
| slagle | derekh: we also hitting the NodeAssociated error a lot in CI | 15:07 |
| slagle | which is the nova/ironic scheduling race | 15:07 |
| slagle | which compounds the cpu usage problem | 15:08 |
| derekh | slagle: ok, is that something within our ability to fix? or something nova/ironic need to sort out? | 15:09 |
| slagle | derekh: not sure...i'm looking up the patch | 15:09 |
| slagle | we had a downstream fix for this, that mitigated it somewhat | 15:09 |
| slagle | i'm not sure that was accepted to Nova | 15:09 |
| dtantsur | thrash, sorry, but no :) actually I don't see 2 images uploaded, so I don't think anything requires fixing | 15:09 |
| thrash | dtantsur: ok. Fair enough. | 15:10 |
| slagle | derekh: yea, it's merged in Nova...https://review.openstack.org/#/c/226235/ | 15:11 |
| derekh | slagle: hmm, so we got a seperate problem | 15:12 |
| slagle | one of the nodes took 20 attempts to schedule :( | 15:12 |
| derekh | ouch | 15:13 |
| slagle | ah ya know what though, https://review.openstack.org/#/c/226235/ is not in liberty nova | 15:13 |
| derekh | slagle: we're inly including the exact number og nodes needed in our nodes.json file, would it make it less likely to happen if we increase that ? | 15:14 |
| trown | derekh: slagle, ya I hit this in RDO quite a bit too | 15:14 |
| slagle | dang | 15:14 |
| slagle | that is killing us! | 15:14 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-common: Properly setup DNS for the container CI job https://review.openstack.org/289966 | 15:14 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 15:14 |
| bnemec | I don't understand how a node could fail scheduling 20 times. | 15:14 |
| bnemec | There aren't 20 nodes, and the retry filter would stop it after 4 at most. | 15:14 |
| dtantsur | bnemec, our retry limit is 30 nowadays | 15:15 |
| slagle | bnemec: because ironicclient has it's own retries | 15:15 |
| dtantsur | (and this too) | 15:15 |
| derekh | slagle: i.e. increase this on line 8 http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/deploy.sh#n8 | 15:15 |
| slagle | derekh: that might help | 15:16 |
| slagle | or we could set up targeted node deployemnt | 15:16 |
| slagle | tag each node controller-1, controller-2, etc | 15:16 |
| bnemec | We need to start testing that anyway. | 15:16 |
| bnemec | Although it doesn't actually work yet. | 15:16 |
| derekh | sounds like it would work | 15:16 |
| derekh | or not | 15:16 |
| slagle | lol | 15:16 |
| bnemec | There are two patches required for predictable placement. | 15:17 |
| bnemec | https://review.openstack.org/288087 | 15:17 |
| bnemec | https://review.openstack.org/288188 | 15:17 |
| bnemec | With those I can tag nodes and force deployment to them. | 15:18 |
| dtantsur | or use profile matching :) | 15:18 |
| * dtantsur hides | 15:18 | |
| trown | bnemec: I use node tagging now | 15:18 |
| trown | I must be missing some nuance here | 15:18 |
| bnemec | dtantsur: That still leaves you three compute nodes to pick from in the ha job. | 15:18 |
| trown | ah, now I see | 15:19 |
| dtantsur | yeah, ha job.. | 15:19 |
| slagle | what's less work, these solutions, or rm'ing nova? :) | 15:20 |
| slagle | i jest | 15:20 |
| trown | if only | 15:20 |
| * dtantsur types 'rm' and gets ready | 15:20 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Forcibly clear parameters now passed as parameter_defaults https://review.openstack.org/256670 | 15:20 |
| slagle | dtantsur: the problem is mitigated somewhat if we have more than the minimum # of nodes available right? | 15:21 |
| dtantsur | slagle, yes. the more nodes to choose from, the less chance to pick the same for several instances | 15:22 |
| slagle | derekh: so if we wrapped http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/deploy.sh#n8 in a "if INTROSPECT=1", we'd have some mitigation | 15:22 |
| slagle | worth trying, i can push it up | 15:22 |
| *** eggmaste` has quit IRC | 15:23 | |
| derekh | slagle: yup, lets give it a shot | 15:23 |
| *** dmacpher has joined #tripleo | 15:23 | |
| derekh | slagle: and line 9 | 15:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: deploy glance using puppet-tripleo manifests https://review.openstack.org/289466 | 15:26 |
| EmilienM | shardy: commit message updated^ | 15:26 |
| openstackgerrit | Imre Farkas proposed openstack/python-tripleoclient: Update baremetal ready state command https://review.openstack.org/289971 | 15:26 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/289757 | 15:27 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Only reduce the node vm's when using introspection https://review.openstack.org/289973 | 15:27 |
| *** oshvartz has quit IRC | 15:29 | |
| jaosorior | bnemec: For the configurable ports change in the loadbalancer manifest; does having a map like this https://review.openstack.org/#/c/287199/7/manifests/loadbalancer.pp address what you were thinking? | 15:29 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 15:31 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Permits configuration of Cinder enabled_backend via hieradata https://review.openstack.org/289979 | 15:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Permits configuration of Cinder enabled_backend via hieradata https://review.openstack.org/269534 | 15:34 |
| jaosorior | By the way, does anyone have time to review this? https://review.openstack.org/#/c/287465/1 it's needed for ssl to work again | 15:35 |
| shardy | jaosorior: does keystone have duplicate endpoints for v3 and non-v3? | 15:37 |
| jaosorior | shardy: What do you mean? | 15:37 |
| shardy | jaosorior: I thought there was a bunch of keystoneclient version discovery code to avoid that? | 15:37 |
| shardy | https://review.openstack.org/#/c/287465/1/environments/enable-tls.yaml | 15:38 |
| shardy | KeystoneV3Admin etc | 15:38 |
| bnemec | shardy: Version discovery wasn't working right in (I think) Nova, so the v3 endpoint was added. | 15:38 |
| shardy | I don't get why | 15:38 |
| jaosorior | shardy: Yeah... well.. the discovery seems to be failing | 15:38 |
| shardy | gah | 15:38 |
| jaosorior | shardy: it started failing with a change introduced in nova | 15:38 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 15:38 |
| bnemec | Yeah, it looks like there are still issues with the new keystoneauth1 library. | 15:39 |
| shardy | per-version endpoints are horrible :( | 15:39 |
| *** aufi_ has quit IRC | 15:39 | |
| jaosorior | shardy: I actually highly suspect this specific CR https://review.openstack.org/#/c/253793/ | 15:39 |
| jaosorior | but I haven't been able to properly debug and check what's up | 15:39 |
| shardy | Ok, I guess maybe worth adding a FIXME referencing the keystone/nova bug if we're going to paper over the bug like this | 15:39 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add a ceph-storage node upgrade script for the upgrade workflow https://review.openstack.org/289896 | 15:39 |
| shardy | I thought keystone was pushing very much away from versioned endpoints | 15:40 |
| jaosorior | shardy: They are | 15:40 |
| bnemec | Yeah, I initially pushed back on the change because I knew it wasn't how things are supposed to work, but I didn't have time to track down the actual problem. | 15:40 |
| *** aufi has joined #tripleo | 15:40 | |
| gfidente | marios, are you guys actually testing that ceph change? | 15:41 |
| EmilienM | :q! | 15:42 |
| EmilienM | oops | 15:42 |
| openstackgerrit | Moshe Levi proposed openstack/diskimage-builder: Add lshw package to ironic-agent https://review.openstack.org/289233 | 15:43 |
| marios | gfidente: eventually yes | 15:43 |
| gfidente | cause you put my name in there | 15:44 |
| gfidente | I was worried | 15:44 |
| gfidente | ahahaha | 15:44 |
| marios | lol | 15:44 |
| *** xinwu has joined #tripleo | 15:44 | |
| gfidente | so the thing is that ceph always tries to keep 3 copies of the data in our default config | 15:45 |
| gfidente | if one of the osds goes down it will create a new mirror thinking that one of the existing failed | 15:45 |
| gfidente | while we're only doing 'maintenance' so we don't want data to be copied around | 15:45 |
| gfidente | the node will supposedly come back | 15:45 |
| gfidente | but from theory to practice ... | 15:46 |
| *** pblaho has quit IRC | 15:46 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add a ceph-storage node upgrade script for the upgrade workflow https://review.openstack.org/289896 | 15:47 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Update enable-tls.yaml with new endpoints https://review.openstack.org/287465 | 15:49 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 15:50 |
| marios | gfidente: makes sense and thanks for that also just fixed up a nit v3 | 15:51 |
| *** links has joined #tripleo | 15:52 | |
| jaosorior | Can anyone check this out? https://review.openstack.org/#/c/287199/ It enables the openstack service ports to be configurable to fix a BZ | 15:53 |
| dmsimard | can't find designate in tripleo at first glance, is it ? | 15:54 |
| jaosorior | dmismard: It's not in tripleo | 15:55 |
| dmsimard | jaosorior: thanks, was just sanity checking :) | 15:56 |
| *** aufi has quit IRC | 15:58 | |
| *** mbound has quit IRC | 15:58 | |
| bnemec | jaosorior: Sorry, that is an improvement. It would still be nice if we documented the available keys in the parameter comments at the top. | 16:02 |
| bnemec | It's much better than when those params were buried throughout the file though. | 16:02 |
| jaosorior | uhm... alright, so you suggest adding the existing defaults to the docstring above? | 16:03 |
| *** mbound has joined #tripleo | 16:05 | |
| jaosorior | bnemec: How about this? http://paste.openstack.org/show/489701/ | 16:08 |
| *** rwsu has quit IRC | 16:09 | |
| bnemec | jaosorior: Yep, perfect. | 16:09 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 16:10 |
| jaosorior | bnemec: Done | 16:10 |
| *** aufi has joined #tripleo | 16:10 | |
| *** tzumainn has quit IRC | 16:14 | |
| *** leanderthal is now known as leanderthal|afk | 16:16 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common: Remove tripleo.sh (now in tripleo-ci repo) https://review.openstack.org/290002 | 16:17 |
| *** pblaho has joined #tripleo | 16:19 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:19 |
| *** olap has quit IRC | 16:19 | |
| *** snecklifter has joined #tripleo | 16:20 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:20 |
| *** jaosorior has quit IRC | 16:21 | |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add 'undercloud upgrade' command https://review.openstack.org/289498 | 16:23 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:23 |
| snecklifter | Hello, networking deployment is failing on controller nodes, is there a user/pass for console access? | 16:23 |
| pradk | jistr, anything else you need me to do for aodh upgrades https://review.openstack.org/#/c/289435/ ? | 16:26 |
| pradk | just need some reviews on that ^^ | 16:27 |
| *** david-lyle_ is now known as david-lyle | 16:28 | |
| *** pblaho has quit IRC | 16:29 | |
| *** absubram has joined #tripleo | 16:29 | |
| *** pblaho has joined #tripleo | 16:31 | |
| *** dmsimard has quit IRC | 16:31 | |
| openstackgerrit | Merged openstack/tripleo-docs: Updated documentation for: tripleo.sh --overcloud-pingtest https://review.openstack.org/289853 | 16:32 |
| jistr | pradk: i would kind of prefer if changes going in were manually tested on a real upgrade scenario (at least when there's a reason to suspect they might affect upgrades), but we probably shouldn't block patches on that ground right now. I just realized we already have other things in Mitaka that make a correct Liberty->Mitaka upgrade not possible right now. For landing in Mitaka only, not fully finished Liberty->Mitaka upgrade support is | 16:37 |
| jistr | probably ok ATM, despite it makes my heart cry, because someone will eventually have to go and stitch all pieces together and fix/fill any omissions :)) | 16:37 |
| *** saneax is now known as saneax_AFK | 16:38 | |
| jistr | pradk: but what you have there looks good to me re upgrades (from visual check only), if we assume aodh is going to be spun up by puppet and not the upgrade script itself | 16:40 |
| snecklifter | To put it another way, is it possible to log into a tripleo node from console? | 16:40 |
| pradk | jistr, right, i think it will be quite a bit of work to standup aodh with bash during upgrades | 16:41 |
| jistr | pradk: yeah, probably not the best endeavor to go into unless all other options have been crossed off | 16:41 |
| pradk | i would think if someone is upgrading their cloud, they will put it in maintenance window of some sort which should be ok for aodh to get in within that window | 16:42 |
| jistr | pradk: when i have some time to breathe, i'd like to consider if running puppet only on controllers (or controllers and cinder nodes) as part of the controller upgrade won't break things | 16:43 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change the CinderISCSIHelper to lioadm https://review.openstack.org/290010 | 16:44 |
| jistr | pradk: so that might get us to a situation where there wouldn't be a big time gap between running the upgrade script and running puppet on conrollers | 16:44 |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Forcibly clear parameters now passed as parameter_defaults https://review.openstack.org/256670 | 16:44 |
| pradk | jistr, cool, that should help but i guess that wont be liberty | 16:44 |
| jistr | yea | 16:44 |
| derekh | snecklifter: not by default, but you can build your image with an element that sets a password for a user | 16:45 |
| derekh | snecklifter: We used to include this element to add a user http://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/stackuser | 16:45 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change the CinderISCSIHelper to lioadm https://review.openstack.org/283712 | 16:45 |
| *** xinwu has quit IRC | 16:45 | |
| snecklifter | derekh, thanks for responding, much appreciated | 16:46 |
| derekh | snecklifter: no prob | 16:46 |
| snecklifter | derekh++ | 16:46 |
| *** mbound has quit IRC | 16:46 | |
| *** Slower has joined #tripleo | 16:46 | |
| *** mikelk has quit IRC | 16:48 | |
| *** yamahata has quit IRC | 16:48 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change the CinderISCSIHelper to lioadm https://review.openstack.org/283712 | 16:48 |
| *** paramite has quit IRC | 16:48 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Forcibly clear parameters now passed as parameter_defaults https://review.openstack.org/256670 | 16:49 |
| gfidente | I did enough mistaked for the day | 16:49 |
| *** rdopiera has quit IRC | 16:53 | |
| *** sshnaidm has quit IRC | 16:53 | |
| *** jprovazn has joined #tripleo | 16:56 | |
| *** lazy_prince has quit IRC | 16:57 | |
| *** rpothier has left #tripleo | 17:00 | |
| jpeeler | thrash: think you'll be able to update your tests today for this review? https://review.openstack.org/#/c/235569/ | 17:02 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job https://review.openstack.org/289445 | 17:03 |
| *** xinwu has joined #tripleo | 17:03 | |
| thrash | jpeeler: yes. working on it. | 17:05 |
| jpeeler | ok sorry to bother! just trying to get my review (dependant on yours) in | 17:05 |
| thrash | i know. | 17:06 |
| thrash | I'm having some issues with the elements, but I don't think it's related to the actual code. | 17:06 |
| *** shakamunyi has joined #tripleo | 17:06 | |
| *** aufi has quit IRC | 17:07 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 17:10 |
| * jpeeler crosses his fingers | 17:10 | |
| *** yamahata has joined #tripleo | 17:12 | |
| *** fgimenez has quit IRC | 17:13 | |
| *** pradk_ has quit IRC | 17:13 | |
| *** cmyster_ has joined #tripleo | 17:13 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: [WIP] Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 17:14 |
| *** admin0 has quit IRC | 17:14 | |
| *** cmyster has quit IRC | 17:14 | |
| *** hjensas has joined #tripleo | 17:18 | |
| *** hjensas has joined #tripleo | 17:18 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 17:21 |
| thrash | jpeeler: requirements check failed. :( | 17:22 |
| *** shakamunyi has quit IRC | 17:23 | |
| jpeeler | yeah, well looks like it was an easy fix | 17:23 |
| thrash | yep | 17:23 |
| * jpeeler double crosses his fingers | 17:23 | |
| jpeeler | i'm assuming at this point that once the tests pass, the review will soon get merged | 17:23 |
| jpeeler | i honestly haven't even been tracking the changes, so i'll likely have some changes to make. has the patch significantly diverged from the original code? | 17:24 |
| *** d0ugal has quit IRC | 17:26 | |
| *** dcain has joined #tripleo | 17:26 | |
| *** ohamada has quit IRC | 17:27 | |
| *** dtantsur is now known as dtantsur|afk | 17:27 | |
| *** mkovacik has quit IRC | 17:30 | |
| *** rcernin has quit IRC | 17:34 | |
| *** ifarkas has quit IRC | 17:36 | |
| *** shakamunyi has joined #tripleo | 17:37 | |
| *** panda has quit IRC | 17:39 | |
| *** panda has joined #tripleo | 17:40 | |
| links | thanks gfidente, shardy https://review.openstack.org/#/c/241606/11/environments/ips-from-pool.yaml worked . | 17:40 |
| gfidente | links, yay :) | 17:40 |
| shardy | \o/ | 17:41 |
| links | :) | 17:41 |
| *** xinwu has quit IRC | 17:41 | |
| links | Now i wished to map each IP specific to %index% , but so far tests have failed . | 17:42 |
| *** jistr has quit IRC | 17:43 | |
| links | presently checking with rh engg internally on bz . | 17:43 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 17:44 |
| shardy | links: each list is accessed by %index% | 17:45 |
| shardy | so e.g the first entry is always for overcloud-controller-0 etc | 17:45 |
| *** lucasagomes is now known as lucas-dinner | 17:46 | |
| links | shardy, yes , i noticed that . but would that order change if nova fails to start & if instance is respawned ? | 17:47 |
| shardy | links: no, it's mapped to the instance resource inside heat | 17:47 |
| shardy | so any scheduler retires are transparent | 17:48 |
| shardy | if you remove a node from the cluster however, you would need to adjust the list | 17:48 |
| shardy | e.g if controller-1 fails and you exclude it by building a controller-3 | 17:48 |
| links | okay , thanks . let me test such scenarios | 17:49 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 17:50 |
| *** derekh has quit IRC | 17:58 | |
| bnemec | slagle: https://review.openstack.org/#/c/288866 has passed CI. Do you want to merge it and discuss setting it on the controller as a followup? | 17:59 |
| slagle | bnemec: yes | 18:01 |
| *** mannidi has joined #tripleo | 18:02 | |
| *** trozet has quit IRC | 18:03 | |
| *** athomas has quit IRC | 18:06 | |
| *** admin0 has joined #tripleo | 18:07 | |
| *** shivrao has joined #tripleo | 18:08 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Set host in nova.conf for compute nodes https://review.openstack.org/288866 | 18:14 |
| *** sshnaidm has joined #tripleo | 18:15 | |
| *** tosky has quit IRC | 18:15 | |
| *** admin0 has quit IRC | 18:16 | |
| *** dprince has quit IRC | 18:21 | |
| *** dprince has joined #tripleo | 18:22 | |
| *** bswartz has quit IRC | 18:23 | |
| *** links has quit IRC | 18:23 | |
| *** admin0 has joined #tripleo | 18:24 | |
| *** bswartz has joined #tripleo | 18:24 | |
| *** gfidente has quit IRC | 18:24 | |
| *** admin0 has quit IRC | 18:28 | |
| *** mannidi has quit IRC | 18:39 | |
| openstackgerrit | Miles Gould proposed openstack/python-tripleoclient: [WIP] Use Ironic API v1.11 to support ENROLL state https://review.openstack.org/272206 | 18:43 |
| *** mgould has quit IRC | 18:44 | |
| bnemec | Ouch: http://paste.openstack.org/show/489717/ | 18:47 |
| bnemec | Even having Ironic provision swap may not be helping. | 18:47 |
| bnemec | 8.7 MB/s is pretty terrible throughput. | 18:47 |
| *** dmacpher is now known as dmacpher-afk | 18:49 | |
| *** trozet has joined #tripleo | 18:50 | |
| *** dmsimard has joined #tripleo | 18:50 | |
| *** trown has quit IRC | 18:51 | |
| *** trown has joined #tripleo | 18:52 | |
| *** xinwu has joined #tripleo | 18:54 | |
| bnemec | Man, it looks like we are destroying CI with the swap provisioning right now. How would people feel about pushing through the changes for https://review.openstack.org/#/c/289085/ ? | 18:56 |
| bnemec | https://review.openstack.org/#/c/289610/ in particular isn't even used in CI, so waiting on results is not accomplishing much. | 18:56 |
| *** jcoufal has quit IRC | 18:56 | |
| slagle | bnemec: is the partition that much faster than the file? | 19:02 |
| bnemec | slagle: I don't know, but we need to change _something_. I'm seeing a ton of gate timeouts and provisioning errors that appear to be related to overloaded IO on the CI nodes. | 19:03 |
| bnemec | Alternatively, we merge the change to shrink to 1 GB of RAM. | 19:03 |
| bnemec | We should maybe do that anyway. | 19:03 |
| bnemec | If we ever get 4 GB into swap we're pretty much hosed anyway. | 19:04 |
| bnemec | *Alternatively, we merge the change to shrink to 1 GB of _swap_ | 19:04 |
| *** Marga_ has quit IRC | 19:05 | |
| slagle | bnemec: how about we switch the swap partition patch to use 1GB of swap, and then just merge that | 19:05 |
| slagle | everyting is basically dead now anyway | 19:05 |
| bnemec | slagle: I'm good with that. | 19:05 |
| bnemec | Yeah, I don't think any HA job is going to complete right now. | 19:05 |
| bnemec | Or in the foreseeable future, looking at the zuul queue. | 19:06 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 19:07 |
| slagle | bnemec: so merge that ^^ and https://review.openstack.org/#/c/289610/ | 19:08 |
| bnemec | slagle: Yeah. I think we should just merge https://review.openstack.org/#/c/289610/ . CI isn't testing anything on it anyway, and it did pass one job so it didn't somehow completely break the repo. | 19:09 |
| slagle | i merged it | 19:09 |
| bnemec | I see you beat me to it. :-) | 19:10 |
| bnemec | slagle: Debating whether to just merge https://review.openstack.org/#/c/289085 too. PS 3 passed CI already, and it's going to be 8+ hours before we get _any_ results back on PS 4. | 19:13 |
| *** nico_auv has quit IRC | 19:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add an environment to use a swap partition https://review.openstack.org/289610 | 19:13 |
| bnemec | My only real concern is if 1 GB of swap isn't enough to fix the OOMs. | 19:13 |
| slagle | bnemec: this passed 8 out of the last 9 times, https://review.openstack.org/#/c/288827/ | 19:15 |
| *** electrofelix has quit IRC | 19:15 | |
| bnemec | slagle: Ah, perfect. | 19:15 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 19:22 |
| bnemec | Now if we could just clear out all of the jobs and re-run them. | 19:22 |
| *** dprince has quit IRC | 19:23 | |
| *** cmyster_ has quit IRC | 19:23 | |
| slagle | we could start killing stuff :) | 19:26 |
| slagle | that wouldn't be very nice | 19:26 |
| *** dprince has joined #tripleo | 19:27 | |
| trown | if they have very little chance to pass anyway... | 19:27 |
| *** Marga_ has joined #tripleo | 19:28 | |
| trown | it is only slightly naughty | 19:28 |
| bnemec | That's the thing. I don't think the ha jobs are going to pass until we stop hammering the host disk. The deploy is just too slow, even if everything else works fine. | 19:30 |
| bnemec | Oh well, maybe by tomorrow jobs will be passing again. :-) | 19:30 |
| slagle | i dont even think it's just the host disk i/o | 19:31 |
| slagle | the cpu load is also very high | 19:31 |
| *** shardy has quit IRC | 19:33 | |
| slagle | there is still a queue, but new stuff is entering | 19:39 |
| slagle | so i'll leave it | 19:39 |
| slagle | hopefully it will start to resolve | 19:39 |
| slagle | https://review.openstack.org/#/c/289279/ entered after we merged the ci patches for instance | 19:40 |
| slagle | bnemec: this ought to help as well: https://review.openstack.org/#/c/289973/ | 19:42 |
| *** akrivoka has quit IRC | 19:44 | |
| *** admin0 has joined #tripleo | 19:53 | |
| *** ccamacho has quit IRC | 20:07 | |
| *** leanderthal|afk has quit IRC | 20:19 | |
| *** yamahata has quit IRC | 20:22 | |
| *** dcain1 has joined #tripleo | 20:32 | |
| *** Goneri has quit IRC | 20:33 | |
| *** dcain has quit IRC | 20:34 | |
| *** admin0 has quit IRC | 20:37 | |
| *** rbrady is now known as rbrady-run | 20:37 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:39 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 20:41 |
| *** dcain1 has quit IRC | 20:45 | |
| *** dcain has joined #tripleo | 20:46 | |
| *** shardy has joined #tripleo | 20:48 | |
| jdob | is there a trick to debugging failures in ControllerAllNodesValidationDeployment? i don't see anything in journalctl | 20:50 |
| *** admin0 has joined #tripleo | 20:53 | |
| *** admin0 has quit IRC | 20:57 | |
| slagle | jdob: where are you seeing that? | 20:59 |
| slagle | those should be fixed if you're seeing it in tripleo-ci | 20:59 |
| jdob | local env | 20:59 |
| slagle | jdob: did it fail pinging some ip's? | 20:59 |
| jdob | which means it's possible I don't have those fixes; were they ci fixes or code patches? | 20:59 |
| jdob | i dunno, that's the problem, I can't see anything besides failing with exit code 1 | 21:00 |
| slagle | jdob: even in the stdout/stderr from the deployment? | 21:00 |
| slagle | this was the fix anyway, https://review.openstack.org/#/c/288747/ | 21:00 |
| slagle | i only ever saw it in ci though. it was racey | 21:01 |
| jdob | ya, I have that fix locally | 21:02 |
| jdob | hrm | 21:02 |
| jdob | even in the stdout all I see is 2016-03-08 20:21:59 [overcloud-ComputeAllNodesValidationDeployment-h2c7o5opencw]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 | 21:03 |
| slagle | what about the deployment-show output? | 21:04 |
| trown | jdob `for failed_deployment in $(heat resource-list --nested-depth 5 overcloud | grep FAILED | grep 'StructuredDeployment ' | cut -d '|' -f3); do heat deployment-show $failed_deployment; done;` | 21:04 |
| trown | I think that shows which host is not pingable... | 21:04 |
| trown | maybe | 21:04 |
| *** akuznetsov has joined #tripleo | 21:05 | |
| *** akuznetsov has quit IRC | 21:05 | |
| jdob | trown: doesn't matter in this case, I don't htink pinging is the issue: "deploy_stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\nImportError: No module named ipaddr\n", | 21:05 |
| trown | ah right | 21:05 |
| trown | I have hit that | 21:05 |
| slagle | that's been fixed too, your images are too old | 21:05 |
| *** xinwu has quit IRC | 21:06 | |
| slagle | https://review.openstack.org/#/c/285858/ | 21:06 |
| *** jprovazn has quit IRC | 21:06 | |
| jdob | ok, cool, I'll double check that I applied the image patch right (spoiler: I think I screwed it up) | 21:07 |
| jdob | thanks trown and slagle | 21:07 |
| trown | np | 21:08 |
| *** jtomasek has quit IRC | 21:20 | |
| *** jayg is now known as jayg|g0n3 | 21:22 | |
| *** dprince has quit IRC | 21:24 | |
| *** lblanchard has quit IRC | 21:38 | |
| *** panda has quit IRC | 21:39 | |
| *** panda has joined #tripleo | 21:40 | |
| *** dcain has quit IRC | 21:40 | |
| *** dcain has joined #tripleo | 21:43 | |
| bnemec | Whoa, I just got two review emails in a row with CI passes. | 21:44 |
| bnemec | Our evil plan is working!!! :-) | 21:44 |
| bkero | Or your CI is broken-passing :) | 21:44 |
| trown | lol, bkero dont be a debbie downer :p | 21:45 |
| bkero | Yeah yeah | 21:45 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/287561 | 21:46 |
| slagle | bnemec: 2h45m :) | 21:46 |
| slagle | barely squeaked by on that one | 21:46 |
| bnemec | Yeah, no kidding. | 21:46 |
| bnemec | It's a start though | 21:47 |
| *** r-mibu has quit IRC | 21:47 | |
| *** r-mibu has joined #tripleo | 21:47 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Store events in Ceilometer https://review.openstack.org/290153 | 21:48 |
| * bnemec can't wait until we can stop backporting _everything_ | 21:48 | |
| slagle | bnemec: we gotta merge that pipefail fix | 21:52 |
| bnemec | slagle: Is that messing something else up? | 21:52 |
| slagle | it's just annoying me | 21:52 |
| bnemec | I mostly needed it because otherwise the undercloud idempotency check would never fail. | 21:52 |
| bnemec | Ah, good enough for me. :-) | 21:52 |
| *** xinwu has joined #tripleo | 21:54 | |
| slagle | i see you rechecked it, so i'll be patient | 21:56 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 21:57 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 22:01 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/289435 | 22:04 |
| *** admin0 has joined #tripleo | 22:08 | |
| *** rbrady-run is now known as rbrady | 22:09 | |
| *** shardy has quit IRC | 22:11 | |
| *** weshay has quit IRC | 22:21 | |
| *** dustins has quit IRC | 22:23 | |
| *** trown is now known as trown|outtypewww | 22:24 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 22:26 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 22:30 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Only reduce the node vm's when using introspection https://review.openstack.org/289973 | 22:31 |
| slagle | another pass with flying colors | 22:32 |
| bnemec | \o/ | 22:32 |
| bnemec | I have 7 of the 19 running jobs in CI. | 22:32 |
| bnemec | See "evil plan" above. ;-) | 22:33 |
| *** admin0 has quit IRC | 22:33 | |
| *** rbrady has quit IRC | 22:36 | |
| *** dcain has left #tripleo | 22:39 | |
| *** morazi has quit IRC | 22:50 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 22:50 |
| bnemec | \o/ just knocked a couple more minutes off our CI runs | 22:51 |
| *** saneax_AFK is now known as saneax | 22:56 | |
| *** rbrady has joined #tripleo | 23:12 | |
| *** yamahata has joined #tripleo | 23:14 | |
| *** trozet has quit IRC | 23:18 | |
| *** thrash is now known as thrash|g0ne | 23:33 | |
| *** absubram has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
