| *** darrenc is now known as darrenc_afk | 00:12 | |
| *** shoutm has quit IRC | 00:19 | |
| *** shoutm has joined #openstack-ansible | 00:23 | |
| *** darrenc_afk is now known as darrenc | 00:29 | |
| *** tlian2 has quit IRC | 04:05 | |
| *** shausy has joined #openstack-ansible | 04:53 | |
| *** markvoelker has joined #openstack-ansible | 05:06 | |
| *** markvoelker has quit IRC | 05:11 | |
| *** markvoelker has joined #openstack-ansible | 05:12 | |
| *** cloudtrainme has joined #openstack-ansible | 05:13 | |
| *** shausy has quit IRC | 05:36 | |
| *** shausy has joined #openstack-ansible | 05:36 | |
| *** markvoelker has quit IRC | 06:23 | |
| *** sdake has joined #openstack-ansible | 06:24 | |
| *** cloudtrainme has quit IRC | 07:08 | |
| *** cloudtrainme has joined #openstack-ansible | 07:08 | |
| *** gparaskevas has joined #openstack-ansible | 07:17 | |
| *** javeriak has joined #openstack-ansible | 07:54 | |
| *** javeriak_ has joined #openstack-ansible | 07:56 | |
| evrardjp | good morning everyone | 07:59 |
|---|---|---|
| *** javeriak has quit IRC | 07:59 | |
| gparaskevas | gm! | 08:04 |
| mattt | howzit guys | 08:04 |
| evrardjp | is it possible to have storage hosts that have container_vars to set them in an availability zone, but limits the cinder backends to some availability zone? | 08:06 |
| evrardjp | for example, I'd like to have 2 storage hosts, one per AZ (let's call them main DC <-> backup DC) | 08:06 |
| evrardjp | and on the first host in mainDC, I'd have 2 backends: one with a replicated ceph (so available on both AZ), the other with local netapp storage (that stays is only available in the AZ mainDC) | 08:07 |
| evrardjp | the second host in backupDC would have 2 backends also: one with the replicated ceph (same as the first one), the other with local netapp storage (that is only available in the AZ backupdc) | 08:09 |
| evrardjp | is that even possible here? | 08:09 |
| *** javeriak_ has quit IRC | 08:13 | |
| evrardjp | (and if possible, in the horizon dashboard have one of the options auto-selected, it's really a pain always change volume_type and availability_zone to make it work | 08:17 |
| odyssey4me | morning all | 08:18 |
| odyssey4me | mattt I'm going to need to pull https://review.openstack.org/219364 into https://review.openstack.org/218572 to unblock the gate - can you then rebase and remove the pin in https://review.openstack.org/215584 as the netaddr bug is addressed by the newer sha in that patch | 08:19 |
| *** javeriak has joined #openstack-ansible | 08:19 | |
| mattt | odyssey4me: i'm not pinning neutron to overcome any gate-related issues | 08:20 |
| mattt | odyssey4me: i did it because it introduces the new --expand / --contract args | 08:20 |
| odyssey4me | mattt my patch will be pinning, so I'm asking you to remove the pin once you've rebased | 08:20 |
| odyssey4me | The choice we have is either to merge my patch and yours (frankenpatch, ugh) - or to add a temporary pin to my patch (which I prefer) | 08:21 |
| mattt | ok sure | 08:21 |
| mattt | why do i need to rebase tho? | 08:22 |
| mattt | i'll just remove the pin | 08:22 |
| mattt | n/m coffee not kicked in yet | 08:22 |
| odyssey4me | mattt if you don't mind, I'll revise the patch with what I mean :) it won't materially change your patch | 08:23 |
| mattt | odyssey4me: hands off | 08:23 |
| odyssey4me | your Neutron sha update is good | 08:23 |
| mattt | you get your grubby hands out of my patch | 08:23 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Wait for container ssh after apparmor profile update https://review.openstack.org/218572 | 08:24 |
| *** javeriak has quit IRC | 08:24 | |
| mattt | odyssey4me: go ahead and do what you need to do :) | 08:24 |
| odyssey4me | lol, ok - then look at the above patch - see the requirements.txt change - that needs to be reverted in your patch | 08:24 |
| mattt | odyssey4me: i'm kidding :) just let me know if you want me to do it and i'll do it, otherwise assuming you're on it | 08:24 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Update how neutron migrations are handled https://review.openstack.org/215584 | 08:26 |
| odyssey4me | mattt I'll do it quickly | 08:26 |
| *** javeriak has joined #openstack-ansible | 08:27 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Update how neutron migrations are handled https://review.openstack.org/215584 | 08:28 |
| odyssey4me | mattt done - now we wait to see if any other surprises have crawled in to cause CI failure in the last 24 hours, otherwise we mergerate and unblock master | 08:30 |
| mattt | odyssey4me: sounds good ... i have more changes to make to that review, but i'm going to let that one go through and create a subsequent patch for additional work | 08:30 |
| mattt | git-harry spotted some issues w/ it late yesterday afternoon | 08:31 |
| odyssey4me | mattt well, you can continue to revise the review if you like - I just wanted to make sure we incorporate the requirements removal as we bump that sha | 08:31 |
| mattt | odyssey4me: but if we are experiencing db syncing issues holding up the gate (even if periodically), i'd say get that guy through | 08:31 |
| odyssey4me | fair enough :) it'll definitely help improve the rate of success | 08:32 |
| *** javeriak has quit IRC | 08:33 | |
| *** javeriak has joined #openstack-ansible | 08:33 | |
| *** shoutm has quit IRC | 08:36 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Change AppArmor profile application order https://review.openstack.org/217014 | 08:41 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Updated kilo to include Neutron netaddr fix - 2 Sep 2015 https://review.openstack.org/219605 | 08:52 |
| odyssey4me | hughsaunders I need some help getting to the bottom of why http://logs.openstack.org/98/217098/3/check/gate-os-ansible-deployment-dsvm-commit/5032990/console.html#_2015-09-01_15_37_49_545 is the resulting tempest output in https://review.openstack.org/217098 | 08:53 |
| odyssey4me | we need to get the juno update out the door | 08:53 |
| hughsaunders | odyssey4me: ok | 08:53 |
| odyssey4me | we were working through how to get into the venv yesterday | 08:54 |
| hughsaunders | yep | 08:54 |
| *** markvoelker has joined #openstack-ansible | 08:57 | |
| odyssey4me | how was that done again? | 08:57 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Disable scatter-gather offload on host bridges https://review.openstack.org/219292 | 09:00 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Disable python buffering for gate checks https://review.openstack.org/218595 | 09:02 |
| evrardjp | osad.readthedocs.org/en/latest doesn't work anymore? | 09:02 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adds the ability to provide user certificates to HAProxy https://review.openstack.org/215525 | 09:02 |
| *** markvoelker has quit IRC | 09:02 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Updated MariaDB to the new release version (10.0) https://review.openstack.org/178259 | 09:02 |
| odyssey4me | evrardjp odd, but http://os-ansible-deployment.readthedocs.org/en/latest/ does | 09:06 |
| evrardjp | this isn't the one documented on the launchpad page ;) (also not the one in my browser history :p) | 09:06 |
| evrardjp | I used a good old make html, so it's not really for me | 09:07 |
| hughsaunders | odyssey4me: cd /opt/tempest_*; . bin/activate | 09:09 |
| odyssey4me | evrardjp fixed the wiki reference, thanks :) | 09:10 |
| *** shoutm has joined #openstack-ansible | 09:11 | |
| odyssey4me | hughsaunders ah, thanks - now I have the issue replicated | 09:12 |
| hughsaunders | odyssey4me: I just created a venv from the package list here: https://github.com/stackforge/os-ansible-deployment/blob/juno/rpc_deployment/roles/tempest/defaults/main.yml#L65-L93 using http://rpc-repo.rackspace.com/python_packages/juno/ and OpenSSL was installed ok :/ | 09:13 |
| evrardjp | odyssey4me: np | 09:14 |
| odyssey4me | hughsaunders so I build a Juno build yesterday and once in the venv you can't import OpenSSL | 09:26 |
| odyssey4me | but once you install pyOpenSSL, then you can | 09:26 |
| odyssey4me | so it seems that we just need to add that into tempest's requirements I guess | 09:27 |
| odyssey4me | hughsaunders what is odd though is that if the openstack clients need it, surely it'd be in their requirements and it would therefore install? | 09:32 |
| hughsaunders | odyssey4me: yeah, I'm doing an AIO as well, so will be in the same position soon | 09:35 |
| odyssey4me | hughsaunders ok - let me know | 09:35 |
| odyssey4me | when you're done with your AIO - I don't want to affect the upstream repo while you're mid build | 09:36 |
| hughsaunders | odyssey4me: hmm aio failed on heat retrieve domain id: ERROR: openstack Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) | 09:43 |
| odyssey4me | hughsaunders did you run the juno build including https://review.openstack.org/217098 ? | 09:44 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Updated juno to include fix for CVE-2015-3241 - 26 Aug 2015 https://review.openstack.org/217098 | 09:44 |
| hughsaunders | no, tip of juno. I'll just run the tempest play manually, to get to this issue | 09:44 |
| odyssey4me | tip of juno is broken, this patch is resolving several issues with it | 09:45 |
| odyssey4me | hughsaunders I'm busy updating http://rpc-repo.rackspace.com/python_packages/juno | 09:48 |
| odyssey4me | I don't think the wheels need updating, but that's my test set of wheels to work with. | 09:49 |
| odyssey4me | To use them requires changing rpc_release in rpc_deployment/inventory/group_vars/all.yml to 'juno' | 09:49 |
| odyssey4me | hughsaunders mattt andymccr we have build success for the master unblocker - please review: https://review.openstack.org/218572 | 09:50 |
| hughsaunders | odyssey4me: I see you capped netaddr, good idea | 10:00 |
| odyssey4me | hughsaunders yep, the cap can be removed with the sha bump in https://review.openstack.org/215584 - the cap removal's already there | 10:01 |
| odyssey4me | the netaddr cap works for https://review.openstack.org/217014 (kilo) too | 10:02 |
| *** javeriak has quit IRC | 10:12 | |
| *** shoutm has quit IRC | 10:19 | |
| odyssey4me | mattt https://review.openstack.org/215584 is still working after my adjustment & rebase on the changed https://review.openstack.org/218572 | 10:20 |
| *** javeriak has joined #openstack-ansible | 10:21 | |
| odyssey4me | hughsaunders it looks like the addition of pyOpenSSL to tempest_pip_packages in the tempest role defaults worked | 10:26 |
| odyssey4me | my test build is successful - waiting for a gate pass now: https://review.openstack.org/217098 | 10:26 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Remove hardcoded config drive enforcement https://review.openstack.org/218480 | 10:31 |
| *** javeriak has quit IRC | 10:31 | |
| hughsaunders | odyssey4me: cool | 10:31 |
| odyssey4me | hughsaunders I'm thinking that the approach we're shifting to in juno where the client versions installed is determined by the service and global requirements is perhaps the best way to go overall - your thoughts? | 10:33 |
| mattt | odyssey4me: are you leaving that change to requirements.txt in https://review.openstack.org/#/c/215584/ ? | 10:33 |
| odyssey4me | ideally I'm thinking that we should avoid carrying sha's if we possibly can | 10:33 |
| odyssey4me | mattt yes, that needs to be changed there | 10:33 |
| odyssey4me | your sha bump removes the need for the cap on netaddr | 10:34 |
| odyssey4me | see point 4 in the commit msg ;) | 10:34 |
| hughsaunders | odyssey4me: always a balance - carrying shas gives stability but forces us to update. Relying on upstream for versions is less work, but less stable | 10:34 |
| odyssey4me | hughsaunders how is the sha for the client more stable? | 10:35 |
| hughsaunders | odyssey4me: because an upstream change can't give us a new version of the client | 10:35 |
| odyssey4me | I would think that relying on the global/service requirements is more stable - as that's what is gate tested throughout the openstack-ci | 10:36 |
| mattt | odyssey4me: ok, asking because you suggested a different change above | 10:36 |
| mattt | which is why i was confused | 10:36 |
| odyssey4me | mattt I added the cap in https://review.openstack.org/218572 which has worked to unblock master (please review) | 10:37 |
| mattt | odyssey4me: right but that's not what you said you were doing :P | 10:37 |
| mattt | anyway reviewing now ! | 10:37 |
| odyssey4me | I'd like to ensure that the cap is removed - and it makes logical sense to me to remove it in https://review.openstack.org/215584 as you're doing a sha bump there. | 10:37 |
| odyssey4me | I'm confused about your confusion now. :) | 10:38 |
| mattt | odyssey4me | mattt I'm going to need to pull https://review.openstack.org/219364 into https://review.openstack.org/218572 to unblock the gate - can you then rebase and remove the pin in https://review.openstack.org/215584 as the netaddr bug is addressed by the newer sha in that patch | 10:39 |
| mattt | odyssey4me | mattt my patch will be pinning, so I'm asking you to remove the pin once you've rebased | 10:39 |
| mattt | from above :) | 10:39 |
| odyssey4me | that's exactly what I did | 10:39 |
| odyssey4me | I added the pin (ie the netaddr version cap in requirements.txt) to https://review.openstack.org/218572 , then rebased https://review.openstack.org/215584 and added the removal of the netaddr cap | 10:40 |
| odyssey4me | did you perhaps think I was going to bump the sha in https://review.openstack.org/215584 ? | 10:40 |
| mattt | yes! | 10:40 |
| mattt | bro operating on little sleep here :P | 10:41 |
| odyssey4me | ah, that's what I wanted to avoid :) | 10:41 |
| odyssey4me | bumping the sha would have meant also needing to add the other fixes in https://review.openstack.org/215584 making a mega-franken-patch | 10:41 |
| odyssey4me | capping netaddr is simpler :) | 10:41 |
| odyssey4me | juno update build success! :) https://review.openstack.org/217098 | 10:42 |
| mattt | odyssey4me: question about https://review.openstack.org/#/c/218572 | 10:44 |
| mattt | odyssey4me: why in some playbooks are we not registering ssh_wait_check and retrying ? | 10:44 |
| odyssey4me | mattt example? | 10:44 |
| mattt | odyssey4me: any that have the wait for container ssh task already, ie. https://review.openstack.org/#/c/218572/5/playbooks/os-glance-install.yml | 10:45 |
| odyssey4me | mattt ah, that's an unintentional omission - let me fix that in a follow-up patch | 10:45 |
| mattt | don't want to do it there, since you're already touching those playbooks ? | 10:46 |
| odyssey4me | but, the reason it doesn't matter so much for those is that their container configs don't change as much | 10:46 |
| odyssey4me | so they tend to come up more quickly | 10:47 |
| odyssey4me | but yes, that needs a follow-up patch to address | 10:47 |
| odyssey4me | good catch! | 10:47 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Additional retries for ssh wait check https://review.openstack.org/219638 | 10:53 |
| odyssey4me | mattt this should cover the missing bits: https://review.openstack.org/219638 | 10:54 |
| odyssey4me | mattt hughsaunders can you please review the kilo & juno unblockers too: https://review.openstack.org/217014 (kilo) and https://review.openstack.org/217098 (juno) | 10:56 |
| hughsaunders | odyssey4me: reviewing 217098 at the moment, trying to figure out what happens when the clients are not specified in repo packages. Presumably that means wheels don't get built so packages have to be pulled from upstream pypi? | 10:58 |
| odyssey4me | hughsaunders yes, the packages do get put into the repo server (our upstream one) but they are pulled from pypi | 10:59 |
| mattt | odyssey4me: thought we shouldn't backport until something has merged ? | 10:59 |
| mattt | that one review references a backport that's still in flight | 10:59 |
| odyssey4me | mattt yeah, don't +w yet - I just want to get any discussion/concerns out the way | 11:00 |
| mattt | odyssey4me: k | 11:00 |
| odyssey4me | the 11.2.0 release requires that patch and is several days overdue, so the sooner we can get the votes in the better | 11:01 |
| odyssey4me | hughsaunders an example of the resulting repo is http://rpc-repo.rackspace.com/python_packages/juno/ | 11:01 |
| mattt | odyssey4me: fair enough | 11:01 |
| odyssey4me | so we still build wheels, just not from git | 11:01 |
| odyssey4me | hughsaunders the critical thing this ends up resolving is this issue: https://bugs.launchpad.net/bugs/1488315 | 11:01 |
| openstack | Launchpad bug 1488315 in openstack-ansible juno "The python-requests package is pulled in by apt via dependency" [High,In progress] - Assigned to Jesse Pretorius (jesse-pretorius) | 11:01 |
| odyssey4me | I think what's happening is that we're pulling in clients that are too new and have greater restrictions in some places for the deps. | 11:03 |
| *** gparaskevas has quit IRC | 11:03 | |
| odyssey4me | Using the service/global requirements resolution of which clients to use means that we get a properly tested and known working client for the services - and likely something that the package maintainers are also working against, so there shouldn't be conflicts like that bug. | 11:04 |
| *** k_stev has joined #openstack-ansible | 11:28 | |
| hughsaunders | OK I get it now, with_py_pkgs gets a list of git repos, one of which is openstack/requirements. Yaprt clones the repos and extracts the requirements in order to build dependant wheels. So by including global requirements and all the services we'll get all the clients without having to specify versions. | 11:33 |
| mattt | man that is a lot of changes to unblock a stable branch | 11:34 |
| odyssey4me | hughsaunders yep, and the versions we get will be based on the upper and lower caps in each service project | 11:34 |
| hughsaunders | I tested with just bumping neutron, but I guess we'd have to bump the rest at some point. | 11:36 |
| odyssey4me | mattt agreed, but unfortunately they're all tied together :/ | 11:36 |
| odyssey4me | mattt assuming you're talking about the juno review? | 11:36 |
| mattt | odyssey4me: yeah 217098 | 11:36 |
| hughsaunders | just reading 217014, its annoying that the service playbooks have to have lxc related tasks. I guess better in the playbooks than the roles. | 11:40 |
| odyssey4me | hughsaunders yep - they were pulled into the playbooks to cut down the down-time | 11:41 |
| odyssey4me | I actually think that putting them into the playbooks is a good thing | 11:41 |
| odyssey4me | but it is a departure from how we were doing things | 11:42 |
| *** bapalm has quit IRC | 11:43 | |
| mattt | odyssey4me: do you have a juno instance up w/ 217098 applied ? | 11:46 |
| odyssey4me | mattt yup | 11:46 |
| odyssey4me | mattt I was thinking of perhaps firing up a fresh one and setting it up for maas | 11:47 |
| mattt | odyssey4me: want me to do that? i want to test something w/ heat | 11:48 |
| mattt | use the stack_heat_domain_name used to not work, i see the bug was fixed but we should be sure before we commit that | 11:48 |
| mattt | because i don't think our gate will catch many heat issues :( | 11:48 |
| odyssey4me | mattt sure - it'd be good to have a more thorough test | 11:48 |
| mattt | ok cool doing now | 11:48 |
| odyssey4me | I did see the upstream issue and resolution of it | 11:48 |
| *** bapalm has joined #openstack-ansible | 11:49 | |
| odyssey4me | mattt you're talking about this one? https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&uact=8&ved=0CDUQFjADahUKEwjDy9bJpNjHAhUCtxQKHfl9Dcc&url=https%3A%2F%2Fbugs.launchpad.net%2Fbugs%2F1376213&usg=AFQjCNFl-TsodddEzf9xG7f3OPox2QIlVA&sig2=LgPrYMveDcM4l-J37mjUyg&bvm=bv.101800829,d.d24 | 11:52 |
| mgariepy | good morning everyone | 11:52 |
| odyssey4me | lolfail | 11:52 |
| odyssey4me | https://bugs.launchpad.net/heat/+bug/1376213 | 11:52 |
| openstack | Launchpad bug 1376213 in heat "stack_user_domain_name not working as expected" [High,Fix released] - Assigned to Ethan Lynn (ethanlynn) | 11:52 |
| mattt | odyssey4me: haha yeah that one | 11:52 |
| mattt | mgariepy: morning, how's quebec doing today | 11:52 |
| mgariepy | doing fine ;) | 11:53 |
| odyssey4me | fixed in heat on nov 6: https://review.openstack.org/133591 | 11:53 |
| mattt | odyssey4me: yeah i just want to be sure, since the gate only tests the heat API iirc | 11:53 |
| odyssey4me | sure thing :) | 11:54 |
| *** k_stev has quit IRC | 11:58 | |
| *** jaypipes has joined #openstack-ansible | 12:08 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 12:08 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Wait for container ssh after apparmor profile update https://review.openstack.org/218572 | 12:13 |
| evrardjp | goog morning mgariepy | 12:25 |
| evrardjp | good* | 12:25 |
| mgariepy | how are you doing ? | 12:25 |
| evrardjp | fine... a little disappointed by cinder, but everything is fine... and you? | 12:25 |
| mgariepy | i'm doing fine. | 12:26 |
| mgariepy | i'll tests your keepalive today if my hardware is fast enough ! | 12:27 |
| mgariepy | keepalived** | 12:27 |
| *** mordred has quit IRC | 12:27 | |
| evrardjp | ok | 12:30 |
| evrardjp | don't hesitate to ask | 12:30 |
| evrardjp | and don't forget to configure variables, but also stuff on your hosts! | 12:31 |
| evrardjp | (like configuring the NIC/bridge where the IP should bind on) | 12:31 |
| evrardjp | Sam-I-Am: You there? | 12:31 |
| evrardjp | I'm reading the documentation here: http://openstack-ansible-deployment.readthedocs.org/en/latest/install-guide/configure-cinder-nfs.html | 12:31 |
| evrardjp | I'm trying to have NetApp NFS (netapp iscsi works fine) on my cinder | 12:32 |
| evrardjp | the page starts with that, so I'm fine | 12:32 |
| evrardjp | however the previous page mentions that I need to write a netapp stanza (or whatever name I put on) in my cinder_backends (with the additional configurations) | 12:33 |
| evrardjp | so I wrote in my netapp stanza a nfs_shares_config: like written on this previous page I just mentionned | 12:34 |
| evrardjp | (it's this one I'm talking about: http://openstack-ansible-deployment.readthedocs.org/en/latest/install-guide/configure-cinder.html) | 12:34 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [OSAD doc] Update link to Cloud Admin Guide https://review.openstack.org/219085 | 12:34 |
| *** KLevenstein has joined #openstack-ansible | 12:35 | |
| evrardjp | however this DOESN'T create/template the shares at all | 12:35 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Removed rpc-repo upstream pip deps https://review.openstack.org/216181 | 12:35 |
| evrardjp | because my nfs_shares_config isn't in cinder_nfs_client | 12:36 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Fixing haproxy-playbook fails when installing on multiple hosts https://review.openstack.org/215579 | 12:40 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Shift irqbalance package from lxc_hosts to openstack_hosts https://review.openstack.org/218354 | 12:41 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add lvm.conf cleanup to teardown script https://review.openstack.org/217827 | 12:41 |
| *** woodard has joined #openstack-ansible | 12:41 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Switch Nova/Tempest to use/test Cinder API v2 https://review.openstack.org/214045 | 12:41 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Update tempest configuration https://review.openstack.org/210107 | 12:42 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow cinder-backup to use ceph https://review.openstack.org/209537 | 12:44 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Implement tox.ini config for bashate and pep8 tests https://review.openstack.org/216170 | 12:45 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 12:45 | |
| *** cloudtrainme has joined #openstack-ansible | 12:46 | |
| *** cloudtrainme has quit IRC | 12:47 | |
| evrardjp | I've dropped nfs just for that | 12:56 |
| mattt | odyssey4me: no issues w/ heat in that patch, a stack created for me fine | 12:58 |
| odyssey4me | mattt awesome :) did you test maas too? | 12:59 |
| mattt | odyssey4me: working on that now | 13:00 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add variable for cirros url https://review.openstack.org/217310 | 13:01 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add sorting_method to swift proxy config as needed https://review.openstack.org/208817 | 13:03 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Implementation of keepalived for haproxy https://review.openstack.org/218818 | 13:04 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Use the pre-defined Ubuntu mirror for the AIO https://review.openstack.org/218611 | 13:05 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Fix the heat stack user create https://review.openstack.org/218184 | 13:06 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Ensure rsync restarts fully during swift setup https://review.openstack.org/217341 | 13:07 |
| *** scarlisle has joined #openstack-ansible | 13:07 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Remove Juno to Kilo logic in upgrade script https://review.openstack.org/215291 | 13:08 |
| cloudnull | morning | 13:09 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add profiling for Ansible tasks https://review.openstack.org/216849 | 13:10 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Used named veth pairs that match container https://review.openstack.org/219457 | 13:10 |
| odyssey4me | o/ cloudnull | 13:11 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow Horizon setup with external SSL termination https://review.openstack.org/214647 | 13:11 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Container create/system tuning https://review.openstack.org/215905 | 13:13 |
| cloudnull | odyssey4me: where are we with things? hows anything i can help out with for a bit | 13:13 |
| cloudnull | or reviews that need doing ? | 13:14 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Container create/system tuning https://review.openstack.org/215905 | 13:14 |
| cloudnull | most of us in the US are out at 10 for a team outing | 13:14 |
| cloudnull | but i can bang on some things for the next couple of hours | 13:14 |
| *** k_stev has joined #openstack-ansible | 13:15 | |
| evrardjp | morning cloudnull | 13:16 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Implement /usr/bin/env as the shebang in all bash scripts https://review.openstack.org/215906 | 13:17 |
| cloudnull | morning evrardjp | 13:17 |
| odyssey4me | cloudnull master is unblocked, and I've just run through everything to rebase for up to date gate checks | 13:18 |
| cloudnull | yay! | 13:18 |
| odyssey4me | kilo needs a final reviewer: https://review.openstack.org/217014 | 13:18 |
| odyssey4me | then it'll be unblocked and I can release 11.2.0 later | 13:18 |
| mattt | odyssey4me: messed up that aio deploy for testing maas, didn't call it aio1 :( | 13:19 |
| odyssey4me | mattt ah, that is a neat trick | 13:19 |
| odyssey4me | mattt I usually do this instead: https://github.com/rcbops/rpc-openstack/blob/master/scripts/deploy.sh#L43-L44 | 13:20 |
| mattt | i thought we had something to update the inventory, must be mistaken | 13:20 |
| cloudnull | andymccr: mattt: hughsaunders: will need to review https://review.openstack.org/217014 im the authhor. | 13:20 |
| hughsaunders | will do | 13:20 |
| odyssey4me | but yeah, I think that needs to be done before deployment | 13:20 |
| odyssey4me | hughsaunders already has :) | 13:20 |
| hughsaunders | already done :) | 13:21 |
| odyssey4me | mattt will probably be happy to approve now that the master commit is merged | 13:21 |
| *** k_stev has quit IRC | 13:21 | |
| cloudnull | also if we can get https://review.openstack.org/#/c/219457 accepted (assuming its something we all want) and backported to kilo itll be great | 13:21 |
| cloudnull | mhayden: ^ - cc | 13:21 |
| mattt | odyssey4me: bah, i knew i saw that somewhere, couldn't remember where it was :-/ | 13:22 |
| odyssey4me | cloudnull so I'm going to revise https://review.openstack.org/218611 to make the AIO use the locally set mirror, but also include changes in the gate check to test whether it's in rax/hpcloud and to set the mirror to the default mirrors for the cloud in question | 13:25 |
| *** k_stev has joined #openstack-ansible | 13:25 | |
| odyssey4me | we can recheck that for a bit and see how it goes, but it seems that even using ubuntu archive is helpful for improving the hpcloud success rate | 13:25 |
| hughsaunders | cloudnull: you want named veth pairs before the kilo release? | 13:25 |
| cloudnull | yes. if possible. | 13:26 |
| cloudnull | right now every contianer restart potentialy creates a new veth leaving an old one behind . | 13:26 |
| odyssey4me | that said, hpcloud-b4 still always breaks - the containers can't speak beyond the host, so I'm suspecting that our iptables clearing isn't helping and will run some tests to target that issue afterwards | 13:26 |
| cloudnull | i have an AIO with 500 + veths on it when it should only have around 120 | 13:27 |
| odyssey4me | cloudnull oh dear, so that's another 11.2.0 blocker? | 13:27 |
| cloudnull | naming them makes that go away. | 13:27 |
| odyssey4me | do we have a bug registered for this? | 13:27 |
| hughsaunders | ahhh, I thought it was just for debug convenience | 13:27 |
| odyssey4me | that is a bit of an issue - evrardjp and svg would probably want to input on the conversation | 13:28 |
| cloudnull | it has the added side effect of providing an easy way to debug for sure but removing the extra load on the network stack is the real benifit | 13:28 |
| mhayden | cloudnull: i'd like to email out some of the dangling veth findings later today -- is that something good for the openstack-dev list with [openstack-ansible] tag? | 13:28 |
| cloudnull | indeed | 13:29 |
| *** gparaskevas has joined #openstack-ansible | 13:29 | |
| odyssey4me | cloudnull but changing them does result in another restart of the containers, right? if so then perhaps holding back 11.2.0 until we have that done is a good plan | 13:29 |
| odyssey4me | otherwise maybe an 11.3.0 should come out when it does | 13:29 |
| odyssey4me | I'm not keen to actually rush another entirely new thing through. | 13:29 |
| evrardjp | odyssey4me: sorry I didn't follow, you want my input for? | 13:30 |
| *** scarlisle has quit IRC | 13:30 | |
| cloudnull | it requires a restart to take effect but if i read the code right it will add the option and when the container restarts it will begin using the named veths. so its not forcing the restart. | 13:30 |
| odyssey4me | evrardjp see the above conversation about leftover veth pairs | 13:30 |
| svg | odyssey4me: not sure on what I should comment? | 13:31 |
| svg | Can't recall I had issues or ideas on veth pairs? | 13:32 |
| cloudnull | its not something we've seen a lot but when it begins to happen and on a long running cluster theyll clean themselves up over time if im not mistaken -cc mhayden | 13:33 |
| * cloudnull needs coffee yet | 13:33 | |
| cloudnull | when it begins to happen they stack up quickly when containers are restarted. | 13:33 |
| odyssey4me | cloudnull honestly I think that it's best that we do two things - one is to implement named veth pairs for liberty, then backport it into kilo in a feature release; the other is to provide some way of cleaning up the leftovers as a workaround until the named veth pairs are implemented | 13:34 |
| evrardjp | odyssey4me: I read it. You (in general) have found something important | 13:34 |
| evrardjp | red* | 13:34 |
| evrardjp | no doc impact on the commit though? | 13:34 |
| evrardjp | what should I look at to give you input about my current drift? | 13:35 |
| odyssey4me | evrardjp I have no idea - the guys in the US are looking into it. | 13:37 |
| evrardjp | the id of the veth: on ip link or the amount of veth on a host? | 13:37 |
| odyssey4me | mhayden can tell you, I think | 13:37 |
| *** tlian has joined #openstack-ansible | 13:39 | |
| evrardjp | I have ids that go up 1600, but only a small amount are really assigned, so I'm sure of the problem... will wait for mhayden input then | 13:40 |
| cloudnull | odyssey4me: im good with waiting on a major release . | 13:41 |
| cloudnull | 11.3 is fine if we cant make it go for 11.2 | 13:42 |
| gparaskevas | odyssey4me: quick question can you define floating range to in the openstack_userconfig? And where does public_network binds? | 13:42 |
| odyssey4me | cloudnull yep - we can perhaps ship a workaround/cleanup script into a hotfix for operational execution until the major release is out | 13:42 |
| mgariepy | evrardjp, concerning keepalived, if I have 3 hosts on which i want to configure vrrp and stuff i guess i would need different priority for all of them ? | 13:43 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Disable python buffering for gate checks https://review.openstack.org/218595 | 13:43 |
| odyssey4me | gparaskevas honestly I have no idea off-hand | 13:43 |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Add profiling for Ansible tasks https://review.openstack.org/216849 | 13:43 |
| gparaskevas | odyssey4me: ok! thanks ! | 13:43 |
| mhayden | evrardjp: the issue we saw is that a container goes down and the host side of the veth is still on the bridge | 13:44 |
| mhayden | but nothing from the container is connected to it | 13:44 |
| mhayden | it's like a dangling ethernet cable ;) | 13:44 |
| openstackgerrit | Matt Thompson proposed stackforge/os-ansible-deployment: Update how neutron migrations are handled https://review.openstack.org/215584 | 13:45 |
| odyssey4me | mhayden what do you think of my suggested approach mentioned to cloudnull ? | 13:46 |
| mhayden | odyssey4me: getting named-veths merged and backported to kilo? or something else? | 13:46 |
| odyssey4me | mhayden a script to clean up and the named veth setup - aimed at master | 13:47 |
| mhayden | sounds good to me | 13:47 |
| mhayden | i need to update the named-veth patch a bit based on cloudnull's good suggestion | 13:47 |
| odyssey4me | then if need be we can backport the script alone to kilo, or do another feature release if it's confirmed to be good and ready for production use | 13:47 |
| cloudnull | Apsu: has one of those. He should PR that to liberty for review | 13:47 |
| mhayden | as for the script, we're looking for something we can run that will auto-remove dangling veths? | 13:47 |
| odyssey4me | mhayden personally I think that perhaps it shouldn't auto-run or even run in ansible | 13:48 |
| odyssey4me | it should just be a tool available to operators - they can choose when to run it | 13:48 |
| cloudnull | if we can make it part of the existing lxc-system-manage command it would be idea. | 13:48 |
| cloudnull | &ideal. | 13:48 |
| cloudnull | ** | 13:48 |
| * cloudnull still needs coffee | 13:48 | |
| odyssey4me | cloudnull yeah, that would be nice if it's at all possible | 13:49 |
| cloudnull | ++ | 13:49 |
| mhayden | odyssey4me / cloudnull: totally agree -- i think ol' Apsu already has quite a nice script ;) | 13:49 |
| cloudnull | yea idk for sure. ive not seen Apsu 's script | 13:49 |
| * mhayden cozies up next to Apsu and asks him if i can toss his script into a gerrit review ;) | 13:49 | |
| odyssey4me | the script may need some baking too - someone ran it during an install the other day and it broke everything :p | 13:49 |
| mhayden | oops | 13:50 |
| Sam-I-Am | mhayden: the vetherator script? | 13:50 |
| cloudnull | Sam-I-Am: obviously . | 13:50 |
| mhayden | hah yeah | 13:50 |
| mhayden | why run the script when you could just reboot the server? | 13:50 |
| cloudnull | the script does more than one thing, thus .*erator | 13:50 |
| * mhayden hides | 13:50 | |
| odyssey4me | anyway - I think the spec should be updated to make it cover both the issue and the convenience, not just the convenience | 13:50 |
| Sam-I-Am | its like terminator, but with veth pairs | 13:51 |
| cloudnull | mhayden: https://gist.github.com/cloudnull/9361461 | 13:51 |
| cloudnull | yw | 13:51 |
| odyssey4me | then we need the script and veth config submitted, probably in seperate commits/reviews to make it easier to selectively backport | 13:51 |
| mhayden | cloudnull: ah, haven't seen that one | 13:51 |
| Sam-I-Am | *hopefully* this named veth pair changes resolves the vethelberry problem, but having the script makes sense for existing deploys | 13:51 |
| cloudnull | fixes openstack guaranteed. | 13:51 |
| mhayden | Apsu's script did some interesting things in bash i haven't seen done much before ;) | 13:51 |
| mhayden | cloudnull: need that in a sticker | 13:51 |
| Sam-I-Am | mhayden: dont look at source, just run as root | 13:52 |
| cloudnull | mhayden: we do | 13:52 |
| *** k_stev has quit IRC | 13:52 | |
| mhayden | Sam-I-Am: that's how it works with Go, right? wget, chmod +x, ./run WOOOO | 13:52 |
| *** k_stev has joined #openstack-ansible | 13:52 | |
| Sam-I-Am | mhayden: pretty much | 13:53 |
| Sam-I-Am | mhayden: thinking i should test the veth namerator in juno, then apply patch to kilo, upgrade | 13:53 |
| Sam-I-Am | see what blows up | 13:53 |
| mhayden | that'd be interesting | 13:54 |
| mhayden | cloudnull mentioned there were some juno-isms that might make named-veths a less than good idea | 13:54 |
| cloudnull | ++ juno.* | 13:54 |
| evrardjp | ok I'm back in the conversation | 13:54 |
| mhayden | evrardjp: howdy! | 13:55 |
| evrardjp | I don't see much dangling veth on my host side | 13:55 |
| evrardjp | is there some kind of garbage collection ? | 13:55 |
| Sam-I-Am | cloudnull: increment juno? | 13:55 |
| Sam-I-Am | gehhh, i replied to a thread on the os mailing list | 13:55 |
| evrardjp | mhayden: I have veth link ids like 1600, but only a few hundreds of veths for the current deployment. but I sometimes reboot, so it cleanes up naturally at that time | 13:56 |
| mhayden | evrardjp: one of the theories (based on an lxc thread) is that open tcp connections might hold the veth open | 13:56 |
| evrardjp | ok | 13:56 |
| mhayden | we were seeing 1-5 veths dangling after rebooting all containers | 13:56 |
| Sam-I-Am | or its something with cgroups breaking on teardown | 13:57 |
| mhayden | some disappeared on their own after a period of time | 13:57 |
| Sam-I-Am | perhaps a timing/race issue | 13:57 |
| mhayden | but some hung out for a long time | 13:57 |
| Sam-I-Am | mhayden: they're social veth pairs | 13:57 |
| evrardjp | after rebooting or after teardown.sh ? | 13:57 |
| mhayden | evrardjp: just after stopping containers | 13:57 |
| Sam-I-Am | rebooting clears up the things | 13:57 |
| evrardjp | do you want me to test on my containers to have a confirmation? | 13:57 |
| Sam-I-Am | evrardjp: the patch? | 13:58 |
| evrardjp | I'll use something to kill the tcp connections | 13:58 |
| mhayden | the funny thing is that naming each named veth seems to eliminate the problem from my testing | 13:58 |
| cloudnull | odyssey4me: in leiberty, IMO this is a priority if we can get it through soon-ish https://review.openstack.org/#/c/215584 | 13:58 |
| Sam-I-Am | mhayden: hence mandelbug | 13:58 |
| cloudnull | mattt: -cc | 13:59 |
| evrardjp | Sam-I-Am: you mean the vextherminator? | 13:59 |
| cloudnull | thers an issue with neutron db migrations still and that resolves it 100%, in my testing | 13:59 |
| evrardjp | or the renaming? | 13:59 |
| evrardjp | ok | 13:59 |
| evrardjp | we can just say that sometimes we need to reboot hosts | 13:59 |
| evrardjp | if it's in the docs, under a known issue section, you don't have to hurry a backport | 14:00 |
| Sam-I-Am | evrardjp: most of the time people don't reboot containers until an upgrade | 14:00 |
| Sam-I-Am | so that seems to be the first time this turns into a problem | 14:00 |
| odyssey4me | cloudnull agreed - although mattt was still busy doing some more work on the patch, so I've held back | 14:00 |
| Sam-I-Am | and only if the dangler has a mac/ip | 14:00 |
| odyssey4me | mattt is that ready for review yet? | 14:00 |
| cloudnull | kk i've got to run. see you all back online late.r | 14:01 |
| Sam-I-Am | cloudnull: running is bad, mmmkay | 14:01 |
| *** markvoelker has joined #openstack-ansible | 14:01 | |
| cloudnull | have a good one guys. | 14:01 |
| evrardjp | thx bye! | 14:01 |
| *** cloudtrainme has joined #openstack-ansible | 14:01 | |
| evrardjp | Sam-I-Am: so it shouldn't be a problem until upgrade, so it could be part of the known issues/procedure | 14:02 |
| mhayden | cloudnull: good call on the string splicing for named-veths | 14:02 |
| mattt | odyssey4me: yeah, there is more work to do but i think that can go through for now | 14:02 |
| odyssey4me | mattt awesome, thanks :) | 14:02 |
| mattt | oh my goodness, the cinder-backup review finally passed | 14:02 |
| evrardjp | :) | 14:03 |
| evrardjp | mattt I'll test it in early october if everything goes fine | 14:03 |
| evrardjp | I'll check the impact of the named veth pairs too, it seems important | 14:03 |
| mattt | evrardjp: nice thanks! | 14:04 |
| Sam-I-Am | evrardjp: definitely in known issues | 14:05 |
| odyssey4me | mattt yep :) | 14:05 |
| odyssey4me | mattt failed_when: false in https://review.openstack.org/#/c/215584/12/playbooks/roles/os_neutron/tasks/neutron_db_setup.yml,cm line 47 | 14:06 |
| *** spotz_zzz is now known as spotz | 14:06 | |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Used named veth pairs that match container https://review.openstack.org/219457 | 14:06 |
| *** markvoelker has quit IRC | 14:06 | |
| *** cloudtrainme has quit IRC | 14:06 | |
| odyssey4me | is that right? | 14:06 |
| Sam-I-Am | odyssey4me: i *think* so | 14:07 |
| Sam-I-Am | but i'd have to poke moar | 14:08 |
| mattt | odyssey4me: i believe that's right assuming i'm using failed_when correctly | 14:08 |
| mattt | we don't want that task to fail if grep finds nothing | 14:08 |
| evrardjp | if the command fails hard before? | 14:10 |
| odyssey4me | mattt if you run the plays on a blank slate, but then also on a db that's been initialised - does it work both ways? | 14:10 |
| evrardjp | I'd rather register the response code, and use another task to check response code and grep content | 14:10 |
| mattt | evrardjp: yeah that was actually waht i needed to do on follow-up work | 14:11 |
| *** phalmos has joined #openstack-ansible | 14:11 | |
| mattt | evrardjp: git-harry was concerned that the command itself would fail and we'd end up doing the wrong thing | 14:11 |
| mattt | odyssey4me: it will work both ways, yep | 14:11 |
| odyssey4me | mattt ok, happy to improve in follow up work - I just didn't want to create another blocker | 14:12 |
| *** cloudtrainme has joined #openstack-ansible | 14:12 | |
| Sam-I-Am | going for a walk bbiab | 14:13 |
| evrardjp | mattt: git-harry I right. And I don't say that because it was my concern too :p | 14:14 |
| hughsaunders | I can't find a bug for excess veth pairs, anyone know if there is one? | 14:14 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Removed rpc-repo upstream pip deps https://review.openstack.org/216181 | 14:15 |
| *** phalmos has quit IRC | 14:20 | |
| mhayden | odyssey4me: bumped the named-veths patch and left the WIP off since it's further along now | 14:21 |
| *** javeriak has joined #openstack-ansible | 14:22 | |
| *** k_stev has quit IRC | 14:25 | |
| mhayden | hughsaunders: imma make that bug for the danglers | 14:26 |
| hughsaunders | mhayden: thanks :) | 14:26 |
| *** sdake has quit IRC | 14:28 | |
| *** cloudtrainme has quit IRC | 14:29 | |
| *** javeriak has quit IRC | 14:29 | |
| *** javeriak has joined #openstack-ansible | 14:30 | |
| *** alextricity has joined #openstack-ansible | 14:30 | |
| *** shausy has quit IRC | 14:30 | |
| *** sdake has joined #openstack-ansible | 14:31 | |
| *** cloudtrainme has joined #openstack-ansible | 14:32 | |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Used named veth pairs that match container https://review.openstack.org/219457 | 14:32 |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Used named veth pairs that match container https://review.openstack.org/219457 | 14:33 |
| *** javeriak_ has joined #openstack-ansible | 14:35 | |
| *** javeriak has quit IRC | 14:36 | |
| mhayden | hughsaunders: not sure what else to tickle on https://bugs.launchpad.net/openstack-ansible/+bug/1491440 -- just assigned it to myself | 14:37 |
| openstack | Launchpad bug 1491440 in openstack-ansible "Restarting containers leads to 'dangling' veth interfaces" [Undecided,New] - Assigned to Major Hayden (rackerhacker) | 14:37 |
| *** phalmos has joined #openstack-ansible | 14:39 | |
| hughsaunders | mhayden: lgtm, for completeness it could be targeted at a milestone probably 11.2.0. Can be pushed back if not ready for some reason. | 14:40 |
| mhayden | got it | 14:42 |
| *** markvoelker has joined #openstack-ansible | 14:42 | |
| *** markvoelker has quit IRC | 14:42 | |
| *** markvoelker has joined #openstack-ansible | 14:43 | |
| *** cloudtrainme has quit IRC | 14:43 | |
| *** javeriak_ has quit IRC | 14:46 | |
| *** cloudtrainme has joined #openstack-ansible | 14:46 | |
| evrardjp | I have a question about cinder configuration of default AZ | 14:48 |
| evrardjp | in the doc we define cinder_storage_availability_zone under the storage_hosts | 14:49 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Change AppArmor profile application order https://review.openstack.org/217014 | 14:49 |
| evrardjp | in container_vars | 14:49 |
| evrardjp | but by default cinder scheduler is NOT on the storage hosts, it's on storage-infra_hosts | 14:50 |
| evrardjp | it could be the same, it's not necessarily, right? | 14:50 |
| *** javeriak has joined #openstack-ansible | 14:51 | |
| evrardjp | so it means the cinder_storage_availability_zone isn't set on the storage-infra_hosts, and so cinder scheduler thinks the cinder_storage_availability_zone is nova | 14:51 |
| evrardjp | (or to be more precise: (storage|default)_availability_zone is nova in cinder.conf for cinder scheduler containers) | 14:52 |
| evrardjp | is this a bug or a feature? | 14:52 |
| evrardjp | will I break everything if I change the AZ in cinder_scheduler? | 14:52 |
| *** galstrom_zzz is now known as galstrom | 14:53 | |
| *** cloudtra_ has joined #openstack-ansible | 14:55 | |
| Sam-I-Am | mhayden: should bug 1491440 be partial-bug in the commit message? | 14:56 |
| openstack | bug 1491440 in openstack-ansible "Restarting containers leads to 'dangling' veth interfaces" [Undecided,In progress] https://launchpad.net/bugs/1491440 - Assigned to Major Hayden (rackerhacker) | 14:56 |
| *** cloudtrainme has quit IRC | 14:56 | |
| odyssey4me | hmm, evrardjp I haven't worked with that in several cycles but as I recall cinder-volume should be assigned an AZ whereas cinder-scheduler schedules volumes into the AZ's | 14:56 |
| odyssey4me | Sam-I-Am yes, as the script is the thing that will remove the dangling veths - the naming of the veths is a feature with the spec | 14:57 |
| *** Mudpuppy has joined #openstack-ansible | 14:59 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Ensure rsync restarts fully during swift setup https://review.openstack.org/217341 | 14:59 |
| *** cloudtra_ has quit IRC | 14:59 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Add sorting_method to swift proxy config as needed https://review.openstack.org/208817 | 15:00 |
| *** KLevenstein has quit IRC | 15:00 | |
| *** woodard has quit IRC | 15:00 | |
| *** cloudtrainme has joined #openstack-ansible | 15:01 | |
| *** KLevenstein has joined #openstack-ansible | 15:03 | |
| *** woodard has joined #openstack-ansible | 15:03 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Enable tempest testing of ceilometer https://review.openstack.org/217600 | 15:03 |
| *** KLevenstein has quit IRC | 15:04 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Additional retries for ssh wait check https://review.openstack.org/219638 | 15:04 |
| mhayden | Sam-I-Am: that might be better | 15:05 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Adds the ability to provide user certificates to HAProxy https://review.openstack.org/215525 | 15:05 |
| evrardjp | odyssey4me: my concern is the following: If I create a volume with horizon, and I define the cinder AZ plus the cinder backend type, it works fine | 15:07 |
| evrardjp | if I create the volume, and don't mention the AZ ("Any AZ"), then the volume fails to create | 15:08 |
| evrardjp | because horizon tries to use "nova" as AZ | 15:08 |
| odyssey4me | evrardjp hmm, isn't there meant to be a default availability zone setting somewhere? | 15:08 |
| odyssey4me | does horizon try to do that, or is that a default either in nova or cinder? | 15:08 |
| odyssey4me | (or both) | 15:09 |
| evrardjp | I think it's how we deploy | 15:09 |
| evrardjp | I think (no confirmation yet), that the variable I explained before isn't set on the scheduler, and therefore horizon gives nothing, and cinder scheduler uses its default, which is nova | 15:10 |
| evrardjp | I'll tell you that when my cinder has finished redeploying | 15:10 |
| evrardjp | is redeployed* | 15:10 |
| odyssey4me | wait, horizon has nothing in the list? | 15:10 |
| evrardjp | nope | 15:11 |
| evrardjp | my system is redeployed | 15:11 |
| evrardjp | and it's confirmed | 15:11 |
| odyssey4me | ok, if you list the zones in cinder - do yuo see any? | 15:11 |
| evrardjp | yes, and in horizon too | 15:12 |
| evrardjp | it's perfectly fine | 15:12 |
| evrardjp | everything works as it should be | 15:12 |
| *** gparaskevas has quit IRC | 15:12 | |
| odyssey4me | oh - so horizon and cinder do show zones? | 15:12 |
| evrardjp | yes | 15:12 |
| evrardjp | however, cinder-scheduler has nova as default AZ | 15:12 |
| *** phalmos has quit IRC | 15:12 | |
| evrardjp | and as AZ | 15:12 |
| odyssey4me | ah | 15:12 |
| evrardjp | which isn't good | 15:12 |
| evrardjp | so the variables that we are setting under storage_hosts | 15:13 |
| evrardjp | for AZ | 15:13 |
| evrardjp | should also be set on storage-infra_hosts | 15:13 |
| evrardjp | not sure how to have a clean version of it though | 15:13 |
| odyssey4me | yeah, that sounds like a bug related to the switch from cohabitating cinder-volume and cinder-scheduler to splitting them out | 15:13 |
| *** k_stev has joined #openstack-ansible | 15:14 | |
| evrardjp | if you put them on the same host, it won't be a problem | 15:14 |
| odyssey4me | so you're definining them in the openstack_user_config? | 15:14 |
| *** phalmos has joined #openstack-ansible | 15:15 | |
| evrardjp | what I had before (and thus failing), was a definition of cinder_storage_availability_zone and cinder_default_availability_zone in container_vars under storage_hosts in openstack_user_config | 15:15 |
| evrardjp | (I'm skipping a few levels, but you understand ;) ) | 15:16 |
| evrardjp | Like described in the doc | 15:16 |
| odyssey4me | so I would think that user_variables should be used to set the default zone site-wide, then each storage host should have a specific zone allocated | 15:16 |
| evrardjp | that would work | 15:16 |
| odyssey4me | I don't know if it makes sense to set a storage zone on a scheduler, nor do I think it makes sense to set a default zone on a storage server. | 15:17 |
| evrardjp | but that's not how it's described on the doc | 15:17 |
| odyssey4me | yep, this is a bug for sure | 15:17 |
| odyssey4me | you aren't going mad :p | 15:17 |
| evrardjp | odyssey4me: still, there are 2 variables in the scheduler | 15:18 |
| evrardjp | cinder scheduler container I mean | 15:18 |
| odyssey4me | yep, I'm just not sure if it gets used | 15:18 |
| evrardjp | storage_availability_zone and default_.* | 15:18 |
| evrardjp | ok | 15:18 |
| evrardjp | let me try and tell you | 15:18 |
| odyssey4me | try only setting the default for the scheduler, and the zone on the storage server | 15:19 |
| hughsaunders | so where is the best place for group vars? Because this could go in group_vars/cinder_all. However we don't seem to have any group vars except for hosts and all. | 15:19 |
| odyssey4me | hughsaunders sure, but then would openstack_user_config be able to override it on a per host basis? | 15:19 |
| odyssey4me | it seems to me more that we just set a default in the role and user_config is used to override the zone var on a host by host basis... and the default zone is set in user_variables | 15:20 |
| hughsaunders | I presume so, as a hostvar should override a group var | 15:20 |
| odyssey4me | I don't think a group var is needed anyway, but if host vars override then that's good. :) | 15:21 |
| evrardjp | I think it could be useful to have a group vars cinder_all if the scheduler has a problem with only have default_ and no storage_availability_zone | 15:28 |
| evrardjp | I need to think of all implications though | 15:29 |
| *** cloudtrainme has quit IRC | 15:29 | |
| *** cloudtrainme has joined #openstack-ansible | 15:30 | |
| evrardjp | ok it works | 15:31 |
| odyssey4me | evrardjp but the role has a default - so I'm not sure what the issue is | 15:31 |
| evrardjp | no need to worry about storage_availability_zone | 15:31 |
| odyssey4me | cool, thought as much | 15:31 |
| odyssey4me | we should perhaps not even bother placing it in cinder.conf for the hosts that don't need it (like we're doing for neutron) | 15:32 |
| evrardjp | the role as a default: nova, for both the storage_availabilty_zone and default_availability_zone | 15:32 |
| evrardjp | like the schedulers? | 15:32 |
| odyssey4me | can you register a bug with the doc issue and your findings | 15:32 |
| evrardjp | yup | 15:32 |
| evrardjp | I'll | 15:32 |
| odyssey4me | yeah - ie don't put default_zone in the storage node's cinder.conf | 15:33 |
| odyssey4me | and don't put storage_zone in the the scheduler's cinder.conf | 15:33 |
| odyssey4me | that makes it less confusing - otherwise you'll see the mismatch and get confused | 15:33 |
| odyssey4me | mattt - are you still around? how's the testing going of maas for https://review.openstack.org/217098 ? | 15:34 |
| *** britthouser has quit IRC | 15:34 | |
| openstackgerrit | Major Hayden proposed stackforge/os-ansible-deployment: Used named veth pairs that match container https://review.openstack.org/219457 | 15:36 |
| mhayden | ^^ switched to Partial-Bug per Sam-I-Am | 15:36 |
| odyssey4me | mhayden I thought it was already. | 15:37 |
| odyssey4me | anyway | 15:37 |
| mattt | odyssey4me: hey, question ... neutron_db_revision ... do we actually still need that now, or will we always point to head/heads ? | 15:39 |
| odyssey4me | mattt I think when liberty releases then it might change - dunno | 15:39 |
| mattt | odyssey4me: yeah it's not clear, kilo points to head, juno points to head | 15:42 |
| mattt | odyssey4me: icehouse seems to point to icehouse tho | 15:42 |
| odyssey4me | mattt well, perhaps the qeuestion should be - is it static through the whole cycle or not, and should a user ever change it? | 15:43 |
| odyssey4me | if it's always static and the user never needs to override it, then no var is needed | 15:43 |
| mattt | odyssey4me: hmm, think that may be a question for cloudnull | 15:46 |
| mattt | i don't believe a user would ever change that | 15:46 |
| odyssey4me | I'm not even sure that a dev would change it? | 15:47 |
| mattt | yeah agreed | 15:47 |
| openstackgerrit | Jean-Philippe Evrard proposed stackforge/os-ansible-deployment: More complete explanation of availability zones https://review.openstack.org/219751 | 15:49 |
| openstackgerrit | Jean-Philippe Evrard proposed stackforge/os-ansible-deployment: More complete explanation of availability zones https://review.openstack.org/219751 | 15:49 |
| *** markvoelker has quit IRC | 16:03 | |
| openstackgerrit | Matt Thompson proposed stackforge/os-ansible-deployment: [WIP] Remove unnecessary tasks in neutron_db_setup.yml https://review.openstack.org/219759 | 16:03 |
| *** k_stev has quit IRC | 16:10 | |
| openstackgerrit | Matthew Kassawara proposed stackforge/os-ansible-deployment-specs: Add spec for the Liberty cycle upgrade path https://review.openstack.org/207713 | 16:11 |
| *** shoutm has joined #openstack-ansible | 16:12 | |
| *** devlaps has joined #openstack-ansible | 16:16 | |
| *** cloudtra_ has joined #openstack-ansible | 16:25 | |
| *** cloudtrainme has quit IRC | 16:27 | |
| *** woodard has quit IRC | 16:28 | |
| *** cloudtra_ has quit IRC | 16:32 | |
| *** galstrom is now known as galstrom_zzz | 16:33 | |
| *** cloudtrainme has joined #openstack-ansible | 16:34 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Set the Ubuntu mirror used based on the environment https://review.openstack.org/218611 | 16:34 |
| *** woodard has joined #openstack-ansible | 16:37 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Set the Ubuntu mirror used based on the environment https://review.openstack.org/218611 | 16:42 |
| *** cloudtrainme has quit IRC | 16:45 | |
| odyssey4me | mattt did you get an AIO up to test maas for https://review.openstack.org/217098 ? | 16:55 |
| mhayden | well, dangling veths came back to get me once the box started swapping | 16:57 |
| mhayden | sigh | 16:57 |
| *** gparaskevas has joined #openstack-ansible | 17:04 | |
| *** cloudtrainme has joined #openstack-ansible | 17:06 | |
| *** phalmos has quit IRC | 17:17 | |
| *** woodard has quit IRC | 17:18 | |
| *** jbweber has joined #openstack-ansible | 17:28 | |
| *** cloudtrainme has quit IRC | 17:29 | |
| *** javeriak has quit IRC | 17:32 | |
| *** markvoelker has joined #openstack-ansible | 17:35 | |
| *** markvoelker has quit IRC | 17:35 | |
| *** markvoelker has joined #openstack-ansible | 17:36 | |
| *** javeriak has joined #openstack-ansible | 17:38 | |
| *** galstrom_zzz is now known as galstrom | 17:45 | |
| *** javeriak_ has joined #openstack-ansible | 17:46 | |
| *** woodard has joined #openstack-ansible | 17:47 | |
| *** javeriak has quit IRC | 17:48 | |
| *** k_stev has joined #openstack-ansible | 17:54 | |
| *** cloudtrainme has joined #openstack-ansible | 18:00 | |
| *** markvoelker has quit IRC | 18:05 | |
| *** woodard has quit IRC | 18:07 | |
| *** sdake_ has joined #openstack-ansible | 18:13 | |
| *** sdake has quit IRC | 18:17 | |
| *** sdake has joined #openstack-ansible | 18:21 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Implement tox.ini config for bashate and pep8 tests https://review.openstack.org/216170 | 18:23 |
| *** jwagner_away is now known as jwagner | 18:24 | |
| *** sdake_ has quit IRC | 18:25 | |
| *** javeriak has joined #openstack-ansible | 18:30 | |
| *** phalmos has joined #openstack-ansible | 18:30 | |
| *** javeriak_ has quit IRC | 18:31 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: [OSAD doc] Update link to Cloud Admin Guide https://review.openstack.org/219085 | 18:39 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: [OSAD doc] Update link to Cloud Admin Guide https://review.openstack.org/219085 | 18:39 |
| *** phalmos has quit IRC | 18:39 | |
| *** phalmos has joined #openstack-ansible | 18:41 | |
| *** sdake_ has joined #openstack-ansible | 18:41 | |
| *** woodard has joined #openstack-ansible | 18:43 | |
| *** sdake has quit IRC | 18:45 | |
| *** ybabenko has joined #openstack-ansible | 18:47 | |
| *** woodard has quit IRC | 18:48 | |
| *** harlowja has quit IRC | 18:54 | |
| *** harlowja has joined #openstack-ansible | 18:58 | |
| *** markvoelker has joined #openstack-ansible | 19:00 | |
| *** phalmos has quit IRC | 19:01 | |
| *** ybabenko has quit IRC | 19:02 | |
| *** aslaen has joined #openstack-ansible | 19:09 | |
| *** phalmos has joined #openstack-ansible | 19:12 | |
| mhayden | so after digging through lxc code, and some fun debugging, it appears that LXC simply sends a halt signal to the init process within the LXC container | 19:14 |
| mhayden | and it expects the system to clean up after the container | 19:14 |
| *** mgariepy has left #openstack-ansible | 19:17 | |
| *** mgariepy has joined #openstack-ansible | 19:17 | |
| *** aslaen has quit IRC | 19:25 | |
| *** Mudpuppy has quit IRC | 19:28 | |
| *** Mudpuppy has joined #openstack-ansible | 19:31 | |
| *** metral is now known as metral_zzz | 19:31 | |
| *** Mudpuppy has quit IRC | 19:33 | |
| *** metral_zzz is now known as metral | 19:33 | |
| *** Mudpuppy has joined #openstack-ansible | 19:35 | |
| *** Mudpuppy has quit IRC | 19:38 | |
| *** Mudpuppy has joined #openstack-ansible | 19:38 | |
| *** jbweber has quit IRC | 19:41 | |
| *** woodard has joined #openstack-ansible | 19:50 | |
| *** javeriak_ has joined #openstack-ansible | 19:58 | |
| *** javeriak has quit IRC | 19:58 | |
| *** jmckind has joined #openstack-ansible | 20:04 | |
| *** alop has joined #openstack-ansible | 20:10 | |
| *** ybabenko has joined #openstack-ansible | 20:10 | |
| *** cloudtrainme has quit IRC | 20:11 | |
| *** Mudpuppy has quit IRC | 20:14 | |
| *** Mudpuppy has joined #openstack-ansible | 20:14 | |
| *** sdake_ is now known as sdake | 20:15 | |
| *** gparaskevas has quit IRC | 20:17 | |
| *** markvoelker has quit IRC | 20:21 | |
| *** cloudtrainme has joined #openstack-ansible | 20:25 | |
| *** galstrom is now known as galstrom_zzz | 20:26 | |
| *** Mudpuppy has quit IRC | 20:29 | |
| *** markvoelker has joined #openstack-ansible | 20:37 | |
| *** Mudpuppy has joined #openstack-ansible | 20:41 | |
| *** harlowja has quit IRC | 20:45 | |
| *** k_stev has quit IRC | 20:46 | |
| *** markvoelker has quit IRC | 20:48 | |
| *** Mudpuppy has quit IRC | 20:53 | |
| *** javeriak_ has quit IRC | 21:01 | |
| *** harlowja has joined #openstack-ansible | 21:21 | |
| *** woodard has quit IRC | 21:27 | |
| *** cloudtrainme has quit IRC | 21:36 | |
| *** aslaen has joined #openstack-ansible | 21:50 | |
| *** jmckind has quit IRC | 21:54 | |
| *** jaypipes has quit IRC | 22:10 | |
| *** shoutm has quit IRC | 22:14 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 22:23 | |
| *** aslaen has quit IRC | 22:26 | |
| *** phalmos has quit IRC | 22:35 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Switch Nova/Tempest to use/test Cinder API v2 https://review.openstack.org/214045 | 22:54 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Fixing haproxy-playbook fails when installing on multiple hosts https://review.openstack.org/215579 | 22:55 |
| *** ybabenko has quit IRC | 22:55 | |
| *** spotz is now known as spotz_zzz | 23:06 | |
| *** arbrandes1 has joined #openstack-ansible | 23:10 | |
| *** arbrandes has quit IRC | 23:11 | |
| *** sdake has quit IRC | 23:15 | |
| *** sdake has joined #openstack-ansible | 23:27 | |
| *** alop has quit IRC | 23:30 | |
| *** arbrandes1 has quit IRC | 23:36 | |
| *** harlowja has quit IRC | 23:42 | |
| *** harlowja has joined #openstack-ansible | 23:43 | |
| *** markvoelker has joined #openstack-ansible | 23:46 | |
| *** shoutm has joined #openstack-ansible | 23:49 | |
| *** arbrandes1 has joined #openstack-ansible | 23:54 | |
| *** markvoelker has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!