| *** Bjoern has joined #openstack-ansible | 00:01 | |
| *** harlowja has left #openstack-ansible | 00:01 | |
| *** baker has joined #openstack-ansible | 00:06 | |
| *** daneyon_ has joined #openstack-ansible | 00:06 | |
| *** Bjoern is now known as Bjoern_zZzZzZzZ | 00:09 | |
| *** sdake has quit IRC | 00:09 | |
| *** daneyon has quit IRC | 00:09 | |
| *** sdake has joined #openstack-ansible | 00:10 | |
| *** markvoelker has quit IRC | 00:11 | |
| *** daneyon_ has quit IRC | 00:11 | |
| *** darrenc_afk is now known as darrenc | 00:12 | |
| *** Bjoern_zZzZzZzZ has quit IRC | 00:15 | |
| *** phalmos has quit IRC | 00:15 | |
| *** phalmos has joined #openstack-ansible | 00:16 | |
| *** phalmos has quit IRC | 00:16 | |
| *** baker has quit IRC | 00:18 | |
| *** markvoelker has joined #openstack-ansible | 00:18 | |
| *** yarkot has joined #openstack-ansible | 00:29 | |
| *** weezS has quit IRC | 00:36 | |
| *** automagically has quit IRC | 00:38 | |
| *** jaypipes has quit IRC | 00:41 | |
| *** baker has joined #openstack-ansible | 00:47 | |
| *** baker has quit IRC | 00:49 | |
| *** jmccrory has quit IRC | 00:52 | |
| *** jmccrory has joined #openstack-ansible | 00:52 | |
| *** eil397 has quit IRC | 00:58 | |
| *** jlvillal has quit IRC | 00:58 | |
| *** daneyon has joined #openstack-ansible | 01:02 | |
| *** jlvillal has joined #openstack-ansible | 01:02 | |
| *** karimb has quit IRC | 01:04 | |
| *** markvoelker has quit IRC | 01:09 | |
| *** challenge has joined #openstack-ansible | 01:16 | |
| *** sdake has quit IRC | 01:17 | |
| *** kjelly has quit IRC | 01:17 | |
| *** yarkot_ has joined #openstack-ansible | 01:18 | |
| *** sdake has joined #openstack-ansible | 01:28 | |
| *** yarkot_ has quit IRC | 01:29 | |
| *** dslev has joined #openstack-ansible | 01:30 | |
| *** dslev_ has joined #openstack-ansible | 01:32 | |
| *** sdake has quit IRC | 01:34 | |
| *** dslev has quit IRC | 01:35 | |
| *** kjelly has joined #openstack-ansible | 01:42 | |
| *** Bjoern has joined #openstack-ansible | 01:45 | |
| *** dslev_ has quit IRC | 01:55 | |
| *** Bjoern has quit IRC | 01:58 | |
| *** baker has joined #openstack-ansible | 02:02 | |
| *** except has joined #openstack-ansible | 02:59 | |
| *** except is now known as Guest3765 | 03:00 | |
| *** challenge has quit IRC | 03:02 | |
| *** sdake has joined #openstack-ansible | 03:06 | |
| *** sdake_ has joined #openstack-ansible | 03:12 | |
| *** sdake has quit IRC | 03:14 | |
| *** v1k0d3n has quit IRC | 03:16 | |
| *** severion has joined #openstack-ansible | 03:16 | |
| *** raddaoui has joined #openstack-ansible | 03:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Kilo SHAs - 21 Jan 2016 https://review.openstack.org/265491 | 03:31 |
|---|---|---|
| *** Guest3765 has quit IRC | 03:46 | |
| *** v1k0d3n has joined #openstack-ansible | 03:46 | |
| *** severion has quit IRC | 03:48 | |
| *** raddaoui has quit IRC | 03:52 | |
| *** tlian has quit IRC | 03:53 | |
| *** daneyon has quit IRC | 04:01 | |
| *** raddaoui has joined #openstack-ansible | 04:18 | |
| *** agireud has quit IRC | 04:20 | |
| *** agireud has joined #openstack-ansible | 04:22 | |
| *** raddaoui has quit IRC | 04:28 | |
| *** baker has quit IRC | 04:35 | |
| *** fawadkhaliq has joined #openstack-ansible | 04:56 | |
| *** sdake_ has quit IRC | 05:14 | |
| *** raddaoui has joined #openstack-ansible | 05:27 | |
| *** phiche has joined #openstack-ansible | 05:29 | |
| openstackgerrit | Merged openstack/openstack-ansible: Update Kilo SHAs - 21 Jan 2016 https://review.openstack.org/265491 | 05:37 |
| *** raddaoui has quit IRC | 05:37 | |
| *** markvoelker has joined #openstack-ansible | 05:44 | |
| *** javeriak has joined #openstack-ansible | 05:49 | |
| *** phiche has quit IRC | 05:58 | |
| *** hybridpollo has quit IRC | 06:02 | |
| *** markvoelker_ has joined #openstack-ansible | 06:04 | |
| *** phiche has joined #openstack-ansible | 06:05 | |
| *** markvoelker has quit IRC | 06:07 | |
| *** phiche has quit IRC | 06:21 | |
| *** phiche has joined #openstack-ansible | 06:23 | |
| *** javeriak_ has joined #openstack-ansible | 06:32 | |
| *** javeriak has quit IRC | 06:32 | |
| *** javeriak has joined #openstack-ansible | 06:39 | |
| *** bryan_att has quit IRC | 06:39 | |
| *** javeriak_ has quit IRC | 06:40 | |
| *** elo has joined #openstack-ansible | 06:57 | |
| *** mpavone has joined #openstack-ansible | 07:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Fixes keystone library incompatibility with the Keystone v3 client https://review.openstack.org/270298 | 07:32 |
| javeriak | hey folks; odyssey4me, for some reason the /var/cache/lxc/ is empty on my hosts; do the playbooks copy over the trusty image? so my setup hosts keep failing | 07:39 |
| javeriak | basically this: msg: Destination directory /var/cache/lxc/trusty/rootfs-amd64/etc/apt does not exist | 07:40 |
| *** mpavone has quit IRC | 07:48 | |
| *** mpavone has joined #openstack-ansible | 07:51 | |
| *** mpavone has quit IRC | 07:52 | |
| mattt | javeriak: replicated, having a look | 08:04 |
| mattt | javeriak: think you need to remove /var/cache/lxc_trusty.tgz, just testing now | 08:11 |
| javeriak | mattt okay, yea thats the only thing in that dir | 08:12 |
| mattt | javeriak: that's not good, that dir shouldn't be empty :P | 08:12 |
| mattt | javeriak: https://gist.github.com/mattt416/a192702638248c418cc9 | 08:13 |
| javeriak | oh man how did that happen | 08:13 |
| javeriak | all i know is that these are fresh hosts with ubuntu 14.0.4.3 installed | 08:14 |
| mattt | javeriak: well, see what happens ... maybe dirs get created on an as-needed basis | 08:15 |
| mattt | but not sure how you got lxc_trusty.tgz but not /var/cache/lxc | 08:15 |
| mattt | anyway removing that tarball and rerunning setup-hosts will do what it needs to do | 08:15 |
| javeriak | mattt yes this is wierd, where does the cache come from in a fresh install; the lxc package i dont think? | 08:15 |
| mattt | javeriak: the cache gets downloaded through the role, so you wouldn't see it on a fresh install until you've deployed ansible | 08:16 |
| javeriak | mattt removing the /var/cache/lxc_trusty.tgz worked, its moved forward; thanks | 08:28 |
| mattt | javeriak: cool, np | 08:30 |
| *** markvoelker_ has quit IRC | 08:38 | |
| *** adac has joined #openstack-ansible | 08:39 | |
| adac | Good morning channel! | 08:39 |
| mattt | adac: howdy! | 08:39 |
| *** javeriak has quit IRC | 08:42 | |
| *** karimb has joined #openstack-ansible | 08:44 | |
| mattt | odyssey4me: any luck w/ bumping keystone servers down to 1 ? | 08:46 |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: config_template: ensure all overides are str https://review.openstack.org/271187 | 08:49 |
| *** javeriak has joined #openstack-ansible | 08:53 | |
| *** admin0 has joined #openstack-ansible | 08:58 | |
| odyssey4me | mattt yep, it pass first time - I kicked off a recheck this morning to validate the finding | 09:07 |
| odyssey4me | but it seems that's an avenue worth looking into | 09:08 |
| mattt | very weird but hey we need to find something :P | 09:10 |
| odyssey4me | hughsaunders https://review.openstack.org/#/q/project:%255Eopenstack/openstack-ansible.*+status:open+starredby:jesse-pretorius | 09:24 |
| odyssey4me | mattt can you look into it? I'm well behind on finalising organisation of the mid cycle | 09:26 |
| mattt | odyssey4me: i've looked into it already, not sure what else i can do at this point | 09:27 |
| mattt | odyssey4me: oh you mean the keystone thing specifically? sure i can explore that | 09:27 |
| odyssey4me | mattt yeah, the keystone thing - it's clearly something to do with the caching... and seems to be something that has changed between liberty and master | 09:29 |
| *** markvoelker has joined #openstack-ansible | 09:34 | |
| *** markvoelker has quit IRC | 09:39 | |
| *** javeriak has quit IRC | 09:39 | |
| *** markvoelker has joined #openstack-ansible | 09:39 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Ceilometer paste file https://review.openstack.org/270807 | 09:44 |
| *** markvoelker has quit IRC | 09:44 | |
| *** markvoelker has joined #openstack-ansible | 09:45 | |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Update plugin for and document per host overrides https://review.openstack.org/266088 | 09:45 |
| *** markvoelker has quit IRC | 09:50 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use slurp to collect the nova ssh keys https://review.openstack.org/266839 | 09:52 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use slurp to collect the swift ssh keys https://review.openstack.org/266840 | 09:52 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use slurp to collect the keystone ssh keys https://review.openstack.org/266842 | 09:53 |
| mattt | openstackgerrit: annoyingly the last 2 AIOs i built didn't run into taht issue :-/ | 09:53 |
| mattt | ^^^ odyssey4me | 09:53 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update plugin for and document per host overrides https://review.openstack.org/266088 | 09:53 |
| odyssey4me | mattt :( | 09:55 |
| *** javeriak has joined #openstack-ansible | 09:55 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Liberty SHAs - 22 Jan 2016 https://review.openstack.org/271212 | 10:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Kilo SHAs - 22 Jan 2016 https://review.openstack.org/271215 | 10:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Neutron ML2 template fix https://review.openstack.org/269394 | 10:11 |
| adac | /j #openvpn | 10:12 |
| adac | sry | 10:12 |
| openstackgerrit | Merged openstack/openstack-ansible: Added a variable so pkg-locations to be redefined https://review.openstack.org/269760 | 10:23 |
| openstackgerrit | Merged openstack/openstack-ansible: Fix HAProxy config docs https://review.openstack.org/269833 | 10:23 |
| openstackgerrit | Merged openstack/openstack-ansible: Fix an invalid YAML example in the Keystone Federation Service provider doc https://review.openstack.org/268266 | 10:23 |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Use http request to check cinder api availability https://review.openstack.org/256414 | 10:32 |
| *** markvoelker has joined #openstack-ansible | 10:35 | |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Fix defcore test list function https://review.openstack.org/270138 | 10:36 |
| *** markvoelker has quit IRC | 10:39 | |
| *** sdake has joined #openstack-ansible | 10:48 | |
| openstackgerrit | Darren Birkett proposed openstack/openstack-ansible: Added a variable so pkg-locations to be redefined https://review.openstack.org/271233 | 10:50 |
| *** electrofelix has joined #openstack-ansible | 11:09 | |
| *** sdake_ has joined #openstack-ansible | 11:12 | |
| *** markvoelker has joined #openstack-ansible | 11:12 | |
| *** sdake has quit IRC | 11:13 | |
| *** markvoelker has quit IRC | 11:18 | |
| *** andrei__ has joined #openstack-ansible | 11:18 | |
| *** fawadkhaliq has quit IRC | 11:22 | |
| *** fawadkhaliq has joined #openstack-ansible | 11:23 | |
| andrei__ | Hi, can someone help with nova_rpc_backend setting. Why it is set to 'nova.openstack.common.rpc.impl_kombu' in Liberty OA insted of 'rabbit'? | 11:30 |
| *** fawadkhaliq has quit IRC | 11:41 | |
| odyssey4me | hughsaunders your thoughts on https://review.openstack.org/269394 ? | 11:43 |
| odyssey4me | andrei__ looking now... | 11:45 |
| javeriak | guys my openstack keystone task is trying to install sync_3.1.0-2ubuntu0.2_amd64.deb which isnt on the rackspace mirror anymore, im installing with 11.2.7; how to do i update my requirements? | 11:46 |
| *** openstackgerrit has quit IRC | 11:47 | |
| odyssey4me | javeriak we don't pin any debs except the rabbitmq debs... so all you should have to do is 'apt-get update' to refresh your apt cache to get the current version from the mirror you're using | 11:47 |
| *** openstackgerrit has joined #openstack-ansible | 11:48 | |
| javeriak | odyssey4me let me try | 11:48 |
| *** fawadkhaliq has joined #openstack-ansible | 11:48 | |
| odyssey4me | javeriak it's a bit odd, because the plays normally do that for you before trying to install anything | 11:48 |
| odyssey4me | unless you happen to have implemented your own pins | 11:48 |
| javeriak | odyssey4me yea i know; but for some reason all my keystone containers are failing out on this exact package | 11:49 |
| javeriak | odyssey4me nope, no pins | 11:49 |
| *** fawadkhaliq has quit IRC | 11:49 | |
| *** fawadkhaliq has joined #openstack-ansible | 11:50 | |
| odyssey4me | andrei__ that's a bug - nice catch :) | 11:50 |
| odyssey4me | andrei__ would you care to either register a bug, or simply submit a patch to our master branch? | 11:50 |
| odyssey4me | javeriak it might be that the mirrors are busy updating | 11:51 |
| odyssey4me | typically around midday they do, which results in apt get failures | 11:51 |
| odyssey4me | the options are to change to use a different mirror, or to wait 30 mins or so for the update to complete :/ | 11:51 |
| javeriak | odyssey4me that would explain it; im getting HttpError404 errors, and apt-get update didnt fix it: Err https://mirror.rackspace.com/ubuntu/ trusty-security/main rsync amd64 3.1.0-2ubuntu0.2 | 11:52 |
| javeriak | okay :( | 11:52 |
| odyssey4me | yup, that's because of the mirror updating | 11:53 |
| javeriak | odyssey4me ok thanks, i'll try again after a while | 11:53 |
| hughsaunders | odyssey4me: looks ok, pattern already in plance for vxlan | 11:54 |
| hughsaunders | *place | 11:54 |
| *** fawadkhaliq has quit IRC | 11:54 | |
| odyssey4me | hughsaunders matt https://review.openstack.org/271215 is to bring https://review.openstack.org/270234 into the repo before we tag later today | 12:00 |
| odyssey4me | mattt ^ | 12:01 |
| *** markvoelker has joined #openstack-ansible | 12:07 | |
| odyssey4me | oh, and hughsaunders the backport is ready for a vote :) https://review.openstack.org/271233 | 12:09 |
| *** markvoelker has quit IRC | 12:12 | |
| mattt | odyssey4me: made a slight bit of progress, i can see that the swift and aodh endpoints aren't getting added, the rest are visible | 12:16 |
| mattt | those are presumably the last two that get created, so not sure why they sometimes don't appear when requesting the catalog | 12:17 |
| *** javeriak has quit IRC | 12:19 | |
| *** javeriak has joined #openstack-ansible | 12:21 | |
| *** sdake_ has quit IRC | 12:27 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** openstackgerrit has joined #openstack-ansible | 12:33 | |
| *** markvoelker has joined #openstack-ansible | 12:42 | |
| *** jaypipes has joined #openstack-ansible | 12:45 | |
| *** jaypipes is now known as leakypipes | 12:45 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Keystone troubleshooting https://review.openstack.org/271278 | 12:50 |
| *** markvoelker has quit IRC | 12:54 | |
| mattt | odyssey4me: don't you usually bump openstack_release when you bump SHAs ? | 12:56 |
| *** sdake has joined #openstack-ansible | 12:58 | |
| openstackgerrit | Ice Yao proposed openstack/openstack-ansible: Update ceph repo https://review.openstack.org/269151 | 13:03 |
| *** javeriak has quit IRC | 13:09 | |
| *** javeriak has joined #openstack-ansible | 13:10 | |
| *** karimb has quit IRC | 13:11 | |
| *** karimb has joined #openstack-ansible | 13:13 | |
| mattt | lbragstad : ping | 13:14 |
| odyssey4me | mattt normally, yes - but in this case no need - it's a fast follow sha bump with no-one depending on it yet | 13:21 |
| mattt | odyssey4me: https://review.openstack.org/#/c/215212/ | 13:22 |
| odyssey4me | mattt hmm, we'll have to catch up with lbragstad and dolphm when they come online | 13:25 |
| mattt | odyssey4me: that makes a lot of sense tho | 13:26 |
| mattt | odyssey4me: because the caching is specific to user, and it seems liek only the glance service user is affected for some reason | 13:26 |
| mattt | which is why nothing else seems to fail | 13:26 |
| *** admin0 has quit IRC | 13:27 | |
| mattt | so it looks like something isn't getting invalidated correctly | 13:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update nova_rpc_backend to correct setting https://review.openstack.org/271285 | 13:33 |
| odyssey4me | mattt yup, that does make sense to me - the validation isn't being done properly to ensure that the updated catalogue is being exposed to that user | 13:34 |
| odyssey4me | mattt re: https://review.openstack.org/271215 - normally, yes - but in this case no need - it's a fast follow sha bump with no-one depending on it yet | 13:35 |
| odyssey4me | mattt it seems that perhaps that cache isn't pushed into the cache backend | 13:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update nova_rpc_backend to correct setting https://review.openstack.org/271285 | 13:37 |
| *** markvoelker has joined #openstack-ansible | 13:38 | |
| dolphm | odyssey4me: mattt: reading back.. | 13:39 |
| dolphm | mattt: 3 keystone nodes, and all 3 are returning a stale catalog for the glance user? | 13:41 |
| *** markvoelker has quit IRC | 13:43 | |
| mattt | dolphm: the AIO gate has 2 keystone nodes, everything in the build works fine but we see about 1/2 of the time uploading an image to glance (backed by swift) fails | 13:43 |
| mattt | dolphm: the error is because the catalog is missing the object-store endpoint | 13:44 |
| dolphm | mattt: can you verify the token against both keystone nodes and see if you get a different response from one? | 13:44 |
| dolphm | mattt: also, what's the cache timeout? | 13:44 |
| dolphm | mattt: in keystone.conf | 13:44 |
| mattt | dolphm: expiration_time is 5400 | 13:45 |
| dolphm | mattt: that's a long time | 13:45 |
| mattt | dolphm: i can check for inconsistent results but it always fails in that exact specific spot, and i'd imagine if the servers were out of sync we'd see issues elsewhere | 13:46 |
| dolphm | mattt: has it been longer than 90 minutes since you saw this issue? | 13:46 |
| dolphm | first* saw this issue | 13:46 |
| mattt | dolphm: well the failure happens at the end, so yeah it's probably just teetering over 90 mins | 13:47 |
| mattt | let me double-check | 13:47 |
| *** tlian has joined #openstack-ansible | 13:47 | |
| dolphm | mattt: and does each keystone node have it's own memcache instance? or are they sharing? | 13:48 |
| mattt | dolphm: actually i have two failures and both happened < 90 mins | 13:49 |
| mattt | dolphm: both are sharing a single instancw | 13:49 |
| mattt | dolphm: how does the caching work, if the catalog is updated does it update the cache or invalidate it? | 13:49 |
| mattt | according ot that patch it should invalidate it right | 13:50 |
| *** woodard has joined #openstack-ansible | 13:51 | |
| *** sdake has quit IRC | 13:51 | |
| dolphm | mattt: IIRC, it'll update the shared cache with new values on create / update, but invalidation is per keystone instance as keystone (dogpile.cache) doesn't write to memcache on invalidation | 13:51 |
| odyssey4me | #success The OpenStack-Ansible Mid Cycle details have been announced on the openstack-dev mailing list: http://lists.openstack.org/pipermail/openstack-dev/2016-January/084716.html | 13:52 |
| openstackstatus | odyssey4me: Added success to Success page | 13:52 |
| dolphm | mattt: the best solution is to reduce the cache expiration to something you're willing to tolerate | 13:52 |
| dolphm | mattt: i think keystone's default is 5 minutes - 90 is a radical change | 13:53 |
| mhayden | ohai | 13:53 |
| mattt | dolphm: cool, i'll give that a shot ... 5 mins sounds more reasonable | 13:53 |
| mattt | dolphm: one more question for you | 13:53 |
| dolphm | mattt: i'd be in favor of anything in the range of 1-15 min without having a use case to go higher / lower | 13:53 |
| *** woodard has quit IRC | 13:54 | |
| kysse | cloudnull: sorry, it's not totally done yet (testing it right now), but we will write something when we are done :-) | 13:54 |
| odyssey4me | dolphm reducing the cache time is perfectly fine, but these actions are happening in under that timeframe... ie the catalogue entry is added and the wrong one given back to the client within 5 mins | 13:54 |
| mattt | dolphm: we front keystone w/ apache (wsgi), should we be using dogpile.cache.memcached or keystone.cache.memcache_pool? | 13:54 |
| *** woodard has joined #openstack-ansible | 13:54 | |
| mhayden | if someone has a moment for a small auditd rule change review, i'd be much obliged -> https://review.openstack.org/#/c/270421/ | 13:54 |
| mattt | dolphm: i'll look at commit messages to see if there is any particular reason it's so high, if i can't find anythign i will use the upstream default | 13:55 |
| dolphm | odyssey4me: after expiration_time passes (90 minutes), the cache should "catch back up" | 13:55 |
| mattt | actually looking i have a build that succeeded under 90 mins | 13:55 |
| dolphm | mattt: good question on dogpile.cache.memcached or keystone.cache.memcache_pool - let me double check | 13:56 |
| mattt | dolphm: there is a nice warning about dogpile in http://docs.openstack.org/developer/keystone/configuration.html | 13:56 |
| mattt | "Do not use dogpile.cache.memcached backend if you are deploying Keystone under eventlet. There are known issues with the use of thread.local under eventlet that can allow the leaking of memcache client objects and consumption of extra sockets." | 13:56 |
| mattt | does it use eventlet when running through wsgi? | 13:56 |
| dolphm | mattt: dogpile.cache.memcached it is! | 13:57 |
| dolphm | mattt: no, the apache httpd + mod_wsgi combo is an alternative to eventlet | 13:57 |
| mattt | alright cool | 13:58 |
| *** admin0 has joined #openstack-ansible | 13:58 | |
| mattt | yeah dropping to 5 mins may not work | 13:58 |
| mattt | hrrmph. | 13:58 |
| odyssey4me | dolphm surely if a new catalogue entry is added, any existing cache entries should be invalidated | 13:59 |
| *** automagically has joined #openstack-ansible | 13:59 | |
| *** markvoelker has joined #openstack-ansible | 13:59 | |
| *** markvoelker_ has joined #openstack-ansible | 14:01 | |
| *** markvoelker has quit IRC | 14:04 | |
| mattt | odyssey4me: 5 mins may work, glance presumably gets teh catalog when it's installed, which happens a fair bit before we install swift and run tempest | 14:05 |
| odyssey4me | mattt sure, I think out timeout is probably way too long anyway - and is most likely a very old default | 14:06 |
| *** dslev_ has joined #openstack-ansible | 14:06 | |
| mattt | odyssey4me: i'm still confused why it's been working w/ keystone scaled down to a single instance tho :-/ | 14:06 |
| odyssey4me | mattt I'd say that we should reduce the default to 5 mins, and if that's not enough then we can override that default for the gate to something more like 30 secs? | 14:06 |
| odyssey4me | mattt agreed, it points to more of the idea that the cache is in-memory and hasn't been flushed | 14:07 |
| odyssey4me | or perhaps the use of the back-end cache is not working at all | 14:07 |
| *** adac has quit IRC | 14:07 | |
| mattt | odyssey4me: so i added a wip to log cache requests | 14:09 |
| mattt | odyssey4me: once that gates we can validate if it's actually working or not | 14:10 |
| dolphm | mattt: one of your keystone instances should be aware of the new catalog, the other isn't (until 90 minutes passes) | 14:11 |
| dolphm | mattt: you could test by verifying the glance token against both nodes separately (skip the LB), and see that the responses are different | 14:11 |
| mattt | dolphm: so memcache isn't used here when querying catalog? | 14:11 |
| *** Bjoern has joined #openstack-ansible | 14:12 | |
| dolphm | mattt: it is | 14:12 |
| *** javeriak has quit IRC | 14:12 | |
| mattt | dolphm: wouldn't i expect they both return the same thing then? | 14:13 |
| odyssey4me | but when memcache is updated, keystone doesn't pickup the change until the current cache expires | 14:13 |
| dolphm | mattt: it's premature cache invalidation that is not shared | 14:13 |
| dolphm | mattt: odyssey4me: i'm digging for dogpile docs to explain better than i - give me a few minutes | 14:14 |
| mattt | dolphm: sorry, i'm slow :P | 14:15 |
| *** KLevenstein has joined #openstack-ansible | 14:15 | |
| dolphm | i haven't actually run into this behavior, but i've read about it. also going to try and produce a demo | 14:16 |
| odyssey4me | mattt I think for master we should just remove the expiration_time setting in our config template, so that the default from upstream is used - a deployer can use a config_override if they wish to adjust the default | 14:16 |
| mattt | odyssey4me: sure, but someone upgrading could get burned by that change ... how are we documenting these changes for upgrade notes? | 14:17 |
| odyssey4me | mattt we should, using the new releasenotes option :) | 14:17 |
| odyssey4me | mattt we could also disable catalog caching entirely, specifically for the AIO | 14:20 |
| automagically | I like the idea of disabling caching altogether for the gate, but not so sure about all AIOs | 14:23 |
| odyssey4me | mattt clear as mud? https://review.openstack.org/271311 | 14:24 |
| mattt | i'd rather the gate SOMEWHAT resemble what a prod deploy is going to do | 14:25 |
| mattt | and that would be enabling caching :) | 14:25 |
| odyssey4me | mattt lemme try and whip up a patch which includes a release note entry - we have yet to make use of reno, so it's about time we do it | 14:25 |
| *** adac has joined #openstack-ansible | 14:25 | |
| mattt | odyssey4me: yeah i was about to ask how that is done :P | 14:25 |
| mattt | i was tracing back the 5400 invalidation time | 14:26 |
| mattt | it goes back very far | 14:26 |
| odyssey4me | mattt I'll make notes, then add a dev doc for how to do it :) | 14:26 |
| mattt | odyssey4me: ok, i'll prep the keystone change for now | 14:26 |
| mattt | then update it with release notes once i know how | 14:26 |
| mattt | unless you want to do that for testing releasenotes | 14:27 |
| odyssey4me | mattt go for it - we may as well know whether it helps resolve the issue or not | 14:27 |
| odyssey4me | 'cos either we fall back to the upstream default, or we do that and also set a lower value for the gate | 14:28 |
| odyssey4me | (or disable caching entirely for the gate) | 14:28 |
| mattt | automagically: we are also considering dropping # of galera/rabbit/keystone containers we deploy in the gate AIO -- i fully stand by that consideration but i don't want us to do it simply when we can't figure out what a specific gate failure is caused by | 14:28 |
| automagically | mattt: Saw that and am completely supportive | 14:29 |
| dolphm | mattt: odyssey4me: run memcached and then run this script: https://gist.github.com/dolph/04bd4984c7d1f34ef821 | 14:33 |
| dolphm | mattt: odyssey4me: this is the bit that is surprising https://gist.github.com/dolph/04bd4984c7d1f34ef821#file-cache_demo-py-L32-L34 | 14:33 |
| mattt | dolphm: one sec | 14:35 |
| *** sdake has joined #openstack-ansible | 14:37 | |
| *** pcaruana has joined #openstack-ansible | 14:37 | |
| mattt | odyssey4me: it defaults to 10 mins actually, so we may need to bump that down in gate | 14:38 |
| *** dslev_ has quit IRC | 14:39 | |
| mattt | dolphm: so you invalidate it in one region and it's still query-able in the remaining regions | 14:44 |
| mattt | (until it expires) | 14:44 |
| mattt | seems like an odd design to me :-/ | 14:44 |
| mattt | brb | 14:46 |
| mattt | actually odyssey4me | 14:46 |
| mattt | what is the process to override user_variables.yml in gate now? | 14:46 |
| mattt | odyssey4me: n/m i see it | 14:48 |
| *** dslev_ has joined #openstack-ansible | 14:52 | |
| dolphm | mattt: yeah, it's not intuitive because it's not writing to memcache on cache invalidation | 14:55 |
| *** markvoelker has joined #openstack-ansible | 14:57 | |
| dolphm | mattt: odyssey4me: "Invalidation works by setting a current timestamp (using time.time()) representing the “minimum creation time” for a value. Any retrieved value whose creation time is prior to this timestamp is considered to be stale. It does not affect the data in the cache in any way, and is also local to this instance of CacheRegion." http://dogpilecache.readthedocs.org/en/latest/api.html#dogpile.cache.region.Cache | 14:58 |
| dolphm | Region.invalidate | 14:58 |
| dolphm | http://dogpilecache.readthedocs.org/en/latest/api.html#dogpile.cache.region.CacheRegion.invalidate * | 14:58 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Doc: Remove 'indices and tables' and correctly spell OpenStack-Ansible https://review.openstack.org/271350 | 14:59 |
| *** Mudpuppy has joined #openstack-ansible | 15:01 | |
| *** markvoelker_ has quit IRC | 15:01 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Reduce keystone cache expiration time https://review.openstack.org/271357 | 15:02 |
| *** sdake_ has joined #openstack-ansible | 15:04 | |
| *** galstrom_zzz is now known as galstrom | 15:05 | |
| *** sdake has quit IRC | 15:06 | |
| *** galstrom is now known as galstrom_zzz | 15:07 | |
| *** spotz_zzz is now known as spotz | 15:08 | |
| mattt | dolphm: but get_or_create() should not retreive a stale value and re-request it right? | 15:09 |
| mattt | are we not using those methods? | 15:09 |
| odyssey4me | mattt so it would sem to me that we should apply the reduced cache time for an AIO generally, not just the gate - thoughts? | 15:11 |
| mattt | hmm, your example did use .get() which should do the same | 15:11 |
| mattt | odyssey4me: not really | 15:11 |
| mattt | odyssey4me: generally people aren't updating the catalog all the time | 15:11 |
| mattt | odyssey4me: so i think it's most applicable in the gate where we update it and then test it immediately | 15:12 |
| mattt | odyssey4me: i'd say if we update it in the AIO, we may as well just update it everywhere :) | 15:12 |
| odyssey4me | ok, fair enough | 15:12 |
| *** admin0 has quit IRC | 15:12 | |
| *** markvoelker has quit IRC | 15:12 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:13 | |
| mattt | odyssey4me: in one of the past cycles i thought we went through and reset all overrides to upstream defaults, wonder how this one was missed | 15:13 |
| mattt | think it was kilo right? | 15:13 |
| odyssey4me | mattt we've done that on a best effort basis - but I'd like to do a lot more of that | 15:13 |
| odyssey4me | we set a lot of things which we don't need to set just because we needed to have a var to override in Juno | 15:14 |
| odyssey4me | so I'd like to, in this cycle, clean anything out that doesn't have to be set - or at least make a good effor tto do so | 15:14 |
| *** baker has joined #openstack-ansible | 15:16 | |
| *** woodard has quit IRC | 15:17 | |
| dolphm | mattt: odyssey4me: BUG FOUND! but I don't think it'll address this specific issue. we're using .invalidate(key) instead of .delete(key) in a couple places. .invalidate() is only useful for invalidating an entire cache, not a specific key (we're not using the API correctly) | 15:18 |
| odyssey4me | woohoo! | 15:19 |
| mattt | dolphm: ah right, invalidate doesn't take a key :( | 15:19 |
| dolphm | mattt: +++ | 15:19 |
| dolphm | we make calls like this in a few places: self.get_service.invalidate(self, service_id) | 15:20 |
| mattt | dolphm: in the context of keystone w/ memcache backend, how do i imagine what a dogpile region is? | 15:20 |
| cloudnull | morning | 15:20 |
| spotz | morning | 15:20 |
| automagically | dolphm: Nice fine | 15:21 |
| mattt | dolphm: so if we have 4 cache threads each would connect to the cache backend, and that connection is essentially a region? | 15:21 |
| dolphm | mattt: threads should share a single region. processes will not. | 15:22 |
| dolphm | 4 processes with 4 threads each all using a single memcached instance - you'll have 4 cache regions. | 15:22 |
| mattt | i see, and 1 region doing an invalidation only invalidates that specific region | 15:23 |
| mattt | which explains why we didn't see this when we took our keystone container count to 1 | 15:23 |
| dolphm | mattt: right, and an invalidation treats the entire region as expired | 15:23 |
| dolphm | mattt: so, by adding an endpoint, we can't possibly know all the keys of catalogs that we've computed per user (because we don't have those user IDs when creating the new endpoint). so the best we can do is to treat the entire region as expired in that process. | 15:24 |
| dolphm | mattt: otherwise, we'd have to know all the user_ids that we've computed and cached catalogs for, then basically call catalog_cache_region.delete(user_id) for each of them | 15:25 |
| *** admin0 has joined #openstack-ansible | 15:30 | |
| mattt | dolphm: yeah updating your snippet to using delete makes it fail as you'd expect | 15:30 |
| odyssey4me | serverascode FYI http://lists.openstack.org/pipermail/openstack-dev/2016-January/084716.html | 15:31 |
| mattt | dolphm: is there a way to cascade that invalidation to all keystone instances? | 15:32 |
| *** iceyao has joined #openstack-ansible | 15:35 | |
| *** weezS has joined #openstack-ansible | 15:39 | |
| *** javeriak has joined #openstack-ansible | 15:42 | |
| *** jwagner is now known as jwagner_wfh | 15:42 | |
| openstackgerrit | David Wilde proposed openstack/openstack-ansible: Convert available and minimum disk sizes to bytes. https://review.openstack.org/271049 | 15:43 |
| *** markvoelker has joined #openstack-ansible | 15:44 | |
| dolphm | mattt: not that i'm aware of | 15:45 |
| *** jthorne has joined #openstack-ansible | 15:45 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Contributor Guidelines https://review.openstack.org/271399 | 15:46 |
| *** alextricity_r has joined #openstack-ansible | 15:47 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Dev Doc: Update Contributor Guidelines https://review.openstack.org/271399 | 15:48 |
| *** admin0 has quit IRC | 15:48 | |
| alextricity_r | morning | 15:48 |
| openstackgerrit | David Wilde proposed openstack/openstack-ansible: Convert available and minimum disk sizes to bytes. https://review.openstack.org/271049 | 15:50 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Reduce keystone cache expiration time https://review.openstack.org/271357 | 15:54 |
| *** fawadkhaliq has joined #openstack-ansible | 15:57 | |
| *** dslev_ has quit IRC | 15:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Doc: Update Contributor Guidelines https://review.openstack.org/271399 | 15:57 |
| *** permalac has joined #openstack-ansible | 16:01 | |
| openstackgerrit | Merged openstack/openstack-ansible: Update Ceilometer paste file https://review.openstack.org/270807 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible: Neutron ML2 template fix https://review.openstack.org/269394 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible: Update plugin for and document per host overrides https://review.openstack.org/266088 | 16:03 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible: Adding openrc_insecure variable into the group vars to make it available across roles. https://review.openstack.org/271406 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible: Added a variable so pkg-locations to be redefined https://review.openstack.org/271233 | 16:03 |
| openstackgerrit | Merged openstack/openstack-ansible: Update Kilo SHAs - 22 Jan 2016 https://review.openstack.org/271215 | 16:03 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-galera_client: Allow sourcing apt_key from ansible host or URL https://review.openstack.org/268303 | 16:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Implement tunable heat api and engine worker overrides https://review.openstack.org/264778 | 16:06 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: set unique galera server-id https://review.openstack.org/267054 | 16:06 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible-galera_server: Allow sourcing apt_key from ansible host or URL https://review.openstack.org/268302 | 16:06 |
| *** sdake_ has quit IRC | 16:07 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update Liberty SHAs - 22 Jan 2016 https://review.openstack.org/271212 | 16:08 |
| alextricity_r | Do we have any docs that go into detail on the hardware requirements for OSA or RPC? | 16:09 |
| *** adac has quit IRC | 16:10 | |
| sigmavirus24 | So I added a trio of variables (like the others) to openstack_services.yml (in playbooks/defaults/repo_packages/) but it's not cloning that service and building the wheel for me. Any ideas? | 16:11 |
| sigmavirus24 | alextricity_r: for AIOs or for actual deployments? | 16:11 |
| alextricity_r | sigmavirus24: Actual deployments | 16:11 |
| odyssey4me | alextricity_r how long is my piece of string? | 16:11 |
| sigmavirus24 | I would guess that the RPC docs are internal only | 16:11 |
| odyssey4me | surely you know, based on your expertise? | 16:11 |
| alextricity_r | odyssey4me: maybe 30-40 characters..but idk really | 16:12 |
| odyssey4me | :p basically the hardware required would be based on your architecture and how you choose to deploy... so that's a tough question | 16:12 |
| *** sdake has joined #openstack-ansible | 16:12 | |
| palendae | git-harry, neillc, mcarden: Here's a rough brain dump of my "run liberty code on top of a kilo install and see what breaks" experiment https://etherpad.openstack.org/p/openstack-ansible-kilo-liberty-upgrade-test2 | 16:13 |
| mcarden | Yep, got it palendae. Is the targeted 'before' SHA purposely before backports? | 16:13 |
| alextricity_r | odyssey4me: What did you mean by "How long is my piece of string?" | 16:14 |
| alextricity_r | is that an expression? | 16:14 |
| palendae | So the first "target" listed was a dumb mistake on my part; git checkout -b liberty just made a copy of my kilo branch at that SHA | 16:14 |
| palendae | So I went to the daebae... SHA for actual liberty | 16:15 |
| sigmavirus24 | nevermind the repo-packages question. I had misnamed a couple vars (destination instead of dest and repository instead of repo) | 16:15 |
| mcarden | Also palendae, I found that it's not enough to add a keystone_rabitmq_password to user_secrets to get going... | 16:15 |
| palendae | mcarden: Correct, we'll need a whole range | 16:16 |
| odyssey4me | alextricity_r basically I mean that just like you can't know how long a piece of string is that I have in my hand (you have insufficient information), there is no way that OSA can specify hardware requirements as the deployment architecture and the load catered for would be different for every environment | 16:16 |
| mcarden | palendae: okay | 16:16 |
| palendae | mcarden: I think low hanging fruit is to add this to the kilo -> liberty process https://github.com/openstack/openstack-ansible/blob/kilo/scripts/upgrade-utilities/playbooks/user-secrets-adjustments.yml | 16:16 |
| palendae | That seems like a logical thing to have for all version migrations, populating new secrets/variables | 16:17 |
| palendae | You'll also notice it runs fairly early in https://github.com/openstack/openstack-ansible/blob/kilo/scripts/run-upgrade.sh#L132 | 16:17 |
| mcarden | so pw-token-gen is up to date with all the things that need to be generated? | 16:18 |
| palendae | mcarden: Looks like it takes the exmaple user secrets, dumps the missing ones in to the "real" file, then pw-token-gen takes a pass at the file and populates any empty keys with values | 16:19 |
| mcarden | Right | 16:19 |
| alextricity_r | odyssey4me: That's definitely true. It would be best if I figure out what kind of load this environment would be used for. | 16:19 |
| palendae | mcarden: And pw-token-gen is our own thing; https://github.com/openstack/openstack-ansible/blob/kilo/scripts/pw-token-gen.py | 16:20 |
| palendae | Basically generating various kinds of secret things for you, if you don't define them, | 16:20 |
| mcarden | Yep - looking at it | 16:20 |
| odyssey4me | alextricity_r that said, it wouldn't hurt to have some sort of general design overview and hardware recommendations somewhere in the install guide | 16:21 |
| *** raddaoui has joined #openstack-ansible | 16:21 | |
| kysse | does any1 know how to get bonding troughput better with NFS | 16:22 |
| kysse | it's like when your destination ip and your source ip is the same it's not really working the way I want it to. | 16:22 |
| alextricity_r | odyssey4me: Right. The openstack docs have an example architecture that has general hardware requirements for the controller, compute, etc. | 16:22 |
| alextricity_r | I could *start* from there, but I wasn't sure if our containerized approach means our machines would need to be more muscular | 16:23 |
| mhayden | does anyone know why we force an apt-get update here? https://github.com/openstack/openstack-ansible/blob/master/scripts/run-playbooks.sh#L103-L112 | 16:24 |
| odyssey4me | alextricity_r I'd suggest putting something up for review, then we can debate/discuss in review. | 16:24 |
| odyssey4me | mhayden https://review.openstack.org/268784 | 16:25 |
| alextricity_r | That would be nice | 16:25 |
| mhayden | odyssey4me: you're too quick for me! | 16:25 |
| mhayden | odyssey4me: i gave you my strongest +1 | 16:26 |
| spotz | heheh | 16:27 |
| *** karimb has quit IRC | 16:30 | |
| *** iceyao has quit IRC | 16:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Arrange Release Notes similarly to the Documentation https://review.openstack.org/271420 | 16:31 |
| mhayden | odyssey4me: does reno populate those release notes? ^^ | 16:35 |
| mhayden | or are those ones that we would write? | 16:35 |
| odyssey4me | mhayden reno generates the skeleton file - you then edit the file to provide the content | 16:35 |
| odyssey4me | reno is then also used in the publishing | 16:35 |
| mhayden | got it -- i need to look at reno at some point | 16:35 |
| odyssey4me | the file is just a simple yaml file, so it's a simple way to add things without worrying about formatting | 16:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Doc: Remove 'indices and tables' and correctly spell OpenStack-Ansible https://review.openstack.org/271350 | 16:37 |
| *** galstrom_zzz is now known as galstrom | 16:38 | |
| *** daneyon has joined #openstack-ansible | 16:38 | |
| openstackgerrit | David Wilde proposed openstack/openstack-ansible: Convert available and minimum disk sizes to bytes. https://review.openstack.org/271049 | 16:39 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-galera_server: Added major version upgrade support https://review.openstack.org/271423 | 16:39 |
| *** mpavone has joined #openstack-ansible | 16:45 | |
| *** mgoddard_ has joined #openstack-ansible | 16:46 | |
| *** mgoddard has quit IRC | 16:48 | |
| *** Mudpuppy has quit IRC | 16:50 | |
| mcarden | palendae: In user-secrets-adjustments.yml, user_secrets.yml is referred to 4 times, once via a relative path and 3 times via absolute. Is that intentional? | 16:50 |
| *** Mudpuppy has joined #openstack-ansible | 16:50 | |
| palendae | mcarden: Dunno :p | 16:52 |
| palendae | mcarden: cloudnull wrote a bunch of that | 16:52 |
| *** mgoddard has joined #openstack-ansible | 16:56 | |
| *** mgoddard_ has quit IRC | 16:56 | |
| mhayden | liberty's HEAD is looking quite stable right now | 16:58 |
| * mhayden could be counting his chickens before they hatch | 16:58 | |
| palendae | Counting your VMs before they launch | 16:58 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Doc: Remove 'indices and tables' and correctly spell OpenStack-Ansible https://review.openstack.org/271350 | 16:59 |
| *** raddaoui has quit IRC | 17:00 | |
| *** mpavone has quit IRC | 17:02 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Doc: Update Contributor Guidelines https://review.openstack.org/271399 | 17:02 |
| *** spotz is now known as spotz_zzz | 17:09 | |
| *** phiche has quit IRC | 17:16 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Consolidating galera recovery documentation https://review.openstack.org/268264 | 17:18 |
| *** phiche has joined #openstack-ansible | 17:21 | |
| openstackgerrit | Merged openstack/openstack-ansible: Dev Doc: Link directly to the bootstrap-host role defaults file https://review.openstack.org/269565 | 17:21 |
| *** spotz_zzz is now known as spotz | 17:21 | |
| openstackgerrit | Merged openstack/openstack-ansible: Removing reference to bug which has been closed in Kilo and beyond https://review.openstack.org/268231 | 17:21 |
| openstackgerrit | Merged openstack/openstack-ansible: Dev Doc: Make the AIO requirements clearer https://review.openstack.org/269557 | 17:21 |
| *** alextricity_r has quit IRC | 17:24 | |
| *** phiche1 has joined #openstack-ansible | 17:24 | |
| *** phiche has quit IRC | 17:26 | |
| odyssey4me | #success OpenStack-Ansible Kilo 11.2.8 has been released! | 17:26 |
| openstackstatus | odyssey4me: Added success to Success page | 17:26 |
| *** sdake has quit IRC | 17:28 | |
| *** javeriak has quit IRC | 17:32 | |
| *** CheKoLyN has joined #openstack-ansible | 17:33 | |
| *** javeriak has joined #openstack-ansible | 17:33 | |
| *** TheIntern has joined #openstack-ansible | 17:37 | |
| *** weezS has quit IRC | 17:38 | |
| *** javeriak_ has joined #openstack-ansible | 17:40 | |
| *** javeriak has quit IRC | 17:41 | |
| spotz | woot | 17:44 |
| odyssey4me | spotz I've changed the topic on all the docs related patches to make them easier to find :) | 17:44 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Allow sourcing apt_key from URL https://review.openstack.org/271447 | 17:44 |
| spotz | Sweet thanks odyssey4me | 17:45 |
| -openstackstatus- NOTICE: Restarting zuul due to a memory leak | 17:49 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Allow sourcing apt_key from ansible host or URL https://review.openstack.org/268676 | 17:50 |
| *** raddaoui has joined #openstack-ansible | 17:51 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible: Update config_template plugin for Ansible 2 https://review.openstack.org/267772 | 17:52 |
| *** fawadkhaliq has quit IRC | 17:59 | |
| *** fawadkhaliq has joined #openstack-ansible | 17:59 | |
| *** sdake has joined #openstack-ansible | 18:01 | |
| *** TheIntern has quit IRC | 18:02 | |
| * mhayden comes back from lunch to find a deployed liberty environment ;) | 18:04 | |
| *** woodard has joined #openstack-ansible | 18:09 | |
| *** jwagner_wfh is now known as jwagner_lunch | 18:10 | |
| *** woodard has quit IRC | 18:10 | |
| *** woodard has joined #openstack-ansible | 18:10 | |
| *** phiche1 has quit IRC | 18:13 | |
| *** phiche has joined #openstack-ansible | 18:17 | |
| sigmavirus24 | mhayden: but does it work? | 18:18 |
| mhayden | it blends | 18:18 |
| sigmavirus24 | don't breath that | 18:18 |
| mhayden | is there an easy way to back out what tempest puts into the environment? | 18:19 |
| *** raddaoui has quit IRC | 18:21 | |
| *** metral is now known as metral_zzz | 18:21 | |
| *** metral_zzz is now known as metral | 18:22 | |
| odyssey4me | mhayden tempest does that itself | 18:26 |
| odyssey4me | the os-tempest role adds a few things in preparation for the testing - that you can simply remove if you must | 18:26 |
| mhayden | oh okay | 18:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update openstack_release version to 11.2.9 https://review.openstack.org/271469 | 18:33 |
| *** adac has joined #openstack-ansible | 18:35 | |
| *** baker has quit IRC | 18:36 | |
| *** admin0 has joined #openstack-ansible | 18:36 | |
| *** admin0 has quit IRC | 18:37 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-galera_server: Added major version upgrade support https://review.openstack.org/271423 | 18:38 |
| *** electrofelix has quit IRC | 18:38 | |
| palendae | mcarden: Re-reading the user-secrets-adjustment.yml file and thinking on your question, the relative references to the file are for the example (source), the absolute are to the in-use, production file (destination) | 18:41 |
| *** baker has joined #openstack-ansible | 18:42 | |
| odyssey4me | cloud https://review.openstack.org/271423 looks good. Ideally we need to add a test that validates that it works. Any thoughts on how we do that? | 18:47 |
| odyssey4me | cloudnull ^ | 18:48 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-galera_server: Added major version upgrade support https://review.openstack.org/271423 | 18:48 |
| cloudnull | we can deploy the role with the base package using mariadb-galera-server-5.5 and then remove the extra variable and run it with the upgrade flags | 18:49 |
| *** adac has quit IRC | 18:51 | |
| mcarden | palendae: Thanks - that's the sort of distinction I was looking for. | 18:54 |
| palendae | mcarden: Yeah, sorry, somehow blanked when I read it the first time | 18:54 |
| mhayden | neutron on a hypervisor is yelling at me about -> Interface eth12 for physical network flat does not exist. Agent terminated! | 18:56 |
| mhayden | but a hypervisor doesn't have eth12 | 18:56 |
| kysse | its a freakin container interface | 18:57 |
| mhayden | i know that much -- just not sure why i'm getting that error on a compute node | 18:57 |
| cloudnull | mhayden: if you need to have the corresponding device on the host | 18:58 |
| palendae | mhayden: Some deployers (RPC) do | 18:58 |
| cloudnull | if your using the aio config you likely have a device specified for it. | 18:58 |
| *** eil397 has joined #openstack-ansible | 18:58 | |
| mhayden | ah, i think i know where i went off the rails | 18:58 |
| mhayden | on a hypervisor, i have -> physical_interface_mappings = flat:eth12,vlan:br-vlan | 18:59 |
| odyssey4me | cloudnull perhaps the test playbook should implement two clusters - one which deploys greenfield, the other which deploys 55 and upgrades it? | 18:59 |
| cloudnull | do you have eth12 ? | 19:01 |
| mhayden | cloudnull: not on the compute nodes | 19:01 |
| odyssey4me | mcarden palendae mhayden here are some docs updates which I'd appreciate some reviews on: https://review.openstack.org/#/q/status:open+project:openstack/openstack-ansible+branch:master+topic:docs | 19:01 |
| cloudnull | if not, youll need to make it or remove it from your openstack_user_config.yml | 19:01 |
| mhayden | odyssey4me: i'll gander | 19:01 |
| odyssey4me | heh, check queue is 244 deep :/ | 19:02 |
| mhayden | cloudnull: https://gist.github.com/major/7826d8ed1437b7a811f8 -- do i need that host_bind_override? | 19:02 |
| mcarden | odyssey4me: Doc reviews. Gotcha. | 19:03 |
| cloudnull | remove https://gist.github.com/major/7826d8ed1437b7a811f8#file-gistfile1-txt-L11-L19 and you should be good | 19:04 |
| mhayden | cloudnull: thanks -- that makes more sense now ;) | 19:04 |
| cloudnull | more data here http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-networking.html#network-configuration | 19:06 |
| *** spotz is now known as spotz_zzz | 19:06 | |
| cloudnull | specifically point 7 | 19:07 |
| odyssey4me | lol, cloudnull actually pointed mhayden at the docs | 19:07 |
| odyssey4me | RTFM :p | 19:07 |
| cloudnull | lololol | 19:07 |
| * mhayden sits in the corner | 19:08 | |
| cloudnull | nah. its a common issue | 19:08 |
| palendae | Yep | 19:09 |
| odyssey4me | it is | 19:10 |
| mhayden | i was scouring the ml2 docs expecting my problem to be there :P | 19:11 |
| palendae | Nah, it's self-inflicted by this project, not a neutron problem | 19:12 |
| mhayden | ah, made a junk veth there right quick and i'm back in business | 19:12 |
| *** alex______ has joined #openstack-ansible | 19:12 | |
| mhayden | thanks, folks | 19:12 |
| alex______ | hi! question on setting up neutron: should the internet facing network be defined as a network block? I think it needs to be so neutron-agent can setup external router, but if I add the block to the config, it applies to all linuxbridge_agents, and so goes on to nova as well (which obviously doesn't connect to public interface). am i missing something simple? | 19:14 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-galera_server: Added major version upgrade support https://review.openstack.org/271423 | 19:15 |
| cloudnull | ^ odyssey4me updated to test an upgrade. | 19:15 |
| alex______ | by defined as a network block, i mean a block in openstack_user_config.yml | 19:18 |
| *** spotz_zzz is now known as spotz | 19:21 | |
| *** CheKoLyN has quit IRC | 19:22 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add upgrade release note about the removal of neutron_service_names https://review.openstack.org/271490 | 19:26 |
| *** markvoelker has quit IRC | 19:27 | |
| mattt | odyssey4me: still got a failure with that change, so either it didn't take properly or 5 mins is still too long | 19:28 |
| mattt | doing a test locally now to see | 19:28 |
| *** weezS has joined #openstack-ansible | 19:29 | |
| *** jwagner_lunch is now known as jwagner | 19:30 | |
| odyssey4me | mattt yeah, I think we may have to turn caching off - or set it way, way down to something like 5 secs? | 19:33 |
| odyssey4me | cloudnull a small bit of feedback on some basics - it'll be interesting to see whether the test works :) | 19:33 |
| jmccrory | cloudnull missing galera_upgrade: true in your test vars or testing for failure first? | 19:35 |
| *** iceyao has joined #openstack-ansible | 19:35 | |
| *** fawadkhaliq has quit IRC | 19:35 | |
| *** CheKoLyN has joined #openstack-ansible | 19:36 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Arrange Release Notes similarly to the Documentation https://review.openstack.org/271420 | 19:37 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-galera_server: Added major version upgrade support https://review.openstack.org/271423 | 19:40 |
| cloudnull | jmccrory: good catch . | 19:40 |
| *** iceyao has quit IRC | 19:40 | |
| mhayden | KLevenstein / odyssey4me: nicely done on the docs changes | 19:40 |
| openstackgerrit | Merged openstack/openstack-ansible: Docs: Update the minor upgrade process https://review.openstack.org/269668 | 19:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Doc: Remove 'indices and tables' and correctly spell OpenStack-Ansible https://review.openstack.org/271350 | 19:44 |
| neillc | cloudnull: should we drop round for more learning? | 19:52 |
| cloudnull | sure thing | 19:52 |
| openstackgerrit | Merged openstack/openstack-ansible: Consolidating foundation playbook instructions https://review.openstack.org/268314 | 19:54 |
| openstackgerrit | Merged openstack/openstack-ansible: Consolidating galera recovery documentation https://review.openstack.org/268264 | 19:54 |
| *** lakerzhou has joined #openstack-ansible | 19:57 | |
| lakerzhou | Hi, I am new to openstack-ansible, just tried the AIO install, but could not find the password to login to horizon. Is there a default one? Can anyone help please? | 19:59 |
| logan- | it will be generated automatically during the aio build.. sudo grep 'auth_admin_pass' /etc/openstack_deploy/user_secrets.yml | 20:02 |
| palendae | lakerzhou: Look in /etc/openstack_deploy/user_secrets.yml | 20:02 |
| palendae | What logan said :) | 20:02 |
| lakerzhou | thanks, looking for keystone_auth_admin_password? | 20:03 |
| logan- | correct | 20:04 |
| lakerzhou | great! thanks. | 20:05 |
| *** hybridpollo has joined #openstack-ansible | 20:09 | |
| *** pcaruana has quit IRC | 20:14 | |
| *** leakypipes has quit IRC | 20:14 | |
| *** sdake_ has joined #openstack-ansible | 20:22 | |
| *** sdake has quit IRC | 20:25 | |
| KLevenstein | mhayden: thanks. will probably do some more soon | 20:27 |
| *** iceyao has joined #openstack-ansible | 20:37 | |
| *** javeriak_ has quit IRC | 20:38 | |
| *** iceyao has quit IRC | 20:41 | |
| *** weezS has quit IRC | 20:47 | |
| *** weezS has joined #openstack-ansible | 20:51 | |
| *** CheKoLyN has quit IRC | 20:52 | |
| *** weezS has quit IRC | 20:54 | |
| *** raddaoui has joined #openstack-ansible | 20:56 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about RabbitMQ SSL communication https://review.openstack.org/271520 | 20:57 |
| *** Nepoc has joined #openstack-ansible | 20:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about haproxy logs feature addition https://review.openstack.org/271522 | 21:04 |
| *** galstrom is now known as galstrom_zzz | 21:06 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: [WIP] Reduce keystone cache expiration time https://review.openstack.org/271357 | 21:06 |
| *** adac has joined #openstack-ansible | 21:07 | |
| *** CheKoLyN has joined #openstack-ansible | 21:07 | |
| *** raddaoui has quit IRC | 21:10 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about RabbitMQ SSL communication https://review.openstack.org/271520 | 21:20 |
| *** lakerzhou has quit IRC | 21:31 | |
| *** KLevenstein has quit IRC | 21:33 | |
| *** automagically has quit IRC | 21:36 | |
| *** markvoelker has joined #openstack-ansible | 21:40 | |
| *** markvoelker has quit IRC | 21:40 | |
| *** adac has quit IRC | 21:40 | |
| *** markvoelker has joined #openstack-ansible | 21:41 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about RabbitMQ SSL communication https://review.openstack.org/271520 | 21:42 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about haproxy logs feature addition https://review.openstack.org/271522 | 21:44 |
| *** pcaruana has joined #openstack-ansible | 21:48 | |
| palendae | https://review.openstack.org/#/c/207713/8 | 21:51 |
| *** sdake_ has quit IRC | 21:54 | |
| *** tlian has quit IRC | 22:08 | |
| odyssey4me | stevelle cloudnull can we bang this through please to open up the next kilo tag: https://review.openstack.org/271469 | 22:18 |
| *** markvoelker has quit IRC | 22:32 | |
| *** phiche1 has joined #openstack-ansible | 22:37 | |
| *** phiche has quit IRC | 22:38 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add release note about restricted .my.cnf distribution https://review.openstack.org/271547 | 22:39 |
| *** phiche has joined #openstack-ansible | 22:51 | |
| *** baker has quit IRC | 22:52 | |
| *** phiche1 has quit IRC | 22:53 | |
| *** Bjoern has quit IRC | 22:56 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:00 | |
| *** sdake has joined #openstack-ansible | 23:00 | |
| *** sdake has quit IRC | 23:01 | |
| *** CheKoLyN has quit IRC | 23:02 | |
| *** phiche has quit IRC | 23:09 | |
| *** jamielennox is now known as jamielennox|away | 23:15 | |
| *** jthorne has quit IRC | 23:26 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible: Reduce keystone cache expiration time https://review.openstack.org/271357 | 23:35 |
| spotz | have a good weekend guys | 23:38 |
| *** spotz is now known as spotz_zzz | 23:39 | |
| *** daneyon has quit IRC | 23:43 | |
| *** sdake has joined #openstack-ansible | 23:47 | |
| *** jeh has quit IRC | 23:56 | |
| *** Mudpuppy has quit IRC | 23:56 | |
| *** Mudpuppy has joined #openstack-ansible | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!