| *** markvoelker has joined #openstack-ansible | 00:01 | |
| *** markvoelker has quit IRC | 00:05 | |
| *** abitha has joined #openstack-ansible | 00:14 | |
| *** annashen has joined #openstack-ansible | 00:48 | |
| *** JTen_ has quit IRC | 00:52 | |
| *** annashen has quit IRC | 01:10 | |
| *** annashen has joined #openstack-ansible | 01:17 | |
| *** annashen has quit IRC | 01:46 | |
| *** annashen has joined #openstack-ansible | 01:49 | |
| *** markvoelker has joined #openstack-ansible | 01:50 | |
| *** markvoelker has quit IRC | 01:55 | |
| *** georgem1 has joined #openstack-ansible | 01:56 | |
| *** stevemar has joined #openstack-ansible | 01:57 | |
| *** annashen has quit IRC | 02:01 | |
| *** galstrom_zzz is now known as galstrom | 02:46 | |
| *** sdake_ has quit IRC | 03:10 | |
| *** jmccrory has quit IRC | 03:14 | |
| *** jmccrory has joined #openstack-ansible | 03:16 | |
| *** galstrom is now known as galstrom_zzz | 03:29 | |
| *** annashen has joined #openstack-ansible | 03:34 | |
| *** galstrom_zzz is now known as galstrom | 03:34 | |
| *** markvoelker has joined #openstack-ansible | 03:38 | |
| *** galstrom is now known as galstrom_zzz | 03:41 | |
| *** markvoelker has quit IRC | 03:43 | |
| *** abitha has quit IRC | 03:49 | |
| *** annashen has quit IRC | 03:54 | |
| *** JRobinson__ is now known as JRobinson__afk | 03:57 | |
| *** sdake has joined #openstack-ansible | 04:04 | |
| *** georgem1 has quit IRC | 04:10 | |
| *** annashen has joined #openstack-ansible | 04:12 | |
| *** JRobinson__afk is now known as JRobinson__ | 04:30 | |
| *** annashen has quit IRC | 04:31 | |
| *** radek has joined #openstack-ansible | 05:07 | |
| *** radek__ has joined #openstack-ansible | 05:18 | |
| *** stevemar has quit IRC | 05:19 | |
| *** radek has quit IRC | 05:21 | |
| *** markvoelker has joined #openstack-ansible | 05:27 | |
| *** shausy has joined #openstack-ansible | 05:27 | |
| *** markvoelker has quit IRC | 05:32 | |
| *** shausy has quit IRC | 05:38 | |
| *** shausy has joined #openstack-ansible | 05:40 | |
| *** shausy has quit IRC | 05:45 | |
| *** jmccrory has quit IRC | 05:52 | |
| *** jmccrory has joined #openstack-ansible | 05:57 | |
| *** javeriak has joined #openstack-ansible | 06:56 | |
| *** JRobinson__ has quit IRC | 07:10 | |
| *** markvoelker has joined #openstack-ansible | 07:16 | |
| *** markvoelker has quit IRC | 07:21 | |
| *** vincent_1dk has quit IRC | 08:13 | |
| *** vincent_vdk has joined #openstack-ansible | 08:14 | |
| *** javeriak has quit IRC | 08:54 | |
| *** markvoelker has joined #openstack-ansible | 09:05 | |
| *** markvoelker has quit IRC | 09:09 | |
| *** sdake has quit IRC | 10:43 | |
| *** britthou_ has quit IRC | 10:43 | |
| *** lbragstad has quit IRC | 10:43 | |
| *** dolphm has quit IRC | 10:43 | |
| *** _d34dh0r53_ has quit IRC | 10:43 | |
| *** odyssey4me_ has quit IRC | 10:43 | |
| *** persia has quit IRC | 10:43 | |
| *** eglute_s has quit IRC | 10:43 | |
| *** sigmavirus24_awa has quit IRC | 10:43 | |
| *** gus has quit IRC | 10:43 | |
| *** bgmccollum has quit IRC | 10:43 | |
| *** jroll has quit IRC | 10:43 | |
| *** persia has joined #openstack-ansible | 10:43 | |
| *** persia has quit IRC | 10:43 | |
| *** persia has joined #openstack-ansible | 10:43 | |
| *** bgmccollum has joined #openstack-ansible | 10:43 | |
| *** gus has joined #openstack-ansible | 10:43 | |
| *** britthouser has joined #openstack-ansible | 10:43 | |
| *** jroll has joined #openstack-ansible | 10:43 | |
| *** odyssey4me has joined #openstack-ansible | 10:43 | |
| *** lbragstad has joined #openstack-ansible | 10:44 | |
| *** sdake has joined #openstack-ansible | 10:44 | |
| *** dolphm has joined #openstack-ansible | 10:44 | |
| *** eglute has joined #openstack-ansible | 10:44 | |
| *** d34dh0r53 has joined #openstack-ansible | 10:45 | |
| *** sigmavirus24_awa has joined #openstack-ansible | 10:46 | |
| *** markvoelker has joined #openstack-ansible | 10:53 | |
| *** subscope has joined #openstack-ansible | 10:56 | |
| *** markvoelker has quit IRC | 10:58 | |
| svg | dstanek: dolphm sigmavirus: testing requests to glance (triggering keystone) while I disable the [memcache] token memcached and the keystone in one to more containers: even after a time some requests fail ('Invalid OpenStack Identity credentials.')..... | 11:00 |
|---|---|---|
| svg | so This is definitely not HA :( | 11:00 |
| svg | to give you an idea: of failure rate: https://dl.dropboxusercontent.com/u/13986042/20150622131444.png | 11:15 |
| svg | given all keystones need to share the same set of caches, I don't see how to solve this (given in the past one got back from useing an LB endpoint) | 11:16 |
| svg | on the plus: this does not trigger long timeouts I saw earlier, so that has to be related to the regular [cache] | 11:16 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow galera wsrep_provider_options to be customised https://review.openstack.org/191106 | 11:17 |
| *** sdake_ has joined #openstack-ansible | 11:27 | |
| *** jlvillal has quit IRC | 11:28 | |
| *** sdake has quit IRC | 11:30 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow galera wsrep_provider_options to be customised https://review.openstack.org/191106 | 11:39 |
| *** andyhky` is now known as andyhky | 11:41 | |
| *** markvoelker has joined #openstack-ansible | 11:54 | |
| *** markvoelker has quit IRC | 11:59 | |
| openstackgerrit | git-harry proposed stackforge/os-ansible-deployment: Add configurable option [cinder]/cross_az_attach https://review.openstack.org/194102 | 12:02 |
| *** markvoelker has joined #openstack-ansible | 12:03 | |
| *** b3rnard0 has left #openstack-ansible | 12:14 | |
| *** b3rnard0 has joined #openstack-ansible | 12:14 | |
| *** britthou_ has joined #openstack-ansible | 12:16 | |
| *** britthouser has quit IRC | 12:17 | |
| *** tobasco_ is now known as tobasco | 12:32 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Spec for multiregion ansible deployment https://review.openstack.org/192421 | 12:40 |
| openstackgerrit | git-harry proposed stackforge/os-ansible-deployment: Add neutron.conf [database] options https://review.openstack.org/194124 | 12:45 |
| *** KLevenstein has joined #openstack-ansible | 12:57 | |
| *** jmccrory has quit IRC | 13:14 | |
| *** jaypipes has joined #openstack-ansible | 13:15 | |
| *** jmccrory has joined #openstack-ansible | 13:15 | |
| *** tlian has joined #openstack-ansible | 13:15 | |
| *** 1JTAAA236 is now known as cloudnull | 13:22 | |
| cloudnull | morning | 13:22 |
| cloudnull | is "https://review.openstack.org/" timing out for folks ? | 13:22 |
| odyssey4me | morning cloudnull | 13:22 |
| odyssey4me | nope, not for me | 13:23 |
| cloudnull | morning odyssey4me | 13:25 |
| odyssey4me | cloudnull hmm, now it is timing out | 13:25 |
| odyssey4me | well, it's very slow | 13:25 |
| cloudnull | yea. i cant even hit it from a cloudserver | 13:25 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated kilo to the latest SHAs - 06.20.2015 https://review.openstack.org/193845 | 13:31 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated juno to the latest SHAs - 06.20.2015 https://review.openstack.org/193846 | 13:33 |
| *** jroll has quit IRC | 13:36 | |
| *** jroll has joined #openstack-ansible | 13:36 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated master to the latest SHAs - 06.20.2015 https://review.openstack.org/193844 | 13:37 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated icehouse to the latest SHAs - 06.20.2015 https://review.openstack.org/193848 | 13:37 |
| *** rromans_ is now known as rromans | 13:48 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment-specs: Cleaned up all specs https://review.openstack.org/193832 | 13:51 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default https://review.openstack.org/193729 | 13:55 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 13:55 |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation unusable deployment https://review.openstack.org/194147 | 13:55 |
| svg | I think I may conclude that keystone + memcache for Tokens is *not* higly available. Not even fault tolerant. | 14:02 |
| dstanek | svg: exactly! | 14:04 |
| dstanek | it's not designed to be | 14:04 |
| *** stevemar has joined #openstack-ansible | 14:04 | |
| svg | it goes even a bit further than that | 14:06 |
| svg | bringing some of the token memcaches down obiously triggers issues | 14:07 |
| svg | but bringing them back up is not enough | 14:07 |
| dstanek | memcached will also silently drop your data when it needs to | 14:07 |
| svg | at that point, with deploying some heat stacks, we kept getting auth errors | 14:07 |
| svg | ... until we erstarted all nova-api-os-compute services.... | 14:07 |
| dstanek | if you cycle a memcache you lose all of your tokens on that node | 14:07 |
| svg | still pretty weird that restarting a nova service helps here? | 14:08 |
| dstanek | is nova doing an auth to get a new token? | 14:09 |
| svg | btw, I was not able to reproducue any troubles wit the plain [cache] - last week we saw terrible slow down in client respones (10's of seconds) | 14:09 |
| svg | I didn't notice / watch that I'm afraid | 14:10 |
| svg | long live the fernet alternative I guess... | 14:10 |
| svg | Is that planned to be backported to kilo? | 14:10 |
| cloudnull | so what im hearing is that we should make fernet a bigger priority for master/kilo | 14:10 |
| cloudnull | svg: yes | 14:11 |
| svg | cloudnull: I would say so, yes. | 14:11 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:12 | |
| svg | right now, if one keystone container, or at least it;s memcaced goed awol, all hell breaks loose. | 14:12 |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation unusable deployment https://review.openstack.org/194147 | 14:13 |
| cloudnull | well that's no good. | 14:13 |
| cloudnull | svg: you could change your driver to use sql backed tokens. that should go without issues. | 14:13 |
| dstanek | fyi...this has been reported several times in the past - just closed another related bug: https://bugs.launchpad.net/keystone/+bug/1436324 | 14:13 |
| openstack | Launchpad bug 1436324 in Keystone "Keystone is not HA with memcache as token persistence driver" [Low,Won't fix] - Assigned to Boris Bobrov (bbobrov) | 14:13 |
| svg | cloudnull: I believe the sql config is not something that osad allows without patching? | 14:16 |
| cloudnull | yes, getting config one sec | 14:16 |
| svg | yeah, looking at http://docs.openstack.org/kilo/config-reference/content/section_keystone.conf.html | 14:19 |
| svg | oh it is, seems like just changing the var keystone_token_driver | 14:21 |
| svg | keystone.token.persistence.backends.sql.Token | 14:22 |
| cloudnull | svg: change `keystone_token_driver: "keystone.token.persistence.backends.sql.Token"` in your user_vars file and you should be good to go . | 14:22 |
| cloudnull | ah. totally late there. | 14:22 |
| cloudnull | you got it. | 14:22 |
| svg | :) | 14:22 |
| svg | thanks for confirmation, was not sure that was the only needed change | 14:22 |
| *** Mudpuppy has joined #openstack-ansible | 14:26 | |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation unusable deployment https://review.openstack.org/194147 | 14:28 |
| mancdaz | cloudnull I added a discussion point around bug/no bug for sha bumps and version bumps, for the next meeting, on the wiki. I can see both arguments, but it's worth chatting about it | 14:30 |
| cloudnull | great | 14:30 |
| cloudnull | so in your opinion are we looking to hold the current sha bumps until that discussion ? | 14:31 |
| mancdaz | cloudnull no not really. We can always add a bug later and reference the reviews that went in | 14:32 |
| mancdaz | it's less about having the bug referenced in the commit message for me, and more about having something that can be 'released' in launchpad | 14:32 |
| mancdaz | so people can go to a milestone page and easily see what changed/was added | 14:32 |
| odyssey4me | cloudnull personally I can see the argument for skipping the bug/bp ref in master (as it's considered unstable)... but not for the other branches | 14:33 |
| mancdaz | cloudnull I don't think it's worth holding up dev for | 14:33 |
| *** galstrom_zzz is now known as galstrom | 14:33 | |
| odyssey4me | it could easily be a bp without a spec | 14:33 |
| cloudnull | so do you think it better to have a bug/bp that we update every in two weeks for two supported branches for 15 months ? also you cant "release" a bp as implemented more than once so it would have to be new bug created every time. | 14:34 |
| cloudnull | food for thought. | 14:35 |
| mancdaz | yeah you're right it would need to be a bug | 14:35 |
| cloudnull | it could be the same bug re-created all the time, with a targeted series. | 14:36 |
| *** mfisch` is now known as mfisch | 14:36 | |
| cloudnull | so one bug per the two/three release branches. | 14:36 |
| *** mfisch is now known as Guest82290 | 14:36 | |
| mancdaz | right | 14:37 |
| cloudnull | but made every two weeks. | 14:37 |
| mancdaz | 'update to latest stable' | 14:37 |
| cloudnull | but isn't that in the git log and the release notes for a released milestone? | 14:37 |
| mancdaz | well, docs team point people at the milestone page for release notes for juno releases currently | 14:38 |
| odyssey4me | hmm, if you re-use the same bug every time then you end up losing the reference for the previous milestone (as you have to re-target it). | 14:39 |
| mancdaz | odyssey4me not the same bug | 14:39 |
| mancdaz | a new bug each 2 weeks | 14:39 |
| odyssey4me | otherwise that would've been great\ | 14:39 |
| odyssey4me | new bug every two weeks obviously caters for the need | 14:40 |
| odyssey4me | ah, you mean a new bug for all software updates - then use partial- instead of closes-bug in the commit? | 14:40 |
| cloudnull | mancdaz: yes the docs team does point people at the milestone page, where they have the ability to click the change log button and read what happened in the release. | 14:41 |
| mancdaz | odyssey4me not sure it matters - closes doesn't actually close the bug anyway | 14:41 |
| odyssey4me | mancdaz closes-bug will mark it as fix-committed whereas partial-bug does not | 14:41 |
| mancdaz | for reals? | 14:42 |
| mancdaz | does that work | 14:42 |
| palendae | Should now that the stable/ fix is in | 14:42 |
| cloudnull | its supposed to | 14:42 |
| mancdaz | across multiple branches? | 14:42 |
| odyssey4me | mancdaz yeah, as far as I've seen that's how it works | 14:42 |
| odyssey4me | partial- and related- are tags designed for that purpose | 14:42 |
| cloudnull | there are going to be things that go into a release that are not "targeted" and while its nice to see all the things that went in from a lp point of view the git log, which we are updating on every release, is the source of truth. | 14:43 |
| mancdaz | cloudnull ok if we're adding a full diff between reelases to that part of the milestone page, I care less | 14:43 |
| cloudnull | https://launchpad.net/openstack-ansible/kilo/11.0.3 | 14:44 |
| mancdaz | as long as we add dates/versions in the commit message title for those bumps | 14:44 |
| odyssey4me | cloudnull mancdaz ah, I had forgotten about the changelog - that's useful | 14:44 |
| odyssey4me | in that case I agree - as long as the subject line is specific enough, then it's great | 14:45 |
| cloudnull | a good example https://launchpad.net/openstack-ansible/icehouse/9.0.10 | 14:45 |
| cloudnull | 3 targeted, 6 changes . | 14:45 |
| cloudnull | mancdaz: +1 on the date change in the subject for the bumps. that was a great suggestion from odyssey4me | 14:46 |
| palendae | I missed it, that's the date the SHAs were captured? | 14:46 |
| mancdaz | done | 14:46 |
| mancdaz | palendae yep | 14:46 |
| palendae | Makes sense | 14:46 |
| *** Mudpuppy_ has joined #openstack-ansible | 14:49 | |
| *** jlvillal has joined #openstack-ansible | 14:49 | |
| *** jlvillal has quit IRC | 14:51 | |
| odyssey4me | cloudnull mancdaz palendae so I'm down with not doing bug/bp commit references ad long as the subject line appropriately covers the change (like package/sha updates with their version/date)... so how far do we extend this? We have a lot of bugs being reported at the moment which aren't bugs - many are simply to facilitate small feature additions or small changes to the way things work. | 14:51 |
| *** jlvillal has joined #openstack-ansible | 14:52 | |
| *** Mudpuppy has quit IRC | 14:52 | |
| palendae | Well, those need tickets/issues/LP bugs still... | 14:52 |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Move Cinder-volumes to "on metal" https://review.openstack.org/194176 | 14:52 |
| odyssey4me | I do think that there should be a conduit for these things, but perhaps some of them should just be committed? | 14:52 |
| *** sdake_ has quit IRC | 14:52 | |
| cloudnull | small feature change we should add a bug, medium to large feature change we should have a bp/spec. | 14:53 |
| odyssey4me | That said, I think the reason I brought up the issue originally was because we had forgotten to backport something because there was no LP reference for it. | 14:53 |
| mancdaz | that's a good argument for having a bug to reference - tracking of backport tags etc | 14:54 |
| odyssey4me | So perhaps the simple rule should be that if it may require a backport, then it should have a bug ref? | 14:54 |
| odyssey4me | But then what about rabbitmq and other such updates - those should be backports, but doing a bug ref every time is silly. | 14:56 |
| cloudnull | idk if rabbitmq or galera should be backports. | 14:56 |
| odyssey4me | cloudnull well, galera gets from the current repo which only contains the latest version... so that's out of our hands | 14:57 |
| cloudnull | especially if they're large bumps in the stack. or if we do backport something like galera + mariadb10 it should be in a feature version . 11.1.x | 14:57 |
| odyssey4me | rabbitmq, however, we control the version | 14:57 |
| cloudnull | thats true galera gets the current version but in the 5.5 series | 14:58 |
| cloudnull | but with one of the updated reviews we're pushing to 10 | 14:58 |
| cloudnull | i'd say the same goes with rabbitmq too. | 14:58 |
| palendae | Would it make sense to only backport if missing version forces us to? | 14:58 |
| palendae | Like an old version drops off upstream repos? | 14:58 |
| odyssey4me | ah yes, so that's good to differentiate - if it's a major version change then there should be a bp/spec | 14:58 |
| cloudnull | id say so | 14:59 |
| odyssey4me | palendae not really, consider security hotfixes | 14:59 |
| palendae | True | 14:59 |
| mancdaz | oh that reminds me, we might be able to move to active/active/active etc on the galera side | 15:01 |
| mancdaz | since the OS projects have largely resolved their bad handling of locks | 15:01 |
| odyssey4me | mancdaz oh, is that in kilo or liberty? | 15:01 |
| mancdaz | well ok, liberty probably | 15:02 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Add global endpoint_type_proto options https://review.openstack.org/193573 | 15:02 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Correct local_settings when AVAILABLE_REGIONS is set https://review.openstack.org/191004 | 15:02 |
| odyssey4me | mancdaz I'm thinking that we should let https://review.openstack.org/189998 through the gate and resolve the key rotation and max_keys issue in later patches once we can have some more testing done to figure out what works best? | 15:04 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Add configurable option [cinder]/cross_az_attach https://review.openstack.org/194102 | 15:06 |
| mancdaz | odyssey4me especially since key rotation and specifically distribution is largely an issue in traditional multi-region type deployments | 15:06 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Add neutron.conf [database] options https://review.openstack.org/194124 | 15:06 |
| mancdaz | so yeah | 15:06 |
| mancdaz | oh wait that wasn't being accounted for anyway - just container - container distribution] | 15:07 |
| *** Mudpuppy_ is now known as Mudpuppy | 15:09 | |
| mancdaz | is there some way to exclude an external CI system from stackalytics? | 15:09 |
| odyssey4me | mancdaz I would guess that the way to do it would be to have it vote properly - ie verify instead of review | 15:12 |
| cloudnull | has the external CI been fixed, it seems to be unhappy all the time. | 15:13 |
| mancdaz | cloudnull it's been turned off I think | 15:14 |
| mancdaz | https://github.com/stackforge/stackalytics/blob/master/etc/default_data.json#L12056 | 15:14 |
| mancdaz | nice alias for rackspace | 15:14 |
| mancdaz | "Korea Telcom, friends with lots of people" | 15:15 |
| mancdaz | ?? | 15:15 |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation ansible deployment https://review.openstack.org/194147 | 15:15 |
| cloudnull | dolphm: dstanek: http://logs.openstack.org/29/193729/6/check/os-ansible-deployment-dsvm-check-commit/20f11a4/console.html#_2015-06-22_14_46_08_643 - seems that fernet is presently incompatible with tempest ? | 15:15 |
| *** daneyon has joined #openstack-ansible | 15:17 | |
| dolphm | cloudnull: hmm, i had a conversation about that check a few weeks ago, but it was the opposite issue then. i'll investigate today | 15:18 |
| cloudnull | in that last run i pulled the latest tempest just to be sure. | 15:18 |
| *** georgem1 has joined #openstack-ansible | 15:19 | |
| cloudnull | same error as before "ValueError: time data '2015-06-21T13:41:25.824964Z' does not match format '%Y-%m-%dT%H:%M:Z'" | 15:19 |
| cloudnull | it seems that fernet is too exact . =) | 15:19 |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation ansible deployment https://review.openstack.org/194147 | 15:21 |
| dolphm | cloudnull: wait, the expected format is missing seconds altogether? | 15:22 |
| cloudnull | thats what it says. | 15:22 |
| *** jwagner is now known as jwagner_away | 15:22 | |
| *** jwagner_away is now known as jwagner | 15:22 | |
| cloudnull | it has seconds. | 15:23 |
| cloudnull | just not microseconds . | 15:23 |
| dolphm | cloudnull: the format you pasted (?) has no seconds at all ('%Y-%m-%dT%H:%M:Z'") | 15:23 |
| cloudnull | i think thats a paste failure | 15:24 |
| cloudnull | http://logs.openstack.org/29/193729/6/check/os-ansible-deployment-dsvm-check-commit/20f11a4/console.html#_2015-06-22_14_46_08_643 | 15:24 |
| dolphm | cloudnull: okay, just wanted to make sure you weren't looking at something i was missing | 15:26 |
| *** daneyon_ has joined #openstack-ansible | 15:26 | |
| *** daneyon has quit IRC | 15:29 | |
| cloudnull | dolphm: do you know of a specific tempest lib bug tracker or is it all wrapped up in https://bugs.launchpad.net/tempest | 15:30 |
| dolphm | cloudnull: i'm not aware of a separate one | 15:30 |
| cloudnull | ok | 15:30 |
| cloudnull | looks like it may be fixed in master temptest-lib | 15:33 |
| cloudnull | https://github.com/openstack/tempest-lib/blob/master/tempest_lib/auth.py#L319-L323 | 15:33 |
| cloudnull | but they've not released a new version since april. | 15:33 |
| palendae | Course not | 15:34 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default https://review.openstack.org/193729 | 15:37 |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated MariaDB to the new release version https://review.openstack.org/178259 | 15:39 |
| odyssey4me | cloudnull it sounds like we may have to track a newer sha for tempest and tempest-lib? | 15:39 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/194194 | 15:41 |
| *** metral is now known as metral_zzz | 15:41 | |
| cloudnull | odyssey4me: i added that to the fernet review . just to see if it goes. | 15:41 |
| cloudnull | but im not all that hopeful | 15:41 |
| cloudnull | we're already on https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/defaults/repo_packages/openstack_other.yml#L41 | 15:41 |
| cloudnull | well maybe not | 15:42 |
| cloudnull | the change to fix was authored may5 | 15:43 |
| cloudnull | but merged june 20 https://review.openstack.org/#/c/180355/ | 15:43 |
| *** rrrobbb has joined #openstack-ansible | 15:45 | |
| cloudnull | dolphm: so all the things may already be fixed. as of the 20th. we're testing now. https://jenkins01.openstack.org/job/os-ansible-deployment-dsvm-check-commit/492/ | 15:47 |
| sigmavirus24 | cloudnull: unfortunately 'recerify' doesn't work ;) | 15:47 |
| cloudnull | fuck spelling . . . | 15:48 |
| cloudnull | thanks for the catch/fix | 15:48 |
| dolphm | cloudnull: i don't think that _verify_expiry method maintains the spirit of that change... it appears to check multiple formats and will fail if either are incorrect?! unless i'm misreading the test, that makes no sense | 15:50 |
| cloudnull | :\ | 15:51 |
| cloudnull | yes it appears that the test for the auth test may be broken. | 15:52 |
| sigmavirus24 | http://logs.openstack.org/50/193850/2/gate/os-ansible-deployment-dsvm-check-commit/fc2ca78/console.html#_2015-06-22_15_42_48_372 btw is why your patch failed cloudnull | 15:54 |
| svg | cloudnull: seems sql now complains about Too many connections :) | 15:54 |
| cloudnull | sigmavirus24: yup and thats related to https://github.com/ansible/ansible-modules-core/issues/1497 | 15:55 |
| cloudnull | svg: welcome to the game | 15:55 |
| cloudnull | svg: how many connections were there ? | 15:56 |
| *** galstrom is now known as galstrom_zzz | 15:57 | |
| svg | still looking | 15:58 |
| *** metral_zzz is now known as metral | 16:02 | |
| svg | The calcculated value is 800, most nodes are around 600, one node has 4857 | 16:04 |
| svg | let's try with another lb method on f5 | 16:05 |
| openstackgerrit | git-harry proposed stackforge/os-ansible-deployment: Add configurable option [cinder]/cross_az_attach https://review.openstack.org/194213 | 16:06 |
| *** galstrom_zzz is now known as galstrom | 16:07 | |
| *** Guest82290 is now known as mfisch | 16:10 | |
| *** mfisch has quit IRC | 16:10 | |
| *** mfisch has joined #openstack-ansible | 16:10 | |
| odyssey4me | hmm, so with regards to https://review.openstack.org/194194 - should this perhaps wait for a major/minor release and not be allowed into a hotfix patch? | 16:11 |
| sigmavirus24 | hm | 16:12 |
| sigmavirus24 | good question odyssey4me | 16:12 |
| palendae | That sounds reasonable to me | 16:12 |
| sigmavirus24 | Also, thanks for inadvertantly pointing out that review -x is different than review -X | 16:12 |
| sigmavirus24 | and I fubar'd that cherry-pick | 16:12 |
| palendae | I thought it was an 11.1.0 thing | 16:12 |
| odyssey4me | yep, I think that backport should be held back for 11.1 | 16:13 |
| openstackgerrit | Ian Cordasco proposed stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/194194 | 16:13 |
| *** javeriak has joined #openstack-ansible | 16:32 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Updated kilo to the latest SHAs - 06.20.2015 https://review.openstack.org/193845 | 16:37 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Updated juno to the latest SHAs - 06.20.2015 https://review.openstack.org/193846 | 16:37 |
| *** javeriak has quit IRC | 16:39 | |
| *** Mudpuppy has quit IRC | 16:43 | |
| *** Mudpuppy_ has joined #openstack-ansible | 16:43 | |
| *** javeriak has joined #openstack-ansible | 16:45 | |
| *** daneyon has joined #openstack-ansible | 16:46 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Allow galera wsrep_provider_options to be customised https://review.openstack.org/191106 | 16:46 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Remove invalid client config option https://review.openstack.org/193833 | 16:46 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Remove invalid client config option https://review.openstack.org/193841 | 16:46 |
| *** daneyon_ has quit IRC | 16:47 | |
| *** javeriak has quit IRC | 16:50 | |
| *** daneyon_ has joined #openstack-ansible | 16:58 | |
| *** daneyon has quit IRC | 16:58 | |
| *** daneyon has joined #openstack-ansible | 16:59 | |
| sigmavirus24 | So, is fernet a pre-requirement for federation or am I making things up/ | 17:01 |
| dolphm | sigmavirus24: not related | 17:01 |
| sigmavirus24 | okay | 17:01 |
| * sigmavirus24 thought it was for some reason | 17:01 | |
| * sigmavirus24 goes to read more docs | 17:02 | |
| sigmavirus24 | oh a summit talk | 17:02 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Allow galera wsrep_provider_options to be customised https://review.openstack.org/194236 | 17:02 |
| * sigmavirus24 goes to watch that | 17:02 | |
| *** daneyon_ has quit IRC | 17:03 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default https://review.openstack.org/193729 | 17:03 |
| *** javeriak has joined #openstack-ansible | 17:03 | |
| odyssey4me | cloudnull sigmavirus24 perhaps launchpad needs a milestone for 11.1.0 to add https://bugs.launchpad.net/openstack-ansible/+bug/1463569 to :) | 17:09 |
| openstack | Launchpad bug 1463569 in openstack-ansible trunk "Add tasks for Keystone deployment using Fernet tokens" [Medium,In progress] - Assigned to Ian Cordasco (icordasc) | 17:09 |
| *** dkalleg has joined #openstack-ansible | 17:12 | |
| *** galstrom is now known as galstrom_zzz | 17:14 | |
| *** galstrom_zzz is now known as galstrom | 17:15 | |
| *** galstrom is now known as galstrom_zzz | 17:16 | |
| *** sdake_ has joined #openstack-ansible | 17:18 | |
| *** daneyon has quit IRC | 17:18 | |
| *** daneyon has joined #openstack-ansible | 17:19 | |
| *** daneyon has quit IRC | 17:20 | |
| openstackgerrit | Dolph Mathews proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation ansible deployment https://review.openstack.org/194147 | 17:21 |
| *** daneyon has joined #openstack-ansible | 17:21 | |
| *** dkalleg has quit IRC | 17:22 | |
| *** daneyon has quit IRC | 17:23 | |
| *** annashen has joined #openstack-ansible | 17:29 | |
| *** daneyon has joined #openstack-ansible | 17:30 | |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Added flag to instruct yaprt to ignore tempest https://review.openstack.org/193850 | 17:31 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet https://review.openstack.org/189998 | 17:31 |
| openstackgerrit | Ken Johnston proposed stackforge/os-ansible-deployment-specs: Spec for accepting ADFS as identity provider https://review.openstack.org/194255 | 17:35 |
| stevemar | dolphm, anything in particular i should look out for re: federationy support? | 17:38 |
| *** Verilium_ is now known as Verilium | 17:38 | |
| openstackgerrit | Miguel Grinberg proposed stackforge/os-ansible-deployment: [WIP] Keystone idp configuration https://review.openstack.org/194259 | 17:40 |
| openstackgerrit | Miguel Grinberg proposed stackforge/os-ansible-deployment: [WIP] Keystone idp configuration https://review.openstack.org/194259 | 17:43 |
| *** dkalleg has joined #openstack-ansible | 17:45 | |
| *** abitha has joined #openstack-ansible | 17:49 | |
| sigmavirus24 | odyssey4me: seems reasonable | 17:50 |
| dolphm | stevemar: the review above ( https://review.openstack.org/194147 ) just covers the use cases we're pursuing against kilo. a sanity check would always be appreciated! | 17:51 |
| *** Mudpuppy_ is now known as Mudpuppy | 17:55 | |
| *** Mudpuppy_ has joined #openstack-ansible | 18:05 | |
| *** Mudpuppy_ has quit IRC | 18:07 | |
| *** Mudpuppy_ has joined #openstack-ansible | 18:07 | |
| *** Mudpuppy_ has quit IRC | 18:08 | |
| *** Mudpuppy has quit IRC | 18:08 | |
| *** Mudpuppy has joined #openstack-ansible | 18:09 | |
| *** alextricity_r has joined #openstack-ansible | 18:49 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation ansible deployment https://review.openstack.org/194147 | 19:00 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Spec for keystone federation ansible deployment https://review.openstack.org/194147 | 19:03 |
| *** galstrom_zzz is now known as galstrom | 19:08 | |
| sigmavirus24 | odyssey4me: since you're clearly still up | 19:08 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Spec for multiregion ansible deployment https://review.openstack.org/192421 | 19:09 |
| odyssey4me | sigmavirus24 you rang, master? :p | 19:09 |
| sigmavirus24 | Are we splitting further up between ADFS work and Keystone federation work? If so, do we know who is going to work on which team? | 19:09 |
| * sigmavirus24 needs to look at miguelgrinberg's stuff to see what he already has but I think etherpads will be helpful once again | 19:09 | |
| odyssey4me | sigmavirus24 I think in general it's you and me on ADFS, with miguelgrinberg and hughsaunders on Keystone-Keystone... but to start with we need miguelgrinberg's starter work up so that we can use it as the base. | 19:10 |
| sigmavirus24 | Good to know :D | 19:10 |
| miguelgrinberg | odyssey4me sigmavirus24: I'm not going through the reviews on the keystone SP stuff on my fork. I will have a review up hopefully soon. | 19:11 |
| miguelgrinberg | s/not/now/ | 19:11 |
| odyssey4me | sigmavirus24 I just volunteered you to work with me, as you're late to the party (ie you didn't volunteer yourself). :p | 19:11 |
| sigmavirus24 | odyssey4me: figures | 19:11 |
| sigmavirus24 | I was looking at keystone federation stuff and saw a clear order of dependence that I thought I'd doc out | 19:11 |
| sigmavirus24 | But I also don't know what miguelgrinberg has finished so I wasn't going to just write down everything he did ;) | 19:12 |
| odyssey4me | sigmavirus24 care to doc that into the spec? | 19:12 |
| sigmavirus24 | oh cool, https://review.openstack.org/#/c/189998/ merged finally | 19:12 |
| sigmavirus24 | odyssey4me: I'll take a gander | 19:12 |
| sigmavirus24 | Or at least, it seems to me that to test this stuff we have a clear order of dependencies | 19:12 |
| sigmavirus24 | that said, I'm still learning keystone federation things and stuff so I could be wrong ;) | 19:13 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Multi-region Compute Deployment https://review.openstack.org/192421 | 19:13 |
| odyssey4me | sigmavirus24 yeah, perhaps we should stab at the spec early next week when we have most of it done and understand it all better? | 19:13 |
| sigmavirus24 | odyssey4me: not a bad idea | 19:13 |
| odyssey4me | or perhaps do it on Wed in time for the meeting on Thu | 19:14 |
| sigmavirus24 | It just seems like we need to be able to deploy a cloud as a Service Provider before we can deploy a cloud that talks to one | 19:14 |
| sigmavirus24 | That way we can test how an OSAD cloud would talk to a Service Provider (that also happens to be OSAD) | 19:14 |
| sigmavirus24 | that was probably obvious to everyone else, but I have 0 experience with this stuff so ¯\_(ツ)_/¯ | 19:15 |
| palendae | sigmavirus24: I'm pretty sure there was a meeting about how's working on what just now | 19:15 |
| odyssey4me | sigmavirus24 yep, that's kinda what miguelgrinberg's work is almost ready to do - with that base we can also then configure an external IDP | 19:15 |
| * sigmavirus24 needs to find miguelgrinberg's work now | 19:16 | |
| sigmavirus24 | palendae: how's working on what just now | 19:16 |
| odyssey4me | sigmavirus24 https://github.com/stackforge/os-ansible-deployment/compare/master...miguelgrinberg:federation | 19:16 |
| palendae | who's | 19:16 |
| stevemar | sigmavirus24, feel free to bug me about federationy bits | 19:16 |
| stevemar | though dolphm is pretty knowledgable about it too | 19:16 |
| odyssey4me | thanks stevemar :) the more SME's the better | 19:17 |
| stevemar | odyssey4me, +1 | 19:17 |
| odyssey4me | stevemar do you happen to also have any expertise on fernet tokens? | 19:17 |
| stevemar | odyssey4me, nope, that's all dolphm and lbragstad | 19:18 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Keystone Federation Deployment https://review.openstack.org/194147 | 19:18 |
| odyssey4me | ah stevemar thanks - I asked the wrong question, actually - I should have asked whether you're aware of anyone running with fernet tokens in production? | 19:19 |
| odyssey4me | We'd like to chat about a few things related to running with fernet tokens in production. | 19:19 |
| dolphm | odyssey4me: time warner is putting them into production in the next couple weeks cc- mfisch | 19:20 |
| *** rrrobbb has quit IRC | 19:24 | |
| *** galstrom is now known as galstrom_zzz | 19:24 | |
| odyssey4me | dolphm it'd be great to have some views on what a good number for 'max_active_keys' is in production, and how key rotation should be managed in a multi-node environment. We're having a discussion about that in the next Thu meeting. cc mfisch | 19:26 |
| *** Mudpuppy has quit IRC | 19:26 | |
| *** Mudpuppy has joined #openstack-ansible | 19:28 | |
| sigmavirus24 | miguelgrinberg: https://github.com/stackforge/os-ansible-deployment/compare/master...miguelgrinberg:federation#diff-921dfcc80a6a5dcf3a922884a5a04c75R70 should have a when keystone_idp is defined, yes? | 19:28 |
| miguelgrinberg | sigmavirus24: yes, I actually caught that in the version I'm preparing to upload to gerrit | 19:28 |
| *** annashen has quit IRC | 19:29 | |
| miguelgrinberg | sigmavirus24: actually that one is already out: https://review.openstack.org/#/c/194259/ | 19:30 |
| *** annashen has joined #openstack-ansible | 19:31 | |
| *** annashen has quit IRC | 19:32 | |
| *** annashen has joined #openstack-ansible | 19:32 | |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Keystone Service Provider with ADFS Identity Provider Deployment https://review.openstack.org/194255 | 19:36 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment-specs: Keystone Service Provider with ADFS Identity Provider Deployment https://review.openstack.org/194255 | 19:36 |
| mfisch | odyssey4me: we're just doing 3 keys for now with the idea we can add more whenever. We are not using keystone's rotation tools | 19:37 |
| odyssey4me | mfisch ah ok - so your plan is quite literally to just run with a small set of keys ad infinitum or until something happens to change your mind? | 19:38 |
| mfisch | we'll probably rotate every 2 weeks or so | 19:38 |
| mfisch | or when a core team member leaves that will also trigger it | 19:38 |
| mfisch | we're storing them in eyaml and deploying with puppet | 19:38 |
| odyssey4me | I'm trying to gauge what sort of security impact there is of having a small number of active keys? | 19:39 |
| mfisch | I have a jenkins job to rotate keys and propose a review of the change to keep humans in the loop | 19:39 |
| mfisch | our token expire is 2 hours | 19:39 |
| mfisch | so as long as we dont do 2 rotations in 2 hours we're ok | 19:39 |
| odyssey4me | so when the rotation is implemented, am I right in saying that one key falls off and another is generated to replace it... and this is done with the specified maximum number of keys always kept active | 19:41 |
| mfisch | yes that sounds right | 19:41 |
| mfisch | the old primary becomes a backup key so you can decode old tokens | 19:41 |
| mfisch | the old old primary goes away | 19:41 |
| mfisch | the new key becomes on-deck | 19:41 |
| mfisch | and the old on-deck becomes primary | 19:41 |
| mfisch | my blog has what I think is a good write up | 19:41 |
| odyssey4me | mfisch ah, that's probably the blog entry that made me aware that these were good questions to ask :) | 19:42 |
| mfisch | we've been in dev for 3-4 weeks and going to prod on Wed | 19:43 |
| *** alextricity_r has quit IRC | 19:44 | |
| odyssey4me | mfisch thank you for that - it has helped :) we'd love to hear your war stories once you've been running it production for a while | 19:44 |
| mfisch | will do | 19:44 |
| mfisch | I'll also have some numbers | 19:44 |
| mfisch | perf #s | 19:44 |
| odyssey4me | yeah, that'll be great - although dolphm's stats post it would seem that the performance will improve quite a bit | 19:45 |
| mfisch | well I'm seeing some of that but not all | 19:46 |
| mfisch | dolph is re-running some numbers for me | 19:46 |
| mfisch | I will have apples to apples numbers on Thursday | 19:46 |
| odyssey4me | mfisch actually, it was lbragstad's post I've seen - do you have the URL to yours? | 19:49 |
| odyssey4me | mfisch this one? http://www.mattfischer.com/blog/?p=648 | 19:50 |
| mfisch | top post here | 19:50 |
| mfisch | https://bfd-gerrit.os.cloud.twc.net/#/c/4763/ | 19:50 |
| mfisch | derp thats a code review | 19:50 |
| mfisch | yeah thats it | 19:50 |
| mfisch | never trust your paste buffer | 19:50 |
| odyssey4me | lol | 19:50 |
| *** rrrobbb has joined #openstack-ansible | 19:51 | |
| mfisch | odyssey4me: FYI from what I've seen we have a 20-30 second keystone outage | 19:53 |
| mfisch | then because we have an old copy of keystone-middleware still I have a reboot the cloud ansible I run | 19:53 |
| odyssey4me | mfisch ouch, that's no fun | 19:54 |
| *** annashen has quit IRC | 19:54 | |
| odyssey4me | but really good to know | 19:54 |
| mfisch | we can live with it | 19:54 |
| mfisch | it's a 2-3 min API outage | 19:54 |
| mfisch | our customers dont mind that but get super angry when they lose net access | 19:54 |
| mfisch | we dont have Rax-like customers | 19:54 |
| *** alextricity_r has joined #openstack-ansible | 19:55 | |
| odyssey4me | mfisch cool, thanks - it sounds like automated rotation is a bad idea... you want to rotate at a known date and time so that you can plan for the outage | 19:57 |
| mfisch | oh sorry thats not it | 19:57 |
| mfisch | its the token provider switch that is the outage | 19:58 |
| mfisch | also we | 19:58 |
| odyssey4me | this is something we should note in documentation, but not implement through cron I mean | 19:58 |
| mfisch | we're dogin a package upgrade | 19:58 |
| mfisch | key rotation is no outage | 19:58 |
| mfisch | I owe a follow-up blog | 19:58 |
| odyssey4me | mfisch ah ok, that's fair enough | 19:58 |
| *** yaya has joined #openstack-ansible | 20:08 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default https://review.openstack.org/193729 | 20:10 |
| *** alextricity_r has quit IRC | 20:20 | |
| *** alextricity_r has joined #openstack-ansible | 20:21 | |
| *** alextricty has joined #openstack-ansible | 20:28 | |
| *** alextricity_r has quit IRC | 20:28 | |
| *** alextricty has quit IRC | 20:31 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default https://review.openstack.org/193729 | 20:35 |
| *** Mudpuppy has quit IRC | 20:42 | |
| *** Mudpuppy has joined #openstack-ansible | 20:46 | |
| *** javeriak has quit IRC | 20:49 | |
| *** georgem1 has quit IRC | 20:53 | |
| *** annashen has joined #openstack-ansible | 20:54 | |
| *** javeriak has joined #openstack-ansible | 20:54 | |
| *** sdake_ has quit IRC | 20:56 | |
| *** annashen has quit IRC | 21:00 | |
| *** rrrobbb has quit IRC | 21:11 | |
| openstackgerrit | Kevin Carter proposed stackforge/os-ansible-deployment: Updated master to the latest SHAs - 06.20.2015 https://review.openstack.org/193844 | 21:14 |
| *** annashen has joined #openstack-ansible | 21:15 | |
| *** javeriak has quit IRC | 21:16 | |
| *** javeriak has joined #openstack-ansible | 21:20 | |
| *** tlian has quit IRC | 21:21 | |
| openstackgerrit | Miguel Grinberg proposed stackforge/os-ansible-deployment: [WIP] Keystone SP configuration https://review.openstack.org/194395 | 21:33 |
| *** yaya has quit IRC | 21:33 | |
| *** KLevenstein has quit IRC | 21:36 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:59 | |
| *** Mudpuppy has quit IRC | 22:02 | |
| *** georgem1 has joined #openstack-ansible | 22:03 | |
| *** georgem1 has quit IRC | 22:03 | |
| *** git-harry has quit IRC | 22:04 | |
| *** mancdaz has quit IRC | 22:04 | |
| *** andymccr has quit IRC | 22:05 | |
| *** git-harry has joined #openstack-ansible | 22:06 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:06 | |
| *** mancdaz has joined #openstack-ansible | 22:06 | |
| *** andymccr has joined #openstack-ansible | 22:07 | |
| *** fawadkhaliq has quit IRC | 22:12 | |
| *** fawadkhaliq has joined #openstack-ansible | 22:24 | |
| *** stevemar has quit IRC | 22:37 | |
| *** JRobinson__ has joined #openstack-ansible | 22:43 | |
| *** radek__ has quit IRC | 22:44 | |
| *** alextricity_r has joined #openstack-ansible | 22:45 | |
| *** alextricity_r has quit IRC | 22:46 | |
| *** daneyon has quit IRC | 22:52 | |
| *** daneyon has joined #openstack-ansible | 22:53 | |
| *** dkalleg has quit IRC | 22:55 | |
| *** dkalleg has joined #openstack-ansible | 22:56 | |
| *** dkalleg has quit IRC | 23:01 | |
| *** markvoelker has quit IRC | 23:16 | |
| *** JRobinson__ has quit IRC | 23:29 | |
| *** dkalleg has joined #openstack-ansible | 23:29 | |
| *** fawadkhaliq has quit IRC | 23:31 | |
| *** annashen has quit IRC | 23:31 | |
| *** annashen has joined #openstack-ansible | 23:34 | |
| *** darrenc is now known as darrenc_afk | 23:44 | |
| *** jaypipes has quit IRC | 23:46 | |
| *** annashen has quit IRC | 23:46 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 23:48 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
