bknudson | I still wonder why I could never recreate in devstack. | 00:07 |
---|---|---|
bknudson | because I can hit it all the time with this vagrant arrrsula install | 00:08 |
*** ddieterly has joined #openstack-keystone | 00:08 | |
*** david-lyle has quit IRC | 00:08 | |
notmorgan | yeaH, i dunno | 00:12 |
*** chlong has quit IRC | 00:13 | |
*** roxanagh_ has joined #openstack-keystone | 00:19 | |
*** chlong has joined #openstack-keystone | 00:26 | |
*** ravelar has quit IRC | 00:32 | |
openstackgerrit | Merged openstack/keystone: Doc fix: license rendered in published doc https://review.openstack.org/359284 | 00:37 |
openstackgerrit | Merged openstack/keystone: Add mapping_populate command https://review.openstack.org/343028 | 00:38 |
*** marekd2 has joined #openstack-keystone | 00:39 | |
*** su_zhang has quit IRC | 00:42 | |
*** su_zhang has joined #openstack-keystone | 00:42 | |
*** marekd2 has quit IRC | 00:44 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 00:45 |
*** su_zhang has quit IRC | 00:46 | |
*** markvoelker has joined #openstack-keystone | 00:55 | |
*** gyee has quit IRC | 00:59 | |
*** sdake has quit IRC | 01:06 | |
*** jamielennox is now known as jamielennox|away | 01:11 | |
*** jamielennox|away is now known as jamielennox | 01:11 | |
*** atod has quit IRC | 01:11 | |
*** esp has quit IRC | 01:18 | |
*** ddieterly has quit IRC | 01:18 | |
*** davechen has joined #openstack-keystone | 01:19 | |
*** ravelar has joined #openstack-keystone | 01:25 | |
*** roxanagh_ has quit IRC | 01:34 | |
*** markvoelker has quit IRC | 01:36 | |
*** sdake has joined #openstack-keystone | 01:36 | |
*** code-R has joined #openstack-keystone | 01:36 | |
*** sdake_ has joined #openstack-keystone | 01:39 | |
*** sdake has quit IRC | 01:42 | |
*** wangqun has joined #openstack-keystone | 01:47 | |
*** ddieterly has joined #openstack-keystone | 01:49 | |
*** code-R_ has joined #openstack-keystone | 01:52 | |
*** sdake_ has quit IRC | 01:54 | |
*** code-R has quit IRC | 01:55 | |
*** ddieterly has quit IRC | 02:02 | |
*** sdake has joined #openstack-keystone | 02:04 | |
*** ddieterly has joined #openstack-keystone | 02:04 | |
*** jrist has quit IRC | 02:05 | |
*** gagehugo has joined #openstack-keystone | 02:05 | |
*** gagehugo has quit IRC | 02:07 | |
*** bigdogstl has joined #openstack-keystone | 02:12 | |
*** jrist has joined #openstack-keystone | 02:18 | |
*** ravelar has quit IRC | 02:26 | |
*** thiagolib has quit IRC | 02:27 | |
*** thiagolib has joined #openstack-keystone | 02:28 | |
*** serverascode has quit IRC | 02:28 | |
*** code-R_ has quit IRC | 02:29 | |
*** code-R has joined #openstack-keystone | 02:30 | |
*** ddieterly has quit IRC | 02:31 | |
*** serverascode has joined #openstack-keystone | 02:31 | |
*** sdake has quit IRC | 02:34 | |
*** sdake has joined #openstack-keystone | 02:35 | |
*** julim has quit IRC | 02:35 | |
*** Ephur has quit IRC | 02:37 | |
*** ddieterly has joined #openstack-keystone | 02:37 | |
*** ddieterly has quit IRC | 02:37 | |
*** sdake has quit IRC | 02:45 | |
*** sdake has joined #openstack-keystone | 02:47 | |
openstackgerrit | Dave Chen proposed openstack/keystone: Handle the exception from creating access token properly https://review.openstack.org/359795 | 02:57 |
*** marekd2 has joined #openstack-keystone | 02:59 | |
*** marekd2 has quit IRC | 03:04 | |
*** code-R has quit IRC | 03:06 | |
*** jamielennox is now known as jamielennox|away | 03:09 | |
*** woodster_ has quit IRC | 03:09 | |
*** sdake has quit IRC | 03:12 | |
*** tonytan4ever has quit IRC | 03:13 | |
*** Ephur has joined #openstack-keystone | 03:16 | |
openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: [WIP] Migrate to keystoneauth https://review.openstack.org/359708 | 03:22 |
*** chlong has quit IRC | 03:23 | |
stevemar | jamielennox|away: looks like ksm gate is broken | 03:23 |
stevemar | we somehow wedged ourselves | 03:23 |
*** jamielennox|away is now known as jamielennox | 03:23 | |
jamielennox | stevemar: oh? | 03:25 |
jamielennox | i was just looking at an old review that was stalled and i wasn't sure why | 03:25 |
jamielennox | what are you seeing? | 03:25 |
openstackgerrit | Jamie Lennox proposed openstack/keystonemiddleware: Refactor audit tests to use create_middleware https://review.openstack.org/336971 | 03:25 |
openstackgerrit | Jamie Lennox proposed openstack/keystonemiddleware: Use oslo_messaging conf fixture https://review.openstack.org/336970 | 03:25 |
openstackgerrit | Jamie Lennox proposed openstack/keystonemiddleware: Extract oslo_messaging specific audit tests https://review.openstack.org/334296 | 03:25 |
openstackgerrit | Jamie Lennox proposed openstack/keystonemiddleware: Return and use an app wherever possible https://review.openstack.org/336972 | 03:25 |
openstackgerrit | Jamie Lennox proposed openstack/keystonemiddleware: Use the mocking fixture in notifier tests https://review.openstack.org/334295 | 03:25 |
stevemar | jamielennox: https://review.openstack.org/#/c/359675/ and https://review.openstack.org/#/c/359514/ | 03:25 |
jamielennox | that was just a rebase | 03:26 |
jamielennox | which i probably shouldn't have done because it was the same failure | 03:26 |
stevemar | eh ? | 03:26 |
jamielennox | nvm | 03:27 |
*** bigdogstl has quit IRC | 03:27 | |
*** atod has joined #openstack-keystone | 03:27 | |
jamielennox | oh - at least i can reproduce it seems | 03:29 |
jamielennox | there's a new requests version about a week ago... | 03:29 |
jamielennox | oh, and there's a ksa more recent than that.. | 03:30 |
*** Ephur has quit IRC | 03:30 | |
*** dkehn_ has quit IRC | 03:31 | |
*** bigdogstl has joined #openstack-keystone | 03:31 | |
*** bigdogstl has quit IRC | 03:35 | |
*** chlong has joined #openstack-keystone | 03:35 | |
stevemar | jamielennox: the ksa changes were tiny | 03:37 |
jamielennox | yea, but i think the relative one might have done something | 03:37 |
stevemar | https://github.com/openstack/keystoneauth/compare/2.11.0...2.12.0 | 03:37 |
stevemar | ah | 03:38 |
stevemar | i was thinking requests-mock, but that didn't change | 03:38 |
*** chlong has quit IRC | 03:38 | |
*** chlong has joined #openstack-keystone | 03:38 | |
stevemar | jamielennox: easy enough to test | 03:39 |
jamielennox | stevemar: yea, it's that one | 03:39 |
jamielennox | stevemar: previously get_endpoint would return None when it couldn't match something | 03:39 |
stevemar | jamielennox: did you want to fix it or shall i revert and release? | 03:39 |
jamielennox | now with the urljoin it does join(original_url, None) | 03:40 |
jamielennox | so it should be an easy fix | 03:40 |
*** bigdogstl has joined #openstack-keystone | 03:40 | |
*** EinstCrazy has joined #openstack-keystone | 03:41 | |
*** dkehn_ has joined #openstack-keystone | 03:43 | |
jamielennox | stevemar: bug 1616720 | 03:45 |
openstack | bug 1616720 in keystoneauth "get_endpoint returns URL where it should return None" [Undecided,New] https://launchpad.net/bugs/1616720 | 03:45 |
*** bigdogstl has quit IRC | 03:47 | |
*** bigdogstl has joined #openstack-keystone | 03:49 | |
*** bigdogstl has quit IRC | 03:51 | |
openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: get_endpoint should return None when no version found https://review.openstack.org/360209 | 03:52 |
jamielennox | stevemar: ^^ | 03:52 |
openstackgerrit | Merged openstack/keystone: Remove unnecessary __init__ https://review.openstack.org/359063 | 03:53 |
*** bigdogstl has joined #openstack-keystone | 03:54 | |
stevemar | samueldmq: one for you in the morning https://review.openstack.org/#/c/360209/1 and we'll need to release a new keystoneauth | 03:55 |
*** esp has joined #openstack-keystone | 03:56 | |
openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: Allow specifying client and service info to user_agent https://review.openstack.org/357633 | 03:59 |
*** dikonoor has joined #openstack-keystone | 04:04 | |
*** tonytan4ever has joined #openstack-keystone | 04:13 | |
*** tonytan4ever has quit IRC | 04:18 | |
*** davechen has left #openstack-keystone | 04:19 | |
*** EinstCrazy has quit IRC | 04:24 | |
*** esp has quit IRC | 04:25 | |
openstackgerrit | Merged openstack/keystone: Doc fix: "keystone-manage upgrade" is not a thing https://review.openstack.org/359281 | 04:26 |
*** iurygregory_ has quit IRC | 04:28 | |
*** bigdogstl has quit IRC | 04:30 | |
*** links has joined #openstack-keystone | 04:37 | |
*** aswadr_ has joined #openstack-keystone | 04:41 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 04:48 |
openstackgerrit | Nam Nguyen Hoai proposed openstack/keystone: Cleaning imports in code https://review.openstack.org/360228 | 04:55 |
openstackgerrit | Merged openstack/keystone: Remove mox from test-requirements https://review.openstack.org/359931 | 05:00 |
openstackgerrit | Merged openstack/keystone: TrivialFix: Remove logging import unused https://review.openstack.org/359760 | 05:01 |
*** ayoung has quit IRC | 05:03 | |
*** EinstCrazy has joined #openstack-keystone | 05:09 | |
*** jaosorior has joined #openstack-keystone | 05:10 | |
*** EinstCrazy has quit IRC | 05:11 | |
*** ravelar has joined #openstack-keystone | 05:24 | |
*** ravelar has quit IRC | 05:28 | |
dikonoor | stevemar:hi steve..are you around? | 05:29 |
*** richm has quit IRC | 05:40 | |
*** david-lyle has joined #openstack-keystone | 05:43 | |
stevemar | dikonoor: only for the next 2 minutes :P | 05:46 |
dikonoor | stevemar:i will be quick.. | 05:47 |
dikonoor | stevemar: I am using fernet token with memcached and I am just wondering if it makes sense to restart memcahced service everytime we restart keystone | 05:48 |
*** davechen has joined #openstack-keystone | 05:48 | |
dikonoor | stevemar: would that be something recommended? (I understand that restarting memcache basically means cleaning up the cache and there's no direct correlation between restart keystone and memcached because both are different services altogether) | 05:49 |
dikonoor | and restarting keystone is generally required only if some configuration parameters have changed or some code has changed etc.. | 05:50 |
stevemar | dikonoor: thats a good question | 05:51 |
dikonoor | but isn't it possible that if we are using a sql identity backend and then switch to ldap, without restarting memcached , the old tokens are going to function for some more time when they actually shouldn;t be | 05:51 |
stevemar | dikonoor: i would think "switching to ldap" would be part of some scheduled down time or that would be expected | 05:52 |
stevemar | it probably doesn't hurt to restart memcache when you restart keystone | 05:52 |
dikonoor | or perhaps some changes in assignments..for eg. some assignments are removed ..I saw some code related to assignments and id_mapping tables etc where the oslo_Cache regions are invalidated whenever there's a db change but I found that that's not reflected until memcahced service is restarted (in the usecase I tried) | 05:53 |
*** atod has quit IRC | 05:55 | |
dikonoor | stevemar: So, there are really no negative effects to restarting memcached while restarting keystone apart from the fact that the cache would get cleared | 05:55 |
stevemar | dikonoor: AFAIK, you are correct | 05:56 |
dikonoor | stevemar: Thanks Steve..thanks for the extra min you spared :D | 05:56 |
stevemar | dikonoor: we've had a few bugs with cache lately, where weird things are being cached, so its probably best to take the performance hit and clear the cache instead of filling your logs with exceptions or just getting weird behaviour | 05:57 |
stevemar | dikonoor: if breton comes online while you're here i recommend bugging him, he's got hands-on experience | 05:58 |
dikonoor | stevemar: ok.. I will try and talk to breton today | 05:59 |
stevemar | dikonoor: cool, sorry for running off, been on 4 flights in 3 days, i'm going to do this now: https://media.giphy.com/media/RkzJzW4xUm0X6/giphy.gif | 05:59 |
dikonoor | stevemar : go ahead.. looks terrible ;-P | 06:00 |
*** atod has joined #openstack-keystone | 06:01 | |
*** edtubill has quit IRC | 06:03 | |
*** adriant has quit IRC | 06:03 | |
*** david-lyle has quit IRC | 06:09 | |
openstackgerrit | Anh Tran proposed openstack/keystone: api-ref: Splitting status lines in API v3. https://review.openstack.org/360264 | 06:19 |
openstackgerrit | Anh Tran proposed openstack/keystone: api-ref: Splitting status lines in API v3-ext. https://review.openstack.org/360267 | 06:23 |
*** rcernin has joined #openstack-keystone | 06:28 | |
*** Gorian has joined #openstack-keystone | 06:32 | |
*** nikhil has quit IRC | 06:36 | |
*** nikhil has joined #openstack-keystone | 06:38 | |
*** zhangjl has joined #openstack-keystone | 07:00 | |
*** tesseract- has joined #openstack-keystone | 07:08 | |
*** atod has quit IRC | 07:11 | |
*** tonytan4ever has joined #openstack-keystone | 07:15 | |
*** tonytan4ever has quit IRC | 07:20 | |
*** AndyWojo has quit IRC | 07:29 | |
*** AndyWojo has joined #openstack-keystone | 07:32 | |
*** sheel has joined #openstack-keystone | 07:37 | |
*** yarkot1 has quit IRC | 07:40 | |
*** zzzeek has quit IRC | 08:00 | |
*** zzzeek has joined #openstack-keystone | 08:00 | |
*** jpena|away is now known as jpena | 08:09 | |
openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 08:10 |
*** asettle has joined #openstack-keystone | 08:20 | |
*** dikonoor has quit IRC | 08:21 | |
*** dikonoor has joined #openstack-keystone | 08:21 | |
*** xiaoyang has joined #openstack-keystone | 08:28 | |
openstackgerrit | Nam Nguyen Hoai proposed openstack/keystone: Cleaning imports in code https://review.openstack.org/360228 | 08:37 |
breton | huh? | 08:41 |
openstackgerrit | Jamie Lennox proposed openstack/keystone: Use egg form of osprofiler in paste pipeline https://review.openstack.org/360337 | 08:41 |
*** marekd2 has joined #openstack-keystone | 08:42 | |
openstackgerrit | Merged openstack/keystone: [api-ref]: Outdated link reference https://review.openstack.org/359631 | 08:42 |
breton | dikonoor: yep, we have a bug with caching that has impact on assignments, catalog and even tokens. You definitely should restart memcached as a workaround for now. | 08:43 |
dikonoor | breton: thanks for pinging..Do you have the LP bug handy so that I can keep track? | 08:43 |
*** openstackgerrit has quit IRC | 08:48 | |
*** openstackgerrit has joined #openstack-keystone | 08:48 | |
*** atod has joined #openstack-keystone | 08:49 | |
*** atod has quit IRC | 08:54 | |
*** root has joined #openstack-keystone | 08:59 | |
*** root is now known as Guest73980 | 08:59 | |
breton | dikonoor: bug 1590779 | 09:04 |
openstack | bug 1590779 in oslo.cache "Cache region invalidation works for local CacheRegion object only" [Undecided,In progress] https://launchpad.net/bugs/1590779 - Assigned to Alexander Makarov (amakarov) | 09:04 |
dikonoor | breton : Thanks ! | 09:04 |
openstackgerrit | Nam Nguyen Hoai proposed openstack/keystone: Cleaning imports in code https://review.openstack.org/360228 | 09:10 |
*** tonytan4ever has joined #openstack-keystone | 09:16 | |
*** Guest73980 has quit IRC | 09:19 | |
*** tonytan4ever has quit IRC | 09:20 | |
*** dkehn_ has quit IRC | 09:22 | |
*** xiaoyang has quit IRC | 09:24 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 09:24 |
openstackgerrit | Dave Chen proposed openstack/keystone: Handle the exception from creating access token properly https://review.openstack.org/359795 | 09:37 |
*** dkehn_ has joined #openstack-keystone | 09:39 | |
*** davechen has left #openstack-keystone | 09:39 | |
*** mkoderer__ has joined #openstack-keystone | 09:49 | |
*** wangqun has quit IRC | 10:05 | |
*** richm has joined #openstack-keystone | 10:07 | |
openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 10:10 |
*** hockeynut has quit IRC | 10:23 | |
*** zhangjl has quit IRC | 10:30 | |
samueldmq | stevemar: ack, approved, waiting on merge to propose a new release | 10:31 |
*** jed56 has quit IRC | 10:35 | |
*** atod has joined #openstack-keystone | 10:38 | |
*** newcomer25 has joined #openstack-keystone | 10:40 | |
*** atod has quit IRC | 10:43 | |
openstackgerrit | Ron De Rose proposed openstack/keystone: Relax the requirement for mappings to result in group memberships https://review.openstack.org/358111 | 10:54 |
*** asettle has quit IRC | 11:01 | |
*** asettle has joined #openstack-keystone | 11:02 | |
*** tonytan4ever has joined #openstack-keystone | 11:17 | |
*** rodrigods has quit IRC | 11:21 | |
*** rodrigods has joined #openstack-keystone | 11:21 | |
*** tonytan4ever has quit IRC | 11:22 | |
*** sheel has quit IRC | 11:26 | |
*** jaosorior has quit IRC | 11:52 | |
*** jaosorior has joined #openstack-keystone | 11:52 | |
*** asettle has quit IRC | 11:55 | |
*** asettle has joined #openstack-keystone | 11:56 | |
*** jpena is now known as jpena|lunch | 12:04 | |
*** tonytan4ever has joined #openstack-keystone | 12:10 | |
*** su_zhang has joined #openstack-keystone | 12:14 | |
*** tonytan_brb has joined #openstack-keystone | 12:17 | |
*** tonytan4ever has quit IRC | 12:19 | |
*** sc68cal has quit IRC | 12:20 | |
*** jaugustine has joined #openstack-keystone | 12:20 | |
*** mfisch has quit IRC | 12:21 | |
*** sc68cal has joined #openstack-keystone | 12:21 | |
*** bradjones has quit IRC | 12:21 | |
*** briancurtin has quit IRC | 12:23 | |
*** vern has quit IRC | 12:23 | |
*** vern has joined #openstack-keystone | 12:24 | |
*** sigmavirus has quit IRC | 12:24 | |
*** mfisch has joined #openstack-keystone | 12:25 | |
*** mfisch has quit IRC | 12:25 | |
*** mfisch has joined #openstack-keystone | 12:25 | |
*** jaosorior is now known as jaosorior_away | 12:25 | |
*** _sigmavirus24 has joined #openstack-keystone | 12:25 | |
*** briancurtin has joined #openstack-keystone | 12:25 | |
*** atod has joined #openstack-keystone | 12:26 | |
*** bradjones has joined #openstack-keystone | 12:27 | |
*** bradjones has quit IRC | 12:27 | |
*** bradjones has joined #openstack-keystone | 12:27 | |
*** mtreinish has quit IRC | 12:27 | |
*** _sigmavirus24 is now known as sigmavirus | 12:30 | |
*** sigmavirus has joined #openstack-keystone | 12:30 | |
*** mtreinish has joined #openstack-keystone | 12:30 | |
*** atod has quit IRC | 12:30 | |
dstanek | rderose: you around? | 12:32 |
*** edmondsw has joined #openstack-keystone | 12:34 | |
samueldmq | stevemar: https://review.openstack.org/#/c/360495/ | 12:35 |
openstackgerrit | henry-nash proposed openstack/keystone: Modify sql banned operations for each of the new repos https://review.openstack.org/358723 | 12:37 |
samueldmq | stevemar: I saw there is a topic in next meeting to talk about versionedobjects vs triggers | 12:39 |
samueldmq | stevemar: how does this impact current proposed code ? should we work for that discussion to merge 355618 and 357789 ? | 12:40 |
samueldmq | stevemar: or perhaps we can work on merging them and refactoring later ? | 12:40 |
samueldmq | stevemar: my only concern are the deadlines | 12:40 |
*** pauloewerton has joined #openstack-keystone | 12:43 | |
dstanek | samueldmq: that is a great question | 12:44 |
dstanek | after the chat yesterday i had the feeling that we were full steam on triggers | 12:44 |
*** ddieterly has joined #openstack-keystone | 12:45 | |
henrynash | lbragstad: ping | 12:46 |
*** markvoelker has joined #openstack-keystone | 12:48 | |
*** code-R has joined #openstack-keystone | 12:52 | |
openstackgerrit | Mikhail Nikolaenko proposed openstack/keystone: [WIP] Move fernet utils to backend https://review.openstack.org/356499 | 12:52 |
samueldmq | dstanek: I missed the convo yesterday, but needing to write DDLs directly seems not ideal given we want to support multiple dbs | 12:53 |
henrynash | lbragstad: I think I know what is going on with the trigger dopping not working with sqlite....it's because in each of our scrips we preceed the drop with a table alter...and sqlite doesn't really support table alter (it creates a new table, copies the data)...I'm betting that it has renamed the trigger in the process | 12:53 |
*** nkinder has joined #openstack-keystone | 12:54 | |
*** mgagne_ is now known as mgagne | 12:54 | |
*** ddieterly has quit IRC | 12:55 | |
*** jpena|lunch is now known as jpena | 12:56 | |
*** woodster_ has joined #openstack-keystone | 12:57 | |
dstanek | samueldmq: we only want to support 2 so it's not terribad | 12:57 |
*** edtubill has joined #openstack-keystone | 12:58 | |
samueldmq | dstanek: postgresql & mysql? | 12:59 |
samueldmq | what about galera? | 12:59 |
dstanek | samueldmq: yessir - galera is mysql | 12:59 |
samueldmq | dstanek: hmm | 13:00 |
*** jrist has quit IRC | 13:00 | |
samueldmq | dstanek: it'd be nice if sqlalchemy supported triggers | 13:00 |
samueldmq | so we wouldn't need to write in ddl | 13:00 |
dstanek | it's a tradeoff. then our code might look horribly complex and hard to understand | 13:00 |
bknudson | sqlalchemy is open source | 13:01 |
bknudson | if you want it to support triggers go ahead and add it | 13:01 |
*** yarkot1 has joined #openstack-keystone | 13:03 | |
samueldmq | perhaps we could get the same effect with ORM events http://docs.sqlalchemy.org/en/latest/orm/events.html | 13:03 |
dstanek | SQL is remarkably declarative and easy to write a DSL for. with triggers you can use many features of an imperative language | 13:04 |
dstanek | even with SQLA you may have to drop down to raw SQL for very complex queries or to make certain queries more performant | 13:04 |
bknudson | if you're going to do that then don't use sqla. | 13:06 |
samueldmq | bknudson: why? | 13:07 |
bknudson | the whole point of sqla is to give you a database abstraction so that you get a consistent interface for any db | 13:08 |
bknudson | so if you're going to do raw sql you don't have an abstract database layer anymore. | 13:09 |
*** julim_ has joined #openstack-keystone | 13:09 | |
lbragstad | henrynash ah - interesting | 13:09 |
henrynash | lbragstad: if I temporarily remove the alter in mine, it works without the "IF EXISTS" | 13:10 |
lbragstad | henrynash where is that? | 13:10 |
henrynash | lbragstad: i just tested it locally by commenting out where I make a column non nullable | 13:11 |
henrynash | lbragstad: in my 002 contract .py script | 13:11 |
dstanek | bknudson: why would you not use it where applicable? | 13:11 |
*** su_zhang has quit IRC | 13:12 | |
henrynash | lbragstad: am trying to see if I can see if sqlite really does do this by testing it outside keytsone | 13:12 |
*** jrist has joined #openstack-keystone | 13:13 | |
bknudson | dstanek: so some parts of the application are database-agnostic and some are database-specific? If that's the case then your application is database-specific anyways. | 13:13 |
lbragstad | henrynash so in your case it is line 27 | 13:13 |
lbragstad | in your contract script that is nuking the triggers | 13:13 |
samueldmq | bknudson: I agree. I was hoping that we could write sqla-like code when using ORM events | 13:13 |
henrynash | lbragstad: well, line 27 of my contract script | 13:14 |
bknudson | we've seen a problem in the past where we or sqla made assumptions about the name of an index. Now we're going to have our own code making assumptions about what sqla does. | 13:14 |
henrynash | lbragstad: ..but the 2nd active line, yes | 13:14 |
lbragstad | henrynash interesting - so i wonder what's doing that in my script? | 13:15 |
henrynash | lbragstad: drop column | 13:15 |
*** julim_ has quit IRC | 13:15 | |
lbragstad | henrynash ah.... | 13:16 |
lbragstad | that sucks | 13:16 |
lbragstad | so sqla will create a new table without the column being dropped | 13:17 |
lbragstad | which is why the trigger doesn't exist. | 13:17 |
henrynash | lbragstad: for sqlite only, I believe so, | 13:17 |
*** newcomer25 has quit IRC | 13:17 | |
*** julim_ has joined #openstack-keystone | 13:17 | |
henrynash | lbragstad: I'd have thought it copies the trigger...maybe giving it a different name? | 13:17 |
henrynash | lbragstad: this is still a theory.... | 13:18 |
*** newcomer25 has joined #openstack-keystone | 13:18 | |
henrynash | lbragstad: sqlite only really properly supports column adding...any other alter results in the table dance | 13:19 |
bknudson | we don't have to worry about people migrating sqlite databases. | 13:21 |
*** links has quit IRC | 13:22 | |
*** Ephur has joined #openstack-keystone | 13:22 | |
lbragstad | henrynash so this is actually a sqlalchemy + sqlite problem? | 13:24 |
dstanek | bknudson: no, even when you have to support only a single DB using something like SQLA is benficial even if you have to dip into SQL for one specific case | 13:29 |
lbragstad | do we have docs on running keystone tests against mysql or postgres? | 13:30 |
*** newcomer25 has quit IRC | 13:30 | |
lbragstad | henrynash doing ^ locally might help figure out is this is strictly an issue with sql-a + sqlite | 13:30 |
*** newcomer25 has joined #openstack-keystone | 13:31 | |
*** roxanagh_ has joined #openstack-keystone | 13:32 | |
henrynash | lbragstad: yep, right now I'm not sure exactly what the issue is, but the gun is smoking.... | 13:32 |
*** asettle has quit IRC | 13:35 | |
*** asettle has joined #openstack-keystone | 13:35 | |
*** newcomer25 has quit IRC | 13:35 | |
*** roxanagh_ has quit IRC | 13:36 | |
*** edtubill has quit IRC | 13:36 | |
*** newcomer25 has joined #openstack-keystone | 13:38 | |
openstackgerrit | Mikhail Nikolaenko proposed openstack/keystone: [WIP] Move fernet utils to backend https://review.openstack.org/356499 | 13:41 |
lbragstad | do we not use ENABLE_LIVE_MYSQL_TEST anymore? | 13:42 |
*** newcomer25 has left #openstack-keystone | 13:45 | |
*** jaosorior_away is now known as jaosorior | 13:48 | |
*** code-R_ has joined #openstack-keystone | 13:49 | |
openstackgerrit | David Stanek proposed openstack/keystone: Removes old, unused code https://review.openstack.org/360561 | 13:50 |
*** code-R_ has quit IRC | 13:51 | |
*** code-R has quit IRC | 13:51 | |
*** code-R_ has joined #openstack-keystone | 13:52 | |
*** ddieterly has joined #openstack-keystone | 13:54 | |
*** ddieterly has quit IRC | 13:56 | |
*** ddieterly has joined #openstack-keystone | 13:57 | |
*** su_zhang has joined #openstack-keystone | 13:59 | |
*** jaosorior has quit IRC | 14:08 | |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Remove unused global variable from unit tests https://review.openstack.org/360575 | 14:10 |
*** hockeynut has joined #openstack-keystone | 14:11 | |
*** amakarov_away is now known as amakarov | 14:11 | |
*** code-R has joined #openstack-keystone | 14:12 | |
*** code-R_ has quit IRC | 14:16 | |
*** ravelar has joined #openstack-keystone | 14:20 | |
*** michauds has joined #openstack-keystone | 14:21 | |
*** esp has joined #openstack-keystone | 14:23 | |
*** slberger has joined #openstack-keystone | 14:26 | |
dstanek | lbragstad: are you talking about running all of the unit tests? | 14:27 |
lbragstad | dstanek yeah - i'm having some issues with a combination of sqla and sqlite... I was going to try and test my credential encryption patch against a mysql database | 14:28 |
dstanek | lbragstad: right now it's not great. a patch i'm working on now deals with the setup/cleanup. lots of fun | 14:29 |
*** dikonoor has quit IRC | 14:29 | |
lbragstad | dstanek got it | 14:33 |
lbragstad | dstanek we test keystone on mysql and postgres in the gate don't we? | 14:33 |
dstanek | lbragstad: not unit tests | 14:34 |
lbragstad | dstanek what do we test? | 14:34 |
lbragstad | with postgres and mysql? | 14:34 |
dstanek | tempest | 14:34 |
lbragstad | oh | 14:34 |
*** edtubill has joined #openstack-keystone | 14:35 | |
dstanek | our unit tests can't currently run against mysql | 14:35 |
*** spedione|AWAY is now known as spedione | 14:35 | |
bknudson | there are unit tests that run against mysql and postgresql in the gate | 14:35 |
bknudson | http://logs.openstack.org/41/359941/1/check/gate-keystone-python27-db-ubuntu-xenial/f4034c0/console.html#_2016-08-24_17_11_36_770971 | 14:36 |
bknudson | http://logs.openstack.org/41/359941/1/check/gate-keystone-python27-db-ubuntu-xenial/f4034c0/console.html#_2016-08-24_17_11_43_827979 | 14:36 |
bknudson | http://logs.openstack.org/41/359941/1/check/gate-keystone-python27-db-ubuntu-xenial/f4034c0/console.html#_2016-08-24_17_16_17_653426 | 14:36 |
bknudson | we only have tests for the database migrations as live tests. | 14:37 |
dstanek | bknudson: i didn't realize that we ran any of them in the gate | 14:39 |
bknudson | I think the nova team got a live database set up for their own testing so it's available to us too. | 14:39 |
bknudson | and then oslo.db added a test fixture for it. | 14:40 |
*** sdake has joined #openstack-keystone | 14:42 | |
*** spedione is now known as spedione|AWAY | 14:43 | |
dstanek | bknudson: i haven't seen anything useful so i started hacking up our database fixture | 14:44 |
bknudson | we probably have some old "live" test code around that doesn't work | 14:46 |
*** openstackgerrit has quit IRC | 14:48 | |
*** openstackgerrit has joined #openstack-keystone | 14:49 | |
*** hockeynut has quit IRC | 14:51 | |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 14:52 |
*** david-lyle has joined #openstack-keystone | 14:53 | |
*** ruoyu has joined #openstack-keystone | 14:59 | |
*** david-lyle has quit IRC | 14:59 | |
dstanek | lbragstad: want to play with the mysql stuff in a little bit? | 15:02 |
lbragstad | dstanek yeah - i would like to do that | 15:02 |
lbragstad | i'm going to have to in order to move forward with credential encryption | 15:03 |
*** tonytan4ever has joined #openstack-keystone | 15:03 | |
*** tonytan_brb has quit IRC | 15:06 | |
*** spedione|AWAY is now known as spedione | 15:07 | |
*** gb21 has joined #openstack-keystone | 15:14 | |
*** su_zhang has quit IRC | 15:16 | |
*** code-R has quit IRC | 15:19 | |
ruoyu | Hello! My name is Ruoyu Chen. I am a newbie in openstack keystone. Our team want to get realtime user information from keystone logs. Logging_context_format_string should print user id in the logs but it doesn’t work. For an example, when I log in with a wrong password, in keystone I only got log message like “2016-08-22 17:00:15.396 2707 WARNING keystone.common.wsgi [req-c9ab9248-4f94-4ed0-9005-10fe3c5e5486 - - | 15:20 |
ruoyu | - - -] Authorization failed. The request you have made requires authentication. from 10.14.37.215”. We think the user id should be next to the request id but we only got ‘-‘. Any instruction is greatly appreciated. Thank you very much! | 15:20 |
*** code-R has joined #openstack-keystone | 15:21 | |
*** links has joined #openstack-keystone | 15:21 | |
*** code-R has quit IRC | 15:26 | |
openstackgerrit | Alexander Makarov proposed openstack/keystone: Unified delegation assignment driver https://review.openstack.org/291318 | 15:26 |
*** hockeynut has joined #openstack-keystone | 15:27 | |
lbragstad | dstanek dolphm see my note here - https://review.openstack.org/#/c/355618/18/keystone/common/sql/contract_repo/versions/002_remove_unencrypted_blob_column_from_credential.py | 15:30 |
*** hockeynu_ has joined #openstack-keystone | 15:37 | |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 15:38 |
openstackgerrit | henry-nash proposed openstack/keystone: Fix issue of password created_at being left as nullable https://review.openstack.org/357789 | 15:39 |
mfisch | dolphm or lbragstad can you point me at the keystone tuning guide for the ops midcycle? I seem to have lost it | 15:40 |
henrynash | lbragstad: I think we should prioritise your patch over mine (since yours is a feature and needs to be merged quick), while mine is a bug fix...I'll rebase mine on to yours to stop migration number clashes | 15:41 |
*** hockeynut has quit IRC | 15:41 | |
lbragstad | henrynash works for me | 15:41 |
*** roxanagh_ has joined #openstack-keystone | 15:42 | |
dolphm | mfisch: http://docs.openstack.org/developer/keystone/performance.html | 15:43 |
mfisch | thanks | 15:43 |
dstanek | bknudson: it looks like it runs all unit tests against the DB in gate-keystone-python27-db-ubuntu-xenial. is that the case? | 15:43 |
lbragstad | henrynash trying to get this running against mysql so we can test real triggers | 15:45 |
lbragstad | versus sqlite | 15:45 |
henrynash | lbragstad: I have all three running on my machine | 15:45 |
lbragstad | henrynash how are you testing all this locally? | 15:46 |
henrynash | lbragstad: key is to add user/password combo of "openstack_citest" to each of your local dbs | 15:46 |
*** BjoernT has joined #openstack-keystone | 15:46 | |
*** roxanagh_ has quit IRC | 15:46 | |
henrynash | lbragstad: then test_sql_upgrade will test on all three | 15:47 |
lbragstad | henrynash are there docs on that process? | 15:47 |
bknudson | dstanek: the -db jobs set up the database servers for us and the keystone tests check to see if the database is there and run if it is. | 15:48 |
henrynash | lbragstad: not really, there are a few lines in the oslo_db documentation.....but I had to deduce it by looking at the code | 15:48 |
dstanek | bknudson: ah, i see. that makes sense | 15:48 |
bknudson | if you set up a database on your dev box it'll run the tests | 15:49 |
henrynash | lbragstad: basically if ensure you have a user called 'openstack_citest' with a password of 'openstack_citest' in each db (mysql and postgresql) then they will get run | 15:49 |
bknudson | I might have an ansible script that sets it up... | 15:49 |
dstanek | bknudson: i was just wondering how all the other tests would run without my patches...but they don't :-) | 15:49 |
henrynash | lbragstad: (and that user has full writes to create a database) | 15:50 |
henrynash | (rights!) | 15:50 |
bknudson | it would be pretty cool if all the functional-style tests ran with a live db, but there's only a few of them. | 15:50 |
henrynash | bknudson: ++ | 15:51 |
henrynash | lbragstad: you are meant to also be able to set an env variable that oslo.db will pick up that gives alternative credentials, but I couldn't make that work | 15:51 |
henrynash | lbragstad: OS_TEST_DBAPI_ADMIN_CONNECTION | 15:52 |
bknudson | it would probably slow the gate down too much | 15:52 |
dstanek | bknudson: yes, orders of magnitude locally :-( | 15:55 |
*** chrisshattuck has joined #openstack-keystone | 15:56 | |
*** ruoyu has quit IRC | 16:00 | |
*** links has quit IRC | 16:00 | |
stevemar | o/ | 16:03 |
*** dikonoor has joined #openstack-keystone | 16:04 | |
*** Marcellin_ has joined #openstack-keystone | 16:04 | |
*** chrisshattuck has quit IRC | 16:12 | |
dstanek | stevemar! | 16:14 |
stevemar | dstanek: yo | 16:15 |
*** tonytan_brb has joined #openstack-keystone | 16:15 | |
lbragstad | dolphm my new test - http://cdn.pasteraw.com/qmbrn5b5za0jz5w4b11ftymaj25l34t | 16:16 |
*** tonytan4ever has quit IRC | 16:17 | |
*** michauds has quit IRC | 16:21 | |
*** michauds has joined #openstack-keystone | 16:21 | |
*** ruoyu has joined #openstack-keystone | 16:22 | |
notmorgan | bknudson: everytjing looking better sans threads? | 16:22 |
notmorgan | ill dig into threading issues with memcache later today if so | 16:22 |
bknudson | notmorgan: I'm working on integrating the change into our deployment tool (arrrsula) | 16:23 |
bknudson | and then I'm going to try it out | 16:23 |
notmorgan | okie | 16:23 |
notmorgan | but so far so good in the drv env? | 16:23 |
notmorgan | dev* | 16:23 |
bknudson | when I ran yesterday I didn't get the error. | 16:24 |
bknudson | my first test today was setting threads=1 and enable-threds=true and that failed like before | 16:24 |
bknudson | so now I'm going to set enable-threads=false which should work | 16:24 |
notmorgan | ok cool ill do some synthetic tests with threadong specifically then | 16:25 |
openstackgerrit | Boris Bobrov proposed openstack/keystone: Faster id mapping lookup https://review.openstack.org/339294 | 16:25 |
bknudson | I was never able to recreate this in devstack. | 16:25 |
bknudson | if you want to try out the arrsula install, it's all open source. | 16:25 |
bknudson | notmorgan: here's my test program: https://github.com/brantlk/keystone_samples/blob/master/revocation_event_test.py | 16:26 |
*** gyee has joined #openstack-keystone | 16:27 | |
*** hockeynu_ has quit IRC | 16:28 | |
*** dikonoor has quit IRC | 16:28 | |
*** hockeynut has joined #openstack-keystone | 16:29 | |
*** Gorian|work has joined #openstack-keystone | 16:29 | |
*** lamt has quit IRC | 16:31 | |
openstackgerrit | Dolph Mathews proposed openstack/keystone: Let upgrade tests control all 4 repositories at once https://review.openstack.org/360667 | 16:31 |
*** roxanagh_ has joined #openstack-keystone | 16:43 | |
openstackgerrit | henry-nash proposed openstack/keystone: Fix issue of password created_at being left as nullable https://review.openstack.org/357789 | 16:45 |
*** rcernin has quit IRC | 16:45 | |
*** roxanagh_ has quit IRC | 16:47 | |
*** david-lyle has joined #openstack-keystone | 16:50 | |
stevemar | rderose: unit test for this patch? https://review.openstack.org/#/c/359675/3 really? | 16:54 |
*** ruoyu has quit IRC | 16:55 | |
*** tesseract- has quit IRC | 16:56 | |
rderose | stevemar: alright, I live without it. but... it wouldn't be hard to add a unit test for that :) | 16:57 |
stevemar | rderose: we've never added unit tests to make sure other things are translated | 16:57 |
*** su_zhang has joined #openstack-keystone | 16:58 | |
stevemar | maybe create a hacking change that detects if you're printing a user facing message, then mark it for translation | 16:58 |
*** dikonoor has joined #openstack-keystone | 16:59 | |
rderose | stevemar: we should check (test) to make sure that it is i18n compatible | 16:59 |
*** marekd2 has quit IRC | 16:59 | |
rderose | stevemar: right, for user facing messages | 17:00 |
*** marekd2 has joined #openstack-keystone | 17:00 | |
rderose | stevemar: but I'm okay with letting that patch go | 17:00 |
*** edtubill has quit IRC | 17:01 | |
*** marekd2 has quit IRC | 17:01 | |
*** marekd2 has joined #openstack-keystone | 17:01 | |
*** edtubill has joined #openstack-keystone | 17:01 | |
*** chrisshattuck has joined #openstack-keystone | 17:03 | |
*** hockeynut has quit IRC | 17:05 | |
*** marekd2 has quit IRC | 17:06 | |
*** hockeynut has joined #openstack-keystone | 17:08 | |
*** asettle has quit IRC | 17:08 | |
*** asettle has joined #openstack-keystone | 17:09 | |
*** asettle has quit IRC | 17:13 | |
*** hockeynut has quit IRC | 17:19 | |
mfisch | stevemar: any value in me adding to the keystone docs on upgrading and adding a section on how to do an upgrade of a multinode cluster with minimal downtime? | 17:28 |
stevemar | mfisch: of course! | 17:29 |
mfisch | ok | 17:29 |
mfisch | I'll leave the existing stuff alone since it explains the base case | 17:29 |
stevemar | mfisch: i guess you just saw our upgrade note on the mailing list? | 17:29 |
mfisch | yeah | 17:29 |
mfisch | id be horrified if anyone used that current process as gospel ;) | 17:29 |
*** code-R has joined #openstack-keystone | 17:29 | |
bknudson | "multinode cluster" -- is there any other kind? | 17:30 |
*** roxanaghe_ has joined #openstack-keystone | 17:30 | |
*** roxanaghe__ has joined #openstack-keystone | 17:31 | |
mfisch | bknudson: sure with docker you can do a cluster on 1 physical node ;) | 17:32 |
bknudson | you can probably do ha too | 17:32 |
*** roxanaghe has quit IRC | 17:33 | |
*** code-R has quit IRC | 17:34 | |
*** roxanaghe_ has quit IRC | 17:34 | |
*** dikonoor has quit IRC | 17:36 | |
*** ddieterly is now known as ddieterly[away] | 17:37 | |
*** roxanaghe_ has joined #openstack-keystone | 17:39 | |
*** roxanaghe has joined #openstack-keystone | 17:41 | |
*** roxanaghe__ has quit IRC | 17:42 | |
*** roxanaghe_ has quit IRC | 17:43 | |
rderose | mfisch stevemar: http://docs.openstack.org/developer/keystone/upgrading.html | 17:48 |
rderose | mfisch: Upgrading with downtime section | 17:49 |
mfisch | I am enhancing it | 17:49 |
rderose | mfisch: cool | 17:49 |
*** Ephur has quit IRC | 17:49 | |
*** roxanaghe_ has joined #openstack-keystone | 17:50 | |
*** hockeynut has joined #openstack-keystone | 17:50 | |
*** hockeynut has quit IRC | 17:51 | |
*** roxanaghe__ has joined #openstack-keystone | 17:51 | |
*** hockeynut has joined #openstack-keystone | 17:53 | |
*** rcernin has joined #openstack-keystone | 17:54 | |
*** roxanaghe has quit IRC | 17:54 | |
*** roxanaghe_ has quit IRC | 17:54 | |
*** roxanaghe has joined #openstack-keystone | 17:59 | |
*** david-lyle has quit IRC | 18:00 | |
*** roxanaghe_ has joined #openstack-keystone | 18:01 | |
*** roxanaghe__ has quit IRC | 18:03 | |
*** sdake has quit IRC | 18:03 | |
*** gyee has quit IRC | 18:03 | |
*** david-lyle has joined #openstack-keystone | 18:04 | |
*** roxanaghe has quit IRC | 18:04 | |
*** sdake has joined #openstack-keystone | 18:04 | |
*** chrisshattuck has quit IRC | 18:04 | |
*** lamt has joined #openstack-keystone | 18:05 | |
*** tqtran has joined #openstack-keystone | 18:05 | |
*** tqtran_ has joined #openstack-keystone | 18:07 | |
*** ravelar has quit IRC | 18:07 | |
*** tqtran has quit IRC | 18:10 | |
*** david-lyle has quit IRC | 18:11 | |
breton | Unfortunately, your submission was not chosen to be part of the official agenda this time around. You submitted: | 18:11 |
breton | PCI via Federation and Keystone | 18:11 |
breton | :( | 18:11 |
breton | rderose: | 18:12 |
*** david-lyle has joined #openstack-keystone | 18:12 | |
*** hockeynut has quit IRC | 18:13 | |
*** david-lyle_ has joined #openstack-keystone | 18:15 | |
*** julim_ has quit IRC | 18:15 | |
*** david-lyle_ has quit IRC | 18:15 | |
*** ravelar has joined #openstack-keystone | 18:17 | |
gagehugo_ | :( | 18:17 |
jaugustine | I think that's one I actually voted on ! | 18:18 |
*** NishaYadav has joined #openstack-keystone | 18:19 | |
*** NishaYadav is now known as Guest59673 | 18:19 | |
*** Guest59673 is now known as nisha_ | 18:19 | |
*** aswadr_ has quit IRC | 18:21 | |
openstackgerrit | Dolph Mathews proposed openstack/keystone: Only use triggers conditionally https://review.openstack.org/360723 | 18:22 |
* nisha_ waves hello | 18:25 | |
*** su_zhang has quit IRC | 18:25 | |
*** su_zhang has joined #openstack-keystone | 18:26 | |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 18:26 |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Let upgrade tests control all 4 repositories at once https://review.openstack.org/360667 | 18:26 |
lbragstad | dolphm ^ | 18:26 |
dolphm | lbragstad: ack | 18:26 |
lbragstad | dolphm for example - checkout line 1656 here https://review.openstack.org/#/c/360667/2/keystone/tests/unit/test_sql_upgrade.py | 18:27 |
*** tqtran_ has quit IRC | 18:28 | |
lbragstad | there are certain spots in the tests that assume they can make assertions on those values | 18:28 |
*** tqtran has joined #openstack-keystone | 18:28 | |
lbragstad | so - to make that happen, i added two helper methods to SqlMigrateBase | 18:28 |
lbragstad | called get_max_version and get_initial_db_version that take the repository you want as an argument and return the values specific to that repository | 18:29 |
*** ruoyu has joined #openstack-keystone | 18:29 | |
dolphm | lbragstad: oh, gotcha | 18:29 |
lbragstad | dolphm i guess repo_path falls into that category too | 18:30 |
*** spedione is now known as chris_hultin | 18:30 | |
dolphm | lbragstad: it looks like the repo_name is only used to get the repo's current version? | 18:31 |
*** roxanaghe has joined #openstack-keystone | 18:31 | |
lbragstad | dolphm looks like it is used to get the repo_path? | 18:32 |
lbragstad | when a MigrationRepository object it initialized | 18:32 |
*** tqtran has quit IRC | 18:35 | |
*** esp has quit IRC | 18:35 | |
*** roxanaghe has quit IRC | 18:35 | |
dolphm | lbragstad: proposing a rev ... | 18:37 |
*** ddieterly[away] has quit IRC | 18:38 | |
lbragstad | dolphm with your patch i'm getting a couple more column assertion errors (wondering if it has something to do with how we are calling self.upgrade()) | 18:38 |
*** asettle has joined #openstack-keystone | 18:38 | |
*** asettle has joined #openstack-keystone | 18:39 | |
lbragstad | dolphm for example - http://cdn.pasteraw.com/i3ywj5pw57swpxb94i980w1hg65uxbx | 18:39 |
*** ddieterly has joined #openstack-keystone | 18:39 | |
*** tqtran has joined #openstack-keystone | 18:45 | |
lbragstad | dolphm otherwise I only have three failures with your patch - http://cdn.pasteraw.com/dz1bkjaojq1dad71meufpvjv8dt97e0 | 18:49 |
dolphm | lbragstad: that's where i'm at too | 18:50 |
openstackgerrit | Dolph Mathews proposed openstack/keystone: Let upgrade tests control all 4 repositories at once https://review.openstack.org/360667 | 18:50 |
lbragstad | awesome | 18:50 |
dolphm | lbragstad: did some other refactors ^ | 18:50 |
lbragstad | dolphm looking | 18:50 |
*** su_zhang has quit IRC | 18:51 | |
*** krotscheck has left #openstack-keystone | 18:51 | |
openstackgerrit | Matt Fischer proposed openstack/keystone: New notes on advanced upgrade/fallback for cluster https://review.openstack.org/360733 | 18:52 |
*** su_zhang has joined #openstack-keystone | 18:53 | |
*** su_zhang has quit IRC | 18:53 | |
stevemar | thanks fishy mfisch | 18:57 |
openstackgerrit | Alexander Makarov proposed openstack/keystone: Moving dependency-related trust logic to manager https://review.openstack.org/360735 | 19:00 |
*** Gorian|work has quit IRC | 19:01 | |
amakarov | stevemar, good day! Are we going to merge token pre-caching patch into Newton? | 19:03 |
lbragstad | dolphm ++ to the refactors | 19:03 |
*** Gorian|work has joined #openstack-keystone | 19:03 | |
*** esp has joined #openstack-keystone | 19:05 | |
stevemar | amakarov: you need someone else to +2 it :) | 19:05 |
*** esp has quit IRC | 19:05 | |
stevemar | dolphm: lbragstad henrynash ^ ? | 19:05 |
stevemar | amakarov: i am +2 on it right? | 19:06 |
*** esp has joined #openstack-keystone | 19:06 | |
amakarov | stevemar, yes. It has -1 from Haneef Ali but I can contact him: looks like in his opinion we don't need it :) | 19:07 |
stevemar | amakarov: lets hope dolphm has a few minutes to spare today | 19:08 |
dolphm | amakarov: performance is not really a subjective thing | 19:09 |
*** david-lyle has quit IRC | 19:10 | |
* stevemar needs food | 19:11 | |
amakarov | dolphm, hi! A bit of history: you asked for shared cache for v2 and v3 - I could do it for v3->v2 only as it's token data, that gets cached | 19:11 |
amakarov | Left a note about it in the code | 19:11 |
dolphm | amakarov: stevemar benchmarked that token "issue" time went down with this patch -- how is that possible? | 19:11 |
amakarov | stevemar, ^ | 19:12 |
dolphm | shouldn't this patch only improve validation performance? | 19:12 |
amakarov | dolphm, I don't know ) | 19:12 |
dolphm | stevemar: what does benchmark.sh do? | 19:12 |
amakarov | My result was as expected | 19:12 |
openstackgerrit | Ron De Rose proposed openstack/keystone: Add check that minimum password age is less than password expires days https://review.openstack.org/360737 | 19:13 |
amakarov | dolphm, downloads patch, checks out HEAD^ runs token issue+validate token N times, gets average timings for both, then does the same for the patch itself | 19:14 |
amakarov | dolphm, restarts apache in the process - here may be the volatility | 19:14 |
openstackgerrit | Ron De Rose proposed openstack/keystone: Adds check that minimum password age is less than password expires days https://review.openstack.org/360737 | 19:14 |
stevemar | i did run it with devstack, so .... | 19:14 |
stevemar | amakarov: dolphm: maybe issue time is now increased because we have to store the token? | 19:15 |
amakarov | stevemar, I've written it with devstack :) | 19:15 |
amakarov | stevemar, yes - it's the expected thing | 19:15 |
dolphm | stevemar: it went *down*, not up (as i would have expected, slightly) | 19:16 |
dolphm | stevemar: Average issue time .20173809523809523809 to .04333333333333333333 | 19:16 |
amakarov | stevemar, are you sure you copy-pasted the line WITH 1st symbol? ;) | 19:17 |
amakarov | otoh it will kill the idea if token issue will start to take seconds... | 19:18 |
lbragstad | dolphm I wonder if those 3 failing tests are because we are sharing metadata? | 19:18 |
lbragstad | across all the repositories? | 19:18 |
dolphm | lbragstad: oh, maybe? | 19:19 |
lbragstad | i can confirm that 109 runs completely | 19:19 |
lbragstad | but the column assertion afterwords doesn't seem to think one of those columns exists | 19:19 |
*** Ephur has joined #openstack-keystone | 19:23 | |
*** Gorian|work has quit IRC | 19:23 | |
dolphm | lbragstad: could definitely be metadata | 19:24 |
*** NishaYadav has joined #openstack-keystone | 19:24 | |
amakarov | stevemar, dolphm I've added my results if it'll help | 19:24 |
*** su_zhang has joined #openstack-keystone | 19:24 | |
*** NishaYadav is now known as Guest88057 | 19:24 | |
*** Gorian|work has joined #openstack-keystone | 19:26 | |
*** nisha_ has quit IRC | 19:27 | |
openstackgerrit | Ron De Rose proposed openstack/keystone: Adds check that minimum password age is less than password expires days https://review.openstack.org/360737 | 19:27 |
*** amakarov is now known as amakarov_away | 19:27 | |
*** su_zhang has quit IRC | 19:29 | |
*** code-R has joined #openstack-keystone | 19:31 | |
*** ruoyu has quit IRC | 19:33 | |
*** nisha_ has joined #openstack-keystone | 19:34 | |
*** edtubill has quit IRC | 19:35 | |
*** slberger has quit IRC | 19:35 | |
*** code-R has quit IRC | 19:36 | |
*** Guest88057 has quit IRC | 19:38 | |
lbragstad | dolphm wait - each MigrationRepository has it's own metadata | 19:39 |
lbragstad | they just share an engine | 19:39 |
lbragstad | so is it the engine that's the problem? | 19:40 |
*** Gorian|work has quit IRC | 19:40 | |
dolphm | lbragstad: uhh | 19:40 |
dolphm | lbragstad: i don't know but i made the tests pass | 19:40 |
lbragstad | ?! | 19:41 |
openstackgerrit | Dolph Mathews proposed openstack/keystone: Let upgrade tests control all 4 repositories at once https://review.openstack.org/360667 | 19:41 |
dolphm | lbragstad: ^ | 19:41 |
dolphm | lbragstad: just keep pushing delete until it works | 19:42 |
*** Gorian|work has joined #openstack-keystone | 19:42 | |
lbragstad | humf | 19:43 |
lbragstad | interesting | 19:43 |
dolphm | lbragstad: but yes, 1 engine and now millions of metadatas | 19:44 |
lbragstad | it looks the other way around/ | 19:44 |
lbragstad | it looks like we're using the same self.metadata everywhere | 19:45 |
lbragstad | gdi | 19:45 |
lbragstad | gerrit throwing a fit for me | 19:45 |
lbragstad | gerrit is* | 19:45 |
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting to implement some performance tuning adjustments, and should return to working order momentarily. | 19:46 | |
bknudson | notmorgan: finally figured out how to get my dev system deployed with use-threads=False ... still got an error | 19:47 |
bknudson | http://paste.openstack.org/show/563451/ | 19:48 |
bknudson | AttributeError: 'RevokeEvent' object has no attribute 'user_id' | 19:48 |
bknudson | Diffferent error this time. | 19:48 |
*** slberger has joined #openstack-keystone | 19:51 | |
notmorgan | bknudson: wtf. | 19:51 |
notmorgan | no really.. what the hell is going on here. | 19:52 |
notmorgan | this makes no sense to me. | 19:52 |
dolphm | bknudson: is that the cache bug? | 19:53 |
dolphm | lbragstad: me too - i changed the definition for self.metadata, check it out | 19:53 |
bknudson | dolphm: yes, these errors are seen when caching is on. | 19:53 |
notmorgan | fwiw, we never saw these errors with caching on until the massive cache refactor. | 19:54 |
dolphm | bknudson: then it's probably the same underlying cause as the bugs that eric brown opened? | 19:54 |
notmorgan | i'm just baffled what happened along the way that is causing this | 19:54 |
lbragstad | dolphm oh - got it! | 19:54 |
* lbragstad https://review.openstack.org/#/c/360667/4/keystone/tests/unit/test_sql_upgrade.py | 19:54 | |
dolphm | notmorgan: good to know, i've been wondering if this was a problem in mitaka | 19:54 |
dolphm | dstanek: fyi ^ | 19:54 |
bknudson | dolphm: I think they're all related. | 19:55 |
lbragstad | you get a metadata and you get a metadata! http://i2.kym-cdn.com/entries/icons/original/000/012/809/oprah-free-car.gif | 19:55 |
notmorgan | bknudson: ... stupid question, if we use pylibmc or something similar does it go away? | 19:56 |
notmorgan | bknudson: (basically anything but python-memcache) | 19:56 |
notmorgan | i'm still wondering if there is some stupid interaction happening there. | 19:57 |
bknudson | notmorgan: that was something I was thinking about trying at some point. The docs seem to say that pylibmc is better. | 19:57 |
notmorgan | dolphm: the unpack values seems to have crept in during mitaka | 19:57 |
openstackgerrit | Ron De Rose proposed openstack/keystone: Adds password regular expression checks to doctor https://review.openstack.org/360757 | 19:57 |
notmorgan | bknudson: because my debugging has led me to guessing at socket errors/re-used sockets when not all data is pulled off them in the python-memcache library | 19:58 |
notmorgan | bknudson: and that feels like a weird place to be. | 19:58 |
openstackgerrit | Dolph Mathews proposed openstack/keystone: Let upgrade tests control all 4 repositories at once https://review.openstack.org/360667 | 19:58 |
dolphm | lbragstad: updated | 19:58 |
dolphm | notmorgan: ooh, i like the pylibmc idea | 19:59 |
lbragstad | dolphm sweet - thanks | 19:59 |
lbragstad | updating my patch | 19:59 |
openstackgerrit | Ron De Rose proposed openstack/keystone: Adds password regular expression checks to doctor https://review.openstack.org/360757 | 20:00 |
*** tonytan_brb is now known as tonytan4ever | 20:02 | |
lbragstad | dolphm so i guess the lessoned learned is that sharing engines is cool - but sharing metadata isn't? | 20:02 |
openstackgerrit | Merged openstack/keystone: Use egg form of osprofiler in paste pipeline https://review.openstack.org/360337 | 20:02 |
*** su_zhang has joined #openstack-keystone | 20:03 | |
*** sdake has quit IRC | 20:03 | |
bknudson | with sqlite in-memory you need to share the engine | 20:03 |
*** sdake has joined #openstack-keystone | 20:04 | |
bknudson | ok, switching to pylibmc if I can. | 20:04 |
* notmorgan nods. | 20:05 | |
*** cheran has joined #openstack-keystone | 20:05 | |
notmorgan | lbragstad: don't share metadata if you expect the schema or anything else to change | 20:05 |
notmorgan | lbragstad: it aggressively caches | 20:05 |
lbragstad | notmorgan huh - interesting | 20:05 |
lbragstad | notmorgan that would explain what dolphm and I were seeing | 20:06 |
notmorgan | lbragstad: the engine is the connection. Share that where needed - absolutely needed for in-mem-sqlite as bknudson said | 20:06 |
cheran | @here When I tried to give access to list_user_projects to member role in policy.json, it's still picking up the default `admin_required` rule | 20:06 |
notmorgan | lbragstad: in non-in-mem (process space store) you can use separate engine instantations as needed | 20:06 |
notmorgan | but since the in-mem stuff is tied to that specific connection, and vaporizes when it's gone, need to share it | 20:07 |
*** ruoyu has joined #openstack-keystone | 20:11 | |
bknudson | oh, I forgot to change the backend from memcache_pool. Let me try that first. | 20:12 |
*** tqtran has quit IRC | 20:12 | |
*** NishaYadav has joined #openstack-keystone | 20:13 | |
*** NishaYadav is now known as Guest52712 | 20:13 | |
notmorgan | bknudson: hehe ok | 20:14 |
notmorgan | bknudson: if it all works after that, yay threads and pool | 20:14 |
*** slberger has quit IRC | 20:15 | |
notmorgan | if it is still b0rked, pylibmc would be thge next test | 20:15 |
*** lamt has quit IRC | 20:15 | |
notmorgan | if that still is an issue, it's something we're doing in keystone, dogpile, or oslo_cache | 20:15 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 20:15 |
notmorgan | but really we aren't doing much fancy stuff at this point | 20:15 |
*** nisha_ has quit IRC | 20:15 | |
ruoyu | Hello! Our team want to get realtime user information from keystone logs. Logging_context_format_string should print user id in the logs but it doesn’t work. For an example, when I log in with a wrong password, in keystone I only got log message like “2016-08-22 17:00:15.396 2707 WARNING keystone.common.wsgi [req-c9ab9248-4f94-4ed0-9005-10fe3c5e5486 - - - - -] Authorization failed. The request you have made requi | 20:16 |
ruoyu | res authentication. from 10.14.37.215”. We think the user id should be next to the request id but we only got ‘-‘. Any instruction is greatly appreciated. Thank you very much! | 20:16 |
*** Marcellin_ has quit IRC | 20:17 | |
*** roxanaghe has joined #openstack-keystone | 20:19 | |
*** lamt has joined #openstack-keystone | 20:21 | |
*** roxanaghe has quit IRC | 20:24 | |
bknudson | notmorgan: dolphm: http://paste.openstack.org/show/563462/ | 20:25 |
bknudson | maybe it's uwsgi? | 20:25 |
notmorgan | maybe | 20:25 |
bknudson | or apache-uwsgi. I was seeing errors there when I tried to use it a while back. | 20:26 |
notmorgan | huh | 20:26 |
notmorgan | can you fire up uwsgi in HTTP (non-proxy) mode on the right ports? | 20:26 |
bknudson | should be able to . | 20:26 |
notmorgan | apache-uwsgi is fairly new compared to the rest of the stack | 20:26 |
notmorgan | but still, that feels like a red herring | 20:26 |
notmorgan | i hope it's not uwsgi. | 20:27 |
bknudson | y, and I know there's bugs in older versions of apache-uwsgi. | 20:27 |
bknudson | it's talking over a pipe to uwsgi so that shouldn't be it. | 20:27 |
bknudson | shouldn't mess with talking to memcached. | 20:27 |
*** slberger has joined #openstack-keystone | 20:27 | |
*** tqtran has joined #openstack-keystone | 20:28 | |
dolphm | bknudson: i doubt eric brown was using uwsgi | 20:28 |
notmorgan | bknudson: exactly | 20:28 |
*** su_zhang has quit IRC | 20:29 | |
*** sdake has quit IRC | 20:29 | |
*** su_zhang has joined #openstack-keystone | 20:29 | |
openstackgerrit | Merged openstack/keystone: Support new osprofiler API https://review.openstack.org/341401 | 20:30 |
bknudson | for some reason we're using uwsgi packages from ubuntu and not pypi. | 20:30 |
bknudson | ii uwsgi-core 1.9.17.1-5build5 amd64 fast, self-healing application container server (core) | 20:30 |
bknudson | vs uWSGI 2.0.13.1 | 20:31 |
bknudson | old-school | 20:31 |
notmorgan | i've never used the distro uwsgi package | 20:31 |
notmorgan | always pypi | 20:31 |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 20:31 |
*** Guest52712 is now known as nisha_ | 20:31 | |
bknudson | I can't imagine why jamielennox decided to break everything and use uwsgi from packages! | 20:31 |
lbragstad | dolphm ^ the sqlite trigger there work now | 20:31 |
lbragstad | triggers* | 20:31 |
*** Ephur has quit IRC | 20:32 | |
lbragstad | according to the special update read only case | 20:32 |
*** nisha_ has quit IRC | 20:32 | |
*** asettle has quit IRC | 20:32 | |
*** sdake has joined #openstack-keystone | 20:35 | |
dolphm | lbragstad: wewt! | 20:36 |
*** asettle has joined #openstack-keystone | 20:36 | |
lbragstad | dolphm gonna run the tests with mysql and postgres in a minute | 20:36 |
*** Gorian|work has quit IRC | 20:40 | |
openstackgerrit | Ron De Rose proposed openstack/keystone: Adds password regular expression checks to doctor https://review.openstack.org/360757 | 20:41 |
*** gyee has joined #openstack-keystone | 20:42 | |
*** ChanServ sets mode: +v gyee | 20:42 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 20:45 |
*** asettle has quit IRC | 20:45 | |
stevemar | bknudson: sounds like something jamielennox would deliberately do | 20:45 |
stevemar | he's an anarchist like that | 20:45 |
*** chris_hultin is now known as spedione|AWAY | 20:46 | |
bknudson | well, maybe he's just smarter than everyone because new uwsgi didn't help. | 20:47 |
bknudson | here's what I got this time: http://paste.openstack.org/show/563464/ | 20:49 |
*** nkinder has quit IRC | 20:50 | |
ruoyu | Hello! Our team want to get realtime user information from keystone logs. Logging_context_format_string should print user id in the logs but it doesn’t work. For an example, when I log in with a wrong password, in keystone I only got log message like “2016-08-22 17:00:15.396 2707 WARNING keystone.common.wsgi [req-c9ab9248-4f94-4ed0-9005-10fe3c5e5486 - - - - -] Authorization failed. The request you have made requi | 20:52 |
ruoyu | res authentication. from 10.14.37.215”. We think the user id should be next to the request id but we only got ‘-‘. Any instruction is greatly appreciated. Thank you very much! | 20:52 |
knikolla | stevemar: ^^ ruoyu is an intern working in our monitoring team | 20:52 |
*** tqtran has quit IRC | 20:53 | |
bknudson | notmorgan: here's a different error from pylibmc: http://paste.openstack.org/show/563466/ | 20:53 |
bknudson | 2016-08-25 20:52:05.710 19409 ERROR keystone.common.wsgi Exception: get keystone.revoke.core:_list_events|None as 1921523d6734d44e88ed58dfc76ef681a36b8e9b failed: error 26 from memcached_get(1921523d6734d44e88ed58dfc76ef681): Operation now in progress | 20:53 |
notmorgan | blink | 20:55 |
notmorgan | wut | 20:55 |
notmorgan | ok i am ready to say we're doing something stupid somewhere | 20:55 |
*** tqtran has joined #openstack-keystone | 20:57 | |
openstackgerrit | Steve Martinelli proposed openstack/keystone: Reduce log level of Fernet key count message https://review.openstack.org/359941 | 20:57 |
openstackgerrit | Steve Martinelli proposed openstack/keystone: Removes old, unused code https://review.openstack.org/360561 | 20:57 |
bknudson | well, one way to do this is start digging and get more debug info. | 20:59 |
notmorgan | yeah =/ | 21:00 |
bknudson | I'll switch back to pure python. The protocol is so simple there shouldn't be any need for a C lib. | 21:00 |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 21:00 |
bknudson | https://github.com/lericson/pylibmc/issues/89 -- says they used libmemcached 1.0-15 and that helped. | 21:04 |
bknudson | still, seems like there's something going on since it's every driver has the problem. | 21:05 |
notmorgan | right | 21:05 |
*** sdake_ has joined #openstack-keystone | 21:05 | |
*** sdake has quit IRC | 21:08 | |
*** ravelar has quit IRC | 21:15 | |
*** sdake_ has quit IRC | 21:20 | |
*** sdake has joined #openstack-keystone | 21:22 | |
lbragstad | henrynash ping | 21:22 |
bknudson | this latest test I was seeing keystone clients hanging, and one of them eventually got the 500 error. Then I saw apache2 process was spinning. | 21:22 |
bknudson | this is why everyone uses nginx now | 21:22 |
*** marekd2 has joined #openstack-keystone | 21:23 | |
breton | what's being debugged? | 21:24 |
*** david-lyle has joined #openstack-keystone | 21:26 | |
*** esp has quit IRC | 21:26 | |
*** marekd2 has quit IRC | 21:27 | |
*** esp has joined #openstack-keystone | 21:28 | |
bknudson | breton: https://bugs.launchpad.net/keystone/+bug/1600394 | 21:29 |
openstack | Launchpad bug 1600394 in OpenStack Identity (keystone) "memcache raising "too many values to unpack"" [Critical,Confirmed] - Assigned to David Stanek (dstanek) | 21:29 |
*** david-lyle has quit IRC | 21:31 | |
breton | > Also, I was never able to recreate on a devstack system, so maybe it's got a fixed version of something memcache related. | 21:33 |
breton | bknudson: have you tried it on devstack with apache2 or with uwsgi? | 21:34 |
bknudson | breton: I haven't been able to track it down. The configuration I'm running with is quite a bit different than what devstack sets up. | 21:34 |
breton | (sorry for taking your attention from debugging) | 21:34 |
bknudson | devstack was with uwsgi | 21:34 |
*** pauloewerton has quit IRC | 21:35 | |
openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 21:36 |
*** esp has quit IRC | 21:36 | |
lbragstad | dolphm dstanek henrynash stevemar ^ passes locally running against sqlite, mysql, and postgres | 21:37 |
lbragstad | I also added the opportunistic tests to run FullMigration if postgres or mysql is enabled to do so | 21:38 |
dolphm | lbragstad: ++! | 21:39 |
*** ruoyu has quit IRC | 21:52 | |
jamielennox | bknudson: because uwsgi is not actually a python thing and comes with C libraries :( | 22:01 |
jamielennox | bknudson: it seemed ilke the right idea | 22:01 |
jamielennox | also because i expected it to be a good idea to be using the same apache plugin version as uwsgi runner | 22:02 |
*** BjoernT has quit IRC | 22:03 | |
*** ddieterly is now known as ddieterly[away] | 22:05 | |
*** michauds has quit IRC | 22:05 | |
*** slberger has left #openstack-keystone | 22:06 | |
*** roxanaghe has joined #openstack-keystone | 22:07 | |
*** ddieterly[away] is now known as ddieterly | 22:10 | |
*** roxanagh_ has joined #openstack-keystone | 22:12 | |
*** roxanaghe has quit IRC | 22:12 | |
*** lamt has quit IRC | 22:16 | |
*** roxanagh_ has quit IRC | 22:16 | |
*** sdake has quit IRC | 22:24 | |
*** ntpttr has quit IRC | 22:31 | |
*** esp has joined #openstack-keystone | 22:32 | |
*** ntpttr has joined #openstack-keystone | 22:36 | |
*** sdake has joined #openstack-keystone | 22:37 | |
*** tqtran has quit IRC | 22:55 | |
*** tqtran has joined #openstack-keystone | 22:55 | |
*** adriant has joined #openstack-keystone | 22:55 | |
*** ddieterly has quit IRC | 22:56 | |
*** chlong has quit IRC | 22:58 | |
*** hockeynut has joined #openstack-keystone | 22:59 | |
*** tqtran has quit IRC | 22:59 | |
*** tqtran has joined #openstack-keystone | 23:00 | |
bknudson | jamielennox: do we need apache? we've already got haproxy | 23:00 |
jamielennox | bknudson: you would still need apache for any federation modules | 23:01 |
jamielennox | bknudson: i'm also not sure you can go direct from haproxy to uwsgi, it's not a straight HTTP redirect i don't think | 23:01 |
bknudson | jamielennox: we're not supporting any federation now, and there's no tests in place anyways | 23:03 |
bknudson | uwsgi supports http | 23:03 |
*** tqtran has quit IRC | 23:04 | |
jamielennox | then yea, i guess it could be done, i don't know if there are issues about exposing uwsgi directly | 23:06 |
jamielennox | it'll make the ursula equivalent harder though | 23:06 |
jamielennox | because there is federation support there | 23:06 |
*** tqtran has joined #openstack-keystone | 23:09 | |
*** hockeynut has quit IRC | 23:10 | |
*** chlong has joined #openstack-keystone | 23:12 | |
*** tqtran_ has joined #openstack-keystone | 23:13 | |
*** tqtran has quit IRC | 23:13 | |
*** chlong has quit IRC | 23:35 | |
*** marekd2 has joined #openstack-keystone | 23:44 | |
*** esp has quit IRC | 23:46 | |
*** marekd2 has quit IRC | 23:48 | |
*** david-lyle has joined #openstack-keystone | 23:52 | |
*** rcernin has quit IRC | 23:53 | |
*** roxanaghe has joined #openstack-keystone | 23:57 | |
*** roxanaghe_ has quit IRC | 23:58 | |
*** woodster_ has quit IRC | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!