| *** felipemonteiro has quit IRC | 00:11 | |
| *** felipemonteiro has joined #openstack-keystone | 00:16 | |
| *** dave-mccowan has joined #openstack-keystone | 00:19 | |
| *** dave-mccowan has quit IRC | 00:30 | |
| *** edmondsw has joined #openstack-keystone | 00:38 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 00:42 | |
| *** edmondsw has quit IRC | 00:43 | |
| *** Dinesh_Bhor has quit IRC | 00:51 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 00:58 | |
| *** liuzz has joined #openstack-keystone | 01:17 | |
| *** gyankum has joined #openstack-keystone | 01:19 | |
| *** harlowja has quit IRC | 01:19 | |
| *** liuzz_ has quit IRC | 01:20 | |
| *** felipemonteiro has quit IRC | 01:21 | |
| *** felipemonteiro has joined #openstack-keystone | 01:24 | |
| *** felipemonteiro has quit IRC | 01:39 | |
| *** lifeless_ has joined #openstack-keystone | 01:40 | |
| *** lifeless has quit IRC | 01:40 | |
| *** felipemonteiro has joined #openstack-keystone | 01:41 | |
| *** namnh has joined #openstack-keystone | 01:48 | |
| *** markvoelker has joined #openstack-keystone | 01:54 | |
| *** markvoelker has quit IRC | 02:02 | |
| *** mvk has quit IRC | 02:06 | |
| *** markvoelker has joined #openstack-keystone | 02:08 | |
| *** annp has joined #openstack-keystone | 02:13 | |
| *** markvoelker has quit IRC | 02:14 | |
| *** markvoelker has joined #openstack-keystone | 02:15 | |
| *** mvk has joined #openstack-keystone | 02:19 | |
| *** markvoelker has quit IRC | 02:19 | |
| *** edmondsw has joined #openstack-keystone | 02:26 | |
| *** edmondsw has quit IRC | 02:31 | |
| *** felipemonteiro has quit IRC | 02:45 | |
| *** xinran__ has joined #openstack-keystone | 02:45 | |
| *** felipemonteiro has joined #openstack-keystone | 02:49 | |
| *** felipemonteiro has quit IRC | 03:00 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Fix the doc CI failure https://review.openstack.org/571369 | 03:22 |
|---|---|---|
| *** annp has quit IRC | 03:25 | |
| *** annp has joined #openstack-keystone | 03:31 | |
| *** rcernin has quit IRC | 03:35 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Fix the doc CI failure https://review.openstack.org/571369 | 03:37 |
| *** rcernin has joined #openstack-keystone | 03:54 | |
| *** mvk has quit IRC | 03:56 | |
| *** sonuk has joined #openstack-keystone | 03:58 | |
| *** edmondsw has joined #openstack-keystone | 04:14 | |
| *** edmondsw has quit IRC | 04:19 | |
| *** gyee has quit IRC | 04:21 | |
| *** rcernin has quit IRC | 04:28 | |
| *** rcernin has joined #openstack-keystone | 04:29 | |
| *** links has joined #openstack-keystone | 04:38 | |
| *** xinran__ has quit IRC | 04:53 | |
| *** Dinesh_Bhor has quit IRC | 04:59 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 05:04 | |
| *** AlexeyAbashkin has quit IRC | 05:23 | |
| *** namnh has quit IRC | 05:26 | |
| *** namnh has joined #openstack-keystone | 05:27 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 05:28 | |
| *** felipemonteiro has joined #openstack-keystone | 05:30 | |
| *** ricolin has joined #openstack-keystone | 05:33 | |
| *** AlexeyAbashkin has quit IRC | 05:35 | |
| *** jaosorior has quit IRC | 05:58 | |
| *** edmondsw has joined #openstack-keystone | 06:03 | |
| *** markvoelker has joined #openstack-keystone | 06:05 | |
| *** edmondsw has quit IRC | 06:07 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 06:10 | |
| *** jaosorior has joined #openstack-keystone | 06:14 | |
| *** felipemonteiro has quit IRC | 06:17 | |
| *** liuzz has quit IRC | 06:24 | |
| *** sonuk has quit IRC | 06:34 | |
| *** sonuk has joined #openstack-keystone | 06:34 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Fix the doc CI failure https://review.openstack.org/571369 | 06:37 |
| *** martinus__ has joined #openstack-keystone | 06:40 | |
| *** pcaruana has joined #openstack-keystone | 06:43 | |
| *** namnh has quit IRC | 07:04 | |
| *** apple01 has joined #openstack-keystone | 07:05 | |
| *** rcernin has quit IRC | 07:11 | |
| *** apple01 has quit IRC | 07:17 | |
| *** tesseract has joined #openstack-keystone | 07:22 | |
| *** apple01 has joined #openstack-keystone | 07:26 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:30 | |
| *** sonuk has quit IRC | 07:37 | |
| *** apple01 has quit IRC | 07:38 | |
| *** sonuk has joined #openstack-keystone | 07:41 | |
| *** edmondsw has joined #openstack-keystone | 07:51 | |
| *** edmondsw has quit IRC | 07:56 | |
| *** hoonetorg has quit IRC | 08:11 | |
| *** rcernin has joined #openstack-keystone | 08:12 | |
| *** rcernin has quit IRC | 08:12 | |
| *** apple01 has joined #openstack-keystone | 08:13 | |
| *** markvoelker has quit IRC | 08:15 | |
| *** hoonetorg has joined #openstack-keystone | 08:24 | |
| *** lifeless_ is now known as lifeless | 08:42 | |
| *** Tahvok has quit IRC | 09:00 | |
| *** markvoelker has joined #openstack-keystone | 09:04 | |
| *** Tahvok has joined #openstack-keystone | 09:12 | |
| *** Dinesh_Bhor has quit IRC | 09:32 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 09:36 | |
| *** Dinesh_Bhor has quit IRC | 09:36 | |
| *** apple01 has quit IRC | 09:37 | |
| *** apple01 has joined #openstack-keystone | 09:37 | |
| *** apple01 has quit IRC | 09:50 | |
| *** lifeless has quit IRC | 09:55 | |
| *** lifeless has joined #openstack-keystone | 09:56 | |
| *** apple01 has joined #openstack-keystone | 09:57 | |
| *** cz2 has joined #openstack-keystone | 10:07 | |
| *** lifeless has quit IRC | 10:08 | |
| *** lifeless_ has joined #openstack-keystone | 10:08 | |
| *** apple01 has quit IRC | 10:13 | |
| *** apple01 has joined #openstack-keystone | 10:15 | |
| *** apple01 has quit IRC | 10:16 | |
| *** nicolasbock has joined #openstack-keystone | 10:33 | |
| *** apple01 has joined #openstack-keystone | 10:57 | |
| *** annp has quit IRC | 10:57 | |
| *** gyankum has quit IRC | 11:00 | |
| *** markvoelker_ has joined #openstack-keystone | 11:04 | |
| *** markvoelker has quit IRC | 11:06 | |
| *** tesseract has quit IRC | 11:07 | |
| *** kaisers has joined #openstack-keystone | 11:07 | |
| *** tesseract has joined #openstack-keystone | 11:08 | |
| *** markvoelker has joined #openstack-keystone | 11:14 | |
| *** markvoelker_ has quit IRC | 11:15 | |
| *** edmondsw has joined #openstack-keystone | 11:27 | |
| *** edmondsw has quit IRC | 11:31 | |
| *** dave-mccowan has joined #openstack-keystone | 11:37 | |
| *** sonuk has quit IRC | 11:37 | |
| *** sonuk has joined #openstack-keystone | 11:39 | |
| *** dave-mcc_ has joined #openstack-keystone | 11:40 | |
| *** dave-mccowan has quit IRC | 11:43 | |
| *** edmondsw has joined #openstack-keystone | 11:47 | |
| *** raildo has joined #openstack-keystone | 12:00 | |
| *** belmoreira has joined #openstack-keystone | 12:07 | |
| *** brad[] has quit IRC | 12:16 | |
| *** panbalag has joined #openstack-keystone | 12:43 | |
| *** panbalag has left #openstack-keystone | 12:44 | |
| *** sonuk has quit IRC | 12:52 | |
| *** jistr is now known as jistr|mtg | 12:59 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystonemiddleware master: Switch coverage tox env to stestr https://review.openstack.org/571470 | 13:12 |
| *** Guest46098 is now known as robcresswell | 13:14 | |
| *** apple01 has quit IRC | 13:19 | |
| lbragstad | looks like https://review.openstack.org/#/c/571369/ fixes the doc build | 13:20 |
| *** rmascena has joined #openstack-keystone | 13:26 | |
| *** cmurphy[m] has left #openstack-keystone | 13:27 | |
| *** apple01 has joined #openstack-keystone | 13:28 | |
| *** links has quit IRC | 13:29 | |
| *** raildo has quit IRC | 13:29 | |
| *** markvoelker_ has joined #openstack-keystone | 13:35 | |
| *** markvoelker has quit IRC | 13:36 | |
| *** markvoelker has joined #openstack-keystone | 13:38 | |
| *** markvoelker_ has quit IRC | 13:40 | |
| *** gyankum has joined #openstack-keystone | 13:42 | |
| *** mvenesio has joined #openstack-keystone | 13:43 | |
| *** belmorei_ has joined #openstack-keystone | 13:49 | |
| *** belmoreira has quit IRC | 13:50 | |
| *** markvoelker has quit IRC | 13:52 | |
| *** brad[] has joined #openstack-keystone | 14:01 | |
| *** tnogisto has joined #openstack-keystone | 14:08 | |
| *** mvenesio has quit IRC | 14:13 | |
| tnogisto | any ideas why keystone on pike ubuntu could be really slow (response time ~10 sec)? | 14:14 |
| *** r-daneel has joined #openstack-keystone | 14:19 | |
| *** jdennis has joined #openstack-keystone | 14:20 | |
| lbragstad | tnogisto: do you have any information about how keystone is configured? | 14:25 |
| tnogisto | what you want to know? | 14:25 |
| lbragstad | do you have caching enabled? which token provider are you using? what kind of data sets are you dealing with? | 14:26 |
| tnogisto | caching yes (memcache), token fernet, poc setup | 14:27 |
| tnogisto | aka 4 compute nodes, 1 controller | 14:27 |
| lbragstad | is keystone's database replicating anywhere, or is it all pretty much local to a single datacenter? | 14:28 |
| tnogisto | single database | 14:28 |
| lbragstad | is there a particular API call that is slow? | 14:28 |
| tnogisto | no | 14:28 |
| lbragstad | is everything slow? | 14:29 |
| tnogisto | pretty much half | 14:29 |
| tnogisto | if I run "openstack endpoint list --timing" | 14:29 |
| tnogisto | total time is ~140 sec | 14:29 |
| lbragstad | do you have the output of that command? | 14:31 |
| tnogisto | yup | 14:31 |
| lbragstad | this is what i'm getting from my test box http://paste.openstack.org/show/722439/ | 14:31 |
| tnogisto | gimme a sec | 14:31 |
| tnogisto | I replied to the paste | 14:34 |
| tnogisto | unsure, perhaps I deleted your previous paste :( | 14:34 |
| tnogisto | http://paste.openstack.org/show/722440/ | 14:35 |
| lbragstad | hmm | 14:39 |
| lbragstad | intersting | 14:39 |
| *** gongysh has joined #openstack-keystone | 14:41 | |
| *** gongysh has quit IRC | 14:42 | |
| lbragstad | do other calls in keystone show similar results? like `openstack user list` or `openstack project list`? | 14:42 |
| tnogisto | project list 18s, user list 2s | 14:43 |
| lbragstad | do you have keystone running behind a load balancer? | 14:43 |
| tnogisto | jup, haproxy | 14:44 |
| lbragstad | how many keystone nodes? | 14:44 |
| tnogisto | only 1 | 14:44 |
| tnogisto | openstack-ansible puts haproxy as default | 14:44 |
| lbragstad | ahh | 14:44 |
| tnogisto | but if I look at keystone logs, then it says that it takes a long time to generate a token | 14:45 |
| lbragstad | do you have a snippet of those logs? | 14:45 |
| lbragstad | tokens contain a catalog by default | 14:46 |
| tnogisto | yup | 14:46 |
| tnogisto | generated 6584 bytes in 10043 msecs | 14:46 |
| lbragstad | so it could be something similar to what you're seeing with endpoint lists | 14:46 |
| lbragstad | can you do a `openstack token issue --timing` | 14:46 |
| tnogisto | gimme a sec | 14:47 |
| tnogisto | get 0.008, post 6.05 | 14:47 |
| tnogisto | seconds | 14:47 |
| *** prashkre has joined #openstack-keystone | 14:47 | |
| tnogisto | generated 6652 bytes in 6048 msecs is the log for the POST | 14:48 |
| lbragstad | so the token creation bit took 6 seconds? | 14:48 |
| lbragstad | http://paste.openstack.org/show/722441/ | 14:48 |
| tnogisto | jup | 14:48 |
| tnogisto | GET part of that was 0.008 seconds, POST part was 6.05 | 14:49 |
| lbragstad | got it - yeah the GET is for version discovery i think | 14:49 |
| lbragstad | how many users and projects are in the deployment? | 14:50 |
| tnogisto | 6 projects 9 users | 14:50 |
| lbragstad | can you verify there is traffic going to memcache? | 14:51 |
| tnogisto | gimme a sec | 14:51 |
| lbragstad | even without memcache, that shouldn't be enough data to slow down keystone | 14:51 |
| lbragstad | how old is this deployment? | 14:52 |
| lbragstad | has it been running for a long time? | 14:52 |
| tnogisto | ~6 months, but I changed all the internal IP-s (both in mysql and hosts) | 14:52 |
| tnogisto | yup, memcache traffic is there, and it all looks pretty correct | 14:53 |
| lbragstad | hmm | 14:53 |
| lbragstad | do you know if there revocation_event table is bloated? | 14:54 |
| lbragstad | s/there/the/ | 14:54 |
| tnogisto | 2 rows in set | 14:55 |
| tnogisto | I guess not :) | 14:55 |
| lbragstad | do you know if you've adjusted password hashing at all? | 14:56 |
| lbragstad | via keystone's configuration file? | 14:56 |
| kmalloc | lbragstad: reading backlog now | 14:57 |
| kmalloc | Sounds like something I should look at. | 14:57 |
| lbragstad | yeah - some weird performance stuff | 14:57 |
| *** belmorei_ has quit IRC | 14:57 | |
| *** jistr|mtg is now known as jistr | 14:57 | |
| *** belmoreira has joined #openstack-keystone | 14:58 | |
| tnogisto | at least not knowingly :) | 14:58 |
| lbragstad | token creation takes about 6 seconds | 14:58 |
| *** spilla has joined #openstack-keystone | 14:58 | |
| lbragstad | do you have the ability to profile keystone to see if the majority of the time is spent in python/memcache/mysql? | 14:59 |
| kmalloc | whoa, that sounds very broken | 14:59 |
| tnogisto | if you could give me nice link how to do it then probably | 15:00 |
| tnogisto | looking at tcpdump, memcache looked pretty responsive | 15:00 |
| lbragstad | https://blog.dolphm.com/performance-profiling-openstack-services-with-repoze-profile/ is an example | 15:01 |
| kmalloc | tnogisto: a ton of users? using LDAP as a user backend? | 15:01 |
| kmalloc | oh 6 users | 15:01 |
| kmalloc | wow, hm. | 15:01 |
| tnogisto | ldap as a backend, but few users | 15:01 |
| lbragstad | oh... | 15:02 |
| kmalloc | ah. ldap backend, ok, there are a lot of pitfalls with LDAP that can slow things down | 15:02 |
| lbragstad | https://www.youtube.com/watch?v=DKOJ-UToCKM might be relevant | 15:02 |
| kmalloc | lbragstad: ++ | 15:02 |
| kmalloc | tnogisto: when you're trying to get a token can you also do (in mysql): show full processlist | 15:03 |
| openstackgerrit | Colleen Murphy proposed openstack/keystonemiddleware master: Switch coverage tox env to stestr https://review.openstack.org/571470 | 15:03 |
| lbragstad | ken and chris did a bunch of work tuning keystone to be more performant with ldap | 15:03 |
| kmalloc | make sure some process isn't hanging and/or IOSTAT/SAR data for the host mysql is running on | 15:03 |
| lbragstad | yeah - connection information might be something to look at too | 15:04 |
| kmalloc | mysql has (sometimes) terrible defaults that can make anything using it very sad, if the mysql server is sad, openstack (and keystone) is likewise sad | 15:04 |
| lbragstad | both ldap connections and mysql connections | 15:04 |
| kmalloc | at first guess, 2 things come to mind: make sure memcache is running and configured for keystone. 2: make sure the queries/filters/LDAP connections (esp. if it's LDAP/AD behind a loadbalancer; don'e use ldappool config in keystone -- possible bugs, ken and chris alluded to and we're trying to look into) are not overloaded. | 15:05 |
| tnogisto | http://paste.openstack.org/show/cz3zDinOWk1ut15EM4EX/ | 15:06 |
| * kmalloc glares sternly at paste.o.o and waits for it to load | 15:06 | |
| kmalloc | ok, that looks sane | 15:06 |
| kmalloc | unless you've hit connection limits in mysql. | 15:07 |
| kmalloc | nothing there stands out as causing issues. | 15:07 |
| kmalloc | but also check the health of your galera cluster, make sure you're not having one node being very very slow (assuming galera due to the wsrep process at the top) | 15:08 |
| tnogisto | I have only one mysql server in the galera cluster | 15:08 |
| kmalloc | ah | 15:08 |
| kmalloc | then no worries there :) | 15:08 |
| tnogisto | :) | 15:08 |
| kmalloc | sometimes galera has a node go out to lunch and it just makes everything behave weird. | 15:08 |
| kmalloc | ok, so i'm not worried about mysql [like i said unless you're getting connections denied], check your LDAP server and make sure it's not overloaded. Make sure you're not using ldappool if your LDAP server is behind a loadbalancer | 15:09 |
| kmalloc | and i highly recommend looking at that youtube video it is one of the best descriptions of what to look for when tuning/optimizing your ldap filters in keystone. | 15:10 |
| tnogisto | LDAP server isnt behind a loadserver, but let me see, does it look happy | 15:10 |
| kmalloc | if you have a ton of users in LDAP [even if theyt aren't used by openstack], the query could be slow | 15:10 |
| kmalloc | or if oyu just have some odd reference chasing. | 15:10 |
| lbragstad | right - optimizing the query might help if you're users are in a specific group | 15:10 |
| lbragstad | s/you're/your/ | 15:11 |
| tnogisto | nope, the ldap server is nicely idling | 15:11 |
| tnogisto | any more ideas :) ? | 15:19 |
| lbragstad | i'd be curious to know if there is any clues with more data about the request (is time getting spent querying ldap? mysql?) | 15:20 |
| tnogisto | how can I check it? | 15:21 |
| tnogisto | tcpdump it? | 15:21 |
| lbragstad | the blog post that i linked has some tips for profiling | 15:21 |
| lbragstad | or you can telnet to the memcache instances and make sure you're seeing hits there | 15:22 |
| tnogisto | pip is not playing ball with me right now :( (im a python noob) | 15:22 |
| lbragstad | it's not missing when it gets queried, is it? | 15:22 |
| *** itlinux has joined #openstack-keystone | 15:25 | |
| *** gyee has joined #openstack-keystone | 15:26 | |
| *** pcaruana has quit IRC | 15:28 | |
| tnogisto | pip says that it cant find a version of repoze.profile :/ | 15:28 |
| lbragstad | there is also osprofiler, which is included in keystone middleware and you should be able to turn that on | 15:28 |
| *** itlinux has quit IRC | 15:31 | |
| *** itlinux has joined #openstack-keystone | 15:31 | |
| *** AlexeyAbashkin has quit IRC | 15:37 | |
| tnogisto | so, I should follow this? https://docs.openstack.org/performance-docs/latest/test_plans/keystone/plan.html | 15:42 |
| *** apple01 has quit IRC | 15:47 | |
| *** felipemonteiro has joined #openstack-keystone | 15:49 | |
| *** felipemonteiro has quit IRC | 15:53 | |
| *** mvk has joined #openstack-keystone | 15:54 | |
| *** felipemonteiro has joined #openstack-keystone | 15:54 | |
| *** felipemonteiro has quit IRC | 15:57 | |
| *** ispp has joined #openstack-keystone | 16:01 | |
| *** belmoreira has quit IRC | 16:05 | |
| *** itlinux has quit IRC | 16:07 | |
| *** markvoelker has joined #openstack-keystone | 16:11 | |
| *** ricolin has quit IRC | 16:12 | |
| *** ispp has quit IRC | 16:14 | |
| *** prashkre has quit IRC | 16:21 | |
| *** itlinux has joined #openstack-keystone | 16:24 | |
| *** markvoelker has quit IRC | 16:24 | |
| lbragstad | tnogisto: yeah - it looks like there are some profiler changes in there that might help | 16:30 |
| tnogisto | does it also work with pike? | 16:30 |
| lbragstad | but i've never profiled keystone with osprofiler | 16:30 |
| lbragstad | i would think it does, but the rally team is who integrated that support | 16:31 |
| lbragstad | i usually use repoze or plop to profile keystone | 16:31 |
| tnogisto | oki, Ill try to get my coworker to help out with that tomorrow. profiling python is really not my cup of tea | 16:35 |
| *** ayoung has joined #openstack-keystone | 16:39 | |
| lbragstad | fwiw - that's just something that i use, if you have another way to find out where that time is getting spent, by all means use it :) | 16:41 |
| *** prashkre has joined #openstack-keystone | 17:04 | |
| knikolla | o/ | 17:06 |
| *** prashkre has quit IRC | 17:09 | |
| *** prashkre has joined #openstack-keystone | 17:10 | |
| *** rmascena has quit IRC | 17:13 | |
| *** raildo has joined #openstack-keystone | 17:13 | |
| kmalloc | ++ to repoze | 17:14 |
| *** felipemonteiro has joined #openstack-keystone | 17:15 | |
| lbragstad | stevemar? https://review.openstack.org/#/c/571397/ | 17:17 |
| kmalloc | wait wut | 17:31 |
| kmalloc | stevemar is doing openstack things and is core on something? | 17:31 |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:32 | |
| *** gyankum has quit IRC | 17:35 | |
| *** felipemonteiro has quit IRC | 17:37 | |
| *** felipemonteiro has joined #openstack-keystone | 17:37 | |
| cmurphy | he still osc's a bit | 17:38 |
| cmurphy | https://review.openstack.org/#/c/536163/ | 17:39 |
| *** markvoelker has joined #openstack-keystone | 17:39 | |
| *** d0ugal_ has joined #openstack-keystone | 17:40 | |
| *** d0ugal has quit IRC | 17:42 | |
| lbragstad | nice... | 17:43 |
| *** AlexeyAbashkin has quit IRC | 17:45 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:45 | |
| *** AlexeyAbashkin has quit IRC | 17:45 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:46 | |
| *** kaisers has quit IRC | 17:49 | |
| *** d0ugal__ has joined #openstack-keystone | 17:50 | |
| *** edmondsw_ has joined #openstack-keystone | 17:51 | |
| *** markvoelker has quit IRC | 17:51 | |
| *** d0ugal_ has quit IRC | 17:53 | |
| *** itlinux has quit IRC | 17:53 | |
| *** edmondsw has quit IRC | 17:54 | |
| *** itlinux has joined #openstack-keystone | 17:57 | |
| *** itlinux has quit IRC | 17:58 | |
| *** d0ugal__ has quit IRC | 17:59 | |
| *** Alexey_Abashkin has joined #openstack-keystone | 18:04 | |
| *** itlinux has joined #openstack-keystone | 18:05 | |
| *** AlexeyAbashkin has quit IRC | 18:07 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 18:07 | |
| *** AlexeyAbashkin has quit IRC | 18:13 | |
| *** d0ugal__ has joined #openstack-keystone | 18:22 | |
| *** prashkre has quit IRC | 18:34 | |
| *** d0ugal__ has quit IRC | 18:34 | |
| *** tesseract has quit IRC | 18:34 | |
| *** lifeless_ has quit IRC | 18:39 | |
| kmalloc | lbragstad: fwiw, we spew a lot of "scope check" warnings in our tests | 18:49 |
| kmalloc | we should probably squash them until we move towards actual system scope use | 18:49 |
| *** prashkre has joined #openstack-keystone | 18:49 | |
| lbragstad | that's part of the reason why i'm working on that testing refactor | 18:50 |
| kmalloc | ahhh | 18:50 |
| lbragstad | i have a patch | 18:50 |
| kmalloc | woo, down to 33 failing tests on flaskification | 18:50 |
| kmalloc | like... everything but ec2 tokens is working | 18:50 |
| kmalloc | this feels good. | 18:50 |
| lbragstad | https://review.openstack.org/#/c/551337/3 | 18:51 |
| lbragstad | ^ that's kinda what i had in mind | 18:51 |
| lbragstad | instead of lumping all the protecting stuff into a single test case | 18:51 |
| lbragstad | i wanted to try and break things up | 18:51 |
| *** mvk has quit IRC | 18:52 | |
| lbragstad | so that 1.) we could write some utilities to make it easy to get tokens for various scopes and use them in tests | 18:52 |
| lbragstad | 2.) isolate the setup of additional resources for those tests to the test classes that actually need them | 18:52 |
| lbragstad | as opposed to creating a bunch of resources for tests that don't use them | 18:53 |
| *** prashkre has quit IRC | 18:58 | |
| *** d0ugal__ has joined #openstack-keystone | 18:59 | |
| *** edmondsw_ is now known as edmondsw | 18:59 | |
| *** dklyle has quit IRC | 19:18 | |
| *** dklyle has joined #openstack-keystone | 19:18 | |
| *** dklyle has quit IRC | 19:19 | |
| *** david-lyle has joined #openstack-keystone | 19:19 | |
| *** david-lyle has quit IRC | 19:29 | |
| *** dklyle has joined #openstack-keystone | 19:29 | |
| *** panbalag has joined #openstack-keystone | 19:31 | |
| *** lifeless has joined #openstack-keystone | 19:38 | |
| kmalloc | lbragstad: sigh, we have some just plain absurd tests. | 19:42 |
| kmalloc | like... assuming that get /v3/OS-TRUSTS with a body of a blank trust gets routed to get_trust instead of list_trusts | 19:43 |
| * kmalloc facepalms | 19:43 | |
| lbragstad | oh - weird... | 19:49 |
| kmalloc | yeah, some very strange behaviors i've found along the way | 19:52 |
| kmalloc | that arent... you know... expected | 19:52 |
| kmalloc | i'm mostly just hacking around them for the moment | 19:52 |
| kmalloc | down to 6 failing tests. | 19:52 |
| kmalloc | it's sooooo close, but getting some weird authorization errors now on some very specific tests that were "just working" | 19:52 |
| kmalloc | lbragstad: i've also had to kill some more v2.0 tests, the final vestiges will pretty much need to just die, it is really unfun to wire it up | 19:53 |
| kmalloc | it's like ... some of our tests didn't actually setup users and just used broken [read: security issues] methods to get around @protected | 19:54 |
| kmalloc | and once i funneled everything through the whole stack it's breaking. | 19:54 |
| kmalloc | lbragstad: we had a bunch of tests that did .admin_request and never grabbed a token | 19:56 |
| kmalloc | just sailed right through and worked. | 19:56 |
| kmalloc | or... were (likely) testing totally the wrong thing | 19:56 |
| kmalloc | like not hitting /v3/<whatever> but hitting /<whatever> and just rolling with the "oh thats a 404, cool!" | 19:58 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Convert Keystone to use Flask https://review.openstack.org/568377 | 19:59 |
| kmalloc | anyway, that is still going to fail tests ^ but ... a lot closer | 19:59 |
| lbragstad | nice | 20:00 |
| *** harlowja has joined #openstack-keystone | 20:01 | |
| kmalloc | lbragstad: this has been a painful process to go through | 20:03 |
| lbragstad | =/ | 20:03 |
| lbragstad | refactors are rarely fun | 20:03 |
| kmalloc | lbragstad: once this is all landed and we have some followup cleanup the cruft, we can start moving things to flask native | 20:03 |
| kmalloc | which will be SO much better | 20:03 |
| kmalloc | i don't even know how this test https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_oauth1.py#L867 ever worked | 20:05 |
| kmalloc | lbragstad: if you can tell me... i'd appreciate it | 20:05 |
| kmalloc | but, afaict, it was magic that it failed with the right error code | 20:05 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Add scope documentation for service developers https://review.openstack.org/554727 | 20:11 |
| kmalloc | yep. was pointing at the wrong path ... *facepalm* | 20:18 |
| kmalloc | lbragstad: i think i want to, for test purposes, make unrouted 404s raise a massive YOU PROGRAMMED THIS WRONG error | 20:19 |
| kmalloc | rather than an actual 404. | 20:19 |
| kmalloc | or i guess i could make it a custom 404 for the sake of testing that we then do an assertIsNot on all requests | 20:19 |
| *** AlexeyAbashkin has joined #openstack-keystone | 20:33 | |
| *** lifeless_ has joined #openstack-keystone | 20:36 | |
| *** lifeless has quit IRC | 20:36 | |
| lbragstad | i wonder if the PTI stuff messed up the doc builds | 20:37 |
| *** AlexeyAbashkin has quit IRC | 20:50 | |
| *** markvoelker has joined #openstack-keystone | 20:51 | |
| *** pcichy has quit IRC | 20:56 | |
| *** harlowja has quit IRC | 21:00 | |
| *** r-daneel has quit IRC | 21:00 | |
| *** martinus__ has quit IRC | 21:00 | |
| lbragstad | i can't seem to wrap my head around how this passed the gate twice six weeks ago but fails the gate today - https://review.openstack.org/#/c/559438/ | 21:02 |
| lbragstad | technically the documentation change in that patch shouldn't have passed since it's not valid json? | 21:02 |
| kmalloc | lbragstad: 2 more tests.... | 21:05 |
| kmalloc | lbragstad: 2 more... | 21:05 |
| lbragstad | nice | 21:05 |
| *** dklyle has quit IRC | 21:17 | |
| *** dklyle has joined #openstack-keystone | 21:18 | |
| *** markvoelker has quit IRC | 21:26 | |
| *** markvoelker has joined #openstack-keystone | 21:26 | |
| *** markvoelker has quit IRC | 21:31 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Fix warnings in documentation https://review.openstack.org/571369 | 21:31 |
| lbragstad | ^ unblocks the failing documentation jobs | 21:31 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Add scope documentation for service developers https://review.openstack.org/554727 | 21:35 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Overhaul the authentication API reference https://review.openstack.org/571309 | 21:35 |
| gagehugo | lbragstad nice | 21:41 |
| *** links has joined #openstack-keystone | 21:42 | |
| *** hoonetorg has quit IRC | 21:55 | |
| *** itlinux has quit IRC | 22:04 | |
| *** gagehugo has quit IRC | 22:05 | |
| *** hoonetorg has joined #openstack-keystone | 22:08 | |
| *** rcernin has joined #openstack-keystone | 22:13 | |
| *** links has quit IRC | 22:16 | |
| *** lbragstad has quit IRC | 22:18 | |
| *** harlowja has joined #openstack-keystone | 22:21 | |
| *** spilla has quit IRC | 22:22 | |
| *** lbragstad has joined #openstack-keystone | 22:23 | |
| *** ChanServ sets mode: +o lbragstad | 22:23 | |
| *** lifeless_ has quit IRC | 22:33 | |
| *** felipemonteiro has quit IRC | 22:33 | |
| *** lifeless has joined #openstack-keystone | 22:34 | |
| *** ricolin has joined #openstack-keystone | 22:39 | |
| *** d0ugal__ has quit IRC | 22:48 | |
| *** gagehugo has joined #openstack-keystone | 22:48 | |
| *** felipemonteiro has joined #openstack-keystone | 23:00 | |
| *** d0ugal__ has joined #openstack-keystone | 23:02 | |
| openstackgerrit | Merged openstack/keystone master: Fix warnings in documentation https://review.openstack.org/571369 | 23:11 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!