| *** ozzzo has quit IRC | 00:06 | |
| *** toabctl has quit IRC | 00:19 | |
| *** toabctl has joined #openstack-oslo | 00:22 | |
| *** tkajinam has quit IRC | 01:49 | |
| *** tkajinam has joined #openstack-oslo | 01:50 | |
| *** larainema has joined #openstack-oslo | 03:44 | |
| *** threestrands has quit IRC | 06:48 | |
| *** jrosser has joined #openstack-oslo | 07:30 | |
| *** tesseract has joined #openstack-oslo | 07:52 | |
| *** iurygregory has joined #openstack-oslo | 07:56 | |
| *** harlowja has quit IRC | 08:15 | |
| *** tkajinam has quit IRC | 08:16 | |
| *** ralonsoh has joined #openstack-oslo | 08:22 | |
| *** rpittau|afk is now known as rpittau | 08:46 | |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 09:02 |
|---|---|---|
| *** gus has quit IRC | 09:08 | |
| *** irclogbot_2 has quit IRC | 09:08 | |
| *** StevenK has quit IRC | 09:08 | |
| mgoddard | hberaud: will do, thanks! | 09:19 |
| hberaud | mgoddard: I updated the patch since my previous msg ^^^^ | 09:20 |
| mgoddard | hberaud: sure, I'll grab the latest | 09:20 |
| hberaud | mgoddard: thanks | 09:21 |
| openstackgerrit | Luigi Toscano proposed openstack/cookiecutter master: Fix links formatting for the contributing documentation https://review.opendev.org/711407 | 09:22 |
| *** tosky has joined #openstack-oslo | 09:25 | |
| *** gus has joined #openstack-oslo | 09:43 | |
| *** irclogbot_2 has joined #openstack-oslo | 09:43 | |
| *** StevenK has joined #openstack-oslo | 09:43 | |
| *** openstackstatus has quit IRC | 09:45 | |
| mgoddard | hberaud: I did some searching, and the only place I can see server_max_value_length used is https://opendev.org/openstack/oslo.cache/src/branch/master/oslo_cache/backends/memcache_pool.py#L38 | 09:50 |
| mgoddard | maybe this problem is easier to solve? | 09:51 |
| hberaud | mgoddard: yep I seen it | 09:51 |
| hberaud | mgoddard: I seen the Radoslav message | 09:51 |
| mgoddard | oh. I didn't | 09:51 |
| hberaud | moguimar: indeed it could be easier to solve | 09:52 |
| mgoddard | hberaud: there is also https://opendev.org/openstack/keystonemiddleware/src/branch/master/keystonemiddleware/auth_token/_cache.py#L108, but I don't think that is our problem as everywhere I can see use_advanced_pool is False so it shouldn't use oslo_cache | 09:54 |
| hberaud | mgoddard: ack | 09:54 |
| mgoddard | hberaud: I'll try modifying those params and running a test job | 09:55 |
| hberaud | ok | 09:55 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 10:03 |
| hberaud | mgoddard: ^^^^ | 10:03 |
| hberaud | mgoddard: with params adapted to our needs | 10:04 |
| hberaud | mgoddard: also I prefered to keep the matrix and deprecations in place, just in case, to surround possible similar issues but it's unlikely | 10:07 |
| mgoddard | hberaud: it's from an old iteration, but in case it helps: https://2e2758d1408492f410e7-6319048316c626def25c176077b7ea65.ssl.cf1.rackcdn.com/711227/4/check/kolla-ansible-centos8-source/4448393/primary/logs/kolla/keystone/keystone.txt | 10:09 |
| hberaud | mgoddard: thanks | 10:10 |
| hberaud | mgoddard: I also fixed the connection_timeout issue in the latest patch set | 10:11 |
| mgoddard | hberaud: ok | 10:11 |
| hberaud | I misspelled it | 10:11 |
| mgoddard | hberaud: is it possible to try creating a client using _create_connection in the unit tests? | 10:12 |
| mgoddard | that way we could be sure creation works | 10:12 |
| mgoddard | hberaud: maybe it could be checked using mock with autospec? | 10:14 |
| mgoddard | if the actual object is too heay | 10:14 |
| mgoddard | heavy | 10:14 |
| hberaud | mgoddard: currently _create_connection is mocked | 10:15 |
| mgoddard | hberaud: is it? I don't see it | 10:16 |
| mgoddard | not in the new tests | 10:16 |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 10:17 |
| hberaud | mgoddard: I think we should add some jobs to our zuul config to test something more than just the base tempest job, like smcginnis suggested | 10:17 |
| hberaud | mgoddard: IIRC it don't use autospec for now | 10:18 |
| yoctozepto | hberaud: yeah, I've just proposed ^ | 10:18 |
| hberaud | yoctozepto: yep I seen thanks :) | 10:18 |
| yoctozepto | pretty nice sync we have :-) | 10:19 |
| hberaud | yoctozepto: yep | 10:19 |
| hberaud | yoctozepto: I asked myself if we need to rebase your changes on mine | 10:20 |
| hberaud | yoctozepto: I suppose your check will fail | 10:20 |
| yoctozepto | hberaud: or vice versa, let's see the fail first | 10:20 |
| hberaud | yoctozepto: yep no rush | 10:20 |
| yoctozepto | hberaud: if it somehow passes the original, then it's not useful :D | 10:20 |
| hberaud | yoctozepto: exact | 10:21 |
| yoctozepto | cross-commented and done for now | 10:22 |
| yoctozepto | let's await results | 10:22 |
| hberaud | yoctozepto: +1 | 10:23 |
| hberaud | yoctozepto: as I explained to mgoddard I think it's worth to keep deprecation and compat matrix in a first time to surround all aspects of this | 10:24 |
| mgoddard | hberaud: mind if I push an update to your patch? | 10:27 |
| yoctozepto | hberaud: totally | 10:27 |
| hberaud | moguimar: feel free | 10:29 |
| openstackgerrit | Mark Goddard proposed openstack/oslo.cache master: fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 10:32 |
| yoctozepto | mgoddard, hberaud: since we know 2.1.0 is broken, could we already blacklist it? | 10:37 |
| hberaud | yoctozepto: not yet from my side | 10:38 |
| hberaud | yoctozepto: not sure how to blacklist things... I suppose through openstack/requirements by pushing a patch there | 10:39 |
| hberaud | yoctozepto: an idea? | 10:39 |
| yoctozepto | hberaud: yes, it would go there with != in reqs and change in uc | 10:40 |
| hberaud | yoctozepto: ack, omw | 10:40 |
| yoctozepto | hberaud: thanks | 10:40 |
| yoctozepto | mgoddard, hberaud: tempest failed as expected, on dead_retry | 10:42 |
| yoctozepto | (/me following the console) | 10:42 |
| hberaud | yoctozepto: https://review.opendev.org/#/c/711427/ | 10:44 |
| openstackgerrit | Jonathan Rosser proposed openstack/oslo.cache master: Fix pymemcache release note python package name https://review.opendev.org/711428 | 10:44 |
| hberaud | yoctozepto: ack good point, so I suppose next step is to rebase on mine? | 10:45 |
| yoctozepto | hberaud: you need to also modify upper constraints | 10:46 |
| hberaud | yoctozepto: ack | 10:46 |
| yoctozepto | hberaud: yeah, I will rebase | 10:46 |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 10:47 |
| hberaud | yoctozepto: upper-constraints updated => https://review.opendev.org/711427 | 10:48 |
| yoctozepto | hberaud: k, thanks | 10:49 |
| openstackgerrit | Mark Goddard proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 10:52 |
| openstackgerrit | Mark Goddard proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 10:52 |
| * hberaud need to address kids stuff bbl | 10:55 | |
| mgoddard | hberaud: another issue, after dropping args in PooledMemcachedBackend: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cc5/711417/1/check/kolla-ansible-centos8-source/cc5b551/primary/logs/kolla/keystone/keystone.txt | 10:57 |
| hberaud | mgoddard: wtf lol | 11:04 |
| * hberaud take a look | 11:04 | |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 11:10 |
| hberaud | mgoddard: ^^^^ | 11:10 |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 11:18 |
| *** rpittau is now known as rpittau|bbl | 11:27 | |
| *** tesseract-RH has joined #openstack-oslo | 11:40 | |
| yoctozepto | mgoddard, hberaud: now: 2020-03-05 11:41:54.236810 | controller | ERROR keystone AttributeError: 'HashClient' object has no attribute 'server' | 11:42 |
| yoctozepto | 😂 | 11:43 |
| *** tesseract has quit IRC | 11:43 | |
| * hberaud back | 11:50 | |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Revert "Switch from python-memcached to pymemcache." https://review.opendev.org/711439 | 11:50 |
| hberaud | sigh... loooool | 11:50 |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 11:51 |
| yoctozepto | mgoddard, hberaud: for now reverted just to test | 11:51 |
| hberaud | yoctozepto: ack | 11:51 |
| hberaud | yoctozepto: ah... I mingled pymemcache base client and hash client code in my previous patch set... this is why we get the "server" error | 11:57 |
| yoctozepto | hberaud: ah, that makes sense; still letting now test the revert | 11:58 |
| yoctozepto | (sanity check above all) | 11:58 |
| hberaud | yoctozepto: and hash client doesn't store servers as a class attribute so this is why we get the first error related to server too | 11:59 |
| hberaud | yoctozepto: sure | 11:59 |
| *** kgiusti has left #openstack-oslo | 12:03 | |
| moguimar | hberaud o/ | 12:05 |
| moguimar | saw a lot of discussion in the patch, havent read it again yet. going for it now | 12:05 |
| hberaud | yoctozepton, mgoddard: hum... I think that migrate to the pymemcache pool module is the best approach to solve the server issue => https://pymemcache.readthedocs.io/en/latest/apidoc/pymemcache.pool.html | 12:06 |
| hberaud | moguimar: o/ | 12:06 |
| hberaud | moguimar: sure np, feedbacks are welcome | 12:06 |
| *** raildo has joined #openstack-oslo | 12:08 | |
| hberaud | moguimar: to summarize the situation, I proposed a patch to blacklist oslo.cache 2.1.0 for now, yoctozepto proposed a patch to introduce backend jobs related to our backend, I merged my 2 previous patches into a single one to bring a all in one fix | 12:09 |
| hberaud | the fix (you already know this patch) => https://review.opendev.org/711220 | 12:09 |
| hberaud | the blacklist of 2.1.0 => https://review.opendev.org/#/c/711427/ | 12:10 |
| hberaud | the new backend job => https://review.opendev.org/#/c/711422/ | 12:10 |
| hberaud | moguimar: and also yoctozepto pushed a revert to test with the backend job https://review.opendev.org/711439 to compare with the fix | 12:12 |
| moguimar | and this is due to the change from python-memcached to pymemecache right? | 12:13 |
| yoctozepto | moguimar: right | 12:15 |
| hberaud | moguimar: yep this is the culprit | 12:17 |
| hberaud | moguimar, yoctozepto: python-memcached stored servers as a list so it's ordered, pymemcache it as a dict, on oslo we store our deaduntil as a list generated from url (https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L191) but dict are ordered since py3.6 so I think we are safe here, do you see something that I may missed? | 12:27 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 12:28 |
| hberaud | moguimar, yoctozepto: this one highlight my previous message ^^^^ | 12:28 |
| *** ricolin has joined #openstack-oslo | 12:29 | |
| hberaud | s/pymemcache it as/pymemcache store it as/ | 12:33 |
| yoctozepto | hberaud: ack | 12:34 |
| *** ansmith_ has quit IRC | 12:40 | |
| yoctozepto | moguimar, mgoddard, hberaud: revert started passing tempest so the job is helpful for testing | 12:45 |
| yoctozepto | console; https://zuul.opendev.org/t/openstack/stream/ddc77c1a9c074c67bbc447fe3cf3ea9b?logfile=console.log | 12:45 |
| hberaud | yoctozepto: ack | 12:45 |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 12:46 |
| yoctozepto | hberaud: now based on yours | 12:46 |
| hberaud | yoctozepto: ack | 12:47 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: [DNM][WIP] fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 13:01 |
| *** dave-mccowan has joined #openstack-oslo | 13:10 | |
| smcginnis | What is the latest thoughts on memcache? Are we reverting the change and going back to how it was, or does it still look like things can be adapted so consumers can have the same interface even though oslo.cache is using a something else underneath? | 13:16 |
| *** ataraday has joined #openstack-oslo | 13:16 | |
| ataraday | Hello everyone! Please review https://review.opendev.org/#/c/710764/ | 13:17 |
| *** takamatsu has joined #openstack-oslo | 13:22 | |
| jrosser | is it necessary to explicitly state pymemcache as a required python package in deployment projects - i.e it doesn't get automatically pulled in as a dependancy of oslo.cache? | 13:24 |
| smcginnis | jrosser: It should get pulled in, but if you are doing something like "pip install oslo.cache" it won't get it. You would need to do "pip install oslo.cache[dogpile]" | 13:27 |
| smcginnis | So if you are installing a different package that pulls in oslo.cache without any optional extras, it won't actually get it. | 13:27 |
| jrosser | right - that hurts :( | 13:28 |
| *** ansmith_ has joined #openstack-oslo | 13:36 | |
| *** ansmith has joined #openstack-oslo | 13:38 | |
| *** ansmith_ has quit IRC | 13:40 | |
| *** rpittau|bbl is now known as rpittau | 13:45 | |
| *** kgiusti has joined #openstack-oslo | 13:47 | |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: [DNM][WIP] fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 13:49 |
| hberaud | smcginnis: we have 2 sure options for now, black list 2.1.0 and try to fix the current code to keep pymemcache alive (I personnaly prefer this one), revert to python-memcached | 14:05 |
| hberaud | smcginnis: I think I'm near from the solution, I need to address dead clients remove | 14:06 |
| hberaud | smcginnis: I've some track to follow to do that properly | 14:06 |
| smcginnis | OK, sounds good. Thanks hberaud. | 14:07 |
| smcginnis | I would prefer getting pymemncache to work over reverting it too. | 14:07 |
| hberaud | smcginnis: I think while 2.1.0 is blacklisted we could wait few days before revert pymemcache to lets a chance to fix that | 14:07 |
| smcginnis | Yeah, I don't think we need to rush this. | 14:07 |
| hberaud | smcginnis: another option could be to revert and keep pymemcache on a separated branch while it doesn't work at 100 percents... | 14:08 |
| smcginnis | I'd say, let's wait a bit and see if it turns out it is going to take a lot more effort and time to get things working right. We have some time. | 14:09 |
| hberaud | smcginnis: one thing we need to take care is to avoid parallel merges on this scope to avoid spaghetti effects | 14:09 |
| hberaud | I don't think we need to continue to manage deaduntil things because pymemcache introduced the `_safely_run_func` which is called each time a memcache command is called on client and so this function will automatically drop the client if is dead https://github.com/pinterest/pymemcache/blob/86fc415d8383c7dbf7bd3442ab9d7d3e36a874d3/pymemcache/client/hash.py#L150 | 14:19 |
| *** ccamel has joined #openstack-oslo | 14:41 | |
| *** camelCaser has quit IRC | 14:42 | |
| *** jaosorior has quit IRC | 15:20 | |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: [DNM][WIP] fix _memcache_pool's backward compatibility https://review.opendev.org/711220 | 15:31 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.cache master: Fix python2 syntaxe in doc https://review.opendev.org/711505 | 16:38 |
| *** rcernin has quit IRC | 16:55 | |
| *** tesseract-RH has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack/oslo.cache master: Fix python2 syntaxe in doc https://review.opendev.org/711505 | 17:11 |
| *** kgiusti has left #openstack-oslo | 17:20 | |
| *** rpittau is now known as rpittau|afk | 17:25 | |
| *** kgiusti has joined #openstack-oslo | 17:27 | |
| openstackgerrit | Radosław Piliszek proposed openstack/oslo.cache master: Add memcache_pool backend job https://review.opendev.org/711422 | 17:48 |
| *** ralonsoh has quit IRC | 18:57 | |
| *** larainema has quit IRC | 19:03 | |
| *** harlowja has joined #openstack-oslo | 19:04 | |
| *** jaosorior has joined #openstack-oslo | 19:08 | |
| *** trident has quit IRC | 20:37 | |
| *** trident has joined #openstack-oslo | 20:47 | |
| *** rcernin has joined #openstack-oslo | 20:53 | |
| *** ansmith has quit IRC | 21:02 | |
| *** rcernin has quit IRC | 21:18 | |
| *** kgiusti has left #openstack-oslo | 21:35 | |
| *** raildo has quit IRC | 21:47 | |
| *** ansmith has joined #openstack-oslo | 22:10 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.log master: Add Victoria and Wallaby releases to versionutils https://review.opendev.org/711561 | 22:12 |
| *** iurygregory has quit IRC | 22:18 | |
| *** tkajinam has joined #openstack-oslo | 22:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!