Tuesday, 2020-08-11

*** diurnalist has quit IRC00:06
*** diurnalist has joined #openstack-keystone00:31
*** diurnalist has quit IRC00:51
*** diurnalist has joined #openstack-keystone00:53
*** diurnalist has quit IRC00:59
*** hemna has quit IRC01:01
*** hemna has joined #openstack-keystone01:01
*** diurnalist has joined #openstack-keystone01:11
*** hemna has quit IRC01:17
*** hemna has joined #openstack-keystone01:18
*** diurnalist has quit IRC01:47
*** diurnalist has joined #openstack-keystone02:10
*** gyee has quit IRC02:11
*** diurnalist has quit IRC02:14
*** rcernin has quit IRC02:37
*** rcernin has joined #openstack-keystone02:48
*** tobberydberg_ has quit IRC03:04
*** bnemec has quit IRC03:04
*** irclogbot_2 has quit IRC03:05
*** hemna has quit IRC03:05
*** bnemec has joined #openstack-keystone03:06
*** hemna has joined #openstack-keystone03:06
*** irclogbot_2 has joined #openstack-keystone03:07
*** tobberydberg has joined #openstack-keystone03:10
*** ayoung has quit IRC03:18
*** ayoung has joined #openstack-keystone03:22
*** diurnalist has joined #openstack-keystone03:23
*** ayoung has quit IRC03:29
*** ayoung has joined #openstack-keystone03:32
*** rcernin has quit IRC03:54
*** diurnalist has quit IRC04:02
*** ayoung has quit IRC04:03
*** ayoung has joined #openstack-keystone04:04
*** diurnalist has joined #openstack-keystone04:13
*** rcernin has joined #openstack-keystone04:21
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-keystone04:33
*** abdysn has joined #openstack-keystone05:21
*** vishakha has joined #openstack-keystone05:53
*** diurnalist has quit IRC06:14
*** Abdallahyas has joined #openstack-keystone06:56
*** abdysn has quit IRC06:59
*** bengates has joined #openstack-keystone07:02
*** Abdallahyas has quit IRC07:54
*** abdysn has joined #openstack-keystone07:55
*** rcernin has quit IRC09:11
*** rcernin has joined #openstack-keystone09:34
*** tkajinam has quit IRC09:36
*** markvoelker has joined #openstack-keystone09:45
*** markvoelker has quit IRC09:49
*** rcernin has quit IRC09:56
*** vishakha has quit IRC10:50
*** markvoelker has joined #openstack-keystone10:52
*** markvoelker has quit IRC10:56
*** rcernin has joined #openstack-keystone11:31
*** markvoelker has joined #openstack-keystone11:35
*** markvoelker has quit IRC11:40
mnaserhrm - is there anyone who has played with using application credentials for openstack services (keystonemiddleware?)11:55
*** raildo has joined #openstack-keystone11:58
knikollamnaser: i haven't, but it should be possible since keystonemiddleware supports using keystoneauth plugins by setting auth_type12:31
mnaserknikolla: ok, we're giving it a try right now and we'll see how that all pans out12:32
fricklermnaser: not yet, but it is on my todo list, so I'd be interested to learn something about it. my main motivation would be credential rotation, which is essentially impossible currently12:32
knikollamnaser: awesome. let me know if there's something that's not working as it should.12:32
mnaserknikolla: i mean, right now there is something not working, we tried the 'provide username and application name/secret' and that did not work, resulting in service giving 403s12:33
mnaserknikolla: we're trying again by explicitly providing id and secret instead12:33
mnaserknikolla: i.e. this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85e/745551/19/check/openstack-operator:functional/85ea9cb/controller/logs/etc/heat/heat_conf.txt12:34
mnaserfrickler: yeah -- that's one of our big goal, right now stage 1 is doing long lived credentails and stage 2 is rotating them automatically :)12:34
mnaserfyi if you're looking at those logs you'll find services running in k8s so -https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85e/745551/19/check/openstack-operator:functional/85ea9cb/docker/12:34
mnaserhmm, interesting.. "Could not find Application Credential: heat.: keystone.exception.ApplicationCredentialNotFound: Could not find Application Credential: heat"12:35
mnaserwe're probably going to use id based things just because its going to be far more performant and avoid extra lookups12:36
knikollaso many pods!12:39
*** Luzi has joined #openstack-keystone12:55
*** Luzi has quit IRC13:00
*** vishalmanchanda has joined #openstack-keystone13:04
*** dave-mccowan has joined #openstack-keystone13:11
knikollagagehugo, cmurphy, lbragstad: i won't be able to host/attend the weekly meeting as i have a medical appt. if either of you wants to run it please go ahead, otherwise we can cancel it for this week. sorry!13:17
*** dave-mccowan has quit IRC13:29
*** abdysn has quit IRC14:03
*** rcernin has quit IRC14:08
*** rcernin has joined #openstack-keystone14:19
*** rcernin has quit IRC14:24
*** rcernin has joined #openstack-keystone14:27
*** rcernin has quit IRC14:31
*** flaviof has joined #openstack-keystone14:35
*** diurnalist has joined #openstack-keystone14:40
flaviofhi frickler et all... grenade jobs in neutron (and nova) in Zuul master seem to be broken. The smoking gun on that failure seems to be related to this: https://zuul.opendev.org/t/openstack/build/5736faa004194fc6af092c63d96e76e3/log/compute1/logs/screen-n-cpu.txt#7724-7727   Any chance the folks here have an understanding on recent changes (after Aug/8/20) that could be related to that?14:43
flaviofERROR nova.compute.manager [instance: 07888380-65b9-41cd-8e8d-6f55624e9204] neutronclient.common.exceptions.ServiceUnavailable: The server is currently unavailable. Please try again at a later time.<br /><br />14:44
*** vishakha has joined #openstack-keystone14:55
*** brtknr has left #openstack-keystone15:08
cmurphyknikolla: i could but if there aren't any topics maybe just cancel it?15:19
knikollacmurphy: ++ that works. sending an email out.15:20
cmurphyflaviof: looks like the keystone error is https://zuul.opendev.org/t/openstack/build/5736faa004194fc6af092c63d96e76e3/log/controller/logs/screen-keystone.txt#37999 I think I've seen that before but not consistently, is it persistent? there are no changes in keystone since august 515:20
vishakhacmurphy knikolla  Could you please look #link https://review.opendev.org/#/c/744940/15:23
flaviofcmurphy: yeas, seems very persistent now: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-grenade-multinode&project=openstack/neutron  +  https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova15:24
gagehugoknikolla: wfm, I will be double booked today15:24
vishakhacmurphy knikolla  gagehugo Its was blocking the requirements repo to update pymysql latest version15:25
knikollavishakha: thanks!15:26
flaviofcmurphy: just another ref point: https://review.opendev.org/#/c/741303/  if you look at the builds or that gerrit change, the grenade jobs began failing after Aug/8, it seems.15:26
vishakhaknikolla gagehugo Also adding the skipped test cases again #link https://review.opendev.org/#/c/745376/15:27
flaviofcmurphy: sorry for the newbie question, but is there any chance that this modification: https://review.opendev.org/#/c/744753/1/lib/keystone  could cause the error in keystone?15:36
cmurphyflaviof: that seems unlikely to me15:38
*** rcernin has joined #openstack-keystone15:40
*** rcernin has quit IRC15:44
fricklercmurphy: flaviof: the keystone error is in msgpack, which was updated here https://review.opendev.org/#/c/745437/2/upper-constraints.txt , I'd bet on that to be the culprit15:51
fricklerprometheanfire: ^^15:51
cmurphyfrickler: ++15:51
prometheanfireya, that just updated a few days ago15:52
*** bengates has quit IRC16:24
*** bengates has joined #openstack-keystone16:25
*** bengates has quit IRC16:30
openstackgerritMerged openstack/keystone master: Skip tests to update u-c for PyMySql to 0.10.0  https://review.opendev.org/74494016:38
*** zzzeek has quit IRC16:47
openstackgerritVishakha Agarwal proposed openstack/keystone master: Add missing test case in test_backends.py for assignments  https://review.opendev.org/74571017:02
openstackgerritVishakha Agarwal proposed openstack/keystone master: Fix PyMySql on latest version 0.10.0  https://review.opendev.org/74537617:13
lbragstadknikolla cmurphy gagehugo the only request i had was reviews on https://review.opendev.org/#/c/744251/17:29
flaviofprometheanfire++ frickler++ please advice on what we should do to remedy this situation.17:43
fricklerflaviof: prometheanfire: I'd propose for someone to propose a revert of the msgpack bump for now, then someone can look into making the update backwards compatible. from the release notes on pypi I fear that that may become difficult17:46
prometheanfireI'm unsure about the nature of the problem, is it something keystone needs to adapt to for msgpack-1.0.0 or a bug in msgpack (that's upstream reported/ack'd preferably)?17:46
prometheanfireso, normal major release bump issue?17:46
fricklerprometheanfire: from https://pypi.org/project/msgpack/ I get that 1.0 is intentionally breaking things, yes17:47
fricklerfrom the backtrace it looks like dogpile.cache will need to handle that somehow, not sure if there are also other consumers17:48
fricklerI'm also not sure why only the grenade-multinode job seems to see this, does that test with only one of the nodes updated?17:49
* prometheanfire shrugs, that I do not know17:52
*** diurnalist has quit IRC18:15
melwitthas anyone noticed this error in the gate? "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte" https://zuul.opendev.org/t/openstack/build/316ecf4d0b69480e95f3c71ceed60690/log/logs/screen-keystone.txt#3062318:57
melwittoh guh sorry, I see that it's already being discussed18:57
fricklermelwitt: yeah, but not much progress yet. it's eod for me, so if you could create a bug and possible a reqs patch to cap msgpack<1.0 that might be helpful19:22
*** vishakha has quit IRC19:44
*** gyee has joined #openstack-keystone19:59
*** openstackgerrit has quit IRC20:52
-openstackstatus- NOTICE: The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server20:52
*** vishalmanchanda has quit IRC21:14
*** raildo has quit IRC21:15
*** zzzeek has joined #openstack-keystone21:38
*** ayoung has quit IRC21:48
*** ayoung has joined #openstack-keystone21:49
*** gregwork has joined #openstack-keystone22:45
*** rcernin has joined #openstack-keystone22:45
*** tkajinam has joined #openstack-keystone22:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!