| *** itlinux has joined #openstack-glance | 00:11 | |
| *** itlinux has quit IRC | 00:11 | |
| *** Liang__ has joined #openstack-glance | 00:16 | |
| *** gyee has quit IRC | 01:08 | |
| *** brinzhang has joined #openstack-glance | 01:08 | |
| *** bhagyashris has joined #openstack-glance | 01:56 | |
| *** itlinux has joined #openstack-glance | 02:18 | |
| *** mhen has quit IRC | 02:27 | |
| *** brinzhang has quit IRC | 03:28 | |
| *** abhishekk has joined #openstack-glance | 04:07 | |
| *** itlinux has quit IRC | 04:11 | |
| *** khomesh has joined #openstack-glance | 04:16 | |
| *** khomesh has quit IRC | 04:16 | |
| *** itlinux has joined #openstack-glance | 04:43 | |
| *** udesale has joined #openstack-glance | 04:49 | |
| *** bhagyashris has quit IRC | 05:01 | |
| *** itlinux has quit IRC | 05:11 | |
| *** _alastor_ has joined #openstack-glance | 05:24 | |
| Liang__ | zuul seems abnormal these days, openstack-tox-functional-py35 always fails | 05:28 |
|---|---|---|
| *** Liang__ is now known as LiangFang | 05:29 | |
| *** ratailor has joined #openstack-glance | 05:40 | |
| *** bhagyashris has joined #openstack-glance | 05:52 | |
| *** _alastor_ has quit IRC | 06:20 | |
| *** _alastor_ has joined #openstack-glance | 06:40 | |
| *** mosulica has joined #openstack-glance | 06:54 | |
| *** _alastor_ has quit IRC | 06:59 | |
| *** rcernin has quit IRC | 07:09 | |
| *** pcaruana has joined #openstack-glance | 07:12 | |
| *** Luzi has joined #openstack-glance | 07:33 | |
| *** _alastor_ has joined #openstack-glance | 07:35 | |
| *** _alastor_ has quit IRC | 07:40 | |
| *** mhen has joined #openstack-glance | 07:53 | |
| *** w|zzy has quit IRC | 07:56 | |
| abhishekk | LiangFang, yes it is | 08:04 |
| abhishekk | I am trying to analyze but so far no luch | 08:04 |
| abhishekk | s/luch/luck | 08:04 |
| *** _alastor_ has joined #openstack-glance | 08:08 | |
| *** _alastor_ has quit IRC | 08:12 | |
| *** imacdonn has quit IRC | 08:23 | |
| *** imacdonn has joined #openstack-glance | 08:24 | |
| *** brinzhang has joined #openstack-glance | 08:45 | |
| *** brinzh has joined #openstack-glance | 08:48 | |
| *** brinzhang has quit IRC | 08:50 | |
| *** abhishekk is now known as abhishekk|lunch | 08:53 | |
| *** trident has quit IRC | 09:16 | |
| *** trident has joined #openstack-glance | 09:19 | |
| *** belmoreira has quit IRC | 09:22 | |
| *** belmoreira has joined #openstack-glance | 09:26 | |
| *** sayalilunkad has quit IRC | 09:31 | |
| *** markvoelker has joined #openstack-glance | 09:46 | |
| *** bhagyashris has quit IRC | 09:47 | |
| *** trident has quit IRC | 10:15 | |
| *** trident has joined #openstack-glance | 10:15 | |
| *** mvkr has quit IRC | 10:24 | |
| *** abhishekk|lunch has quit IRC | 10:29 | |
| *** w|zzy has joined #openstack-glance | 10:33 | |
| *** sayalilunkad has joined #openstack-glance | 10:35 | |
| *** mvkr has joined #openstack-glance | 10:56 | |
| *** LiangFang has quit IRC | 11:03 | |
| *** mvkr has quit IRC | 11:16 | |
| *** mvkr has joined #openstack-glance | 11:16 | |
| *** _alastor_ has joined #openstack-glance | 11:25 | |
| *** _alastor_ has quit IRC | 11:30 | |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:41 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 11:42 | |
| *** tobias-urdin is now known as tobias-urdin_afk | 11:43 | |
| *** markvoelker has quit IRC | 12:24 | |
| *** udesale has quit IRC | 12:25 | |
| *** udesale has joined #openstack-glance | 12:26 | |
| *** brinzh has quit IRC | 12:32 | |
| *** ratailor has quit IRC | 12:45 | |
| *** tobias-urdin_afk is now known as tobias-urdin | 12:53 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/glance master: Add irrelevant-files for integrated test jobs https://review.openstack.org/624983 | 13:01 |
| *** MattMan_ has quit IRC | 13:02 | |
| *** MattMan_ has joined #openstack-glance | 13:02 | |
| *** markvoelker has joined #openstack-glance | 13:10 | |
| *** irclogbot_1 has quit IRC | 13:30 | |
| *** zul has quit IRC | 13:32 | |
| *** zul has joined #openstack-glance | 13:40 | |
| *** irclogbot_1 has joined #openstack-glance | 13:44 | |
| *** adam_zhang has joined #openstack-glance | 13:44 | |
| *** abhishekk has joined #openstack-glance | 13:45 | |
| *** irclogbot_1 has quit IRC | 13:52 | |
| *** Liang__ has joined #openstack-glance | 13:54 | |
| Liang__ | abhishekk: ok, thanks | 13:58 |
| *** Liang__ is now known as LiangFang | 13:59 | |
| *** markvoelker has quit IRC | 14:03 | |
| *** irclogbot_1 has joined #openstack-glance | 14:08 | |
| *** irclogbot_1 has quit IRC | 14:27 | |
| *** _alastor_ has joined #openstack-glance | 14:33 | |
| *** irclogbot_1 has joined #openstack-glance | 14:33 | |
| *** markvoelker has joined #openstack-glance | 14:36 | |
| *** _alastor_ has quit IRC | 14:38 | |
| *** priteau has joined #openstack-glance | 14:42 | |
| *** abhishekk has quit IRC | 14:49 | |
| *** adam_zhang has quit IRC | 14:50 | |
| *** mosulica has quit IRC | 15:05 | |
| *** dpawlik has quit IRC | 15:06 | |
| *** itlinux has joined #openstack-glance | 15:06 | |
| *** dpawlik has joined #openstack-glance | 15:07 | |
| *** dpawlik has quit IRC | 15:07 | |
| *** markvoelker has quit IRC | 15:38 | |
| *** dpawlik has joined #openstack-glance | 15:40 | |
| *** dpawlik has quit IRC | 15:44 | |
| *** Luzi has quit IRC | 15:52 | |
| jmlowe | So, can we talk about show_multiple_locations? | 16:15 |
| *** LiangFang has quit IRC | 16:16 | |
| jmlowe | The whole "This option will be removed in the Pike release or later because the same functionality can be achieved with greater granularity by using policies." doesn't really ring true as there are explicit checks for this config option to be set True in order to update an image location, this is part of the workflow for shelving when using ceph | 16:18 |
| rosmaita | jmlowe: yes, that statement is false | 16:19 |
| jmlowe | not to put too fine of a point on it, but why is my queens glance making false statements in its logs? | 16:20 |
| rosmaita | we thought it was true at the time it was written | 16:21 |
| rosmaita | and forgot to go back and patch the deprecation | 16:21 |
| rosmaita | we still want to get rid of it, but it's going to be pretty complicated | 16:21 |
| jmlowe | if you would be so kind, it might save the next person a few hours of troubleshooting | 16:23 |
| rosmaita | ok, do me a favor and check in some time tomorrow to make sure i do it | 16:24 |
| jmlowe | ok, will do, thanks | 16:24 |
| rosmaita | cool | 16:24 |
| *** _alastor_ has joined #openstack-glance | 16:34 | |
| *** udesale has quit IRC | 16:41 | |
| *** mriedem has joined #openstack-glance | 16:42 | |
| mriedem | someone should look at this http://logs.openstack.org/41/617941/31/gate/grenade-py3/7909f30/logs/screen-g-api.txt.gz#_Dec_13_14_43_21_412172 | 16:42 |
| mriedem | there is an infinite loop in the quota code | 16:42 |
| mriedem | File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__ | 16:42 |
| *** _alastor_ has quit IRC | 16:43 | |
| mriedem | is that happening in a thread? if so, why? | 16:43 |
| *** cdent has joined #openstack-glance | 16:51 | |
| mriedem | rosmaita: ^ | 16:54 |
| mriedem | looks like this is probably contributing to gate failures | 16:54 |
| mriedem | http://status.openstack.org/elastic-recheck/#1808063 | 16:54 |
| mriedem | hmm, monotonic import was removed around the time this showed up b/c it wasn't used - does it monkey patch something on import? | 16:56 |
| jokke_ | :o | 16:57 |
| jokke_ | just trying to look into it. Seemed very odd as by the looks of it that code stuck looping has been pretty much untouched for years | 16:57 |
| mriedem | is that quota code getting hit in a thread? i tried following where that QuotaImageTagsProxy code gets used but got lost fast | 17:00 |
| cdent | same | 17:00 |
| *** abhishekk has joined #openstack-glance | 17:03 | |
| mriedem | so what sets self.image.tags here? https://github.com/openstack/glance/blob/master/glance/quota/__init__.py#L348 | 17:03 |
| mriedem | is that tags attribute itself an QuotaImageTagsProxy? | 17:03 |
| cdent | jokke_: you still looking/around? | 17:07 |
| jokke_ | mriedem: I have no idea. Trying to figure it out myself atm. That onion layer model with 382 nested proxies is pain in the dick to debug | 17:07 |
| cdent | t-shirt | 17:08 |
| mriedem | you might just need some penicilin | 17:08 |
| jokke_ | mriedem: no what i need is army of great developers and 2 cycles to refactor all that crap out of glance ;) | 17:10 |
| jokke_ | I think that ImageProxy.iamge.tags is coming from the database and is wrapped into the QuotaImageTagsProxy on row 348 | 17:13 |
| cdent | the __getattr__ is not hit by unit tests | 17:17 |
| cdent | oh wait, yes it is, sorry, I can't read | 17:17 |
| mriedem | and the ImageProxy.image is an ImageProxy which wraps a glance.domain.Image object | 17:18 |
| mriedem | which contains db sqla objects | 17:18 |
| mriedem | actually tags looks like it'd be a list of strings from image_tag_get_all in the sqlalchemy api code | 17:18 |
| mriedem | but if QuotaImageTagsProxy.tags is a list, how is getattr(self.tags, name) working? | 17:20 |
| abhishekk | jokke_, I am in to get rid of onion :D | 17:22 |
| cdent | mriedem, jokke_ : For the looping behavior that we're seeing to happen, the easy way to trigger it is for the self.tags on QuotaImageTagsProxy to be deleted. if that happens there is a loop | 17:33 |
| cdent | so one option is to inject a hasattr there and guard against that | 17:35 |
| *** zul has quit IRC | 17:35 | |
| cdent | of course why self.tags would ever go awry is unclear | 17:35 |
| cdent | I gotta get dinner | 17:36 |
| abhishekk | jokke_, rosmaita http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000911.html | 17:40 |
| *** abhishekk has quit IRC | 17:41 | |
| cdent | mriedem: re ^ dec 7th is the day the monotonic removal merged | 17:42 |
| cdent | which makes little sense, but... | 17:42 |
| cdent | two other changes merged as well | 17:43 |
| cdent | when did this bug with the bad response in the proxy start showing up? | 17:43 |
| mriedem | around dec 7 | 17:43 |
| mriedem | http://status.openstack.org/elastic-recheck/#1808063 | 17:43 |
| mriedem | the other changes were removing i18n lazy translation (not used) and adding the upgrade status cli | 17:44 |
| mriedem | neither of which should have anything to do with this | 17:45 |
| cdent | the functional tests do some thread manipulation, so perhaps both problems are caused by the same thing | 17:46 |
| cdent | i haven't got a working py35 env handy, but i'll see if functional-py37 tests fail in the ways abhishekk is describing | 17:47 |
| cdent | and then put monotonic back | 17:47 |
| * jokke_ is very confused how the monotonic or rather lack of it can cause this | 17:55 | |
| cdent | indeed, but we're grasping at straws | 17:57 |
| cdent | in 3.7 there's no difference | 17:57 |
| cdent | I can't repeat abhishekk's issue in 3.7, and don't have a 3.5 | 17:58 |
| * cdent is at a loss for the time being | 17:58 | |
| mriedem | i can try glance functional on py35 | 17:58 |
| jokke_ | we do depend on it 'though | 18:01 |
| jokke_ | glance/common/timeutils.py:from monotonic import monotonic as now # noqa | 18:01 |
| jokke_ | lower-constraints.txt:monotonic==0.6 | 18:01 |
| jokke_ | requirements.txt:monotonic>=0.6 # Apache-2.0 | 18:01 |
| cdent | jokke_: sounds like your checkout is out of date? | 18:01 |
| cdent | regardless of the cause of the problem, it is the case that the __getattr__ code in QuotaImageTagsProxy is unsafe, and needs a guard | 18:02 |
| mriedem | jokke_: yeah you need to rebase | 18:02 |
| cdent | __getattr__ always needs a guard | 18:02 |
| jokke_ | indeed ... I think I even approved that change | 18:03 |
| jokke_ | now thinking of it | 18:03 |
| *** MattMan has joined #openstack-glance | 18:03 | |
| *** MattMan_ has quit IRC | 18:03 | |
| mriedem | so glance functional py35 did fail for me | 18:05 |
| * cdent holds breath | 18:05 | |
| mriedem | figuring out wtf failed is another matter... | 18:05 |
| mriedem | {5} glance.tests.functional.v2.test_images.TestImagesWithRegistry.test_owning_tenant_can_delete_image [] ... inprogress | 18:06 |
| mriedem | anywho, let me revert monotonic and try again | 18:06 |
| *** gyee has joined #openstack-glance | 18:09 | |
| jokke_ | sure interested to see if that was the case really. If so I would really like to understand wtf breaks it | 18:09 |
| jokke_ | eventlet added external dependency for monotonic on 0.24.0 | 18:15 |
| mriedem | with the monotonic revert the functional py35 run passed | 18:17 |
| mriedem | i'll run it again | 18:17 |
| jokke_ | I'm just wondering how that patch got through if it breaks eventlet and our tests | 18:17 |
| mriedem | yeah idk the patch ran the func 35 job | 18:18 |
| mriedem | http://logs.openstack.org/64/615664/3/check/openstack-tox-functional-py35/a59717d/tox/ | 18:18 |
| mriedem | hmm, monotonic was installed anyway http://logs.openstack.org/64/615664/3/check/openstack-tox-functional-py35/a59717d/tox/functional-py35-2.log | 18:18 |
| mriedem | something transitive? | 18:19 |
| jokke_ | http://logs.openstack.org/64/615664/3/check/openstack-tox-functional-py35/a59717d/tox/functional-py35-3.log | 18:20 |
| jokke_ | yeah just looking the same | 18:20 |
| jokke_ | so educated guess, some dependency was still depending on monotonic and it got pulled in even we did not explicitely list it, then that dependency dropped the need as well just after we got rid of it | 18:21 |
| mriedem | 2nd local func py35 run failed | 18:22 |
| mriedem | so could just be flaky tests | 18:22 |
| jokke_ | did that fail with the same sig as before? | 18:22 |
| mriedem | umm | 18:22 |
| mriedem | {4} glance.tests.functional.v2.test_images.TestImages.test_owning_tenant_can_delete_image [] ... inprogress | 18:23 |
| mriedem | yes? | 18:23 |
| mriedem | http://paste.openstack.org/show/737247/ | 18:23 |
| mriedem | that's just part of the noise | 18:23 |
| mriedem | i don't know where the actual failures are | 18:23 |
| mriedem | the job doesn't dump a summary of what tests failed | 18:23 |
| cdent | mriedem: are you seeing the assertError after fork in _threading? | 18:24 |
| mriedem | definitely looks like should be using wsgi_intercept rather than eventlet | 18:24 |
| mriedem | i do see | 18:26 |
| mriedem | {1} subunit.parser [0.000232s] ... FAILED | 18:26 |
| mriedem | i don't see _threading | 18:26 |
| mriedem | but there are a ton of deprecation warnings cluttering up my console | 18:26 |
| cdent | in 3.7 i was seeing many of these: http://paste.openstack.org/show/737248/ | 18:27 |
| mriedem | anyway, this is with monotonic imported so i don't think that matters | 18:27 |
| * cdent nods | 18:27 | |
| mriedem | no i don't see those | 18:27 |
| * mriedem gets lunch | 18:27 | |
| *** _alastor_ has joined #openstack-glance | 18:31 | |
| *** _alastor_ has quit IRC | 18:39 | |
| *** mvkr has quit IRC | 18:40 | |
| *** jmlowe has quit IRC | 18:47 | |
| openstackgerrit | Erno Kuvaja proposed openstack/glance master: DO NOT MERGE https://review.openstack.org/625085 | 18:55 |
| openstackgerrit | Chris Dent proposed openstack/glance master: Guard __getattr__ on QuotaImageTagsProxy https://review.openstack.org/625086 | 19:04 |
| cdent | jokke_, mriedem : I need to do something other stuff now, but sticking that ^ in somewhere may help at least debug the problem a bit better | 19:05 |
| *** jmlowe has joined #openstack-glance | 19:12 | |
| jokke_ | cdent: thanks a million! | 19:16 |
| *** mvkr has joined #openstack-glance | 19:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/glance master: Ignore MoxStubout deprecation warnings https://review.openstack.org/625092 | 19:41 |
| mriedem | let's have some log sanity ^ | 19:41 |
| *** pcaruana has quit IRC | 19:47 | |
| jokke_ | mriedem: could you do me a favor and try to run the functional-py35 with oslo.policy!=1.43.1? | 20:02 |
| mriedem | sure | 20:04 |
| jokke_ | it was released around the same time, knowing our policies and how dodgy that piece of kode looks like and the oslo policy starting to do deepcopy on target, might be just the key here, not the monotonic removal | 20:06 |
| jokke_ | s/kode/code/ | 20:06 |
| mriedem | jokke_: i still hit this http://paste.openstack.org/show/737259/ | 20:11 |
| mriedem | oh wait a minute, | 20:11 |
| mriedem | it looks like my blacklist on oslo.policy didn't work | 20:11 |
| mriedem | weird, | 20:12 |
| mriedem | +oslo.policy>=1.30.0,!=1.43.1 # Apache-2.0 | 20:12 |
| mriedem | anyway i'll do it from within the venv | 20:12 |
| *** priteau has quit IRC | 20:15 | |
| jokke_ | yeah, my test patch didn't do it in the gate either :( | 20:16 |
| mriedem | passed | 20:16 |
| mriedem | running again | 20:16 |
| mriedem | not much in here https://github.com/openstack/oslo.policy/compare/1.43.0...1.43.1 | 20:19 |
| mriedem | unless https://github.com/openstack/oslo.policy/commit/b9fd10e2612f26c93d49c168a0408aba6d20e5bf | 20:19 |
| jokke_ | Well that target deepcopy got me worried with how that nesting works | 20:20 |
| mriedem | yeah so maybe the target is one of these proxy object? | 20:20 |
| jokke_ | exactly | 20:20 |
| mriedem | func passed again | 20:20 |
| mriedem | i think that's probably it | 20:20 |
| mriedem | cdent: for your pleasure ^ | 20:20 |
| jokke_ | mriedem: I totally owe you a beverage next time we meet | 20:21 |
| jokke_ | thanks for your help | 20:21 |
| cdent | hrmm. are we thinking that's both problems? or just the 3.5 being a dick? | 20:22 |
| mriedem | not sure about 3.5 | 20:23 |
| mriedem | or how that plays into it | 20:23 |
| mriedem | i can pull cdent's change down and run that with func py35 with oslo.policy 1.43.1 and see if that fixes it | 20:23 |
| cdent | my change isn't a fix: it's simply a "make a problem more tractable" | 20:24 |
| jokke_ | cdent: I'm gonna follow that up when we get the gate moving again to make sure we catch stuff easier | 20:25 |
| jokke_ | the problem is that I have currently no idea how else to fix this than block using 1.43.1 oslo.policy and prevent that deepcopy to be done | 20:26 |
| cdent | is the oslo.policy fix likely to have any impact on https://bugs.launchpad.net/glance/+bug/1808063 ? | 20:27 |
| openstack | Launchpad bug 1808063 in Glance "glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot - infinite loop in File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__" [High,Confirmed] | 20:27 |
| jokke_ | cdent: yes based on mriedem's couple of manual test runs it fixes that | 20:29 |
| jokke_ | or causes that to be correct | 20:30 |
| cdent | I thought we weren't seeing that infinite loop problem localy, just the functional tests failing in the way that abhishekk's described in his email message? | 20:30 |
| mriedem | yes the latter for me | 20:31 |
| cdent | oh, nevermind, I think I get it from the description mriedem has written on the bug | 20:31 |
| mriedem | i pulled cdent's change down and ran func py35 and they passed | 20:32 |
| mriedem | with oslo.policy 1.43.1 | 20:32 |
| cdent | or not | 20:32 |
| * cdent still confused | 20:32 | |
| cdent | a deepcopy on the nested object would definitely mess things up if self.tags needed to be copied | 20:33 |
| cdent | untangling that is going to be challenging | 20:35 |
| jokke_ | so if the problem abhishek described is triggered by excessive logging/stdout this would explain why we are constantly hitting it as that infinite loop seems to be hammering the outputs heavily | 20:35 |
| jokke_ | cdent: indeed ... that's what I'm mostly afraid of | 20:36 |
| jokke_ | and taken that our policy enforcement is down from the API layer there is high likelyhood that these proxy objects are the target before the policy enforcer is called | 20:37 |
| cdent | sigh. and for the integrated gate we have to not use the newer oslo.policy for the entire install. | 20:39 |
| * cdent must retire for tonight | 20:42 | |
| cdent | and the rest of the year, I'm not very present, but will try to keep tabs | 20:43 |
| jokke_ | https://review.openstack.org/625097 that should help ... if that doesn't unblock the gate we have issues likely somewhere else, if it does, we know where we need to look at for solutions | 20:44 |
| cdent | If that one object is the only place that uses __getattr__ we might be able to hack something in there to use __getattribute__ when 'name' == 'tags' | 20:46 |
| cdent | there are two other places, but they look like they may not be a problem | 20:47 |
| * cdent tries it | 20:49 | |
| *** lpetrut has joined #openstack-glance | 20:53 | |
| openstackgerrit | Chris Dent proposed openstack/glance master: Make QuotaImageTagsProxy deep-copyable https://review.openstack.org/625114 | 20:59 |
| cdent | jokke_, mriedem : i reckon that should fix it, with olso.policy 1.43.1, but I can't trigger the error for myself (my 3.5 install on this machine is busted) | 20:59 |
| cdent | also, I keep trying to step away and getting sucked back in | 21:00 |
| cdent | the above doesn't include tests because of needing to step away | 21:00 |
| *** jmlowe has quit IRC | 21:01 | |
| jokke_ | cdent: Thanks, lets see what zuul says | 21:01 |
| *** mriedem has quit IRC | 21:02 | |
| *** markvoelker has joined #openstack-glance | 21:11 | |
| *** jmlowe has joined #openstack-glance | 21:21 | |
| openstackgerrit | Chris Dent proposed openstack/glance master: Make QuotaImageTagsProxy deep-copyable https://review.openstack.org/625114 | 21:25 |
| openstackgerrit | Chris Dent proposed openstack/glance master: Make QuotaImageTagsProxy deep-copyable https://review.openstack.org/625114 | 21:25 |
| *** jmlowe has quit IRC | 21:26 | |
| cdent | fixing commit message ^ | 21:27 |
| *** tobias-urdin has quit IRC | 21:32 | |
| *** jmlowe has joined #openstack-glance | 21:50 | |
| openstackgerrit | Matt Riedemann proposed openstack/glance master: Fix DeprecationWarnings for RequestContext.tenant/user usage https://review.openstack.org/625125 | 21:55 |
| *** lpetrut has quit IRC | 22:00 | |
| *** markvoelker has quit IRC | 22:06 | |
| *** rcernin has joined #openstack-glance | 22:21 | |
| *** itlinux has quit IRC | 22:36 | |
| *** _alastor_ has joined #openstack-glance | 23:30 | |
| *** w|zzy has quit IRC | 23:41 | |
| *** w|zzy has joined #openstack-glance | 23:43 | |
| *** w|zzy has quit IRC | 23:47 | |
| *** w|zzy has joined #openstack-glance | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!