| *** takedakn has joined #openstack-glance | 00:12 | |
| *** sabari is now known as zz_sabari | 00:33 | |
| *** takedakn has quit IRC | 00:45 | |
| *** buzztroll has joined #openstack-glance | 01:24 | |
| *** buzztroll has quit IRC | 01:29 | |
| *** julim has joined #openstack-glance | 01:35 | |
| *** julim has quit IRC | 01:40 | |
| *** zz_sabari is now known as sabari | 01:44 | |
| *** spzala has joined #openstack-glance | 02:00 | |
| *** HenryG has joined #openstack-glance | 02:44 | |
| *** TravT has joined #openstack-glance | 02:54 | |
| *** TravT has quit IRC | 02:58 | |
| *** spzala has quit IRC | 03:15 | |
| *** buzztroll has joined #openstack-glance | 03:23 | |
| *** buzztroll has quit IRC | 03:30 | |
| *** slDabbler has joined #openstack-glance | 04:08 | |
| *** TravT has joined #openstack-glance | 04:10 | |
| *** TravT has quit IRC | 04:14 | |
| *** Longgeek has joined #openstack-glance | 04:15 | |
| *** sabari is now known as zz_sabari | 04:38 | |
| *** zz_sabari is now known as sabari | 05:12 | |
| *** TravT has joined #openstack-glance | 05:26 | |
| *** sabari is now known as zz_sabari | 05:26 | |
| *** TravT has quit IRC | 05:30 | |
| *** pcaruana has quit IRC | 05:32 | |
| *** zz_sabari is now known as sabari | 05:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/glance: Imported Translations from Transifex https://review.openstack.org/146765 | 06:04 |
|---|---|---|
| *** alex_xu has quit IRC | 06:21 | |
| *** bkopilov has joined #openstack-glance | 06:23 | |
| *** diegows has quit IRC | 06:34 | |
| *** TravT has joined #openstack-glance | 06:42 | |
| *** buzztroll has joined #openstack-glance | 06:42 | |
| *** TravT has quit IRC | 06:46 | |
| *** buzztroll has quit IRC | 06:53 | |
| *** buzztroll has joined #openstack-glance | 06:55 | |
| *** groen692 has joined #openstack-glance | 06:56 | |
| *** belmoreira has joined #openstack-glance | 06:57 | |
| *** buzztroll has quit IRC | 07:04 | |
| *** buzztrol_ has joined #openstack-glance | 07:07 | |
| *** chlong has quit IRC | 07:10 | |
| *** sgotliv_ has quit IRC | 07:16 | |
| *** buzztrol_ has quit IRC | 07:22 | |
| *** buzztroll has joined #openstack-glance | 07:23 | |
| *** buzztroll has quit IRC | 07:28 | |
| *** buzztrol_ has joined #openstack-glance | 07:28 | |
| *** tshefi has joined #openstack-glance | 07:28 | |
| *** pkoniszewski has joined #openstack-glance | 07:28 | |
| *** slDabbler has quit IRC | 07:35 | |
| *** pcaruana has joined #openstack-glance | 07:42 | |
| *** Krast has joined #openstack-glance | 07:47 | |
| *** Krast has quit IRC | 07:47 | |
| *** sabari is now known as zz_sabari | 07:47 | |
| *** buzztrol_ has quit IRC | 07:48 | |
| *** buzztroll has joined #openstack-glance | 07:48 | |
| *** zz_sabari is now known as sabari | 07:50 | |
| *** buzztroll has quit IRC | 07:54 | |
| *** markus_z has joined #openstack-glance | 08:04 | |
| *** nlevinki has joined #openstack-glance | 08:07 | |
| junhongl | zhiyan: ping | 08:15 |
| *** markus_z has quit IRC | 08:19 | |
| *** sabari is now known as zz_sabari | 08:19 | |
| *** zz_sabari is now known as sabari | 08:21 | |
| junhongl | hi all, i opened a bug https://bugs.launchpad.net/glance/+bug/1408940, which is about the lack of support to https in glance-replicator clis | 08:21 |
| junhongl | i read the implementation of glance-replicator | 08:22 |
| junhongl | and found that the api call is initiated by httplib.HTTPConnection only without considering the https case | 08:24 |
| *** markus_z has joined #openstack-glance | 08:24 | |
| junhongl | i think we can enhance it by adding httplib.HTTPSConnection into it. but i'm hesitating about it because i noticed the implementation of glance-replicator | 08:25 |
| *** nellysmitt has joined #openstack-glance | 08:26 | |
| junhongl | is kind of different from other clis, i don't see other clis like nova, cinder use httplib, so i would like to see any comments from your guys | 08:27 |
| junhongl | about how to enhance glance-replicator to let it support https | 08:27 |
| *** TravT has joined #openstack-glance | 08:30 | |
| *** jamielennox is now known as jamielennox|away | 08:34 | |
| *** TravT has quit IRC | 08:35 | |
| *** sgotliv_ has joined #openstack-glance | 08:44 | |
| *** sabari is now known as zz_sabari | 08:48 | |
| *** flwang1 has joined #openstack-glance | 08:48 | |
| *** zz_sabari is now known as sabari | 08:48 | |
| *** krykowski has joined #openstack-glance | 08:53 | |
| *** nlevinki has quit IRC | 09:04 | |
| *** sgotliv_ has quit IRC | 09:10 | |
| *** nlevinki has joined #openstack-glance | 09:19 | |
| *** krykowski has quit IRC | 09:22 | |
| *** pdb has joined #openstack-glance | 09:23 | |
| *** jistr has joined #openstack-glance | 09:24 | |
| *** sgotliv_ has joined #openstack-glance | 09:27 | |
| *** krykowski has joined #openstack-glance | 09:28 | |
| *** sabari is now known as zz_sabari | 09:42 | |
| *** krykowski has quit IRC | 09:49 | |
| *** MattMan has joined #openstack-glance | 09:54 | |
| *** MattMan has left #openstack-glance | 09:54 | |
| *** MattMan has joined #openstack-glance | 09:55 | |
| *** alex_xu has joined #openstack-glance | 09:58 | |
| *** belmoreira has quit IRC | 10:02 | |
| *** groen693 has joined #openstack-glance | 10:07 | |
| *** andrearo1 has joined #openstack-glance | 10:07 | |
| *** andrearosa has quit IRC | 10:07 | |
| *** groen692 has quit IRC | 10:08 | |
| *** nlevinki has quit IRC | 10:11 | |
| *** krykowski has joined #openstack-glance | 10:14 | |
| *** TravT has joined #openstack-glance | 10:19 | |
| *** TravT has quit IRC | 10:24 | |
| *** nlevinki has joined #openstack-glance | 10:24 | |
| *** takedakn has joined #openstack-glance | 10:28 | |
| *** nellysmi_ has joined #openstack-glance | 10:34 | |
| *** nellysmitt has quit IRC | 10:35 | |
| *** andrearo1 is now known as andrearosa | 10:37 | |
| *** takedakn has quit IRC | 10:38 | |
| *** sgotliv_ has quit IRC | 10:38 | |
| *** sgotliv_ has joined #openstack-glance | 10:42 | |
| *** aix has joined #openstack-glance | 10:43 | |
| openstackgerrit | Pawel Koniszewski proposed openstack/glance: Add operators to provide multivalue support https://review.openstack.org/148213 | 10:44 |
| *** sgotliv_ has quit IRC | 10:47 | |
| *** sgotliv_ has joined #openstack-glance | 10:59 | |
| openstackgerrit | Fei Long Wang proposed openstack/glance: Ensure digital signatures are a minimum of SHA2 https://review.openstack.org/80178 | 11:03 |
| *** pkoniszewski has quit IRC | 11:13 | |
| *** flwang1 has quit IRC | 11:16 | |
| *** nellysmitt has joined #openstack-glance | 11:32 | |
| *** nellysmi_ has quit IRC | 11:36 | |
| openstackgerrit | Inessa Vasilevskaya proposed openstack/glance: Artifacts Domain https://review.openstack.org/132898 | 11:39 |
| *** krykowski has quit IRC | 11:43 | |
| *** krykowski has joined #openstack-glance | 11:46 | |
| openstackgerrit | Mike Fedosin proposed openstack/glance: Artifacts API https://review.openstack.org/136629 | 11:47 |
| *** sgotliv_ has quit IRC | 11:54 | |
| *** sgotliv has joined #openstack-glance | 11:54 | |
| openstackgerrit | Louis Taylor proposed openstack/glance-specs: Improvements to swift store retry behaviour https://review.openstack.org/146437 | 12:06 |
| *** krykowski has quit IRC | 12:09 | |
| openstackgerrit | Mike Fedosin proposed openstack/glance: Artifacts API https://review.openstack.org/136629 | 12:10 |
| *** pkoniszewski has joined #openstack-glance | 12:16 | |
| *** belmoreira has joined #openstack-glance | 12:27 | |
| *** belmoreira has quit IRC | 12:29 | |
| *** takedakn has joined #openstack-glance | 12:37 | |
| *** takedakn has quit IRC | 12:39 | |
| *** krykowski has joined #openstack-glance | 12:56 | |
| *** sgotliv has quit IRC | 12:58 | |
| *** takedakn has joined #openstack-glance | 13:10 | |
| *** takedakn has quit IRC | 13:12 | |
| *** takedakn has joined #openstack-glance | 13:12 | |
| openstackgerrit | Inessa Vasilevskaya proposed openstack/glance: Artifacts Domain https://review.openstack.org/132898 | 13:13 |
| *** takedakn has quit IRC | 13:13 | |
| *** sgotliv has joined #openstack-glance | 13:13 | |
| *** takedakn has joined #openstack-glance | 13:13 | |
| openstackgerrit | Inessa Vasilevskaya proposed openstack/glance: Artifacts API https://review.openstack.org/136629 | 13:20 |
| *** EmilienM|afk is now known as EmilienM | 13:22 | |
| *** pkoniszewski has quit IRC | 13:36 | |
| *** nellysmitt has quit IRC | 13:52 | |
| *** nellysmitt has joined #openstack-glance | 13:53 | |
| *** TravT has joined #openstack-glance | 13:57 | |
| *** sileht has quit IRC | 13:58 | |
| *** diegows has joined #openstack-glance | 13:59 | |
| *** TravT has quit IRC | 14:02 | |
| *** takedakn has quit IRC | 14:03 | |
| *** julim has joined #openstack-glance | 14:06 | |
| *** julim has quit IRC | 14:08 | |
| *** diegows has quit IRC | 14:08 | |
| *** takedakn has joined #openstack-glance | 14:11 | |
| *** julim has joined #openstack-glance | 14:11 | |
| *** sileht has joined #openstack-glance | 14:11 | |
| *** spzala has joined #openstack-glance | 14:18 | |
| *** spzala has quit IRC | 14:20 | |
| *** sgotliv has quit IRC | 14:20 | |
| *** sgotliv has joined #openstack-glance | 14:20 | |
| *** mjturek has joined #openstack-glance | 14:22 | |
| *** peristeri has joined #openstack-glance | 14:27 | |
| *** sileht has quit IRC | 14:32 | |
| *** krykowski has quit IRC | 14:36 | |
| *** sileht has joined #openstack-glance | 14:37 | |
| *** krykowski has joined #openstack-glance | 14:48 | |
| *** EmilienM is now known as EmilienM|afk | 14:57 | |
| *** pkoniszewski has joined #openstack-glance | 14:59 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:08 | |
| sigmavirus24 | junhongl: if you're still around can you show me the bug? | 15:13 |
| openstackgerrit | Pawel Koniszewski proposed openstack/glance: Add operators to provide multivalue support https://review.openstack.org/148213 | 15:15 |
| sigmavirus24 | nikhil_k_: are you around? | 15:16 |
| *** esheffield has joined #openstack-glance | 15:21 | |
| ativelkov | Folks, any experts on swift store around? | 15:26 |
| kragniz | ativelkov: define expert | 15:27 |
| ativelkov | anybody who knows the chunking approach of the swift sotre | 15:28 |
| nikhil_k_ | sigmavirus24: o/ | 15:29 |
| nikhil_k_ | ativelkov: what's the issue? | 15:29 |
| ativelkov | I have some strange behavior with it and cannot understand what's going on | 15:29 |
| *** krykowski has quit IRC | 15:30 | |
| nikhil_k_ | hmm, mind elaborating a bit more? | 15:30 |
| sigmavirus24 | ativelkov: or providing a link to what you're looking at? | 15:30 |
| ativelkov | moment. typing... ) | 15:30 |
| ativelkov | nikhil_k_: I am creating an image with "copy-from" command. The image is larger then 'swift_store_large_object_size', i.e. the image is stored as multiple swift objects. The sizes of objects in switch does not match the size of the original image. i.e. the data seems corrupt | 15:31 |
| sigmavirus24 | nikhil_k_: https://blueprints.launchpad.net/glance/+spec/software-metadefs was submitted Friday and the author targetted it to k-2/kilo which seems to be a stretch since no one's even had the chance to review the spec yet. Might be a good idea to remove the target until it's been discussed some and then we can decide on the milestone/release | 15:31 |
| sigmavirus24 | ativelkov: larger or smaller? | 15:31 |
| ativelkov | sigmavirus24: size in swift is smaller. I.e. some data is missing | 15:32 |
| sigmavirus24 | hm | 15:32 |
| *** krykowski has joined #openstack-glance | 15:33 | |
| ativelkov | no errors in glance logs till I attempt to download the image. When I do, there is an obvious error of checksum mismatch | 15:33 |
| ativelkov | Issue seems to be reproducible on latest devstack. | 15:34 |
| ativelkov | (and on stable/icehouse as well) | 15:35 |
| sigmavirus24 | huh | 15:35 |
| ativelkov | There is no problem if creating image directly from local file (i.e. no --copy-from param) or if the swift_store_large_object_size is larger then the size of the image | 15:37 |
| * sigmavirus24 goes to investigate | 15:37 | |
| *** vijendar has joined #openstack-glance | 15:38 | |
| ativelkov | sigmavirus24: thanks! | 15:38 |
| sigmavirus24 | ativelkov: don't expect much | 15:38 |
| sigmavirus24 | I know nothing about this code | 15:38 |
| sigmavirus24 | but I'm curious =P | 15:38 |
| ativelkov | Sure :) But any help is appreciated | 15:38 |
| nikhil_k_ | ativelkov: err, got distracted | 15:39 |
| nikhil_k_ | ativelkov: there was a bug which mclaren opened for copy-from not working after grizzly | 15:39 |
| nikhil_k_ | lemme check | 15:39 |
| nikhil_k_ | hmm, your issue seems different - though here it is https://bugs.launchpad.net/glance/+bug/1398903 | 15:40 |
| *** nlevinki has quit IRC | 15:41 | |
| ativelkov | yes, this looks different. In my case there are no errors, and the image successfully moves to "active" state | 15:42 |
| ativelkov | but sizes in swift differs from the expected size | 15:42 |
| nikhil_k_ | ativelkov: are you measuring the summation of sizes? | 15:46 |
| nikhil_k_ | ativelkov: also, the checksum in glance is a checksum or checksum (etag) | 15:46 |
| nikhil_k_ | checksum _of_ checksum | 15:47 |
| ativelkov | nikhil_k_: yes, summation of sizes | 15:48 |
| ativelkov | also, swift has some special object which is (probably) corresponding to all the chunks | 15:48 |
| nikhil_k_ | sigmavirus24: Thanks for raising that one. Will definitely look int this | 15:49 |
| nikhil_k_ | seems like a small change, mostly document library of the descriptions | 15:49 |
| nikhil_k_ | so might be do-able | 15:49 |
| nikhil_k_ | ativelkov: right, it by default falls back to a Dynamic Large Object (DLO) | 15:50 |
| nikhil_k_ | it has a file called manifest file stored in swift proxy to locate the chunks across the ecosystem of storage drives | 15:50 |
| nikhil_k_ | you'd see it to be an empty 0 bytes file while uploading from glance | 15:51 |
| ativelkov | ok, so the size of this object is the same as the sum of all chunks | 15:51 |
| nikhil_k_ | ativelkov: btw, does glance give out error while downloading that image? | 15:51 |
| ativelkov | and is different from the image which was passed as the copy-from source | 15:51 |
| nikhil_k_ | ativelkov: ah that's a relief | 15:51 |
| ativelkov | nikhil_k_: yes, it does | 15:51 |
| nikhil_k_ | hmm | 15:51 |
| nikhil_k_ | do you mind sharing the version of swiftclient and glance_store too? | 15:52 |
| *** tellesnobrega has quit IRC | 15:52 | |
| ativelkov | sure | 15:52 |
| ativelkov | one moment | 15:52 |
| nikhil_k_ | looking at glance/master there does not seem to be any issue right up front | 15:52 |
| ativelkov | right now python-swiftclient is at 2.2.0, but that's after I've triyed playing with it. Initially it installed most recent version (2.3.1) and it had the same issue | 15:54 |
| ativelkov | glance-store==0.1.10 | 15:54 |
| *** nlevinki has joined #openstack-glance | 15:55 | |
| ativelkov | nikhil_k_: some output here: http://paste.openstack.org/show/158899/ | 15:58 |
| ativelkov | swift_large_object_size is set to 200 megs | 15:59 |
| *** sileht has quit IRC | 16:01 | |
| ativelkov | seems like it lost 25984 K of data. | 16:01 |
| ativelkov | Which is a multiple of 16K, where 16K is size of datachunk to be transferred by swiftclient | 16:02 |
| *** nlevinki has quit IRC | 16:02 | |
| ativelkov | 1624 chunks lost. May be just a coincidence, of course | 16:02 |
| *** krykowski has quit IRC | 16:02 | |
| sigmavirus24 | ativelkov: I thought you said each of the swift chunks added up to the original image size | 16:03 |
| sigmavirus24 | did I misunderstand you? | 16:03 |
| ativelkov | sigmavirus24: no, these are different "chunks" | 16:03 |
| ativelkov | there are "chunks" as different objects used by glance to split large object into smaller ones | 16:04 |
| ativelkov | these are configured to be 200 Mb | 16:04 |
| ativelkov | (not exactly, because we use some strange constant for 1 Mb here: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L46) | 16:05 |
| *** tellesnobrega has joined #openstack-glance | 16:06 | |
| sigmavirus24 | yeah I saw that | 16:07 |
| ativelkov | So, for object of 536412160 bytes the first object in swift is 204800000, and so is the second | 16:08 |
| *** EmilienM|afk is now known as EmilienM | 16:08 | |
| ativelkov | But the third one is 100204544, while it should be 126 812 160 | 16:09 |
| *** nlevinki has joined #openstack-glance | 16:15 | |
| openstackgerrit | Louis Taylor proposed openstack/glance_store: Add option to wait between swift retries https://review.openstack.org/141803 | 16:15 |
| openstackgerrit | Louis Taylor proposed openstack/glance_store: Default number of swift download retries to 5 https://review.openstack.org/145907 | 16:15 |
| kragniz | nikhil_k_: reckon we could start getting some of the oslo namespace patches merged? | 16:22 |
| kragniz | sigmavirus24: I ended up just removing the eventlet stuff from the glance_store swift patch. This means the consumer of glance_store should do the monkey_patching | 16:25 |
| kragniz | sigmavirus24: reckon this is a better way to go? | 16:25 |
| sigmavirus24 | kragniz: given that eventlet will monkey-patch the world I think that's okay | 16:25 |
| * sigmavirus24 isn't 100% certain though | 16:25 | |
| kragniz | I'm pretty sure it does, given how python modules work | 16:26 |
| *** olc has joined #openstack-glance | 16:26 | |
| sigmavirus24 | also how eventlet works | 16:26 |
| sigmavirus24 | I think we just need to check that something does monkey patch the world with eventlet before assuming it will just work | 16:27 |
| kragniz | sigmavirus24: https://github.com/openstack/glance/blob/master/glance/cmd/api.py#L32 | 16:28 |
| kragniz | same for the registry | 16:28 |
| olc | hi - sri for crosspost :) Did someone build an image for jessie successfully using build-openstack-debian-image? | 16:28 |
| * sigmavirus24 didn't olc | 16:29 | |
| kragniz | olc: not I, although I haven't attempted it | 16:30 |
| *** groen693 has quit IRC | 16:30 | |
| *** sileht has joined #openstack-glance | 16:33 | |
| *** sileht has quit IRC | 16:33 | |
| *** tshefi has quit IRC | 16:34 | |
| *** sileht has joined #openstack-glance | 16:34 | |
| nikhil_k_ | ativelkov: the upload_utils has a method to check if the checksum back from the store is same as that in the image meta | 16:34 |
| nikhil_k_ | ativelkov: how did your upload happen successfully? | 16:35 |
| olc | hmm ok, thanks guys! No inittab is created by default in jessie and build-openstack-debian-image complains about that. Anyway I'll investigate further. | 16:37 |
| ativelkov | nikhil_k_: well, I don't know. May be because the checksum being computed at swift-store is correct, and the problem happens after it is computed? | 16:39 |
| nikhil_k_ | ativelkov: yeah, my guess is that something wrong with download; in the case of this particular way of image data upload to your backend storage | 16:40 |
| nikhil_k_ | ativelkov: mind cross checking the checksum entity on image with that on swift http://paste.openstack.org/show/158899/ (<- not here) | 16:40 |
| *** cpallares has joined #openstack-glance | 16:40 | |
| ativelkov | nikhil_k_: sorry, didn't understand what do you suggest | 16:41 |
| nikhil_k_ | ativelkov: nvm | 16:43 |
| nikhil_k_ | though verifying the checkum might give us an indication on | 16:44 |
| ativelkov | What is the source of the "checksum" value which is stored in glance? | 16:44 |
| nikhil_k_ | whether the issue is in the downloading process or the storage is behaving inconsistently | 16:44 |
| ativelkov | Is it computed when the image is being uploaded? | 16:45 |
| *** tshefi has joined #openstack-glance | 16:46 | |
| *** TravT has joined #openstack-glance | 16:47 | |
| *** TravT has quit IRC | 16:51 | |
| *** nlevinki has quit IRC | 16:52 | |
| *** tshefi has quit IRC | 16:55 | |
| *** markus_z has quit IRC | 16:59 | |
| *** rwsu has joined #openstack-glance | 16:59 | |
| nikhil_k_ | ativelkov: a bit distracted this morning. can you please create a bug for this? we may get some more info from other folks | 17:04 |
| nikhil_k_ | ativelkov: and yeah, the checksum is set after upload process is complete | 17:04 |
| ativelkov | nikhil_k_: sure, will do in about 30 mins | 17:05 |
| ativelkov | nikhil_k_: the checksum is correct, btw. I mean, when I md5 the original file it has the same value as in Glance | 17:05 |
| *** spzala has joined #openstack-glance | 17:05 | |
| nikhil_k_ | however, the weird thing is it's set based on what is recieved back from the add_to_backend method | 17:05 |
| nikhil_k_ | ativelkov: ah k | 17:06 |
| ativelkov | so, turns out that the code in glance_store reads all the data from remote source correctly - and computes the sum correctly as well. But something gets lost on the way to swift | 17:08 |
| sigmavirus24 | weird | 17:09 |
| mfedosin | can it be because 1 mb in glance store is ONE_MB = 1000 * 1024 https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L46 instead of 1024 * 1024? | 17:09 |
| sigmavirus24 | mfedosin: yeah we all noticed that | 17:09 |
| ativelkov | mfedosin: no, that is not the reason: I've just changed it to 1024*1024, but the problem is still there | 17:10 |
| nikhil_k_ | ativelkov: do you have access to the store logs? might be worth grepping for that X-Trans-Id: tx01a19f7476a541808c9a1-0054bd28e1 | 17:10 |
| nikhil_k_ | or whatever corres. to that upload (this might be your stat id) | 17:11 |
| ativelkov | nikhil_k_: it's a devstack on my VM, so I have any access, But I don't know anything about swift and its logging, don't know where to look | 17:11 |
| nikhil_k_ | try starting from swift proxy service | 17:12 |
| *** pkoniszewski has quit IRC | 17:13 | |
| *** EmilienM is now known as EmilienM|afk | 17:27 | |
| kragniz | sigmavirus24: that was a harsh -1 :P | 17:28 |
| sigmavirus24 | kragniz: yw | 17:29 |
| openstackgerrit | Louis Taylor proposed openstack/glance-specs: Improvements to swift store retry behaviour https://review.openstack.org/146437 | 17:29 |
| sigmavirus24 | kragniz: thoughts on https://bugs.launchpad.net/glance/+bug/1412394 ? | 17:39 |
| sigmavirus24 | kragniz: it's related to flaper87's work on https://review.openstack.org/#/c/147912/ | 17:39 |
| flaper87 | what did I do now? | 17:40 |
| flaper87 | haven't you learned you shouldn't be summoing ghosts ? | 17:40 |
| sigmavirus24 | flaper87: you grievanced me last | 17:42 |
| *** pdb has quit IRC | 17:43 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:50 | |
| kragniz | sigmavirus24_awa: /win 43 | 18:06 |
| kragniz | grr | 18:06 |
| kragniz | sigmavirus24_awa: I don't think he's proposing the schema is made more permissive, just more common parameters are added to it | 18:07 |
| kragniz | which I think is fairly sane | 18:08 |
| *** pcaruana has quit IRC | 18:08 | |
| openstackgerrit | Mike Fedosin proposed openstack/glance: Add the ability to specify the sort dir for each key https://review.openstack.org/148326 | 18:15 |
| ativelkov | nikhil_k_: I've checked the swift-proxy logs.. No errors there, but seems like it does not receive all the needed data. The log contains messages like "Reading chunk..." and "Read 65536 bytes". If we sum all these bytes we will get 100204544 bytes for the last (third) object. That is exactly what is actually stored in swift. But it should be 116981760 instead. Seems like I have 16 megs missing | 18:16 |
| mfedosin | client support for multiple sort dirs will be added tomorrow with a new syntax | 18:17 |
| *** TravT has joined #openstack-glance | 18:19 | |
| *** sgotliv has quit IRC | 18:20 | |
| *** TravT has quit IRC | 18:24 | |
| *** EmilienM|afk is now known as EmilienM | 18:24 | |
| *** openstackgerrit has quit IRC | 18:37 | |
| *** openstackgerrit has joined #openstack-glance | 18:37 | |
| *** aix has quit IRC | 18:41 | |
| *** Longgeek has quit IRC | 18:47 | |
| nikhil_k_ | ativelkov: that's weird. don't want to increase your work, however if you get a chance - might be worth looking at the g-api logs to cross-verify how much data was written | 18:47 |
| ativelkov | nikhil_k_: Doing just this: I am adding some logging to python-swiftclient | 18:48 |
| ativelkov | to log every 64 chunk which it sends | 18:48 |
| nikhil_k_ | it will have a log like msg = ("Wrote chunk %(chunk_name)s (%(chunk_id)d/" | 18:48 |
| ativelkov | nikhil_k_: afair these "chunks" are the large ones | 18:49 |
| nikhil_k_ | yeah, glance chunks | 18:49 |
| ativelkov | 200 Mb or whatever is configured | 18:49 |
| nikhil_k_ | yes | 18:49 |
| ativelkov | And I am looking for smaller pieaces of lost data | 18:49 |
| nikhil_k_ | no, good point to increase logging to swiftclient! | 18:49 |
| * nikhil_k_ hides | 18:50 | |
| *** nikhil_k_ is now known as nikhil_k | 18:50 | |
| ativelkov | wait, no. Iven the logging messages you speak about show the problem | 18:51 |
| ativelkov | Even* | 18:51 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:51 | |
| sigmavirus24 | kragniz: I have no objections but they were in fact complaining that we don't allow os_type which we do... if we're told to | 18:52 |
| ativelkov | Wrote chunk faf778a5-84d8-48a4-bf64-963fe977c72d-00001 (1/3) of length 218103808 to Swift returning MD5 of content: 1296786a9aa728616adf4235f6a8d2df add /usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py:525 | 18:52 |
| ativelkov | then Wrote chunk faf778a5-84d8-48a4-bf64-963fe977c72d-00002 (2/3) of length 218103808 to Swift returning MD5 of content: 0b95b310aa2c532bbf4b15522f0c2334 add /usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py:525 | 18:52 |
| ativelkov | Wrote chunk faf778a5-84d8-48a4-bf64-963fe977c72d-00003 (3/3) of length 100204544 to Swift returning MD5 of content: 75d857ea350ff04b9836e1deb85f3d7e add /usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py:525 | 18:53 |
| sigmavirus24 | wut | 18:53 |
| ativelkov | It looks like this is it | 18:53 |
| ativelkov | According to config glance chunks should be 209715200 bytes (and actually swift reports of receiving exact this amount). But Glance thinks that it has sent 218103808 | 18:54 |
| sigmavirus24 | which is the size of the entire image? | 18:55 |
| kragniz | sigmavirus24: I guess, I was reading it as changing the default schema to explicitly list the allowed properties | 18:57 |
| ativelkov | sigmavirus24: entire image is 536412160 | 18:57 |
| sigmavirus24 | kragniz: with the context from the review it seemed like they were complaining that it wasn't there to begin with because nova sends extra cruft that the service isn't necessarily going to use or expect | 18:58 |
| nikhil_k | ativelkov: the chunk addition from the logs seem right and come to 536412160 | 18:58 |
| sigmavirus24 | because nova's never heard the part about "be conservative in what you send" they only heard "be liberal" | 18:58 |
| *** TravT has joined #openstack-glance | 18:58 | |
| ativelkov | nikhil_k: right | 18:58 |
| ativelkov | nikhil_k: but it does not actually send 218103808 - it should send 204800000 | 18:59 |
| ativelkov | as it is configured to create chunks of 200 "megabytes" | 18:59 |
| ativelkov | and it actually sends these 204800000 (according to swift) | 19:00 |
| sigmavirus24 | ah that's the ONE_MB problem :( | 19:00 |
| ativelkov | nope | 19:00 |
| ativelkov | if ONE_MB is set to real one mb, glance is supposed to create chunks of 209715200 bytes. | 19:01 |
| ativelkov | But it still reports the same 218103808 per chunk | 19:01 |
| sigmavirus24 | odd | 19:01 |
| ativelkov | you know what? These 218103808 are exactly 13*16*Mb | 19:02 |
| ativelkov | 16 is the READ_CHUNK_SIZE | 19:02 |
| ativelkov | for the external source | 19:03 |
| ativelkov | (HTTP) | 19:03 |
| *** bkopilov has quit IRC | 19:03 | |
| *** TravT has quit IRC | 19:03 | |
| ativelkov | So, I now see what is going on | 19:03 |
| ativelkov | Glance copies data from remote source in 16 Mb chunks | 19:04 |
| ativelkov | and "flushes" this data to swift in large 200 "megabyte" chunks | 19:04 |
| ativelkov | but 200 "megabytes" (and even 200 megabytes if we had a correct constant) is not a multiple of 16Mb | 19:05 |
| ativelkov | so, some (significant) part of the last 16 mb chunk which was read from external source is discarded at the end of each 200 Mb chunk | 19:06 |
| ativelkov | thus, we are loosing data | 19:06 |
| nikhil_k | ativelkov: that is an excellent catch | 19:06 |
| nikhil_k | and does show loop hole into not having checksum for that upload chunk process! | 19:07 |
| nikhil_k | checksum verification :( | 19:07 |
| ativelkov | no, the checksum verifies correctly | 19:07 |
| ativelkov | as it verifies it against datat which was READ, not against which was sent | 19:07 |
| nikhil_k | even after upload to store? | 19:07 |
| nikhil_k | right, yes | 19:07 |
| nikhil_k | so, it does that partially | 19:08 |
| *** zul has quit IRC | 19:09 | |
| nikhil_k | This may mean we need a common module for store drivers to declare constants | 19:09 |
| d34dh0r53 | nikhil_k: sigmavirus24 pointed me in your direction, is there any magic required to get glance client from master to auth against rackspace public cloud? | 19:09 |
| ativelkov | nikhil_k: I'd prefer to decouple constan sizes from each other | 19:10 |
| ativelkov | we just don't need to loose the last part of previous read chunk | 19:10 |
| nikhil_k | d34dh0r53: hmm, tricky one | 19:10 |
| nikhil_k | d34dh0r53: what error do you see? | 19:10 |
| d34dh0r53 | Unauthorized 401 | 19:11 |
| kragniz | sigmavirus24: yeah, I don't think it should accept any cruft by default | 19:11 |
| sigmavirus24 | kragniz: we can accept it but that doesn't mean it should be in the schema | 19:11 |
| sigmavirus24 | Do we use/need os_type even? | 19:11 |
| nikhil_k | ativelkov: basically, in future if some store driver adds constants - we will get a inconsistent chunk size ratio | 19:11 |
| d34dh0r53 | nikhil_k: don't see any debug or verbose output either | 19:12 |
| *** zul has joined #openstack-glance | 19:12 | |
| kragniz | sigmavirus24: we don't need it to be | 19:12 |
| kragniz | as far as I know | 19:12 |
| sigmavirus24 | kragniz: trick question. https://github.com/openstack/glance/blob/5a74548f4649e1cbb8914f4ac5e1d889be54efc4/etc/metadefs/compute-xenapi.json#L13 is the only reference to it | 19:12 |
| * kragniz not sure | 19:18 | |
| kragniz | anyone know if there's a list of the properties used by nova drivers? | 19:18 |
| kragniz | these seem to be the only 'blessed' properties http://docs.openstack.org/developer/glance/common-image-properties.html | 19:19 |
| *** zz_sabari is now known as sabari | 19:20 | |
| sigmavirus24 | nikhil_k: we can probably mark this as rejected right? https://blueprints.launchpad.net/glance/+spec/remove-simple-driver (before the meeting) | 19:26 |
| *** bkopilov has joined #openstack-glance | 19:26 | |
| nikhil_k | sigmavirus24: +1 | 19:27 |
| nikhil_k | sigmavirus24: can you do that? | 19:28 |
| sigmavirus24 | nikhil_k: would you mind doing that then? I dont' have permissions to | 19:28 |
| sigmavirus24 | heh | 19:28 |
| nikhil_k | ahk | 19:28 |
| nikhil_k | :) | 19:28 |
| nikhil_k | (that's what I meant btw) (confusing IRC <.<) | 19:28 |
| mfedosin | read here ignores the length param https://github.com/openstack/glance_store/blob/master/glance_store/common/utils.py#L127 may be it's our problem? | 19:29 |
| nikhil_k | d34dh0r53: may be this should help https://github.com/emonty/rackspace-auth-openstack | 19:31 |
| nikhil_k | d34dh0r53: Monty was using the CLI a few weeks back and did not have any issues.. | 19:32 |
| sigmavirus24 | ^ | 19:34 |
| *** bkopilov has quit IRC | 19:34 | |
| d34dh0r53 | nikhil_k: ty sir | 19:34 |
| nikhil_k | np | 19:34 |
| *** bkopilov has joined #openstack-glance | 19:36 | |
| ativelkov | nikhil_k: ok, I've got the root cause of the issue | 19:40 |
| ativelkov | https://github.com/openstack/glance/blob/master/glance/common/utils.py#L153 here, the cooperative reader ignores the length we asked for | 19:41 |
| ativelkov | which is understandable, as its underlying is not a file or a stream, but just a generator | 19:42 |
| ativelkov | which may return only fixed-size chunks | 19:42 |
| nikhil_k | gotcha | 19:44 |
| nikhil_k | ativelkov: Though, the issue really is with global constants not being multiples of each other right? | 19:45 |
| ativelkov | no | 19:45 |
| ativelkov | they don't have to be | 19:45 |
| nikhil_k | hmm | 19:45 |
| ativelkov | we just dontt need to rely on this | 19:46 |
| nikhil_k | implement a carry over? | 19:46 |
| ativelkov | We actually do stuff wrong in our ChunkReader | 19:46 |
| ativelkov | when the consumer code (swift-client in this case) ask our datat stream for a next chunk, it specifies the size | 19:46 |
| ativelkov | in my case it is 8K per chunk | 19:47 |
| ativelkov | but we return 6Mb instead, assuming that they will maintain the buffer | 19:47 |
| ativelkov | not 6Mb, 16Mb | 19:47 |
| nikhil_k | ativelkov: the put_object specifies a content length of 200M (by default) | 19:54 |
| ativelkov | nikhil_k: ok, let's put it this way. Look here: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L894 | 19:55 |
| ativelkov | what value would you expect for i? | 19:55 |
| nikhil_k | ativelkov: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L518 | 19:58 |
| nikhil_k | chunk_size | 19:58 |
| ativelkov | nikhil_k: no. Read is being called by the code which tries to read from this ChunkReader | 19:58 |
| ativelkov | in our case this is httplib which is called by swiftclient | 19:59 |
| ativelkov | this is how it looks like: http://paste.openstack.org/show/158958/ | 20:00 |
| ativelkov | (line 14) | 20:00 |
| ativelkov | the lib requests next 8K of data from the stream | 20:01 |
| ativelkov | this value (8192) gets to the read method of ChunkReader | 20:01 |
| ativelkov | And ChunkReader actually ignores this value and returns the amount of data it is configured to read from the remote source | 20:05 |
| *** sabari is now known as zz_sabari | 20:05 | |
| nikhil_k | ativelkov: ok, might be best discussing this on a bug. | 20:07 |
| sigmavirus24 | nikhil_k +1 | 20:08 |
| nikhil_k | Trying to co-relate where exactly the 16M factor is coming in :| | 20:08 |
| nikhil_k | Seems like we are dealing with 2 streams | 20:08 |
| ativelkov | nikhil_k: I'll file a bug - and a patchset | 20:08 |
| nikhil_k | one using reading from remote source (store type A) and other dest. (store type B( | 20:09 |
| nikhil_k | if you can please include the flow of things looking correct on download and not on upload and the stream mis-match, it would be really appreciated! | 20:10 |
| *** flwang1 has joined #openstack-glance | 20:18 | |
| flwang | sigmavirus24: ping | 20:20 |
| sigmavirus24 | flwang: pong | 20:20 |
| flwang | I'm looking the comment from you on https://review.openstack.org/#/c/80178/12/glance/common/config.py | 20:20 |
| flwang | sigmavirus24: I'm wondering why it deserved a -1 | 20:21 |
| sigmavirus24 | flwang: I'm more concerned with using more secure defaults. So if we're not using a more secure default, I'm -1 until a good reason has been supplied | 20:22 |
| flwang | as for your question in the comment, the reason why i'm using sha1 as the default is because we want to get a smooth upgrade | 20:22 |
| *** jistr has quit IRC | 20:23 | |
| flwang | sigmavirus24: did you see my previous patch sets ? | 20:23 |
| flwang | did you see the other change in the same patch? | 20:23 |
| flwang | we use sha1 since it's the hard code value we're using currently | 20:24 |
| flwang | after discussed with nikhil_k, we believe set sha1 as the default is better for smooth upgrade from the operator/deployer perspective | 20:24 |
| flwang | and we will change it in next release(L) | 20:25 |
| sigmavirus24 | flwang: that makes sense. I didn't look at the earlier versions of the patch though | 20:25 |
| sigmavirus24 | I left other comments iirc too and kragniz had some comments that were worth addressing | 20:25 |
| flwang | kragniz's -1 is asking change (Using -> Use) | 20:26 |
| flwang | that's scaring me :P | 20:27 |
| sigmavirus24 | =P | 20:28 |
| *** sgotliv has joined #openstack-glance | 20:28 | |
| flwang | it's ok for me, but I think it will scare the new comer for glance team | 20:29 |
| openstackgerrit | Fei Long Wang proposed openstack/glance: Ensure digital signatures are a minimum of SHA2 https://review.openstack.org/80178 | 20:33 |
| kragniz | flwang: to be fair, I'm being very paranoid about differences between config files and the help text on the cfg options at the moment | 20:36 |
| kragniz | flwang: sorry for the -1 :P | 20:37 |
| flwang | kragniz: no worries :) i'm kidding | 20:37 |
| flwang | but it would be great if you guys can be nice for those new comer to avoid scaring them :D | 20:38 |
| *** zz_sabari is now known as sabari | 20:38 | |
| kragniz | flwang: of course :) | 20:39 |
| flwang | awesome :) | 20:39 |
| sigmavirus24 | flwang: I know why that default of sha1 threw me off | 20:42 |
| *** spzala_ has joined #openstack-glance | 20:42 | |
| sigmavirus24 | Your commit message says "Ensure digital signatures are a minimum of SHA2" | 20:42 |
| sigmavirus24 | Which is no longer true in the current patchset so I had that in mind while reviewing the rest of it | 20:43 |
| *** spzala has quit IRC | 20:43 | |
| *** spzala_ is now known as spzala | 20:43 | |
| sigmavirus24 | flwang: mind updating the commit message to represent reality? | 20:46 |
| *** sabari is now known as zz_sabari | 21:04 | |
| flwang | sigmavirus24: ah, good point | 21:04 |
| sigmavirus24 | flwang: I'd -1 you but I don't want new contributors to be afraid of us making them write accurate commit messages =P | 21:05 |
| flwang | that's true | 21:06 |
| openstackgerrit | Fei Long Wang proposed openstack/glance: Make digest algorithm configurable https://review.openstack.org/80178 | 21:31 |
| *** julim has quit IRC | 21:32 | |
| sigmavirus24 | flwang: ++ | 21:32 |
| flwang | sigmavirus24: cheers, man | 21:35 |
| cpallares | sigmavirus24: That's really nice of you to consider new contributors. | 21:37 |
| cpallares | sigmavirus24: +2 | 21:37 |
| *** diegows has joined #openstack-glance | 21:41 | |
| kragniz | we need a glance PR campaign to attract new contributors | 21:43 |
| kragniz | this should be in the form of a cool logo | 21:44 |
| * kragniz likes cool logos | 21:44 | |
| sigmavirus24 | kragniz: should be an illuminati eye | 21:45 |
| sigmavirus24 | get it. glance. eye. | 21:45 |
| sigmavirus24 | Ooooh. Eye of sauron! | 21:45 |
| sigmavirus24 | no wait, that's ceilometer | 21:45 |
| *** peristeri has quit IRC | 21:45 | |
| kragniz | heh | 21:45 |
| kragniz | getting eyes to fit the openstack branding style is kind of hard | 21:46 |
| kragniz | I should give it a go | 21:46 |
| sigmavirus24 | kragniz: use inkscape | 21:46 |
| cpallares | LOL | 21:46 |
| kragniz | of course | 21:46 |
| * cpallares likes the eye of sauron | 21:46 | |
| cpallares | I'd be great at marketing. | 21:46 |
| cpallares | Although my freddie kruger idea didn't go so well in Zaqar. | 21:47 |
| sigmavirus24 | cpallares: it'd always be able to find the OpenStack O | 21:47 |
| sigmavirus24 | lol | 21:47 |
| kragniz | haha | 21:47 |
| sigmavirus24 | One hypervisor to rule them all and in the darkness boot them | 21:47 |
| cpallares | sigmavirus24: You are a marketing genius. | 21:48 |
| kragniz | nine rings for the swift nodes under the cloud | 21:48 |
| sigmavirus24 | lmao | 21:48 |
| cpallares | We don't need the marketing team. We can do this all by ourselves. | 21:49 |
| cpallares | which reminds me: http://www.smbc-comics.com/comics/20120501.gif | 21:50 |
| kragniz | cpallares: haha | 21:50 |
| __hemanthm | There is some good in this cloud, and its worth developing for :P | 21:51 |
| sigmavirus24 | cpallares++ | 21:51 |
| *** __hemanthm is now known as hemanthm | 21:51 | |
| *** nellysmitt has quit IRC | 22:05 | |
| *** TravT has joined #openstack-glance | 22:33 | |
| *** rwsu has quit IRC | 22:35 | |
| *** TravT has quit IRC | 22:38 | |
| *** zz_sabari is now known as sabari | 22:39 | |
| *** echevemaster has joined #openstack-glance | 22:40 | |
| *** sabari is now known as zz_sabari | 22:40 | |
| *** zz_sabari is now known as sabari | 22:42 | |
| *** buzztroll has joined #openstack-glance | 22:44 | |
| *** buzztroll has quit IRC | 22:54 | |
| *** echevemaster has quit IRC | 22:54 | |
| *** bkopilov has quit IRC | 22:57 | |
| *** vijendar has quit IRC | 23:01 | |
| *** vijendar has joined #openstack-glance | 23:03 | |
| kragniz | hmm, that didn't really come out right | 23:05 |
| kragniz | http://a.pomf.se/ypyetb.png | 23:05 |
| kragniz | diskuminati! | 23:06 |
| *** vijendar has quit IRC | 23:07 | |
| *** sabari is now known as zz_sabari | 23:09 | |
| *** chlong has joined #openstack-glance | 23:11 | |
| *** takedakn has quit IRC | 23:12 | |
| cpallares | kragniz: haha | 23:15 |
| cpallares | that is amazing :) | 23:15 |
| cpallares | that could be the A in glAnce :P | 23:16 |
| sigmavirus24 | kragniz: that's hilarious | 23:16 |
| sigmavirus24 | kragniz: +2 +W | 23:17 |
| cpallares | kragniz: That eye looks a lot like a CD | 23:17 |
| sigmavirus24 | cpallares: I think that's the point of kragniz's "diskuminati" comment =P | 23:19 |
| cpallares | sigmavirus24: Ah! Reading the word makes sense. I didn't understand the reference at first glance. | 23:20 |
| cpallares | No pun intended. | 23:21 |
| sigmavirus24 | uhuh | 23:21 |
| sigmavirus24 | sure you didn't | 23:21 |
| *** jamielennox|away is now known as jamielennox | 23:21 | |
| sigmavirus24 | (intend the pun) | 23:21 |
| * cpallares intended the pun | 23:21 | |
| cpallares | jk | 23:21 |
| kragniz | :) | 23:24 |
| kragniz | cpallares: it was supposed to look like a CD! | 23:24 |
| *** zz_sabari is now known as sabari | 23:25 | |
| *** takedakn has joined #openstack-glance | 23:34 | |
| *** takedakn has quit IRC | 23:35 | |
| *** sabari is now known as zz_sabari | 23:42 | |
| *** zz_sabari is now known as sabari | 23:43 | |
| *** diegows has quit IRC | 23:45 | |
| *** TravT has joined #openstack-glance | 23:49 | |
| *** EmilienM is now known as EmilienM|afk | 23:52 | |
| *** sabari is now known as zz_sabari | 23:52 | |
| *** TravT has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
