| *** thrash is now known as thrash|g0ne | 00:07 | |
| *** gildub_ has joined #openstack-sdks | 00:22 | |
| *** gildub has quit IRC | 00:25 | |
| *** openstack has joined #openstack-sdks | 00:35 | |
| *** pm90_ has quit IRC | 00:35 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:46 | |
| *** pm90_ has joined #openstack-sdks | 00:50 | |
| *** steveortiz is now known as zz_steveortiz | 00:58 | |
| jamielennox | terrylhowe: still here? | 01:08 |
|---|---|---|
| *** Qiming has joined #openstack-sdks | 01:10 | |
| *** pm90_ has quit IRC | 01:10 | |
| *** pm90_ has joined #openstack-sdks | 01:19 | |
| *** liangbo has joined #openstack-sdks | 01:22 | |
| *** bnemec has quit IRC | 01:26 | |
| *** gildub_ has quit IRC | 01:27 | |
| *** gildub_ has joined #openstack-sdks | 01:43 | |
| *** Yanyanhu has joined #openstack-sdks | 01:45 | |
| *** gildub_ has quit IRC | 01:49 | |
| *** rm_work|away is now known as rm_work | 02:00 | |
| *** gildub_ has joined #openstack-sdks | 02:06 | |
| *** gildub_ has quit IRC | 02:10 | |
| *** rm_work is now known as rm_work|away | 02:10 | |
| *** rm_work|away is now known as rm_work | 02:17 | |
| *** boris-42 has quit IRC | 02:20 | |
| openstackgerrit | Everett Toews proposed stackforge/python-openstacksdk: Image import https://review.openstack.org/199318 | 02:41 |
| *** gildub has joined #openstack-sdks | 02:51 | |
| openstackgerrit | Everett Toews proposed stackforge/python-openstacksdk: Image import https://review.openstack.org/199318 | 02:52 |
| *** Caleb-- has quit IRC | 03:28 | |
| *** gildub has quit IRC | 03:38 | |
| *** boris-42 has joined #openstack-sdks | 03:45 | |
| *** gildub has joined #openstack-sdks | 03:55 | |
| *** Caleb-- has joined #openstack-sdks | 04:00 | |
| *** gildub has quit IRC | 04:49 | |
| *** gildub has joined #openstack-sdks | 04:49 | |
| *** Yanyanhu has quit IRC | 05:26 | |
| *** Yanyanhu has joined #openstack-sdks | 05:27 | |
| *** aufi has joined #openstack-sdks | 06:26 | |
| *** aufi has quit IRC | 06:47 | |
| *** e0ne has joined #openstack-sdks | 06:52 | |
| *** gildub has quit IRC | 06:52 | |
| *** aufi has joined #openstack-sdks | 07:30 | |
| *** e0ne has quit IRC | 07:45 | |
| *** dguerri` is now known as dguerri | 08:21 | |
| *** Yanyanhu has quit IRC | 08:25 | |
| *** Yanyanhu has joined #openstack-sdks | 08:25 | |
| openstackgerrit | Qiming Teng proposed stackforge/python-openstacksdk: add scheduler_hints support for server creation https://review.openstack.org/204637 | 08:38 |
| *** liangbo has quit IRC | 09:08 | |
| *** e0ne has joined #openstack-sdks | 09:08 | |
| *** e0ne has quit IRC | 09:18 | |
| *** e0ne has joined #openstack-sdks | 09:23 | |
| *** dguerri is now known as dguerri` | 09:40 | |
| *** liangbo has joined #openstack-sdks | 10:05 | |
| *** Yanyanhu has quit IRC | 10:20 | |
| openstackgerrit | Terry Howe proposed openstack/python-openstackclient: Create log configuration class https://review.openstack.org/211291 | 10:23 |
| openstackgerrit | Terry Howe proposed openstack/python-openstackclient: Allow debug to be set in configuration file https://review.openstack.org/211293 | 10:24 |
| *** Qiming has quit IRC | 10:26 | |
| *** Qiming has joined #openstack-sdks | 11:53 | |
| *** trown|outttypeww is now known as trown | 12:09 | |
| *** thrash|g0ne is now known as thrash | 12:12 | |
| *** petertr7_away is now known as petertr7 | 12:16 | |
| *** jaosorior has joined #openstack-sdks | 12:31 | |
| *** petertr7 is now known as petertr7_away | 12:53 | |
| *** petertr7_away is now known as petertr7 | 12:56 | |
| terrylhowe | I am no jamielennox | 12:57 |
| terrylhowe | s/no/now | 12:57 |
| *** jose4183 has joined #openstack-sdks | 13:03 | |
| *** jose4183 has quit IRC | 13:05 | |
| *** jose4183 has joined #openstack-sdks | 13:08 | |
| *** pm90__ has joined #openstack-sdks | 13:09 | |
| *** pm90_ has quit IRC | 13:12 | |
| *** thrash has quit IRC | 13:30 | |
| *** thrash has joined #openstack-sdks | 13:31 | |
| *** liangbo has quit IRC | 13:32 | |
| *** bknudson has joined #openstack-sdks | 13:38 | |
| *** zz_steveortiz is now known as steveortiz | 13:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/cliff: Updated from global requirements https://review.openstack.org/217142 | 13:53 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:05 | |
| *** petertr7 is now known as petertr7_away | 14:16 | |
| *** steveortiz is now known as zz_steveortiz | 14:38 | |
| *** petertr7_away is now known as petertr7 | 14:38 | |
| *** jose4183 has quit IRC | 14:42 | |
| *** jose4183 has joined #openstack-sdks | 14:45 | |
| *** jose4183 has quit IRC | 14:51 | |
| *** jose4183 has joined #openstack-sdks | 14:52 | |
| *** Qiming has quit IRC | 14:54 | |
| *** thrash has quit IRC | 15:08 | |
| *** thrash has joined #openstack-sdks | 15:09 | |
| *** boris-42 has quit IRC | 15:10 | |
| *** bnemec has joined #openstack-sdks | 15:17 | |
| *** GheRivero is now known as GheRivero_Out | 15:25 | |
| etoews | briancurtin: did you have any feels about the elif hasattr(data, 'read') in https://review.openstack.org/#/c/199318/ ? | 15:27 |
| briancurtin | etoews: looking at that right now, actually. i need a similar thing in object_store and am seeing if i can write a small util func to handle it once and for all. generally you don't want to do the hasattr thing -- just try to read and handle the exception if it can't be read, and fallback from there | 15:28 |
| etoews | briancurtin: but just trying to read seems dangerous "When size is omitted or negative, the entire contents of the file will be read and returned; it’s your problem if the file is twice as large as your machine’s memory." | 15:31 |
| briancurtin | etoews: eh, going to hold off on a common thing for right now as what i need in object_store needs a few turnable knobs and buttons | 15:31 |
| etoews | okay. then how about we leave the hasattr as is for now but update it when you've got something more generic? | 15:32 |
| briancurtin | etoews: with what you're working with in image, can it take segmented images for upload or does it need the whole thing at once? | 15:32 |
| briancurtin | etoews: use getattr(ob, attr, <False or None>) instead of hasattr | 15:33 |
| *** aufi has quit IRC | 15:33 | |
| briancurtin | in that case, None, and then check is not None | 15:34 |
| etoews | hmmm...not sure how it handles large image files yet. | 15:34 |
| etoews | need to do some testing. | 15:34 |
| briancurtin | etoews: for object_store.upload_object, i'm going to have to add some params somewhere for segment size and then open the file in those chunks and upload in those chunks | 15:35 |
| etoews | but i don't know how practical it would be to actually add a large image to the functional tests. | 15:35 |
| briancurtin | etoews: the util func I'm thinking of probably does the following order: try to .read() the `data` param as a file-like obj, try to open a file with the `data` param if os.path.exists works, or just use what is contained in the `data` param | 15:36 |
| etoews | briancurtin: this looks promising. http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads | 15:37 |
| etoews | i'm going to just see how well it works. it'll probably break my devstack though. | 15:37 |
| briancurtin | etoews: i wonder if the server is able to work with that | 15:38 |
| -openstackstatus- NOTICE: restarting gerrit due to a slow memory leak | 15:45 | |
| etoews | briancurtin: just uploaded a 327.5 MB file using only data=open('image-file', 'r') | 15:55 |
| briancurtin | etoews: is that big? :) (i don't know what constitutes a big image) | 15:55 |
| briancurtin | etoews: i have to do some work on the object_store one because it can't accept more than a 5GB file, so if you want more than that it needs segmentation (or if you *want* it segmented smaller than that) | 15:56 |
| etoews | no but it's a start. in my experience, svelte images are 1-2 GB but they can get up to 5 GB or larger. | 15:56 |
| briancurtin | etoews: also was this with the requests streaming? | 15:56 |
| etoews | this was with the requests streaming. it also supports chunking by giving it a generator. | 15:57 |
| *** liangbo has joined #openstack-sdks | 15:58 | |
| etoews | if we can avoid writing any "chunking" code, that's a huge plus. i've seen (not written) that code before and it's nasty. | 15:58 |
| briancurtin | etoews: i'm going to have to do something like it for object_store since you have to create the segments yourself, e.g., /container/object/00001.../container/object/00005 | 16:05 |
| etoews | ah yes. ick. once you've chunked it yourself, now i believe you're also responsible for efficiently getting all of those chunks to swift. e.g. multi-threading the upload for efficient bandwidth usage. | 16:08 |
| etoews | i'd advise talking to sigmavirus24 on this. he's probably got good advice on it and maybe there's even something in requests that will make your life easier. | 16:10 |
| *** petertr7 is now known as petertr7_away | 16:10 | |
| sigmavirus24 | what's up? | 16:10 |
| etoews | briancurtin is chunking | 16:10 |
| etoews | maybe more accurately...segmenting. | 16:10 |
| sigmavirus24 | so let me provide one warning when using a file with streaming | 16:11 |
| briancurtin | sigmavirus24: object_store large object segmentation. i think it's above requests | 16:11 |
| briancurtin | not using streaming | 16:11 |
| sigmavirus24 | we pass that file down to httplib and that reads 8192 byte chunks at a time | 16:11 |
| sigmavirus24 | so it's super slow | 16:11 |
| sigmavirus24 | briancurtin: okay | 16:11 |
| sigmavirus24 | yeah I think the way swift does it's segmenting is outside of requests | 16:12 |
| briancurtin | sigmavirus24: from what I can tell: take a big file, open segments, upload them with some ordered suffix, so /cont/ob/0001, /cont/ob/0002, etc | 16:12 |
| sigmavirus24 | For as sad as reading its code makes me, I would think you could extract the logic from swiftclient | 16:13 |
| sigmavirus24 | Fwiw, swiftclient will use threading (I think) to do concurrent uploads too if possible | 16:13 |
| sigmavirus24 | (to be clear threading is fine with http because threading works well with IO) | 16:13 |
| etoews | sigmavirus24: how does glance handle large file streams? say 5 GB and above? | 16:13 |
| briancurtin | yep, and we might have to implement some of that, although I'm not too sure where it fits in with what we're doing off the top of my head | 16:13 |
| briancurtin | (especially since what we have now is probably going away with KSA) | 16:14 |
| etoews | doh. i forgot about ksa. | 16:14 |
| sigmavirus24 | etoews: glance server or client? | 16:15 |
| etoews | server | 16:15 |
| sigmavirus24 | etoews: I forget the exact workflow but 5GB images work fine | 16:15 |
| sigmavirus24 | It's 30GB ones that cause problems with expired tokens etc | 16:15 |
| etoews | actually, knowing what the client does might be helpful too. | 16:15 |
| etoews | i just uploaded a 1.21 GB image using only data=open('image-file', 'r') successfully | 16:16 |
| sigmavirus24 | Also keep in mind if you're planning on streaming large things with requests, you may want something like https://github.com/sigmavirus24/requests-toolbelt/pull/84 | 16:16 |
| sigmavirus24 | etoews: yeah I bet that took a while though | 16:16 |
| etoews | yep | 16:16 |
| sigmavirus24 | Yeah that's because of the 8192 byte sized reads | 16:17 |
| etoews | ~15 min. | 16:17 |
| sigmavirus24 | yep | 16:17 |
| etoews | briancurtin: at this point i'm comfortable with the image upload code the way it is for handling large files. considering we're going to ksa, i don't think it's worthwhile to put any effort in to optimize it at this point in time. | 16:19 |
| briancurtin | etoews: yeah, makes sense to me. we can work on KSA when we need to tweak | 16:19 |
| terrylhowe | catching up on the backscroll, sounds reasonable etoews waiting on optimization | 16:20 |
| etoews | sigmavirus24: FastererererWrapper++ | 16:20 |
| etoews | sigmavirus24: thanks for the info. we'll probably revisit this w.r.t. requests in the future. | 16:21 |
| sigmavirus24 | etoews: sure | 16:23 |
| sigmavirus24 | Just a word of warning, even urllib3 behaves this way (because it all just gets passed to httplib which is awful) | 16:24 |
| etoews | sigmavirus24: the size of the reads isn't configurable in urllib/httplib? | 16:25 |
| sigmavirus24 | NOPE | 16:25 |
| sigmavirus24 | we don't use urllib either | 16:25 |
| etoews | wft :*( | 16:25 |
| sigmavirus24 | urllib3 is a separate library that we do use | 16:25 |
| sigmavirus24 | etoews: it's fine | 16:25 |
| sigmavirus24 | No one uploads big files with python | 16:25 |
| sigmavirus24 | Nope | 16:25 |
| sigmavirus24 | No one ever | 16:25 |
| sigmavirus24 | Not once has someone ever wanted to | 16:25 |
| etoews | why would anyone ever want to do that? | 16:26 |
| sigmavirus24 | What's funnier is that this person who has 2 commits to CPython was hired to work on httplib and hasn't done squat | 16:26 |
| sigmavirus24 | *hired by a company | 16:26 |
| etoews | i don't do it so no one else must do it. | 16:26 |
| openstackgerrit | Brian Curtin proposed stackforge/python-openstacksdk: object_store switch create/save to upload/download https://review.openstack.org/217293 | 16:26 |
| sigmavirus24 | I love people that get jobs based off projects they might become core developers of and then never do anything with | 16:26 |
| sigmavirus24 | It's borrowing against work for the community that never gets done | 16:26 |
| openstackgerrit | Everett Toews proposed stackforge/python-openstacksdk: Image import https://review.openstack.org/199318 | 16:30 |
| *** liangbo has quit IRC | 16:34 | |
| *** liangbo has joined #openstack-sdks | 16:37 | |
| *** trown is now known as trown|lunch | 16:53 | |
| *** petertr7_away is now known as petertr7 | 16:56 | |
| *** boris-42 has joined #openstack-sdks | 17:07 | |
| openstackgerrit | Merged stackforge/python-openstacksdk: Updated from global requirements https://review.openstack.org/217226 | 17:07 |
| *** pm90_ has joined #openstack-sdks | 17:09 | |
| *** liangbo has quit IRC | 17:09 | |
| *** jose4183 has quit IRC | 17:09 | |
| *** pm90_ has quit IRC | 17:10 | |
| *** petertr7 is now known as petertr7_away | 17:11 | |
| *** pm90__ has quit IRC | 17:11 | |
| *** jose4183 has joined #openstack-sdks | 17:13 | |
| *** e0ne has quit IRC | 17:15 | |
| openstackgerrit | Brian Curtin proposed stackforge/python-openstacksdk: object_store switch create/save to upload/download https://review.openstack.org/217293 | 17:18 |
| *** jose4183 has quit IRC | 17:18 | |
| *** jose4183 has joined #openstack-sdks | 17:19 | |
| *** zz_steveortiz is now known as steveortiz | 17:22 | |
| openstackgerrit | Merged openstack/python-openstackclient: default OS_VOLUME_API_VERSION to v2 https://review.openstack.org/216305 | 17:23 |
| *** jose4183 has quit IRC | 17:24 | |
| *** trown|lunch is now known as trown | 17:47 | |
| *** pm90_ has joined #openstack-sdks | 17:55 | |
| *** hitalia has joined #openstack-sdks | 18:08 | |
| *** openstackgerrit has quit IRC | 18:17 | |
| *** openstackgerrit has joined #openstack-sdks | 18:17 | |
| *** ameybhide has joined #openstack-sdks | 18:22 | |
| *** jose4183 has joined #openstack-sdks | 18:23 | |
| *** e0ne has joined #openstack-sdks | 18:25 | |
| *** jose4183 has quit IRC | 18:28 | |
| *** e0ne has quit IRC | 18:50 | |
| *** jose4183 has joined #openstack-sdks | 18:59 | |
| *** hitalia has quit IRC | 19:00 | |
| *** jose4183 has quit IRC | 19:04 | |
| *** jose4183 has joined #openstack-sdks | 19:04 | |
| *** e0ne has joined #openstack-sdks | 19:13 | |
| *** jose4183 has quit IRC | 19:18 | |
| *** jose4183 has joined #openstack-sdks | 19:19 | |
| *** jose4183 has quit IRC | 19:21 | |
| *** jose4183 has joined #openstack-sdks | 19:21 | |
| *** petertr7_away is now known as petertr7 | 19:23 | |
| *** jose4183 has quit IRC | 19:52 | |
| openstackgerrit | Terry Howe proposed stackforge/python-openstacksdk: Keystone auth integration https://review.openstack.org/186140 | 19:54 |
| *** jose4183 has joined #openstack-sdks | 20:18 | |
| *** petertr7 is now known as petertr7_away | 20:20 | |
| *** hitalia has joined #openstack-sdks | 20:26 | |
| *** e0ne has quit IRC | 20:28 | |
| *** etoews has quit IRC | 20:41 | |
| *** ameybhide has quit IRC | 20:55 | |
| *** openstackgerrit has quit IRC | 21:01 | |
| *** openstackgerrit has joined #openstack-sdks | 21:02 | |
| *** steveortiz is now known as zz_steveortiz | 21:02 | |
| *** jaosorior has quit IRC | 21:15 | |
| *** trown is now known as trown|outttypeww | 21:15 | |
| *** jamielennox has quit IRC | 21:36 | |
| *** jamielennox has joined #openstack-sdks | 21:38 | |
| *** stevemar has joined #openstack-sdks | 21:40 | |
| *** pm90__ has joined #openstack-sdks | 21:51 | |
| *** pm90__ has quit IRC | 21:51 | |
| *** pm90__ has joined #openstack-sdks | 21:52 | |
| *** pm90__ has quit IRC | 21:52 | |
| *** pm90_ has quit IRC | 21:55 | |
| *** jose4183 has quit IRC | 21:58 | |
| *** bknudson has quit IRC | 22:08 | |
| *** stevemar has quit IRC | 22:32 | |
| *** pm90_ has joined #openstack-sdks | 22:39 | |
| *** pm90__ has joined #openstack-sdks | 22:40 | |
| *** pm90_ has quit IRC | 22:41 | |
| *** pm90_ has joined #openstack-sdks | 22:44 | |
| *** pm90__ has quit IRC | 22:47 | |
| *** hitalia has quit IRC | 22:58 | |
| *** gildub has joined #openstack-sdks | 23:09 | |
| *** thrash is now known as thrash|g0ne | 23:20 | |
| *** Qiming has joined #openstack-sdks | 23:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!