*** chlong has joined #openstack-swift | 00:20 | |
*** mat128 has joined #openstack-swift | 00:28 | |
*** tovin07_ has joined #openstack-swift | 00:41 | |
timburke | mattoliverau: fyi, i've got a patch that puts back a call to broker.get_info() in https://review.openstack.org/#/c/504981/1/swift/container/sharder.py -- maybe should have broke that out as its own thing, but the rest of the patch is fairly straight-forward so i figured... eh, why not? | 00:59 |
---|---|---|
patchbot | patch 504981 - swift (feature/deep) - Move sharding header translation to the proxy | 00:59 |
timburke | i think the if and comment will make me less likely to want to remove it again later :-) | 00:59 |
mattoliverau | lol | 00:59 |
mattoliverau | I'll take a look :) | 01:00 |
*** klrmn has quit IRC | 01:03 | |
timburke | if you merge it before acoles wakes up, he'll never need to figure out why hash_path is raising `TypeError: sequence item 0: expected string, NoneType found` ;-) | 01:03 |
timburke | also, i think that probe test should be ready to merge now. still not sure why our time mock bled over from container/sharder.py to common/db_replicator.py though | 01:06 |
timburke | maybe it has to do with us importing the time module in both, rather than importing just the time *function*? | 01:07 |
* timburke shrugs | 01:07 | |
*** Sukhdev has quit IRC | 01:12 | |
*** cshastri has joined #openstack-swift | 01:21 | |
*** itlinux has joined #openstack-swift | 01:48 | |
*** chlong has quit IRC | 02:03 | |
mattoliverau | timburke: oh yeah, you need to call get_info to get the .container and .account. that burnt me once :) | 02:18 |
*** itlinux has quit IRC | 02:37 | |
-openstackstatus- NOTICE: Gerrit is being restarted to feed its insatiable memory appetite | 02:41 | |
*** itlinux has joined #openstack-swift | 04:03 | |
*** mat128 has quit IRC | 04:10 | |
*** itlinux has quit IRC | 04:36 | |
*** cnf has joined #openstack-swift | 04:45 | |
*** mabrams has joined #openstack-swift | 04:54 | |
*** gyee has quit IRC | 04:58 | |
*** ChubYann has quit IRC | 05:52 | |
*** Jeffrey4l has quit IRC | 06:05 | |
*** mwheckmann has quit IRC | 06:05 | |
*** afazekas has quit IRC | 06:05 | |
*** dhellmann has quit IRC | 06:05 | |
*** _d34dh0r53_ has quit IRC | 06:05 | |
*** StevenK has quit IRC | 06:05 | |
*** dr_gogeta86 has quit IRC | 06:05 | |
*** tdasilva has quit IRC | 06:05 | |
*** acoles has quit IRC | 06:05 | |
*** d34dh0r53 has joined #openstack-swift | 06:05 | |
*** dhellmann has joined #openstack-swift | 06:05 | |
*** skudlik has joined #openstack-swift | 06:05 | |
*** Jeffrey4l has joined #openstack-swift | 06:05 | |
*** StevenK has joined #openstack-swift | 06:05 | |
*** afazekas has joined #openstack-swift | 06:05 | |
*** dr_gogeta86 has joined #openstack-swift | 06:05 | |
*** dr_gogeta86 has quit IRC | 06:05 | |
*** dr_gogeta86 has joined #openstack-swift | 06:05 | |
*** csmart has quit IRC | 06:09 | |
*** mwheckmann has joined #openstack-swift | 06:10 | |
*** tdasilva has joined #openstack-swift | 06:10 | |
*** acoles has joined #openstack-swift | 06:10 | |
*** barjavel.freenode.net sets mode: +v acoles | 06:10 | |
*** kei_yama has quit IRC | 06:34 | |
*** kei_yama has joined #openstack-swift | 06:43 | |
*** hseipp has joined #openstack-swift | 06:52 | |
*** hseipp has quit IRC | 06:52 | |
*** hseipp has joined #openstack-swift | 06:52 | |
*** hseipp has quit IRC | 06:53 | |
*** hseipp has joined #openstack-swift | 06:57 | |
*** hseipp has quit IRC | 06:57 | |
*** hseipp has joined #openstack-swift | 06:57 | |
*** hseipp has quit IRC | 06:57 | |
*** hseipp has joined #openstack-swift | 06:57 | |
*** rcernin has joined #openstack-swift | 07:18 | |
*** csmart has joined #openstack-swift | 07:19 | |
*** pcaruana has joined #openstack-swift | 07:22 | |
-openstackstatus- NOTICE: Post jobs are not executed currently, do not tag any releases | 07:24 | |
*** ChanServ changes topic to "Post jobs are not executed currently, do not tag any releases" | 07:24 | |
*** tesseract has joined #openstack-swift | 07:24 | |
*** cbartz has joined #openstack-swift | 07:25 | |
*** cschwede has joined #openstack-swift | 07:34 | |
*** ChanServ sets mode: +v cschwede | 07:34 | |
*** oshritf has joined #openstack-swift | 07:38 | |
openstackgerrit | Chenghui Yu proposed openstack/swift master: Update os-api-ref>=1.4.0 https://review.openstack.org/505095 | 07:41 |
*** mingyu has joined #openstack-swift | 07:42 | |
*** SkyRocknRoll has joined #openstack-swift | 07:45 | |
*** geaaru has joined #openstack-swift | 07:47 | |
acoles | good morning | 07:54 |
mahatic | acoles: o/ good morning | 07:58 |
acoles | hi mahatic | 07:58 |
acoles | lol mattoliverau +/-800 line patch to change the name from pivot :) | 07:59 |
acoles | ok, let's get that merged before bothering to push other patches. | 07:59 |
*** d0ugal has quit IRC | 08:01 | |
mattoliverau | acoles: I hope its right :P | 08:05 |
acoles | mattoliverau: it is forwards progress, thanks for doing it | 08:06 |
mattoliverau | acoles: seems I used the term pivot alot. I kept some instances, as a pivot being serching for a place in the data to split sounds right.. (in the doc glossary) but hit that last so maybe I didn't have to remove _all_ of them | 08:06 |
mattoliverau | also, not sure that made sense, /me might still be jetlagged :P | 08:07 |
acoles | mattoliverau: did you mean to use shardrange (no spaces) like here https://review.openstack.org/#/c/505097/2/swift/common/db_replicator.py@374 | 08:08 |
patchbot | patch 505097 - swift (feature/deep) - Stop using the term Pivot | 08:08 |
*** d0ugal has joined #openstack-swift | 08:09 | |
mattoliverau | I think whats suppose to read, the shard ranges stored in the container... | 08:11 |
acoles | mattoliverau: ok no worries I will fix up as I review | 08:12 |
*** mvk has quit IRC | 08:21 | |
*** mvk has joined #openstack-swift | 08:50 | |
mattoliverau | acoles: I was thinking maybe the main shard functions to test, misplassed objects, audit, meld and explode etc, for ease of testing could simply take a "job". Then when searching for jobs we just generate jobs and pass them to methods.. this would hopefully make it easier to write tests, remove dodginess like the self.node_idx and take us a step toward the concurrently or goal, in a sharer or replicator daemon. | 08:57 |
mattoliverau | *concurrency goal | 08:59 |
acoles | mattoliverau: yep, I was working on not hanging node_idx etc off self. That's going to need rebasing on the name change patch :) | 08:59 |
mattoliverau | I think we talk about just providing a broker, which could work too, but the job could also contain other things, like node is etc | 09:00 |
mattoliverau | Cool | 09:00 |
acoles | also was looking at moving the static methods to be broker methods?? | 09:00 |
acoles | so you could instantiate a broker and ask it for it's shard points/ranges | 09:01 |
mattoliverau | Cool, in my concurrent sharder tests I've been using job dicts which work clean things up nice. | 09:01 |
mattoliverau | Yeah, good idea. | 09:01 |
acoles | yeah that's a nice pattern too | 09:01 |
mattoliverau | The broker is shard aware, so makes sense | 09:01 |
acoles | right, seems like broker already has some shard aware stuff | 09:01 |
*** joeljwright has joined #openstack-swift | 09:09 | |
*** ChanServ sets mode: +v joeljwright | 09:09 | |
*** mvk has quit IRC | 09:32 | |
*** kei_yama has quit IRC | 09:32 | |
*** mvk has joined #openstack-swift | 09:45 | |
*** tovin07_ has quit IRC | 10:09 | |
*** SkyRocknRoll_ has joined #openstack-swift | 10:15 | |
*** skudlik has left #openstack-swift | 10:23 | |
*** joeljwright has quit IRC | 10:26 | |
*** joeljwright has joined #openstack-swift | 10:27 | |
*** ChanServ sets mode: +v joeljwright | 10:27 | |
*** SkyRocknRoll_ has quit IRC | 10:29 | |
*** mingyu has quit IRC | 10:50 | |
*** cshastri has quit IRC | 11:13 | |
*** SkyRocknRoll has quit IRC | 11:29 | |
*** ypbg has joined #openstack-swift | 11:30 | |
*** mingyu has joined #openstack-swift | 11:34 | |
*** abhitechie has joined #openstack-swift | 11:45 | |
*** saint_ has joined #openstack-swift | 11:53 | |
*** abhitechie has quit IRC | 11:57 | |
*** abhitechie has joined #openstack-swift | 11:57 | |
*** oshritf_ has joined #openstack-swift | 11:58 | |
*** oshritf has quit IRC | 12:01 | |
*** abhitechie has quit IRC | 12:10 | |
*** abhitechie has joined #openstack-swift | 12:10 | |
*** lucasxu has joined #openstack-swift | 12:15 | |
acoles | mattoliverau: fixed up and merged the name changes. I hope I correctly interpreted your intentions. | 12:18 |
*** MVenesio has joined #openstack-swift | 12:19 | |
acoles | timburke: rebased probe test on ^^ and merged | 12:19 |
acoles | torgomatic: note there has been a significant churn on feature/deep to rename pivots to shard ranges | 12:19 |
acoles | hmmm, this has merged https://review.openstack.org/#/c/504127/ but gerritbot is still not alerting for feature/deep | 12:20 |
patchbot | patch 504127 - openstack-infra/project-config - Update gerritbot branch alerts in #openstack-swift (MERGED) | 12:20 |
*** abhitechie has quit IRC | 12:26 | |
*** catintheroof has joined #openstack-swift | 12:49 | |
*** skudlik has joined #openstack-swift | 12:55 | |
*** mat128 has joined #openstack-swift | 12:58 | |
*** mabrams has quit IRC | 13:14 | |
*** chlong has joined #openstack-swift | 13:19 | |
*** _ix has joined #openstack-swift | 13:48 | |
_ix | Hey, good morning folks. I had a quick question about swift and splitting larger objects. We take some fairly large compressed db backups, and as a rule, we compress and split them. | 13:49 |
_ix | But, it looks like swift does splitting natively. Is the split step that we do necessary? | 13:49 |
*** skudlik has left #openstack-swift | 13:50 | |
_ix | Further, I don't know how it works exactly, but, if swift does the splitting, does that mean we don't have to download the segments, cat them together, and uncompress; instead, we just do a swift download for the object and it takes care of all of the stitching for us? | 13:50 |
tdasilva | _ix: the answer is kind of twofold: the swift api (server side) has support for large file objects, but the client is responsible for splitting up the segments and uploading. Read here for more: https://docs.openstack.org/swift/latest/api/large_objects.html | 13:57 |
_ix | Thanks, tdasilva. | 13:58 |
tdasilva | _ix: the swift *client*, has support for splitting the objects automatically for you, so that's what you might be seeing... | 13:58 |
tdasilva | _ix: but you don't have to use the python swiftclient per se, you could be sending requests to the API in a different way, so it's really up to your client application how to send those segments to the server | 13:59 |
tdasilva | _ix: see here for swift client upload options: https://docs.openstack.org/python-swiftclient/latest/cli/index.html#swift-upload | 14:00 |
*** SkyRocknRoll has joined #openstack-swift | 14:22 | |
mwheckmann | Has anyone seen a lot of "Too many open files" errors with the object-server after switching to "servers_per_port" config? i.e with the a port per disk in the ring. | 14:25 |
mwheckmann | This is happening to us under load testing since moving over to per disk ports. | 14:26 |
mwheckmann | systemd sets a 4k file descriptor limit on the object server. | 14:26 |
mwheckmann | At first glance, it seems to trigger a little bit quicker on nodes w/ less CPU cores available | 14:27 |
mwheckmann | Running 2.13.0 BTW | 14:27 |
tdasilva | mwheckmann: some of the guys on the PST might have more ops experience running servers_per_port | 14:34 |
mwheckmann | tdasilva: sorry, what do you mean by PST? The timezone? | 14:43 |
tdasilva | mwheckmann: sorry, yeah, PST timezone. I'd ask the question again in a couple of hours | 14:43 |
mwheckmann | will do. thanks for the tip | 14:44 |
*** cbartz has quit IRC | 15:00 | |
*** cbartz has joined #openstack-swift | 15:00 | |
*** joeljwright has quit IRC | 15:00 | |
*** joeljwright has joined #openstack-swift | 15:05 | |
*** ChanServ sets mode: +v joeljwright | 15:05 | |
*** gyee has joined #openstack-swift | 15:05 | |
*** MVenesio has quit IRC | 15:30 | |
*** chsc has joined #openstack-swift | 15:38 | |
*** oshritf_ has quit IRC | 15:42 | |
*** chsc has quit IRC | 15:45 | |
*** joeljwright has quit IRC | 15:59 | |
*** joeljwright has joined #openstack-swift | 16:02 | |
*** ChanServ sets mode: +v joeljwright | 16:02 | |
*** silor has joined #openstack-swift | 16:03 | |
*** hamzy has quit IRC | 16:11 | |
*** itlinux has joined #openstack-swift | 16:11 | |
*** hseipp has quit IRC | 16:19 | |
*** geaaru has quit IRC | 16:22 | |
timburke | good morning | 16:28 |
*** mat128 has quit IRC | 16:32 | |
acoles | timburke: o/ ... got some sharding patches merged today | 16:36 |
*** joeljwright has quit IRC | 16:36 | |
timburke | acoles: just taking a look at the dashboard :) | 16:37 |
timburke | i see there are a few that need rebasing; i think i'll start there | 16:37 |
acoles | timburke: inevitable merge conflicts with the big rename | 16:38 |
*** pcaruana has quit IRC | 16:38 | |
timburke | thanks for getting the probe test in. i keep carrying it around so i can do an integration test when i've got another patch i'm working on | 16:38 |
acoles | timburke: right! I was about to cherry-pick it for same reason :) so just got it merged | 16:39 |
acoles | until we get unit test coverage up it's the best check we have | 16:39 |
timburke | yup. still need to come up with some other edge cases for it... and think about whether the skip limit of 100 is too low | 16:40 |
acoles | I added notes here for anyone wanting to run the sharding probe test https://etherpad.openstack.org/p/deep-containers | 16:41 |
acoles | the sample internal-client.conf does not have account_autocreate | 16:41 |
timburke | yeah... i ran into a bit of trouble with that yesterday, too. remind me: when did we introduce internal-client.conf? was that for EC stuff (so we expect accounts/containers to already exist)? | 16:43 |
acoles | timburke: I asked in -infra about the gerritbot alerts for the new feature branches - sounds like they're working through gerrit upgrade fallout before taking in new config | 16:43 |
timburke | ah, yeah, that seems fair | 16:43 |
*** chsc has joined #openstack-swift | 16:44 | |
*** chsc has joined #openstack-swift | 16:44 | |
acoles | so the dashboard is the place to go, and trello | 16:44 |
*** chsc has quit IRC | 16:44 | |
acoles | timburke: internal-client.conf came in with a container sync change to use it | 16:45 |
timburke | huh. so i guess you can't sync to an account unless it already exists... i suppose that's reasonable... | 16:46 |
acoles | I think it is only used for the get side of sync, was needed to support EC gets | 16:50 |
*** mat128 has joined #openstack-swift | 16:51 | |
*** klrmn has joined #openstack-swift | 16:51 | |
*** rcernin has quit IRC | 16:52 | |
*** tesseract has quit IRC | 16:52 | |
*** mvk has quit IRC | 17:01 | |
*** cbartz has quit IRC | 17:06 | |
acoles | timburke: what should I be looking to review/merge tomorrow? | 17:17 |
acoles | listing middleware on master perhaps? | 17:17 |
timburke | good plan. https://review.openstack.org/#/c/503938/ would also be worht a peek once i've got it rebased | 17:18 |
patchbot | patch 503938 - swift (feature/deep) - Always use node_idx 0 as scanner | 17:18 |
acoles | timburke: ok | 17:19 |
timburke | i'm trying to remember if there was anything else we wanted to land on master to use for sharding... | 17:20 |
acoles | timburke: might be worth getting this done on master in case we end up making changes that would conflict...but it needs more than review :/ https://review.openstack.org/#/c/448480/ | 17:23 |
patchbot | patch 448480 - swift - DB replicator cleanup | 17:23 |
timburke | will take a look. https://review.openstack.org/#/c/441520/ would probably be good to have, too, but for feature/s3api | 17:24 |
patchbot | patch 441520 - swift - Use swift3's check_signature function | 17:24 |
timburke | oh! we should maybe think about https://review.openstack.org/#/c/464084/ | 17:25 |
patchbot | patch 464084 - swift - Apply remote metadata in _handle_sync_response | 17:25 |
*** chsc has joined #openstack-swift | 17:25 | |
*** chsc has joined #openstack-swift | 17:25 | |
acoles | k | 17:25 |
timburke | it does a two-way sync on the metadata before we start shipping object rows around, which seems like something we may want to start doing... | 17:27 |
mwheckmann | Has anyone seen a lot of "Too many open files" errors with the object-server after switching to "servers_per_port" config? i.e with the a port per disk in the ring. | 17:46 |
mwheckmann | This is happening to us under load testing since moving over to per disk ports. | 17:46 |
mwheckmann | I asked this morning (EST) but since there are more folks active in the channel now | 17:47 |
*** MVenesio has joined #openstack-swift | 17:49 | |
mwheckmann | lsof output shows the object server processes w/ a bunch of open pipes as well as some AF_ALG sockets. The latter seems odd since those are for the crypto API which we are not using. | 17:51 |
mwheckmann | this is on 2.13.0 BTW | 17:51 |
*** mvk has joined #openstack-swift | 17:53 | |
*** ChubYann has joined #openstack-swift | 17:59 | |
*** saint_ has quit IRC | 18:08 | |
*** itlinux has quit IRC | 18:22 | |
*** silor has quit IRC | 18:40 | |
*** mingyu has quit IRC | 18:45 | |
*** mingyu has joined #openstack-swift | 18:46 | |
*** mingyu has quit IRC | 18:47 | |
*** mingyu has joined #openstack-swift | 18:48 | |
*** cschwede has quit IRC | 18:55 | |
*** itlinux has joined #openstack-swift | 18:59 | |
*** Sukhdev has joined #openstack-swift | 19:17 | |
openstackgerrit | Merged openstack/swift master: Remove all post_as_copy related code and configes https://review.openstack.org/504069 | 19:19 |
*** lucasxu has quit IRC | 19:26 | |
*** mat128 has quit IRC | 19:46 | |
*** rcernin has joined #openstack-swift | 20:14 | |
*** _ix has quit IRC | 20:18 | |
-openstackstatus- NOTICE: Zuul and Gerrit are being restarted to address issues discovered with the Gerrit 2.13 upgrade. review.openstack.org will be inaccessible for a few minutes while we make these changes. Currently running jobs will be restarted for you once Zuul and Gerrit are running again. | 20:27 | |
*** _ix has joined #openstack-swift | 20:28 | |
*** Sukhdev has quit IRC | 20:41 | |
*** catintheroof has quit IRC | 20:50 | |
*** catintheroof has joined #openstack-swift | 20:50 | |
*** catintheroof has quit IRC | 20:51 | |
*** itlinux has quit IRC | 20:56 | |
*** itlinux has joined #openstack-swift | 20:57 | |
*** itlinux has quit IRC | 20:58 | |
*** itlinux has joined #openstack-swift | 20:59 | |
*** _ix has quit IRC | 21:21 | |
*** itlinux has quit IRC | 21:21 | |
*** itlinux has joined #openstack-swift | 21:25 | |
*** klrmn1 has joined #openstack-swift | 21:29 | |
*** klrmn has quit IRC | 21:31 | |
*** jistr has quit IRC | 21:41 | |
*** Sukhdev has joined #openstack-swift | 21:50 | |
*** jistr has joined #openstack-swift | 22:01 | |
*** Sukhdev has quit IRC | 22:12 | |
*** MVenesio has quit IRC | 22:12 | |
*** chlong has quit IRC | 22:21 | |
-openstackstatus- NOTICE: Gerrit is being restarted to address some final issues, review.openstack.org will be inaccessible for a few minutes while we restart | 22:34 | |
*** vint_bra has quit IRC | 22:45 | |
*** catintheroof has joined #openstack-swift | 22:45 | |
*** openstackgerrit has quit IRC | 22:47 | |
*** itlinux has quit IRC | 22:49 | |
*** rcernin has quit IRC | 23:11 | |
mattoliverau | Morning | 23:18 |
timburke | good morning mattoliverau | 23:18 |
timburke | lots of sharding stuff in the gate | 23:19 |
timburke | poor -infra's been overworked with the zuul/gerrit issues :-( | 23:19 |
*** openstackgerrit has joined #openstack-swift | 23:19 | |
openstackgerrit | Tim Burke proposed openstack/swift feature/deep: Reduce the number of queries in get_shard_range https://review.openstack.org/505439 | 23:19 |
timburke | yay! we get our branch notifications now! | 23:19 |
*** hoonetorg has quit IRC | 23:23 | |
*** kei_yama has joined #openstack-swift | 23:26 | |
mattoliverau | mwheckmann: in 2.15.0+ we new change the default eventlet threadpool size, because of the amount of threads servers_per_port where using. This might have something to do with your problem. see: https://github.com/openstack/swift/blob/2.15.0/swift/obj/server.py#L202-L228 | 23:33 |
*** chsc has quit IRC | 23:33 | |
mattoliverau | *we changed the default.. | 23:33 |
mattoliverau | timburke: nice! | 23:34 |
timburke | looks like p 505321 is going to hit a conflict once p 505329 lands -- but with all the patches in flight, i'm inclined to let acoles handle the rebase after things have settled a bit | 23:34 |
patchbot | https://review.openstack.org/#/c/505321/ - swift (feature/deep) - remove node-id, node_idx and part instance vars in... | 23:34 |
patchbot | https://review.openstack.org/#/c/505329/ - swift (feature/deep) - Remove some instance vars from _one_shard_cycle | 23:34 |
mattoliverau | yeah, I see a bunch of gerrit messages in backlog. as always I suspect some teathing issues with the new verison. | 23:35 |
mattoliverau | timburke: wow, you both have been busy! | 23:35 |
*** hoonetorg has joined #openstack-swift | 23:36 | |
clarkb | email sucks in the new version | 23:39 |
clarkb | we are trying to debug it | 23:39 |
mattoliverau | bugger. | 23:39 |
timburke | clarkb: no worries man, thanks for all the hard work! | 23:39 |
*** ChanServ changes topic to "Queens PTG planning https://etherpad.openstack.org/p/swift-ptg-queens | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Review Dashboard: http://not.mn/reviews.html" | 23:41 | |
-openstackstatus- NOTICE: Gerrit is once again part of normal puppet config management. Problems with Gerrit gitweb links and Zuul post jobs have been addressed. We currently cannot create new gerrit projects (fixes in progress) and email sending is slow (being debugged). | 23:41 | |
mattoliverau | I wish the 'f' button worked in the new version liked it worked a few versions ago (open the list of changed files). | 23:41 |
mattoliverau | timburke: FYI I hope to get to some shard work (reviewing and dev) later today, but need to work on some suse tasks first :( | 23:42 |
timburke | no worries. acoles and i can bounce between each other for a while -- we'll be sure to raise up anything that seems like it really needs your attention | 23:44 |
mattoliverau | timburke: thanks man | 23:44 |
*** Sukhdev has joined #openstack-swift | 23:46 | |
openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Symlink implementation. https://review.openstack.org/232162 | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!