torgomatic | notmyname: PUT calls broker.put_record(), which only sometimes calls _commit_puts() depending on file size | 00:00 |
---|---|---|
torgomatic | GET calls list_objects_iter calls _commit_puts_stale_ok, which tries to flush but doesn't complain if it can't get the lock | 00:01 |
notmyname | torgomatic: I don't know about your fancy "this is what the code does"; I'm just going off of what I saw on my SAIO ;-) | 00:02 |
torgomatic | lol | 00:03 |
mattoliverau | keep it up guys, I'm learning heaps without having to read the code :) | 00:03 |
*** occupant has quit IRC | 00:04 | |
*** occupant has joined #openstack-swift | 00:09 | |
zaitcev | sphinx.errors.ExtensionError: Could not import extension oslosphinx (exception: No module named oslosphinx) -- god dammit | 00:21 |
zaitcev | Swift succumbed too | 00:21 |
zaitcev | doc/source/conf.py: 'sphinx.ext.ifconfig', 'oslosphinx'] | 00:22 |
*** echevemaster has joined #openstack-swift | 00:28 | |
*** dmorita has joined #openstack-swift | 00:29 | |
*** rmcall has quit IRC | 00:30 | |
*** kyles_ne has quit IRC | 00:44 | |
*** tdasilva has joined #openstack-swift | 00:46 | |
*** mitz_ has joined #openstack-swift | 00:46 | |
*** StevenK_ is now known as StevenK | 00:51 | |
*** conor has joined #openstack-swift | 00:51 | |
clayg | zaitcev: mfw oslo -> http://memegenerator.net/instance/54370440 | 00:56 |
openstackgerrit | Clay Gerrard proposed a change to openstack/swift: wip: make /info configurable https://review.openstack.org/121723 | 01:01 |
clayg | umm... how do you mark changes wip now? | 01:02 |
clayg | nm, review, workflow, wip -1 | 01:04 |
*** addnull has joined #openstack-swift | 01:04 | |
*** conor has quit IRC | 01:21 | |
openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Let eventlet.wsgi.server log tracebacks when eventlet_debug is enabled https://review.openstack.org/105918 | 01:23 |
clayg | man I'm loving the abandon change alert | 01:23 |
*** gyee has quit IRC | 01:27 | |
*** miqui has quit IRC | 01:29 | |
mattoliverau | clayg: glad your finding it helpful :) | 01:32 |
*** occupant has quit IRC | 01:37 | |
*** occupant has joined #openstack-swift | 01:37 | |
openstackgerrit | Thiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver https://review.openstack.org/121728 | 01:39 |
*** tdasilva has quit IRC | 01:59 | |
*** haomaiw__ has quit IRC | 02:16 | |
*** haomaiwang has joined #openstack-swift | 02:17 | |
*** IgnacioCorderi has quit IRC | 02:26 | |
*** haomaiw__ has joined #openstack-swift | 02:33 | |
*** haomaiw__ has quit IRC | 02:33 | |
*** haomaiwang has quit IRC | 02:34 | |
*** haomaiw__ has joined #openstack-swift | 02:34 | |
*** haomaiwang has joined #openstack-swift | 02:34 | |
*** bkopilov has quit IRC | 02:37 | |
*** haomaiw__ has quit IRC | 02:38 | |
*** vu has joined #openstack-swift | 02:40 | |
*** vu has quit IRC | 02:41 | |
*** vu has joined #openstack-swift | 02:41 | |
briancline | notmyname: torgomatic: thanks for the discussion earlier - a human being oddly found its way to my desk, so I had to tend to that | 03:08 |
briancline | I think I found part of the issue. One of the 3 nodes storing that db had an extended network partition, during which there were numerous puts, gets, and deletes | 03:09 |
briancline | the kicker, however, is that it was that node's replication network connection that suffered the partition, not the proxy-facing network | 03:10 |
briancline | each of the three .pending's are empty, but it seems each .db reflects a different state | 03:12 |
briancline | so I guess my next step is figuring out how to get what appears to be a three-way split to reconcile, if that's even possible | 03:14 |
briancline | (I'd hate to cheat and replay the official sequence of requests) | 03:15 |
*** mrsnivvel has joined #openstack-swift | 03:29 | |
*** mahatic has joined #openstack-swift | 03:31 | |
*** haomaiw__ has joined #openstack-swift | 03:39 | |
*** haomaiwang has quit IRC | 03:40 | |
*** haomaiw__ has quit IRC | 03:41 | |
*** haomaiwa_ has joined #openstack-swift | 03:41 | |
*** bkopilov has joined #openstack-swift | 03:43 | |
*** bkopilov has quit IRC | 03:48 | |
*** haomaiwa_ has quit IRC | 03:50 | |
*** haomaiwang has joined #openstack-swift | 03:51 | |
*** haomaiwang has quit IRC | 03:58 | |
*** bkopilov has joined #openstack-swift | 04:00 | |
*** haomaiwang has joined #openstack-swift | 04:09 | |
notmyname | briancline: if gholt were still around, he's who I'd ask. he's done that sort of thing for cloud files before | 04:14 |
*** oomichi has joined #openstack-swift | 04:16 | |
briancline | notmyname: does he still frequent the channel? | 04:22 |
notmyname | briancline: doubt it | 04:23 |
notmyname | briancline: maybe dfg would be able to provide some pointers on getting DBs back in sync | 04:24 |
briancline | any pointers would be great - the container and its contents aren't important as the correct state is its contents should be fully deleted, it's more that I'd like to dive as deep as necessary to at least get it to a consistent state | 04:32 |
briancline | one of those things my brain can't let go :) | 04:32 |
notmyname | yup. I understand | 04:35 |
*** kyles_ne has joined #openstack-swift | 04:43 | |
*** kyles_ne has quit IRC | 04:46 | |
*** kyles_ne has joined #openstack-swift | 04:46 | |
*** kyles_ne has quit IRC | 04:51 | |
*** haomaiwang has quit IRC | 05:09 | |
*** zhiyan_ is now known as zhiyan | 05:10 | |
*** zaitcev has quit IRC | 05:22 | |
*** echevemaster has quit IRC | 05:23 | |
*** kopparam has joined #openstack-swift | 05:30 | |
*** ppai has joined #openstack-swift | 05:35 | |
*** RockKuo_Office has joined #openstack-swift | 05:46 | |
*** bkopilov has quit IRC | 05:47 | |
*** bkopilov has joined #openstack-swift | 06:03 | |
*** nshaikh has joined #openstack-swift | 06:30 | |
*** kopparam has quit IRC | 07:27 | |
*** geaaru has joined #openstack-swift | 07:33 | |
*** yuan has joined #openstack-swift | 07:40 | |
mattoliverau | I'm calling it a night, need to go cook some dinner. Have a great one all! | 07:45 |
*** homegrown has joined #openstack-swift | 08:02 | |
*** Dafna has joined #openstack-swift | 08:05 | |
*** ledeveloper has quit IRC | 08:16 | |
*** cschwede has joined #openstack-swift | 08:17 | |
*** aix has joined #openstack-swift | 08:19 | |
*** mkollaro has joined #openstack-swift | 09:07 | |
*** haomaiwang has joined #openstack-swift | 09:10 | |
*** RockKuo_Office has quit IRC | 09:27 | |
*** jasondotstar has quit IRC | 09:44 | |
hugokuo | notmyname: morning !!! | 10:32 |
*** addnull_ has joined #openstack-swift | 10:43 | |
*** cschwede has quit IRC | 10:45 | |
*** addnull has quit IRC | 10:46 | |
*** addnull_ has quit IRC | 10:47 | |
*** addnull has joined #openstack-swift | 10:48 | |
*** anticw has quit IRC | 10:50 | |
*** anticw has joined #openstack-swift | 10:50 | |
*** addnull has quit IRC | 11:00 | |
*** AbyssOne has quit IRC | 11:14 | |
*** AbyssOne has joined #openstack-swift | 11:16 | |
*** cschwede has joined #openstack-swift | 11:18 | |
*** anticw has quit IRC | 11:20 | |
*** zul has quit IRC | 11:21 | |
*** zul has joined #openstack-swift | 11:22 | |
*** smart_developer has quit IRC | 11:29 | |
*** smart_developer has joined #openstack-swift | 11:30 | |
*** mahatic has quit IRC | 11:35 | |
*** dmorita has quit IRC | 12:08 | |
*** nshaikh has quit IRC | 12:33 | |
*** vr1 has joined #openstack-swift | 12:44 | |
openstackgerrit | Christian Schwede proposed a change to openstack/swift: Return correct number of changed partitions https://review.openstack.org/120713 | 12:59 |
openstackgerrit | Alistair Coles proposed a change to openstack/swift: Early checks for data migration config and headers https://review.openstack.org/121831 | 13:24 |
acoles | tdasilva: ^^ | 13:29 |
*** tdasilva has joined #openstack-swift | 13:37 | |
*** judd7 has joined #openstack-swift | 13:37 | |
*** nshaikh has joined #openstack-swift | 13:39 | |
*** miqui has joined #openstack-swift | 13:42 | |
*** tdasilva has quit IRC | 13:42 | |
*** mkollaro has quit IRC | 13:43 | |
*** r-daneel__ has joined #openstack-swift | 13:48 | |
*** dmsimard_away is now known as dmsimard | 13:55 | |
*** oomichi has quit IRC | 13:59 | |
*** bkopilov has quit IRC | 14:00 | |
*** mkollaro has joined #openstack-swift | 14:00 | |
*** bkopilov has joined #openstack-swift | 14:14 | |
openstackgerrit | David Goetz proposed a change to openstack/swift: Small bug with targetted reload. https://review.openstack.org/121851 | 14:22 |
*** bkopilov has quit IRC | 14:22 | |
*** ZBhatti has joined #openstack-swift | 14:29 | |
openstackgerrit | A change was merged to openstack/swift: Treat 404s as 204 on object delete in proxy https://review.openstack.org/114120 | 14:29 |
openstackgerrit | A change was merged to openstack/python-swiftclient: Work toward Python 3.4 support and testing https://review.openstack.org/118807 | 14:30 |
openstackgerrit | A change was merged to openstack/swift: Fix delete versioning objects when previous is expired https://review.openstack.org/88204 | 14:35 |
smart_developer | What are some suggestions for tweaking the MSR register for performance optimization? | 14:38 |
*** ppai has quit IRC | 14:39 | |
smart_developer | or in general, what are some ways that you can tweak the kernel to optimize Swift performance? | 14:42 |
*** sandywalsh has joined #openstack-swift | 14:46 | |
*** Dafna has quit IRC | 14:57 | |
*** Dafna has joined #openstack-swift | 14:58 | |
openstackgerrit | David Goetz proposed a change to openstack/swift: Small bug with swift-init and targetted configs https://review.openstack.org/121851 | 14:58 |
openstackgerrit | Christian Schwede proposed a change to openstack/swift: Fix bug in swift-ring-builder list_parts https://review.openstack.org/121893 | 15:01 |
*** homegrown has quit IRC | 15:04 | |
*** ZBhatti has quit IRC | 15:18 | |
smart_developer | Or rather, does anyone know how to optimize the Swift configurations themselves, for performance? | 15:21 |
*** nshaikh has quit IRC | 15:26 | |
*** vu has quit IRC | 15:37 | |
openstackgerrit | Christian Schwede proposed a change to openstack/swift: Add containeralias middleware https://review.openstack.org/62494 | 15:38 |
*** ZBhatti has joined #openstack-swift | 15:48 | |
*** bkopilov has joined #openstack-swift | 15:50 | |
*** vr1 has quit IRC | 15:56 | |
*** Trixboxer has joined #openstack-swift | 15:59 | |
*** mahatic has joined #openstack-swift | 15:59 | |
*** cschwede has quit IRC | 16:00 | |
*** nshaikh has joined #openstack-swift | 16:14 | |
*** kyles_ne has joined #openstack-swift | 16:15 | |
*** mwstorer has joined #openstack-swift | 16:22 | |
*** kyles_ne has quit IRC | 16:31 | |
*** kyles_ne has joined #openstack-swift | 16:32 | |
*** bsdkurt has quit IRC | 16:33 | |
*** kyles_ne has quit IRC | 16:36 | |
*** kyles_ne has joined #openstack-swift | 16:36 | |
*** geaaru has quit IRC | 16:40 | |
*** tdasilva has joined #openstack-swift | 16:46 | |
*** zaitcev has joined #openstack-swift | 16:54 | |
*** ChanServ sets mode: +v zaitcev | 16:54 | |
openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Move multipart MIME parser into utils https://review.openstack.org/120652 | 17:11 |
*** tongli has joined #openstack-swift | 17:14 | |
*** nshaikh has quit IRC | 17:14 | |
openstackgerrit | Alistair Coles proposed a change to openstack/swift: Early checks for data migration config and headers https://review.openstack.org/121831 | 17:21 |
acoles | torgomatic: was that just rebase/conflict resolution to 120652 ? | 17:24 |
torgomatic | acoles: yeah, there was a really easy conflict with a recent formpost fix. no other code changes. | 17:25 |
acoles | torgomatic: cool | 17:25 |
*** kevinbenton has quit IRC | 17:27 | |
*** jdaggett has quit IRC | 17:28 | |
*** jdaggett has joined #openstack-swift | 17:28 | |
tdasilva | acoles: checking 121831 now | 17:29 |
tdasilva | acoles, thanks for your comments on 121728 | 17:29 |
acoles | tdasilva: ok i hope i didn't conflict with anything you did | 17:31 |
tdasilva | acoles: not at all...I like the direction it is going | 17:32 |
*** kevinbenton has joined #openstack-swift | 17:32 | |
acoles | tdasilva: good :) i'm done for today but will try to look more closely at your driver tomorrow | 17:33 |
tdasilva | acoles: ok, have a good night. I hope I can clean up some stuff there and add some comments in Gil's patch, but you explained things very well | 17:34 |
*** judd7 has quit IRC | 17:38 | |
*** bsdkurt has joined #openstack-swift | 17:41 | |
*** aix has quit IRC | 17:48 | |
openstackgerrit | A change was merged to openstack/swift: Allow filtering by region in swift-recon https://review.openstack.org/121634 | 18:02 |
*** nshaikh has joined #openstack-swift | 18:10 | |
smart_developer | If your middleware configurations are showing up in whenever you do " curl http://swift-proxy-ip:8080/info " does that mean that the list of middleware you've added to your proxy-server.conf pipeline is working? | 18:28 |
smart_developer | or do you need to further test each middleware and actually try to use each one of them? | 18:29 |
clayg | smart_developer: I'd recommend testing them ;) | 18:44 |
smart_developer | Is there a guide on how to test each of the Swift middleware that is available? | 18:45 |
*** NM has joined #openstack-swift | 18:47 | |
*** echevemaster has joined #openstack-swift | 18:48 | |
*** kevinc has joined #openstack-swift | 18:55 | |
*** Trixboxer has quit IRC | 18:57 | |
*** ZBhatti has quit IRC | 18:58 | |
*** ZBhatti has joined #openstack-swift | 18:59 | |
openstackgerrit | Christian Schwede proposed a change to openstack/swift: Fix bug in swift-ring-builder list_parts https://review.openstack.org/121893 | 19:02 |
*** kevinc has quit IRC | 19:07 | |
*** kyles_ne has quit IRC | 19:14 | |
*** kyles_ne has joined #openstack-swift | 19:14 | |
*** kyles_ne has quit IRC | 19:19 | |
openstackgerrit | Thiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver https://review.openstack.org/121728 | 19:28 |
*** Victor__ has joined #openstack-swift | 19:30 | |
Victor__ | Hi everyone. I'm studying how to deploy a reliable and fault tolerant swift cluster. How could I add a extra proxy node in a already deployed swift cluster? Thanks in advance. | 19:34 |
glange | Victor__: just put it behnind the load balancer in front of swift and when it is ready have the load balancer start sending it requests | 19:36 |
NM | Victor__: 1 - install swift 2 - copy the ring files 3 - have fun :) | 19:37 |
*** kyles_ne has joined #openstack-swift | 19:42 | |
Victor__ | Glange: Nice! Should I replace the endpoint of swift on Keystone? | 19:42 |
Victor__ | NM: Good. Should I add the new proxy nodes address on keystone catalog? | 19:43 |
NM | Victor__: usually all of your proxies should be behind a Load Balancer | 19:45 |
NM | In my case I have: http://myswift.mydomain.com pointing to all my proxy servers. And my keystone points to myswift.mydomain.com | 19:46 |
*** kevinc has joined #openstack-swift | 19:46 | |
glange | Victor__: typically the load balancer is your end point | 19:48 |
*** lpabon has joined #openstack-swift | 19:57 | |
Victor__ | NM, glange: Nice! Do I need to configure something in the proxies? | 19:59 |
*** fbo_ has quit IRC | 19:59 | |
NM | Victor__: Copy the ring files from other clusters. | 20:04 |
NM | Victor__: what about your memcached? Are you running then at the same machine as the proxies? If yes you should pay attention to its configuration. | 20:04 |
*** fbo has joined #openstack-swift | 20:05 | |
*** kevinc has quit IRC | 20:06 | |
*** kevinc has joined #openstack-swift | 20:07 | |
smart_developer | Is anyone able to share a performance-optimized proxy-server.conf, account-server.conf, container-server.conf, object-server.conf for a small, 3-5 node cluster? | 20:08 |
*** smart_developer has quit IRC | 20:08 | |
Victor__ | NM: Right now, my proxies are in different machines. Should I use only one memcache for both? | 20:09 |
*** smart_developer has joined #openstack-swift | 20:09 | |
*** fbo has quit IRC | 20:10 | |
glange | Victor__: typically you the proxies don't need different code or confs -- they should use the same memcache server pool | 20:10 |
smart_developer | Or for any cluster, if small is not available. | 20:12 |
Victor__ | glange: What is the memcached used for? | 20:13 |
glange | Victor__: caching certain kinds of information to lighten the req load on swift and sometimes things like auth | 20:14 |
glange | like some container information is cached | 20:14 |
*** nshaikh has quit IRC | 20:14 | |
glange | it speeds up requests and lightens the load | 20:15 |
Victor__ | glange, NM: Allright! Thank you so very much guys. You've helped me a lot. | 20:15 |
glange | Victor__: good luck! | 20:16 |
NM | Victor__: Anytime :) | 20:16 |
NM | smart_developer: I think one of the most importants tunning you can do you to set the number of workers correctly. And this number depends on the number of processors that you have. | 20:30 |
smart_developer | so, # of workers = # of processors ? | 20:38 |
smart_developer | ( NM ) | 20:39 |
*** fbo has joined #openstack-swift | 20:39 | |
smart_developer | Thanks in advance. | 20:40 |
openstackgerrit | Thiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver https://review.openstack.org/121728 | 20:41 |
mahatic | torgomatic, Does the patch now look ok? (https://review.openstack.org/#/c/119193/) | 20:42 |
*** dencaval has joined #openstack-swift | 20:48 | |
NM | smart_developer: It's a good start. But if you can, always run a benchmark test. | 20:50 |
smart_developer | NM : All right, so for my 3-node cluster I am running all the services (proxy, account, container, object) on each node. So node1, node2, node3, each have their own proxy-server, account-server, container-server, and object-server. | 20:52 |
smart_developer | NM : So for the number of workers that I specify in proxy-server.conf, account-server.conf, container-server.conf, and object-server.conf should add up (at least initially before benchmarking) to the number of processers that I have on that particular node? | 20:53 |
smart_developer | Thanks. | 20:53 |
smart_developer | Thanks in advance | 20:54 |
*** kyles_ne has quit IRC | 20:58 | |
*** kyles_ne has joined #openstack-swift | 20:59 | |
*** kyles_ne has quit IRC | 21:01 | |
*** kyles_ne has joined #openstack-swift | 21:01 | |
*** lpabon has quit IRC | 21:09 | |
*** smart_developer1 has joined #openstack-swift | 21:11 | |
*** smart_developer has quit IRC | 21:11 | |
*** marcusvrn has quit IRC | 21:11 | |
*** miqui has quit IRC | 21:23 | |
NM | smart_developer1: I'd use that approach | 21:24 |
smart_developer1 | NM : Okay. | 21:28 |
smart_developer1 | NM : But how would you distribute the # of processors to each service on that node? | 21:28 |
smart_developer1 | NM : Let's say that you have about ~30 processors. | 21:30 |
smart_developer1 | By processor, we mean individual cores as well, right? | 21:31 |
smart_developer1 | NM : So I guess technically you mean # of workers = roughly the # of cores ? | 21:31 |
NM | smart_developer1: Yes. I mean it. Is there anyway that you can separate the proxies from the ACO (Account/Container/Object) ? | 21:32 |
*** ZBhatti has quit IRC | 21:33 | |
smart_developer1 | NM : Unfortunately, that's not an option here. | 21:37 |
NM | smart_developer1: Ok. Usually proxies and object servers are more busy than account and container servers. I'd start with something like 10 Proxy, 10 object, 5 account and 5 container. | 21:37 |
NM | smart_developer1: As a smart developer that you are, you should now that there is no silver bullet :) | 21:38 |
smart_developer1 | NM : Thanks. But don't the account and container servers have to traverse through their DBs? | 21:38 |
smart_developer1 | NM : Which seems it would relatively need a lot of compute power? | 21:38 |
NM | What do you mean by that? Accounts and containers response might be cached on memcached. | 21:38 |
smart_developer1 | NM : I thought searching through a lot of database entries was a very computational intensive process ? | 21:39 |
*** morganfainberg is now known as CaptainMorgan | 21:39 | |
smart_developer1 | NM : Thus some people also use SSDs for their account and container services, while just HDDs for object services... | 21:39 |
*** CaptainMorgan is now known as morganfainberg | 21:39 | |
NM | smart_developer1: Yes, ssd are great solution. But they speed up the disk I/O. I'm not sure if more workers would be a good solution because you would have more concurrency. | 21:42 |
smart_developer1 | NM : and concurrency doesn't necessarily help with account/container DB reads, is what you're saying? | 21:46 |
smart_developer1 | NM : (in terms of reads-performance) | 21:47 |
smart_developer1 | searching/reads. | 21:47 |
NM | smart_developer1: Yes. I think that more process trying to read the same file wouldn't help the performance. And I'd apply the same thought to write. | 21:48 |
smart_developer1 | NM : Would that actually be more likely to *degrade* the performance? | 21:54 |
smart_developer1 | NM : if more processes are trying to read, or to write, to the same file | 21:55 |
NM | smart_developer1: I think so. | 21:56 |
*** NM has quit IRC | 22:01 | |
mattoliverau | Morning all | 22:06 |
*** judd7 has joined #openstack-swift | 22:07 | |
*** r-daneel__ has quit IRC | 22:09 | |
*** bnelson has quit IRC | 22:11 | |
*** dmsimard is now known as dmsimard_away | 22:12 | |
*** peluse_ has joined #openstack-swift | 22:18 | |
*** smart_developer1 has quit IRC | 22:20 | |
*** peluse_away has quit IRC | 22:21 | |
*** smart_developer has joined #openstack-swift | 22:21 | |
*** mahatic has quit IRC | 22:36 | |
*** r-daneel__ has joined #openstack-swift | 22:39 | |
*** r-daneel__ has quit IRC | 22:44 | |
*** tdasilva has quit IRC | 22:52 | |
*** r-daneel__ has joined #openstack-swift | 22:56 | |
*** acoles has quit IRC | 23:08 | |
*** tongli has quit IRC | 23:09 | |
*** acoles has joined #openstack-swift | 23:19 | |
*** ChanServ sets mode: +v acoles | 23:19 | |
openstackgerrit | Billy Olsen proposed a change to openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 23:20 |
*** kevinc has quit IRC | 23:28 | |
*** dmsimard_away is now known as dmsimard | 23:32 | |
*** Dafna has quit IRC | 23:33 | |
*** mkollaro1 has joined #openstack-swift | 23:33 | |
*** mkollaro has quit IRC | 23:33 | |
*** bill_az has joined #openstack-swift | 23:45 | |
*** ekarlso has quit IRC | 23:48 | |
*** ekarlso has joined #openstack-swift | 23:48 | |
*** yuan has quit IRC | 23:55 | |
*** yuan has joined #openstack-swift | 23:56 | |
*** cebruns_ has quit IRC | 23:57 | |
*** cebruns has joined #openstack-swift | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!