*** haomaiwang has quit IRC | 00:01 | |
*** haomaiwang has joined #openstack-swift | 00:02 | |
*** darrenc_afk is now known as darrenc | 00:14 | |
*** jasondotstar has quit IRC | 00:23 | |
*** dmorita has joined #openstack-swift | 00:25 | |
*** marcusvrn has quit IRC | 00:25 | |
*** tellesnobrega has joined #openstack-swift | 00:33 | |
*** tellesnobrega has quit IRC | 00:34 | |
*** h00327910__ has joined #openstack-swift | 00:58 | |
*** jasondotstar has joined #openstack-swift | 00:58 | |
*** haomaiwang has quit IRC | 01:01 | |
*** haomaiwang has joined #openstack-swift | 01:02 | |
*** kei_yama has quit IRC | 01:02 | |
*** kei_yama has joined #openstack-swift | 01:03 | |
*** janonymous_ has joined #openstack-swift | 01:14 | |
*** chlong has joined #openstack-swift | 01:29 | |
*** ccavanna has joined #openstack-swift | 01:30 | |
*** ccavanna has quit IRC | 01:33 | |
*** jasondotstar has quit IRC | 01:35 | |
flwang1 | ho: still around? | 01:45 |
---|---|---|
flwang1 | clayg: ping | 01:45 |
flwang1 | anybody around? | 01:59 |
ho | flwang!: yes | 01:59 |
ho | sorry s/flwang!/flwang1/ | 01:59 |
flwang1 | ho: cooool | 01:59 |
flwang1 | i just give it a try for my 1st use case | 02:00 |
flwang1 | i mean add a role which has the swift only access | 02:00 |
flwang1 | it 'works' | 02:00 |
flwang1 | but the problem is, swift will take all the roles in operator_roles as an admin | 02:00 |
*** haomaiwang has quit IRC | 02:01 | |
flwang1 | that said, the container acl won't work for this role | 02:01 |
*** haomaiwang has joined #openstack-swift | 02:02 | |
*** Kennan has quit IRC | 02:02 | |
*** Kennan has joined #openstack-swift | 02:02 | |
ho | flwang1: i understand. anonymous access with acl is a possibility | 02:04 |
flwang1 | ho: so is it possible to create a swift-only role, and make it honour container ACL? | 02:04 |
ho | flwang1: let me summarize your situation. wait a minute. | 02:05 |
flwang1 | ho: ok | 02:05 |
*** annegentle has quit IRC | 02:06 | |
ho | flwang1: you created two roles for swift. one is for read, other one is for write. then you configured them in operator_roles in proxy-server.conf | 02:07 |
janonymous_ | o/ | 02:08 |
flwang1 | ho: no, for now, I just created one role in Keystone, named swift_only, and i add it for "operator_roles" in proxy-server.conf | 02:08 |
flwang1 | because all the other openstack components don't know this new role, so it's swift only | 02:09 |
ho | flwang1: then you access to a container which allows read only access with a user who has a role for write. | 02:09 |
ho | flwang1: but the user will be allowed to access because the role in operator_roles. | 02:10 |
flwang1 | ho: i found all the user has 'swift-only' role can delete any object | 02:10 |
flwang1 | ho: yep | 02:10 |
*** kota_ has joined #openstack-swift | 02:10 | |
*** ChanServ sets mode: +v kota_ | 02:10 | |
flwang1 | so i'm trying to figure out if there is a way to create a role in keystone and configure it in swift, which can still honour the container ACL | 02:11 |
flwang1 | that's my current question | 02:11 |
mattoliverau | flwang1: anyone user in keystone with a role defined in operator_roles, will be an admin for there own account. | 02:11 |
flwang1 | mattoliverau: yep, i just realized that | 02:11 |
mattoliverau | flwang1: so the ACL wont work when accessing there own account | 02:11 |
flwang1 | mattoliverau: so my new question is "so i'm trying to figure out if there is a way to create a role in keystone and configure it in swift, which can still honour the container ACL" | 02:11 |
mattoliverau | flwang1: if that user was to access another account, or its another user in keystone, then the ACL will work. | 02:12 |
hrou | flwang1: Just create another tenant in keystone => different account in swift, and try to access that from the original tenant/account, you won't be able to without ACL | 02:13 |
mattoliverau | flwang1: what hrou said ^ :) | 02:13 |
hrou | Which is what mattoliverau was saying too, I just reworded it ; ) | 02:14 |
flwang1 | hrou: any way we don't have to create a new tenant? | 02:14 |
flwang1 | because it will may the billing process hard | 02:14 |
mattoliverau | you can create a user in keystone that doesn't have an role in operator_roles, then the ACLs will work on them. | 02:14 |
flwang1 | mattoliverau: but what's the role of the new user? | 02:15 |
hrou | Ah but without a new tenant, your going to be using the same account in swift, or try to change the role. | 02:15 |
flwang1 | ok, guys, let me repeat my user case | 02:15 |
*** nakagawamsa has joined #openstack-swift | 02:15 | |
mattoliverau | flwang1: so long as its a valid keystone user (so the token can be verified) and no roles that are in operator_roles, then anything | 02:15 |
flwang1 | 1. we want a swift only role, all the users with that role can only access swift, no other openstack components | 02:16 |
flwang1 | 2. based on #1, we would like to set read only access for some containers | 02:16 |
*** janonymous_ has quit IRC | 02:17 | |
flwang1 | mattoliverau: ok | 02:17 |
ho | flwang1: you create userA who has access role and userB who doesn't have access role in a project. you use userB for access and configure the userB info on a container. | 02:17 |
flwang1 | ho: access role = swift_only role? | 02:20 |
ho | flwang1: yes. | 02:21 |
ho | flwang1: the result said swift can't distinguish users' access by acl if the users are configured in operator-roles. | 02:21 |
flwang1 | ho: so seems we don't have to add the swift_only role in to the proxy-server.conf, right? | 02:24 |
mattoliverau | flwang1: so until ho's patch merges, to give someone a read/write account they must have a role in operator_roles (and access to others via ACLs). Everyoneelse in keystone can have access to different containers, etc, by ACLs. | 02:25 |
ho | flwang1: i said so. | 02:25 |
flwang1 | mattoliverau: i'm confused now | 02:26 |
mattoliverau | flwang1: you'll need to create containers and set the ACLs, so you need someone to have an account. | 02:26 |
flwang1 | mattoliverau: based on my test with the devstack master branch | 02:26 |
flwang1 | anyway, I will do the test again, unless i missed something. but before that, i'm going to write down my steps, so that you guys can review it | 02:27 |
mattoliverau | good idea, maybe we're all getting confused ;) | 02:27 |
flwang1 | 1. create role 'swift_only' in keystone | 02:28 |
flwang1 | 2. add the role in operator_role of proxy-server.conf, and restart proxy-server service | 02:28 |
flwang1 | 3. keystone user-create --name user_A --tenant demo | 02:30 |
flwang1 | 4. keystone user-create --name user_B --tenant demo | 02:30 |
*** bill_az has quit IRC | 02:30 | |
flwang1 | 5. keystone user-role-add --user user_A --tenant demo --role swiftoperator | 02:31 |
*** jasondotstar has joined #openstack-swift | 02:31 | |
flwang1 | 6. keystone user-role-add --user user_B --tenant demo --role swiftoperator | 02:31 |
flwang1 | i'm using swiftoperator instead of 'swift_only' sorry :) | 02:31 |
mattoliverau | flwang1: dont you want to add them to swift_only | 02:31 |
mattoliverau | lol | 02:31 |
flwang1 | hah | 02:31 |
mattoliverau | flwang1: but you only want to add one of them to swift_only | 02:31 |
flwang1 | mattoliverau: really? | 02:32 |
flwang1 | so I don't have to add user_B for the role, right? | 02:32 |
ho | flwang1: you dont need step 6 | 02:33 |
mattoliverau | flwang1: well if you want to test ACLs | 02:33 |
flwang1 | mattoliverau: ok, until now, what i can see from keystone for user_B is like this | 02:33 |
flwang1 | http://paste.openstack.org/show/406331/ | 02:34 |
mattoliverau | because user_A will create a container and set the ACLs for suer_B. then user_B can access it | 02:34 |
flwang1 | it has a ROLE '_member_' | 02:34 |
flwang1 | so that means user_B can play with the other openstack components | 02:34 |
flwang1 | i think user_A has the similar issue | 02:35 |
*** jrichli has joined #openstack-swift | 02:35 | |
*** yuanzz has quit IRC | 02:37 | |
flwang1 | now i'm going to remove the _member_ role from user_A and user_B, any concern? guys? | 02:37 |
*** yuanzz has joined #openstack-swift | 02:37 | |
ho | flwang1: dont need to do it. i think swift doesn't care | 02:37 |
flwang1 | ho: yep, i believe swift doesn't care, but the other components care | 02:38 |
flwang1 | that's my basic line | 02:38 |
ho | flwang1: i see | 02:38 |
flwang1 | we need a role which is swift only, that means the user with that role can't talk with the other components | 02:38 |
flwang1 | does that make any sense? | 02:38 |
flwang1 | we have several customers requiring that | 02:39 |
ho | flwang1: make sense for me. you mange roles for each component. | 02:41 |
flwang1 | the result is shit | 02:41 |
*** jasondotstar has quit IRC | 02:41 | |
flwang1 | for now, user_A only has one role, swiftoperator, but it can login, and create instance :( | 02:41 |
flwang1 | anybody know what i missed? | 02:42 |
flwang1 | seems by default, the policy of openstack components give a default user too much permissions | 02:42 |
ho | flwang1: what is login? you use horizon something? | 02:43 |
flwang1 | ho: horizon | 02:43 |
flwang1 | i think command line will do the same thing | 02:43 |
flwang1 | and i just checked the policy.json of nova, i got "compute:create": "", | 02:43 |
flwang1 | i think it means any user can create instance | 02:44 |
ho | flwang1: i'm not sure but it's better to re-configure __menber__ on userA for horizon :) | 02:44 |
flwang1 | ho: __member__ has been removed from user_A | 02:44 |
flwang1 | now user_A only has the swiftoperator role | 02:44 |
ho | flwang1: yeah, but swift can work with the user but other component not right? | 02:46 |
flwang1 | ho: i haven't try swift side :) | 02:46 |
flwang1 | let's do it now | 02:46 |
flwang1 | i will figure out how to keep the base line | 02:46 |
ho | flwang1: better to separator the problem :-) | 02:46 |
openstackgerrit | Tim Burke proposed openstack/swift: Improve content negotiation https://review.openstack.org/207276 | 02:47 |
flwang1 | actually, i confirmed my way with some keystone cores, they told me it works, but seems no | 02:47 |
flwang1 | ho: let's rock on | 02:47 |
*** changbl has quit IRC | 02:47 | |
flwang1 | ho: now i'm going to create container_demo with user_A | 02:47 |
*** annegentle has joined #openstack-swift | 02:50 | |
flwang1 | swift post --header 'x-container-read: AUTH_user_b' container_C | 02:50 |
flwang1 | is this the right command i should use? | 02:50 |
ho | flwang1: you will create container_c? | 02:51 |
flwang1 | i have created | 02:51 |
flwang1 | and i'm going to set its acl | 02:52 |
flwang1 | with x-container-read | 02:52 |
ho | flwang1: 'x-container-read: <project_id for demo>:<user_id for user_b> is better | 02:52 |
flwang1 | cool | 02:53 |
flwang1 | ho: http://paste.openstack.org/show/406332/ | 02:54 |
flwang1 | it should be user_B, i have fixed it | 02:55 |
*** breitz has quit IRC | 02:55 | |
ho | flwang1: one question you specify id for project and user. i see "demo:user_b" for read-acl | 02:56 |
flwang1 | ho: oh, ok | 02:57 |
ho | flwang1: both cases, name and id, work for your test but better to use id for keystone v3 | 02:58 |
flwang1 | ho: ok | 02:59 |
*** haomaiwang has quit IRC | 03:01 | |
*** haomaiwang has joined #openstack-swift | 03:01 | |
openstackgerrit | Tim Burke proposed openstack/swift: Improve content negotiation https://review.openstack.org/207276 | 03:01 |
flwang1 | ho: yep, it woks | 03:01 |
flwang1 | so in my previous test, the key step is i shouldn't add role 'swiftoperator' for user_B | 03:02 |
*** SkyRocknRoll has quit IRC | 03:02 | |
ho | flwang1: i think so. the access to be authorized by acl in swift. | 03:03 |
flwang1 | ho: and I think we even don't need add the 'swiftoperator' into proxy-server.conf | 03:04 |
flwang1 | but there is another problem | 03:04 |
ho | flwang1: yep! | 03:04 |
flwang1 | from the swift side, we still can't only accept a role access swfit | 03:05 |
flwang1 | because we don't have the policy.json support | 03:05 |
flwang1 | so we have to 'ask' the other components reject to accept a specific role, we named it as 'swiftonly' or something like that | 03:06 |
flwang1 | right? | 03:06 |
*** changbl has joined #openstack-swift | 03:06 | |
ho | flwang1: i have same understanding with you now. | 03:07 |
flwang1 | ho: cool, happy to see we're on the same page | 03:07 |
*** breitz has joined #openstack-swift | 03:07 | |
flwang1 | ho: btw, do you think it's possible to hack something in keystone to achieve that? | 03:07 |
flwang1 | s/keystone/swift | 03:09 |
*** sanchitmalhotra has joined #openstack-swift | 03:09 | |
ho | flwang1: sorry achieve that means policy support? | 03:09 |
flwang1 | ho: we need it in juno :( | 03:12 |
flwang1 | so if there is no good way to achieve that, we may have to hack it a bit | 03:12 |
ho | flwang1: commit to upstream then backport to juno. the change of current code is almost keystoneauth.py so it's possible i think :-) | 03:14 |
flwang1 | ho: ok, will see. | 03:14 |
ho | i have a business trip this afternoon so i will out of office soon. thanks for the info (what you want to do). it's useful for me. thanks! | 03:14 |
flwang1 | ho: thank you soooooooooooooooooo much for your support on my question | 03:15 |
flwang1 | have a good trip | 03:15 |
*** annegentle has quit IRC | 03:20 | |
*** kei_yama has quit IRC | 03:21 | |
*** kei_yama has joined #openstack-swift | 03:21 | |
*** annegentle has joined #openstack-swift | 03:26 | |
*** annegentle has quit IRC | 03:26 | |
*** sanchitmalhotra1 has joined #openstack-swift | 03:29 | |
*** sanchitmalhotra has quit IRC | 03:31 | |
notmyname | hello, again | 03:33 |
*** annegentle has joined #openstack-swift | 03:40 | |
*** jamielennox|away is now known as jamielennox | 03:47 | |
janonymous | Hi, Please vote on : https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/Presentation/4570 , Thanks in advance :) | 03:50 |
*** esker has joined #openstack-swift | 03:53 | |
*** sanchitmalhotra has joined #openstack-swift | 03:59 | |
*** haomaiwang has quit IRC | 04:01 | |
*** sanchitmalhotra1 has quit IRC | 04:01 | |
*** 16WAAEKLZ has joined #openstack-swift | 04:02 | |
*** ppai has joined #openstack-swift | 04:04 | |
*** aluria has quit IRC | 04:05 | |
*** tamizh_g_ has joined #openstack-swift | 04:09 | |
*** jrichli has quit IRC | 04:14 | |
*** proteusguy has quit IRC | 04:16 | |
mattoliverau | notmyname: hi again! | 04:19 |
clayg | oh my good ness | 04:19 |
mattoliverau | clayg: whats up? | 04:19 |
clayg | mattoliverau: no i was saw a bunch of scrollback | 04:20 |
notmyname | this was my view tonight https://twitter.com/notmyname/status/626588976753721345 | 04:20 |
mattoliverau | notmyname: wow, hard life your leading | 04:21 |
notmyname | lol | 04:21 |
notmyname | it's a really incredible view | 04:22 |
portante | would that be mount hood? | 04:22 |
mattoliverau | it really is | 04:22 |
notmyname | mt ranier | 04:22 |
portante | active volcano,right? | 04:22 |
notmyname | not yet ;-) | 04:22 |
clayg | notmyname: don't get dead | 04:23 |
notmyname | super clear today and as the sun was starting to go down the colors really popped | 04:23 |
portante | "Geologists consider this mountain to be an 'episodically active' volcano, meaning one that will erupt again some time in the future even though it may be quiet now. Mount Rainier is the tallest volcano and fifth highest peak in the contiguous United States." | 04:23 |
notmyname | clayg: I'll do my best | 04:23 |
mattoliverau | Looks a bit like fuji-san (mount fuji) | 04:23 |
portante | may you have no episodes | 04:24 |
notmyname | portante: if you want to be really freaked out, read http://www.newyorker.com/magazine/2015/07/20/the-really-big-one | 04:24 |
mattoliverau | may you be up wind if it does (and far enough away) | 04:25 |
portante | notmyname: okay, I really really love New England | 04:28 |
notmyname | lol | 04:28 |
*** ccavanna has joined #openstack-swift | 04:30 | |
notmyname | ok, so what do I need to look at in swift tonight before going to bed? | 04:30 |
notmyname | ah, ccavanna just joined. his patch would be easy to review | 04:30 |
ccavanna | hi. | 04:30 |
ccavanna | Yes, I was hoping for that :-) | 04:30 |
mattoliverau | ccavanna: o/ | 04:30 |
ccavanna | Actually, I have two patches. | 04:30 |
ccavanna | One for documentation, another for the improvements to statsD. | 04:31 |
ccavanna | And I wanted to ask for some suggestions on what I could review myself. | 04:31 |
*** blmartin has joined #openstack-swift | 04:32 | |
notmyname | ccavanna: do you see the "review dashboard" link in the channel topic? | 04:33 |
ccavanna | Now that you mention it, yes. | 04:33 |
notmyname | ccavanna: when you open that, you'll see a bunch of swift-related patches. the top section there is "starred patches". those are the priority patches to review | 04:33 |
*** jamielennox is now known as jamielennox|away | 04:33 | |
ccavanna | Great. Thanks. | 04:34 |
openstackgerrit | John Dickinson proposed openstack/swift: Swift documentation for first-time contributors. https://review.openstack.org/206861 | 04:40 |
notmyname | ccavanna: it looks great | 04:40 |
notmyname | I took the RDO link fix that tdasilva had and rolled it in | 04:41 |
notmyname | and approved | 04:42 |
*** sakaYK has joined #openstack-swift | 04:42 | |
ccavanna | Great, thank you. | 04:43 |
ccavanna | One question, how do I move forward my other change? | 04:43 |
ccavanna | I addressed all the comments as well. | 04:43 |
ccavanna | Do I just wait? | 04:44 |
*** annegentle has quit IRC | 04:44 | |
notmyname | yup | 04:44 |
notmyname | and if/when you don't see anything on it, you come in here and bug people ;-) | 04:44 |
ccavanna | notmyname: That was the dilemma, how much "bugging" was reasonable :-) | 04:46 |
clayg | ccavanna: if you don't make it to targeted it will sorta roll around | 04:48 |
clayg | like if you jump in channel every day and say "hey can someone look at xyz" that will normally spur someone who is feeling particularlly guilty about being behind | 04:49 |
clayg | if someone leaves a review and you address *their* comments in particular - it's not really "bugging" if you say "can you validate that I addressed your concern" | 04:49 |
clayg | it's just a friendly and helpful reminder | 04:49 |
clayg | which we all need from time to time | 04:49 |
* clayg hopes i'm not setting myself up to review ccavanna's patch tonite | 04:50 | |
ccavanna | Well ... I can be very convincing ... and since you offered to ... :-) but fortunately for you it is late in Toronto. | 04:52 |
*** darrenc is now known as darrenc_afk | 04:56 | |
*** blmartin has quit IRC | 04:58 | |
ccavanna | Well, in case somebody (more awake than I am) sees this ... | 04:58 |
ccavanna | Can anyone review https://review.openstack.org/#/c/202657/ ? | 04:58 |
*** tamizh_g_ has quit IRC | 04:58 | |
ccavanna | It's a proposed change to improve statistics in Graphite. | 04:59 |
mattoliverau | notmyname: if you still want another patch to review, patch 195131 is important for my fellow rackspace private cloud guys. (they've poked me). | 04:59 |
patchbot | mattoliverau: https://review.openstack.org/#/c/195131/ | 04:59 |
ccavanna | clayg: thanks for the tips. I'm still getting used to the dynamic. | 04:59 |
notmyname | mattoliverau: ah, thanks for the info on that. (to be fair, I'm not going to be reviewing any keystone stuff tonight--too tired to do the necessary setup to test it) | 05:00 |
mattoliverau | ccavanna: I'll take a look, I'm suppose to be more awake.. but not really firing on all cylinders :) | 05:00 |
mattoliverau | lol | 05:00 |
mattoliverau | fair enough :) | 05:00 |
*** 16WAAEKLZ has quit IRC | 05:01 | |
ccavanna | mattoliverau: me neither, long week + it's late. | 05:01 |
mattoliverau | ccavanna: well its only 3pm here | 05:01 |
*** haomaiwang has joined #openstack-swift | 05:02 | |
clayg | this seems right, but really needs some functional testing. | 05:04 |
clayg | w/c | 05:04 |
ccavanna | mattoliverau: I don't want to look at my watch ... :-) | 05:04 |
*** flwang1 has quit IRC | 05:04 | |
ccavanna | Anyway, I'm really going to sleep. Up early tomorrow. On course all day. Thanks for the tips. | 05:05 |
clayg | mattoliverau: does devstack setup v3 and domains and stuff? | 05:05 |
ccavanna | I'll start reviewing one patch. | 05:05 |
mattoliverau | ccavanna: night | 05:05 |
*** tamizh_g_ has joined #openstack-swift | 05:08 | |
mattoliverau | clayg: devstack.. *shudder* I'm not sure, I just set up my own keystone servers. (yay public cloud), which I re-use for testing. | 05:09 |
notmyname | mattoliverau: have you talked to alex or lana lately? there was talk in vancouver about reviewing the swift docs and getting a plan for improving them, but I haven't seen anything | 05:10 |
clayg | do you have a playbook or something that will setup install keystone - or you just set it up once and then use it? | 05:10 |
clayg | whoa - double barrell questions | 05:11 |
hrou | clayg yep ! Fairly sure it does, you just need to change openrc to use the V3 end points, I'll give it a go. | 05:11 |
clayg | hrou: whoa - awesome - I think | 05:12 |
mattoliverau | clayg: dplph had a github repo what set up keystone (but it is old) so used that and updated it a little to make it work. I'll clone it in my github and push up my fixes so it can be used to set up keystone ready to play with swift. | 05:13 |
*** sakaYK has quit IRC | 05:14 | |
clayg | mattoliverau: sure - could work | 05:15 |
hrou | mattoliverau, that'd be great, seems a lot of folks start off with swift SAIO, and it's nicer just to add keystone. | 05:15 |
openstackgerrit | Charles Hsu proposed openstack/swift: Remove error_suppression_interval, error_suppression_limit options. https://review.openstack.org/207296 | 05:15 |
hrou | In theory this should do it but I haven't tried it: https://github.com/Open-I-Beam/swift-install | 05:15 |
clayg | charz: remove options!? that's not nearly as fun as adding them! | 05:15 |
mattoliverau | notmyname: alex is holidaying in vietnam.. that's probably the delay | 05:16 |
notmyname | ok | 05:16 |
mattoliverau | I can poke lana tho ;) | 05:16 |
notmyname | this "holiday" concept. what is that? seems that all the non-US people seem to think it's a thing | 05:16 |
notmyname | and clayg and I are here after 10pm | 05:16 |
notmyname | (and I'm technically on half-day PTO this week) ;-) | 05:17 |
clayg | notmyname: we're the weird ones tho - the rest of the world is normal | 05:17 |
mattoliverau | lol, yeah you guys are crazy | 05:17 |
mattoliverau | we Aussies love our holidays | 05:17 |
notmyname | mattoliverau: but it's my only chance to talk to you | 05:17 |
* notmyname is sad that his work hours rarely overlap with cschwede these days | 05:17 | |
charz | clayg: That's in account-server.conf-sample. I think that's belong to proxy-server. | 05:18 |
clayg | charz: yeah i think so too - are their other references - man pages, docs? | 05:19 |
clayg | they're in the proxy conf example yeah? | 05:19 |
charz | clayg: yes, they're in the proxy conf, and deployment docs is good. | 05:19 |
clayg | charz: did you see my notes on the .durable bug thread? | 05:20 |
clayg | well... *your* the one taking notes - I'm just throwing out ideas from the side lines :\ | 05:20 |
charz | clayg: yeah, I saw that. | 05:20 |
clayg | charz: you buying it? | 05:20 |
charz | clayg: yes, I'm thinking how to repro that in a small cluster even VM. | 05:21 |
hrou | clayg, yep just replace OS_AUTH_URL=*/v3 and OS_IDENTITY_API_VERSION=v3 . Of course the keystone CLI doesn't support v2, only the openstack CLI does. | 05:22 |
notmyname | charz: looks like those options are also in man/account-server.conf.5 and need to be removed there too...or I could just fix the patch | 05:22 |
charz | clayg: first, we need to figure out why the .durables missing in high concurrency or load, and able to reproduce that. | 05:23 |
charz | notmyname: I can do that. | 05:23 |
notmyname | charz: no, you do the EC stuff with clayg. that's more important ;-) | 05:23 |
notmyname | I've already got this open | 05:23 |
hrou | * keystone CLI only supports v2. | 05:23 |
charz | notmyname: k | 05:23 |
clayg | charz: you want me to fix the man page - or do you wanna fix it and I'll +2 :P | 05:24 |
clayg | charz: for patch 207296 | 05:24 |
patchbot | clayg: https://review.openstack.org/#/c/207296/ | 05:24 |
clayg | notmyname: do you have any idea how to build manpages? there's all that weird syntax - every time I touch one I try to copy whats near by - but I have no idea | 05:24 |
charz | clayg: I do that. | 05:24 |
clayg | hrou: nice! | 05:25 |
clayg | notmyname: oh you beat me too it | 05:25 |
notmyname | clayg: nope. I do the same | 05:25 |
clayg | notmyname: how did they even get there!? | 05:25 |
clayg | notmyname: marcelo? | 05:25 |
notmyname | yup :-) | 05:25 |
clayg | nice | 05:25 |
clayg | what was his handle... btorch? | 05:26 |
openstackgerrit | John Dickinson proposed openstack/swift: Remove error_suppression_interval, error_suppression_limit options. https://review.openstack.org/207296 | 05:26 |
notmyname | yes | 05:26 |
clayg | redbo: what ever happened to btorch | 05:26 |
notmyname | fortunately in this case it was just deleting some parts of the man pages | 05:26 |
notmyname | it's really easy to delete stuff | 05:26 |
charz | notmyname: you're winner. :) | 05:26 |
clayg | notmyname: also - correction - charz is not doing "EC stuff with clayg" - charz is doing "critical EC testing" - clayg is goofing off - as usual | 05:26 |
*** SkyRocknRoll has joined #openstack-swift | 05:26 | |
notmyname | :-) | 05:27 |
notmyname | I just didn't want to distract the conversation with this. bigger fish, ya know? | 05:27 |
clayg | BIG OL FREAKING BASS BABY! | 05:27 |
charz | clayg: I'll try to test in with benchmark tool on VMs and disable reconstructors. | 05:27 |
*** zaitcev has quit IRC | 05:28 | |
clayg | charz: is that our best bet? did we alreay repurpose everyhting in the lab? I had assumed those 5 nodes were still available to you. | 05:28 |
clayg | maybe it's cheap and easy to reproduce on vms - in which case we'll do that | 05:29 |
mattoliverau | notmyname: well in a week and a bit well all be locking in an airconditioned room in Texas ;) (sorry for late reply got stuck on a call). | 05:29 |
clayg | I could probably crank up some knobs on my swift-bench config until my saio starts dropping .durables | 05:29 |
charz | clayg: yeah, if I can reproduce in VMs, you can fix it in a easy way. is it correct? | 05:30 |
notmyname | clayg: the 5 nodes are back in as gate checks (and charz is adding another for functests w/ default EC) right now | 05:30 |
clayg | umm... yeah - let's go with that | 05:30 |
clayg | well.. | 05:30 |
notmyname | so they could be used for the EC stuff again, but they aren't set up for that currently | 05:30 |
clayg | i mean I really just want full transaction log for one of those requests that resulted in a frag with no durable | 05:31 |
clayg | notmyname: charz: ok sure np | 05:31 |
notmyname | charz: any chance we could pull it from the archived logs? I'm not sure if we have enough to find it | 05:32 |
clayg | notmyname: i was thinking you'd have to work your way up from a hashdir with no .durable -> swift-object-info -> name -> log -> transaction_id | 05:32 |
clayg | you know - like you do | 05:32 |
notmyname | as one does | 05:33 |
charz | notmyname: I'll try, but no guarantee. Maybe I'll turn qacluster to test this if I need. | 05:34 |
clayg | charz: do what you think is best | 05:34 |
notmyname | charz: no, we'd have to start with data in the cluster. no need to start with logs to guess at something like this | 05:34 |
redbo | clayg: He's still around. He moved to austin and works remote. | 05:34 |
clayg | redbo: nice! does he know how man pages work? can he come to the hack-a-thon and do a brown bag. you're up late. | 05:35 |
*** kei_yama_ has joined #openstack-swift | 05:36 | |
*** kei_yama has quit IRC | 05:36 | |
redbo | It's not late yet. I'll be up late here in a few hours. | 05:36 |
clayg | well look at that - a GET that 404's has a body (html w/e) and a content-length (obviously) but if you make the same request with the verb HEAD - you get a different content-length (0 this time, obviously) | 05:37 |
charz | notmyname: got it. | 05:38 |
clayg | i guess the rule about GET's and HEAD's having the same headers only applies to 200 series responses | 05:38 |
redbo | isn't that a SHOULD, not a MUST? | 05:39 |
*** darrenc_afk is now known as darrenc | 05:40 | |
clayg | redbo: having the same headers - yeah that's a should - but I don't think it applies to 404's? or yeah - maybe it just doesn't matter | 05:47 |
notmyname | it's late for me. I'm going to bed. talk to you tomorrow | 05:48 |
*** ppai has quit IRC | 05:51 | |
openstackgerrit | Azhagu Selvan SP proposed openstack/swift: Respect 'Accept' header in error responses https://review.openstack.org/204196 | 05:54 |
*** tamizh_g_ has quit IRC | 05:57 | |
*** tamizh_g_ has joined #openstack-swift | 06:00 | |
*** haomaiwang has quit IRC | 06:01 | |
*** jith_ has quit IRC | 06:01 | |
*** haomaiwang has joined #openstack-swift | 06:02 | |
*** Kennan2 has joined #openstack-swift | 06:03 | |
*** Kennan has quit IRC | 06:03 | |
*** ppai has joined #openstack-swift | 06:05 | |
clayg | kota_: looking at the fix content-lenght on 416 for ec | 06:06 |
clayg | kota_: working on a test to describe teh failure i'm seeing | 06:07 |
*** hrou has quit IRC | 06:09 | |
clayg | did we ever get to run functests against non-default policy or do we still just change the default policy? | 06:13 |
*** esker has quit IRC | 06:13 | |
clayg | ok SWIFT_TEST_POLICY=ec seems to do the trick | 06:18 |
*** sakaYK has joined #openstack-swift | 06:25 | |
kota_ | clayg: I noticed your comment now. | 06:26 |
kota_ | clayg: night man | 06:26 |
kota_ | oh yeah, the trick recently landed on master. | 06:29 |
kota_ | i guess, we did but not sure. ok let's run functional agaisnt to ec now. | 06:30 |
kota_ | or if you have failure tests already, please feel free to comment on gerrit. I wanna try to make sure also in my env. | 06:33 |
*** sanchitmalhotra1 has joined #openstack-swift | 06:43 | |
*** david-lyle has quit IRC | 06:43 | |
*** sanchitmalhotra has quit IRC | 06:45 | |
clayg | kota_: oh hi :) | 06:47 |
*** david-lyle has joined #openstack-swift | 06:49 | |
kota_ | i just was reading versioned_writes middleware. | 06:50 |
kota_ | in paralell, I launched an all-in-one with vagrant at AWS for the functional checking :P | 06:51 |
kota_ | clayg: and hi :) (might be too late) | 06:52 |
*** bkopilov has quit IRC | 06:52 | |
clayg | kota_: nice ;) | 06:53 |
clayg | kota_: do you have ... oh AWS + vagrant!? you're in crazy town man! | 06:53 |
kota_ | yey | 06:55 |
kota_ | clayg: based on your vagrant-all-in-one | 06:55 |
clayg | heh, i've never tried the vagrant cloud stuff - very cool | 06:55 |
*** bkopilov has joined #openstack-swift | 06:55 | |
kota_ | clayg: here, https://github.com/bloodeagle40234/vagrant-swift-all-in-one/tree/aws | 06:56 |
kota_ | clayg: note I didn't make enough documentation for that :( | 06:57 |
clayg | hey - it has a commit message | 06:57 |
kota_ | ya, just memo for me :P | 06:58 |
clayg | kota_: looks pretty sweet - i bet we could get that baked into the mainline with just a few tweeks and some extra localrc options? | 07:00 |
clayg | neway - yeah versioned middleware - good call | 07:00 |
*** haomaiwang has quit IRC | 07:01 | |
clayg | I'm fartin' around with haypo's patch now - let me know if you have some thoughts on the 416 ec range etag thing (i left you a comment to ponder) | 07:01 |
kota_ | which one? | 07:01 |
kota_ | mine? | 07:01 |
*** haomaiwang has joined #openstack-swift | 07:02 | |
kota_ | oh, you mean you moved from the 416 thing to haypos | 07:02 |
*** mahatic has joined #openstack-swift | 07:02 | |
clayg | kota_: yessir | 07:03 |
clayg | kota_: I'm in pep8 crazy town now! | 07:03 |
kota_ | clayg: oic, I think so too, that's crazy town looking at today's meeting. | 07:04 |
clayg | yeah it was pretty nuts | 07:05 |
*** rledisez has joined #openstack-swift | 07:10 | |
haypo | hello | 07:17 |
kota_ | haypo: hello | 07:20 |
haypo | does swift juno support Python 2.6? | 07:22 |
haypo | hum, i still see python26 on https://review.openstack.org/#/c/207132/ ("Fixes for mock 1.1" for juno) | 07:22 |
haypo | oh, i remember that mock 1.1 dropped python 2.6 support, but i came back later :) | 07:23 |
haypo | so it's ok, nevermind ;) | 07:23 |
*** mahatic has quit IRC | 07:23 | |
kota_ | i think so. python2.6 support should drop after Kilo release. | 07:24 |
kota_ | oh, interesting for patch 207132 | 07:24 |
patchbot | kota_: https://review.openstack.org/#/c/207132/ | 07:24 |
*** chlong has quit IRC | 07:25 | |
haypo | kota_: "python26: SUCCESS", it's ok ;) | 07:25 |
openstackgerrit | Ondrej Novy proposed openstack/python-swiftclient: Test auth params together with --help option. https://review.openstack.org/207322 | 07:25 |
kota_ | lol | 07:26 |
kota_ | haypo: sounds good. | 07:26 |
haypo | kota_: yeah, i reviewed my own patch, it looks good to me :-D | 07:27 |
haypo | kota_: (well, it's a backport made by notmyname, and juno is different) | 07:27 |
*** ho has quit IRC | 07:28 | |
clayg | so is jchroll not the guy for pep8 anymore? PyCQA? | 07:29 |
openstackgerrit | Victor Stinner proposed openstack/swift: Fix pep8 E265 warning of hacking 0.10 https://review.openstack.org/207242 | 07:34 |
openstackgerrit | Victor Stinner proposed openstack/swift: Fix warning pep8 E128 warning of hacking 0.10 https://review.openstack.org/207243 | 07:34 |
openstackgerrit | Victor Stinner proposed openstack/swift: Update hacking to 0.10.0 https://review.openstack.org/205977 | 07:34 |
openstackgerrit | Victor Stinner proposed openstack/swift: Fix pep8 E warning for hacking 0.10 https://review.openstack.org/207237 | 07:34 |
haypo | clayg: you are right for "# ####" and E265: https://review.openstack.org/207242 -- i modified my patch | 07:35 |
onovy | hi, just quick review, someone pls? :) https://review.openstack.org/#/c/207322/ | 07:35 |
clayg | haypo: yeah but I +2'd already - now I gotta go back and do it again - i was hoping to sleep tonite | 07:36 |
kota_ | wow, I'm running out of time to stay my office, see you again. | 07:36 |
haypo | clayg: it's the morning for me :) | 07:37 |
haypo | clayg: next time, approve directly the patch :-D | 07:38 |
*** ppai has quit IRC | 07:38 | |
haypo | clayg: (thanks for reviewing these stuff) | 07:38 |
haypo | clayg: what's your timezone? (where do you live?) | 07:38 |
haypo | i live at the south of france (CEST) | 07:39 |
onovy | CEST and morning here too :) | 07:39 |
haypo | i'm asking because i remember someone saying that the swift meeting was at 02:00pm if i recall correctly | 07:40 |
haypo | i hate timezones :) | 07:40 |
haypo | 21:22:13 <peluse_> Im' tired too - its 2:21pm here :) | 07:41 |
clayg | notmyname: acoles_away: if I get frustrated and start +A'ing hypos stuff you guys can kick me out of the club | 07:41 |
*** kota_ has quit IRC | 07:41 | |
clayg | i'ts 7:50 UTC here same as everywhere else | 07:41 |
clayg | my residence is in SF, CA, USA - I don't keep very normalized hours | 07:42 |
haypo | ah wait, pm if before noon. i also hate am/pm, sorry :) | 07:42 |
haypo | clayg: haha | 07:42 |
-openstackstatus- NOTICE: Our CI system is broken again today, jobs are not getting processed at all. | 07:42 | |
*** ChanServ changes topic to "Our CI system is broken again today, jobs are not getting processed at all." | 07:42 | |
*** flwang1 has joined #openstack-swift | 07:46 | |
openstackgerrit | Ondrej Novy proposed openstack/python-swiftclient: Test auth params together with --help option. https://review.openstack.org/207322 | 07:49 |
*** flwang1 has quit IRC | 07:50 | |
-openstackstatus- NOTICE: CI system is broken and very far behind. Please do not approve any changes for a while. | 07:52 | |
*** ChanServ changes topic to "CI system is broken and very far behind. Please do not approve any changes for a while." | 07:52 | |
*** ppai has joined #openstack-swift | 07:52 | |
*** geaaru has joined #openstack-swift | 07:54 | |
*** jordanP has joined #openstack-swift | 07:55 | |
openstackgerrit | Ondrej Novy proposed openstack/python-swiftclient: Block comment PEP8 fix. https://review.openstack.org/205849 | 07:56 |
*** jamielennox|away is now known as jamielennox | 07:57 | |
*** tamizh_g_ has quit IRC | 08:00 | |
*** haomaiwang has quit IRC | 08:01 | |
*** haomaiwang has joined #openstack-swift | 08:02 | |
*** ppai has quit IRC | 08:05 | |
*** tamizh_g_ has joined #openstack-swift | 08:07 | |
*** ho has joined #openstack-swift | 08:11 | |
*** acoles_away is now known as acoles | 08:15 | |
acoles | clayg: np i'm going to look at them again today | 08:19 |
*** ppai has joined #openstack-swift | 08:19 | |
acoles | haypo: ack will review today. thanks again for the work. | 08:19 |
haypo | acoles: no problem | 08:20 |
haypo | acoles: i'm sorry for my behaviour, i put too much energy on python3, i'm getting tired. i'm in holiday in two days, i want to reach the milestone of a voting py34 gate for swift :-p | 08:21 |
acoles | haypo: no need for apologies! sounds like a good time for a holiday :) | 08:23 |
*** SkyRocknRoll has quit IRC | 08:25 | |
*** jistr has joined #openstack-swift | 08:25 | |
haypo | acoles: it takes me extra efforts to justify my work, and acecpt that other developers want to modify the code differently than me :) | 08:25 |
clayg | haypo: acoles puts up with me - so you're probably fine | 08:25 |
haypo | acoles: i only fixed E checks, 2 F checks and many H checks are still ignored. are you fine with that? | 08:26 |
haypo | acoles: i don't know if the 2 F checks are new or not | 08:27 |
acoles | haypo: i expect i will be fine with that | 08:27 |
acoles | ;) | 08:27 |
acoles | clayg: up late again? | 08:28 |
haypo | (note: it looks like the CI is sick, the infra team asked to "not approve any changes for a while" :-p) | 08:29 |
acoles | haypo: if you have N swift developers in a room you will get at least N+1 opinions :P | 08:29 |
haypo | acoles: lol | 08:29 |
acoles | haypo: and I'm probably responsible for the extra 1 | 08:29 |
haypo | i may propose a patch to sort and group imports. when i had to add six imports, it took me a while to find where to put it | 08:30 |
haypo | but i know that some people hate coding styles :) | 08:33 |
*** ppai has quit IRC | 08:45 | |
*** aix has joined #openstack-swift | 08:47 | |
*** ppai has joined #openstack-swift | 08:57 | |
*** sasi has joined #openstack-swift | 08:57 | |
*** sasi has left #openstack-swift | 08:58 | |
*** haomaiwang has quit IRC | 09:01 | |
*** ChanServ changes topic to "Review Dashboard: http://goo.gl/8IUcKl | Summary Dashboard: http://goo.gl/qHus5v | Hackathon topics: https://etherpad.openstack.org/p/swift-midcycle-aug-2015 | https://etherpad.openstack.org/p/swift_encryption_issues | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/" | 09:01 | |
-openstackstatus- NOTICE: CI is back online but has a huge backlog. Please be patient and if possible delay approving changes until it has caught up. | 09:01 | |
*** haomaiwang has joined #openstack-swift | 09:02 | |
*** silor has joined #openstack-swift | 09:02 | |
*** chlong has joined #openstack-swift | 09:06 | |
*** jamielennox is now known as jamielennox|away | 09:25 | |
*** tamizh_geek has quit IRC | 09:27 | |
*** ppai is now known as ppai|afk | 09:29 | |
*** ig0r_ has joined #openstack-swift | 09:31 | |
*** tamizh_geek has joined #openstack-swift | 09:33 | |
*** sakaYK has quit IRC | 09:34 | |
*** sanchitmalhotra has joined #openstack-swift | 09:41 | |
*** tamizh_g_ has quit IRC | 09:41 | |
*** tamizh_g_ has joined #openstack-swift | 09:41 | |
*** sanchitmalhotra1 has quit IRC | 09:43 | |
*** marzif_ has joined #openstack-swift | 09:57 | |
*** haomaiwang has quit IRC | 10:01 | |
*** haomaiwang has joined #openstack-swift | 10:01 | |
openstackgerrit | Hiroshi Miura proposed openstack/python-swiftclient: Update flake8 config https://review.openstack.org/207372 | 10:02 |
acoles | clayg: you still here? | 10:08 |
*** SkyRocknRoll has joined #openstack-swift | 10:12 | |
*** jasondotstar has joined #openstack-swift | 10:15 | |
*** ho has quit IRC | 10:19 | |
*** sanchitmalhotra1 has joined #openstack-swift | 10:27 | |
*** sanchitmalhotra has quit IRC | 10:29 | |
openstackgerrit | Hiroshi Miura proposed openstack/python-swiftclient: Update flake8 config https://review.openstack.org/207372 | 10:33 |
*** sanchitmalhotra has joined #openstack-swift | 10:37 | |
*** aix has quit IRC | 10:39 | |
*** sanchitmalhotra1 has quit IRC | 10:39 | |
*** sanchitmalhotra1 has joined #openstack-swift | 10:42 | |
*** sanchitmalhotra has quit IRC | 10:44 | |
*** sanchitmalhotra has joined #openstack-swift | 10:49 | |
*** sanchitmalhotra1 has quit IRC | 10:51 | |
*** haomaiwang has quit IRC | 11:01 | |
*** 7GHAAT9JB has joined #openstack-swift | 11:02 | |
*** sanchitmalhotra1 has joined #openstack-swift | 11:09 | |
*** sanchitmalhotra has quit IRC | 11:11 | |
*** aix has joined #openstack-swift | 11:13 | |
*** jasondotstar has quit IRC | 11:19 | |
openstackgerrit | Alistair Coles proposed openstack/swift: Refactor diskfile https://review.openstack.org/198429 | 11:31 |
openstackgerrit | Alistair Coles proposed openstack/swift: Add ability for GET path to see/select alternate frag archs https://review.openstack.org/207165 | 11:31 |
acoles | peluse: ^^ fixed a merge conflict. I guess if you like the refactor you could just abandon your existing review and adopt this one. | 11:32 |
*** cdelatte has quit IRC | 11:35 | |
*** dmorita has quit IRC | 11:38 | |
*** swat30 has quit IRC | 11:45 | |
*** jordanP has quit IRC | 11:46 | |
*** swat30 has joined #openstack-swift | 11:46 | |
*** marcusvrn has joined #openstack-swift | 11:47 | |
*** jordanP has joined #openstack-swift | 11:47 | |
*** tsubic has quit IRC | 11:51 | |
*** cdelatte has joined #openstack-swift | 11:51 | |
*** delattec has joined #openstack-swift | 11:51 | |
peluse | acoles, thanks. that's the plan as soon as I get some tie to go through it - looking like tomorrow or this weekend | 11:52 |
*** 7GHAAT9JB has quit IRC | 12:01 | |
*** haomaiwang has joined #openstack-swift | 12:02 | |
*** cdelatte has quit IRC | 12:03 | |
*** delattec has quit IRC | 12:03 | |
*** tsubic has joined #openstack-swift | 12:05 | |
*** lpabon has joined #openstack-swift | 12:07 | |
*** km has quit IRC | 12:09 | |
tsubic | `2` | 12:16 |
*** tsubic has left #openstack-swift | 12:17 | |
*** jasondotstar has joined #openstack-swift | 12:22 | |
*** SkyRocknRoll has quit IRC | 12:35 | |
*** mahatic has joined #openstack-swift | 12:35 | |
*** jamielennox|away is now known as jamielennox | 12:35 | |
*** ccavanna has quit IRC | 12:49 | |
*** ppai|afk is now known as ppai | 12:56 | |
*** jkugel has joined #openstack-swift | 12:59 | |
*** haomaiwang has quit IRC | 13:01 | |
*** haomaiwang has joined #openstack-swift | 13:02 | |
*** jamielennox is now known as jamielennox|away | 13:06 | |
*** nakagawamsa has quit IRC | 13:09 | |
*** ig0r_ has quit IRC | 13:10 | |
*** hrou has joined #openstack-swift | 13:12 | |
*** lpabon has quit IRC | 13:13 | |
*** ig0r_ has joined #openstack-swift | 13:16 | |
*** NM has joined #openstack-swift | 13:16 | |
*** marzif_ has quit IRC | 13:21 | |
*** marzif_ has joined #openstack-swift | 13:22 | |
*** marzif_ has quit IRC | 13:22 | |
*** lpabon has joined #openstack-swift | 13:22 | |
*** rbrooker has joined #openstack-swift | 13:23 | |
*** bill_az has joined #openstack-swift | 13:25 | |
*** jasondotstar has quit IRC | 13:30 | |
*** ig0r_ has quit IRC | 13:31 | |
*** kei_yama_ has quit IRC | 13:33 | |
*** ccavanna has joined #openstack-swift | 13:36 | |
*** annegentle has joined #openstack-swift | 13:43 | |
*** joeljwright has joined #openstack-swift | 13:51 | |
*** ChanServ sets mode: +v joeljwright | 13:51 | |
*** annegentle has quit IRC | 13:55 | |
*** haomaiwang has quit IRC | 14:01 | |
*** thurloat_isgone is now known as thurloat | 14:02 | |
*** haomaiwang has joined #openstack-swift | 14:02 | |
thurloat | has anyone launched a reasonably scaled swift setup using SSDs for storage? | 14:02 |
*** sanchitmalhotra1 has quit IRC | 14:04 | |
*** Kennan2 has quit IRC | 14:16 | |
*** Kennan has joined #openstack-swift | 14:16 | |
*** breitz has quit IRC | 14:22 | |
*** ppai has quit IRC | 14:23 | |
*** annegentle has joined #openstack-swift | 14:24 | |
*** ig0r_ has joined #openstack-swift | 14:31 | |
*** mahatic has quit IRC | 14:34 | |
*** jlhinson has joined #openstack-swift | 14:35 | |
*** rbrooker has quit IRC | 14:38 | |
*** jrichli has joined #openstack-swift | 14:42 | |
*** joeljwright has quit IRC | 14:43 | |
*** bapalm_ has joined #openstack-swift | 14:43 | |
*** lcurtis has joined #openstack-swift | 14:52 | |
*** breitz has joined #openstack-swift | 14:55 | |
*** minwoob has joined #openstack-swift | 14:59 | |
*** haomaiwang has quit IRC | 15:01 | |
*** rbrooker has joined #openstack-swift | 15:02 | |
*** haomaiwang has joined #openstack-swift | 15:02 | |
*** ig0r_ has quit IRC | 15:03 | |
*** ig0r__ has joined #openstack-swift | 15:03 | |
*** ig0r_ has joined #openstack-swift | 15:08 | |
*** logan2 has quit IRC | 15:08 | |
*** ig0r__ has quit IRC | 15:09 | |
*** ig0r__ has joined #openstack-swift | 15:12 | |
*** ig0r__ has quit IRC | 15:15 | |
lcurtis | q for all...if i lose a drive...does swift auditor detect the failure and start replicating elswhere or do i have to replace drive first and remount as same device for replication to start occuring | 15:15 |
*** ig0r_ has quit IRC | 15:15 | |
jrichli | lcurtis: replication to the handoff node is triggered at that point. so you don't have to replace the drive to get full replication again. | 15:19 |
lcurtis | jrichli...what exact action kicks off that replication | 15:20 |
lcurtis | i am checking logs on server with failed drive | 15:20 |
lcurtis | dont see anything replicating | 15:21 |
glange | the replicator daemon is the part that does replication -- it walks the disk and tries to push everything it finds to the right places | 15:21 |
glange | replication on a particular partition happens when it is found on disk | 15:22 |
lcurtis | okay...so replicator on another server would find partition and notice missing and replicate? | 15:22 |
redbo | Maybe. The drive has to be unmounted, and you have to have mount checks enabled. | 15:23 |
glange | it would check the to see if the file is in the right three places, and try to push to places where the file should be | 15:23 |
redbo | Also having data on handoff nodes for long periods of time isn't ideal. You should replace the drive or remove it from the ring. | 15:23 |
lcurtis | definitely...we are replacing immediately | 15:23 |
glange | then the files should be written back to it when a replication pass is done | 15:24 |
glange | nothing really "knows" that the drive is gone or empty -- the replicator just tries to dumbly push objects to the right place over and over again | 15:24 |
lcurtis | but the replicator might run on another server and check that replica is not available on deviceX? | 15:26 |
lcurtis | and THAT would force a replication? | 15:26 |
tdasilva | glange: so until you replace that drive, you might only have 2 copies? | 15:26 |
lcurtis | sorry...trying to understand in detail | 15:26 |
redbo | If the replicator sees that a drive is unmounted, it'll switch to the handoff. | 15:26 |
lcurtis | the replicator on that server with failed drive or from another | 15:27 |
redbo | from another server that has a copy of that data | 15:27 |
lcurtis | awesome | 15:28 |
lcurtis | thank you | 15:28 |
lcurtis | very much folks | 15:28 |
glange | it's always best to replace the drive fast :) | 15:28 |
lcurtis | yep! we are indeed | 15:28 |
*** cppforlife_ has quit IRC | 15:29 | |
lcurtis | we do have mount checks enabled | 15:29 |
notmyname | good morning | 15:29 |
*** zul has quit IRC | 15:31 | |
lcurtis | hello notmyname! | 15:33 |
peluse | mornin' | 15:36 |
peluse | so FYI Mahati and one of our guys here did some S3 compatiblity testing with Swift and Ceph and have engaged the swift3 folks (Kota and others) as well as the S3 test repo maintainer... | 15:37 |
peluse | interesting results, they're still comparing notes on configs, etc., but once that's straightened out and there's some confidence in the results we'll post here somewhere | 15:37 |
peluse | clayg, take a look at the missing .durables bug when you get online. looks like we might be seeing the same thing here in Phx on the performance cluster | 15:38 |
lcurtis | @all...what would be most reliable way to determine all partitions back to normal on a replaced drive? | 15:41 |
*** jasondotstar has joined #openstack-swift | 15:42 | |
*** rbrooker has quit IRC | 15:45 | |
lcurtis | or i should say objects | 15:46 |
notmyname | lcurtis: you can't really know that, since that would require finding things that are not there. so in practice you look at the replication logs | 15:49 |
*** joeljwright has joined #openstack-swift | 15:49 | |
*** ChanServ sets mode: +v joeljwright | 15:49 | |
lcurtis | notmyname okay great...thats what i am doing...just wanted to make sure not overlooking something | 15:50 |
glange | lcurtis: there is also something called the dispersion report -- basically you seed a bunch of small objects all over your cluser and it can check to see if they are in the right place | 15:51 |
lcurtis | thank you | 15:51 |
lcurtis | glange...ah okay | 15:51 |
lcurtis | makes sense | 15:51 |
glange | http://docs.openstack.org/developer/swift/admin_guide.html <-- look at the "cluster health" section | 15:51 |
*** ig0r_ has joined #openstack-swift | 15:52 | |
notmyname | oh yeah. the dispersion report would be good for that | 15:54 |
ahale | general system graphs are useful too, seeing a drives disk usage increase and plateau with ring weight increases etc. | 15:54 |
lcurtis | make sense | 15:55 |
*** Protux has quit IRC | 15:55 | |
*** h00327910__ has quit IRC | 15:55 | |
*** marcusvrn has quit IRC | 15:55 | |
*** rbrooker has joined #openstack-swift | 15:58 | |
*** minwoob_ has joined #openstack-swift | 15:59 | |
*** minwoob has quit IRC | 16:00 | |
*** haomaiwang has quit IRC | 16:01 | |
*** haomaiwa_ has joined #openstack-swift | 16:02 | |
*** thumpba has joined #openstack-swift | 16:04 | |
*** nadeem has joined #openstack-swift | 16:04 | |
*** nadeem has quit IRC | 16:05 | |
*** nadeem has joined #openstack-swift | 16:06 | |
*** rbrooker has quit IRC | 16:06 | |
*** mfalatic has quit IRC | 16:07 | |
*** rbrooker has joined #openstack-swift | 16:08 | |
*** zul has joined #openstack-swift | 16:10 | |
tdasilva | peluse: were the s3 tests a comparison between the compatibility differences between swift and ceph rgw? or just swift3 and the latest s3 api? | 16:12 |
*** rbrooker has quit IRC | 16:14 | |
*** bhakta has quit IRC | 16:16 | |
*** zul has quit IRC | 16:16 | |
*** ajiang has quit IRC | 16:16 | |
*** logan2 has joined #openstack-swift | 16:19 | |
*** rledisez has quit IRC | 16:21 | |
*** ajiang has joined #openstack-swift | 16:21 | |
*** bhakta has joined #openstack-swift | 16:22 | |
*** jasondotstar has quit IRC | 16:23 | |
*** mfalatic has joined #openstack-swift | 16:23 | |
*** jistr has quit IRC | 16:24 | |
openstackgerrit | Michael Barton proposed openstack/swift: go: fix expecttransport memory usage https://review.openstack.org/206276 | 16:25 |
*** zhill has joined #openstack-swift | 16:25 | |
*** marcusvrn has joined #openstack-swift | 16:26 | |
*** zaitcev has joined #openstack-swift | 16:26 | |
*** ChanServ sets mode: +v zaitcev | 16:26 | |
*** zul has joined #openstack-swift | 16:28 | |
*** aix has quit IRC | 16:31 | |
*** cppforlife_ has joined #openstack-swift | 16:33 | |
*** zul has quit IRC | 16:33 | |
*** jordanP has quit IRC | 16:36 | |
*** zul has joined #openstack-swift | 16:42 | |
*** zhill has quit IRC | 16:48 | |
clayg | peluse: yeah I think we're onto something - dunno why with all this hashdirs that don't have durables in them we can't get back to the transaction ids that put them there? | 16:52 |
clayg | acoles: nope | 16:53 |
*** jordanP has joined #openstack-swift | 16:55 | |
acoles | clayg: heh. np it wasn't important | 16:56 |
clayg | acoles: k | 16:57 |
clayg | acoles: did we merge any pep8/flake8/hacking fixes last night!? | 16:57 |
acoles | clayg: I +A'd all the pep8 ones and i'm +2 on the hacking version bump patch 205977 | 16:59 |
patchbot | acoles: https://review.openstack.org/#/c/205977/ | 16:59 |
acoles | torgomatic had a +2 on that a few versions back ^^ | 16:59 |
acoles | don't think i have seen any land but zuuls been struggling | 17:00 |
clayg | acoles: oh right | 17:00 |
*** haomaiwa_ has quit IRC | 17:01 | |
*** haomaiwang has joined #openstack-swift | 17:02 | |
acoles | looks like 205977 is the only one on haypo's pep8 topic that is not WIP and not approved | 17:03 |
acoles | yet | 17:03 |
*** robefran has joined #openstack-swift | 17:04 | |
notmyname | 12 patches in the gate queue | 17:05 |
haypo | acoles: yes | 17:05 |
haypo | i will rebase my other pep8 patches when this first patch serie is merged | 17:06 |
*** silor has quit IRC | 17:07 | |
*** Protux has joined #openstack-swift | 17:08 | |
*** h00327910__ has joined #openstack-swift | 17:08 | |
peluse | clayg, yeah, odd. Just asked our BM guy to up the timeouts (conn and node both) and restart the tests, will see if that makes our errors go away | 17:13 |
peluse | BTW you may recall some discussion ages ago wrt the ecrecon at least about possibly bumping these based on increased connections. there's even still a comment in the code there in the ecrecon... | 17:15 |
peluse | # defaults subject to change after beta | 17:16 |
peluse | self.conn_timeout = float(conf.get('conn_timeout', 0.5)) | 17:16 |
peluse | self.node_timeout = float(conf.get('node_timeout', 10)) | 17:16 |
acoles | ho: i took a look at your test patch - can you tell me which scenario you are concerned about? | 17:16 |
*** bapalm_ has quit IRC | 17:25 | |
*** zhill_desktop has quit IRC | 17:29 | |
clayg | peluse: I don't think it's the reconstructor that's iming out tho? | 17:32 |
openstackgerrit | Thiago da Silva proposed openstack/swift: versioned writes middleware https://review.openstack.org/134347 | 17:34 |
clayg | why does everything think their linting changes are helping? https://review.openstack.org/#/c/207372/2 | 17:34 |
clayg | i'm heading in | 17:34 |
*** zhill_desktop has joined #openstack-swift | 17:36 | |
redbo | I made a thing like hacking one time that all it did was look for "if not x == y:" and told you to change it to "if x != y:". I think we should gate on that. | 17:39 |
*** joeljwright has quit IRC | 17:40 | |
*** robefran_ has joined #openstack-swift | 17:41 | |
*** robefran_ has quit IRC | 17:41 | |
*** acoles is now known as acoles_away | 17:42 | |
openstackgerrit | Minwoo Bae proposed openstack/swift: EC: Handoff node to push existing fragment to the correct location. https://review.openstack.org/196848 | 17:56 |
*** haomaiwang has quit IRC | 18:01 | |
*** haomaiwang has joined #openstack-swift | 18:02 | |
*** nadeem has quit IRC | 18:14 | |
*** nadeem has joined #openstack-swift | 18:15 | |
*** ig0r__ has joined #openstack-swift | 18:24 | |
*** ccavanna_ has joined #openstack-swift | 18:29 | |
*** jasondotstar has joined #openstack-swift | 18:30 | |
*** thumpba_ has joined #openstack-swift | 18:30 | |
*** briancurtin_ has joined #openstack-swift | 18:31 | |
*** bill_az_ has joined #openstack-swift | 18:33 | |
*** marcusvrn has quit IRC | 18:33 | |
*** blair has quit IRC | 18:34 | |
*** briancurtin has quit IRC | 18:34 | |
*** acoles_away has quit IRC | 18:34 | |
*** tdasilva has quit IRC | 18:34 | |
*** ccavanna has quit IRC | 18:34 | |
*** bill_az has quit IRC | 18:34 | |
*** thumpba has quit IRC | 18:34 | |
*** briancurtin_ is now known as briancurtin | 18:34 | |
*** blair has joined #openstack-swift | 18:40 | |
*** changbl has quit IRC | 18:43 | |
ccavanna_ | Hi. I proposed a stats improvement change a few days ago (to show stats by policy) and addressed all comments. Does anyone have a moment to re-review? It's a small change. | 18:48 |
ccavanna_ | https://review.openstack.org/#/c/202657/ | 18:48 |
*** ig0r_ has quit IRC | 18:50 | |
*** changbl has joined #openstack-swift | 18:56 | |
*** bapalm_ has joined #openstack-swift | 19:00 | |
*** haomaiwang has quit IRC | 19:01 | |
*** geaaru has quit IRC | 19:01 | |
*** haomaiwang has joined #openstack-swift | 19:02 | |
*** ig0r_ has joined #openstack-swift | 19:04 | |
*** bapalm_ has quit IRC | 19:05 | |
*** lpabon has quit IRC | 19:10 | |
*** marcusvrn has joined #openstack-swift | 19:14 | |
*** jasondotstar has quit IRC | 19:29 | |
*** tdasilva has joined #openstack-swift | 19:29 | |
*** ChanServ sets mode: +v tdasilva | 19:29 | |
*** acoles_away has joined #openstack-swift | 19:29 | |
*** acoles_away is now known as acoles | 19:30 | |
*** ChanServ sets mode: +v acoles | 19:30 | |
*** openstackgerrit has quit IRC | 19:46 | |
*** openstackgerrit has joined #openstack-swift | 19:46 | |
*** jasondotstar has joined #openstack-swift | 19:50 | |
*** wer_ has joined #openstack-swift | 19:53 | |
*** wer_ has quit IRC | 19:54 | |
*** ig0r_ has quit IRC | 19:56 | |
*** haomaiwang has quit IRC | 20:01 | |
*** haomaiwang has joined #openstack-swift | 20:02 | |
*** thurloat is now known as thurloat_isgone | 20:10 | |
*** annegentle has quit IRC | 20:13 | |
*** annegentle has joined #openstack-swift | 20:13 | |
*** openstackgerrit has quit IRC | 20:16 | |
*** openstackgerrit has joined #openstack-swift | 20:16 | |
openstackgerrit | Minwoo Bae proposed openstack/swift: EC: Handoff node to push existing fragment to the correct location. https://review.openstack.org/196848 | 20:21 |
*** blmartin has joined #openstack-swift | 20:22 | |
*** si1v3r has joined #openstack-swift | 20:26 | |
si1v3r | Out of curiousity, is there any preference against using raid cards when they are available? | 20:27 |
ctennis | yes, use them in jbod mode | 20:28 |
si1v3r | Why not raid 10? | 20:28 |
ctennis | because swift works better when it knows about each individual disk so it can place data specifically. | 20:29 |
si1v3r | Does it? | 20:29 |
ctennis | when you have a failed drive, swift will work around it. when you have a failed raid drive, swift doesn't know. | 20:29 |
occupant | I do that with an LSI raid card and I ran into weirdo performance issues with the CentOS kernel that I could never get anyone to care about. Ended up using the elrepo v3 kernel instead. | 20:29 |
occupant | A lot of raid cards won't let you have a raid array AND use jbods at the same time. | 20:30 |
si1v3r | ctennis: I understand that, but replacing the drive in the same manner as any of our other machines would make swift's working around completly unnessicary. | 20:30 |
si1v3r | I'm seeing big performance numbers in my head. | 20:30 |
occupant | when your app can directly address drives, there's no performance benefit in creating a logical group | 20:31 |
occupant | the bandwidth of your controller is the only thing that matters | 20:31 |
si1v3r | Well, it's a sas controller, and I'm pretty sure you can write to a raid faster than 2 drives of the same speed. | 20:32 |
si1v3r | Using the controllers data queing | 20:32 |
si1v3r | (having a bad typing day here...) | 20:32 |
ctennis | i don't recommend that either | 20:33 |
si1v3r | Why? | 20:33 |
ctennis | because you have a time window where you could lose data | 20:33 |
*** robefran has quit IRC | 20:33 | |
si1v3r | We have that already | 20:33 |
ctennis | I guess in the end it depends on your use case. Swift is designed for durability, not necessarily write performance. | 20:33 |
si1v3r | Would the data not be spread to 3 raids? | 20:34 |
si1v3r | We have 30 machines with 12 drives each. The sas controllers are useless in this configuration. | 20:34 |
ctennis | yes, so present all 360 disks individually to swift | 20:35 |
si1v3r | Why | 20:35 |
si1v3r | also, did. It sucks. | 20:35 |
ctennis | sorry to hear | 20:35 |
si1v3r | Wouldn't swift be happy as a clam never knowing of drive failure? | 20:36 |
ctennis | can you be more specific? | 20:36 |
ctennis | sure, but then why use Swift at that point? | 20:36 |
si1v3r | I want to know why many single drive jbods would be better than raid. | 20:36 |
occupant | used to work at a place that wrote data raw to disk. no filesystem, no parition even. | 20:36 |
ctennis | notmyname: can you articulate this better than I? ^ | 20:37 |
ctennis | can you quantify how it sucked? | 20:37 |
si1v3r | Swift uses the kernel to tell if a drive is bad. It's causing false positives. | 20:37 |
si1v3r | I'd rather trust LSI. | 20:38 |
ctennis | Okay, well, there's no technical reason you *can't* do it | 20:38 |
si1v3r | So instead of each box being a zone with 12 drives, we could just make each a zone with one drive. | 20:38 |
si1v3r | Is there a technical reason I should not? | 20:39 |
si1v3r | all I'm hearing is that it's abnormal and not as designed. | 20:39 |
si1v3r | has anyone even tried? | 20:39 |
occupant | you're getting false kernel errors about your drives? | 20:39 |
ctennis | sure, I've seen people do lots of things. I've seen clusters of all raid0 single drives. | 20:39 |
ctennis | you may never have an issue. | 20:39 |
ctennis | I'd say if you have good success with it, it would be an interesting blog post or story to read. | 20:40 |
*** annegentle has quit IRC | 20:40 | |
si1v3r | So there is no reason to believe that it works better with individual drives then? | 20:40 |
ctennis | I guess it depends on what you mean by "works better" | 20:41 |
*** ccavanna_ has quit IRC | 20:41 | |
si1v3r | If this is unexplored territory that is fine, but using hardware wrong based on bad assumptions is a damn shame. | 20:41 |
ctennis | It sounds like you have a specific error you're trying to avoid | 20:41 |
occupant | I dunno, maybe you know better than the people who wrote this shit | 20:41 |
ctennis | which is okay | 20:41 |
si1v3r | ctennis: I'm trying to make it go faster than 1gbps. | 20:42 |
si1v3r | it takes forever to upload 400GB right now. | 20:42 |
ctennis | that may be doable with just some performance tuning | 20:42 |
*** openstackgerrit has quit IRC | 20:46 | |
*** openstackgerrit has joined #openstack-swift | 20:46 | |
occupant | if you're trying to upload large files, you'd probably be happier doing more aggressive chunking | 20:47 |
occupant | that way your writes are more distributed with less chance of saturating the IO of any one drive | 20:47 |
ctennis | you should easily be able to 1Gbps, if not 10. | 20:47 |
si1v3r | yeah these boxes are on 40gb links I think, they should be able to go a lot faster. | 20:48 |
*** jasondotstar has quit IRC | 20:56 | |
*** jamielennox|away is now known as jamielennox | 20:57 | |
*** thumpba_ has quit IRC | 21:00 | |
*** haomaiwang has quit IRC | 21:01 | |
*** openstack has joined #openstack-swift | 21:10 | |
*** annegentle has joined #openstack-swift | 21:16 | |
*** hrou has quit IRC | 21:18 | |
*** robefran has joined #openstack-swift | 21:18 | |
*** zul has joined #openstack-swift | 21:18 | |
*** wer_ has joined #openstack-swift | 21:19 | |
*** wer_ has quit IRC | 21:19 | |
clayg | swifterdarrell: https://bugs.launchpad.net/swift/+bug/1479972 | 21:23 |
openstack | Launchpad bug 1479972 in OpenStack Object Storage (swift) "HUP signal doesn't shutdown wsgi servers" [Undecided,New] | 21:23 |
*** marzif_ has joined #openstack-swift | 21:26 | |
*** Fin1te has joined #openstack-swift | 21:28 | |
*** marzif_ has quit IRC | 21:32 | |
*** blmartin has quit IRC | 21:33 | |
*** marzif_ has joined #openstack-swift | 21:34 | |
*** Fin1te has quit IRC | 21:35 | |
*** marzif_ has quit IRC | 21:35 | |
*** marzif_ has joined #openstack-swift | 21:36 | |
openstackgerrit | Darrell Bishop proposed openstack/swift: Fix regression in WSGI server SIGHUP behavior https://review.openstack.org/207637 | 21:37 |
openstackgerrit | Darrell Bishop proposed openstack/swift: Fix regression in WSGI server SIGHUP behavior https://review.openstack.org/207637 | 21:40 |
*** jrichli has quit IRC | 21:40 | |
swifterdarrell | acoles: clayg: cschwede: glange: mattoliverau: notmyname: peluse: redbo: tdasilva: torgomatic: zaitcev: Gah! I broke SIGHUP for the WSGI servers https://review.openstack.org/#/c/207637/2 | 21:42 |
redbo | well stop it. oh you can't. | 21:43 |
swifterdarrell | redbo: haha, train. station. left. | 21:43 |
swifterdarrell | redbo: fix is ready for review, though | 21:43 |
clayg | anyone got any problems getting this merged quickly? It's a) really gross and b) fairly obvious fix - we can do a post morten after we start working on new packages. | 21:45 |
redbo | done | 21:45 |
*** jasondotstar has joined #openstack-swift | 21:48 | |
*** annegentle has quit IRC | 21:49 | |
*** ig0r__ has quit IRC | 21:51 | |
peluse | neat | 21:53 |
*** marzif_ has quit IRC | 21:54 | |
clayg | peluse: no - NOT neat | 21:55 |
*** jlhinson has quit IRC | 21:57 | |
MooingLemur | this reminds me of having to bounce between event loop handlers every couple hundred ms in a very old project. | 21:58 |
clayg | :'( | 21:58 |
clayg | I honestly don't think I really understood how the *non*-green os.wait was acctually *blocking* the parent pids event loop and the pythons moving to sig handlers HUP was falling out out of the wait | 21:59 |
clayg | it's all very crazy | 21:59 |
clayg | the way it's written now I acctually understand - and if I'd have thought about the old implementation I would have realized I don't know how it would work if the wait blocks with no timeout - but that "exactly" what the old code did! :'( | 22:00 |
*** annegentle has joined #openstack-swift | 22:00 | |
*** haomaiwang has quit IRC | 22:01 | |
*** NM has quit IRC | 22:01 | |
clayg | but I guess that's the EINTR :'( | 22:01 |
clayg | god damn it | 22:01 |
*** haomaiwang has joined #openstack-swift | 22:02 | |
redbo | did someone say python's getting rid of eintr? that'll be nice. | 22:02 |
redbo | well hiding it | 22:02 |
clayg | lol | 22:02 |
*** jkugel has quit IRC | 22:06 | |
redbo | I've had problems on modwsgi before where signals interrupt i/o calls, causing an unrecoverable exception. But apache uses signals to talk between workers, so you can't just disable them. | 22:08 |
clayg | can't win for loosing :'( | 22:08 |
*** jamielennox is now known as jamielennox|away | 22:15 | |
*** jordanP has quit IRC | 22:22 | |
clayg | can't win for loosing :'( | 22:37 |
clayg | ^ or for alt-tab-up-entering | 22:37 |
*** annegentle has quit IRC | 22:44 | |
openstackgerrit | Clay Gerrard proposed openstack/swift: Add a probetest for HUP/reload https://review.openstack.org/207656 | 22:52 |
clayg | love me some probetests | 22:59 |
*** haomaiwang has quit IRC | 23:01 | |
*** km has joined #openstack-swift | 23:01 | |
*** haomaiwa_ has joined #openstack-swift | 23:02 | |
*** ho has joined #openstack-swift | 23:14 | |
*** kei_yama has joined #openstack-swift | 23:17 | |
mattoliverau | Morning | 23:22 |
openstackgerrit | paul luse proposed openstack/swift: Add ability for GET path to see/select alternate frag archs https://review.openstack.org/207165 | 23:32 |
peluse | acoles, ^ EC func tests fixed | 23:34 |
*** marcusvrn has quit IRC | 23:43 | |
*** lcurtis has quit IRC | 23:46 | |
clayg | peluse: so is patch 201283 to be abandonded? | 23:48 |
patchbot | clayg: https://review.openstack.org/#/c/201283/ | 23:48 |
peluse | yeah, acoles made one that had diskfile refactor as a dependecny so we moved to that one | 23:49 |
peluse | so he is the owner, same patch name, etc. All working/rebased/bla bla bla. Ready for eyes | 23:49 |
peluse | clayg, BTW our perf cluster has been running a while now w/large (30 sec) timeouts and no strange partial PUT failures that lead to Ec recon failures. Testing continues of course | 23:50 |
peluse | clayg, might be worth charles trying to crank those up also as a test | 23:51 |
peluse | notmyname, or anyone really... does the Openstack community generally consider Swift as HA? I don't see it mentioned in http://docs.openstack.org/high-availability-guide/high-availability-guide.pdf | 23:58 |
peluse | but seems to me that we're already HA... | 23:58 |
peluse | ... no single POF, rolling upgrades w/o downtime, etc., etc. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!