| *** vint_bra has joined #openstack-swift | 00:18 | |
| *** psachin has joined #openstack-swift | 00:32 | |
| -openstackstatus- NOTICE: due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:33 | |
| *** tovin07_ has joined #openstack-swift | 00:40 | |
| tdasilva | kota_: any issues with landing hp 512284 and p 512277 | 01:12 |
|---|---|---|
| patchbot | https://review.openstack.org/#/c/512277/ - swift (feature/s3api) - Merge 'remotes/origin/master' into s3api | 01:12 |
| *** two_tired has joined #openstack-swift | 01:15 | |
| *** m_kazuhiro has joined #openstack-swift | 01:26 | |
| m_kazuhiro | timburke: Are you here now? | 01:28 |
| timburke | m_kazuhiro: o/ | 01:28 |
| m_kazuhiro | timburke: Hi. I want to know detail of your opinion about symlink POST. | 01:29 |
| m_kazuhiro | timburke: Do versioned_write middleware need "symlink=post"? | 01:31 |
| *** kei_yama has quit IRC | 01:32 | |
| timburke | i think the current behavior (accept the post, because we have to in order to maintain eventual consistency) is good. the ?symlink=post api is ... meh. might be a useful escape match if a developer doesn't have full control over what the HTTP layer does? | 01:34 |
| timburke | versioned_writes won't need it -- it'll basically follow the 307 and reapply the POST (if i ever get around to trying to rewrite that...) | 01:34 |
| tdasilva | just wondering..playing devil's advocate here....what's the problem with having 'symlink=post'? | 01:37 |
| tdasilva | trying to understand what the concern is... | 01:37 |
| timburke | i don't really care; either way's fine. i guess the advantage it to have some 2xx response ... *shrug* | 01:39 |
| timburke | biggest downside i can think of is just cognitive load and maintenance burden. both of which should be pretty light | 01:40 |
| m_kazuhiro | tdasilva: timburke: I think having "symlink=post" makes POST explanation of symlink difficult a little. I want to make it simple. | 01:40 |
| timburke | and that -- the "crap, i have to explain it" burden ;-) | 01:41 |
| tdasilva | m_kazuhiro: i don't know...i think the argument of having a way for users that DO want to post to a symlink is a good one. If the issue is explaining that, then we can work on the wording | 01:41 |
| *** chetna has quit IRC | 01:43 | |
| timburke | tdasilva: but alternatively, they could just *not* follow the 307. we *have* to accept the post regardless, so there's no real impediment to that workflow | 01:43 |
| tdasilva | yeah, but that I find it a bit more weird, like..."I know what I am doing, i'm postiong to the 'object' I want, I get an error, but I can ignore it, cause I know it worked as I expected it. idk... | 01:46 |
| m_kazuhiro | tdasilva: I think if we don't have "symlink=post", we can update metadata of symlinks by re-PUTting symlinks with new metadata... because symlinks are zero-byte objects. | 01:47 |
| clayg | tdasilva: I hear you, seems "cleaner" somehow... but you can always throw it (and docs) on after the fact? the effect of POST w/o symlink=post is the same as POST w/ symlink=post the difference is only in the response code | 01:47 |
| clayg | it's like you HAVE to write the code that sees the 2XX and translates to error (or 3XX) - then later you can add the code that doesn't translate the 2XX iff you have symlink=post qs? | 01:48 |
| tdasilva | clayg, m_kazuhiro: yes, I agree and did think that this is something that can be added later, it's much easier than removing something we no longer want to support... | 01:49 |
| tdasilva | OTOH, it's just that the argument for not having is because it's hard to explain that I'm not a big fan of | 01:49 |
| *** kei_yama has joined #openstack-swift | 01:49 | |
| clayg | tdasilva: is that the last remaining issue to address on the symlink patch? "should we include/explain symlink=post"?!?! | 01:50 |
| tdasilva | well..it's already there. we are discussing if we should remove it | 01:50 |
| timburke | let's think about swiftclient, as an example application that might want to add support for POSTing to symlinks | 01:50 |
| timburke | either, we add an exception to the is-success logic, in the case of POST | 01:51 |
| timburke | or we add a query param to our POSTs | 01:51 |
| timburke | both require code changes | 01:51 |
| timburke | idk, maybe it's that i just don't see 307 == error? 3xx is kinda informational -- it's not 4xx (my fault) or 5xx (your fault)... it's saying "hey, you might not be done yet" | 01:55 |
| clayg | +1 3XX isn't really an error - which is probably good since we're kinda acctually doing the POST? | 01:57 |
| m_kazuhiro | According to rfc, 3xx indicates that further action needs to be taken by the user agent in order to fulfill the request. So applications should handle them. | 02:04 |
| *** vint_bra has quit IRC | 02:07 | |
| m_kazuhiro | Umm, RFC continues with "The action required MAY be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD." So if response is 3xx in POST, application cannot handle. | 02:11 |
| clayg | yeah that's the updated 7230 stuff - 3XX used to be a lot more loosey goosey and browsers just learned that 3XX in response to POST meant they should GET the Location header, but honestly for a native client like swiftclient I wouldn't think it too unreasonable to say "POST responded 3XX do you want to try to apply the POST to the location?" with support for post --follow-symlinks (i.e. pre-emptive interaction | 02:14 |
| clayg | with the user) to be added later | 02:14 |
| clayg | I'm still ok with 3XX response for POST to symlink | 02:15 |
| clayg | ... but it might be worth re-reading the rfc - do you have a link to the relevant section(s) handy? | 02:15 |
| timburke | https://tools.ietf.org/html/rfc7231#section-6.4.7 for the 307 i've been advocating... | 02:16 |
| clayg | that's new too! awesome! | 02:17 |
| m_kazuhiro | timburke: thank you | 02:17 |
| clayg | yeah timburke is the best | 02:19 |
| timburke | https://tools.ietf.org/html/rfc7231#section-6.4 explicitly says "Automatic redirection needs to done with care for methods not known to be safe" | 02:20 |
| *** chetna has joined #openstack-swift | 02:33 | |
| tdasilva | sounds like removing is the way to go | 02:35 |
| tdasilva | if needed later it can be added | 02:35 |
| m_kazuhiro | Yes. We can add it later. | 02:37 |
| m_kazuhiro | timburke: clayg: Is this fine with you? | 02:38 |
| clayg | wait... so it's already in there? I mean we're all in agreement that symlink=post returning 2xx instead of 307 is not a hard requirement | 02:39 |
| clayg | it the the only remaining issue to discuss? because it seems somewhat minor in the grand scheme of things - is the patch otherwise ready? | 02:39 |
| m_kazuhiro | clayg: "symilnk=post" is in current implementation. I have 2 other remaining issue... I think they are minor too. Patch is in updating for comments from kota by me. | 02:42 |
| m_kazuhiro | clayg: Remaining issues are in https://etherpad.openstack.org/p/swift_symlink_remaining_discussion_points | 02:42 |
| clayg | ok, that looks great - i'd call this issue closed for now - I think the comments on the etherpad should be sufficient to get someone context after the fact if they ask about it during review - if the implementation drops symlink=post entirely I think that's a pretty defensible design decision - definitely shouldn't hold up any work/review | 02:44 |
| clayg | KUDOS! | 02:44 |
| *** chlong has quit IRC | 02:47 | |
| *** tovin07 has quit IRC | 02:47 | |
| *** tovin07 has joined #openstack-swift | 02:48 | |
| m_kazuhiro | timburke: How about you? | 02:48 |
| *** chetna has quit IRC | 02:56 | |
| *** tovin07 has quit IRC | 02:56 | |
| *** tovin07 has joined #openstack-swift | 02:57 | |
| m_kazuhiro | timburke seems not to be in IRC now. | 03:06 |
| *** wes_dillingham has quit IRC | 03:06 | |
| m_kazuhiro | timburke: We conclude that we remove "symlink=post". If you have opinion for that, please write it on the etherpad page or send me message. | 03:07 |
| m_kazuhiro | I have to go to lunch now. Everyone, thank you for discussion. See you. | 03:08 |
| *** openstackgerrit has quit IRC | 03:22 | |
| *** m_kazuhiro has quit IRC | 03:31 | |
| *** klrmn has quit IRC | 03:32 | |
| *** links has joined #openstack-swift | 03:45 | |
| *** gkadam has joined #openstack-swift | 03:55 | |
| *** Jeffrey4l has quit IRC | 04:02 | |
| *** Jeffrey4l has joined #openstack-swift | 04:03 | |
| *** chetna has joined #openstack-swift | 04:26 | |
| *** chetna has quit IRC | 04:28 | |
| *** m_kazuhiro has joined #openstack-swift | 04:29 | |
| *** openstackgerrit has joined #openstack-swift | 04:51 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Stop logging tracebacks on bad xLOs https://review.openstack.org/496493 | 04:51 |
| *** fguillot has joined #openstack-swift | 04:51 | |
| *** fguillot has left #openstack-swift | 04:52 | |
| *** two_tired has quit IRC | 05:15 | |
| *** ChubYann has quit IRC | 05:28 | |
| *** hseipp has joined #openstack-swift | 05:36 | |
| *** hseipp has quit IRC | 05:40 | |
| *** chetna has joined #openstack-swift | 05:48 | |
| *** chetna has quit IRC | 05:49 | |
| *** chetna has joined #openstack-swift | 05:50 | |
| *** chetna has quit IRC | 05:51 | |
| mahatic | acoles: ack for the meeting, thanks. | 06:02 |
| *** m_kazuhiro has quit IRC | 06:38 | |
| *** m_kazuhiro has joined #openstack-swift | 06:39 | |
| *** pcaruana has joined #openstack-swift | 06:45 | |
| *** hseipp has joined #openstack-swift | 06:52 | |
| *** oshritf has quit IRC | 07:00 | |
| *** oshritf has joined #openstack-swift | 07:08 | |
| *** tesseract has joined #openstack-swift | 07:17 | |
| *** oshritf has quit IRC | 07:23 | |
| *** m_kazuhiro_ has joined #openstack-swift | 07:25 | |
| *** tovin07_ has quit IRC | 07:28 | |
| *** m_kazuhiro has quit IRC | 07:28 | |
| *** hoonetorg has joined #openstack-swift | 07:32 | |
| *** wasmum has joined #openstack-swift | 07:34 | |
| *** geaaru has joined #openstack-swift | 07:41 | |
| *** oshritf has joined #openstack-swift | 07:43 | |
| *** oshritf has quit IRC | 07:48 | |
| *** chetna has joined #openstack-swift | 07:52 | |
| *** chetna has quit IRC | 07:53 | |
| *** jidar has quit IRC | 08:03 | |
| *** oshritf has joined #openstack-swift | 08:05 | |
| *** oshritf has quit IRC | 08:07 | |
| *** portante has quit IRC | 08:08 | |
| *** portante has joined #openstack-swift | 08:09 | |
| *** chetna has joined #openstack-swift | 08:14 | |
| *** cbartz has joined #openstack-swift | 08:15 | |
| *** chetna has quit IRC | 08:16 | |
| acoles | good morning | 08:19 |
| *** jrichli- has joined #openstack-swift | 08:25 | |
| *** oshritf has joined #openstack-swift | 08:25 | |
| acoles | mahatic: hi! | 08:25 |
| mahatic | acoles: hello! good morning | 08:28 |
| *** ujjain has quit IRC | 08:28 | |
| *** jrichli has quit IRC | 08:28 | |
| *** ujjain has joined #openstack-swift | 08:30 | |
| *** ujjain has joined #openstack-swift | 08:30 | |
| *** kei_yama has quit IRC | 08:32 | |
| *** nikivi has quit IRC | 08:38 | |
| *** nikivi has joined #openstack-swift | 08:42 | |
| *** oshritf has quit IRC | 09:35 | |
| *** chetna has joined #openstack-swift | 09:36 | |
| *** chetna has quit IRC | 09:37 | |
| *** hseipp has quit IRC | 09:38 | |
| *** abhitechie has joined #openstack-swift | 09:48 | |
| *** mvk has quit IRC | 09:50 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: tighten up drop_privileges unit tests https://review.openstack.org/512992 | 10:01 |
| *** abhitechie has quit IRC | 10:02 | |
| *** m_kazuhiro_ has quit IRC | 10:29 | |
| *** oshritf has joined #openstack-swift | 10:32 | |
| *** mat128 has joined #openstack-swift | 10:33 | |
| *** oshritf has quit IRC | 10:46 | |
| *** bob_cheesey has joined #openstack-swift | 10:53 | |
| openstackgerrit | Merged openstack/swift feature/deep: Change ShardRange to take a created_at arg https://review.openstack.org/512867 | 10:54 |
| openstackgerrit | Merged openstack/swift master: Drop group comparison from drop_privileges test https://review.openstack.org/512797 | 10:55 |
| bob_cheesey | could anyone help me out as to why i'm still seeing some traffic (and specifically some 500s) to a node which doesn't exist in any of the rings? | 10:55 |
| bob_cheesey | i'm going round in circles a little trying to figure it out | 10:55 |
| acoles | bob_cheesey: where do you see the 500s? did all the nodes get the new ring file(s)? | 11:00 |
| *** chlong has joined #openstack-swift | 11:01 | |
| bob_cheesey | acoles: yes, i've checked that it's up to date everywhere. i'm tcpdumping 601{0,1,2} to make sure it's safe to remove the node from the cluster (it's been decomissioned) | 11:02 |
| bob_cheesey | https://gist.github.com/analbeard/be7ca3d3d849b8ac22127c308c6c6319 is an example of what i'm seeing (using -A to view contents) | 11:03 |
| bob_cheesey | 10.0.137.104 is the decomissioned node | 11:04 |
| *** oshritf has joined #openstack-swift | 11:06 | |
| *** abhitechie has joined #openstack-swift | 11:07 | |
| bob_cheesey | watching the logs further, i can see that the packets seem to happen when the account auditor runs | 11:08 |
| bob_cheesey | (i think) | 11:09 |
| *** hseipp has joined #openstack-swift | 11:10 | |
| acoles | bob_cheesey: I'm not sure the auditor would generate an http request. could be the container updater | 11:11 |
| bob_cheesey | acoles: i'm just waiting to see if it happens again - it's over 6012 though which is what we have configured for the account-server | 11:13 |
| *** NM has joined #openstack-swift | 11:14 | |
| *** oshritf has quit IRC | 11:26 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Symlink implementation. https://review.openstack.org/232162 | 11:31 |
| acoles | bob_cheesey: that traceback would be for a PUT to the account server which occurs either during handling of a container PUT request, or from the container updater daemon | 11:31 |
| *** chetna has joined #openstack-swift | 11:37 | |
| *** chetna has quit IRC | 11:39 | |
| bob_cheesey | acoles: that would make sense as i'm occasionally seeing a failed put request too | 11:45 |
| acoles | bob_cheesey: it shouldn't cause a client container PUT to fail, it is just the update from container server to account server to let the account know about the container's state | 11:47 |
| bob_cheesey | acoles: so this put i'm seeing is showing up in the tcpdumps on this decomissioned node - is that unexpected behaviour? | 11:49 |
| bob_cheesey | additionally, it's the same account/container showing up each time, and it's only every 10 minutes or so - the cluster is extremely busy which is why i find it odd that it's a put to the same account/container each time | 11:50 |
| bob_cheesey | sorry for the questions, just trying to get my head around what i'm seeing | 11:51 |
| acoles | bob_cheesey: is it possible that there is a container updater process running on the decommissioned node itself? and that has an old ring file? | 11:53 |
| bob_cheesey | acoles: i haven't stopped any swift processes yet so yes, it will be running. the ring has been updated though | 11:54 |
| *** oshritf has joined #openstack-swift | 11:56 | |
| *** oshritf has quit IRC | 11:57 | |
| *** hseipp has quit IRC | 12:02 | |
| *** caiobrentano_ has joined #openstack-swift | 12:04 | |
| *** mvk has joined #openstack-swift | 12:07 | |
| *** Dinesh_Bhor has quit IRC | 12:09 | |
| *** wes_dillingham has joined #openstack-swift | 12:14 | |
| *** abhinavtechie has joined #openstack-swift | 12:15 | |
| *** abhitechie has quit IRC | 12:16 | |
| *** oshritf has joined #openstack-swift | 12:23 | |
| *** oshritf has quit IRC | 12:24 | |
| *** MVenesio has joined #openstack-swift | 12:32 | |
| *** gkadam has quit IRC | 12:34 | |
| *** hseipp has joined #openstack-swift | 12:39 | |
| *** hseipp has quit IRC | 12:43 | |
| *** hseipp has joined #openstack-swift | 12:43 | |
| *** links has quit IRC | 12:44 | |
| *** mat128 has quit IRC | 12:45 | |
| *** ujjain has quit IRC | 12:46 | |
| *** ujjain has joined #openstack-swift | 12:51 | |
| *** ujjain has quit IRC | 12:51 | |
| *** ujjain has joined #openstack-swift | 12:51 | |
| *** chetna has joined #openstack-swift | 13:01 | |
| *** oshritf has joined #openstack-swift | 13:02 | |
| *** chetna has quit IRC | 13:03 | |
| *** oshritf has quit IRC | 13:08 | |
| *** oshritf has joined #openstack-swift | 13:09 | |
| *** oshritf has quit IRC | 13:16 | |
| *** oshritf has joined #openstack-swift | 13:22 | |
| *** chetna has joined #openstack-swift | 13:22 | |
| *** chetna has quit IRC | 13:23 | |
| *** mat128 has joined #openstack-swift | 13:47 | |
| *** vint_bra has joined #openstack-swift | 13:50 | |
| *** vint_bra has quit IRC | 13:57 | |
| *** mabrams has joined #openstack-swift | 13:59 | |
| *** hseipp has quit IRC | 14:02 | |
| *** psachin has quit IRC | 14:05 | |
| *** mjturek has joined #openstack-swift | 14:08 | |
| openstackgerrit | Andrey Groshev proposed openstack/swift master: Removed all translations https://review.openstack.org/513049 | 14:20 |
| *** chetna has joined #openstack-swift | 14:26 | |
| *** abhinavtechie has quit IRC | 14:35 | |
| *** vinsh has joined #openstack-swift | 14:38 | |
| notmyname | good morning | 14:45 |
| *** chlong has quit IRC | 14:52 | |
| *** hseipp has joined #openstack-swift | 14:59 | |
| *** klrmn has joined #openstack-swift | 15:13 | |
| *** catintheroof has joined #openstack-swift | 15:13 | |
| *** pcaruana has quit IRC | 15:19 | |
| *** tesseract has quit IRC | 15:20 | |
| *** chetna has quit IRC | 15:25 | |
| *** chetna has joined #openstack-swift | 15:25 | |
| *** MVenesio has quit IRC | 15:28 | |
| *** cbartz has quit IRC | 15:34 | |
| *** vint_bra has joined #openstack-swift | 15:44 | |
| *** Sukhdev has joined #openstack-swift | 15:51 | |
| *** links has joined #openstack-swift | 15:54 | |
| *** guimaluf has quit IRC | 15:55 | |
| *** gkadam has joined #openstack-swift | 16:02 | |
| *** mabrams has quit IRC | 16:07 | |
| *** klrmn has quit IRC | 16:08 | |
| *** guimaluf has joined #openstack-swift | 16:15 | |
| *** gkadam has quit IRC | 16:22 | |
| *** hseipp has quit IRC | 16:24 | |
| *** links has quit IRC | 16:28 | |
| *** mat128 has quit IRC | 16:38 | |
| *** mvk has quit IRC | 16:42 | |
| *** blair has quit IRC | 16:42 | |
| *** gyee has joined #openstack-swift | 16:43 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Stop logging tracebacks on bad xLOs https://review.openstack.org/496493 | 16:43 |
| timburke | good morning | 16:44 |
| *** chsc has joined #openstack-swift | 16:50 | |
| *** Sukhdev has quit IRC | 16:50 | |
| *** wes_dillingham has quit IRC | 16:53 | |
| *** NM has quit IRC | 16:58 | |
| notmyname | timburke: what's the right answer on patch 336323 | 17:01 |
| patchbot | https://review.openstack.org/#/c/336323/ - swift - Add checksum to object extended attributes | 17:01 |
| notmyname | I rebased, you raised a question, I'm not sure | 17:02 |
| timburke | i don't know! i'm not necessarily opposed to adding checksums after the fact, but it seemed like torgomatic made a conscious decision to *not* do it for now... so it makes me nervous that we *sometimes* do it | 17:04 |
| *** blair has joined #openstack-swift | 17:06 | |
| torgomatic | It's tricky. Any given un-checksummed metadata is very likely to be fine, but it's not guaranteed. If we toss checksums on everything, then we'll get some guy showing up with a checksummed bogon in his cluster. If we don't, then all old objects remain at risk until they go through ssync. | 17:09 |
| torgomatic | Or get replaced, or whatever it is that makes them get checksums. | 17:10 |
| *** klrmn has joined #openstack-swift | 17:11 | |
| torgomatic | I opted to not add checksums because it's tricky; if we decide that it's a good idea to add them, we can always do it later. If I add checksums and then later we decide it was a bad idea, there's nothing we can do about it. | 17:15 |
| timburke | so, you're fine with ssync putting checksums on old data? don't we have the same "maybe it's good, maybe it isn't" problem? | 17:25 |
| *** mat128 has joined #openstack-swift | 17:29 | |
| *** geaaru has quit IRC | 17:30 | |
| *** silor has joined #openstack-swift | 17:34 | |
| *** geaaru has joined #openstack-swift | 17:35 | |
| torgomatic | timburke: yes, that's the same problem. I hadn't considered ssync. | 17:38 |
| *** guimaluf has quit IRC | 17:42 | |
| *** NM has joined #openstack-swift | 17:42 | |
| timburke | so are we fine with it or not? should we have some header in the ssync protocol to tell us whether or not to write the checksum? | 17:42 |
| *** wes_dillingham has joined #openstack-swift | 17:44 | |
| *** mvk has joined #openstack-swift | 17:48 | |
| *** NM has quit IRC | 17:49 | |
| *** NM has joined #openstack-swift | 17:50 | |
| * torgomatic has no idea | 17:52 | |
| *** catinthe_ has joined #openstack-swift | 17:53 | |
| *** catintheroof has quit IRC | 17:53 | |
| *** NM has quit IRC | 17:54 | |
| *** MVenesio has joined #openstack-swift | 17:56 | |
| *** catintheroof has joined #openstack-swift | 17:59 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift feature/s3api: Import all swift3 code as s3api https://review.openstack.org/512284 | 18:00 |
| *** catinthe_ has quit IRC | 18:01 | |
| *** NM has joined #openstack-swift | 18:05 | |
| *** catinthe_ has joined #openstack-swift | 18:05 | |
| *** geaaru has quit IRC | 18:05 | |
| *** catintheroof has quit IRC | 18:07 | |
| *** catintheroof has joined #openstack-swift | 18:14 | |
| *** catinthe_ has quit IRC | 18:15 | |
| tdasilva | as we merge swift3 into swift, i'm planning to merge AUTHORS file, does that make sense? | 18:18 |
| timburke | seems good to me | 18:18 |
| *** Sukhdev has joined #openstack-swift | 18:20 | |
| *** mat128 has quit IRC | 18:25 | |
| *** newmember has joined #openstack-swift | 18:30 | |
| *** chsc has quit IRC | 18:42 | |
| *** ChubYann has joined #openstack-swift | 18:44 | |
| *** jidar has joined #openstack-swift | 18:45 | |
| notmyname | tdasilva: sounds good. keep the top of swift's authors file the same. sort the list below by first character. I've got a script that may help that I use at release time, but it's certainly not a blocker now for anything | 18:49 |
| tdasilva | notmyname: yep, it was pretty straight forward...swift3 author's list was not that different from swift | 18:50 |
| notmyname | timburke: I think either I'm missing something or we're talking about two separate things | 18:50 |
| tdasilva | only a few more names | 18:50 |
| notmyname | tdasilva: the only "trickiness" will be any swift3 authors that aren't in the git history | 18:51 |
| tdasilva | mmm..why so? | 18:51 |
| tdasilva | your script will pick it up? | 18:51 |
| notmyname | yeah | 18:51 |
| tdasilva | oic | 18:51 |
| notmyname | to be fair, we already have a few like that | 18:52 |
| tdasilva | yeah, there will be a 'few' more :) | 18:54 |
| openstackgerrit | Merged openstack/swift feature/s3api: Merge 'remotes/origin/master' into s3api https://review.openstack.org/512277 | 18:54 |
| *** silor1 has joined #openstack-swift | 18:58 | |
| *** jidar has quit IRC | 18:59 | |
| *** silor has quit IRC | 19:00 | |
| *** silor1 is now known as silor | 19:00 | |
| *** jidar has joined #openstack-swift | 19:00 | |
| *** jidar has left #openstack-swift | 19:02 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: WIP only make single request to get container listings https://review.openstack.org/512756 | 19:10 |
| *** chsc has joined #openstack-swift | 19:14 | |
| *** chetna has quit IRC | 19:20 | |
| timburke | torgomatic: does your metadata checksumming patch handle POST data, too? or just whatever came with the original PUT? | 19:25 |
| openstackgerrit | Thiago da Silva proposed openstack/swift feature/s3api: Continue merge swift3 middleware https://review.openstack.org/513170 | 19:41 |
| *** joeljwright has joined #openstack-swift | 19:44 | |
| *** ChanServ sets mode: +v joeljwright | 19:44 | |
| torgomatic | timburke: I'm pretty sure it includes POSTs. It happens at a fairly low level in diskfile. | 19:44 |
| *** wes_dillingham has quit IRC | 19:52 | |
| *** NM has quit IRC | 19:58 | |
| drewn3ss | clayg: if you get a moment, I'm playing with that classify_handoff_parts.py script. I'm trying to figure out the meaning of the --request-node-count variable and how I should set it given a 9 server environment, one node directory per server with 20 devices mounted on each and replicas=3 on the ring. should it just be set to 1 since there's only one node per server? | 19:58 |
| drewn3ss | I see default == 3, which I'm guessing would be the three replicas | 19:59 |
| clayg | basically just use whatever you have for request_node_count in your proxies... | 19:59 |
| drewn3ss | gotchya | 19:59 |
| drewn3ss | thanks | 19:59 |
| clayg | the default value for your proxy to dig on 404's is like 2 * replicas - but that's total including primaries... i think... pretty sure | 19:59 |
| clayg | so if you have a bunch of "misplaced" parts - it's helpful to consider if *any* of the replicas of a part are on a primary location - if things go *way* south you can turn up request_node_count == <really_big_number> - a lot of times you'll find something *eventually* - assuming you can trade latency for more correctness-ish | 20:01 |
| clayg | or in a global cluster with write affinity sometimes you have a bunch of handoffs - which is different than mispalce parts from a rebalance... idk, YMMV | 20:01 |
| drewn3ss | intriguing. | 20:01 |
| clayg | increasing request_node_count makes more parts be "handoffs" instead of "misplaced" | 20:02 |
| drewn3ss | I did notice that. | 20:02 |
| drewn3ss | is that because rings keep copies of previous locations of the parts? | 20:02 |
| clayg | it does *not* track "previous locations" of parts - but eventually every disk in the cluster is a handoff for any given part - the question isn't "if" it's "when" | 20:03 |
| drewn3ss | got it | 20:04 |
| clayg | https://youtu.be/ger20cqOypE?t=538 maybe helpful? | 20:04 |
| drewn3ss | thanks again! | 20:04 |
| *** joeljwright has quit IRC | 20:05 | |
| clayg | sorry you're having a tough time of it - but you know... at least it's all open source - you'll get it figured out! | 20:06 |
| drewn3ss | lol +1k to that | 20:06 |
| *** gyee has quit IRC | 20:07 | |
| *** MVenesio has quit IRC | 20:12 | |
| drewn3ss | that vid was super helpful | 20:12 |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** Sukhdev has quit IRC | 20:21 | |
| *** joeljwright has joined #openstack-swift | 20:24 | |
| *** ChanServ sets mode: +v joeljwright | 20:24 | |
| *** patchbot has quit IRC | 20:38 | |
| *** patchbot has joined #openstack-swift | 20:39 | |
| *** joeljwright has quit IRC | 20:39 | |
| *** PagliaccisCloud has quit IRC | 20:39 | |
| *** PagliaccisCloud has joined #openstack-swift | 20:45 | |
| *** joeljwright has joined #openstack-swift | 20:47 | |
| *** ChanServ sets mode: +v joeljwright | 20:47 | |
| *** notmyname has quit IRC | 20:48 | |
| *** m_kazuhiro has joined #openstack-swift | 20:53 | |
| *** catintheroof has quit IRC | 20:53 | |
| *** catintheroof has joined #openstack-swift | 20:54 | |
| *** silor has quit IRC | 20:54 | |
| *** notmyname has joined #openstack-swift | 20:55 | |
| *** ChanServ sets mode: +v notmyname | 20:55 | |
| notmyname | meeting in a few minutes | 20:56 |
| notmyname | just was having some bouncer connectivity issues, so I hope this works | 20:56 |
| kota_ | morning | 20:57 |
| timburke | kota_: o/ | 20:58 |
| kota_ | timburke: o/ | 20:58 |
| *** catintheroof has quit IRC | 20:58 | |
| *** joeljwright has quit IRC | 21:00 | |
| notmyname | wooo! meeting time! yay! | 21:00 |
| torgomatic | timburke: so it seems that, at cluster-upgrade time, there will be N items in the cluster, and 0 <= F << N broken items in the cluster. If we add checksums, we protect all (N - F) good objects, at a cost of also protecting the F bad objects. | 21:04 |
| *** joeljwright has joined #openstack-swift | 21:07 | |
| *** ChanServ sets mode: +v joeljwright | 21:07 | |
| torgomatic | So, I could see my way to adding checksums to everyone. Worst case, we put our stamp of approval on a very few objects with corrupt metadata, but in exchange, we get to prevent silent metadata corruption for the vast majority of things | 21:07 |
| timburke | yeah, my gut feeling is that it's probably a net-gain to put it on everything we can -- i just didn't like the inconsistent application | 21:09 |
| torgomatic | agreed | 21:09 |
| *** gyee has joined #openstack-swift | 21:10 | |
| *** gyee has quit IRC | 21:10 | |
| *** gyee has joined #openstack-swift | 21:11 | |
| *** gyee has quit IRC | 21:11 | |
| *** newmember has quit IRC | 21:27 | |
| *** newmember has joined #openstack-swift | 21:27 | |
| *** Sukhdev has joined #openstack-swift | 21:29 | |
| timburke | clayg: it's frozen, except for it's not, and i totally see where it's *also* not going to be frozen in the near term | 21:41 |
| torgomatic | frozen isn't necessarily a binary state... maybe it's more of a thick slush | 21:42 |
| timburke | i guess upstream is frozen? but i've gotta keep going, on a timeline that's shorter than we could hope to get s3api on master | 21:42 |
| clayg | soft freeze + maintanace mode + we're liars - I wasn't laking words so much as clear picture of reality - reality is muddy | 21:42 |
| *** joeljwright has quit IRC | 21:59 | |
| notmyname | FWIW, as I look to the next release, ideally I'd like to see the slow-SLO patch, the check xattrs patch, and the SLO data-segments patch | 22:00 |
| timburke | that should make joeljwright happy :-) | 22:00 |
| clayg | i wonder if in general using sub manifests can help ... how long does 1K HEAD requests really take? oh... hrm... yeah I guess it can take awhile - but we use some parallelization? | 22:01 |
| acoles | good night | 22:02 |
| timburke | clayg: for my purposes, it really doesn't. i want it for swift3, where i wanna do 10k segments in a single client-request-cycle | 22:02 |
| timburke | good night acoles! | 22:02 |
| kota_ | acoles: g'night | 22:03 |
| acoles | timburke: I just noticed that there are no pending reviews on feature/deep today (other than WIP patches) | 22:03 |
| timburke | cranking concurrency helped, but not enough | 22:03 |
| kota_ | tdasilva: do | 22:03 |
| kota_ | sorry | 22:04 |
| timburke | acoles: guess it's time to merge? ;-) | 22:04 |
| kota_ | do we have something to discuss more? | 22:04 |
| tdasilva | kota_: hi, just wanted to check on your plans for the work on the s3api feature branch | 22:04 |
| acoles | timburke: I'll head to bed, you take care of that ;) | 22:04 |
| tdasilva | acoles: o/ | 22:04 |
| timburke | acoles: no, i'll try to get some more probe tests written. i really want to see a shard split | 22:05 |
| kota_ | tdasilva: i see | 22:05 |
| acoles | timburke: +1 | 22:05 |
| timburke | but i think i might need to get some plumbing into the sharder to limit a run to particular parts | 22:05 |
| kota_ | tdasilva: the basic plan in near future is starting to work on the item in the commit message at https://review.openstack.org/#/c/512284/ | 22:05 |
| patchbot | patch 512284 - swift (feature/s3api) - Import all swift3 code as s3api | 22:05 |
| timburke | ...which will be a little tricky to use given our current probe test framework | 22:06 |
| kota_ | tdasilva: as you already did for docs migration, right? | 22:06 |
| timburke | (i think. idk. i'll look around for other precedents) | 22:06 |
| tdasilva | kota_: yeah, I've basically started on points 1 and 2 on that list | 22:06 |
| clayg | **10k segments** !~!!!! | 22:06 |
| timburke | YUP | 22:06 |
| kota_ | tdasilva: oh you did 1 and 2 too? | 22:07 |
| kota_ | i see | 22:07 |
| tdasilva | kota_: that's patch https://review.openstack.org/#/c/513170/1 | 22:07 |
| patchbot | patch 513170 - swift (feature/s3api) - Continue merge swift3 middleware | 22:07 |
| clayg | ok then, great! no wonder they designed their api the way they did! | 22:07 |
| timburke | we crank through 'em remarkably well. i think i was seeing 6k or 8k getting processed in like 1m10s or so? but the client timed out after 50s :-( | 22:07 |
| kota_ | i just looked at a few changes on the top of change list :P | 22:07 |
| tdasilva | kota_: I have a new patchset almost ready, but not yet... | 22:07 |
| *** caiobrentano_ has quit IRC | 22:07 | |
| timburke | of course, i think that was probably the *only* thing really going on in that cluster, so... | 22:08 |
| timburke | ymmv | 22:08 |
| kota_ | tdasilva: k, let me know. If the new patch set is ready. | 22:09 |
| clayg | timburke: what concurrency? roughly? | 22:09 |
| timburke | 10, but it only really helped by like 2x | 22:09 |
| *** openstackgerrit has joined #openstack-swift | 22:09 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Remove swift-temp-url man page. https://review.openstack.org/513194 | 22:09 |
| tdasilva | kota_: so I started moving the unit tests around, but I still have some unit tests failing. I'm not sure if I will be able to get back to it tonight. | 22:10 |
| timburke | i should see what happens when you go to delete one of those beasts... with ?multipart-manifest=delete, that is... | 22:10 |
| clayg | eventually you have a few outstanding requests to every disk... with 8K segments even on a 20 node cluster... | 22:10 |
| *** m_kazuhiro has quit IRC | 22:10 | |
| tdasilva | kota_: I'm going to submit the patchset and if you would like and have the cycles you can continue with that, otherwise I will continue in my morning | 22:10 |
| clayg | zohno!? more heartbeat=on? | 22:11 |
| kota_ | tdasilva: if you push the patch as it is, i could look at the errors and get the reasons | 22:11 |
| tdasilva | kota_: is that a good plan? | 22:11 |
| clayg | or... it already does the bulk delete api? | 22:11 |
| timburke | we definitely piggyback off the bulk delete infrastructure -- but i'm not sure if the eventlet min write size things gets set properly? | 22:11 |
| clayg | coolio! | 22:12 |
| kota_ | tdasilva: sounds good | 22:12 |
| kota_ | oh, you deleted the lines for pep8 errors https://review.openstack.org/#/c/512284/1..2/swift/common/middleware/s3api/swift3/test/functional/test_object.py | 22:12 |
| patchbot | patch 512284 - swift (feature/s3api) - Import all swift3 code as s3api | 22:12 |
| kota_ | :/ | 22:12 |
| clayg | shit, patch 499634 still needs to be merged :'( | 22:13 |
| patchbot | https://review.openstack.org/#/c/499634/ - swift - Respect co-builder partition moves when min_part_h... | 22:13 |
| kota_ | tdasilva: sort of https://review.openstack.org/#/c/512543/ would be nice change to happen. | 22:13 |
| patchbot | patch 512543 - swift3 - Don't merge: Check if email can be used to replace... | 22:13 |
| kota_ | timburke had better idea though | 22:13 |
| timburke | kota_: i'd glanced at that -- yeah, what do you think about just using what's in swift/common/utils? | 22:14 |
| openstackgerrit | Thiago da Silva proposed openstack/swift feature/s3api: Continue merge swift3 middleware https://review.openstack.org/513170 | 22:14 |
| kota_ | timburke: sounds excellent | 22:14 |
| kota_ | I just picked up some test code from tests :P | 22:14 |
| kota_ | better module is nice to me | 22:14 |
| tdasilva | kota_: +1 | 22:16 |
| *** gyee has joined #openstack-swift | 22:17 | |
| tdasilva | kota_: I had thought about merging code as is, and then applying the change to fix that test, but either way works for me... | 22:17 |
| kota_ | ok, I'll update the patch 512284 and look at patch 513170 | 22:17 |
| patchbot | https://review.openstack.org/#/c/512284/ - swift (feature/s3api) - Import all swift3 code as s3api | 22:17 |
| patchbot | https://review.openstack.org/#/c/513170/ - swift (feature/s3api) - Continue merge swift3 middleware | 22:17 |
| tdasilva | kota_: cool, thanks! | 22:17 |
| tdasilva | kota_: and I will try to continue in my morning from wherever you have left off | 22:17 |
| kota_ | tdasilva: thx! | 22:17 |
| *** patchbot has quit IRC | 22:19 | |
| *** notmyname has quit IRC | 22:19 | |
| *** notmyname has joined #openstack-swift | 22:21 | |
| *** ChanServ sets mode: +v notmyname | 22:21 | |
| *** patchbot has joined #openstack-swift | 22:21 | |
| * kota_ is leaving to get breakfast | 22:21 | |
| kota_ | see you later | 22:21 |
| patchbot | back | 22:22 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Stop logging tracebacks on bad xLOs https://review.openstack.org/496493 | 22:40 |
| *** Sukhdev has quit IRC | 23:00 | |
| *** vint_bra has quit IRC | 23:12 | |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Be more tolerant of exception messages from sqlite https://review.openstack.org/513206 | 23:15 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Add checksum to object extended attributes https://review.openstack.org/336323 | 23:18 |
| torgomatic | timburke: notmyname: for xattr checksums, for existing objects, I was going to make the object auditor add metadata checksums, as well as an additional attribute that says when the checksum was added. Is it important to make ssynced objects behave exactly the same way, or can we let them get checksums like new objects do and call it close enough? | 23:23 |
| notmyname | "... like new objects"? if ssync finds an object without an xattr checksum and moves it to a new node, does it write down an xattr checksum? | 23:25 |
| torgomatic | It would unless we took steps to prevent it from doing so | 23:26 |
| *** chsc has quit IRC | 23:27 | |
| notmyname | seems like we should make ssync either not write it down (and wait for the auditor to do its thing) or write it down with the added "this is when the xattr checksum was added". | 23:27 |
| torgomatic | I mean, for perfection's sake, yes. I want to do the slightly lazier thing and not bother with it, figuring that most objects aren't going to get moved to new nodes before the auditor gets a chance to visit them. It's one less code path to maintain. | 23:28 |
| timburke | hmm... as it is, we're writing down a new metadata checksum *every time* we move data via ssync, which it almost certainly *needs* to do, because the exact on-disk serialization may have changed (dictionary iteration being what it is) | 23:28 |
| torgomatic | Right; the checksum isn't going to be the same between all 3 replicas. It's a purely local property. | 23:29 |
| *** mjturek has quit IRC | 23:31 | |
| torgomatic | If we want everything perfect, then newly-PUT objects would get metadata + checksum + no checksum-added timestamp, and existing objects would have metadata and get checksum + checksum-added timestamp | 23:31 |
| notmyname | so the difference is that with an object that already has xattr checksums.... it's validated on read? | 23:31 |
| notmyname | like the data xhecksums | 23:32 |
| torgomatic | and that requires code in ssync to propagate the fact that an object either has or has not a metadata checksum when we ship it to the other node | 23:32 |
| notmyname | so that means that the new stuff , the ones without the checksum on xattrs, would be read, sent via ssync, and written down with an xattr checksum without checking if the xattrs are right (becauyse there's no way to check it) | 23:33 |
| torgomatic | if we can be lazy and *not* put that code in ssync, then we can only think about checksums in three places: diskfile.read_metadata(), diskfile.write_metadata(), and the object auditor | 23:33 |
| notmyname | which doesn't seen that different that what we're doing today with all data | 23:33 |
| torgomatic | notmyname: correct; it's better than what we're doing today, which is hoping that the old metadata is good, writing it down, and then hoping that it stays good in its new home | 23:33 |
| notmyname | this disadvantage is that for an object that gets ssync'd before auditing, we have no way to know if the metadata is what the user set. (like today with no checksums) | 23:34 |
| torgomatic | yep | 23:34 |
| notmyname | just that the ssync-set check doesn't have the "this is when we set the xattr checksum" value | 23:34 |
| torgomatic | but the corresponding advantage is that we don't have to think about checksums nearly as often | 23:35 |
| notmyname | the extra checksum set-at date is only for a migration period. for a cluster that's deployed after your patch lands, 100% of the data will never have that value, because the xattr checksum will always be there when the auditor looks | 23:36 |
| torgomatic | correct | 23:36 |
| notmyname | so we're talking about the period of time between when old clusters upgrade and finish an audit cycle | 23:37 |
| notmyname | which we hope to be about 1 month | 23:37 |
| notmyname | (hope being the operative word) | 23:37 |
| torgomatic | yes | 23:37 |
| clayg | WHO AMOUNG US STILL HAS HOPE!!! | 23:38 |
| clayg | i will crush you | 23:38 |
| torgomatic | although now that I think about it, I'm starting to wonder if we need the timestamp at all | 23:38 |
| notmyname | ok. I can live with that. I think it needs some fairly clear descriptions to set expectations | 23:38 |
| torgomatic | an operator will know about when they upgraded their Swift version, so if a user complains about metadata being wrong, they can check the object's X-Timestamp and see if it predates the upgrade | 23:38 |
| notmyname | hmm.. true | 23:39 |
| torgomatic | if it does, they can just be like "sorry dude, guess you got your bits flipped" | 23:39 |
| notmyname | and I'd definitely like to make sure briancline onovy rledisez and other big deployers know about this when they think about upgrades | 23:40 |
| notmyname | torgomatic: yeah, I'm coming around to the "don't need the timestamp at all" view | 23:40 |
| thurloat | quick q: I'm seeing an increasing number of connection timeouts to container servers, will upping the workers combat this? there's no IO wait, container and account data is on SSDs | 23:43 |
| torgomatic | thurloat: might help | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!