Thursday, 2016-11-24

*** catintheroof has joined #openstack-swift00:10
*** catintheroof has quit IRC00:11
-openstackstatus- NOTICE: Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved.00:13
*** ChanServ changes topic to "Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved."00:13
openstackgerritMerged openstack/liberasurecode: Fix reconstruct to return an error when memory allocation failed  https://review.openstack.org/38524000:18
*** arch-nemesis has quit IRC00:31
*** arch-nemesis has joined #openstack-swift00:34
openstackgerritMerged openstack/swift: go: Update README.md to clarify policies allowed  https://review.openstack.org/40028600:42
*** dmorita has quit IRC00:50
*** dmorita has joined #openstack-swift00:52
*** dmorita has quit IRC00:55
*** dmorita has joined #openstack-swift00:55
*** tqtran has quit IRC01:03
*** siva_krish has quit IRC01:06
*** diogogmt has quit IRC01:09
*** clu_ has quit IRC01:11
*** vint_bra has joined #openstack-swift01:23
*** nikivi has quit IRC01:31
*** arch-nemesis has quit IRC01:35
*** vint_bra has quit IRC01:38
*** vint_bra has joined #openstack-swift01:43
*** catintheroof has joined #openstack-swift01:50
timburkemattoliverau: on the py3 thing, i was using ascii as a nothing-up-my-sleeves kind of a move -- it should only ever be used when dealing with data that should be a very limited set of chars, like numeric (for the policy index header or the expiration time in tempurl's hmac) or hex (i think that was generally for etag-related things?)01:52
*** catintheroof has quit IRC01:55
*** catintheroof has joined #openstack-swift01:55
timburkethe latin1/utf8 stuff is a mess, though; pep-3333 seems to say env vars are native strings, and if that means unicode, then they have to be encodable to latin1. which means that when we finally have some py3 code speaking http, it'll be with some crazy "wsgi string" (which i have distinct in my head from a byte string, unicode string, or even native string) getting passed around as headers01:55
timburke(fwiw, "wsgi string" in my head means "native (byte) string for py2, or latin1-decoded utf8 string for py3")01:57
*** catinthe_ has joined #openstack-swift01:58
timburkeit's gonna be a mess, and it's gonna touch most things. so may as well start with something vaguely small-ish, like making ring building work. but that dragged in utils... so here we are.01:59
*** catintheroof has quit IRC02:00
*** vint_bra has quit IRC02:07
mattoliverautimburke: ahh ok, makes more sense, yeah you are only touching a few things with ascii, so that makes sense. Yeah this if going to touch everything and could look a little messy, which is why a this is the plan/how it works docs might be in order so we're all on the same page when dev or reviewing :)02:13
mattoliverauThanks for all the hard work here!02:13
*** diogogmt has joined #openstack-swift02:15
timburkemattoliverau: yeah, definitely. it's not a terribly high priority, though, so i'm not entirely sure when i'll get to it ;-)02:16
mattoliverau;)02:19
timburkebut on the plus side, that patch triples the number test files for the py3 jobs! (if you count the empty test/unit/common/ring/__init__.py)02:23
timburkebug 1644387 is unfortunate, though02:24
openstackbug 1644387 in OpenStack Object Storage (swift) "Ring builder files need a consistent format" [Undecided,New] https://launchpad.net/bugs/164438702:24
mattoliverauI know I count it.. we should add all the empty __init__.py's!02:38
mattoliverauyeah02:38
openstackgerritchen.xing proposed openstack/swift: Update a more simple rabbitmq configuration  https://review.openstack.org/39203002:57
openstackgerritchen.xing proposed openstack/swift: Make rabbitmq configuration much simpler  https://review.openstack.org/39203003:00
*** catinthe_ has quit IRC03:14
*** _JZ_ has joined #openstack-swift03:32
*** EmilienM has quit IRC03:50
*** EmilienM has joined #openstack-swift03:53
*** psachin has joined #openstack-swift04:03
*** dmorita has quit IRC04:28
*** SkyRocknRoll has joined #openstack-swift04:31
*** links has joined #openstack-swift04:45
*** smaktub has joined #openstack-swift05:14
smaktubHi!05:15
smaktubWe have been previously using Icehouse setup of SWIFT and CEILOMETER.05:15
smaktubIn Icehouse ceilometer, for incoming and outgoing bytes, Sample class structure with following fields was published onto ceilometer: name, type, unit, volume, user_id, project_id, resource_id, timestamp and resource_metadata.05:15
smaktubIs this structure changed in Mitaka version of ceilometer?05:15
smaktubI could see only the result(volume) and Metric being raised as an event and notified to ceilometer?05:15
*** dmorita has joined #openstack-swift05:18
*** ppai has joined #openstack-swift05:23
*** dmorita has quit IRC05:23
*** ChanServ changes topic to "Let's talk, we're nice. | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews"05:39
-openstackstatus- NOTICE: The affected filesystems on the log server are repaired. Please leave 'recheck' comments on any changes which failed with POST_FAILURE.05:39
*** sams-gleb has joined #openstack-swift06:24
*** ChubYann has quit IRC06:34
*** tqtran has joined #openstack-swift07:01
*** tqtran has quit IRC07:05
*** tesseract has joined #openstack-swift07:18
*** tesseract is now known as Guest1349107:19
*** rcernin has joined #openstack-swift07:21
*** hseipp has joined #openstack-swift07:38
openstackgerritMerged openstack/swift: Handle deleted files with if-none-match  https://review.openstack.org/39558208:11
*** smaktub has quit IRC08:32
donaghsmaktub: Swift does not send notifications to Ceilimeter. Instead Ceilimiter polls Swift to get account (and if you really must) container size data. So I suggest you ask in ceilometer room if there have been any changes.08:42
*** amoralej|off is now known as amoralej08:44
donaghsmaktub: Are you sure the fields you mention are related to Swift? One of the fields is "volume" --- that sounds more like Cinder than Swift.08:46
*** rledisez has joined #openstack-swift08:48
*** geaaru has joined #openstack-swift09:01
*** cbartz has joined #openstack-swift09:14
*** asettle has joined #openstack-swift09:24
*** asettle has quit IRC09:26
*** asettle has joined #openstack-swift09:27
*** admin6 has joined #openstack-swift09:29
*** _JZ__ has joined #openstack-swift09:34
*** bkopilov has quit IRC09:37
*** _JZ_ has quit IRC09:37
*** acoles_ is now known as acoles09:38
*** abalfour has quit IRC09:53
*** abalfour has joined #openstack-swift09:54
*** sams-gleb has quit IRC10:14
*** sams-gleb has joined #openstack-swift10:15
*** kei_yama has quit IRC10:15
*** sams-gleb has quit IRC10:19
*** 17SAAM6B5 has joined #openstack-swift10:47
*** sams-gleb has joined #openstack-swift10:52
*** Jeffrey4l has quit IRC10:52
*** SkyRocknRoll has quit IRC10:53
*** Jeffrey4l has joined #openstack-swift11:04
*** Jeffrey4l has quit IRC11:05
*** Jeffrey4l has joined #openstack-swift11:06
*** Jeffrey4l has quit IRC11:07
*** Jeffrey4l has joined #openstack-swift11:07
*** laci has joined #openstack-swift11:24
lacihi11:24
laciacoles, I managed my proxy<->keystone problem, thank you for the help11:25
laciacoles, I would like to use my swift as a s3 provider, how can I acquire access_key ans secret_key11:27
*** dmorita has joined #openstack-swift11:29
onovycschwede: yep, as i sad: don't lost your data guys :)11:30
*** geaaru has quit IRC11:31
cschwedeonovy: yes; i’m still looking into this, it  seems to get even more worse - will update the burgreport soon11:31
acoleslaci: glad you got keystone working. I'm not familiar with swift311:32
onovyi think every 10th run: https://github.com/openstack/swift/blob/stable/mitaka/swift/obj/replicator.py#L409 are synced all suffixies11:32
onovylaci: hi, try #swift3 channel, there are upstream authors of swift s3 plugin11:33
laciacoles, Thank you!11:33
*** dmorita has quit IRC11:33
*** 17SAAM6B5 has quit IRC11:35
*** links has quit IRC11:39
psachinping #openstack-swift, can any please review https://review.openstack.org/#/c/349199/ ?11:41
patchbotpatch 349199 - swift - Fix swift-get-nodes arg parsing for missing ring11:41
openstackgerritAlistair Coles proposed openstack/swift: Add pip and virtualenv upgrade to development guidelines  https://review.openstack.org/40198611:44
*** geaaru has joined #openstack-swift11:45
onovycschwede: do you want any help from us?11:47
cschwedeonovy: not yet, thx!11:48
onovyok11:48
cschwedeonovy: but it would be great if you could have a look later, if i’m right it’s even worse…11:48
onovydata lost already?11:48
onovywe didn't have disk/servers failure sync upgrade to mitaka...11:49
openstackgerritAlistair Coles proposed openstack/swift: Drop "--allow-insecure netifaces" from tox pip install  https://review.openstack.org/40199411:53
*** links has joined #openstack-swift11:54
*** vint_bra has joined #openstack-swift12:00
*** davideagle_ has joined #openstack-swift12:01
*** mvk has quit IRC12:15
cschwedeacoles: Hi! If you have some time, please have a log at the following bug: https://bugs.launchpad.net/swift/+bug/163496712:17
openstackLaunchpad bug 1634967 in OpenStack Object Storage (swift) "object-replicator in 2.7.0 and later doesn't replicate all suffixes" [Critical,In progress]12:17
cschwedeacoles: this seems critical to me12:17
*** catintheroof has joined #openstack-swift12:17
acolescschwede: ack. I was just reading the bug report12:20
cschwedethx12:21
onovycschwede: is Pavel patch fixies it? according to 'redo.sh'12:23
cschwedeonovy: yes, it fixes that for new suffixes, but not for missing suffixes which could take a long time12:23
onovyso applying patch + rm *.pkl will fix it :)12:24
cschwedeonovy: sure, that’s the quickest way to fix this :)12:24
cschwedeor maybe based on time? if hashes.pkl wasn’t changed for X hours/days, re-list the dir12:25
acolescschwede: I can't look immediately, will try to look later today12:26
onovycschwede: i think we could split it into two parts12:26
onovy1. fix it for new upgrades12:26
onovy2. found solution for already upgraded12:26
*** catintheroof has quit IRC12:26
onovy1b. found workaround for alread upgraded (rm *.pkl)12:26
cschwedeacoles: alright, thx a lot! i’ll be around today for some time, ping me if you want to discuss later on12:26
cschwedeonovy: sure; i tend to +2 the patch for now, and then we’ll add some follow ups12:27
*** amoralej is now known as amoralej|lunch12:29
*** catintheroof has joined #openstack-swift12:29
*** catintheroof has quit IRC12:31
*** tqtran has joined #openstack-swift13:04
*** geaaru has quit IRC13:05
*** oshritf has quit IRC13:08
*** tqtran has quit IRC13:09
*** links has quit IRC13:14
*** mvk has joined #openstack-swift13:17
*** geaaru has joined #openstack-swift13:22
openstackgerritPavel Kvasnička proposed openstack/swift: WIP: optimize hash calculation when suffix hash invalidated  https://review.openstack.org/40204313:30
openstackgerritMathias Bjoerkqvist proposed openstack/swift: Storing encryption root secret in Barbican  https://review.openstack.org/36487813:38
-openstackstatus- NOTICE: OpenStack CI has taken a Thanksgiving break; no new jobs are currently launched. We're currently hoping for a friendly admin to come out of Thanksgiving and fix the system.13:39
*** ppai has quit IRC13:55
davideagle_join /#openstack-swift13:56
patchbotError: You don't have the admin capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.13:56
onovycschwede: are you going to do follow up for (2), or should we work on it?13:57
*** amoralej|lunch is now known as amoralej14:00
*** oshritf has joined #openstack-swift14:03
*** laci has quit IRC14:06
davideagle_has anyone worked with swift3 (amazon s3 middleware) and tempauth, the proxy responds with a 403 Access denied error when I try to create a bucket, "s3curl --id credentials --createBucket --debug  -- http://localhost:8080/bucket1"14:07
*** Jeffrey4l has quit IRC14:13
*** Jeffrey4l has joined #openstack-swift14:17
*** Jeffrey4l has quit IRC14:18
davideagle_proxy-server.conf http://paste.openstack.org/show/590316/  and .s3curl file http://paste.openstack.org/show/590317/14:18
*** Jeffrey4l has joined #openstack-swift14:18
*** silor has joined #openstack-swift14:21
onovydavideagle_: hi, try #swift3 channel :)14:23
davideagle_thanks :)14:31
*** silor1 has joined #openstack-swift14:31
cschwedeonovy: i’ll propose a patch for an enforced do_listdir if hashes.pkl has not been updated within X days14:31
onovycschwede: cool! Pavel had same suggestion. "X days" should be lower than reclaim_age14:32
onovyideally < 1/2 of reclaim_age14:33
*** silor has quit IRC14:33
*** silor1 is now known as silor14:33
*** mvk has quit IRC14:39
-openstackstatus- NOTICE: OpenStack CI is processing jobs again. Thanks to the Canadian admin "team" that had their Thanksgiving holiday already ;) Jobs are all enqueued, no need to recheck.14:48
*** mvk has joined #openstack-swift14:53
*** oshritf has quit IRC14:58
*** oshritf has joined #openstack-swift14:59
*** oshritf has quit IRC15:01
*** oshritf has joined #openstack-swift15:02
*** openstackgerrit has quit IRC15:03
*** openstackgerrit has joined #openstack-swift15:04
*** _JZ__ has quit IRC15:05
*** sams-gleb has quit IRC15:05
*** sams-gleb has joined #openstack-swift15:06
*** oshritf has quit IRC15:07
*** oshritf has joined #openstack-swift15:08
*** PetrKubena has joined #openstack-swift15:09
*** sams-gleb has quit IRC15:11
*** PetrKubena has quit IRC15:12
*** oshritf has quit IRC15:13
*** sams-gleb has joined #openstack-swift15:24
*** silor1 has joined #openstack-swift15:27
*** silor has quit IRC15:27
*** silor has joined #openstack-swift15:28
*** diogogmt has quit IRC15:28
*** oshritf has joined #openstack-swift15:29
*** oshritf has quit IRC15:29
*** PetrKube_ has joined #openstack-swift15:30
*** Jeffrey4l has quit IRC15:30
*** PetrKube_ is now known as PetrKubena15:30
*** Jeffrey4l has joined #openstack-swift15:30
*** silor1 has quit IRC15:32
*** Jeffrey4l has quit IRC15:32
*** Jeffrey4l has joined #openstack-swift15:32
cschwedeonovy: so i have that patch ready, but i just realized this won’t fix it. the mtime of the hashes.pkl is likely to change quite often, and therefore the listdir will never be retriggered15:33
cschwedeonovy: on the other side, if you have partitions with archive data that never changes, it will recompute the hashes.pkl every reclaim_age/2, which is totally overkill then15:33
onovycschwede: ah, right15:34
onovyidea: add option: force_sync=True to config15:34
cschwedeonovy: i think a better solution is to simply list suffixes, and if they are not yet contained in the hashes.pkl then add them - this won’t recompute existing ones15:34
onovyor forst_listdir15:34
onovy*force15:34
onovycschwede: hmm15:35
onovyi think we need this15:35
onovy1. hashes can be not-consistent15:35
onovydoesn't matter why15:36
onovy2. sometimes recount it15:36
onovysometimes is run % 10 == 0 now15:36
onovywhich is not working in every case15:36
onovywe should recount 1/10 of partitions every replicator run, right?15:36
cschwedeonovy: yes; however the problem right now with the issue you found is that the suffixes are totally unknown, and that needs to be fixed quickly imo15:36
onovycschwede: yep, but we could fix both at once15:37
cschwedetrue15:37
onovyif we have counter of "replicator runs", why not recount partition_id%10=replication run?15:37
*** Jeffrey4l has quit IRC15:37
onovywe recount exact 1/10 of partitions every run15:37
cschwedeyeah, i was thinking about this already15:37
onovyafter 10th run, everything will be recounted15:37
onovyalmost this is already there15:38
onovybut it's 1/10 of random partitions15:38
onovyadd determinism here and we are done15:38
cschwedeexactly15:38
cschwedelet’s see15:38
onovy                do_listdir=(self.replication_count % 10) == 0,15:39
onovywe need new counter, replication_phases15:39
onovyincrement it every whole phase15:39
cschwedei think i have a solution, testing15:39
onovyok :)15:40
*** Jeffrey4l has joined #openstack-swift15:49
*** links has joined #openstack-swift15:52
*** links has quit IRC15:52
*** diogogmt has joined #openstack-swift16:12
acolesonovy: cschwede I am looking at patch 410330. I can clearly see the bug but I want to make sure I understand the impact16:15
patchbothttps://review.openstack.org/#/c/410330/16:15
acoleshey patchbot didn't you used to print the patch subject?16:16
cschwedepatch 40133016:16
patchbothttps://review.openstack.org/#/c/401330/ - swift - Fixed regression in consolidate_hashes16:16
acolesaaahh, thanks cschwede !16:17
cschwedeacoles: yw :)16:17
acolesso, on a node that has a replica of new object in a new suffix, consolidated_hashes loses the suffix for the new object (the bug), so the replicator is unaware it has a new object to replicate and its suffix hashes appears the same as a node that does not have the object :/16:19
*** Guest13491 has quit IRC16:19
acolesbut, every 10% or replication passes a recalculate is forced, at which point the replicator will spot the difference with the node that is missing the object, and will sync the object.16:20
*** rcernin has quit IRC16:20
cschwedeacoles: well, it’s non-deterministic - and during my testing i spotted a partition that was due too bad luck only fixed after 32 replication cycles16:21
acolescschwede: understood16:21
cschwedeacoles: so in addition to the fixed bug, i’m working on making the process deterministic, to ensure after 10 replication cycles all hashes.pkl are updated once16:21
acolesthen the REPLICATE after the rsync will force the receiving side to recalculate its suffixes so its hashes.pkl should be good, correct?16:22
acolesso what causes the reported increase in rsync traffic?16:23
cschwedeacoles: i think it was not increased if you take into account the whole sum of traffic over a period of time. however, due to the bug the replication traffic was a bit lower, and every 10th replication pass was syncing a whole partition then, because its suffix wasn’t replicated for some time16:26
*** oshritf has joined #openstack-swift16:26
cschwedeacoles: honestly, i didn’t check the rsync traffic during my tests, i was already worried enough about the missing suffixes and focused on that16:26
*** oshritf has quit IRC16:28
*** dmorita has joined #openstack-swift16:29
cschwedeonovy: you spotted the increased rsync traffic, right? ^^16:31
*** geaaru has quit IRC16:32
acolescschwede: reading bug report again, the increase is only when there is a mix of object server versions - which would be because get_hashes reports incorrect results on newer nodes (when there is a "lost suffix") so replicator fires off an rsync process (even though objects are in sync). when all nodes are upgraded then they all report the wrong get_hashes :/ so they think they are in sync and rsync is called less often16:33
acoles^^ that's my guess at least16:34
*** dmorita has quit IRC16:34
cschwedeacoles: ah yes, that’s the one issue if running a mixed cluster. however the underlying issue happens also on master, without running a mixed cluster16:34
acolesso there's two bugs really - 1. inconsistent suffix hashing between versions 2. failure to replicate lost suffixes16:35
*** oshritf has joined #openstack-swift16:35
acoles#2 being worse than #1 :(16:35
cschwedeacoles: yeah, i should have opened a second bug maybe…16:36
acolescschwede: IDK, they are related and there's only one fix so i think it's ok to leave as one16:36
acolescschwede: at first i didn't understand the rsync stats increase and was concerned it was a separate issue, but i think that explains it16:37
acolescschwede: I'll comment on launchpad16:37
*** geaaru has joined #openstack-swift16:44
*** mvk has quit IRC16:49
*** two_tired has joined #openstack-swift16:50
acolescschwede: I ownder if we could have a gate job that runs a script to create a bunch of objects then checkout the last N releases, call get_hashes and check all results are consistent?16:53
acoleswonder*16:54
*** hseipp has quit IRC16:54
cschwedeacoles: thx for looking into the bug and responding on lp16:55
cschwedeacoles: hmm, sounds a bit tricky due to different requirements? pyeclib comes to my mind. but might be possible?16:55
acolescould run the get_hashes in a venv?? anyway, add that to my list of "better testing" :)16:56
acolescschwede: I'll continue with review of patch now...16:56
*** asettle has quit IRC16:59
*** two_tired has quit IRC17:01
*** davideagle_ has quit IRC17:03
*** tqtran has joined #openstack-swift17:06
*** tqtran has quit IRC17:10
*** cbartz has quit IRC17:10
*** davideagle_ has joined #openstack-swift17:20
onovycschwede: yep17:31
onovya attached new graph17:31
onovywe have 2.7.0 only in production, and traffic is still increased17:31
onovythat information about "drop to normal" in bug is wrong17:32
onovybug updated17:33
onovyi think reason is simple: one of replica have different hashes.pkl (sender have wrong, receiver have right)17:34
onovyand hashes are fixed undeterministicly, so this situation is fixed in 1 of 10 runs17:36
onovyand i think it's one bug with one fix :)17:36
onovyafk, on the way to home17:37
*** rledisez has quit IRC17:37
*** jordanP has joined #openstack-swift17:39
*** davideagle_ has quit IRC17:46
*** jordanP has quit IRC17:48
*** geaaru has quit IRC17:49
*** jordanP has joined #openstack-swift17:49
*** davideagle_ has joined #openstack-swift17:50
*** two_tired has joined #openstack-swift17:52
*** oshritf has quit IRC18:03
*** psachin has quit IRC18:07
onovyback18:17
*** davideagle_ has quit IRC18:17
*** amoralej is now known as amoralej|off18:47
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873618:51
*** two_tired has quit IRC18:56
*** Jeffrey4l has quit IRC18:58
*** siva_krish has joined #openstack-swift19:07
openstackgerritAlistair Coles proposed openstack/swift: Add assertions to invalidate_hashes tests  https://review.openstack.org/40219919:11
acolescschwede: onovy I'll be around tomorrow to discuss patch 401330 more19:17
patchbothttps://review.openstack.org/#/c/401330/ - swift - Fixed regression in consolidate_hashes19:17
* acoles afk now19:17
*** acoles is now known as acoles_19:18
*** jordanP has quit IRC19:26
*** jordanP has joined #openstack-swift19:40
*** jordanP has quit IRC19:46
openstackgerritChristian Schwede proposed openstack/swift: Add missing suffixes to hashes.pkl during replication  https://review.openstack.org/40220419:48
*** amoralej|off is now known as amoralej19:56
onovycschwede: here?19:57
*** tqtran has joined #openstack-swift20:07
*** tqtran has quit IRC20:12
*** Renich has quit IRC20:38
*** siva_krish has quit IRC20:40
*** asettle has joined #openstack-swift20:48
*** asettle has quit IRC20:50
*** McMurlock1 has joined #openstack-swift21:07
*** McMurlock1 has quit IRC21:21
mattoliveraumorning21:22
*** sams-gleb has quit IRC21:37
*** sams-gleb has joined #openstack-swift21:37
*** vint_bra has quit IRC21:42
*** sams-gleb has quit IRC21:42
*** amoralej is now known as amoralej|off21:49
*** silor has quit IRC21:50
*** diogogmt has quit IRC21:58
*** davideagle has joined #openstack-swift22:31
*** vint_bra has joined #openstack-swift22:43
*** mvk has joined #openstack-swift23:09
*** vint_bra has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!