Thursday, 2018-12-06

*** gyee has quit IRC00:17
openstackgerritTim Burke proposed openstack/swift master: Support long-running multipart uploads
*** Belgar81 has joined #openstack-swift00:55
timburkeclayg: tdasilva:
patchbotpatch 617046 - swift - s3api: Change default location to us-east-1 - 3 patch sets00:56
openstackgerritTim Burke proposed openstack/swift master: s3api: Allow some forms of server-side-encryption
timburkegood call clayg -- that's way better ^^^01:19
notmynameoh cool. is that the first time we're using admin swift info?01:22
timburkei think maybe01:22
timburkei love the idea of that being more than a client-facing tool though01:23
notmynametimburke: ugh. the check_drive patch failed after 6 hours with a probetest failure01:48
notmynamedo you remember if I can recheck now, or do I have to wait until after zuul reports back?01:48
*** itlinux has joined #openstack-swift01:50
notmynameI'll check it later tonight. hopefully we can have it landed by our morning tomorrow01:53
*** mahatic has quit IRC02:01
mattoliveraunotmyname: I'll keep an eye on it and recheck it at the end of my day if it hasn't landed by then.02:02
*** psachin has joined #openstack-swift03:32
openstackgerritMerged openstack/python-swiftclient master: Change openstack-dev to openstack-discuss
openstackgerritMerged openstack/swift master: fixed _check_node() in the container sharder
openstackgerritMerged openstack/python-swiftclient master: Add delimiter to get_account().
*** e0ne has joined #openstack-swift07:38
*** rcernin has quit IRC07:56
*** pcaruana has joined #openstack-swift07:58
*** pcaruana is now known as muttley07:58
*** rcernin has joined #openstack-swift08:03
*** gkadam has joined #openstack-swift08:16
*** rcernin has quit IRC08:33
*** admin6_ has joined #openstack-swift08:34
*** admin6 has quit IRC08:37
*** admin6_ is now known as admin608:37
zigoIs there an Android app to do a phone backup using swift natively?08:48
*** mikecmpbll has joined #openstack-swift08:51
*** mahatic has joined #openstack-swift08:59
*** ChanServ sets mode: +v mahatic08:59
*** admin6_ has joined #openstack-swift09:28
*** admin6 has quit IRC09:31
*** admin6_ is now known as admin609:31
*** admin6_ has joined #openstack-swift09:36
*** admin6 has quit IRC09:39
*** admin6_ is now known as admin609:39
*** e0ne has quit IRC10:24
*** e0ne has joined #openstack-swift10:27
*** mvkr has joined #openstack-swift10:36
*** Belgar81 has quit IRC10:54
*** admin6_ has joined #openstack-swift11:07
*** admin6 has quit IRC11:10
*** admin6_ is now known as admin611:10
*** admin6_ has joined #openstack-swift11:34
*** admin6 has quit IRC11:37
*** admin6_ is now known as admin611:37
*** NM has joined #openstack-swift11:55
*** gkadam_ has joined #openstack-swift12:00
*** gkadam has quit IRC12:03
*** gkadam_ has quit IRC12:03
*** gkadam has joined #openstack-swift12:05
*** psachin has quit IRC12:08
*** gkadam_ has joined #openstack-swift12:55
*** gkadam has quit IRC12:58
*** admin6_ has joined #openstack-swift13:00
*** admin6 has quit IRC13:03
*** admin6_ is now known as admin613:03
*** muttley has quit IRC13:08
*** rcernin has joined #openstack-swift13:17
*** muttley has joined #openstack-swift13:21
*** muttley has quit IRC13:25
*** muttley has joined #openstack-swift13:26
*** rcernin has quit IRC13:29
*** muttley has quit IRC13:29
*** pcaruana has joined #openstack-swift13:34
*** kota_ has quit IRC13:36
*** kota_ has joined #openstack-swift13:38
*** ChanServ sets mode: +v kota_13:38
*** pcaruana has quit IRC13:39
*** pcaruana has joined #openstack-swift13:43
*** pcaruana has quit IRC13:47
*** hseipp has joined #openstack-swift13:50
*** dr_gogeta86 has joined #openstack-swift13:55
openstackgerritMerged openstack/swift master: api-ref: Clean up container and object response code formatting
*** psachin has joined #openstack-swift14:32
*** admin6_ has joined #openstack-swift15:16
*** admin6 has quit IRC15:18
*** admin6_ is now known as admin615:18
*** gkadam_ has quit IRC15:19
*** admin6_ has joined #openstack-swift16:04
*** admin6 has quit IRC16:07
*** admin6_ is now known as admin616:07
*** takamatsu has quit IRC16:10
*** SkyRocknRoll has joined #openstack-swift16:23
*** e0ne has quit IRC16:30
*** admin6 has quit IRC16:32
*** psachin has quit IRC16:41
*** NM has quit IRC17:01
*** NM has joined #openstack-swift17:04
*** hseipp has quit IRC17:21
notmynamegood morning17:41
notmynamepatch 59223117:41
patchbot - swift - s3api: Include '-' in S3 ETags of normal SLOs - 6 patch sets17:41
*** NM has quit IRC17:43
*** mikecmpbll has quit IRC17:47
claygdoes s3api really just hide any container that happens to end in `_segments`?17:49
*** NM has joined #openstack-swift17:52
*** gyee has joined #openstack-swift17:55
claygtimburke: p 618301 has no tests 🙄17:58
patchbot - swift - s3api: Allow concurrent multi-deletes - 3 patch sets17:58
timburkeclayg: but it didn't *break* any tests, either! :P18:07
*** e0ne has joined #openstack-swift18:08
timburkeclayg: and no, it's not _segments that's special (though maybe it'll hide that, too?) -- it uses +segments for MPUs specifically because + isn't valid under either the old or new bucket naming rules18:08
timburkeif a bucket name isn't valid, why return it?18:08
*** e0ne has quit IRC18:09
claygTIL "The bucket name cannot contain underscores"18:11
timburkebu it *used* to be able to, iirc. set dns_compliant_bucket_names=false and it should show up18:16
timburke"The legacy rules for bucket names in the US East (N. Virginia) Region allowed bucket names to be as long as 255 characters, and bucket names could contain any combination of uppercase letters, lowercase letters, numbers, periods (.), hyphens (-), and underscores (_)."18:16
timburkeso there are some ceph s3 tests that still test 250 chars, 251, 252, 253, 254, 255, 256, 280, and 3000 :P18:18
*** SkyRocknRoll has quit IRC18:29
NMHi everyone!  Is anyone using any feature/tool/middleware to undelete objects? I found swift_undelete on Github but it has been while since the project was updated.18:34
timburkeNM: i'd go with something like the history mode for versioned writes, personally -- are you looking to have it apply cluster-wide, or just for a container or two?18:38
timburkeif we ever get the defaulter merged (see p 342857) we could even have versioned_writes look for and apply a default x-delete-after header as part of the copy...18:40
patchbot - swift - Add defaulter middleware - 7 patch sets18:40
notmynamezigo: I have no idea if there's any native android->swift backup tool. but weirdly I did just see this on twitter:
notmynamezigo: seems like maybe they do some sort of backup to swift?18:48
NMtimburke: right now, just for a container. Someone made a implementation of x-delete-at and after some bad results people are asking if we could offer an undelete feature. I think versioning won't protect the objects for this case…18:48
openstackgerritTim Burke proposed openstack/swift master: s3api: Allow some forms of server-side-encryption
timburkeNM: yeah, versioning and object expiration don't exactly mesh well...18:56
openstackgerritMerged openstack/swift master: s3api: Include '-' in S3 ETags of normal SLOs
NMtimburke: thank you anyway. 👍🏼19:21
* clayg I'm trying to boto and seeing:19:26
claygDEBUG:boto:encountered gaierror exception, reconnecting19:26
clayggaierror is like a get hostname call or something right?  I tried `host=''` and saio... i don't know what the actual error is - but my syslog isn't showing any requests19:27
notmynamefungi: I'm seeing some odd things in zuul. specifically, I've seen two patches with tests reported failed in the dashboard but they've both merged with no rechecks19:29
notmyname merged just find with no reported errors, but this morning clarkb pointed me at which is a failed job19:29
patchbotpatch 592231 - swift - s3api: Include '-' in S3 ETags of normal SLOs (MERGED) - 6 patch sets19:29
notmynameand last night landed but I saw while it was in the gate queue with a big red x next to it19:29
patchbotpatch 621727 - swift - fixed _check_node() in the container sharder (MERGED) - 5 patch sets19:30
*** mikecmpbll has joined #openstack-swift19:30
timburkeclayg: probably trying to connect to <bucket>.saio or something?19:30
notmynamesorry.. saw
claygaparently that's the failure mode if you try and skip the `calling_format=boto.s3.connection.OrdinaryCallingFormat()` bit from examples!?19:30
clayg@timburke brilliant!  DEBUG:boto:Host: lots-of-files.
notmynamefungi: the hash after the test name is different for the ones reporting errors (comparing the failing log with the reported log that passed), but I don't know what that means19:31
timburkenotmyname: most likely another ahead of it in the queue failed (most likely for another project), so *all* the jobs for the patch that merged got re-enqueued without the other project's patch applied, and *then* the jobs all passed19:31
notmynametimburke: ah, maybe so.19:32
notmynameI think I've noticed it because it's happened twice in the last 12 hours or so19:32
timburkesometimes i kinda wish we didn't get all the auto-retries, just to force us to fix our flaky tests... not real often though ;-)19:32
funginotmyname: looks like 592231 failed build 7bde795 but then another change ahead of it failed and so all the jobs for that change were rerun again without the failing change ahead of it. if you look at the items list for you'll see what other changes were included with that run19:32
notmynamefungi: ah ok. so what timburke said about the job being re-enqueued19:33
fungiright. zuul won't consider job results from a dependent pipeline valid if there's a change with failing jobs "ahead" of that one19:34
timburkeclayg: i wonder if i could convince boto to take a patch to check whether a host is actually an IP address and if so, skip straight to the bucket-in-path request...19:34
timburkewas it boto or boto3?19:34
fungiso it moves the earliest failing change aside and then reparents all the subsequent changes to the nearest non-failing change19:34
notmyname...which gives a definitive answer to something I've been wondering. always wait for zuul to report back with failure *before* sending a recheck19:35
fungiwell, yes that too. unless zuul completely dequeues a change first it's going to ignore recheck comments19:35
notmynamebecause if it's a transient failure, failing stuff ahead of it will reenqueue it (and earlier than a human recheck would place it)19:36
fungiright, for example you don't want a bad nova patch in a change ahead of yours to cause a devstack failure in your swift change and have it be ejected as a bad patch19:37
fungiso we try the switch change again without the failing nova change to make sure that wasn't the reason the swift change was also failing19:37
notmynamefungi: so what you're saying is that as long as our patches have the most stable tests, then the system automatically works most optimally for us :-)19:37
notmynamestable tests + few dependencies = best possible openstack gate queues19:38
timburkenotmyname: but few dependencies won't necessarily help us -- as long as we're in the integrated gate queue, we'll have ready-to-merge code getting re-enqueued because of someone else's failure19:40
notmynametimburke: I just mean that if we don't have a dependency on nova, then it's impossible for a nova patch to break our tests. meaning that given the reality that nova and swift patches are tested together, we only need to "handle" failures based on our own code19:42
timburkemmm -- true19:42
*** mikecmpbll has quit IRC19:54
*** rcernin has joined #openstack-swift20:12
openstackgerritMerged openstack/swift master: Change openstack-dev to openstack-discuss
*** NM has quit IRC20:45
*** NM has joined #openstack-swift20:47
*** NM has quit IRC20:53
sorrisonHi all, we upgraded our swift cluster to 2.15 about a week ago and now our accounts don't seem to be updating their object count or size, container count works fine though. Any ideas where to look?21:14
timburkesorrison: have you looked at the container-updater logs?21:16
timburkealso, sorry -- i've been meaning to look at p 603529 -- i think i've even got a decent setup to test it ATM...21:17
patchbot - swift - s3 secret caching - 10 patch sets21:17
sorrisontimburke: nothing in logs for container updater. ran it manually on one host and got:21:21
sorrisonContainer update single threaded sweep completed: 8.64s, 20 successes, 0 failures, 414 with no changes21:21
sorrison[updater_last_sweep] low: 2, high: 34, avg: 21.3, total: 361, Failed: 0.0%, no_result: 0, reported:21:22
sorrisontimeburke: So the container-updater is the process that updates the account with the number of objects and size of objects?21:41
sorrisontimburke: ^21:41
timburkesorrison: k, pretty sure you're getting bit by bug 1710328 which was fixed in -- can you upgrade to 2.15.1 instead of 2.15.0? alternatively, i know we did some backports for that guy -- tip of stable/ocata should have
openstackbug 1710328 in OpenStack Object Storage (swift) "object server deadlocks when a worker thread logs something" [High,Fix released] - Assigned to Samuel Merritt (torgomatic)21:42
timburkeyeah, container-updater takes the stats that got rolled up at the container level and sends them on to the account21:43
sorrisonWe are running 2.15.1 from ubuntu cloud archive package21:43
sorrison2.15.1-0ubuntu3~cloud0 to be precise21:43
sorrison(which is pike)21:43
sorrisonJust checked and the code for  is in the version we running21:45
timburkeah! i was thinking of :-)21:45
openstackLaunchpad bug 1722951 in OpenStack Object Storage (swift) "Container updater may be stuck and not make progress" [High,Fix released] - Assigned to Samuel Merritt (torgomatic)21:45
sorrisonok I see, fixed in 2.16.0 we are going to upgrade to the queens version next week so should all be good then21:46
timburkenotmyname: should we maybe tag some stable releases so downstream knows that we've got some fixes for ocata and pike? 'cause this was fixed for pike, too, in
notmynametimburke: yeah, when we do the next release, we should just do a whole round of tagging for all the stable branches22:06
timburkenotmyname: in that case, i ought to take another look at what's landed since the various stable branches were cut and see if there's anything *else* we ought to backport22:07
notmynametimburke: there's *always* something else to land22:14
openstackgerritTim Burke proposed openstack/swift master: s3api: Allow concurrent multi-deletes

Generated by 2.15.3 by Marius Gedminas - find it at!