| *** rcernin has quit IRC | 02:02 | |
| *** threestrands has joined #openstack-swift | 02:06 | |
| *** gyee has quit IRC | 02:09 | |
| *** diablo_rojo has quit IRC | 02:48 | |
| *** rcernin has joined #openstack-swift | 02:52 | |
| kota_ | hello world | 03:24 |
|---|---|---|
| kota_ | mattoliverau: working at home is sometime hard to be online constantly because of interruption by kids :/ | 03:26 |
| *** psachin has joined #openstack-swift | 03:33 | |
| *** diablo_rojo has joined #openstack-swift | 05:04 | |
| zaitcev | How's Mutsuru | 05:06 |
| zaitcev | You tweeted he's got sick a few days ago | 05:06 |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-swift | 05:35 | |
| kota_ | zaitcev: thx zaitcev. he've been well. | 06:07 |
| kota_ | He got FLU and i received FLU from him :/ | 06:08 |
| kota_ | so we were at home for a while. | 06:08 |
| openstackgerrit | Merged openstack/swift stable/train: tests: Use timedelta to adjust dates, not string manipulations https://review.opendev.org/710901 | 06:15 |
| *** diablo_rojo has quit IRC | 06:17 | |
| *** threestrands has quit IRC | 06:34 | |
| *** threestrands has joined #openstack-swift | 06:35 | |
| *** threestrands has quit IRC | 06:36 | |
| *** threestrands has joined #openstack-swift | 06:36 | |
| *** threestrands has quit IRC | 06:37 | |
| *** threestrands has joined #openstack-swift | 06:38 | |
| *** threestrands has quit IRC | 06:39 | |
| *** zaitcev has quit IRC | 06:55 | |
| *** zaitcev has joined #openstack-swift | 07:12 | |
| *** ChanServ sets mode: +v zaitcev | 07:12 | |
| openstackgerrit | Merged openstack/swift master: Use float consistently for proxy timeout settings https://review.opendev.org/710475 | 07:29 |
| openstackgerrit | Merged openstack/swift master: Fix up some Content-Type handling in account/container listings https://review.opendev.org/533027 | 07:45 |
| *** tesseract has joined #openstack-swift | 08:09 | |
| *** rpittau|afk is now known as rpittau | 08:16 | |
| *** rdejoux has quit IRC | 08:20 | |
| *** rdejoux has joined #openstack-swift | 08:47 | |
| *** tkajinam has quit IRC | 09:30 | |
| alecuyer | DHE: didn't notice you were using ZFS before, are your top-level vdevs single disks, or do you have mirror/raidz ? Really interested to hear about your setup if that's something you can share | 09:56 |
| *** ccamacho has quit IRC | 10:02 | |
| *** ccamacho has joined #openstack-swift | 10:58 | |
| DHE | alecuyer: I'm migrating from ZFS to Swift because it's a better fit. the current array is 13 vdevs of 10-disks raidz2 plus a pair of SSDs for metadata/special allocation classes... | 10:59 |
| DHE | the intention is to do: zfs create pool/swift1 -o mountpoint=/srv/node/zfs1 # and slowly expand until it makes sense to nuke the pool and format each disk like a normal swift cluster | 11:00 |
| DHE | however today (unless you use the git version of zfs) any time an O_TMPFILE file is hardlinked into the filesystem, it's accompanied by an effective `zpool sync` which just killed performance so I had to disable that entirely | 11:01 |
| *** rcernin has quit IRC | 11:08 | |
| *** rpittau is now known as rpittau|bbl | 11:32 | |
| *** psachin has quit IRC | 11:33 | |
| *** psachin has joined #openstack-swift | 11:37 | |
| *** psachin has quit IRC | 11:49 | |
| *** openstackstatus has joined #openstack-swift | 11:51 | |
| *** ChanServ sets mode: +v openstackstatus | 11:51 | |
| *** mvkr has quit IRC | 11:58 | |
| *** jvisser has joined #openstack-swift | 12:17 | |
| *** mvkr has joined #openstack-swift | 12:19 | |
| *** zaitcev has quit IRC | 13:03 | |
| *** jvisser has quit IRC | 13:16 | |
| *** jvisser has joined #openstack-swift | 13:17 | |
| *** jvisser has quit IRC | 13:25 | |
| *** jvisser has joined #openstack-swift | 13:37 | |
| *** rpittau|bbl is now known as rpittau | 13:44 | |
| alecuyer | DHE: yes the metadata allocation class is great. One issue we found tho, is if you're going to be having one HDD = one ZFS pool, when the disk dies, you can't get rid of the zpool until you reboot. (maybe you can leave it be for a while and use another name for the replacement pool?). Have you had any issue with memory/CPU usage? | 14:00 |
| *** psachin has joined #openstack-swift | 14:16 | |
| DHE | alecuyer: we're not doing one disk per pool. yeah ZFS has a fit when the pool dies. I'm using the same pool storing my existing data and adding a swift dataset as I migrate. "swift" dataset grows, old dataset shrinks | 14:17 |
| DHE | although I'm going to be adding multiple swift datasets because per-"device" locks in the swift replicator are clearly a bottleneck and having tens of thousands of partitions on a single device is really really bad | 14:17 |
| DHE | the pool can take it, but swift locking can't | 14:17 |
| *** ccamacho has quit IRC | 14:18 | |
| alecuyer | That's right, good point :) | 14:19 |
| *** ccamacho has joined #openstack-swift | 14:20 | |
| *** ccamacho has quit IRC | 14:20 | |
| *** ccamacho has joined #openstack-swift | 14:23 | |
| *** psachin has quit IRC | 15:00 | |
| DHE | yeah. for ZFS I would make larger redundant pools, and then add multiple datasets depending on what size of disk you want to simulate. but ultimately I'm moving away from ZFS so this is just a stopgap while I move half a petabyte of data around | 15:31 |
| *** psachin has joined #openstack-swift | 15:41 | |
| alecuyer | oh moving away.. I understood you were moving to. but thanks for the feedback! | 15:52 |
| *** psachin has quit IRC | 15:56 | |
| *** psachin has joined #openstack-swift | 16:05 | |
| *** gyee has joined #openstack-swift | 16:40 | |
| *** psachin has quit IRC | 16:53 | |
| *** rdejoux has quit IRC | 17:12 | |
| *** openstackgerrit has quit IRC | 17:20 | |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-swift | 17:35 | |
| *** rpittau is now known as rpittau|afk | 17:44 | |
| *** zaitcev has joined #openstack-swift | 18:00 | |
| *** ChanServ sets mode: +v zaitcev | 18:00 | |
| *** diablo_rojo has joined #openstack-swift | 18:11 | |
| *** guimaluf has joined #openstack-swift | 18:38 | |
| *** tesseract has quit IRC | 18:55 | |
| guimaluf | hi timburke & clayg any ideas on the O_TMPFILE issue? | 19:17 |
| timburke | guimaluf, so the new volume worked on the old server, and the old volume didn't work on the new server, is that right? | 19:25 |
| timburke | makes it seem like an issue with the filesystem... but then i don't have a good explanation for why it *used to* work, pre-upgrade... | 19:26 |
| guimaluf | timburke, no. old and new volume worked on new server. old and new volume didn't work in the old server. I think is this. but I'll recheck | 19:26 |
| timburke | ok! starts to make more sense :-) | 19:26 |
| timburke | have you compared the mount options? sorry, i'm kinda grasping at straws | 19:27 |
| guimaluf | timburke, that's what I'm trying to discover :/ what was left on this upgraded server that mess with the filesystem | 19:27 |
| guimaluf | timburke, don't worry. every recheck worth it. but yeah, I'm using the same mount options. I'll redo the volume swappping between servers just to ensure. if you have any advice on things to look for, please let me know :) | 19:28 |
| timburke | does the write checker work on the root drive? the amount of metadata should be small enough that the fs shouldn't need to be xfs, just support xattrs | 19:30 |
| zaitcev | rledisez: is there anything I can do to entice you to look at https://review.opendev.org/706653 versus https://review.opendev.org/212824 ? | 19:32 |
| patchbot | patch 706653 - swift - Let developers/operators add watchers to object au... - 4 patch sets | 19:32 |
| patchbot | patch 212824 - swift - Let developers/operators add watchers to object audit - 21 patch sets | 19:32 |
| guimaluf | timburke, hmmm no. I can't write on ext4... same error | 19:32 |
| guimaluf | same mode | 19:32 |
| zaitcev | rledisez: you made a comment "I prefer the process to crash than to not do what the operator think it is doing. At least if the process is not running, the operator has a chance to see it in his monitoring", so I thought you'd like it when I dropped the separate process, because that one can be behind a queue and silently hang. | 19:33 |
| zaitcev | timburke: I'm making another go at xattr in /tmp: https://www.spinics.net/lists/linux-mm/msg203569.html | 19:35 |
| timburke | guimaluf, (sanity checking) but it *does* work on the new server, right? | 19:35 |
| timburke | zaitcev, that seems remarkably small -- they should go merge it! ;-) | 19:36 |
| guimaluf | timburke, nice check! yes it does. its fails with: `AttributeError: 'NoneType' object has no attribute 'warning'` but the data is there with the metadata. | 19:39 |
| timburke | :-( i was hoping i wouldn't have to muck with loggers much... sorry, i was being expedient passing that None to DiskFileManager | 19:41 |
| guimaluf | timburke, nothing to sorry about! :D is just an unrelated warning.. the core of its function is working ;) | 19:43 |
| guimaluf | timburke, so yes. repro-script can write both ext4 and xfs in the new-server. but can't in both FS in the old server | 19:43 |
| timburke | makes me wonder a bit about what the warning was. i think you could just replace the None with logging.getLogger() | 19:43 |
| zaitcev | There's almost a complete fake logger in unit tests, BTW. I saw it when I expanded it a little for Dark Data. | 19:43 |
| zaitcev | Very easy to steal. FYI | 19:44 |
| guimaluf | timburke, could you point me auto the meaning of this mode=0o100000? | 19:50 |
| guimaluf | auto == out hehehe | 19:50 |
| timburke | so 100000 says it's regular file, then *all the chmod bits* are zero, so no (non-root) user -- not even the owner -- is allowed to read, write, or execute | 19:53 |
| timburke | i'm fairly certain that's leading to the EACCES error on put(), but then i still don't have an explanation for why the write() succeeds! | 19:54 |
| zaitcev | It only looks at r/w bits in the fd. Permissions are checked at open and then it just rolls using those bits, I think. | 19:54 |
| zaitcev | This can be tested by open, chmod, write, I suppose... | 19:55 |
| guimaluf | timburke, but where does this come from ? is this from the inode mode? that's what I'm confuse off | 19:56 |
| timburke | zaitcev, yeah -- i suppose we could try adding a `os.fchmod(dfw._fd, 0o600 | os.fstat(dfw._fd).st_mode)` right inside the create() context manager | 19:57 |
| timburke | guimaluf, me too -- i have all the same questions :-/ | 19:57 |
| guimaluf | timburke, nice! I'll search for it and share with you ;) | 19:58 |
| guimaluf | timburke, yes. this mode is related with the file inode stat. reading over inodes now hahaha | 20:13 |
| *** jvisser has quit IRC | 20:51 | |
| timburke | almost meeting time! | 20:54 |
| kota_ | morning | 20:57 |
| mattoliverau | morning | 21:00 |
| *** rdejoux has joined #openstack-swift | 21:20 | |
| *** jvisser has joined #openstack-swift | 21:22 | |
| *** openstackgerrit has joined #openstack-swift | 21:38 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: wip: asyc concurrent ecfragfetcher https://review.opendev.org/711342 | 21:38 |
| zaitcev | tadaa | 21:38 |
| zaitcev | asyc? | 21:39 |
| alecuyer | wow thats quite a change :) | 21:39 |
| zaitcev | 600+ in just obj controller | 21:40 |
| timburke | seems like it's mostly from cloning/tweaking ResumingGetter, yeah? | 21:41 |
| clayg | timburke: yes very much so, hopefully the line count will go down when I can get that stitched back together | 21:42 |
| timburke | oh, looks like after Victoria, we'll have Wallaby: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013006.html | 21:42 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Replace all "with Chunk*Timeout" by a watchdog https://review.opendev.org/697653 | 21:43 |
| alecuyer | oh wrt to LOSF, forgot to mention that, if the key value store lives on SSDs, it may be possible to use SMR drives, for some use cases | 21:47 |
| openstackgerrit | Merged openstack/swift stable/stein: tests: Use timedelta to adjust dates, not string manipulations https://review.opendev.org/710902 | 21:47 |
| mattoliverau | sounds very Australian, Victoraa (state in Oz) and now Wallaby :) | 21:47 |
| *** jvisser has quit IRC | 22:20 | |
| *** rcernin has joined #openstack-swift | 22:39 | |
| guimaluf | timburke, I have no f*king clue on what's happening... couldn't find anything related to o_tmpfile mode 000, etc. I give up... if that happens in production I'll keep the volume and destroy the instance, create a new one and attach the old volume to it... | 22:46 |
| timburke | seems reasonable -- sorry i couldn't be more help! | 22:47 |
| guimaluf | timburke, you were incredible helpful :) no words to thanks your attention and support | 22:52 |
| *** tkajinam has joined #openstack-swift | 22:59 | |
| seongsoocho | Hi ~ Does recon middleware count quarantine only suffix directory (not count of quarantined objects/container db/account db) ?? | 23:01 |
| seongsoocho | https://github.com/openstack/swift/blame/master/swift/common/middleware/recon.py#L279 | 23:01 |
| seongsoocho | I had confusion over the result of swift-recon. The swift-recon report to me that there are 2 quarantined objects, but actually there are more than 2 quarantined objects. | 23:01 |
| timburke | seongsoocho, are they whole suffixe dirs instead? makes me think of alecuyer's https://review.opendev.org/#/c/653748/ | 23:03 |
| patchbot | patch 653748 - swift - Fix quarantine when object path is not a directory (MERGED) - 2 patch sets | 23:03 |
| seongsoocho | timburke: I think the object quarantine was done seperately (same suffix). The whole suffix dirs are not quarantined at the same time. | 23:06 |
| seongsoocho | If the 2 objects are quarantined with same suffix, the recon middleware report there are 1 object in quarantined directory. | 23:08 |
| timburke | oh, interesting... i guess i need to play with quarantining some more.. | 23:10 |
| seongsoocho | Yes. In the code, the recon middleware seems to only counting suffix directory. And I don't know why `linkcount - 2` does it. | 23:12 |
| timburke | well, it seems to expect the layout to be like <dev>/quarantined/objects[-N]/<hash>, in which case the link count would be the number of hash dirs quarantined +2 (one as a self-ref for the directory, one for the parent) | 23:15 |
| timburke | that's why i was thinking about alecuyer's change -- we used to (mistakenly) quarantine whole suffix dirs if there was fs corruption such that a hash dir wasn't actually a dir | 23:16 |
| seongsoocho | oh.. ok.. The most of quarantined objects created in ocata version (in my cluster). So the layout is '<dev/quarantined/objects/<suffix>/<hash>' . I will test it on train version. | 23:20 |
| timburke | keep in mind that even after upgrading, any old quarantines will hang around with their old layout | 23:21 |
| timburke | one thing you might want to do is check the quarantined suffix dir and look for the should-be-a-dir-but-actually-is-a-file condition that we know would trip it | 23:22 |
| timburke | a little more info: https://bugs.launchpad.net/swift/+bug/1827412 | 23:23 |
| openstack | Launchpad bug 1827412 in OpenStack Object Storage (swift) "Swift may quarantine all objects within a suffix if a single one is invalid" [Undecided,Fix released] | 23:23 |
| seongsoocho | timburke: Thanks for the info. I'll check some more. | 23:27 |
| *** threestrands has joined #openstack-swift | 23:28 | |
| *** guimaluf has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!