Monday, 2018-01-29

*** andybotting has joined #openstack-swift00:24
*** andybotting has quit IRC00:31
*** andybotting has joined #openstack-swift00:33
*** andybotting has quit IRC00:34
*** andybotting has joined #openstack-swift00:35
*** andybotting has quit IRC00:35
*** andybotting has joined #openstack-swift00:36
andybottingHey notmyname - we've just heard back from our remote site with the IDS and they have no long term logs00:37
andybottingSo we think our options are: 1) script something to get all objects and test those or 2) just start nuking one disk at a time at that site and forcing it to resync00:39
andybottingwe're also are a bit unsure about etags on containers too - the bad etags at the problem site match the bad etags on problem objects00:40
*** sorrison_laptop has joined #openstack-swift00:42
*** jakeyip has joined #openstack-swift00:42
*** tovin07_ has joined #openstack-swift00:47
*** yashmurty has joined #openstack-swift01:07
*** tovin07 has joined #openstack-swift01:16
*** yashmurty has quit IRC01:37
kota_morning01:37
*** m_kazuhiro has joined #openstack-swift01:46
m_kazuhirogood morning01:47
kota_o/01:48
m_kazuhiroo/01:50
*** yashmurty has joined #openstack-swift01:52
*** tovin07__ has joined #openstack-swift01:55
*** tovin07_ has quit IRC01:58
mattoliveraukota_, m_kazuhiro morning o/01:59
kota_mattoliverau: o/01:59
m_kazuhiromattoliverau: morning. Do you have time to discuss task queue now? I want to discuss upgrade impact of expirer's task queue udpate. I wrote about it on etherpad page https://etherpad.openstack.org/p/swift_general_task_queue02:02
mattoliveraum_kazuhiro: I need to finish expenses and catching up on what I missed last week at work. So I little busy right now. I'll ping you once I'm done :)02:14
m_kazuhiromattoliverau: OK! :)02:17
notmynameandybotting: that's unfortunate02:46
notmynameandybotting: I was just hoping to find a way to reduce the list of objects to check to something smaller than "every object in the cluster"02:49
notmynameyou may be able to start with a container listing in the good region, especially if the objects went to the same container02:49
notmynamehmm.. or actually find all the container DBs in the bad region. then find the other replicas of those containers and compare the listings. if you find a mismatch, then also query the objects02:50
notmynamethat won't get all of them, necessarily, because a container being in a region doesn't mean that an object is in that region, and it was the objects that were corrupted02:51
andybottingnotmyname: any reason why we shouldn't just nuke the data, disk by disk at the affected site?03:00
sorrison_laptopOne of the known bad objects was listed in a container in a different region with the bad etag, the container was also in the bad region with the correct etag03:00
notmynamesorrison_laptop: of course!03:01
notmynamehow's the dispersion on the ring? do you have more than 3 regions and no overload or anything. IOW, is the stuff that's in the bad region only have one replica in the bad region?03:02
notmynameandybotting: I'm not normally a fan of "just delete it" strategies :-)03:02
sorrison_laptopwe have 5 regions03:04
andybottingyeah, it's a bit of a sledgehammer approach03:05
sorrison_laptop3 object copies, 5 for container and account03:05
notmynameoh! how about this...03:05
notmynamehack up the ZBF auditor. instead of just checking for zero-byte-files, it grabs the saved etag for every object and dumps it in a local csv file (eg name, etag). then by the power of Python, find the mismatched etags. that's then your working list for restoring the non-corrupted data03:06
notmynamedownside is you have to walk every disk03:07
notmynameoh, would have to be (name, date, etag) so you know if a mismatched etag is an overwrite or actually corrupted data03:08
*** rcernin has quit IRC03:08
andybottingyeah, that could work03:12
*** cshastri has joined #openstack-swift03:32
andybottingnotmyname: We'll have a play and get back to you. Thanks for your help03:35
*** yashmurty has quit IRC03:48
*** gkadam_ has quit IRC03:58
*** rcernin has joined #openstack-swift04:04
*** yashmurty has joined #openstack-swift04:09
*** psachin has joined #openstack-swift04:17
*** m_kazuhiro_ has joined #openstack-swift04:37
*** m_kazuhiro has quit IRC04:39
*** m_kazuhiro_ has quit IRC04:43
*** m_kazuhiro has joined #openstack-swift04:44
openstackgerritKota Tsuyuzaki proposed openstack/swift feature/s3api: Fix s3api middleware to work with functests  https://review.openstack.org/53783804:59
kota_tdasilva, timburke, (and perhaps, acoles knows?): do you know where we have object-server.conf in the swift-tox-func gate?05:00
kota_I'm still working on s3api func tests to pass the gate, which the patch works fine in my local.05:01
kota_in the way, I found s3api requires some extra allow_headers in object-server.conf and it's default off so I'm looking for the place I can inject the config.05:02
* kota_ is thinking... Or perhaps, we could make it in default at object-server???05:03
openstackgerritKota Tsuyuzaki proposed openstack/swift feature/s3api: Change object-server default allowed_headers  https://review.openstack.org/53875005:10
mattoliveraukota_: look in test/funtional/__init__.py. It uses _in_process_find_conf_file which by default will look in the <swift source>/etc dir and will also look for -sample files. So by default <swift-source>/etc/object-server.conf-sample.05:40
mattoliveraukota_: more detailed. .zuul.yaml a job uses a toc_envlist. So you could add one for s3api05:41
mattoliveraukota_: then the tox.ini env would point to a SWIFT_TEST_IN_PROCESS_CONF_LOADER=s3api or something05:42
kota_mattoliverau: thx the info.05:43
mattoliveraukota_: then add an 's3api' to the conf_loaders in test/functional/__init__.py. Which I think points to a function that will tweek the settings.05:43
kota_mattoliverau: I was thinking so "editing etc/object-sever.conf-sample affects the result" but, i could not get expected result.05:44
kota_let me check the conf loader process.05:44
kota_it looks like the entry point is around _in_process_find_conf_file ?05:46
*** gkadam has joined #openstack-swift05:47
mattoliverauyeah, when it looks for a conf file to use.05:48
*** SkyRocknRoll has quit IRC05:48
mattoliverauconf_loaders on line (377) and in_process_setup (383) (on master) is where the conf_loaders happen, and around line 434 is where the modification to the configs happen. So where an s3api funtion could insert itself into the pipeline.05:50
mattoliverauoh that's the proxy and swift.confs not object-server.05:51
kota_yeah, IIRC, current all in-process functests requires to modify only proxy conf or swift.conf.05:56
mattoliveraukota_: it looks like the object config used is built up from the config dictionany definted at the root level of the file. then global config is used. So you might be able to just add extra config to that in your s3api function you use to insert the s3api middelware to the proxy.. if that's what your doing05:56
mattoliverau*config dictionary defined05:57
mattoliverau(sorry failing at typing)05:57
kota_where file the config dict is in?05:57
kota_functional/__init__.py?05:58
mattoliverauyeah line 91 (on master)05:58
mattoliverauthen global on 44705:59
mattoliverauobject servers are created with it on 55906:00
kota_oic, and it's in "def in_process_setup(the_object_server=object_server)"06:00
mattoliverauyeah, that's what it looks like.06:00
kota_I may hack the code, let me take a while to try it.06:00
*** SkyRocknRoll has joined #openstack-swift06:01
mattoliverau:)06:01
kota_mattoliverau: much appreciated!06:01
mattoliveraunps, I hope it helps06:01
*** stewie925 has quit IRC06:28
* kota_ is playing with 'SWIFT_TEST_IN_PROCESS=1'06:54
*** sorrison_laptop has quit IRC06:57
*** kazsh has joined #openstack-swift06:58
*** geaaru_ has quit IRC07:06
openstackgerritKota Tsuyuzaki proposed openstack/swift feature/s3api: Fix s3api middleware to work with functests  https://review.openstack.org/53783807:10
*** rcernin has quit IRC07:11
kota_^ worked well in my local with `tox -e func-s3api` and s3api is only in the env. i.e. I'm expecting that will pass the gate check.07:11
* kota_ is praying07:12
*** cbartz has joined #openstack-swift07:16
*** armaan has joined #openstack-swift07:26
*** hoonetorg has quit IRC07:35
*** m_kazuhiro_ has joined #openstack-swift07:38
*** hseipp has joined #openstack-swift07:41
*** m_kazuhiro has quit IRC07:41
*** m_kazuhiro_ has quit IRC07:42
*** m_kazuhiro has joined #openstack-swift07:42
openstackgerritKota Tsuyuzaki proposed openstack/swift master: WIP: Move swift-drive-audit code to swift/cli/drive_audit  https://review.openstack.org/51120507:43
*** Supun has joined #openstack-swift07:44
*** pcaruana has joined #openstack-swift07:44
*** neonpastor has quit IRC07:45
*** neonpastor has joined #openstack-swift07:49
*** hoonetorg has joined #openstack-swift07:52
*** links has joined #openstack-swift08:09
*** stewie925 has joined #openstack-swift08:15
*** Supun has quit IRC08:22
*** d0ugal has quit IRC08:22
*** tesseract has joined #openstack-swift08:26
*** d0ugal has joined #openstack-swift08:28
*** geaaru has joined #openstack-swift08:42
*** yashmurty has quit IRC08:43
*** kei_yama has quit IRC08:45
*** cbartz has quit IRC08:49
*** d0ugal has quit IRC09:00
acolesgood morning09:13
acoleskota_: did you get it figured out? iirc tox -e func runs on devstack but the the other tox func-* jobs use in process testing09:18
kota_acoles: good morning. And mattoliverau helps me to get it.09:18
kota_acoles: it looks like the new zuul job with in-process is running well (looking at http://zuul.openstack.org/)09:19
acoleskota_: great. BTW here's an example of adding a custom in process test conf loader https://review.openstack.org/#/c/503262/09:19
patchbotpatch 503262 - swift - Functional tests for staticweb middleware09:19
kota_except, it got timeout :/09:20
*** d0ugal has joined #openstack-swift09:20
kota_but looking at the logs (http://logs.openstack.org/38/537838/5/check/swift-tox-func-s3api/40a6d7d/job-output.txt.gz) it looks like that had been going well....09:20
kota_I'm not sure if it's zuul issue or not, yet.09:21
kota_acoles: yeah, i just did the customization (https://review.openstack.org/#/c/537838/5/test/functional/__init__.py) ;-)09:22
patchbotpatch 537838 - swift (feature/s3api) - Fix s3api middleware to work with functests09:22
acolesOIC, nice! just catching up in my scrollback09:22
acoleswe need to thank jrichli_ for pioneering the in process custom loader pattern09:23
kota_acoles: absolutely09:23
kota_but one thing I didn't catch up what happens in the swift-tox-func gate.09:24
*** cbartz has joined #openstack-swift09:24
kota_it looks like tox -e func is not "in process" that means it will test against to the existing cluster09:25
acoleskota_: yes I think that job uses devstack09:25
kota_acoles: oh, it's in dsvm?09:25
acolesI was just looking for a recent log to confirm but I'm getting 'File not found' on all the job results :/09:25
kota_too bad :/09:26
kota_recently, zuul is unstable :(09:26
acolesmaybe something to do with the zuul issues at end of last week, perhaps we lost a load of logs09:26
kota_kk. I'll check the stats tomorrow morning. maybe we will get NOTICE or something here (or @openstackinfra at twitter).09:29
kota_thanks acoles, mattoliverau, and jrichli_!09:30
*** Supun has joined #openstack-swift09:42
*** Supun has quit IRC09:47
*** m_kazuhiro has quit IRC09:55
*** tovin07__ has quit IRC10:04
*** threestrands_ has joined #openstack-swift10:33
*** Supun has joined #openstack-swift10:35
*** threestrands has quit IRC10:36
*** jappleii__ has joined #openstack-swift10:37
*** jappleii__ has quit IRC10:38
*** jappleii__ has joined #openstack-swift10:39
*** threestrands_ has quit IRC10:41
-openstackstatus- NOTICE: Zuul is currently under heavy load. Do not *recheck* or *approve* any changes.11:04
*** shohra has joined #openstack-swift11:29
shohraWe are facing openstack swift problem where some segments are missing every time we upload some file11:30
shohraThe log is as following ClientException at /api/swift/containers/cassandra_jill_data_segments/object/zeele/cassandra-ac-cfm:18.01.0000_jill_20180125.tar.gz/1516878130.261379/6062520072/500000000/00000000 Object GET failed: http://wharf-openstack.internal.worksap.com:8088/v1/AUTH_49d9a14594674e3991be8c9dbe6f0567/cassandra_jill_data_segments/zeele/cassandra-ac-cfm%3A18.01.0000_jill_20180125.tar.gz/1516878130.261379/6062520072/11:30
shohraException Value: Object GET failed: http://wharf-openstack.internal.worksap.com:8088/v1/AUTH_49d9a14594674e3991be8c9dbe6f0567/cassandra_jill_data_segments/zeele/cassandra-ac-cfm%3A18.01.0000_jill_20180125.tar.gz/1516878130.261379/6062520072/500000000/00000000 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.< Exception Location:/usr/lib/python2.7/dist-packages/swiftclie11:31
*** mvk has quit IRC12:02
*** cshastri has quit IRC12:43
*** armaan_ has joined #openstack-swift12:49
*** armaan has quit IRC12:52
*** mvk has joined #openstack-swift13:17
openstackgerritAlistair Coles proposed openstack/swift feature/deep: Fix misplaced_objects method and align with cleave method  https://review.openstack.org/53832713:27
-openstackstatus- NOTICE: Zuul is currently under heavy load. Do not *recheck* or *approve* any changes until we give the go ahead.13:34
*** ChanServ changes topic to "Zuul is currently under heavy load. Do not *recheck* or *approve* any changes until we give the go ahead."13:34
*** d0ugal has quit IRC13:38
*** d0ugal has joined #openstack-swift13:38
*** bkopilov has quit IRC13:40
*** SkyRocknRoll has quit IRC13:49
*** cbartz has quit IRC13:53
*** cbartz has joined #openstack-swift14:01
*** SkyRocknRoll has joined #openstack-swift14:05
*** Sourav has joined #openstack-swift14:06
SouravHello guys, I am facing some problem with Swift. I am using 5 data nodes. 4 of them are full and last one has 1.3TB left. But swift can't store files.14:09
SouravWhile uploading it shows that segments are uploaded but with `stat` command it can't find that segments.14:09
SouravError message just showing that `rsync` error and write error. No disk space left.14:10
Sourav!!!! Please help !!!!14:10
openstackSourav: Error: "!!!" is not a valid command.14:10
Sourav@#openstack-swift14:10
*** psachin has quit IRC14:15
-openstackstatus- NOTICE: we've been able to restart zuul, and re-enqueue changes for gate. Please hold off on recheck or approves, we are still recovering. More info shortly.14:30
*** gkadam has quit IRC14:33
*** SkyRocknRoll has quit IRC14:42
*** SkyRocknRoll has joined #openstack-swift14:56
*** Supun has quit IRC15:22
*** Supun has joined #openstack-swift15:25
*** cbartz has quit IRC15:48
*** tesseract has quit IRC16:03
*** jappleii__ has quit IRC16:05
*** ukaynar has joined #openstack-swift16:09
*** SkyRocknRoll has quit IRC16:11
*** links has quit IRC16:20
*** ChanServ changes topic to "https://bugs.not.mn/project/Swift | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Review Dashboard: http://not.mn/reviews.html"16:22
-openstackstatus- NOTICE: zuul.o.o is back online, feel free to recheck / approve patches.16:22
*** pcaruana has quit IRC16:25
*** SkyRocknRoll has joined #openstack-swift16:29
*** armaan has joined #openstack-swift16:36
*** brimestone has joined #openstack-swift16:36
notmynamegood morning16:39
*** armaan_ has quit IRC16:39
*** tesseract has joined #openstack-swift16:47
openstackgerritOpenStack Release Bot proposed openstack/python-swiftclient master: Update reno for stable/queens  https://review.openstack.org/53897216:54
*** Supun has quit IRC17:20
*** Supun has joined #openstack-swift17:21
openstackgerritTim Burke proposed openstack/python-swiftclient master: Update reno for stable/queens  https://review.openstack.org/53897217:24
*** hseipp has quit IRC17:26
*** gyee has joined #openstack-swift17:29
*** armaan_ has joined #openstack-swift17:29
*** armaan_ has quit IRC17:31
*** armaan_ has joined #openstack-swift17:32
*** armaan has quit IRC17:32
*** ukaynar has quit IRC17:33
timburkenotmyname, andybotting, sorrison: we keep finding reasons that it'd be nice to have https://review.openstack.org/#/c/212824/ ...17:34
patchbotpatch 212824 - swift - Let developers/operators add watchers to object audit17:34
*** Supun has quit IRC17:41
*** Supun has joined #openstack-swift17:42
*** ukaynar has joined #openstack-swift17:44
*** mvk has quit IRC17:56
*** ukaynar has quit IRC18:16
*** ukaynar has joined #openstack-swift18:16
*** geaaru has quit IRC18:17
*** ukaynar has quit IRC18:20
*** ukaynar has joined #openstack-swift18:24
*** mvk has joined #openstack-swift18:35
*** brimestone has quit IRC19:09
*** hseipp has joined #openstack-swift19:23
*** hseipp has quit IRC19:25
*** tesseract has quit IRC19:31
openstackgerritSamuel Merritt proposed openstack/swift master: Cleanup for iterators in SegmentedIterable  https://review.openstack.org/53836619:59
*** ukaynar has quit IRC20:09
*** ukaynar has joined #openstack-swift20:10
*** ukaynar has quit IRC20:14
*** ukaynar has joined #openstack-swift20:20
*** awkwardpenguin has joined #openstack-swift21:16
*** Supun has quit IRC21:18
openstackgerritTim Burke proposed openstack/swift master: tempurl: Make the digest algorithm configurable  https://review.openstack.org/52577021:25
*** ukaynar has quit IRC21:36
*** ukaynar has joined #openstack-swift21:47
*** awkwardpenguin has quit IRC21:53
*** awkwardpenguin has joined #openstack-swift22:03
openstackgerritTim Burke proposed openstack/swift master: Fix up some Content-Type handling in account/container listings  https://review.openstack.org/53302722:04
mattoliveraumorning22:05
notmynamemattoliverau: hello22:05
*** threestrands has joined #openstack-swift22:05
*** threestrands has quit IRC22:05
*** threestrands has joined #openstack-swift22:05
notmynamemattoliverau: so walking down the sidewalk at lunch in SF, and I am still trying to go to the left of oncoming people. my brain hasn't switched back yet :-)22:05
mattoliveraulol, the first road I cross whenever I arrive in the US is always the most  dangerous ;P22:07
notmynamejust look everywhere, all the time22:07
*** threestrands_ has joined #openstack-swift22:07
*** awkwardpenguin has quit IRC22:08
*** threestrands has quit IRC22:08
*** threestrands_ has quit IRC22:08
*** threestrands has joined #openstack-swift22:08
*** vint_bra has joined #openstack-swift22:12
*** awkwardpenguin has joined #openstack-swift22:14
*** awkwardpenguin has quit IRC22:19
kota_yey, https://review.openstack.org/#/c/537838/ works at the gate!22:19
patchbotpatch 537838 - swift (feature/s3api) - Fix s3api middleware to work with functests22:19
mattoliveraukota_: nice!22:20
tdasilvakota_: great!22:20
kota_mattoliverau: thx! you helped me a lot!22:20
*** threestrands_ has joined #openstack-swift22:21
kota_tdasilva: o/22:21
*** vint_bra has quit IRC22:22
*** threestrands has quit IRC22:23
*** awkwardpenguin has joined #openstack-swift22:24
*** geaaru has joined #openstack-swift22:25
*** awkwardpenguin has quit IRC22:29
tdasilvakota_: does that mean I can abandon p 53254522:30
patchbothttps://review.openstack.org/#/c/532545/ - swift (feature/s3api) - WIP: merge func tests22:30
*** awkwardpenguin has joined #openstack-swift22:34
*** jappleii__ has joined #openstack-swift22:35
*** vint_bra has joined #openstack-swift22:35
*** jappleii__ has quit IRC22:36
*** jappleii__ has joined #openstack-swift22:37
*** threestrands_ has quit IRC22:37
*** awkwardpenguin has quit IRC22:39
*** awkwardpenguin has joined #openstack-swift22:45
*** rcernin has joined #openstack-swift22:46
*** awkwardpenguin has quit IRC22:50
openstackgerritTim Burke proposed openstack/swift master: Improve content negotiation  https://review.openstack.org/20727622:52
*** vint_bra has quit IRC22:53
*** awkwardpenguin has joined #openstack-swift22:55
*** awkwardpenguin has quit IRC23:00
openstackgerritTim Burke proposed openstack/swift master: Disallow fractional replicas in EC policies  https://review.openstack.org/50345423:01
*** ukaynar has quit IRC23:02
*** awkwardpenguin has joined #openstack-swift23:05
*** awkwardpenguin has quit IRC23:10
kota_tdasilva: no it depends on your patch23:15
*** awkwardpenguin has joined #openstack-swift23:16
kota_so i wonder if you could check mine and squash it to yours23:16
*** vint_bra has joined #openstack-swift23:18
*** awkwardpenguin has quit IRC23:21
*** awkwardpenguin has joined #openstack-swift23:26
*** awkwardpenguin has quit IRC23:31
*** awkwardpenguin has joined #openstack-swift23:36
tdasilvakota_: ah ok, I will do that...23:41
*** awkwardpenguin has quit IRC23:41
*** briancline has quit IRC23:45
*** StevenK has quit IRC23:45
*** blair has quit IRC23:45
*** jlvillal has quit IRC23:45
*** jistr has quit IRC23:45
*** awkwardpenguin has joined #openstack-swift23:47
*** briancline has joined #openstack-swift23:51
*** StevenK has joined #openstack-swift23:51
*** blair has joined #openstack-swift23:51
*** jlvillal has joined #openstack-swift23:51
*** jistr has joined #openstack-swift23:51
*** awkwardpenguin has quit IRC23:51
*** awkwardpenguin has joined #openstack-swift23:57
*** awkwardpenguin has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!