Tuesday, 2020-08-04

« Monday, 2020-08-03 Index Wednesday, 2020-08-05 »
*** k_mouza has joined #openstack-glance00:01
*** k_mouza has quit IRC00:06
*** k_mouza has joined #openstack-glance00:16
*** k_mouza has quit IRC00:21
*** rcernin has quit IRC02:27
*** gyee has quit IRC02:29
*** rcernin has joined #openstack-glance02:30
*** keper7 has quit IRC03:00
*** k_mouza has joined #openstack-glance04:17
*** ratailor has joined #openstack-glance04:21
*** k_mouza has quit IRC04:21
*** ratailor has quit IRC04:26
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-glance04:33
*** m75abrams has joined #openstack-glance04:46
*** nikparasyr has joined #openstack-glance05:00
*** udesale has joined #openstack-glance05:36
openstackgerritMerged openstack/glance master: [goal] Migrate glance jobs to focal  https://review.opendev.org/74309205:55
*** belmoreira has joined #openstack-glance06:15
openstackgerritAbhishek Kekane proposed openstack/glance-specs master: Make cinder driver compatible with multiple stores  https://review.opendev.org/69515207:00
*** rcernin has quit IRC07:22
*** rcernin has joined #openstack-glance07:48
*** ralonsoh has joined #openstack-glance07:50
*** ralonsoh has quit IRC07:50
*** ralonsoh has joined #openstack-glance07:52
*** rcernin has quit IRC07:53
*** rcernin has joined #openstack-glance08:26
*** rcernin has quit IRC08:31
*** priteau has joined #openstack-glance08:43
*** k_mouza has joined #openstack-glance08:53
*** rcernin has joined #openstack-glance08:59
*** lpetrut has joined #openstack-glance09:12
*** rcernin has quit IRC09:16
*** ralonsoh has quit IRC09:40
*** ralonsoh has joined #openstack-glance09:41
* abhishekk will be back at 1400 hrs09:44
*** k_mouza has quit IRC09:47
*** k_mouza has joined #openstack-glance09:59
*** udesale_ has joined #openstack-glance10:12
*** tkajinam has quit IRC10:12
*** udesale has quit IRC10:14
*** belmoreira has quit IRC10:17
*** k_mouza has quit IRC10:21
*** k_mouza has joined #openstack-glance10:27
openstackgerritMerged openstack/glance stable/ussuri: Releasenotes for Ussuri point release  https://review.opendev.org/74409310:37
*** rcernin has joined #openstack-glance10:55
*** jv_ has joined #openstack-glance11:15
*** Luzi has joined #openstack-glance11:20
*** jv__ has joined #openstack-glance11:26
*** udesale has joined #openstack-glance11:27
*** jv_ has quit IRC11:27
*** udesale_ has quit IRC11:27
*** udesale_ has joined #openstack-glance11:31
*** udesale has quit IRC11:32
*** udesale_ has quit IRC11:38
*** udesale has joined #openstack-glance11:41
*** k_mouza has quit IRC11:47
*** belmoreira has joined #openstack-glance11:51
*** rcernin has quit IRC11:53
*** takamatsu has joined #openstack-glance12:04
*** k_mouza has joined #openstack-glance12:12
*** belmoreira has quit IRC12:23
*** ralonsoh_ has joined #openstack-glance12:26
*** ralonsoh__ has joined #openstack-glance12:28
*** ralonsoh has quit IRC12:30
*** ralonsoh_ has quit IRC12:31
*** njohnston has joined #openstack-glance12:33
*** ralonsoh__ is now known as ralonsoh12:44
abhishekkdansmith, around?13:13
dansmithabhishekk: just got back13:21
dansmithmnaser: devstack or osc? the former maybe but not the latter that I know of13:22
abhishekkdansmith, ack, welcome to the core group13:22
dansmithheh, thanks :)13:22
abhishekkcould you please confirm you have +2 ability?13:22
dansmithyup, looks like I do13:23
abhishekkcool13:23
dansmiththanks for that :)13:33
dansmithI've got some big cleanups to the format inspector almost done which make it a lot easier to follow,13:34
dansmithand will get some tests in place today13:34
abhishekkdansmith, ack13:35
abhishekkso now we could have virtual size set for those images whoever uses upload or import api13:36
*** hoonetorg has quit IRC13:36
abhishekkOnly problem is with nova snapshot with copy-on-write as those will use locations API13:36
dansmithmeaning when nova does a snapshot direct in rbd and doesn't upload?13:39
*** hoonetorg has joined #openstack-glance13:39
abhishekkyes13:39
dansmithI thought you said something about nova setting the virtual_size in that case?13:40
abhishekkdansmith, no13:41
dansmithnova *should* only be using raw on rbd, so the virtual_size should be the same as the raw size anyway right?13:43
abhishekkyes13:43
mnaserdansmith: it ended up being that when osc fails to upload, it tries to import, so that was really a distraction13:46
mnaseri fixed my failure but the osc behaviour of import-if-upload-fails was reverted cause it was very unpredictable13:46
mnaseresp if its against a cloud that is running under uwsgi which there are many13:47
mnaserdansmith: https://review.opendev.org/#/c/744254/13:47
dansmithmnaser: ack, yeah.. that might've been monty trying to future-proof it for a pending removal of upload, but agree that's confusing behavior for most13:47
dansmithgmann: do you know, if I need to use qemu-img from unit tests (or code for that matter), do I need to put qemu-something into bindep.txt? I think nova probably gets it from the libvirt things it has there13:51
abhishekkrosmaita, fixed your comments on the cinder multi-store spec13:55
rosmaitaabhishekk: thanks, will take a look a bit later today13:56
rosmaitahope they were reasonable?13:56
abhishekkrosmaita, no problem, just wanted to let you know13:56
abhishekkrosmaita, yes,13:56
* abhishekk in meeting14:01
*** Luzi has quit IRC14:27
*** udesale_ has joined #openstack-glance14:35
*** udesale has quit IRC14:38
*** donnyd has joined #openstack-glance14:40
*** nikparasyr has left #openstack-glance14:47
*** jv__ has quit IRC14:58
*** jv__ has joined #openstack-glance15:10
openstackgerritAbhishek Kekane proposed openstack/glance-specs master: Calculate virtual size of image  https://review.opendev.org/74112115:11
* abhishekk dinner break, back in 45 mins15:12
gmanndansmith: you can add explicitly also like ironic does - https://opendev.org/openstack/ironic/src/branch/master/bindep.txt15:14
dansmithgmann: ah perfect, thanks15:15
*** lpetrut has quit IRC15:15
jokkedansmith: welcome welcome and thanks for helping us out!15:16
dansmith\o/15:17
*** openstackgerrit has quit IRC15:20
*** jv__ has quit IRC15:22
*** k_mouza has quit IRC15:40
*** lpetrut has joined #openstack-glance15:43
*** k_mouza has joined #openstack-glance15:54
* abhishekk back from break15:57
*** m75abrams has quit IRC16:04
*** lpetrut has quit IRC16:21
*** openstackgerrit has joined #openstack-glance16:21
openstackgerritDan Smith proposed openstack/glance master: Stream-friendly disk format inspection module  https://review.opendev.org/74420516:21
dansmithabhishekk: ^16:21
abhishekkdansmith, ack16:22
abhishekkdansmith, lee yarwood from nova has reported one bug against nova-ceph job16:25
abhishekkI guess it is related to race condition, do you get any chance to have a look at it?16:25
abhishekk#link https://bugs.launchpad.net/bugs/189019316:26
openstackLaunchpad bug 1890193 in Glance "nova-ceph-multistore fails with rbd.OSError: [errno 116] error removing image" [Undecided,New]16:26
dansmithabhishekk: I saw and asked you a question on there.. I haven't dug into the logs yet, but I figured you must have because you said that16:26
abhishekkohh, I didn't got notification about it16:26
dansmithdo you know that glance can fail to delete an rbd if something is still writing or?16:26
abhishekknot sure16:28
abhishekkbut from log it looks like that it can16:28
dansmithokay did you see a second glance task not yet finished before the delete happens?16:31
abhishekkI see failure message which states that no file present in staging area16:32
abhishekkNow the file in staging area can be deleted only by DeleteFromFS task or revert operation or image is deleted while operation is in progress16:33
dansmithokay and that's your indication that the staging file was deleted by the first thread when it completed but that there's likely another thread still going?16:36
abhishekki think so16:39
dansmithack16:40
*** gregwork has joined #openstack-glance16:40
dansmithI have been pretty much striking out trying to figure out a good way to test for this, more than just mocked unit tests,16:40
dansmithso maybe if we're hitting it reasonably reliably in the gate, we should just merge that stack as soon as possible so we have time to watch it in the gate well in advance of the release in case we need to fix or revert16:41
dansmithare there comments I need to address on it, or are we just waiting for wider review?16:41
abhishekkdansmith, one of the suggestion is busted states16:42
abhishekkif task state some how remains in pending state then it is not possible to unlock the image without admin intervention16:43
abhishekksame thing can be said about reverting/reverted state16:43
abhishekkreverted will not come into picture as once task is successfully reverted that means we have released the lock as well16:44
dansmithabhishekk: in my current thing there is no pending state and the lock can be busted by the API as soon as it becomes old enough16:44
abhishekkanother one is should we increase the logging of progress by some more time?16:44
dansmithso I'm confused I guess16:44
abhishekklet me have a look again16:44
dansmithmeaning less than once per minute?16:45
dansmithis this feedback collected somewhere? if there are comments on the patches I think I'm missing them16:46
abhishekknope I was going to add them16:46
dansmithah okay16:46
abhishekklast friday I linked you flow diagram of task flow, remember?16:47
dansmithyeah16:47
dansmithI was scrambling to get ready for my trip, so I don't remember if something actionable came out of that...16:47
*** gyee has joined #openstack-glance16:49
*** priteau has quit IRC16:51
abhishekkdansmith, ack, I was also on PTO yesterday so totally forget about that16:51
*** priteau has joined #openstack-glance16:52
*** ralonsoh has quit IRC17:00
*** ralonsoh has joined #openstack-glance17:01
*** ralonsoh has quit IRC17:06
*** ralonsoh has joined #openstack-glance17:07
*** udesale_ has quit IRC17:16
openstackgerritDan Smith proposed openstack/glance master: Stream-friendly disk format inspection module  https://review.opendev.org/74420517:30
openstackgerritDan Smith proposed openstack/glance master: Inspect upload stream and set virtual_size on upload  https://review.opendev.org/74423417:30
jokkethat was the other thing I was suppose to get done. review those patches17:31
*** priteau has quit IRC17:33
dansmithabhishekk: ah *that* pending, I see.. will reply :)17:35
abhishekkack17:36
*** k_mouza has quit IRC17:36
*** lpetrut has joined #openstack-glance17:49
*** lpetrut has quit IRC17:50
*** jv__ has joined #openstack-glance17:51
* abhishekk leaving for the day18:02
dansmithrosmaita: so I know you're already in glance review debt, but your feedback on this, at least at a high level might be useful: https://review.opendev.org/#/c/744205/618:30
dansmithrosmaita: note the small patch after that to integrate it into the upload data pipeline18:30
dansmithmight also be useful for cinder?18:30
rosmaitadansmith: i'll take a look18:31
dansmithsomething could use this to examine a disk that isn't stored locally by just streaming enough of it until the module finds what it needs and then stops18:33
dansmithso, like read chunks from ceph until that happens, without having to download the whole thing, in the case where we just need to inspect something stored in ceph already18:33
rosmaitayes, that could be very useful18:46
*** k_mouza has joined #openstack-glance18:48
*** k_mouza has quit IRC19:07
jokkedansmith: rosmaita: I mostly like that. Only problem I'm seeing is those exceptions are not caught anywhere. As we have never done actual verification of the image data (and lots of people are just using bunch of random bytes as part of their canary tests) IIUC this will start throwing 500s out of the API like no tomorrow?19:15
dansmithjokke: nope, it swallows any/all failures to detect or process the image19:15
dansmithjokke: right here: https://review.opendev.org/#/c/744205/6/glance/common/format_inspector.py@51719:16
dansmithtested in both cases here: https://review.opendev.org/#/c/744205/6/glance/tests/unit/common/test_format_inspector.py@23519:16
rosmaitatests!!!19:16
dansmithrosmaita: ikr?19:16
rosmaitadansmith: it is a pleasure to have you working on glance19:16
rosmaita:)19:16
dansmith:P19:17
jokkeaha, yet another wrapper ... missed that. Thanks for taking it in cnsideration19:17
dansmiththose tests also feed each format thingy random data19:17
jokkecool ... yeah, maybe debug rather than error logs there but indeed that looks nifty19:19
dansmithI still need to remove a bunch of debug from those format handlers btw19:19
dansmiththat's too much debug for actual use I think19:19
dansmithmaybe I can have a separate knob to turn those on if we need to debug and repro a problem or something19:20
jokkeit's quite a lot indeed19:20
dansmithwell, I blame MSFT's overly complicated VHDX format for making me add all that in the first place :P19:20
jokkeLog level debugdebug ;)19:20
dansmithheh19:21
jokkedansmith: nono, I totally understand why you would want to have it in there while actually writing the code, in gate and production it indeed might be bit much19:21
dansmithdefinitely.. I also hesitate to throw it out because in a year, it'll be easier to turn it on to debug things than re-read the spec docs19:22
dansmithI'll add a separate knob and you can see if what I have is reasonable19:22
dansmiths/see/judge/19:22
jokkeHonestly it would be great if we had level in between debug and info we could use in the gate19:22
dansmithwell, we used to have logging.TRACE for uuberdebug like this19:23
dansmithbut it was non-standard and thus shunned19:23
jokkeand if there is something which needs it, one could just flip that19:23
jokkedansmith: I know ... missing those times19:23
dansmithyeah19:23
dansmithI think the misunderstanding in that effort was "no, people run with debug=True all the time, so we *do* need another"19:23
jokkecause tbh, gate in normal day doesn't need to debug on that level, but like you said, in certain times it would be beneficial19:23
jokkewell it was the times when OpenStack logging was all over the place19:24
jokkeIt was actually one of the early things I took on when I joined the Glance team. I wen't through every single logging line, made sure that the level and message made at least somewhat sense19:25
jokkeThat' why I also pay a lot of attention on how people log in their changes :P19:25
jokkeit's an exercise I don't want to do again19:25
dansmithaye19:27
jokkeI learned hell of a lot of the code and drove rosmaita and the rest of the cores totally mental trying to review those :P19:29
jokkedansmith: rosmaita: smcginnis: any chance to get those bugfixes https://review.opendev.org/744024 https://review.opendev.org/744030 https://review.opendev.org/744011 https://review.opendev.org/744010 looked over. Would be great to get them backported before we pull the trigger of releasing ussuri and train19:35
jokkeif you're too busy for it, just say. Version numbers are cheap19:37
dansmithI will in a sec19:37
openstackgerritDan Smith proposed openstack/glance master: Stream-friendly disk format inspection module  https://review.opendev.org/74420519:38
openstackgerritDan Smith proposed openstack/glance master: Inspect upload stream and set virtual_size on upload  https://review.opendev.org/74423419:38
dansmith^with tracing knob19:38
jokkelets hpe grenade likes them this time. Tried to look, but have no idea why it failed19:41
jokkehope19:41
dansmithjokke: grenade is failing for swift reasons, AFAICT19:47
dansmithjokke: if there's a rush on these, let me know and I'll write these tests for you19:47
*** ralonsoh has quit IRC19:47
dansmithfor a bug, surely seems like it's reasonable to require tests... :)19:47
jokkeI'll have a look. one of them misses the tests all together ... it's not exactly trivial to get there. Abhishek found the behaviour by pausing the execution with debugger and whacking the data :P19:50
jokkeso kind of those that one should not face in real life19:51
dansmithpretty sure each of these things are at least trivially testable just to make sure the exception handler runs19:51
dansmithbut let me know and I'll take a go at it :)19:51
dansmithapproved that client one as my first glance +W!19:52
jokkethnx19:54
jokkehttps://review.opendev.org/#/c/744011/ this is the one we really struggled to have any meaningful tests for19:55
jokkeAnd Abhishek is in general pretty good mocking enough things to get some kind of starting point :P19:55
dansmithwhy, isn't that trivial to run such that delete() will explode and we run the log and assert we didn't re-raise?19:57
dansmithlike, if someone changed the log content and didn't add a new parameter to the arg dict, we wouldn't notice until it exploded in real life19:57
dansmiththat's not meaningful in the larger integration sense, but it is meaningful in making sure that we catch such a trivial error way before it generates a stack trace in someone's log instead of a useful error message :)19:58
jokkegraah indeed .... you might not know yet ... I'm really bad doing unit tests and I hate them in general.19:58
dansmithheh, well, I might've guessed, but.. still :)19:59
dansmithbut again, if you're EOD or out of time, I'll happily write you something19:59
jokkeit's 9pm for me ... but tbh I don't think there's such panic if grenade is not working anyways. Meaning I think it's better to get the releases done this week than keep them waiting for these.20:01
jokkelike siad ersion numbers are cheap20:01
dansmithwell, up to you, it won't take me long and it's notably *not* 9pm here :)20:02
jokkeif you're up for it, sure, I'll take it. Otherwise I jump into the wonderous world of black magic called mocking once I've had my dinner :P20:03
jokkeIt's funny. Still after 5 years mocking stuff for testing if true is true makes absolutely no sense to me :P20:04
dansmithjokke: what's your answer about the task naming?20:04
jokkedamn it's actually way over that20:04
jokkedansmith: I really don't care what it's called apart from SaveImage in it's current form was very much off to warrant changing it20:05
dansmithack20:05
jokkeVerifyState, VerifyImageState, ConfirmActive ... any would work for me20:06
smcginnisJokkesTask20:06
jokkePreferably nothing crazy long as that name gets stuff added areound it when it's logged20:06
jokkenot fun having 200 character strings in your logs20:07
*** jv__ has quit IRC20:29
openstackgerritDan Smith proposed openstack/glance master: Fix active image when all uploads fail  https://review.opendev.org/74401020:38
openstackgerritDan Smith proposed openstack/glance master: Fix active image without data  https://review.opendev.org/74401120:38
dansmithjokke: unless I'm missing something, your log message would always explode in the second patch, so I fixed it in the above ^ ... let me know if I'm missing why that's not right.20:38
dansmithsmcginnis: since I have my fingerprints on those I'm hesitant to ninja approve, but they had a +2 from abhishekk before I came along, in case you have change to skim and approve :)21:26
*** freerunner has quit IRC22:55
*** sorrison has quit IRC22:55
*** dansmith has quit IRC22:55
*** frickler has quit IRC22:55
*** tkajinam has joined #openstack-glance22:55
*** freerunner has joined #openstack-glance22:57
*** sorrison has joined #openstack-glance22:57
*** dansmith has joined #openstack-glance22:57
*** frickler has joined #openstack-glance22:57
« Monday, 2020-08-03 Index Wednesday, 2020-08-05 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!