Wednesday, 2021-02-24

*** tosky has quit IRC00:10
jokkedansmith: dropped to -1 -(W) hopefully that works, if not we just need to throw the empty migration as follow up00:58
jokkeworld will not end to that missing ;)00:59
jokkeI think we miss the migration script about every cycle as they are under data_migrations vs. versions01:01
dansmithanything that does not add up to "should merge" will make it not merge, so missing a +W is all that is needed01:07
jokkekk01:08
jokkenow I don't need to remember un-2 it tomorrow :)01:08
jokkeI have a feeling that I've done this exactly sam thing before for same reason :D01:11
*** zzzeek has quit IRC01:15
*** zzzeek has joined #openstack-glance01:17
*** rcernin has quit IRC02:37
openstackgerritDan Smith proposed openstack/glance master: Add housekeeping module and staging cleaner  https://review.opendev.org/c/openstack/glance/+/77701203:06
openstackgerritDan Smith proposed openstack/glance master: Test restarting glance for staging cleanup  https://review.opendev.org/c/openstack/glance/+/77701403:06
openstackgerritDan Smith proposed openstack/glance master: Make functional tests set node_staging_uri  https://review.opendev.org/c/openstack/glance/+/77727703:06
*** rcernin has joined #openstack-glance04:44
*** rcernin has quit IRC04:54
*** rcernin has joined #openstack-glance04:55
*** ratailor has joined #openstack-glance05:01
*** rcernin has quit IRC05:14
*** udesale has joined #openstack-glance05:21
*** rcernin has joined #openstack-glance05:21
openstackgerritLance Bragstad proposed openstack/glance master: Implement project personas for image actions  https://review.opendev.org/c/openstack/glance/+/76475405:29
openstackgerritLance Bragstad proposed openstack/glance-tempest-plugin master: Implement API protection testing for images  https://review.opendev.org/c/openstack/glance-tempest-plugin/+/77356805:30
lbragstadabhishekk ok - that *should* be in good shape, i reworked everything to only include the project-member and project-reader personas05:31
abhishekklbragstad, ack05:31
abhishekklbragstad, so we need same for tasks and metadefs right?05:32
lbragstadyeah05:32
abhishekkack05:32
lbragstadbut it should be less05:32
abhishekkright05:32
lbragstadhttps://review.opendev.org/c/openstack/glance/+/765210/13 might be worth abandoning05:32
abhishekkyeah, we could revisit it in X05:33
abhishekkI will try to work on tasks later today05:33
lbragstadhttps://review.opendev.org/c/openstack/glance/+/775741 and https://review.opendev.org/c/openstack/glance-tempest-plugin/+/775742/3 should take care of the rest of the image API05:33
abhishekkack05:33
lbragstadi plan on doing those tomorrow05:34
*** gyee has quit IRC05:34
abhishekkcool05:34
lbragstadthen it should just be tasks and metadata, right?05:34
abhishekkyes05:34
lbragstadok05:34
abhishekkthank you for your efforts (tireless)05:35
lbragstadthose patches i just pushed pass all tests for me locally and the new tests against gtp05:35
lbragstadbut - feel free to tweak or recheck if something breaks05:35
lbragstadi wouldn't say tirelessly, because i'm going to go pass out :)05:35
lbragstadthanks for all the reviews abhishekk05:35
abhishekkack, I will try some checks as well05:36
abhishekkso last thing we need is to run gtp on glance jobs, am I right ?05:37
lbragstadyeah - i think so05:37
abhishekkok, its late for you, good night05:38
lbragstador we cna enable a gtp job after we get the first patch into glance for the images api (and make it non-voting)05:38
lbragstadbut - that's up to you05:38
*** whoami-rajat has joined #openstack-glance05:41
abhishekkack05:46
*** ricolin has quit IRC06:00
openstackgerritAbhishek Kekane proposed openstack/glance master: Expand tasks database table to add more columns  https://review.opendev.org/c/openstack/glance/+/76373906:00
openstackgerritAbhishek Kekane proposed openstack/glance master: Extract req.context in a variable  https://review.opendev.org/c/openstack/glance/+/77559606:00
openstackgerritAbhishek Kekane proposed openstack/glance master: Utilize newly added tasks database fields  https://review.opendev.org/c/openstack/glance/+/77461506:00
openstackgerritAbhishek Kekane proposed openstack/glance master: New API /v2/images/{id}/tasks  https://review.opendev.org/c/openstack/glance/+/77483006:00
openstackgerritAbhishek Kekane proposed openstack/glance master: Update API docs for new /v2/images/{image_id}/tasks API  https://review.opendev.org/c/openstack/glance/+/77559706:00
openstackgerritAbhishek Kekane proposed openstack/glance master: Bump Images API version to 2.12  https://review.opendev.org/c/openstack/glance/+/77586006:00
*** m75abrams has joined #openstack-glance06:03
*** ricolin has joined #openstack-glance06:28
openstackgerritMerged openstack/glance master: Add basic/common personas to base policies  https://review.opendev.org/c/openstack/glance/+/76424107:05
dirtwashabhishekk: u didnt have time to take a look at my xx bytes written out of issue right?07:27
dirtwashwonder if something is just counted wrong07:27
dirtwashbut i cant imagine being the first to hit that issue, seems unlikely07:27
abhishekkdirtwash, sorry, I tried, but not able to reproduce it locally07:27
dirtwashd you think its a glance issue or ceph related somehow07:28
abhishekkis it possible for you to add reproduce steps (what exactly you are doing step by step) to launchpad?07:29
abhishekkI didn't see anything wrong in glance code yet07:29
dirtwashliterally just " create snapshot by any means, ui/cli"07:29
dirtwashnot many steps there07:29
abhishekkyou are using glanceclient ?07:29
dirtwash tried with the normal openstack cli commands to make a new image and otherwies via webui "create image"07:30
dirtwashso not sure glanceclient07:30
dirtwashim not a openstack expert, new to it07:30
dirtwashbut it always fails, only thing that works is importing a new image to the rbd backend, there is no issue then07:31
abhishekkcould you do it with glance image-create ?07:32
dirtwashyes I'll try now07:33
abhishekkSo please add both the CLI commands you are using07:33
abhishekk1 for creating snapshot07:33
abhishekk2 for importing new image07:33
dirtwashhm online it says for snapshot to use 'nova image-create'07:34
dirtwashtat it what i used last time07:35
*** rcernin has quit IRC07:39
dirtwashI can create images just fine, just not sure how to make a snapshot via cli, they all use nova in examples07:40
*** ralonsoh has joined #openstack-glance07:48
abhishekkdirtwash, nova image-create means you are trying to take snapshot of an instance08:05
abhishekkI need to check that08:05
dirtwashis it not the same?08:07
dirtwashisnt it using glance to do this?08:07
* dirtwash is confused by openstack08:07
abhishekkAFAIK, nova image-create is used for taking snapshot of instance08:10
dirtwashyea and that works08:12
dirtwashnot08:12
dirtwashI meant, it brings the same error08:12
dirtwashIm sure that is what te webgui does when I click 'create image'08:12
dirtwashso nova imgage-create fails08:13
dirtwashand glance logs that error as said08:13
*** rcernin has joined #openstack-glance08:15
*** rcernin has quit IRC08:31
*** rcernin has joined #openstack-glance08:33
*** rcernin has quit IRC09:08
*** hoonetorg has joined #openstack-glance09:16
*** tosky has joined #openstack-glance09:18
*** rcernin has joined #openstack-glance10:27
dirtwashabhishekk: the nova snapshot command doesnt trow an error, it says it made the image, but the image doesnt exist , it disappears after the error10:34
*** rcernin has quit IRC10:42
*** k_mouza has joined #openstack-glance10:48
*** rcernin has joined #openstack-glance10:52
*** k_mouza has quit IRC10:55
*** k_mouza has joined #openstack-glance10:56
*** k_mouza_ has joined #openstack-glance11:03
*** k_mouza has quit IRC11:06
*** rcernin has quit IRC11:12
*** udesale_ has joined #openstack-glance11:14
*** udesale has quit IRC11:18
*** k_mouza_ has quit IRC11:45
*** k_mouza has joined #openstack-glance11:46
*** ratailor has quit IRC12:07
*** k_mouza has quit IRC12:08
*** k_mouza has joined #openstack-glance12:21
jokkedirtwash: you need to look into your Nova logs, sounds like Nova is having issue and after that it cleans up after itself12:44
*** ajitha has joined #openstack-glance12:47
dirtwashjokke: nova says pipe error but i suspect because the rbd write failure happens in glance12:55
dirtwashit cleans up because of that12:56
jokkeYou're not using the same ceph for ephemeral, I take?12:57
dirtwashwe only use ceph rbd backend for images, not sure what you mean with ephemeral12:58
dirtwashotherwise local disk on hypervisors12:58
jokkeyeah, that's what I meant, nova is not sharing that same ceph12:58
dirtwashim confused13:00
dirtwashbut it is, there is only 1?13:00
dirtwashand it worked before vitoria13:00
dirtwashwe didnt change anyting on ceph13:00
dirtwashand it only affects snapshots13:00
jokkeAre you using multi-store config (do you configure the ceph store in it's own section in the gapi config or do you have the config just under ceph in the glance_store section?13:01
dirtwashI didnt set this up but te glanci-api.conf has [rbd] mentionnig ceph stuff, glance_store only has default_backend=rbd13:04
jokkeok13:04
dirtwashjokke: [rbd] has like pool and user naname and conf location and chunk size, nothing else13:06
jokkeyep13:06
dirtwashim totally new to openstack, someone else set this up, im just tryin to help with debug, i m the ceph admin13:07
jokkeI was just thinking if we have opportunity to test something, but we really don't with that configuration13:07
dirtwashall I know is thatit worked before the upgrade13:08
jokkeyeah, only thing forward, I'd put Glance and Nova to debug logging and see if the logs reveals what and why is not being happy13:11
jokkeIn glance point of view there is no difference between image and snapshot13:12
jokkeso creating snapshot is just like creating image, if either works on Glance both _should_ work13:12
openstackgerritLance Bragstad proposed openstack/glance master: Implement project personas for image actions  https://review.opendev.org/c/openstack/glance/+/76475413:14
dirtwashjokke: creating an image works fine, importing an image works fine, snashot doesnt, I alraedy set to debug, neither showed more info13:17
dirtwashjokke: thats all the log I got https://bugs.launchpad.net/glance/+bug/191648213:18
openstackLaunchpad bug 1916482 in Glance "rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes) since Victoria Upgrade, ceph v nautilus" [Undecided,New]13:18
dirtwashim happy to try something and paste u the logs13:18
*** k_mouza has quit IRC13:31
*** k_mouza has joined #openstack-glance13:32
jokkeyeah, I'm not sure what's going on there. Wondering if the very small size and the resizing is affecting that somehow13:32
jokkehave you tried if you get same error creating image with just few kilos of data?13:34
*** Luzi has joined #openstack-glance13:47
openstackgerritDan Smith proposed openstack/glance master: Test restarting glance for staging cleanup  https://review.opendev.org/c/openstack/glance/+/77701414:05
*** Luzi has quit IRC14:28
dirtwashjokke: i havent tried that, i know its creating size0 image for snapshot14:31
dirtwashsoi should make one thats just few kb big?14:31
jokkedirtwash: I would try just using small amout of random data as image file and I think the snapshot rezise message in logs was very small14:36
dirtwash100K testfile maybe?14:38
dirtwashor smaller14:38
dirtwashso far making images from file worked14:40
dirtwashworked fine for 100Kb14:41
*** Luzi has joined #openstack-glance14:43
dansmithI don't really know stuff about alembic, but what can we add to a job to verify that we have this empty migrate thing if/when we need it? Is there some db manage thing we can run in post that will fail to contract if we are missing it or something?15:12
dansmithsounds like it's a bit of tribal knowledge at the moment, and we should really make the computers do our work for us15:12
dansmithabhishekk: jokke rosmaita ^15:12
*** lpetrut has joined #openstack-glance15:13
rosmaitayeah, i am actually not 100% clear on that15:14
rosmaitaour dev docs say we should do it15:14
abhishekkthere is db_check command for glance-manage but not sure it will help15:14
rosmaitabut it could just be for consistency15:14
dansmithokay, if nothing breaks I'm not sure why it's important either :)15:14
rosmaitawell, there's "nothing breaks in devstack" vs "nothing breaks"15:15
dansmithrosmaita: well, that's what I'm asking.. if something breaks not in devstack, we should make devstack or our jobs tickle whatever will break :)15:15
dansmithmaybe just a functional test to make sure we have a matching migrate for each version we have an expand for?15:15
rosmaitathat would be good15:16
rosmaita"A given database change may not require actions in each of the expand, migrate, contract phases, but nonetheless, we require a script for each phase for every change. In the case where an action is not required, a no-op script, described below, MUST be used."15:16
rosmaitahttps://docs.openstack.org/glance/latest/contributor/database_migrations.html15:16
rosmaitabut i don't recall if that's a shit can break MUST or a working in glance MUST15:17
dansmithokay, so maybe just convention but nothing will actually break?15:17
dansmithokay15:17
*** bhagyashris is now known as bhagyashri|ruck15:20
rosmaitayeah, it may be one of those nothing will break until it does15:20
*** abhishekk is now known as abhishekk|afk15:20
rosmaitaand this is a belt-and-suspenders approach15:21
*** lpetrut has quit IRC15:24
dansmithrosmaita: the test I'm working on tells me that ocata doesn't have the migrate phase (which is true)15:37
dansmithso (a) we need a test and (b) I imagine we would have heard about a problem by now if there was one :)15:37
rosmaitai think ocata was community images, there should be a real migrate phase15:38
dansmiththe only ocata files are:15:39
dansmith-rw-rw-r-- 1 dan dan 2.3K Aug 17  2020 ocata_contract01_drop_is_public.py15:39
dansmith-rw-rw-r-- 1 dan dan 5.7K Aug 17  2020 ocata_expand01_add_visibility.py15:39
rosmaitaneed to look in /glance/db/sqlalchemy/alembic_migrations/data_migrations/15:39
rosmaitadon't ask me why, there was a good reason, but i don't remember15:39
dansmiththose should be different than schema migrations right?15:41
rosmaitayeah, those are the actual data migrations15:41
rosmaitathe schema changes are expand and contract15:42
dansmithah, so all the migrates should be in the data_migrations directory and expand/contract in versions?15:42
dansmithI see, that looks like how it is15:42
rosmaitayeah15:42
*** abhishekk|afk is now known as abhishekk15:47
openstackgerritDan Smith proposed openstack/glance master: Add a test for migration naming and phase rules  https://review.opendev.org/c/openstack/glance/+/77741315:52
dansmithrosmaita: how's that ^ ?15:52
dansmithwe have a lot of these "developer trap" tests in nova which helps people submit patches with these sorts of things already sorted, and avoids every core having to remember these kinds of mechanical rules15:53
rosmaitagreat idea, will look at the patch15:55
rosmaitawhile you are thinking about this ... i think our code will break when openstack goes from 'zoo' to 'aardvark' release15:57
dansmithdo we have some release[0] ordering somewhere?15:57
*** m75abrams has quit IRC15:57
rosmaitapretty sure it's assumed in the migration code somewhere15:58
dansmithokay I thought the whole point of alembic was to avoid things like migration numbers15:58
dansmithbut like I said, I'm alembic stupid15:58
rosmaitayeah, i think this is a glance-imposed restriction when it orders the files to apply15:59
dansmithhuh, okay15:59
dansmithI've looked briefly at neutron things and they have uuids for some of that, but I guess I thought alembic looks at the "down_revision" or "parent" or something to construct the ordering16:00
rosmaitalooking at your test, i think we ran into a problem with mitaka and that's why the naming convention and all-3-files inclusion was introduced16:02
dansmithack, I figured something like that16:03
toskywhile you discuss this, may I ask for a review on this zuul stable/train backport (legacy->native job)? https://review.opendev.org/c/openstack/glance_store/+/74923516:05
rosmaitatosky: you are so polite, how can we refuse?16:05
*** k_mouza_ has joined #openstack-glance16:14
dansmithabhishekk: just checking, but did you notice that housekeeping patch has a test cleanup below it?16:15
toskyehm :)16:16
toskydansmith: thanks, answered :)16:16
dansmithtosky: just checking to make sure you're paying attention :P16:17
dansmithI don't have +2 on stable so I can't help you further16:17
*** k_mouza has quit IRC16:17
*** gyee has joined #openstack-glance16:41
toskyrosmaita: aaand thanks!16:43
rosmaitayw16:44
*** udesale_ has quit IRC16:54
abhishekkdansmith, yes, I noticed16:55
abhishekkjust waiting for gate results16:55
dansmithoh, gate reported last night, but that's cool, I just slipped that in since you looked yesterday so wasn't sure if you noticed16:56
abhishekkyep, you have changed to use glanceclient from openstackclient16:59
abhishekkand that's why waiting to make sure it passes again17:00
dansmithabhishekk: ah, no I mean the patch underneath17:00
dansmiththe one on the top is probably just DNM17:00
abhishekkohh, that one17:01
abhishekkcan we have related bug tag in commit, so if we decided to backport it?17:02
openstackgerritLance Bragstad proposed openstack/glance master: Implement project personas for image actions  https://review.opendev.org/c/openstack/glance/+/76475417:03
openstackgerritLance Bragstad proposed openstack/glance-tempest-plugin master: Add tests for image membership, deactivation, and reactivation  https://review.opendev.org/c/openstack/glance-tempest-plugin/+/77574217:03
abhishekklbragstad, you can abandon other patches which are not required now17:04
lbragstadsure17:05
lbragstadso - from what i can tell, https://review.opendev.org/c/openstack/glance/+/764754/14 implements project-reader and project-member for all image APIs17:07
lbragstadi obviously left the project-admin stuff alone17:07
abhishekkack17:10
abhishekkproject-admin is == current rule:admin ?17:10
*** k_mouza_ has quit IRC17:14
*** k_mouza has joined #openstack-glance17:14
lbragstadabhishekk yeah17:24
lbragstadtechnically - rule:admin works for system-admin, too17:25
abhishekkmakes sense17:25
dansmithabhishekk: sorry, call.. sure, I'll tag it17:31
abhishekkdansmith, no problem17:31
openstackgerritDan Smith proposed openstack/glance master: Make functional tests set node_staging_uri  https://review.opendev.org/c/openstack/glance/+/77727717:32
openstackgerritDan Smith proposed openstack/glance master: Add housekeeping module and staging cleaner  https://review.opendev.org/c/openstack/glance/+/77701217:32
openstackgerritDan Smith proposed openstack/glance master: MDNM: Test restarting glance for staging cleanup  https://review.opendev.org/c/openstack/glance/+/77701417:33
dansmithabhishekk: marked my intentions to probably not merge the top patch, unless we change our mind ^17:33
abhishekkdansmith, ack, I guess we should avoid the post checking (as after successful run if something went wrong then we need to run it again)17:34
dansmithwell, this post check doesn't actually check anything, it just does some setup and restarts the service.. you have to look at the log file to see the behavior anyway17:35
dansmithI just wanted some way to demonstrate that it was working for all three cases in reality,17:35
dansmithbut once I was able to restart the functional worker without destroying the database, that need was solved there17:36
dansmiths/was able/figured out how to/17:36
abhishekkack17:40
* abhishekk signing off for the day17:41
openstackgerritMerged openstack/glance_store stable/train: zuul: glance_store-src-ceph-tempest replaces a legacy job  https://review.opendev.org/c/openstack/glance_store/+/74923517:43
*** Luzi has quit IRC17:56
*** ralonsoh has quit IRC18:00
*** k_mouza has quit IRC19:00
*** gregwork has joined #openstack-glance19:06
*** rcernin has joined #openstack-glance19:10
*** rcernin has quit IRC19:15
*** k_mouza has joined #openstack-glance20:14
*** k_mouza has quit IRC20:19
*** rcernin has joined #openstack-glance20:33
*** zzzeek has quit IRC20:34
*** zzzeek has joined #openstack-glance20:34
*** rcernin has quit IRC20:47
*** jdillaman has joined #openstack-glance21:04
*** whoami-rajat has quit IRC21:12
*** rcernin has joined #openstack-glance21:14
*** rcernin has quit IRC21:19
*** zzzeek has quit IRC21:26
*** zzzeek has joined #openstack-glance21:28
*** rcernin has joined #openstack-glance21:32
lbragstadrandom question - is it possible for metadef sub-objects to have different owners than the namespace?21:34
*** rcernin has quit IRC21:37
*** rcernin has joined #openstack-glance21:40
*** rcernin has quit IRC21:45
*** rcernin has joined #openstack-glance21:46
*** zzzeek has quit IRC21:49
*** rcernin has quit IRC21:51
*** zzzeek has joined #openstack-glance21:53
*** rcernin has joined #openstack-glance22:15
*** zzzeek has quit IRC23:14
*** zzzeek has joined #openstack-glance23:15
*** ajitha has quit IRC23:26
*** jv_ has quit IRC23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!