| *** gyee has quit IRC | 00:20 | |
| *** k_mouza has joined #openstack-glance | 01:12 | |
| *** k_mouza has quit IRC | 01:16 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 02:44 |
|---|---|---|
| *** baojg has quit IRC | 03:10 | |
| *** Liang__ has joined #openstack-glance | 03:31 | |
| openstackgerrit | Merged openstack/glance master: Make our tempest job use import, standalone, and inject_metadata https://review.opendev.org/741276 | 04:12 |
| *** ratailor has joined #openstack-glance | 04:14 | |
| *** ratailor_ has joined #openstack-glance | 04:26 | |
| *** ratailor has quit IRC | 04:26 | |
| *** m75abrams has joined #openstack-glance | 04:53 | |
| *** ajitha has joined #openstack-glance | 05:09 | |
| *** udesale has joined #openstack-glance | 05:27 | |
| *** rosmaita has quit IRC | 05:41 | |
| *** rosmaita has joined #openstack-glance | 05:41 | |
| *** ajitha has quit IRC | 06:09 | |
| *** nikparasyr has joined #openstack-glance | 06:31 | |
| *** amoralej|off is now known as amoralej | 07:22 | |
| nikparasyr | goodmorning. any1 using https://docs.openstack.org/glance/stein/admin/interoperable-image-import.html for auto-convert and/or injecting default image properties? I have some issues using it | 07:23 |
| *** ralonsoh has joined #openstack-glance | 07:29 | |
| *** rcernin has quit IRC | 07:32 | |
| *** baojg has joined #openstack-glance | 08:26 | |
| openstackgerrit | Abhishek Kekane proposed openstack/glance_store master: PoC Cinder multiple stores https://review.opendev.org/742371 | 08:44 |
| openstackgerrit | Abhishek Kekane proposed openstack/glance master: PoC for lazy loading cinder store https://review.opendev.org/742373 | 08:49 |
| *** ratailor__ has joined #openstack-glance | 09:04 | |
| *** ratailor_ has quit IRC | 09:06 | |
| * abhishekk will be back around 1400 UTC | 09:07 | |
| *** rcernin has joined #openstack-glance | 09:10 | |
| *** rcernin has quit IRC | 09:35 | |
| *** Liang__ has quit IRC | 09:40 | |
| *** k_mouza has joined #openstack-glance | 09:53 | |
| *** udesale_ has joined #openstack-glance | 09:58 | |
| *** udesale has quit IRC | 10:01 | |
| *** priteau has joined #openstack-glance | 10:33 | |
| *** tkajinam has quit IRC | 10:59 | |
| jokke | nikparasyr: what's up | 11:06 |
| nikparasyr | jokke hello :). I have enabled the image_conversion and inject_image_metadata plugins. The config can be seen here: https://pastebin.com/GKYkgAX3. after reconfiguring I create an image via: glance image-create-via-import --disk-format raw --container-format bare --name conversion_test --import-method web-download --uri https://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img. The image is still qcow2, that is what | 11:38 |
| nikparasyr | image show indicates and also if i download and check the format its qcow2. properties are also not added. I'm running a 3-node glance-api with haproxy on top. based on documentation the underlying filesystem between the nodes should be shared, which is not the case in my setup. I switched off the glance-api from 2 of the nodes and tried with the same results. | 11:38 |
| nikparasyr | debug level logs do not have any error, they show the flow transitioning from imported etc but no "convert" step | 11:38 |
| nikparasyr | question is am i possibly missing something in the configuration? | 11:39 |
| nikparasyr | store is ceph | 11:39 |
| *** rcernin has joined #openstack-glance | 11:49 | |
| *** baojg has quit IRC | 11:49 | |
| *** baojg has joined #openstack-glance | 11:50 | |
| *** amoralej is now known as amoralej|lunch | 12:05 | |
| *** baojg has quit IRC | 12:11 | |
| *** baojg has joined #openstack-glance | 12:12 | |
| jokke | nikparasyr: I don't see the staging path in your config | 12:27 |
| nikparasyr | isnt the default /tmp/staging used? | 12:27 |
| nikparasyr | sorry im using kolla-ansible to deploy, and the snippet i sent is basically what im overwriting to the default configuration | 12:28 |
| jokke | ah ok ... sorry | 12:29 |
| jokke | so you have that folder and it creates the image, you have your metadata injected but no convert? | 12:29 |
| nikparasyr | no covnert and also no metadata injected | 12:29 |
| jokke | how is kolla starting glance (if you check `ps aux | grep glance-api` is it specifying config file or config folder? | 12:31 |
| nikparasyr | gimme a sec to check | 12:31 |
| jokke | sure | 12:31 |
| jokke | iirc we have had problems getting the extra configs picked up unless the config dir is used so that might be a thing if it just points to glance-api.conf | 12:32 |
| nikparasyr | runs inside a docker container. ps aux | grep glance on the container gives: glance 24 0.0 0.1 549944 110196 ? S Jul21 0:00 /usr/bin/python2 /usr/bin/glance-api | 12:33 |
| nikparasyr | the config files are under /etc/glance | 12:33 |
| nikparasyr | yday i also configured adding property policy file and that worked well, the difference though is that for that one you explicitely define the property policy file location on the glance-api.conf | 12:36 |
| *** rm_work has quit IRC | 12:38 | |
| jokke | yeah | 12:39 |
| *** mordred has quit IRC | 12:40 | |
| jokke | so if you start glance-api with debug logging, it lists all it's config options it picks up at the start, I'm pretty sure you won't see image_import_opts.image_import_plugins nor inject_metadata_properties & image_conversion groups there | 12:41 |
| jokke | let me see if we managed to fix the config pickup issue and which branches we actually backported it | 12:41 |
| *** rm_work has joined #openstack-glance | 12:42 | |
| nikparasyr | ok. i was going through the logs yday after reconfiguring and setting debug mode. i didnt notice something like that and i think i would. I'll run it again in a bit to doublecheck | 12:43 |
| *** ratailor__ has quit IRC | 12:45 | |
| nikparasyr | btw in case that this is useful: | 12:46 |
| nikparasyr | (glance-api)[root@control-t01 /]# glance-api --version | 12:46 |
| nikparasyr | 18.0.1 | 12:46 |
| *** mordred has joined #openstack-glance | 12:49 | |
| jokke | nikparasyr: https://bugs.launchpad.net/glance/+bug/1775782 I think you're hitting this one | 12:52 |
| openstack | Launchpad bug 1775782 in Glance "glance-image-import.conf not parsed when running under wsgi" [Critical,In progress] - Assigned to Erno Kuvaja (jokke) | 12:52 |
| jokke | or actually pretty sure you are | 12:52 |
| jokke | nikparasyr: so indeed the config file is not being picked up | 12:53 |
| jokke | nikparasyr: so 2 quick approaches to get you moving with that. 1st and least intrusive is to change the glance startup script to run it with --config-dir parameter or 2nd option would be to modify the glance/cmd/api.py as abhishekk mentioned in https://bugs.launchpad.net/glance/+bug/1775782/comments/2 | 12:56 |
| openstack | Launchpad bug 1775782 in Glance "glance-image-import.conf not parsed when running under wsgi" [Critical,In progress] - Assigned to Erno Kuvaja (jokke) | 12:56 |
| nikparasyr | ok. i'll have a look. thanks a lot for the help. i'll keep you updated. Another question is: for these features to work is it a hard requirement to have either a single-node glance or if multinode have a shared fs below? | 13:04 |
| *** amoralej|lunch is now known as amoralej | 13:19 | |
| *** baojg has quit IRC | 13:23 | |
| *** baojg has joined #openstack-glance | 13:24 | |
| *** rcernin has quit IRC | 13:32 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 13:32 |
| dansmith | abhishekk: I forgot to set the interpreter config in ^ but it _did_ try image conversion in wsgi in CI at least | 13:33 |
| jokke | nikparasyr: for glance-direct it is atm. | 13:34 |
| jokke | nikparasyr: we're trying to find our ways around that but the problem with glance-direct is that the stage and the import calls might land on different nodes, which will fail the import if it doesn't have access to the staged data. web-download works just fine even with split staging | 13:35 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make glance-api able to do async tasks in WSGI mode https://review.opendev.org/742065 | 13:38 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 13:38 |
| jokke | dansmith: that starts to look promising | 13:38 |
| dansmith | jokke: as of that second patch I don't know of anything that doesn't work under uwsgi | 13:39 |
| dansmith | so, yes :) | 13:39 |
| nikparasyr | jokke: thanks for this as well. in our case these features are quite important but switching to single-node only to have them might be a big trade-off that we might not be willing to make. in any case. figuring out how to config kolla to have glance-api start with --config-dir | 13:41 |
| *** baojg has quit IRC | 13:41 | |
| *** baojg has joined #openstack-glance | 13:43 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 13:45 |
| smcginnis | dansmith: You're just doing whatever you can to avoid working on that fence. | 13:47 |
| dansmith | hehe | 13:47 |
| jokke | :P | 13:50 |
| abhishekk | dansmith, ack | 13:57 |
| dansmith | abhishekk: what else do you think we need to see to get the wsgi stuff merged? | 13:57 |
| *** Liang__ has joined #openstack-glance | 13:58 | |
| abhishekk | dansmith, I remember we introduced that async pool for graceful shutdown of api service, I need to check if it is working with uwsgi (which last time I checked was not working) | 13:59 |
| *** Liang__ is now known as LiangFang | 13:59 | |
| abhishekk | But that can be fixed afterwards as well | 14:00 |
| dansmith | abhishekk: oh do you drain the thread pool on exit? point me at that and I can look too, although yeah, I don't think this makes it any worse :) | 14:00 |
| abhishekk | Also as commented on the patch we need to add periodic job support for prefetcher | 14:00 |
| abhishekk | dansmith, give me some time, I will let you know | 14:01 |
| dansmith | we need that before we merge it or we need that before it's considered done? ;) | 14:01 |
| dansmith | just asking because there are so many inter-dependent patches up right now and it would help to merge things so we can start to unwind that | 14:02 |
| abhishekk | we need that before it is considered as done | 14:06 |
| dansmith | ack | 14:06 |
| dansmith | abhishekk: stores-info is an api call I can make to glance? | 14:09 |
| abhishekk | dansmith, yes | 14:09 |
| dansmith | is that info/stores like info/import ? | 14:09 |
| abhishekk | yes | 14:10 |
| dansmith | okay | 14:10 |
| *** LiangFang has quit IRC | 14:12 | |
| dansmith | abhishekk: so stores is either an empty array or an array with some things, and if it's anything other than empty, then multistore is enabled? | 14:12 |
| abhishekk | dansmith, yes | 14:12 |
| dansmith | okay | 14:12 |
| abhishekk | https://docs.openstack.org/api-ref/image/v2/?expanded=list-stores-detail#list-stores | 14:13 |
| *** Liang__ has joined #openstack-glance | 14:14 | |
| * abhishekk still need to work on policy doc :( | 14:14 | |
| dansmith | abhishekk: yeah I saw that, it just isn't clear what it looks like if stores is not enabled, AFAICT | 14:16 |
| abhishekk | dansmith, if stores is not enabled then it will return {"stores": []} | 14:16 |
| dansmith | roger | 14:16 |
| dansmith | abhishekk: it's not clear to me if the same rule applies to all_stores | 14:21 |
| dansmith | I assume it does | 14:22 |
| *** baojg has quit IRC | 14:22 | |
| abhishekk | dansmith, it applies for all_stores_must_succeed only | 14:23 |
| dansmith | okay that's ... confusing | 14:23 |
| *** baojg has joined #openstack-glance | 14:24 | |
| dansmith | gmann: if I need to rise an exception from images_client because the test asked for something the server can't do due to config, is InvalidConfiguration legit to use? | 14:25 |
| openstackgerrit | Erno Kuvaja proposed openstack/glance stable/train: Fix reloading under PY3 https://review.opendev.org/713720 | 14:32 |
| abhishekk | dansmith, jokke sorry I need to drop early today, medical emergency, will try to come back in 2-3 hours | 14:33 |
| dansmith | abhishekk: np, thanks for your help | 14:33 |
| abhishekk | thanks | 14:34 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make glance-api able to do async tasks in WSGI mode https://review.opendev.org/742065 | 14:35 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 14:35 |
| *** nikparasyr has left #openstack-glance | 14:54 | |
| *** k_mouza has quit IRC | 15:00 | |
| *** k_mouza has joined #openstack-glance | 15:01 | |
| *** Liang__ has quit IRC | 15:15 | |
| *** Liang__ has joined #openstack-glance | 15:19 | |
| *** Liang__ has quit IRC | 15:23 | |
| *** rcernin has joined #openstack-glance | 15:24 | |
| *** rcernin has quit IRC | 15:29 | |
| *** k_mouza has quit IRC | 15:41 | |
| *** k_mouza has joined #openstack-glance | 15:44 | |
| *** gyee has joined #openstack-glance | 15:48 | |
| *** baojg has quit IRC | 15:54 | |
| *** k_mouza has quit IRC | 15:55 | |
| *** baojg has joined #openstack-glance | 15:56 | |
| *** m75abrams has quit IRC | 16:21 | |
| *** baojg has quit IRC | 16:34 | |
| *** udesale_ has quit IRC | 16:47 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make wsgi_app support graceful shutdown https://review.opendev.org/742493 | 17:05 |
| dansmith | jokke: your config-reload expectation is that SIGHUP triggers the reload, is that right? | 17:06 |
| *** amoralej is now known as amoralej|off | 17:08 | |
| dansmith | AFAICT, we get that for free with the way uwsgi does its worker recycling, so if I HUP the master process, I see it barf out new config values that I've changed since the first start | 17:15 |
| dansmith | abhishekk: ^ | 17:15 |
| dansmith | smcginnis: I will reward your ninja-like review with a fix of your nit :) | 17:18 |
| smcginnis | Haha, great. | 17:18 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make wsgi_app support graceful shutdown https://review.opendev.org/742493 | 17:18 |
| dansmith | smcginnis: bad news, the lumber truck just showed up :/ | 17:23 |
| dansmith | I'm running out of excuses | 17:23 |
| smcginnis | You need to let the lumber adapt to the local heat and humidity. So really you should wait a few days before you work with it. :) | 17:25 |
| smcginnis | Don't want a warped fence. | 17:25 |
| dansmith | ooh, like a goldfish? | 17:25 |
| smcginnis | Haha | 17:25 |
| *** k_mouza has joined #openstack-glance | 17:55 | |
| *** k_mouza has quit IRC | 17:59 | |
| gmann | dansmith: from client we can raise what ever API reply. and config based request param we can handle on test side. i mean client call API with whatever param test want and return what API return | 18:00 |
| dansmith | gmann: well, I wanted to raise an exception that isn't an API exception, because abhishekk says we need to avoid putting things into the request body if the server doesn't support something, | 18:01 |
| dansmith | so we need to probe | 18:01 |
| dansmith | gmann: here https://review.opendev.org/#/c/741425/10/tempest/lib/services/image/v2/images_client.py@243 | 18:02 |
| gmann | dansmith: yeah, that we can handle on test side not to add in request body | 18:02 |
| gmann | that is how we do for all other client where method accept **kwargs so if we need to write negative test for unknown param, then test should be able to pass it till API and get what API return | 18:03 |
| dansmith | gmann: this is just related to your ask for including all the params in the client | 18:03 |
| dansmith | gmann: ah, so you want the client to put them in there if the test asks, and make the test check to see if it should or not first? | 18:03 |
| gmann | yeah, we can add those in request body in client based of 'If not None' | 18:04 |
| gmann | like url i added in web-download case | 18:04 |
| dansmith | okay | 18:07 |
| dansmith | I'll change it again | 18:07 |
| gmann | dansmith: you can just add those param in client method with None default and when we will write copy test then we can add those param with /info/store checks | 18:11 |
| dansmith | yep, that's what I'm doing.. is it okay to leave the has_multistore check there for later since I already wrote it? | 18:11 |
| gmann | dansmith: few comment, +1 on keeping that method but let's return resp body to give store names to test | 18:19 |
| dansmith | ah yea | 18:19 |
| dansmith | I guess I'll remove the caching then | 18:19 |
| dansmith | I was going to do that since I was checking inside the import function, but no reason now | 18:19 |
| gmann | dansmith: /info/stores urls need to be corrected - https://review.opendev.org/#/c/741425/12/tempest/lib/services/image/v2/images_client.py@194 | 18:23 |
| dansmith | cripes | 18:24 |
| dansmith | gmann: sorry | 18:24 |
| jokke | dansmith: yeah, it basically spins the workers around without killing any existing connections | 18:30 |
| dansmith | jokke: you mean without dropping tasks I assume? it seems to just propagate the HUP to the children, but then they also seem to ignore further HUP signals after the first one, | 18:32 |
| dansmith | so I assume things are supposed to die off after that too | 18:32 |
| dansmith | uwsgi will SIGTERM the current processes and send new requests to fresh children so I think we don't even need the oslo.config reload stuff for that to work | 18:33 |
| jokke | dansmith: iirc how it works is that it basically keeps X workers running and tells the workers to go and die when you have nothing to do. When the worker finally dies it spawns a new one to replace it | 18:34 |
| dansmith | jokke: you're talking about the standalone based approach right? | 18:34 |
| jokke | yeah | 18:34 |
| dansmith | yeah, okay | 18:34 |
| dansmith | uwsgi just does that for us | 18:35 |
| dansmith | so with the graceful shutdown stuff, we'll get that for free without dropping anything that is mid-flight | 18:35 |
| jokke | so if the worker is streaming image to client, that worker dies when it's finished, istead of interrupting the request it's currently serving | 18:35 |
| dansmith | yup | 18:35 |
| dansmith | jokke: while I have you here... I'm seeing functional test jobs in CI that just hang until timeout, which I think I've seen outside of this async work, but logstash is fubar right now so I can't check | 18:36 |
| dansmith | does that sound familiar? | 18:36 |
| dansmith | I haven't been able to repro locally so I can get a stack trace or anything | 18:36 |
| jokke | yeah, it has been ongoing for couple of weeks. It looked like it corrected itself mid last week but have seen it reappearing again this week | 18:37 |
| jokke | we neither ... have no idea what's causing it | 18:37 |
| dansmith | okay cool, just wanted to make sure it wasn't me causing that with the threading stuff | 18:37 |
| jokke | I don't think so | 18:38 |
| dansmith | jokke: you know anything about the _after_fork warnings in functional? | 18:38 |
| dansmith | I wonder if that's related, but I do see that locally | 18:38 |
| jokke | those are eventlet monkeypatching threading for greenthreads ... | 18:39 |
| dansmith | ah, I see there's an eventlet bug open for it | 18:39 |
| dansmith | with py37 | 18:39 |
| jokke | started py37 when python changed the threading | 18:39 |
| jokke | yup | 18:39 |
| jokke | non related as well as the functional test timeouts happens in both py36 and py38 variants | 18:40 |
| dansmith | well, it seemed like it could be since it's warning about having live threads across fork I think, which could maybe lead to a deadlock sometimes, | 18:41 |
| dansmith | but I guess if 37 brought the problem and the deadlock happens on 36 then probably not | 18:42 |
| jokke | if our py36 job actually runs py36 :P | 18:42 |
| jokke | wouldn't be first time | 18:42 |
| jokke | I think we ran our py27 on py3 for a year or so before noticing | 18:42 |
| jokke | :P | 18:42 |
| dansmith | I thought there was only a 37 and 38 job now | 18:43 |
| * dansmith looks | 18:43 | |
| dansmith | oh, 36 and 38 I see | 18:43 |
| dansmith | I run 37 locally | 18:43 |
| jokke | ok, dinner time. I'll poke my head here after so feel free to ping me if you need anything | 18:44 |
| dansmith | ack | 18:44 |
| dansmith | gmann: do you know any tricks for getting a better state snapshot of a stalled functional/unit test in the gate? we don't get a stack trace from the aborted tox run that I can see | 18:45 |
| *** ralonsoh has quit IRC | 18:45 | |
| gmann | dansmith: afaik job-output.txt is all we can get for tox runs | 18:48 |
| gmann | for aborted, we do not get any of job log even they were finished | 18:49 |
| dansmith | hmm, I dunno what to do then | 18:49 |
| openstackgerrit | Sean McGinnis proposed openstack/glance stable/train: Fix reloading under PY3 https://review.opendev.org/713720 | 18:51 |
| *** baojg has joined #openstack-glance | 19:07 | |
| *** baojg has quit IRC | 19:31 | |
| *** baojg has joined #openstack-glance | 19:32 | |
| openstackgerrit | Merged openstack/glance master: Add image_set_property_atomic() helper https://review.opendev.org/737868 | 20:16 |
| *** priteau has quit IRC | 20:47 | |
| *** baojg has quit IRC | 21:22 | |
| *** baojg has joined #openstack-glance | 21:23 | |
| *** rosmaita has quit IRC | 21:44 | |
| *** rosmaita has joined #openstack-glance | 21:46 | |
| *** rcernin has joined #openstack-glance | 22:51 | |
| *** rcernin has quit IRC | 22:58 | |
| *** rcernin has joined #openstack-glance | 23:01 | |
| *** rcernin has quit IRC | 23:04 | |
| *** rcernin has joined #openstack-glance | 23:05 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!