| *** jmlowe has joined #openstack-glance | 00:24 | |
| *** jmlowe has quit IRC | 00:33 | |
| openstackgerrit | kayluu proposed openstack/glance-specs master: Update proposal for duplication image download https://review.opendev.org/734683 | 00:55 |
|---|---|---|
| *** Liang__ has joined #openstack-glance | 01:18 | |
| *** Liang__ has quit IRC | 02:06 | |
| *** Liang__ has joined #openstack-glance | 02:10 | |
| *** Liang__ has quit IRC | 02:15 | |
| *** Liang__ has joined #openstack-glance | 02:21 | |
| *** jmlowe has joined #openstack-glance | 02:41 | |
| *** Liang__ has quit IRC | 02:44 | |
| *** rcernin has quit IRC | 03:08 | |
| *** rcernin has joined #openstack-glance | 03:54 | |
| *** rcernin has quit IRC | 03:55 | |
| *** rcernin has joined #openstack-glance | 03:55 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-glance | 04:33 | |
| *** Liang__ has joined #openstack-glance | 04:56 | |
| *** gyee has quit IRC | 05:11 | |
| *** Liang__ has quit IRC | 05:38 | |
| *** Liang__ has joined #openstack-glance | 05:39 | |
| *** Liang__ has quit IRC | 05:43 | |
| *** Liang__ has joined #openstack-glance | 05:44 | |
| *** ratailor has joined #openstack-glance | 05:44 | |
| *** m75abrams has joined #openstack-glance | 05:54 | |
| *** m75abrams has joined #openstack-glance | 05:56 | |
| *** udesale has joined #openstack-glance | 06:02 | |
| *** udesale has quit IRC | 06:17 | |
| *** udesale has joined #openstack-glance | 06:18 | |
| *** belmoreira has joined #openstack-glance | 06:38 | |
| *** jmlowe has quit IRC | 06:45 | |
| *** jawad_axd has joined #openstack-glance | 07:21 | |
| openstackgerrit | Rajat Dhasmana proposed openstack/glance_store master: Don't allow image creation with encrypted nfs volumes https://review.opendev.org/732506 | 07:29 |
| *** rcernin has quit IRC | 08:00 | |
| *** rcernin has joined #openstack-glance | 08:50 | |
| *** rcernin has quit IRC | 09:17 | |
| *** Liang__ has quit IRC | 09:31 | |
| *** ratailor has quit IRC | 09:38 | |
| *** ratailor has joined #openstack-glance | 09:38 | |
| *** threestrands has quit IRC | 11:10 | |
| *** rcernin has joined #openstack-glance | 11:37 | |
| *** rcernin has quit IRC | 11:43 | |
| *** ratailor has quit IRC | 12:02 | |
| *** tkajinam has quit IRC | 12:25 | |
| *** vkmc has quit IRC | 12:32 | |
| *** mnaser has quit IRC | 12:32 | |
| *** gmann has quit IRC | 12:33 | |
| *** nicolasbock has quit IRC | 12:33 | |
| *** vkmc has joined #openstack-glance | 12:33 | |
| *** nicolasbock has joined #openstack-glance | 12:33 | |
| *** gmann has joined #openstack-glance | 12:33 | |
| *** mnaser has joined #openstack-glance | 12:33 | |
| *** m75abrams has quit IRC | 12:35 | |
| *** m75abrams has joined #openstack-glance | 12:40 | |
| dansmith | abhishekk: did you happen to see my query from yesterday? | 13:38 |
| *** jmlowe has joined #openstack-glance | 13:38 | |
| abhishekk | dansmith, sorry I was on leave yesterday | 13:47 |
| dansmith | abhishekk: no problem, sorry to bug you anyway, but you have "correct answer affinity" :) | 13:48 |
| abhishekk | :D | 13:48 |
| abhishekk | dansmith, yes that config is sufficient | 13:49 |
| dansmith | abhishekk: okay, it doesn't seem to be working in a ci job | 13:49 |
| abhishekk | dansmith, you are using devstack right? | 13:49 |
| dansmith | abhishekk: it's in zuul, so, yeah :) | 13:50 |
| dansmith | does this require wsgi_mode too? | 13:50 |
| abhishekk | yes | 13:50 |
| abhishekk | WSGI_MODE=mod_wsgi | 13:50 |
| abhishekk | otherwise it will not trigger taskflow | 13:50 |
| dansmith | ah, okay.. and is there some delay between uploading and when it is converted? | 13:50 |
| abhishekk | so with taskflow it first uploads image to staging area | 13:50 |
| abhishekk | then it converts it there itself | 13:51 |
| abhishekk | and then it will import it to glance backend | 13:51 |
| abhishekk | so you need to specify os_glance_staging_store in glance-api.conf as well | 13:51 |
| dansmith | I have that, copied from my own config, so coo | 13:52 |
| abhishekk | great | 13:52 |
| dansmith | I was just wondering if there is a race between the upload saying it's done and something trying to use it, | 13:52 |
| dansmith | but devstack probably takes longer to get tempest going so probably not a problem | 13:52 |
| abhishekk | yes, there will not be any problem | 13:53 |
| abhishekk | because the image will be marked active only when everything is done | 13:53 |
| dansmith | well, that only matters if something watches and waits | 13:54 |
| abhishekk | yes | 13:54 |
| abhishekk | will be around next couple of hours, let me know if you have any questions | 13:55 |
| abhishekk | jokke_, rosmaita, smcginnis glance meeting at #openstack-glance | 13:59 |
| *** rosmaita has quit IRC | 13:59 | |
| *** rosmaita has joined #openstack-glance | 14:00 | |
| dansmith | thanks! | 14:06 |
| *** udesale_ has joined #openstack-glance | 14:19 | |
| *** jawad_axd has quit IRC | 14:20 | |
| *** udesale has quit IRC | 14:22 | |
| *** jdillaman has joined #openstack-glance | 15:13 | |
| *** sean-k-mooney has joined #openstack-glance | 15:13 | |
| dansmith | abhishekk: when I turn on mod_wsgi in a devstack job in upstream CI, g-api starts but devstack never finds it | 15:21 |
| dansmith | looks like it's getting 503 from apache failing to proxy | 15:22 |
| dansmith | this worked in my local devstack, but is there something else I need to do? | 15:22 |
| dansmith | apache tls-proxy log shows this: | 15:23 |
| dansmith | 2020-06-11 14:59:39.532624 [proxy:error] [pid 15204:tid 140023944947456] (111)Connection refused: AH00957: HTTP: attempt to connect to 10.0.1.13:19292 (10.0.1.13) failed | 15:23 |
| abhishekk | dansmith, need to check, not sure at this moment :( | 15:26 |
| dansmith | abhishekk: does mod_wsgi not get tested in the gate? I don't see any jobs that use it | 15:26 |
| dansmith | how do task-based features get tested? | 15:26 |
| abhishekk | dansmith, they are not getting tested at the moment for glance | 15:27 |
| dansmith | oye :/ | 15:27 |
| dansmith | g-api is showing listening on port 19191, but that's for registry right? | 15:27 |
| abhishekk | yes | 15:27 |
| dansmith | should g-api be listening on 19292 and 19191 or what? | 15:27 |
| abhishekk | 9292 | 15:27 |
| dansmith | bind_port in glance-api.conf seems to be 19191, so maybe it's just misconfiguring that in the mod_wsgi case? | 15:28 |
| dansmith | or does registry run as a standalone python service and api runs inside mod_wsgi, via apache? | 15:29 |
| abhishekk | yes api runs inside mod_wsgi via apache | 15:29 |
| dansmith | okay I see the glance tls proxy conf which listens on 9292, which proxies to 19292 | 15:30 |
| dansmith | er, 19292 | 15:30 |
| dansmith | and the 9191 to 19191 | 15:30 |
| dansmith | and I see in g-api log that it listens on 19191 | 15:31 |
| dansmith | but I don't see the wsgi config to actually load the glance api, like I do for nova and cinder | 15:31 |
| abhishekk | sounds confusing, because 9191 is used for registry :/ | 15:32 |
| dansmith | that's what I'm saying | 15:32 |
| dansmith | this totally worked for me locally in devstack | 15:33 |
| abhishekk | yeah, I am using same for development since its introduced | 15:34 |
| dansmith | should g-api show listening on 19191 and 19292 in the same log file? | 15:35 |
| dansmith | or should there be a g-reg for 19191 and g-api for 19292? | 15:35 |
| dansmith | I don't even see any config that configures apache to run either | 15:35 |
| abhishekk | thats what I am checking now | 15:35 |
| dansmith | okay, glance-api.conf is configured for 19292 and registry.conf is set for 19191 | 15:36 |
| dansmith | but the g-api log shows 19191 only | 15:36 |
| abhishekk | when I restart g-api locally it shows only log message for 9191 | 15:37 |
| abhishekk | Jun 11 15:36:27 akekane-victoria glance-api[4463]: DEBUG glance.common.config [-] registry_port = 9191 {{(pid=4463) log_opt_values /usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py:2581}} | 15:37 |
| dansmith | okay that's what I see in the log | 15:38 |
| dansmith | and you see apache is listening on 19292? | 15:38 |
| abhishekk | and for 9292 it shows wsgi | 15:38 |
| abhishekk | Jun 11 15:38:13 akekane-victoria glance-api[4688]: INFO eventlet.wsgi.server [-] (4734) wsgi starting up on http://0.0.0.0:9292 | 15:38 |
| dansmith | what log is that? | 15:39 |
| abhishekk | just pasted | 15:39 |
| dansmith | I mean g-api also? | 15:39 |
| abhishekk | yes | 15:40 |
| dansmith | what does your /etc/systemd/system/devstack@g-api.service look like? | 15:40 |
| abhishekk | let me checl | 15:40 |
| abhishekk | http://paste.openstack.org/show/794649/ | 15:41 |
| dansmith | looks to me like the devstack job is configuring it to run /usr/local/bin/glance-api | 15:41 |
| dansmith | yeah, so isn't that weird? that's not mod_wsgi right? | 15:42 |
| abhishekk | yes | 15:42 |
| abhishekk | hmm, yes | 15:43 |
| dansmith | I'm thinking maybe it's *not* doing mod_wsgi.. it's either uwsgi or ... standalone python http server ? | 15:43 |
| dansmith | and the tls proxy config of the upstream devstack job isn't starting or proxying to that standalone service or something | 15:43 |
| abhishekk | https://review.opendev.org/#/c/545483 | 15:45 |
| rosmaita | dansmith: yes, the mod_wsgi is a badly named option | 15:45 |
| dansmith | rosmaita: so confirming that "not uwsgi" results in a standalone? | 15:45 |
| abhishekk | so its running as a standalone service? | 15:45 |
| rosmaita | yes, though i haven't looked at that in a while | 15:46 |
| dansmith | and I'm guessing that nobody runs with tls proxy enabled in their local devstack, and so the lack of a job for this means nobody noticed it was broken in that arrangement? | 15:46 |
| rosmaita | probably | 15:47 |
| dansmith | bummer | 15:47 |
| dansmith | I still don't understand why glance-api as run by systemd isn't listening on both ports like it seems to when abhishekk runs it locally | 15:47 |
| dansmith | is there something in glance-api.conf that says "start an api and a registry" ? | 15:49 |
| abhishekk | nope | 15:49 |
| dansmith | I figured since glance seems to take the config dir, it would start up whatever was configured | 15:49 |
| dansmith | abhishekk: this is the job I'm looking at: https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/logs | 15:49 |
| abhishekk | looking | 15:50 |
| dansmith | here's the g-api log, you can see it doesn't even parse 19292 in configs: https://zuul.opendev.org/t/openstack/build/8afb850f7844450389823d7f84b90c61/log/controller/logs/screen-g-api.txt | 15:50 |
| dansmith | even though 19292 is in the glance-api.conf and it claims to be reading glance-api.conf | 15:50 |
| abhishekk | let me see, I guess I can explain this | 15:51 |
| abhishekk | give me 5 minutes | 15:51 |
| dansmith | thanks | 15:51 |
| abhishekk | so its a mess of how config options are read in glance | 15:53 |
| abhishekk | if you see api and registry conf both have bind_port option defined under default section | 15:53 |
| abhishekk | and we parse entire /etc/glance directory at service start which overrides the config option value to 19191 from registry for g-api | 15:54 |
| dansmith | yup | 15:55 |
| dansmith | ...okay | 15:55 |
| abhishekk | and this is the only reason why its listening on 19191 | 15:56 |
| dansmith | okay are you saying I could just point apache at 19191 and it would work for api? the ports don't need to be separate? | 15:57 |
| *** belmoreira has quit IRC | 15:57 | |
| abhishekk | you could try that as registry is not used any more | 15:59 |
| dansmith | okay I'm confused | 15:59 |
| abhishekk | wait | 15:59 |
| dansmith | I could also just set the registry port to the same value in devstack localrc | 15:59 |
| abhishekk | the public_endpoint in glance is showing https://10.0.1.13:9292 | 15:59 |
| abhishekk | not sure that will help | 16:01 |
| abhishekk | is it possible to ignore registry at all? | 16:01 |
| dansmith | I dunno what that means.. we're not ever really starting g-reg that I know of.. sounds like the glance devstack module needs some serious work | 16:02 |
| abhishekk | +1 | 16:02 |
| *** gyee has joined #openstack-glance | 16:03 | |
| dansmith | lemme just try setting the bind_port in g-reg.conf to be the same thing and see if that at least unblocks me | 16:03 |
| abhishekk | ack | 16:03 |
| dansmith | but sounds like the g-reg part of the devstack module needs to be gutted so we're not even generating that config | 16:03 |
| abhishekk | I have submitted patch for removal of registry but its stuck in controversy | 16:04 |
| dansmith | removal of it in glance? | 16:04 |
| abhishekk | from devstack | 16:04 |
| dansmith | oh, okay | 16:05 |
| abhishekk | https://review.opendev.org/#/c/708062/ | 16:05 |
| abhishekk | Because we cannot remove it from glance unless its removed from devstack, otherwise everyone will be (no comments) | 16:06 |
| dansmith | right | 16:06 |
| abhishekk | dansmith, I will be back in short after dinner | 16:06 |
| dansmith | ack thanks | 16:07 |
| *** udesale_ has quit IRC | 17:19 | |
| abhishekk | dansmith, any progress? | 17:23 |
| dansmith | job just finished, will check in a sec | 17:23 |
| abhishekk | ack | 17:23 |
| dansmith | abhishekk: my job didn't work, but it finished devstack and has a bunch of glance accesses in the log | 17:28 |
| dansmith | so that's progress | 17:28 |
| abhishekk | dansmith, I am looking at glance-api.conf | 17:29 |
| dansmith | it's still not converting the image on upload though | 17:29 |
| abhishekk | wants to point out some issues there | 17:29 |
| abhishekk | enabled_backends = cheap:file, robust:rbd | 17:29 |
| dansmith | that works for me locally, btw | 17:30 |
| abhishekk | there should not be space between two options | 17:30 |
| dansmith | okay | 17:30 |
| abhishekk | Ohh, earlier it not used to work | 17:30 |
| dansmith | seemed to work fine for me | 17:30 |
| abhishekk | as DictConfigOpt does not support it | 17:30 |
| abhishekk | not tested recently though | 17:31 |
| abhishekk | now look at last section where rbd config is defined | 17:31 |
| abhishekk | [robust] config section | 17:31 |
| abhishekk | nope its good | 17:31 |
| abhishekk | sorry | 17:31 |
| dansmith | so, devstack probably never does a show on the image later to see if the conversion worked, | 17:32 |
| dansmith | but later nova complained that the image is not raw | 17:32 |
| dansmith | nova is configured to require raw images because of rbd | 17:32 |
| abhishekk | where can I find devstack logs? | 17:32 |
| dansmith | https://zuul.opendev.org/t/openstack/build/22908a60c2de4fe6ba2bcdbef2191d75/logs | 17:33 |
| abhishekk | found it | 17:33 |
| abhishekk | is it because devstack uses openstack image create command? | 17:34 |
| abhishekk | I guess that's the reason | 17:34 |
| dansmith | oh jeez | 17:35 |
| dansmith | instead of what? | 17:35 |
| abhishekk | glance image-create | 17:35 |
| sean-k-mooney | the clance client | 17:35 |
| dansmith | I did this manually on my local test, didn't use the conversion but was trying to make this automatic | 17:35 |
| dansmith | haha | 17:35 |
| dansmith | yet another thing osc doesn't do? | 17:35 |
| abhishekk | for conversion purpose we need to use glance image-create-via-import command | 17:35 |
| abhishekk | osc is not uptodate with glance | 17:36 |
| sean-k-mooney | abhishekk: well nova is useing the api | 17:36 |
| sean-k-mooney | so i dont think that is related | 17:37 |
| abhishekk | nova is not using import workflow | 17:37 |
| abhishekk | its using traditional image upload api | 17:37 |
| sean-k-mooney | for snapshots | 17:37 |
| dansmith | right, this is the original image upload in devstack | 17:38 |
| abhishekk | yep | 17:38 |
| *** rosmaita has left #openstack-glance | 17:38 | |
| dansmith | unrelated to nova, so I'm sure abhishekk is right | 17:38 |
| sean-k-mooney | the orginal image uplad is goingt too the file backend | 17:38 |
| dansmith | abhishekk: so can we change devstack to use glance client for the image upload all the time | 17:38 |
| dansmith | sean-k-mooney: yes it is, it has to | 17:38 |
| abhishekk | yes, lots of work | 17:38 |
| sean-k-mooney | so we dont care about the format right | 17:38 |
| dansmith | we do | 17:38 |
| dansmith | it needs to be raw | 17:39 |
| dansmith | so that when we copy it to rbd it's raw, which is necessary | 17:39 |
| sean-k-mooney | we are expecting the image convertion to happen as part of the copy-form-image import later | 17:39 |
| dansmith | no | 17:39 |
| sean-k-mooney | *i was expecting ^ | 17:39 |
| dansmith | no, an image can't be in two different formats in two different stores, AFAIK | 17:39 |
| dansmith | those would have to be separate images I think | 17:39 |
| sean-k-mooney | oh ok thats much more limiting | 17:40 |
| dansmith | I dunno about that.. the multistore thing is just saying "I have copies of this file in [$places]" | 17:40 |
| abhishekk | dansmith, yes those should be separate images | 17:40 |
| dansmith | right | 17:40 |
| sean-k-mooney | i guess that make sense | 17:40 |
| sean-k-mooney | the format is a property of the image not where its stored | 17:41 |
| *** rosmaita has joined #openstack-glance | 17:41 | |
| sean-k-mooney | so ya i guess the conversion would have to happen on inital upload | 17:41 |
| abhishekk | yep | 17:41 |
| abhishekk | dansmith, I can spend some time to check whether glanceclient can be used instead of osc in devstack | 17:42 |
| dansmith | abhishekk: okay thanks, that'd be much appreciated | 17:42 |
| sean-k-mooney | abhishekk: it could be used but long term we should actully update osc | 17:42 |
| dansmith | abhishekk: I guess the other option is to make the ceph plugin upload a second raw copy of it and configure tempest to do that | 17:43 |
| abhishekk | dansmith, +1 | 17:43 |
| sean-k-mooney | you dont even need to have the ceph plugin do that | 17:43 |
| sean-k-mooney | you can specify a different image url to use | 17:43 |
| abhishekk | I also need to work on changing ceph plugin so that it can be used in multistore configuration | 17:44 |
| sean-k-mooney | or addtional image depending on what you want | 17:44 |
| dansmith | sean-k-mooney: there is no cirros raw image to fetch that I know of | 17:44 |
| abhishekk | what is the priority, I am having long weekend this time starting from tomorrow | 17:44 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/master/stackrc#L655-L717 | 17:44 |
| abhishekk | if its urgent, I will put some time tomorow | 17:44 |
| sean-k-mooney | dansmith: ah ok | 17:44 |
| dansmith | abhishekk: me too of course, no not that urgent :) | 17:45 |
| dansmith | abhishekk: working in the next week or so would be great | 17:45 |
| abhishekk | dansmith, cool, will keep you posted | 17:45 |
| dansmith | abhishekk: thanks a lot for your continued help and patience :) | 17:45 |
| abhishekk | dansmith, no problem :D | 17:46 |
| sean-k-mooney | dansmith: isnt http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img a raw image | 17:46 |
| dansmith | no | 17:46 |
| dansmith | it's a qcow | 17:46 |
| dansmith | that's what we use currently | 17:46 |
| sean-k-mooney | oh ok .img is ment to be used for raw image but ok we could have devstack convert it | 17:47 |
| sean-k-mooney | or just use glance client i guess | 17:47 |
| abhishekk | ack | 17:47 |
| abhishekk | dansmith, just one question out of curiosity, where did you changed/override the port setting? | 17:49 |
| abhishekk | saw it | 17:49 |
| dansmith | abhishekk: https://review.opendev.org/#/c/734184/11/.zuul.yaml | 17:50 |
| abhishekk | I was still looking at PS 10 :D | 17:50 |
| dansmith | sean-k-mooney: right, could convert it (which is what I did in my local test), I was just hoping to have glance magically do it for me so we don't end up with more changes required | 17:50 |
| sean-k-mooney | dansmith: we could convert it with a preplay book but this is where the current image uplad is happneing https://github.com/openstack/devstack/blob/e0f051ced7c7cd760b6d502e8d95027f5029d6e8/functions#L396 | 17:51 |
| dansmith | yeah | 17:52 |
| dansmith | sean-k-mooney: is that something I can control with just our zuul config? | 17:52 |
| sean-k-mooney | a pre playbook yes | 17:52 |
| sean-k-mooney | you have to do a trick with creating a local.sh | 17:53 |
| sean-k-mooney | but i can show you an example | 17:53 |
| sean-k-mooney | so here is an example https://review.opendev.org/#/c/681807/3/playbooks/nfv/pinning.yaml | 17:54 |
| sean-k-mooney | it create a local.sh that create flavor for numa testing | 17:55 |
| sean-k-mooney | that runs right at the end of devstack | 17:55 |
| sean-k-mooney | and this is how you use it in the job https://review.opendev.org/#/c/681807/3/.zuul.yaml@413 | 17:55 |
| sean-k-mooney | so the pre playbook runs before devstack starts and creates the local.sh which devstack then run right before if finishes and tempest starts | 17:56 |
| sean-k-mooney | you can then just change the image ref to use in the tempest config https://review.opendev.org/#/c/681807/3/.zuul.yaml@459 | 17:57 |
| sean-k-mooney | like i was changing the flavor | 17:57 |
| dansmith | ack yeah, okay.. I think probably better in the long run to make the ceph plugin just do this, but good to know that I have that as an option | 17:57 |
| sean-k-mooney | the local.sh is sourced by devstack so all variables and function are avaiable to it | 17:58 |
| dansmith | if for some reason that becomes contentious | 17:58 |
| sean-k-mooney | ya or just add a flag to devstack to have it convet to a given format before upload | 17:58 |
| sean-k-mooney | devstack is already using qemu-img info | 17:58 |
| sean-k-mooney | so calling convert is not really a big change | 17:59 |
| dansmith | yeah | 17:59 |
| abhishekk | good to know | 17:59 |
| sean-k-mooney | this is how i hack together jobs quirckly when i neeed to tweak thing that are a pain in the ass to set up otherwise. long term there are better options but is nice to use locally too | 18:00 |
| sean-k-mooney | e.g. the local.sh | 18:00 |
| dansmith | yeah | 18:01 |
| abhishekk | signing out for the day | 18:09 |
| dansmith | o/ | 18:18 |
| abhishekk | o/ | 18:23 |
| *** sean-k-mooney has quit IRC | 18:33 | |
| *** m75abrams has quit IRC | 20:20 | |
| *** gyee has quit IRC | 20:36 | |
| *** rcernin has joined #openstack-glance | 21:48 | |
| *** rcernin has quit IRC | 21:48 | |
| *** rcernin has joined #openstack-glance | 21:49 | |
| *** threestrands has joined #openstack-glance | 22:20 | |
| *** rosmaita has left #openstack-glance | 22:40 | |
| *** tkajinam has joined #openstack-glance | 22:46 | |
| *** rcernin has quit IRC | 23:48 | |
| *** rcernin has joined #openstack-glance | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!