Monday, 2024-09-09

*** bauzas_ is now known as bauzas00:08
*** bauzas_ is now known as bauzas00:21
*** bauzas_ is now known as bauzas01:01
*** bauzas_ is now known as bauzas01:26
*** bauzas_ is now known as bauzas01:34
*** __ministry is now known as Guest293601:43
*** bauzas_ is now known as bauzas02:24
*** bauzas_ is now known as bauzas02:40
*** bauzas_ is now known as bauzas02:48
*** bauzas_ is now known as bauzas02:56
*** bauzas_ is now known as bauzas04:05
*** __ministry is now known as Guest294404:14
*** bauzas_ is now known as bauzas04:26
*** bauzas_ is now known as bauzas04:58
*** bauzas_ is now known as bauzas06:20
bauzasmorning07:17
*** noonedeadpunk_ is now known as noonedeadpunk07:29
opendevreviewzhou zhong proposed openstack/nova master: nova-manage: modify image properties in request_spec  https://review.opendev.org/c/openstack/nova/+/92431908:37
opendevreviewzhou zhong proposed openstack/nova master: nova-manage: modify image properties in request_spec  https://review.opendev.org/c/openstack/nova/+/92431908:39
opendevreviewzhou zhong proposed openstack/nova master: nova-manage: modify image properties in request_spec  https://review.opendev.org/c/openstack/nova/+/92431908:42
opendevreviewBalazs Gibizer proposed openstack/os-traits master: Add support for hw_vif_model igb image property  https://review.opendev.org/c/openstack/os-traits/+/92858209:12
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92858309:14
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858409:14
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858409:15
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92858509:15
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92845609:16
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858409:16
*** bauzas- is now known as bauzas09:54
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92845610:05
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858410:05
opendevreviewBalazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity  https://review.opendev.org/c/openstack/nova/+/92859010:05
opendevreviewBalazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity  https://review.opendev.org/c/openstack/nova/+/92859010:18
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92845610:18
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858410:18
opendevreviewBalazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity  https://review.opendev.org/c/openstack/nova/+/92859011:36
opendevreviewBalazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property  https://review.opendev.org/c/openstack/nova/+/92845611:36
opendevreviewBalazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb  https://review.opendev.org/c/openstack/nova/+/92858411:36
ratailor_Hi All, can I get reviews on my open patches, https://review.opendev.org/q/owner:ratailor@redhat.com+status:open12:46
ratailor_some of those are open since long. 12:46
sean-k-mooneyhave you put them on the review etherpad12:48
sean-k-mooneyor pingged us about them recently12:48
sean-k-mooneythey should all be listed here https://etherpad.opendev.org/p/nova-dalmatian-status#L6012:49
sean-k-mooneywe will be creating an epoxy version of that soon12:49
sean-k-mooneyso maybe we should add them there12:50
sean-k-mooneybauzas: is there a nova-epoxy-status etherpad yet12:50
sean-k-mooneyor nova-2025.1-status12:50
sean-k-mooneyi would prefer 2025.1 personally like ubuntu i prefer the number over the name but either is fine12:51
ratailor_sean-k-mooney, no yet, I will do it asap. Thanks!12:59
bauzassean-k-mooney: not yet, I'm literrally catching up with all pre-RC1 PTL tasks13:01
bauzaswhich include rpcapin pin patch and other fancy prelude writing13:01
sean-k-mooneyno worries13:01
sean-k-mooneythe only thing you really need to decied is which one to use and then we can just create it13:02
bauzasI'm rushing over deadlines, since RC1 is planned on Thur13:02
sean-k-mooneyill create it for you13:02
sean-k-mooneywhich would you prefer name or number13:02
bauzaswell, creating an etherpad isn't really a problem13:02
sean-k-mooneyright its not so i can do it quickly13:02
bauzasI just discussed with the Foundation and they were amazed we were having some tracking process13:02
sean-k-mooneywell we have alwasy had one13:03
bauzasbut given we don't document it actually, this isn't really useful13:03
sean-k-mooneywell we document that we shoudl use review priorty in gerrit13:03
sean-k-mooneywhich i still perfer13:03
sean-k-mooneybut since you wanted to use ether pad we document that currently in the channel topic13:03
sean-k-mooneybut not in our contubutors guide13:04
ratailor_sean-k-mooney, also could you please move this to Triaged, its still showing in new. https://bugs.launchpad.net/nova/+bug/205892813:05
sean-k-mooneyok although you stared on it a while ago so should i put it in inprogress13:06
ratailor_sean-k-mooney, yes, sure. Thanks!13:06
sean-k-mooneynote the traige team is open so you can join it and do this yourself too13:06
ratailor_sean-k-mooney, ack. sure. 13:08
sean-k-mooneyi set it to inprogress and low beacuse of how long this has been broken13:09
bauzassean-k-mooney: sorry was afk for family matters13:19
bauzassean-k-mooney: fwiw, after that discussion with the foundation folks, I now have an AI on me, which is to update our contrib docs (and how to reach them)13:20
bauzaslike our wikipage is the first POC for nova but it's fucking old and stale13:20
bauzasso I'll do all of that stuff shortly after RC113:21
sean-k-mooneybauzas: ok if we really are not going to back to using gerrit for this13:21
sean-k-mooneythen ill update our gerrit config to remvoe the review priorty flag13:21
bauzasabout the etherpad, we usually use codenames for etherpads, but I'm not opposed to use release numbers if people agree13:21
sean-k-mooneybauzas: we proposed stoping using in the last two ptgs13:21
sean-k-mooneyboth for the ptg and this status one13:21
sean-k-mooneyalso for the launchpad series13:22
sean-k-mooneywe made the change in lauchpad form caracal13:22
sean-k-mooneywe didnt make the change for the ptg or status page however13:23
sean-k-mooneyin the retos both dan and i have prevsiouly expressed our desire to stop using names13:23
sean-k-mooneyfor all our nova trackers/processes13:23
sean-k-mooneybut its not a hill i want to die on13:24
sean-k-mooneymy personal recolection (which might be wrong) is we agreed to use the number in caracal but that change was not implemnted13:26
sean-k-mooneywe started disucssing htis as far back as antelope13:27
bauzasthe written resolution on release names says that numbers are the official way to describe a release13:30
sean-k-mooneyyep13:30
bauzashence why we use them in both launchpad, stable branches and specs repo13:30
bauzasthat said, the resolution says we can use codenames for internal use13:31
bauzaswhich is why we were naming the etherpad that way13:31
sean-k-mooneyright but we as a nova team discussed this and at lease dan and i advocated for using the number for lanchpad as well13:31
bauzasnow that I realize that this tracking etherpad is useful for new contributors, I think we can start using release numbers13:31
sean-k-mooneyand i tought we agreed we shoudl do that13:31
bauzaslaunchpad *is* using numbers13:32
sean-k-mooneybut the etherpad is not13:32
bauzasfrom 2023.1 caracal at least :)13:32
bauzascorrect, that's what I just said,13:32
sean-k-mooneyi was gong to bring this up as a low light in the rethro athat we had not chagned to the numeber for the epther pad for a second release in a row13:32
bauzasI'm just creating the new etherpad, that's quick13:33
bauzashttps://etherpad.opendev.org/p/nova-2025.1-status13:35
bauzassean-k-mooney: ralonsoh ^13:35
bauzasI'm also creating a second etherpad for rc tracking13:35
bauzas(which will use the codename format since it's for dalmatian)13:35
sean-k-mooneyi think you ment ratailor_ https://etherpad.opendev.org/p/nova-2025.1-status13:36
bauzasmy bad, typed too fast and autocompletion hit the road13:36
* bauzas goes back into datamining mode for RC prep13:37
opendevreviewMerged openstack/nova stable/2023.1: hardware: Correct log  https://review.opendev.org/c/openstack/nova/+/92848513:39
*** ykarel_ is now known as ykarel14:02
ratailor_sean-k-mooney, bauzas Thanks!14:08
ralonsohbauzas, thanks for th elink14:23
opendevreviewRajesh Tailor proposed openstack/nova master: Fix KeyError on assisted snapshot call  https://review.opendev.org/c/openstack/nova/+/90078314:31
opendevreviewSylvain Bauza proposed openstack/nova master: doc: mark the maximum microversion for 2024.2 Dalmatian  https://review.opendev.org/c/openstack/nova/+/92866016:07
opendevreviewSylvain Bauza proposed openstack/nova master: Update compute rpc alias for dalmatian  https://review.opendev.org/c/openstack/nova/+/92866116:10
opendevreviewSylvain Bauza proposed openstack/nova master: Add service version for Dalmatian  https://review.opendev.org/c/openstack/nova/+/92866216:17
noonedeadpunkhey folks! is there any way to suppli a custom URI to the VNC console? As I'm obviously getting `404 File not found` when tried to define `https://domain.com/novnc/vnc_lite.html`16:19
noonedeadpunkI tried to find inspiration in devstack, but seems vnc there is just port-oriented?16:20
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/092ca27659d5466aa3c19120b58b5031/log/controller/logs/etc/nova/nova-cpu_conf.txt#15516:20
noonedeadpunkso basically console is not under /compute while nova overall is16:22
opendevreviewSylvain Bauza proposed openstack/placement master: Update 2024.2 reqs to support os-traits 3.1.0 as min version We're now close to RC1, we need to ensure that 2024.2 Placement will support the new traits.  https://review.opendev.org/c/openstack/placement/+/92866316:28
opendevreviewSylvain Bauza proposed openstack/placement master: Update 2024.2 reqs to support os-traits 3.1.0 as min version  https://review.opendev.org/c/openstack/placement/+/92866316:28
opendevreviewSylvain Bauza proposed openstack/nova-specs master: Move Dalmatian implemented specs  https://review.opendev.org/c/openstack/nova-specs/+/92866616:38
opendevreviewribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call  https://review.opendev.org/c/openstack/nova/+/90886416:57
opendevreviewribaudr proposed openstack/nova master: Amend DB model add a unique constraint.  https://review.opendev.org/c/openstack/nova/+/91251816:57
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119416:57
opendevreviewribaudr proposed openstack/nova master: Use client token when talking to manila  https://review.opendev.org/c/openstack/nova/+/92527716:57
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309016:57
opendevreviewribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process  https://review.opendev.org/c/openstack/nova/+/88007516:57
opendevreviewribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion  https://review.opendev.org/c/openstack/nova/+/88147216:57
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050016:57
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/85482316:57
opendevreviewribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/85482416:57
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/86028416:57
opendevreviewribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/86028516:57
opendevreviewribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares  https://review.opendev.org/c/openstack/nova/+/86028616:57
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares  https://review.opendev.org/c/openstack/nova/+/86028716:57
opendevreviewribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol  https://review.opendev.org/c/openstack/nova/+/88386216:57
opendevreviewribaudr proposed openstack/nova master: Check shares support (compute manager)  https://review.opendev.org/c/openstack/nova/+/88575116:57
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API)  https://review.opendev.org/c/openstack/nova/+/83683016:57
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/88575316:57
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050116:57
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102816:57
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102916:57
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach_error notification  https://review.opendev.org/c/openstack/nova/+/86028216:57
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach_error notification  https://review.opendev.org/c/openstack/nova/+/86028316:57
opendevreviewribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share  https://review.opendev.org/c/openstack/nova/+/92708816:57
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208616:57
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208716:57
opendevreviewribaudr proposed openstack/nova master: Docs about Manila shares API usage  https://review.opendev.org/c/openstack/nova/+/87164216:57
noonedeadpunkit feels like the usecase of custom URI for consoleproxy is just not supported16:57
noonedeadpunkas I'd guess I'd expect to see some api-paste for nova-novncproxy but obviously there's none17:02
sean-k-mooneynoonedeadpunk: you can customise it to a degree 17:25
sean-k-mooneywhat are you tryign to do17:25
noonedeadpunktrying to have smth like `https://domain.com/novnc/vnc_lite.html`17:26
noonedeadpunkinstead of `https://domain.com:6080/vnc_lite.html`17:26
noonedeadpunkand feels like the way forward would be to place novnc install folder somewhere under DEFAULT.web17:27
noonedeadpunkie make a subtree or smth17:27
sean-k-mooneyya so you can do that17:27
sean-k-mooneywe have conifg option for tha tlet me find it17:28
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#vnc.novncproxy_base_url17:28
sean-k-mooneywe should really upgrade our default to default to vnc_lite.html at some point17:28
noonedeadpunkyeah, so ^ is not helpful17:28
sean-k-mooneyhow so17:29
noonedeadpunkas I have that and novncproxy ends up with 40417:29
noonedeadpunkopenstack console url show returns expected URL though17:29
noonedeadpunkie https://domain.com/novnc/vnc_lite.html?path=%3Ftoken%3D880916e7-2e8a-4d0c-a814-9e16f663319417:30
sean-k-mooneyright so to use that you need to aslo modify your reverse proxy17:30
noonedeadpunkbut then it's just 404 from nova-novncproxy service itself17:30
noonedeadpunkwith File not found17:30
noonedeadpunkI'd guess I would need to do rewrite17:30
sean-k-mooneyyep that what i used to do with kolla ansible17:31
noonedeadpunkbut I'm using haproxy which can't really do rewrites17:31
sean-k-mooneyyou could set https://docs.openstack.org/nova/latest/configuration/config.html#vnc.novncproxy_port to 80 or 44317:31
noonedeadpunkit's probably easier when you have apache though17:31
sean-k-mooneyso i used trafix to do it17:31
sean-k-mooneyit was doing all my tlscert stuff with letencypt and doing the rewrite17:32
noonedeadpunkI don't think that port will anyhow influence URI after it?17:32
noonedeadpunkregardless on which port nova-novncproxy.service will listen17:32
noonedeadpunkas long as balancer passes traffic to it - it should work right?17:32
noonedeadpunkfor nova-api I can do rewrites with uWSGI17:33
noonedeadpunkor, using api-paste for eventlet17:33
sean-k-mooneyso if you want to have vnc in a sub folder17:33
sean-k-mooneyyou need to strip that in your revirse proxy17:33
sean-k-mooneyapi-paste is not really used for this17:33
noonedeadpunkoh, well, it kind of is>17:33
noonedeadpunkhttps://opendev.org/openstack/nova/src/branch/master/etc/nova/api-paste.ini#L20-L2917:34
sean-k-mooneykind of but we dont have midelware enabled by default for rewrites17:34
sean-k-mooneyits better to do that in the webserver/reverse_proxy17:34
noonedeadpunkah, for this - novnc - yeah, it does not17:34
noonedeadpunkso in the webserver - yeah, that would be fine. But reverse proxies not designed for rewrites, imo17:35
sean-k-mooney ya so the novnc proxy is useing websockify and the eventlet webserver17:35
sean-k-mooneynoonedeadpunk: tell that to kubernetes17:35
sean-k-mooneythat what ingress is17:35
noonedeadpunkI don't think k8s is good bfor lb...17:35
noonedeadpunkit's nginx17:35
noonedeadpunkwhich is technically not a reverse proxy17:36
sean-k-mooneyits the runciton its used for however17:36
noonedeadpunkit's a web server with hugely reduced feature list that acts as a reverse proxy17:36
sean-k-mooneyi would consier it to be both a reverse proxy and a webserver17:36
sean-k-mooneyyep but even haproxy can do this too17:36
noonedeadpunkcan it?17:36
sean-k-mooneyi belive so when used as a http loadbalbnce17:37
noonedeadpunkAs it can do maps, sure, but it's not really rewrites17:37
noonedeadpunkso it can pass /console to correct backend17:37
noonedeadpunkbut not alter uri. I guess17:37
noonedeadpunknot 100% sure17:37
sean-k-mooneyya. so that is how i would personaly do it. i generall prefer using subdomain insteda of subpaths17:38
noonedeadpunkI was actually hoping to see answer in devstack, but it's jsut using port :D17:38
noonedeadpunkoh, yes, sure, subdomain works nicely17:38
sean-k-mooneyim currently in the process of rebuildign my home lab17:38
noonedeadpunkjsut decided to practise on subpaths as that's what devstack does17:38
jrosserdoes any of this help https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/17:38
sean-k-mooneybut my home openstack did everything behind traefix exposing everything over port 44317:39
noonedeadpunkoh17:39
sean-k-mooney openshifts defautl ingress controller(route contoler) is haproxy and it does supprot this to at least some degree17:40
noonedeadpunkjrosser: I guess I tried to have a look at https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#set-the-url-path but failed to implement so dropped this idea a while ago17:42
noonedeadpunkbut maybe for consoles it's actually worth checking17:42
sean-k-mooney https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#replace-part-of-a-header-by-using-a-regular-expression17:44
sean-k-mooneyyou can proably use the regex replace value funtionality17:44
sean-k-mooneyin your case set-path17:45
sean-k-mooneyis proably simpler17:45
noonedeadpunkI've jsut tried to place novnc actually in subfolder, but then it's also failing on `wss://domain.com/?token=4741df38-cc54-4be6-b63f-1d7b7b1b336d`17:45
noonedeadpunkas somehow wss is not adjusted or trimms subpath /o\17:46
noonedeadpunkand that I'm even not sure how to catch 17:47
noonedeadpunkso eventually what creates wss should also pass a path17:57
noonedeadpunkor at least respect passed one17:57
sean-k-mooneywell you would need to also rewrite it 17:58
noonedeadpunkit's tough to rewrite just /?token given that / is serving horizon17:59
noonedeadpunkbut yeah17:59
noonedeadpunkI'd say there should be code in place respecting passed URIs when returning wss18:00
sean-k-mooneyunfortuetly i dont have a config to had for that. i tought i had that working in my previous home cloud. i.e. the one i had pre  202018:00
sean-k-mooneybut my current one is forwarding 6080 and everything else was using subdomains18:01
noonedeadpunkyeah, I managed to get it working with subdomain jsut out of the box18:02
noonedeadpunkbut then it's a bit tougher with let's encrypt as for wildcards I'd need dns-0118:04
sean-k-mooneyyep thats what i use now although before i didnt use wildcards i added a dns entry for each serice18:15
sean-k-mooneyi use cloudfail as my dns provider mainly for there restapi adn the fact it supproted in a lot of things18:15
sean-k-mooney*cloudflare18:16
noonedeadpunkwell. these are quite long-shot workarounds for smth which should be just possible to do quite simply. And I have pretty much everything working with subpaths except console...18:18
sean-k-mooneyhonestly none of this is inscope of nova18:20
sean-k-mooneynoonedeadpunk: a webseocket is just a upgradded http connection18:20
sean-k-mooneymy guess is haproxy does not offcially supprto them or apply the rewite to them18:20
noonedeadpunkyes, but it's https://opendev.org/openstack/nova/src/branch/master/nova/console/websocketproxy.py which replies with wss URL I assume?18:21
noonedeadpunkas when I open novnc under specific path, and it fully loads (I reached this point), now I assume nova provides wss url that is not under same path at all18:22
noonedeadpunkSo I'd argue about scope here18:22
sean-k-mooneywe ues a websocket between the proxy and the html5 applciation that part of novnc18:22
noonedeadpunkyeah, but it's created with no respect to where html5 is running I assume?18:23
noonedeadpunkas it always thinks it runs in `/`18:24
sean-k-mooneythe websocket is created regardless yes18:24
sean-k-mooneybut we do not supprot subpaths18:24
sean-k-mooneywe only supprot runing it on a port18:24
sean-k-mooneyanythign beyond that is out so scope18:24
noonedeadpunkso regardless of any rewrites, except trying to catch some GETs under `/` which is really weird - it won't work, yes18:25
sean-k-mooneyi belive it can be made work if you rewirte teh websocet too18:26
sean-k-mooneyhttps://www.haproxy.com/documentation/haproxy-configuration-tutorials/load-balancing/websocket/#configure-websockets ha porxy has some supprot for this18:26
noonedeadpunkbut then what's the whole point of devstack to be serving services under subpaths....18:26
sean-k-mooneyto avoid using domain18:26
noonedeadpunkyeah, but jsut use ports as used to - would be way more simple to configure...18:27
sean-k-mooneythat it we wanted to test that we can run all the rest apis over https on port 443418:27
sean-k-mooney*44318:27
sean-k-mooneythere is nothing stoping you using ports18:27
sean-k-mooneybtu we did have a lot of operator want to run everything on 44318:28
noonedeadpunk*all except consoles18:28
sean-k-mooneyyou can run the console on 44318:28
noonedeadpunk(that is not tested in devstack)18:28
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/092ca27659d5466aa3c19120b58b5031/log/controller/logs/etc/nova/nova-cpu_conf.txt#152-15818:28
sean-k-mooneysure but devstack does not test everything18:28
noonedeadpunkI can recall some tempest tests for consoles?18:28
sean-k-mooneythere are some yes18:29
sean-k-mooneyits not that devstack could not test this18:29
sean-k-mooneyits that its not in scope to test every combination of how openstack is deployed18:29
noonedeadpunkyeah, fair18:30
noonedeadpunkthat I agree with18:30
noonedeadpunkI guess I just not fully in agreement that serving consoles under subpath should be out of scope of nova-novncproxy service18:30
sean-k-mooneyfor our new installer we deploy novnc on a subdomain18:31
sean-k-mooneyhttps://logserver.rdoproject.org/57/857/e3248dbf95dfb1878d852beef4475bfb68f4e811/github-check/nova-operator-tempest-multinode/616a00d/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/routes/nova-novncproxy-cell1-public.yaml18:31
noonedeadpunkyeah, subdomain works really nicely....18:32
noonedeadpunkok, gotcha18:32
sean-k-mooneyyep so again using subpath need a reversproxy that support both http and websockets18:32
sean-k-mooneyas in it need to be able to rewirte both properly18:32
sean-k-mooneyit might be posibel to do some other way18:33
noonedeadpunkthe problem here of reverseproxy, that then you can't serve anything on root18:33
noonedeadpunkso if you want to have smth like skyline/horizon on root - you can't do that jsut because of vnc console18:33
sean-k-mooneyright but normally you sever that on /dashboard18:33
sean-k-mooneyand have a redirect for root to it18:33
noonedeadpunkbut then redirecting is also fishy a bit, as you need to account for wss18:34
sean-k-mooneyyou shoudl be able to have the proxy on /vnc by the way and use / for horizon aslong as there isnt a collition18:34
noonedeadpunkwhich is just /?token=18:34
noonedeadpunkbut probably possible to come up with some regex18:35
sean-k-mooneyyou shoudl be able to have the websoceket be wss://<domain>/novnc?token=...18:35
noonedeadpunkso I just got `https://domain.com/console/vnc_lite.html` loading but still wss was just  wss://<domain>/?token=18:36
noonedeadpunkso it ended up with "something went wrong"18:37
sean-k-mooneyright but the websocket is not beeing writhe properly18:37
sean-k-mooneyyou said the url looked correct form nova right18:38
sean-k-mooneydo you have an exmaple of the consol url returned18:38
noonedeadpunkyes, but what does return wss URL?18:38
sean-k-mooneyat least for the serial console oepnstack server console show returns it18:39
noonedeadpunkopenstack console url show returns `https://domain.com/console/vnc_lite.html?path=%3Ftoken%3D76a1303f-c40c-4169-912d-19c05432dbc5` is what is returned18:39
sean-k-mooneybut i think we get https back by defualt for novnc18:39
noonedeadpunkit loads, vnc console renders18:39
sean-k-mooneyso with devstack the dash board load at "http://192.168.16.127/dashboard/auth/login/?next=/dashboard/"18:40
noonedeadpunkbut then html5 tries to connect to wss on `wss://domain.com/?token=76a1303f-c40c-4169-912d-19c05432dbc5`18:40
noonedeadpunkok, I gues I'm jsut confused where this wss path is coming from18:41
noonedeadpunkas indeed it's probably not from nova at all18:41
sean-k-mooneyits part of the html content provided to the javascript app18:41
sean-k-mooneyso in my case it returns  http://192.168.16.127:6080/vnc_lite.html?path=%3Ftoken%3Dd8a68169-dbd6-4333-8b4a-f57bc3beb53c by edfault18:43
sean-k-mooneyif i open that direclty that is what nova is serving and what ironic is embding in an iframe18:43
noonedeadpunkyup, true, and that's working as well (if clients don't have 6080 firewalled)18:43
noonedeadpunkbyt heir isp18:44
noonedeadpunkok, I'll try to do some rewrites indeed.18:44
noonedeadpunkand look one more time what does construct wss uri18:44
sean-k-mooneytehre are two path to how htis works. we use websoxkify to wrap th raw tcp connectoin to the vnc console create by qemu18:45
sean-k-mooneyand we use novnc ot provide the webpage that conenct to that and provde the vnc clinet18:45
noonedeadpunkas htmlk5 console does have `const path = readQueryVariable('path', 'websockify');`18:45
noonedeadpunkand it does `url += '/' + path;` - that's why I kind of complained that smth not passing html5 all details for it to perform18:46
sean-k-mooneyok but that novnc that doing that18:47
sean-k-mooneynot nova18:47
sean-k-mooneythere might eb some header you can set like x_forwared_for or similar18:47
sean-k-mooneyin my case the path query arge is empty18:48
noonedeadpunkI  think it's just because path being used to pass token18:48
sean-k-mooneywell technialy path=%3F18:48
noonedeadpunkah18:48
noonedeadpunktrue18:49
noonedeadpunkbut it's what nova replies then18:49
sean-k-mooneyyou may be able to add it to novncproxy_base_url18:49
sean-k-mooneyso instead of http://127.0.0.1:6080/vnc_auto.html18:50
sean-k-mooneyhttp://domain/subpath/vnc_auto.html?path=/subpath18:50
sean-k-mooneycan you try setting that in your broser and see if setting path=/subpath is enouch18:51
noonedeadpunkhm, I somehow don't think that will work?18:51
noonedeadpunkyeah, I'm trying now18:52
noonedeadpunkin terms won't work - regarding defining novncproxy_base_url that way18:52
sean-k-mooneywell its being used by vnc_lite.html18:52
sean-k-mooneywhic is part of novnc18:52
sean-k-mooneynova might not allow you pas qurey sting arge in baseurl18:52
sean-k-mooneybut it could be added by the proxy18:53
sean-k-mooney*reverse_proxy when rewriteing18:53
sean-k-mooneywe coudl maybe add a cofnig option for that if it works18:53
sean-k-mooneybased on https://github.com/novnc/noVNC/blob/master/vnc_lite.html#L134-L15318:54
sean-k-mooneyi dont see why it woudl not18:54
noonedeadpunkyeah, adding manually shiould work18:54
noonedeadpunkwhat probably wont - adjustin novncproxy_base_url as you said18:55
sean-k-mooneyya we likely dont have logic to parse the query arges and append them18:55
noonedeadpunkI don't think that nova gonna respect already defined path and wont'd add default path=%3F18:55
sean-k-mooneybut should be doable exterally18:55
noonedeadpunkyeah, this should be not hard to do indeed18:56
noonedeadpunk(I'd say would make sense for config option though)18:56
noonedeadpunkor indeed parse for the option18:56
sean-k-mooneythe other thing you said is there isp blockes 608019:00
sean-k-mooneywe do hwave supprot for using alternitive ports alredy19:00
sean-k-mooneyso that might be an option in that case 19:00
sean-k-mooneyit would be a pretty minor feature to add novncproxy_extra_query_args19:01
noonedeadpunkok, so `https://domain.com/console/vnc_lite.html?path=console%3Ftoken%3Dd7bf7a47-f081-4500-a961-05ddb7994e13`19:05
noonedeadpunkdoes work indeed19:05
noonedeadpunkso thanks a ton for your time19:07
noonedeadpunkI know I was annoying but quite happy now as you said - it should be trivial both to rewrite and implement a "native" way19:07
sean-k-mooneyyour bringing a pain point to our attention that not annoying19:08
sean-k-mooneyi still think subdomains are preferable19:08
sean-k-mooneybut if someone wants to add this minor feature im not agaisnt it19:08
sean-k-mooneyi just rpaobly wont go implemtnet it in my spare time19:08
sean-k-mooneynoonedeadpunk: you may have hit https://github.com/haproxy/haproxy/issues/829 by the way19:10
sean-k-mooneythere is a set-pathq to set items in teh query string now19:12
sean-k-mooneyi guess that migh now be https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#set-the-query-string19:14
sean-k-mooneyso i think you can jsut add "http-request set-query path=/subpath"19:15
sean-k-mooneynoonedeadpunk: im going to call it a day. if that works you could porpsoe a docs update if you were so inclidned or a devstack path19:16
sean-k-mooney*patch19:16
noonedeadpunk++ thanks for the help as usual!19:20
noonedeadpunkyeah, so here it goes: https://opendev.org/openstack/nova/src/branch/master/nova/objects/console_auth_token.py#L7819:30
samcat116Hi all, whats a good place to start debugging the GET /servers api being slow? Running `openstack server list --all-projects --no-name-lookup` on a stack with about 1500 instances takes over 48s to return, with the GET https://example.com:8774/v2.1/servers/detail?deleted=False&all_tenants=True call taking over 30s itself. CPU usage across our 3 controllers is around 50% and I'm running with 10 api_workers, so don't think its a raw19:49
samcat116horsepower issue19:49
*** bauzas_ is now known as bauzas21:15
*** bauzas_ is now known as bauzas21:56
sean-k-mooneysamcat116: the server detail list need to hit neutron for each server to lookup secuirty groups, but openstack client is doing more then jsut /v2.1/servers/detail?deleted=False&all_tenants=Tr22:10
sean-k-mooney                       | ue22:10
sean-k-mooneythat specific query however does require conenctin to all cell db and pulling a lot of info so its one of the more expensive api calls you can make to nova22:12
sean-k-mooneyits not as expensieve as simple tenant usage but it s more expensive then just about any other read only call22:13
samcat116Ok I guess I’ll look on the neutron side as we are pretty port and network heavy as well22:19
samcat116I just don’t know why it would be that slow22:19
sean-k-mooneyit might not be related to that but its on eof tthe peices of data we lookup form other services22:20
sean-k-mooneywe cache alotof the info reatled to other services but not all22:20
sean-k-mooneytehre are some profiling tools liek os-profiler22:20
sean-k-mooneybut im not famialr with who to properly use them 22:21
samcat116Ok. I do have memcache all setup so I would assume this could be heavily cached22:21
sean-k-mooneyno22:22
sean-k-mooneyi think we have almsot no caching for this22:22
sean-k-mooneywe use memcache very sparingly22:22
sean-k-mooneyits main use in nova is for the metadata api22:23
sean-k-mooneywe generally dont cache api responces22:24
sean-k-mooneythe other main use of memcafhe is for keytone auth22:25
samcat116Good to know22:27
samcat116How far off of an estimate could this be. Just trying to figure out if this is an order of magnitude slower than it should be or not22:30
sean-k-mooneythat kind of depend on other factors like how many cells you have22:30
sean-k-mooneyyou not the first person to ask about this 22:30
sean-k-mooneysamcat116: can you try making the same query to /server instead of /server/detials22:31
sean-k-mooneythe delta is /server just hits the api db and list all the server we know about without going to the cell dbs or calling other services22:31
sean-k-mooneythat should be very fast as it basicaly a select instance_uuid from istance_mappings22:32
sean-k-mooneyif /servers/ is fast and /servers/detail is slow then one thing to check is if any of the cells are slow to respond22:35
sean-k-mooneyinternally the api when repondign ot /servers/detail is calling into each cell db and geting the data form the instances table22:35
*** bauzas_ is now known as bauzas22:37
sean-k-mooneythe security group info i thin need a call to neutron for every port22:38
samcat116Ok I’ll try that. We only have one cell22:38
sean-k-mooneythat should simplfy thing then as the api is not making parralel requests22:39
sean-k-mooneyi think the security gorups are the only thing not cached in that responce. i.e. that is not just a direct db lookup22:41
sean-k-mooneythis has been know for a long time https://bugs.launchpad.net/nova/+bug/1923560 and a fix was prospoed here https://review.opendev.org/c/openstack/nova/+/78634822:42
sean-k-mooneyi was personaly on the side of removing the security groups form the instance obejct entirly 22:43
sean-k-mooneybtu if we dont then we shoudl cache them22:43
*** bauzas_ is now known as bauzas23:10

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!