| * stephenfin waves | 06:09 | |
| * gibi waves back | 07:21 | |
| bauzas | good morning Nova | 07:37 |
|---|---|---|
| bauzas | gibi: stephenfin: hola folks | 07:37 |
| * bauzas had a longer dogwalk this morning :p | 07:37 | |
| lyarwood | \o morning | 07:48 |
| *** rpittau|afk is now known as rpittau | 07:55 | |
| gibi | o. | 07:58 |
| gibi | o/ | 07:58 |
| * kashyap waves | 07:58 | |
| oklhost | sean-k-mooney: thanks, seems we got less log entries. | 08:56 |
| opendevreview | Thomas Goirand proposed openstack/nova master: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/809980 | 09:14 |
| zigo | bauzas: Good morning, there's a ooopsy there, no ? ^ | 09:15 |
| * bauzas looks | 09:15 | |
| bauzas | zigo: well, good question, I'm not sure we need it | 09:15 |
| bauzas | what kind of issue you have ? | 09:16 |
| zigo | bauzas: When I later on generate nova.conf with oslo-config-generator (with nova installed debian/tmp/usr/lib/python3/dist-package) I get a stack dump, with this folder not installed ... | 09:16 |
| sean-k-mooney | bauzas: without __init__.py its technially not a python module | 09:17 |
| bauzas | yup | 09:17 |
| bauzas | I know | 09:17 |
| bauzas | but I wonder if it's an issue | 09:17 |
| zigo | So basically, nova/db/api isn't getting installed when doing python3 setup.py install ... | 09:17 |
| bauzas | that changed a bit with py3 IIRC | 09:17 |
| * bauzas rereads https://docs.python.org/3/tutorial/modules.html#packages | 09:19 | |
| bauzas | but I wonder why we have this issue now | 09:19 |
| bauzas | and not before | 09:19 |
| zigo | bauzas: 2 things: first, that folder didn't exist in Wallaby, 2/ it's the symptoms of "it works in devstack" ... | 09:21 |
| zigo | (ie: setup.py install isn't being run...) | 09:21 |
| sean-k-mooney | bauzas: presumably because steph change thing with his alembic module | 09:21 |
| sean-k-mooney | work | 09:21 |
| sean-k-mooney | https://github.com/openstack/nova/commit/bf8b5fc7d05e0a66031a03e50e8f6bb76a921046#diff-6137249efc22cd455ac118bde1598a27beea93adcfcd49b36a2329318fc33c6e | 09:21 |
| zigo | Yeah, also this ... | 09:21 |
| sean-k-mooney | The two remaining modules, 'api_models' and 'api_migrations', are | 09:21 |
| sean-k-mooney | moved to the new 'nova.db.api' module. | 09:21 |
| sean-k-mooney | so bauzas stephenfin created that module in august | 09:21 |
| sean-k-mooney | but missed that file | 09:22 |
| sean-k-mooney | bauzas: so this is a xena release regressions | 09:22 |
| sean-k-mooney | zigo: devstack will be installing this more or less the same way as the distro | 09:23 |
| zigo | Ok. | 09:23 |
| zigo | Well, I don't know, but my patch needs to be merged ! :) | 09:23 |
| sean-k-mooney | we do not install with -e in devstack | 09:23 |
| sean-k-mooney | so it should be copying the files to the site-packages directory | 09:24 |
| sean-k-mooney | and using it form there | 09:24 |
| sean-k-mooney | zigo: did you file a bug | 09:24 |
| zigo | For a single "touch __init__.py" ?!? Seriously ? | 09:24 |
| zigo | :) | 09:24 |
| gibi | in the past (stable/wallaby) nova/db/api contained only the migration scripts that was always independently executed to the main nova services but now it contains code that nova services try to import. So I agree we need to fix this | 09:24 |
| sean-k-mooney | yes | 09:24 |
| sean-k-mooney | zigo: we will need to do an RC2 | 09:24 |
| zigo | Ok, filing the bug. | 09:25 |
| gibi | zigo: thank you for catching this | 09:26 |
| opendevreview | Thomas Goirand proposed openstack/nova master: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/809980 | 09:29 |
| zigo | There you go... | 09:29 |
| zigo | Bug filled, PR closing it. | 09:30 |
| zigo | I'm used to often do single char patches, this one will be ZERO chars ! :0 | 09:30 |
| sean-k-mooney | zigo: technially master is now yoga, which is why we need the bug for backporting and if we want to do an RC2 | 09:32 |
| sean-k-mooney | if you saw this last week before we created RC1 we proably could have just merged it | 09:32 |
| zigo | sean-k-mooney: Ok, thanks for letting me know. Should I wait until my first patch is merged before opening the backport PR ? | 09:33 |
| sean-k-mooney | am we should jsut be able to cherry pick this via the api so i think its safe to do it now | 09:34 |
| zigo | Oh also, is there anything to know from the user's point of view about the sqla-migrate -> alembic switch? | 09:34 |
| zigo | Or is it fully transparent? | 09:34 |
| sean-k-mooney | i dont expect this to take long to review given its size :) | 09:34 |
| sean-k-mooney | zigo: well other then the packageing impact no | 09:34 |
| zigo | Ok, cheers. | 09:35 |
| sean-k-mooney | i.e. the nova-mange command is still the same but you obvioulsy need alembic installed | 09:35 |
| sean-k-mooney | zigo: ok that has the flags and marked it as critical since it blocks packaging https://bugs.launchpad.net/nova/+bug/1944111 | 09:38 |
| zigo | The stable/xena branch is missing a defaultbranch=stable/rocky in the .gitreview file no ? | 09:41 |
| gibi | https://review.opendev.org/c/openstack/nova/+/809759 | 09:42 |
| gibi | the setup of stable/xena is not fully done yet as RC1 and the branch was cut last Friday | 09:42 |
| zigo | Ok, so I guess I must wait for that one to merge ... :/ | 09:43 |
| gibi | lyarwood, bauzas, elodilles: you you look at https://review.opendev.org/q/topic:create-xena+project:openstack/nova ? | 09:45 |
| gibi | s/you/could/ | 09:46 |
| lyarwood | Yup happy to | 09:47 |
| * lyarwood clicks | 09:47 | |
| bauzas | gibi: +Wd | 09:50 |
| bauzas | fwiw, also +wd https://review.opendev.org/c/openstack/nova/+/809761/1 and the above one | 09:51 |
| bauzas | so we will have the xena release notes | 09:52 |
| bauzas | we miss a second core on https://review.opendev.org/c/openstack/nova/+/809762 | 09:52 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Fix the wrong exception used to retry detach API calls https://review.opendev.org/c/openstack/nova/+/809934 | 09:52 |
| gibi | bauzas: done | 09:53 |
| bauzas | https://review.opendev.org/q/project:openstack/nova+owner:infra-root%2540openstack.org+is:open shows me all the xena paperwork for our jobs and reno are done | 09:53 |
| bauzas | gibi: thanks | 09:53 |
| bauzas | zigo: sorry was taxidriving my daughter from school | 09:53 |
| * bauzas now has to cook for her :) | 09:54 | |
| bauzas | zigo: +Wd your change | 09:55 |
| bauzas | zigo: please provide a backport change for stable/xena too | 09:56 |
| bauzas | so we will create a RC2 | 09:56 |
| * bauzas goes to the kitchen | 09:58 | |
| gibi | bauzas: after you are back placement also needs care after RC1 https://review.opendev.org/c/openstack/placement/+/809366 | 10:05 |
| gibi | bauzas: I will look into the lower constraints failre in placemenet stable/xena setup patches, probably that impacts placement master too | 10:06 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: [DNM]: Trigger lower-constaints job https://review.opendev.org/c/openstack/placement/+/809994 | 10:07 |
| gibi | yepp it seems master lower-constraints also times out in placement too ^^ | 10:46 |
| gibi | will fix it based on how neutron fixed it | 10:46 |
| gibi | lyarwood, stephenfin: is it an RC critical fix https://review.opendev.org/c/openstack/nova/+/809934 ? | 11:19 |
| stephenfin | Ah, whoops, probably not (though I'll defer to lyarwood to be sure). I was thinking all bugfixes were fair play right now since we'd branched already /o\ | 11:21 |
| stephenfin | gibi: feel free to pull it back out | 11:21 |
| bauzas | gibi: ack, taxying back my kid but I'll be around in 15 mins | 11:21 |
| lyarwood | I was under the same impression | 11:21 |
| lyarwood | gibi: isn't master open for Yoga now? | 11:22 |
| gibi | I think until the final RC we need to keep master close to stable/xena for any last minute backport | 11:22 |
| lyarwood | that's fair, yeah it isn't critical so feel free to yank it out if you can | 11:23 |
| lyarwood | and FWIW I'm not a huge fan of policy we can't enforce in the tooling like this | 11:23 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix the wrong exception used to retry detach API calls https://review.opendev.org/c/openstack/nova/+/809934 | 11:23 |
| gibi | yanked | 11:23 |
| lyarwood | thanks | 11:23 |
| bauzas | gibi: all placement Xena changes are now +Wd https://review.opendev.org/q/project:openstack/placement+owner:infra-root%2540openstack.org+is:open | 12:07 |
| gibi | bauzas: thanks but lower constraints fix will be still be needed | 12:07 |
| gibi | as the job will time out | 12:07 |
| gibi | I'm working on it | 12:08 |
| bauzas | gibi: ack thanks | 12:08 |
| bauzas | gibi: what is the change ? | 12:08 |
| gibi | lower constraint bump on master first | 12:08 |
| gibi | as it effects master | 12:08 |
| gibi | then we can lament on either we bump lower on stable/xena too, or look into somehow pinning setuptools version on stable branches | 12:09 |
| gibi | the thing is that we dont pin setuptools coming from virtualenv package on stable so we use basically the latest on every stable | 12:10 |
| gibi | an the latest setuptools removed support for some features old packages are depends on | 12:10 |
| bauzas | (14:08:46) gibi: lower constraint bump on master first | 12:10 |
| bauzas | can't see it | 12:10 |
| gibi | haven't proposed yet | 12:11 |
| bauzas | oh ok | 12:11 |
| gibi | 14:08 < gibi> I'm working on it | 12:11 |
| gibi | 14:08 < gibi> I'm working on it | 12:11 |
| gibi | 14:08 < gibi> I'm working on it | 12:11 |
| gibi | ups | 12:11 |
| gibi | sorry | 12:11 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001 | 12:15 |
| gibi | bauzas: now here it is | 12:15 |
| bauzas | gibi: heh sorry | 12:15 |
| gibi | I was lost couple of hour figuring out what happened | 12:15 |
| gibi | especially as I don't like the ide to bump a lower constraint on stable branch | 12:15 |
| gibi | s/ide/idea/ | 12:16 |
| bauzas | agreed | 12:16 |
| * bauzas goes taying the second kid now | 12:16 | |
| bauzas | taxying | 12:16 |
| * bauzas should be a Kardashian | 12:16 | |
| gibi | as like https://en.wikipedia.org/wiki/Cardassian ? :) | 12:18 |
| opendevreview | Merged openstack/nova master: Update master for stable/xena https://review.opendev.org/c/openstack/nova/+/809761 | 12:18 |
| opendevreview | Merged openstack/nova master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/nova/+/809762 | 12:19 |
| gibi | interestingly nova stable/xena is not effected | 12:20 |
| gibi | hm in nova we already have decorator >= 4.1.0 since https://review.opendev.org/c/openstack/nova/+/744506/2/lower-constraints.txt#20 | 12:22 |
| belmoreira | Hi, I need your help to understand if I'm missing something in the new vnc configuration | 12:52 |
| belmoreira | I can finally move nova to "train" release and I'm digging again into the vncproxy changes that were introduced by this time. The vncproxy does now the token validation from the cell DB. In stein I'm running with "workarounds/enable_consoleauth" | 12:52 |
| belmoreira | Currently, I have the same vncproxy for all cells. Means that the user gets the same console_url and I only need to open 1 port in the firewall. Also, haproxy configuration is trivial | 12:53 |
| belmoreira | With the new arch, the console_url needs to be redirected to the vncproxy of the cell for the token validation | 12:53 |
| belmoreira | This means that deployments with a large number of cells need to be creative in the way they expose the different console_urls (per cell) | 12:53 |
| belmoreira | maybe I'm missing something something here... | 12:53 |
| bauzas | belmoreira: sorry, I saw your pings but I don't know how to help you | 15:11 |
| belmoreira | Hi bauzas. Thanks, to me this new approach seems really heavy for deployments with a lot cells. For now I'm hacking something similar to [1] to not have a vncproxy per cell. | 15:18 |
| belmoreira | [1] https://github.com/openstack/nova/blob/0bd61915ee1d96ca339f342a190e395a39afbcf9/nova/api/openstack/compute/console_auth_tokens.py#L42 | 15:18 |
| belmoreira | maybe we can discuss this in the PTG | 15:18 |
| dansmith | it seems strange to me that someone with lots of cells would want to *not* shard that service across cells | 15:26 |
| dansmith | especially with geo-distributed cells | 15:27 |
| bauzas | agreed with dansmith | 15:31 |
| dansmith | (he dropped) | 15:31 |
| bauzas | hah, my internal meeting trampled this discussion | 15:32 |
| bauzas | -ETOOMANYMEETINHS | 15:32 |
| bauzas | :) | 15:32 |
| kashyap | bauzas: Drop the needless ones on the floor like hot potatoes. And embrace JOMO (joy of missing out) | 15:57 |
| bauzas | hah | 15:59 |
| bauzas | nah, I'm still digesting my Friday-late meeting :p | 15:59 |
| *** rpittau is now known as rpittau|afk | 16:00 | |
| sean-k-mooney | if the central site has direct connectivity to the edge site then you could just centralise the novnc proxy instnace | 16:06 |
| sean-k-mooney | but ya i would have assumed you would want them at each edge site too | 16:07 |
| sean-k-mooney | well each cell | 16:07 |
| sean-k-mooney | not nessisarly edge | 16:07 |
| sean-k-mooney | i was assuming you would run the novnc proxy on the same host as the cell conductor | 16:08 |
| dansmith | sean-k-mooney: I think the change he's referring to was one to make the service only look in one cell, which means you can centralize services, but not unify them (i.e. you need multiple ports and endpoints, regardless of where they are) | 16:09 |
| sean-k-mooney | ah i see | 16:13 |
| sean-k-mooney | unless we moved this to the api db, or allowed the proxy to connect to multiple cell dbs im not sure how we would adress that | 16:14 |
| dansmith | it used to I think, that's the point | 16:15 |
| dansmith | IIRC we removed that ability when we eliminated the consoleauth service | 16:15 |
| melwitt | it (nova-consoleauth) used to use memcache (one instance) to store token auths for the entire deployment | 16:15 |
| dansmith | I imagine that we could add back in just the api db lookup part (like metadata) but I think the expectation was was to make it shard, which I think is a better design, personally | 16:16 |
| dansmith | ah right | 16:16 |
| sean-k-mooney | so really without some way to pass the cell mapping info to a web server there is really no way to use a reverse proxy to expose it over one port/endpoint now | 16:18 |
| melwitt | but yeah, adding a console_auth_token_mappings table would be one way to make it so you only need one console proxy | 16:18 |
| sean-k-mooney | how i would proably try and set it up personaly is have it use a subdomain per cell in the url and have a reverse proxy bind to the single port | 16:18 |
| dansmith | oh, do we not get the instance id as well? | 16:18 |
| sean-k-mooney | then have it delegate to the correct backedn | 16:18 |
| dansmith | that would suck to have to add another mapping :/ | 16:18 |
| melwitt | no we don't, token only | 16:18 |
| dansmith | well then I'm pretty -1 on that plan | 16:19 |
| sean-k-mooney | we dont need to add anything in nova | 16:19 |
| dansmith | you could scatter/gather to find it | 16:19 |
| sean-k-mooney | to find the inial url | 16:19 |
| sean-k-mooney | ya you could | 16:19 |
| dansmith | sean-k-mooney: that's not necessary, because we given them the url from the proxy anyway | 16:19 |
| dansmith | sean-k-mooney: belmiro just doesn't want that | 16:20 |
| dansmith | presumably because he doesn't want to run multiple services and have multiple firewall rules | 16:20 |
| melwitt | oh yeah, I guess he said as much already (scatter gather) | 16:20 |
| sean-k-mooney | ya if he does not want to run multiple proxy instances | 16:21 |
| dansmith | melwitt: ah, I hadn't even clicked the link, but yeah | 16:21 |
| melwitt | we could add another config option! for choosing whether you want a central console auth | 16:21 |
| dansmith | melwitt: I think we'd want that as a toggle | 16:21 |
| dansmith | yeah | 16:21 |
| dansmith | because if you want most efficient and least-shared, you don't want it doing that | 16:22 |
| melwitt | yeah | 16:22 |
| * bauzas shutdowns for the day | 16:32 | |
| *** efried1 is now known as efried | 16:44 | |
| belmoreira | bauzas dansmith melwitt sean-k-mooney I see that you discussed the vncproxy topic. Sorry I needed to leave the office (end of the working day here). | 18:47 |
| sean-k-mooney | no worries. did you add it to the ptg adgenda | 18:48 |
| belmoreira | not yet | 18:48 |
| sean-k-mooney | was dansmith correct when ne assumed you did not want to run multiple novnc proxy instance (1 per cell) | 18:48 |
| belmoreira | let me explain my concern. | 18:49 |
| sean-k-mooney | or are you just concerned about how many port you need to open in the fire wall | 18:49 |
| sean-k-mooney | sure | 18:49 |
| belmoreira | having the vncproxy per cell in theory is good, because we are sharding the service per cell. But it depends in the deployment... For deployments that only expose the console_url in the internal network is ok. | 18:49 |
| belmoreira | However, in my case I need to expose the vncproxy externally. Having only one set os vncproxies allow me to open only one port in the external firewall and have only one console_url address masked by the load balancer. | 18:50 |
| belmoreira | The current approach of having a vncproxy per cell, means that I will have a different console_url per cell. Mapping this with the LB I will need at least to have a different frontend per cell. If I do it per port is a lot of open ports... | 18:50 |
| sean-k-mooney | well you could | 18:50 |
| sean-k-mooney | you can use a reverse proxy instead of a loadblance | 18:50 |
| sean-k-mooney | and expose only one port and have it route the reuest to the backend based on a partil path match | 18:51 |
| sean-k-mooney | or using a subdomain per cell | 18:51 |
| sean-k-mooney | so the reverse proxy is the only thing you open the firewall too and have it dispatch internally to the per cell proxy based on a part of the url | 18:51 |
| belmoreira | true, but in those cases we are also exposing the cell architecture to the user | 18:51 |
| sean-k-mooney | yes at least to the extend needed to match on the url | 18:52 |
| belmoreira | the console_url will be different per cell | 18:52 |
| sean-k-mooney | ya it would be | 18:52 |
| sean-k-mooney | so if we allowed a singel vnc proxy to connect to any of the cell dbs that would be your preference | 18:53 |
| sean-k-mooney | belmoreira: dansmith and melwitt can correct me if i get this wrong but i think what they were suggesting was add a config option to denote if the novnc proxy should connect to on celldb or multiple and having it do a scater gater request to each cell db in the case of multi cell mode | 18:54 |
| belmoreira | having that possibility would be great. I already patch it and have it working in my test infrastructure | 18:55 |
| sean-k-mooney | belmoreira: that would assume that the novnc proxy can actully connect to all the hyperviors in any cell but i belive that is the cause for your env right | 18:55 |
| belmoreira | yes | 18:56 |
| belmoreira | my ideal setup is to have a set of vncproxies that can connect to any hypervisor in the region | 18:58 |
| sean-k-mooney | ya which this would give you | 19:00 |
| sean-k-mooney | i think the main issue is from the consol url we dont know which instance it for without looking up the token and to do that we need to check the cell db | 19:00 |
| sean-k-mooney | so in this case we would need to check multiple cell dbs which increase the load on the db since only one will have the token | 19:01 |
| sean-k-mooney | so we would not want to do that by default but we could allow you to opt into it | 19:01 |
| belmoreira | basically is something similar to the code that I pointed earlier | 19:02 |
| belmoreira | I agree that this shouldn't be the default. Small deployments would not benefit from it | 19:03 |
| dansmith | yeah that's what I meant.. scatter/gather to find the cell that a token is in is not very efficient, but I think the alternative is a lot more work for the few people that might want it | 19:04 |
| dansmith | I'd definitely prefer solving that at the load balancer level with a url suffix or something like that, | 19:05 |
| dansmith | so I think adding a config to scatter/gather is okay and let's not add a new mapping table just for this until/unless performance becomes an issue | 19:05 |
| belmoreira | I think it would be ok. Consoles are not a popular api call. I can report after in terms of performance. | 19:08 |
| belmoreira | thank you all. I can add this into the ptg agenda we need to discuss it more | 19:11 |
| belmoreira | I need to leave now. thank you again | 19:13 |
| zigo | What's the problem with https://review.opendev.org/c/openstack/nova/+/809759 ? (ie what's happening with this nova-tox-validate-backport check?) | 21:15 |
| artom | zigo, it checks that the "source" hash is merged in an upstream branch | 21:57 |
| artom | zigo, so if you cherry pick from wallaby to victoria, the victoria check will fail until the wallaby one merged | 21:57 |
| artom | zigo, ah, no, I had the completely wrong idea without even opening the link | 21:59 |
| artom | "Stable branch requires either cherry-pick -x headers or [stable-only] tag!" is what explains it | 21:59 |
| artom | In the job output | 21:59 |
| clarkb | sounds like someone needs to update the bot or fix the job. Cherrypicking the gitreview change doesn't make sense as it is different than masters | 22:28 |
| clarkb | in my personal opinion it seems like overkill to make people explicitly tag stuff stable only | 22:28 |
| clarkb | its clearly stable only and reviewers can see that why do we need CI to -1? | 22:28 |
| artom | clarkb, it's really a commit message linter, if you think about it | 23:08 |
| artom | I'm very *shrug* about it, though I'd tend to err on the side of "more linting" over "less linting" | 23:09 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!