| *** abhishekk is now known as akekane|home | 03:55 | |
| *** akekane|home is now known as abhishekk | 03:55 | |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Move metadef property policy checks in the API https://review.opendev.org/c/openstack/glance/+/799635 | 03:59 |
|---|---|---|
| opendevreview | Abhishek Kekane proposed openstack/glance master: Move metadef tag policy checks in the API https://review.opendev.org/c/openstack/glance/+/799636 | 03:59 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Implement project personas for metadef namespaces https://review.opendev.org/c/openstack/glance/+/798700 | 04:02 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Implement project personas for metadef objects https://review.opendev.org/c/openstack/glance/+/802054 | 04:02 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Implement project personas for metadef resource-types https://review.opendev.org/c/openstack/glance/+/799671 | 04:02 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Implement project personas for metadef properties https://review.opendev.org/c/openstack/glance/+/802055 | 04:02 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Implement project personas for metadef tags https://review.opendev.org/c/openstack/glance/+/802056 | 04:02 |
| opendevreview | Mridula Joshi proposed openstack/glance master: Fix overwriting of existing tags while creating new tags https://review.opendev.org/c/openstack/glance/+/804966 | 04:40 |
| opendevreview | Merged openstack/glance master: Check policies for Image Tags in API https://review.opendev.org/c/openstack/glance/+/804588 | 04:45 |
| opendevreview | Mridula Joshi proposed openstack/glance master: Fix overwriting of existing tags while creating new tags https://review.opendev.org/c/openstack/glance/+/804966 | 05:25 |
| opendevreview | Abhishek Kekane proposed openstack/glance-tempest-plugin master: Add protection testing for metadef namespaces https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 05:57 |
| *** akekane_ is now known as abhishekk | 06:01 | |
| opendevreview | Abhishek Kekane proposed openstack/glance-tempest-plugin master: Add protection testing for metadef namespaces https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 08:51 |
| opendevreview | Abhishek Kekane proposed openstack/glance-tempest-plugin master: Add protection testing for metadef namespaces https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 09:38 |
| opendevreview | HanGuangyu proposed openstack/glance master: Make "killed" images list available using the CLI https://review.opendev.org/c/openstack/glance/+/806134 | 10:28 |
| *** jokke_ is now known as jokke | 11:07 | |
| *** jokke is now known as jokke_ | 11:07 | |
| opendevreview | Abhishek Kekane proposed openstack/glance-tempest-plugin master: Implement API protection testing for metadef objects https://review.opendev.org/c/openstack/glance-tempest-plugin/+/802793 | 11:45 |
| opendevreview | Mridula Joshi proposed openstack/glance master: Fix overwriting of existing tags while creating new tags https://review.opendev.org/c/openstack/glance/+/804966 | 13:23 |
| opendevreview | Mridula Joshi proposed openstack/glance master: Add doc support for delete-from-store API https://review.opendev.org/c/openstack/glance/+/806180 | 13:23 |
| rajiv_muchelii | hi, how can we calculate glance-api memory requirement? | 13:29 |
| rajiv_muchelii | I would like to understand, glance-api process patterns when several images of different sizes are uploaded together | 13:30 |
| rajiv_muchelii | I see approx 300Mb is required but when I try upload several images, this values spikes to 3.8Gb and kills the process | 13:31 |
| opendevreview | Rajat Dhasmana proposed openstack/glance master: Add doc support for delete-from-store API https://review.opendev.org/c/openstack/glance/+/806180 | 13:35 |
| opendevreview | Rajat Dhasmana proposed openstack/glance master: Fix overwriting of existing tags while creating new tags https://review.opendev.org/c/openstack/glance/+/804966 | 13:35 |
| opendevreview | Erno Kuvaja proposed openstack/glance master: Cache management API endpoints https://review.opendev.org/c/openstack/glance/+/792022 | 13:50 |
| abhishekk | jokke_, rosmaita, dansmith, croelandt, smcginnis glance weekly meeting in 10 minutes | 13:50 |
| abhishekk | at #openstack-meeting | 13:50 |
| jokke_ | rajiv_: so yeah the worker count is one factor on the memory consumption. I think that defaults to number of logical cores in the system capped to 4, but each of those workers has pool of 1000 greenlet threads on their disposal | 15:01 |
| rajiv_ | in my prod, 5 workers are running in parallel | 15:02 |
| jokke_ | these are spun up as needed. but in general you might have 4000 requests being processed at any given time as default config and those will consume decent amount of memory just on their own | 15:02 |
| jokke_ | kk | 15:02 |
| jokke_ | so that means you might potentially handling 5000 connections per server | 15:02 |
| jokke_ | we capped that default few years back as the machines started to get big and suddenly 64 logical cores meant that gapi had 64 workers running and the memory consumption went through the roof just on idle | 15:03 |
| rajiv_ | okay, as you aware, my prod runs on k8s. if any image_upload job is assigned to a pod, does the upload get shared among all the workers ? i guess its not ? there is a dedicated PID to follow the process, correct ? | 15:03 |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Add protection testing for metadef namespaces https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 15:04 |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Implement API protection testing for metadef objects https://review.opendev.org/c/openstack/glance-tempest-plugin/+/802793 | 15:04 |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Implement API protection testing for metadef resource types https://review.opendev.org/c/openstack/glance-tempest-plugin/+/802792 | 15:04 |
| jokke_ | The upload is single socket that is attached to a single greenlet thread on a single worker | 15:05 |
| jokke_ | and that thread will get eventually garbage collected/recycled after the socket is closed | 15:05 |
| jokke_ | I dont think the greenlet threads gets their own pids 'though. Workers definitely do as they are proper child processes | 15:06 |
| rajiv_ | yes, i see a PID for the worker jobs in the DEBUG logs | 15:07 |
| rajiv_ | but i am still figuring out on how to set a value or limit upload processes to prevent another crash. | 15:08 |
| rajiv_ | are there any tempest tests which can assist here ? | 15:09 |
| rajiv_ | can we use the glance-tempest-plugin repo now ? i still dont see tests added here. | 15:09 |
| jokke_ | I don't think there is any performance/scale testing happening upstream at all. we used to have rally jobs but I think those haven't been ran for years anymore | 15:10 |
| jokke_ | you might need to just set the memory limits very liberal and hope none of your apis gets hit by basically all your uploads | 15:11 |
| jokke_ | somewhere like you know 32-64 gigs might give you pretty good margin | 15:12 |
| jokke_ | I don't know how k8s deals with that 'though does it allocate that memory or is it just limit out of shared pool? | 15:12 |
| jokke_ | Might be worth of probing other operators on the mailing list too for their experiences and tips | 15:13 |
| rajiv_ | FYR : https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits | 15:13 |
| rajiv_ | sure, any names ? | 15:14 |
| rajiv_ | to be precise : https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory | 15:15 |
| jokke_ | rajiv_: yeah just got to that point. It's tricky ... like said I have no experience imposing artificial limits to the process, so won't be much of help unfortunately :( | 15:17 |
| rajiv_ | okay, i will set the maximum value possible and monitor for few days, something like hit and try method. | 15:18 |
| jokke_ | openstack-discuss mailing list and I think the subject tag is [ops] that lots of the operators are following there. Shoot an e-mail and hopefully there is someone round who would have better experience running the services under k8s | 15:18 |
| rajiv_ | thanks a lot jokke_ like always :) | 15:19 |
| jokke_ | NP ... I might have some interesting numbers in the future as I'll be looking closer into the perfscale, but it won't help now | 15:20 |
| jokke_ | feel free to share your findings too. I'm sure your experiences will help others to move on with their deployments too | 15:21 |
| rajiv_ | roger that! | 15:21 |
| abhishekk | lbragstad, could you please have a look at metadef rbac protection test patch ? | 15:46 |
| abhishekk | #link https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 15:47 |
| lbragstad | abhishekk yeah - i can take a look today | 15:50 |
| abhishekk | lbragstad, ack, thank you | 15:50 |
| *** akekane_ is now known as abhishekk | 16:18 | |
| * abhishekk going for dinner | 16:18 | |
| * abhishekk signing out for the day | 17:13 | |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Add protection testing for metadef namespaces https://review.opendev.org/c/openstack/glance-tempest-plugin/+/800902 | 18:58 |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Implement API protection testing for metadef objects https://review.opendev.org/c/openstack/glance-tempest-plugin/+/802793 | 18:58 |
| opendevreview | Pranali Deore proposed openstack/glance-tempest-plugin master: Implement API protection testing for metadef resource types https://review.opendev.org/c/openstack/glance-tempest-plugin/+/802792 | 18:58 |
| lbragstad | i generated a sample policy file for glance and noticed that this policy has nested, unescaped double quotes - is that a problem? | 20:28 |
| lbragstad | https://github.com/openstack/glance/blob/master/glance/policies/base.py#L68 | 20:28 |
| lbragstad | "download_image": "role:admin or (role:member and (project_id:%(project_id)s or project_id:%(member_id)s or "community":%(visibility)s or "public":%(visibility)s or "shared":%(visibility)s))" | 20:28 |
| dansmith | policy json? I'm confused | 21:14 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!