| opendevreview | Gage Hugo proposed openstack/openstack-helm master: Remove cinder v1/v2 defaults https://review.opendev.org/c/openstack/openstack-helm/+/801014 | 02:45 |
|---|---|---|
| opendevreview | Gage Hugo proposed openstack/openstack-helm-infra master: WIP - Testing bitnami memcached chart https://review.opendev.org/c/openstack/openstack-helm-infra/+/787788 | 02:52 |
| opendevreview | Gage Hugo proposed openstack/openstack-helm-infra master: WIP - Testing bitnami memcached chart https://review.opendev.org/c/openstack/openstack-helm-infra/+/787788 | 03:42 |
| opendevreview | zhen proposed openstack/openstack-helm-infra master: Modify the rbac_role tpl to make secrets accessible https://review.opendev.org/c/openstack/openstack-helm-infra/+/802434 | 08:10 |
| opendevreview | Susanta gautam proposed openstack/openstack-helm master: Set default workers count for controller workers to one https://review.opendev.org/c/openstack/openstack-helm/+/802451 | 10:34 |
| opendevreview | Susanta gautam proposed openstack/openstack-helm master: Change default workers for rpc_workers to one https://review.opendev.org/c/openstack/openstack-helm/+/802455 | 10:53 |
| *** reddy is now known as Guest2435 | 13:23 | |
| reddy1 | o/ | 15:01 |
| gagehugo | #startmeeting openstack-helm | 15:01 |
| opendevmeet | Meeting started Tue Jul 27 15:01:29 2021 UTC and is due to finish in 60 minutes. The chair is gagehugo. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'openstack_helm' | 15:01 |
| gagehugo | cool that works | 15:01 |
| gagehugo | o/ | 15:01 |
| reddy1 | o/ | 15:01 |
| sangeet | o/ | 15:01 |
| reddy1 | I'd like to discuss: https://review.opendev.org/c/openstack/openstack-helm/+/801043 | 15:05 |
| reddy1 | The PS calculates the workers for nova services and there are similar PS for other services. | 15:06 |
| gagehugo | yeah I saw that | 15:06 |
| gagehugo | #topic worker count nova | 15:06 |
| gagehugo | #link https://review.opendev.org/c/openstack/openstack-helm/+/801043 | 15:06 |
| reddy1 | I'm of the opinion that we can set the workers to 1 and let the tenant scale the replicas | 15:07 |
| lamt | \o sorry. I was running late | 15:07 |
| puppetsock | reddy1: a long time ago thats where i was too | 15:07 |
| puppetsock | but its an approach that leads to a lot of overhead | 15:08 |
| puppetsock | 32 worker pods of 1 thread each != 1 pod of 32 threads (and the linear between them) | 15:08 |
| puppetsock | that said, this sort of automagic worker calc, also makes me a bit worried | 15:08 |
| sangeet | I agree .. it should not be auto. Can this be overwritten? | 15:08 |
| puppetsock | the only time i think it really adds value is when you have nodes of diff cpu thread counts in your cluster | 15:09 |
| puppetsock | in which case, you'll have some pods running x threads and others running y | 15:09 |
| sangeet | I mean if it is auto, we can still overwrite it, correct | 15:09 |
| reddy1 | I agree with the auto calculation as well. Moreover, nproc returns the number of cpus on the pods, not from the pod resource limits | 15:09 |
| reddy1 | * Moreover, nproc returns the number of cpus on the host, not from the pod resource limits | 15:10 |
| puppetsock | is the rationalle behind this ps in the commit somewhere | 15:11 |
| puppetsock | i think that would be fantastic to have, it may be trying to solve a issue we have not thought of | 15:11 |
| gagehugo | puppetsock: I asked why in the ps I believe | 15:11 |
| miniroy | o/ sorry I was running late also | 15:12 |
| gagehugo | I am dubious about hard-coding n/4 into the chart as "auto" | 15:12 |
| gagehugo | what if we wanna do n/2? or n/8? | 15:12 |
| puppetsock | i see your comment gagehugo | 15:12 |
| puppetsock | id push to the an answer to your 1st q in your comment | 15:12 |
| reddy1 | @Sangeet, yes auto can be overwritten but having this as default may not be optimal solution | 15:12 |
| lamt | miniroy: I thought you saw issues before with this autoscaling? | 15:13 |
| lamt | something about talking to rabbit? | 15:13 |
| lamt | s/autoscaling/# of worker threads/ | 15:13 |
| gagehugo | I don't think this change will do what they think it will | 15:14 |
| puppetsock | gagehugo: i assume you thinking that its intended to allow 'hands free' deployment of osh | 15:15 |
| miniroy | lamt: yeah there is still that rabbitmq issue when nova osapi workers are set to > 1 | 15:15 |
| puppetsock | and have it right size to the env? | 15:15 |
| gagehugo | puppetsock: yeah, I am assuming that is the intended goal | 15:15 |
| puppetsock | in which case i agree | 15:15 |
| puppetsock | i think to reddy1's point above | 15:15 |
| puppetsock | it should scale to the resources on the pod | 15:15 |
| puppetsock | rather than the underlying host in that case | 15:16 |
| puppetsock | eg - if you allocate 2vcpu you get a multiple of threads | 15:16 |
| puppetsock | would also be much easier/cleaner to implement | 15:16 |
| puppetsock | not requireing init containers etc | 15:17 |
| puppetsock | but could be done purely through the config template | 15:17 |
| gagehugo | yeah | 15:19 |
| gagehugo | but also miniroy's point still stands (for nova anyway) | 15:20 |
| gagehugo | bumping the worker count has unintended side-effects with oslo.messaging | 15:20 |
| gagehugo | There is also a ps for cinder as well for this | 15:22 |
| gagehugo | #link https://review.opendev.org/c/openstack/openstack-helm/+/802015 | 15:22 |
| puppetsock | id provide some fb in the reviews with a summary or link to this meeting | 15:23 |
| gagehugo | sure | 15:24 |
| puppetsock | also miniroy it may be worth updating the docs, or putting a comment in the values yaml | 15:24 |
| puppetsock | for what you have observed | 15:24 |
| puppetsock | when workers > 1 | 15:24 |
| miniroy | puppetsock: yeah will probably do that, we were just noticing and investigating this issue recently | 15:25 |
| puppetsock | nice | 15:25 |
| miniroy | seeing lots of connection timeout messages in rabbitmq when more than 1 worker is defined | 15:25 |
| miniroy | seems something is interfering with the heartbeat message is we are guessing but not overly certain on the reason | 15:26 |
| gagehugo | yeah, we might wanna add a comment then if that turns out to be an issue | 15:27 |
| gagehugo | #topic Yoga PTG | 15:28 |
| gagehugo | So the sign up for timeslots for the next PTG is available, does this timeslot work for everyone? | 15:28 |
| gagehugo | It's what we've done in the past | 15:29 |
| gagehugo | just use the meeting time now instead for the PTG | 15:29 |
| miniroy | works for me | 15:29 |
| gagehugo | We can verify next week as well and get signed up | 15:32 |
| gagehugo | #topic open discussion | 15:32 |
| gagehugo | Anyone have anything else they wanna discuss? | 15:32 |
| reddy1 | Thank you for the discussion around the workers count. Appreciate it. | 15:33 |
| opendevreview | Merged openstack/openstack-helm-infra master: [ceph-osd] Mount /var/crash inside ceph-osd pods https://review.opendev.org/c/openstack/openstack-helm-infra/+/801536 | 15:41 |
| opendevreview | Gage Hugo proposed openstack/openstack-helm master: Use bitnami memcached chart https://review.opendev.org/c/openstack/openstack-helm/+/802461 | 18:12 |
| opendevreview | chinasubbareddy mallavarapu proposed openstack/openstack-helm master: [OSH] Add make utlity to list of required tools https://review.opendev.org/c/openstack/openstack-helm/+/802595 | 18:13 |
| opendevreview | Gage Hugo proposed openstack/openstack-helm master: Use bitnami memcached chart https://review.opendev.org/c/openstack/openstack-helm/+/802461 | 18:17 |
| opendevreview | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: [ceph-provisioner] Add ceph mon v2 port for ceph csi provisioner https://review.opendev.org/c/openstack/openstack-helm-infra/+/802597 | 18:26 |
| opendevreview | Nafiz Haider proposed openstack/openstack-helm master: [nova-novnc] Mount rabbitmq tls certs for nova-novnc https://review.opendev.org/c/openstack/openstack-helm/+/802598 | 18:34 |
| opendevreview | Christopher Straut proposed openstack/openstack-helm-images master: Add retry around Nagios Request https://review.opendev.org/c/openstack/openstack-helm-images/+/802600 | 18:50 |
| opendevreview | Nafiz Haider proposed openstack/openstack-helm master: [nova-novnc] Mount rabbitmq tls certs for nova-novnc https://review.opendev.org/c/openstack/openstack-helm/+/802598 | 18:52 |
| opendevreview | Sangeet Gupta proposed openstack/openstack-helm-infra master: cert-rotation: New chart for certificate rotation https://review.opendev.org/c/openstack/openstack-helm-infra/+/788061 | 18:52 |
| opendevreview | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: [ceph-provisioner] Add check for empty ceph endpoint https://review.opendev.org/c/openstack/openstack-helm-infra/+/802605 | 19:01 |
| opendevreview | Sangeet Gupta proposed openstack/openstack-helm-infra master: cert-rotation: New chart for certificate rotation https://review.opendev.org/c/openstack/openstack-helm-infra/+/788061 | 19:02 |
| opendevreview | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: [ceph-provisioner] Add check for empty ceph endpoint https://review.opendev.org/c/openstack/openstack-helm-infra/+/802605 | 19:17 |
| opendevreview | Christopher Straut proposed openstack/openstack-helm-images master: Add retry around Nagios Request https://review.opendev.org/c/openstack/openstack-helm-images/+/802600 | 19:19 |
| opendevreview | Nafiz Haider proposed openstack/openstack-helm master: [nova-novnc] Mount rabbitmq tls cert for nova-novnc communication https://review.opendev.org/c/openstack/openstack-helm/+/802598 | 19:29 |
| opendevreview | Sangeet Gupta proposed openstack/openstack-helm-infra master: Helm-Toolkit: Make Rabbit-init job more robust https://review.opendev.org/c/openstack/openstack-helm-infra/+/800097 | 20:20 |
| opendevreview | Nafiz Haider proposed openstack/openstack-helm master: Mount rabbitmq tls cert for nova-novnc communication https://review.opendev.org/c/openstack/openstack-helm/+/802598 | 21:02 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!