| *** gyee has quit IRC | 00:36 | |
| *** ykarel has joined #openstack-rpm-packaging | 02:34 | |
| *** ykarel has quit IRC | 05:21 | |
| *** ykarel has joined #openstack-rpm-packaging | 05:21 | |
| *** pcaruana|elisa| has joined #openstack-rpm-packaging | 05:29 | |
| *** pcaruana|elisa| has quit IRC | 05:37 | |
| *** ykarel has quit IRC | 05:51 | |
| *** chkumar|off is now known as chandankumar | 07:13 | |
| *** ykarel has joined #openstack-rpm-packaging | 07:42 | |
| *** pcaruana|elisa| has joined #openstack-rpm-packaging | 07:45 | |
| openstackgerrit | Dirk Mueller proposed openstack/rpm-packaging master: update oslo.config to 6.6.2 https://review.openstack.org/613072 | 07:48 |
|---|---|---|
| *** ykarel has quit IRC | 07:49 | |
| *** ykarel has joined #openstack-rpm-packaging | 07:50 | |
| *** gkadam has joined #openstack-rpm-packaging | 07:54 | |
| *** ykarel has quit IRC | 07:56 | |
| *** ykarel has joined #openstack-rpm-packaging | 08:02 | |
| *** ykarel has quit IRC | 08:10 | |
| *** ykarel has joined #openstack-rpm-packaging | 08:27 | |
| *** jpich has joined #openstack-rpm-packaging | 08:28 | |
| *** ykarel has quit IRC | 08:28 | |
| *** ykarel has joined #openstack-rpm-packaging | 08:29 | |
| *** ykarel_ has joined #openstack-rpm-packaging | 08:31 | |
| *** ykarel has quit IRC | 08:34 | |
| *** ykarel has joined #openstack-rpm-packaging | 08:34 | |
| *** ykarel_ has quit IRC | 08:38 | |
| *** ykarel has quit IRC | 08:40 | |
| *** jpena|off is now known as jpena | 08:52 | |
| openstackgerrit | Merged openstack/rpm-packaging master: Switch keystone pyldap dependency to python-ldap https://review.openstack.org/614209 | 09:04 |
| *** pcaruana|elisa| has quit IRC | 09:13 | |
| *** pcaruana|elisa| has joined #openstack-rpm-packaging | 09:15 | |
| *** zul has joined #openstack-rpm-packaging | 09:57 | |
| *** jpena is now known as jpena|lunch | 12:46 | |
| *** gkadam has quit IRC | 12:57 | |
| *** gkadam has joined #openstack-rpm-packaging | 13:03 | |
| *** jpena|lunch is now known as jpena | 13:48 | |
| openstackgerrit | Johannes Grassler proposed openstack/rpm-packaging master: monasca-agent: switch to systemd supervision https://review.openstack.org/613918 | 14:02 |
| dirk | #startmeeting rpm_packaging | 14:06 |
| openstack | Meeting started Wed Oct 31 14:06:56 2018 UTC and is due to finish in 60 minutes. The chair is dirk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:06 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:06 |
| *** openstack changes topic to " (Meeting topic: rpm_packaging)" | 14:06 | |
| openstack | The meeting name has been set to 'rpm_packaging' | 14:07 |
| dirk | ping toabctl, dirk, apevec, aplanas, IgorYozhikov, jpena, jruzicka, number80, kaslcrof, ykarel | 14:07 |
| dirk | #topic roll call | 14:07 |
| *** openstack changes topic to "roll call (Meeting topic: rpm_packaging)" | 14:07 | |
| * dirk struggles with etherpad | 14:08 | |
| jpena | o/ | 14:11 |
| cmurphy | o/ | 14:16 |
| dirk | #chair jpena cmurphy | 14:18 |
| openstack | Current chairs: cmurphy dirk jpena | 14:18 |
| dirk | does anyone have topics? I can't access etherpad for some reason | 14:19 |
| jpena | nothing special for me | 14:19 |
| dirk | jpena: are you coming to the berlin summit? | 14:19 |
| jpena | dirk: no, not this time | 14:19 |
| dirk | anyone else coming that is worth talking to ? | 14:22 |
| dirk | I mean, sorry, in the context of contributions for the openstack rpm packaging | 14:22 |
| jpena | from the Red Hat side I'm not aware of many people going to Berlin | 14:23 |
| dirk | k, thanks | 14:30 |
| dirk | #topic reviews | 14:30 |
| *** openstack changes topic to "reviews (Meeting topic: rpm_packaging)" | 14:30 | |
| dirk | do we have reviews to talk about? | 14:30 |
| dirk | https://review.openstack.org/#/c/613652/ looks like an easy merge | 14:30 |
| dirk | ah, I see cmurphy and jpena would need to look at https://review.openstack.org/#/c/610011/ again - I would like to close out this topic | 14:31 |
| dirk | any strong opinions on how to get this merged? | 14:31 |
| dirk | or if it should be merged? | 14:31 |
| jpena | I don't have a strong opinion on the limit increase. I saw cmurphy didn't like the increase in TasksMax | 14:32 |
| cmurphy | I'm still unclear on why raising both nproc and tasksmax is okay | 14:32 |
| jpena | dirk: how did you come up with those numbers? | 14:33 |
| dirk | jpena: I copied them from ceph-osd.service | 14:33 |
| cmurphy | when i happened to check this on a customer's production site the number of tasks was at 22 for cinder-volume | 14:33 |
| dirk | the issue we had was with a user that was running cinder-volume with rbd at scale. | 14:33 |
| dirk | and librbd just asserts() (which crashes all of cinder-volume) when it can't start a new thread | 14:34 |
| dirk | so TasksMax=infinity basicall removes the systemd cgroup pid controller | 14:34 |
| dirk | and the rest is just bumping the limits accross the sanity imho | 14:34 |
| dirk | cmurphy: so you considered tasksmax=infinity the unsafe part? or the NPROC? | 14:35 |
| cmurphy | dirk: the combination of raising both is what seems unsafe to me | 14:38 |
| cmurphy | but i'm not an expert | 14:38 |
| cmurphy | but i noted that ubuntu doesn't bother with these limits http://paste.openstack.org/show/732149/ | 14:39 |
| dirk | so LimitNPROC=1234\nTasksMax=500 means "this service and its children are allowed to use 500 pids (threads, processes), all of the same uid are allowed 1234" | 14:39 |
| dirk | LimitNPROC=1234\nTasksMax=inifinity means "this service is allowed to use up to the user rlimit (1234)" | 14:40 |
| jpena | oh, so nproc still affects | 14:40 |
| dirk | yes | 14:40 |
| dirk | nproc is number of processes for this *user* (not this process tree) | 14:40 |
| dirk | so e.g. cinder-api and cinder-volume share limits | 14:41 |
| dirk | (as they run as the same user) | 14:41 |
| dirk | tasksMax is a systemd feature to limit cinder-volume so that its not "eating" the quota of cinder-api | 14:41 |
| dirk | but exactly this feature caused the issue for the customer (he was having 8191 threads in ceph and then it asserted) | 14:41 |
| dirk | when it tried to spawn 8192 | 14:41 |
| dirk | given the tradeoff "all of cinder volume is down for every user" and "well, ceph goes slightly beyond sanity but if the hardware is powerful enough we allow it" this change is changing it towards "we keep cinder volume up at all cost" | 14:42 |
| cmurphy | is librbd supposed to be using that many threads? | 14:43 |
| dirk | cmurphy: I have to admit I ignore the ubuntu reason because it just means they eithe rhave some other way of overriding the defaults (e.g. via a limits drop in from their orchestration) or they never had the issue so far (for example because they use the older systemd or have a DefaultTasksMax=infinity configured) | 14:43 |
| dirk | so if they don't set it, the global default catches, and if that one is infinity then they never have cgroup pid limits | 14:43 |
| dirk | cmurphy: well.. thats a whole other debate.. :-) | 14:44 |
| dirk | IMHO 8192 is plenty but then again we saw that this limit is being reached | 14:44 |
| cmurphy | but is it reached because it's a runaway, or because it's functioning normally? if it's a runaway and malfunctioning then raising the limits for it is just going to delay the inevitable crash | 14:45 |
| dirk | so, to draw a conclusion here, a simple TasksMax=infinity change without raising LimitNOFILE and limitNPROC would be acceptable? | 14:45 |
| dirk | cmurphy: there wasn't a leak, its just a load spike, it is normal a few seconds later | 14:46 |
| cmurphy | okay | 14:46 |
| dirk | e.g. the user is spawning a heat stack with a few dozenn volumes inside | 14:46 |
| dirk | and while I agree librbd could be less agressively spawning threads, still any decent hardware should be capable of handling 8192 threads, so its not that the machine would be crawling death with that many threads | 14:47 |
| dirk | (although it will eat like 64GB of RAM just for stack space) | 14:48 |
| cmurphy | okay, tasksmax=infinity without the other limit changes sounds fine to me | 14:48 |
| dirk | #topic open floor | 14:53 |
| *** openstack changes topic to "open floor (Meeting topic: rpm_packaging)" | 14:53 | |
| openstackgerrit | Dirk Mueller proposed openstack/rpm-packaging master: cinder-volume: Raise limits and disable cgroup limtis https://review.openstack.org/610011 | 14:54 |
| dirk | cmurphy: done ^^ :) | 14:54 |
| dirk | anything else? | 14:54 |
| dirk | T-2 min before ending ;) | 14:54 |
| dirk | (happy halloween) | 14:54 |
| * cmurphy nothing | 14:54 | |
| jpena | nope | 14:56 |
| dirk | thanks :) | 14:56 |
| dirk | #endmeeting | 14:57 |
| *** openstack changes topic to "https://etherpad.openstack.org/p/openstack-rpm-packaging - Regular IRC Meeting Wednesdays 1:00 PM UTC in openstack-rpm-packaging" | 14:57 | |
| openstack | Meeting ended Wed Oct 31 14:57:00 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/rpm_packaging/2018/rpm_packaging.2018-10-31-14.06.html | 14:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/rpm_packaging/2018/rpm_packaging.2018-10-31-14.06.txt | 14:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/rpm_packaging/2018/rpm_packaging.2018-10-31-14.06.log.html | 14:57 |
| *** gyee has joined #openstack-rpm-packaging | 15:01 | |
| *** gkadam has quit IRC | 16:20 | |
| openstackgerrit | Dirk Mueller proposed openstack/rpm-packaging master: WIP Restore monasca-ceilometer package https://review.openstack.org/611982 | 16:22 |
| *** pcaruana|elisa| has quit IRC | 16:37 | |
| *** gyee has quit IRC | 16:50 | |
| openstackgerrit | Johannes Grassler proposed openstack/rpm-packaging master: monasca-agent: switch to systemd supervision https://review.openstack.org/613918 | 16:52 |
| *** chandankumar is now known as chkumar|off | 16:53 | |
| *** jpich has quit IRC | 16:59 | |
| *** gyee has joined #openstack-rpm-packaging | 17:44 | |
| *** jpena is now known as jpena|off | 18:04 | |
| *** nicolasbock has joined #openstack-rpm-packaging | 19:19 | |
| *** gyee has quit IRC | 22:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!