| *** huntxu has joined #openstack-qinling | 01:15 | |
| *** caoyuan has joined #openstack-qinling | 01:49 | |
| *** jiangyuan has joined #openstack-qinling | 01:57 | |
| *** caoyuan_ has joined #openstack-qinling | 02:02 | |
| *** caoyuan has quit IRC | 02:04 | |
| jiangyuan | hi, kong! Do you have any related documents about ' the ulimit will limit the resource uasge in current process *AND* its subprocesses'? I just saw 'A child process created via fork(2) inherits its parent's resource limits' in Linux man page. | 02:10 |
|---|---|---|
| kong | jiangyuan: for your testing here https://paste.ubuntu.com/p/sKdP3VgJtM/, which user did you use? | 02:16 |
| jiangyuan | yuan@yuan-ThinkPad-T430:~/pythonText$ python testRLIMIT.py | 02:20 |
| jiangyuan | (209715200, 209715200) | 02:20 |
| jiangyuan | and huntxu provided this test | 02:21 |
| kong | jiangyuan: have you tested the process number? | 02:24 |
| kong | i need to refresh my mind from k8s to linux rlimit and i need to test the memory eating behavor again | 02:26 |
| *** yufei has joined #openstack-qinling | 02:32 | |
| *** yufei has quit IRC | 02:37 | |
| huntxu | kong: while subprocesses are sleeping(line 10 of the script), ps aux shows 50 more processes with the same commandline | 02:37 |
| jiangyuan | Test result:https://paste.ubuntu.com/p/wG5ptjG5f3/ | 02:38 |
| huntxu | kong: and via free, I also observed ~2G more memory occupied and then freed after the running finished | 02:38 |
| *** yufei has joined #openstack-qinling | 02:39 | |
| huntxu | jiangyuan: wow, the VSZ on your machine is quite smaller than mine, I guess you have to lower the limit to 100MiB so that the non-forking path would hit MemoryError | 02:41 |
| kong | huntxu, jiangyuan, thanks for your investigation, please give me some time before I do the test, i have to some other higher prioritiy things need to do | 02:41 |
| kong | but, besides the memory, could you please also test the process number if you are available? | 02:42 |
| kong | and make sure the script is running using non-root user | 02:42 |
| huntxu | kong: Yes I'm running it non-root. For the process number, you mean the 128 NPROC limit? I would give it a try | 02:43 |
| kong | huntxu: yes, that's what i'm talking about, thanks! | 02:43 |
| huntxu | kong: Update some information, for the NPROC_LIMIT, I tested it and it is verified that fork bomb will be prevented, no more that the NPROC_LIMIT processes can be created | 03:26 |
| yufei | hello, qinlingers, I'm from china insea research center, I and my colleagues will work on qinling recently. :) | 03:27 |
| huntxu | kong: then I believe that the doc of the setrlimit manpanges is correct, RLIMIT_NPROC is in fact limiting The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. | 03:27 |
| huntxu | kong: FYI https://linux.die.net/man/2/setrlimit | 03:27 |
| huntxu | kong: so I created a new qinling function to fork 80 processes and then sleep for a while, start two function executions asynchronously. They are scheduled to run on the same pod | 03:28 |
| huntxu | kong: and unfortunately, the later created execution failed for the reason of too much resource consumption | 03:29 |
| *** caoyuan_ has quit IRC | 03:29 | |
| huntxu | kong: so that means if more than one execution is running on the pod, they will share the NPROC_LIMIT together. | 03:31 |
| huntxu | jiangyuan: ^ | 03:31 |
| kong | huntxu: ok, i will do more test during the weekend as i will get off work soon. i will let you know my findings so we could work together about how to restrict the resource usage for individial function exeuction | 03:33 |
| kong | yufei: welcome to qinling :-) | 03:34 |
| yufei | (๑¯◡¯๑) | 03:35 |
| kong | yufei: feel free to ask if you have any question, but be sure to stay here (using irccloud or znc) otherwise you will miss the reply :-) | 03:36 |
| huntxu | kong: ok | 03:38 |
| yufei | thanks for reminding, I will invite my partener to this channel too, we now start from review qinling and will try to commit latter. | 03:38 |
| kong | yufei: awesome | 03:38 |
| *** liusheng has quit IRC | 03:47 | |
| *** Steven_Gao has joined #openstack-qinling | 03:47 | |
| *** Steven_Gao has quit IRC | 03:47 | |
| *** Steven_Gao has joined #openstack-qinling | 03:47 | |
| *** liusheng has joined #openstack-qinling | 03:47 | |
| *** caoyuan has joined #openstack-qinling | 04:05 | |
| *** Steven_Gao has quit IRC | 04:36 | |
| jiangyuan | huntxu: Did you set NPROC limit in the parent process(the flask app) when testing? | 05:20 |
| huntxu | jiangyuan: the current implementation python2.7 runtime sets the limit to 129 | 05:37 |
| huntxu | 128 | 05:37 |
| huntxu | jiangyuan: so I fork 80 processes in a function and create two async executions | 05:38 |
| jiangyuan | huntxu: ok, I see | 05:40 |
| jiangyuan | huntxu:So, based on your test, each execution in its individual subprocess will share RLIMIT_NPROC number which is set in every subprocesses | 05:44 |
| huntxu | jiangyuan: the NPROC limit should be considered based on user, it limits how many processes a user can create | 05:46 |
| huntxu | jiangyuan: two qinling executions run on the same pod, so they are run by the same user in the k8s pod | 05:47 |
| huntxu | jiangyuan: I guess if they run in different pods, they won't affect each other, as different pods should be seperated by namespaces(pid, user...) | 05:48 |
| jiangyuan | huntxu: So, no matter in which subprocess we set the NPROC limit, the NPROC limit will work on the user . But RLIMIT_CPU/AS will not? | 05:51 |
| huntxu | jiangyuan: yes, I think so, IIUC from the setrlimit(2) manpages | 05:52 |
| jiangyuan | huntxu: ok, thx | 05:52 |
| *** cctopusZhang has joined #openstack-qinling | 05:53 | |
| *** yufei has quit IRC | 05:54 | |
| *** cctopusZhang is now known as yufei | 05:54 | |
| *** yufei has quit IRC | 06:18 | |
| *** yufei has joined #openstack-qinling | 08:22 | |
| *** cctopusZhang has joined #openstack-qinling | 09:01 | |
| *** yufei has quit IRC | 09:04 | |
| *** cctopusZhang is now known as yufei | 09:04 | |
| *** yufei has quit IRC | 09:45 | |
| *** yufei has joined #openstack-qinling | 09:46 | |
| *** caoyuan has quit IRC | 09:59 | |
| *** yufei has quit IRC | 10:11 | |
| *** larainema has quit IRC | 10:15 | |
| openstackgerrit | Merged openstack/python-qinlingclient master: Updated from global requirements https://review.openstack.org/553258 | 10:26 |
| *** jiangyuan has quit IRC | 10:34 | |
| *** larainema has joined #openstack-qinling | 10:47 | |
| *** caoyuan has joined #openstack-qinling | 10:52 | |
| *** caoyuan has quit IRC | 11:08 | |
| *** caoyuan has joined #openstack-qinling | 11:20 | |
| *** caoyuan has quit IRC | 11:25 | |
| openstackgerrit | Hunt Xu proposed openstack/qinling master: engine: delete workers from etcd after removing https://review.openstack.org/555744 | 11:38 |
| openstackgerrit | Hunt Xu proposed openstack/qinling master: engine: delete workers from etcd after removing https://review.openstack.org/555744 | 11:44 |
| *** caoyuan has joined #openstack-qinling | 11:59 | |
| *** caoyuan has quit IRC | 12:04 | |
| *** huntxu has quit IRC | 12:13 | |
| *** caoyuan has joined #openstack-qinling | 14:28 | |
| *** caoyuan_ has joined #openstack-qinling | 14:34 | |
| *** caoyuan has quit IRC | 14:37 | |
| -openstackstatus- NOTICE: zuul.o.o has been restarted to pick up latest code base and clear memory usage. Both check / gate queues were saved, be sure to check your patches and recheck when needed. | 14:52 | |
| *** rbrady is now known as rbrady-afk | 15:12 | |
| *** caoyuan_ has quit IRC | 15:31 | |
| -openstackstatus- NOTICE: Gerrit will be temporarily unreachable as we restart it to complete the rename of some projects. | 15:50 | |
| *** openstackgerrit has quit IRC | 19:34 | |
| *** liusheng has quit IRC | 23:32 | |
| *** liusheng has joined #openstack-qinling | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!