Wednesday, 2017-03-15

openstackgerritKota Tsuyuzaki proposed openstack/storlets master: Set concurrency 2 for less gate failures with 35 mins Timeout
*** sagara has joined #openstack-storlets07:57
*** akihito has joined #openstack-storlets08:01
sagaraeranrom: Hi !08:01
eranromI have left comments in
sagarayesterday, thank you for adding your comment.08:02
akihitoeranrom: HI!08:03
eranromakihito: Hi08:03
eranromHave also left comments here:
sagaraeranrom: year, I confirmed that, and I wrote a reply comment.08:03
eranromsagara: oh, I missed that. Let me have another look. min.08:04
akihitoeranrom: Thank you. I am looking it.08:04
sagaraeranrom: Sorry, I was writing until just a little while ago08:05
eranromsagara: no problem. I have added replies.08:08
sagaraeranrom: thank you, I will check it right away08:09
eranromThere is one think I am not yet sure about:08:09
eranromWhat is the relationship between per account configuration and per node/proxy configuration.08:10
eranromPer node there is the mechanism defined by reserved_host_cpu_cores, storlets_cpu_cores_fraction (and same for memory)08:11
eranromNow for account there is also cores_per_node (and memory). How do the two should work together?08:12
sagara'per account' means how much resources user can request as containers spec, and 'per node/proxy' is how much resources admin(operator) reserved to host.08:12
sagara'cpu_cores_per_node', I assume we do per account resouce limit with newly creating API which is used by only admin.08:14
eranromSo given both the per-account and the per node/proxy how is the container ends up being executed?08:14
eranromCan I try and write down my understanging?08:15
sagaraI'll glad if you write it down.08:16
eranromok, let me know if I got it right:08:16
sagaraand I also write example case.08:16
eranromSay a user says my account requires 1.5 CPU cores. and the admin says overall storlets can have 10 cores in a proxy/object node08:17
eranromWhen we need to start a container on a given node, we see if there are 1.5 CPUs available from the total 10 cores.08:18
eranromIs that correct?08:18
sagarayes. that's correct08:21
sagarabut I don't think float cpu number request08:22
eranromI think that float cpu is supported, but its not very important for the design.08:22
eranromSo what happens if there are no cores left? Do we retry another node?08:23
eranromAlso, what if other containers on the same node are idle?08:24
sagaraSure, float is just a magnification.08:24
sagaraIf there are no cores left, it returns 409 or 5xx.08:24
sagaraI don't think idle containers, I assumed Akihito's idle resource collection feature.08:25
sagaras/I don't think/I did not consider it/08:26
sagaras/I don't think/I did not consider/08:26
eranromThis might work, although I am afraid that even with a reasonable timeout for Akihito's resource collection feature we may end up with either seconds to tens of seconds of idle08:28
eranromor even with not fully utilized CPUs.08:28
eranromThe fact that a container is not idle does not mean it uses its CPU to their full capacity.08:28
eranromanother option would be to use priorities...I agree its not the same.08:30
sagaraI agree priority about CPU, Disk I/O.08:30
sagaraWe don't need reservation model without memory.08:31
sagaraIs it correct?08:31
eranromyes, I think that for memory this is different. My understanding for memory is:08:32
eranromwhen you give 1GB of ram to a container, it means that it can use up to 1GB memory, but it does not guarantee that there will be 1GB of memory kept only for the container08:33
eranromso it actually means the container has <=1GB08:33
eranromI do need to confirm that08:34
sagaraSure. cgroup may be upper constraint, I also think it is not reservation. but I will confirm that.08:35
sagaraSo we calculate all storlets memory in a node from reserved_host_memory_mb or storlets_memory_fraction, and set it to docker cgroup constraint value.08:38
sagarait can keep docker container's total memory to some upper value, and can keep host stable.08:39
eranromthat was my thinking. and this can be done by passing the constraints to the docker run command.08:39
eranromI guess the more tricky part is doing the priorities, but IMHO this can be done later...08:40
eranromunless of corse you have some strong requirement here...08:40
sagaraPlease confirm your sayng for me, I write it down now08:41
eranromFor the per node we calculate the reserved memory / CPU for storlets and enforce it when starting a container using e.g. the --memory and --cpu-set08:43
eranromMake sense?08:43
sagaraYour saying is, user storlet request will create docker run with --memory.limit_in_bytes=(total_host_memory - reserved_host_memory)08:44
sagarauser storlet request will create docker instance with 'docker run --memory.limit_in_bytes=(total_host_memory - reserved_host_memory)'08:45
eranromsomething like that (there is also the franction option)08:45
sagarasure, I'll implement fraction together.08:46
sagaraSo all docker has memory cap, and their memory request are 'first come, first served'08:47
eranromand similarly docker-run --cpu-set=(total_host_set / reserved_host_set)08:47
eranromwhere / is minus for sets :-)08:47
sagaraShould we limit cpu cores just only cpu.shares?08:48
sagaraI think both idea are good.08:50
eranromThis is also an option. Although swapping has some overhead so IMHO you get better segregation when using cpu-set although I am not sure how different this is...08:50
eranromThe merit of using shares is that when the host is idle you can make use of it...08:51
eranromperhaps Takashi or Kota have an opinion here.08:51
kota_sorry i was missing08:52
eranromnp :-) welcome.08:52
sagaraSorry I cannot understand the relation between 'swapping' and 'cpuset', something like NUMA?08:52
eranromso we have two options for CPU sharing between host and storlets08:53
eranrom(1) use cpu share where storlets get e.g. 0.6 of all cores and host gets 0.4 of all cores08:53
eranrom(2) use sets, suppose there are 10 cores altogether. give storlets 6 and host 408:53
eranromoption 1 is better in the sense that if storlets or host are idle the other one can use the cpus08:54
eranromoption 2 has better assurances on what the host ends up getting.08:54
eranromthis is because CPU swapping between processes has some price,08:55
eranromMy guess is that operators would want the assurances given by option 2 (but this is just a guess)08:56
sagaraI see, maybe (2) is better, we talked no API model, so current design doesn't control cpu.share, so we had better controlling it with cpuset. is that right?08:56
kota_i like option (2) for the first step because it makes deployers easy to estimate how many cores used for users and swift backends08:56
sagaraah, I see, you said about CPU pinning08:56
eranromsagara: right!08:57
eranromregarding the API, I think this can be defined using config files, and 'enforced' using docker run command line params08:58
eranromspecifically, --memory and --cpuset-cpus (
sagaraYes, I know that.09:00
eranromsagara: right, sorry09:01
eranromsagara: nothing to be sorry about :-)09:03
sagaraOK, so all my question is resolved, we have config file to reserve host cpu cores and memory.09:05
sagarawhich are storlets_cpu_cores_fraction, reserved_host_cpu_cores, storlets_memory_fraction, reserved_host_memory_mb.09:05
sagaraUser do storlets request, and if there isn't docker container yet, 'docker run --cpuset.cpus=xxx, --memory.limit_in_bytes=yyy'09:07
sagarawhere xxx is 'total_host_cpus' - 'culculated by storlets_cpu_cores_fraction or reserved_host_cpu_cores'.09:08
sagaraand memory is same as cpu09:09
sagaratherefore, we don't need API09:10
eranromsagara: right. There is also the per account settings which we may want to use shares for, but I suggest to discuss this later - is that ok?09:11
eranromakihito: still here?09:12
sagaraeranrom: thanks a lot!09:12
eranromsagara: sure, my pleasure09:12
eranromakihito: I have commented on the etherpad09:13
akihitooh! Thank you! I look it. and I comment too.09:14
eranromI think that we still need to think more on the long running storlet problem (stop storlet app)09:15
eranromand how to delete files2 and files309:15
eranromjust let me know when09:16
akihitoI see. I will update my opinion on etherpad.09:18
eranromakihito: sure.09:29
akihitosorry. I don't assemble my opinion. I will update it tonight.09:38
*** akihito has quit IRC09:39
*** openstackgerrit has quit IRC10:18
*** sagara has quit IRC10:30
*** akihito has joined #openstack-storlets15:50
*** akihito has quit IRC15:51

Generated by 2.14.0 by Marius Gedminas - find it at!