| *** ianychoi has quit IRC | 00:07 | |
| *** ianychoi has joined #openstack-containers | 00:08 | |
| *** ondrejburian has quit IRC | 00:50 | |
| *** ondrejburian has joined #openstack-containers | 00:51 | |
| *** ianychoi has quit IRC | 01:30 | |
| *** ianychoi has joined #openstack-containers | 01:32 | |
| *** ianychoi has quit IRC | 02:10 | |
| *** ianychoi has joined #openstack-containers | 02:18 | |
| *** hongbin has joined #openstack-containers | 02:20 | |
| *** hongbin has quit IRC | 02:39 | |
| *** ricolin_ has joined #openstack-containers | 03:01 | |
| *** ianychoi has quit IRC | 03:35 | |
| *** ianychoi has joined #openstack-containers | 03:37 | |
| *** ricolin_ has quit IRC | 03:38 | |
| *** ykarel|away is now known as ykarel | 03:44 | |
| *** tkaprol has joined #openstack-containers | 03:53 | |
| *** tkaprol has quit IRC | 04:01 | |
| *** ricolin has quit IRC | 04:44 | |
| *** ianychoi has quit IRC | 05:00 | |
| *** ianychoi has joined #openstack-containers | 05:03 | |
| *** udesale has joined #openstack-containers | 05:07 | |
| *** ricolin has joined #openstack-containers | 05:51 | |
| *** rcernin has quit IRC | 05:58 | |
| *** ianychoi has quit IRC | 06:01 | |
| *** ianychoi has joined #openstack-containers | 06:03 | |
| *** ianychoi has quit IRC | 06:11 | |
| *** ianychoi has joined #openstack-containers | 06:13 | |
| *** ianychoi has quit IRC | 06:29 | |
| *** ianychoi has joined #openstack-containers | 06:31 | |
| *** ianychoi has quit IRC | 06:39 | |
| *** ianychoi has joined #openstack-containers | 06:41 | |
| *** ianychoi has quit IRC | 06:58 | |
| *** ianychoi has joined #openstack-containers | 07:05 | |
| *** pcaruana has joined #openstack-containers | 08:16 | |
| *** openstackgerrit has joined #openstack-containers | 08:16 | |
| openstackgerrit | Merged openstack/magnum master: Fix duplicated words issue like "meaning meaning that" https://review.opendev.org/701166 | 08:16 |
|---|---|---|
| *** flwang1 has joined #openstack-containers | 08:34 | |
| *** sapd1_x has quit IRC | 08:36 | |
| *** ykarel is now known as ykarel|lunch | 08:38 | |
| *** gouthamr has quit IRC | 08:49 | |
| *** mgoddard has quit IRC | 08:49 | |
| *** gouthamr has joined #openstack-containers | 08:50 | |
| brtknr | Morning folks | 08:54 |
| *** vishalmanchanda has joined #openstack-containers | 08:56 | |
| *** ykarel has joined #openstack-containers | 08:56 | |
| *** ykarel|lunch has quit IRC | 08:57 | |
| flwang1 | hello | 08:58 |
| strigazi | hello | 08:59 |
| flwang1 | #startmeeting magnum | 09:00 |
| openstack | Meeting started Wed Mar 11 09:00:11 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: magnum)" | 09:00 | |
| openstack | The meeting name has been set to 'magnum' | 09:00 |
| brtknr | o/ | 09:00 |
| strigazi | ο/ | 09:00 |
| flwang1 | o/ | 09:00 |
| brtknr | My topics can go last, I’m still 5 mins away from work | 09:01 |
| brtknr | not easy to type on the phone | 09:01 |
| flwang1 | ok | 09:02 |
| flwang1 | #topic Allow updating health_status, health_status_reason https://review.opendev.org/710384 | 09:02 |
| *** openstack changes topic to "Allow updating health_status, health_status_reason https://review.opendev.org/710384 (Meeting topic: magnum)" | 09:02 | |
| flwang1 | strigazi: brtknr: i'd like to propose above change to allow updating the health_status and health_status_reason via the update api | 09:02 |
| strigazi | flwang1: would it make sense to configure who can do this by policy? | 09:03 |
| flwang1 | i'm still doing testing, but i'd like to get your guys comment | 09:03 |
| flwang1 | strigazi: that's a good idea | 09:04 |
| flwang1 | i can do that | 09:04 |
| flwang1 | the context is, all the k8s cluster on our cloud are private, which are not accessible by the magnum control plane | 09:04 |
| brtknr | Would it make sense to make all health updates using this rather than magnum making api calls to k8s end point? | 09:04 |
| flwang1 | so we would like to let the magnum-auto-healer to send api call to update the health status from the cluster inside | 09:05 |
| brtknr | I.e. do we need 2 health monitoring mechanism side by side? | 09:05 |
| strigazi | I think we need to options, not two running together | 09:05 |
| strigazi | s/to/two/ | 09:05 |
| flwang1 | strigazi: +1 | 09:05 |
| flwang1 | brtknr: the two options work for different scenarios | 09:06 |
| flwang1 | if the cluster is a private cluster, then currently we don't have option to update the health status | 09:06 |
| flwang1 | but if it's a public cluster, then magnum can handle it correctly | 09:06 |
| brtknr | But the api would work for both types of clusters | 09:08 |
| flwang1 | brtknr: yes | 09:08 |
| flwang1 | you can disable the magnum server side health monitoring if you want | 09:09 |
| flwang1 | but the problem is, not all vendors will deploy magnum auto healer | 09:10 |
| flwang1 | make sense? | 09:10 |
| brtknr | Ok | 09:11 |
| brtknr | yeah i am happy to have the option, its a nice workaround for private clusters | 09:12 |
| flwang1 | strigazi: except the role control, any other comments? | 09:13 |
| brtknr | Do we have the different roles that magnum expects documented somewhere? | 09:14 |
| flwang1 | for this case or general? | 09:14 |
| brtknr | e.g. only a heat_stack_owner can deploy a cluster for example | 09:14 |
| strigazi | no, as a genetal comment, out-of-tree things should be opt-in. Only kubernetes can not be opt-in | 09:14 |
| strigazi | magnum has it's own policy. We can tune policy-in-code or policy fie | 09:16 |
| strigazi | s/fie/file/ | 09:16 |
| flwang1 | in general, magnum has the policy.json, and you can define any role and update the file based on your need | 09:18 |
| strigazi | +1 | 09:18 |
| flwang1 | shall we move on? | 09:18 |
| strigazi | brtknr: have you arrived? | 09:19 |
| brtknr | yep ive been at my desk for 10 mins :D | 09:19 |
| brtknr | seamless transition | 09:19 |
| flwang1 | #topic Restore deploy_{stdout,stderr,status_code} https://review.opendev.org/#/c/710487/ | 09:19 |
| *** openstack changes topic to "Restore deploy_{stdout,stderr,status_code} https://review.opendev.org/#/c/710487/ (Meeting topic: magnum)" | 09:19 | |
| flwang1 | brtknr: ^ | 09:20 |
| brtknr | Ok basically it was bugging me that deploy_stderr was empty and Rico also pointed this out that this breaks backward compatibility | 09:20 |
| brtknr | threading seems like the only way to handle two input streams simultaneously without weird buffering behaviour you get when you use "select" | 09:21 |
| strigazi | put the same thing as stderr and stdout \o/ | 09:21 |
| strigazi | brtknr: what do you think is the best option? | 09:22 |
| brtknr | i think using threading to write to the same file but capture the two streams separately is the winner for me | 09:22 |
| brtknr | since there might be a genuine deploy_stderr which will always resolve to being empty | 09:23 |
| strigazi | I'm only a little scared that we make a critical component for us more complicated | 09:23 |
| strigazi | The main reason for this is backwards compatibility? | 09:23 |
| brtknr | it works though... threading is not a complicated thing... | 09:24 |
| strigazi | ok if it works | 09:24 |
| flwang1 | i kind of share the same concern as strigazi | 09:25 |
| flwang1 | and personally, i'd like to see we merge all the things back to the heat-agents repo | 09:25 |
| *** mgoddard has joined #openstack-containers | 09:25 | |
| flwang1 | it would be nice if we can share the maintainance of the heat-container-agents | 09:26 |
| brtknr | The parallel change Rico suggested doesnt work as it reads stderr first and then consumes stdout all at once at the end | 09:26 |
| strigazi | +1 ^^ | 09:26 |
| *** udesale_ has joined #openstack-containers | 09:27 | |
| brtknr | I'm happy to share the maintenance burden with the heat team... looks like they have even incorporated some tests | 09:27 |
| strigazi | IMO The best options are: two files and different outputs (as proposed by brtknr intially) or one file and duplicated output in heat | 09:28 |
| flwang1 | so my little tiny comment for this patch is, please collaborate with heat team to make sure we are not far away from the original heat-agents code | 09:28 |
| strigazi | we can try threading if you want, it produces exactly what we need. | 09:29 |
| brtknr | That is my concern, removing deploy_stderr feels like cutting off an arm from the original thing | 09:29 |
| strigazi | flwang1: I think heat follows us in this case | 09:29 |
| *** udesale has quit IRC | 09:30 | |
| flwang1 | strigazi: good | 09:30 |
| flwang1 | i don't really care who follows who, TBH, i just want to see we're synced | 09:30 |
| strigazi | flwang1: they follow == it was low priority for them | 09:31 |
| flwang1 | ok, fair enough | 09:31 |
| strigazi | while for us it is high, we can sync of course. | 09:31 |
| brtknr | please test the threading implementation, its available at brtknr/heat-container-agent:ussuri-dev | 09:32 |
| brtknr | i have tested with both coreos and atomic and it works | 09:32 |
| strigazi | i will, let's move this way then. | 09:33 |
| strigazi | last thing to add | 09:33 |
| strigazi | I was hesitant because the number of bugs user found instead of us was very high in train. | 09:34 |
| strigazi | let's move on | 09:34 |
| flwang1 | strigazi: should we try to enable the k8s functional testing again? | 09:34 |
| strigazi | no | 09:34 |
| flwang1 | or at least put it on our high priority? | 09:34 |
| strigazi | we can't | 09:35 |
| flwang1 | still blocked by the nested virt? | 09:35 |
| strigazi | lack of infra/slow infra | 09:35 |
| flwang1 | CatalystCloud hasn't enable the nested virt yet | 09:36 |
| flwang1 | but we will do in the near future, then we maybe able to contribute the infra for testing | 09:36 |
| flwang1 | CI i mean | 09:36 |
| strigazi | sounds good | 09:36 |
| flwang1 | move on? | 09:37 |
| strigazi | yeap | 09:37 |
| flwang1 | #topic Release 9.3.0 | 09:38 |
| *** openstack changes topic to "Release 9.3.0 (Meeting topic: magnum)" | 09:38 | |
| strigazi | We need some fixes for logging | 09:39 |
| strigazi | disable zincati updates | 09:39 |
| strigazi | and fix cluster resize (we found a corner case) | 09:39 |
| flwang1 | if so, we can hold it a bit? brtknr | 09:40 |
| brtknr | flwang1: yeah no problem | 09:40 |
| brtknr | thats why I wanted to ask you guys if there was anything | 09:40 |
| flwang1 | i appreciate that | 09:41 |
| flwang1 | strigazi: anything else? | 09:41 |
| strigazi | the logging issue is too serious for us. Heavy services break nodes (fill up the disk) | 09:42 |
| strigazi | that's it | 09:42 |
| brtknr | how did you choose a value of 50 million? | 09:42 |
| strigazi | it is strange you haven'e encountered it | 09:42 |
| strigazi | 50μ | 09:42 |
| strigazi | 50m | 09:42 |
| brtknr | Ah 50m | 09:43 |
| strigazi | mega bytes | 09:43 |
| strigazi | I think it is reasonable. Can't explode nodes | 09:43 |
| brtknr | should this be configurable? | 09:43 |
| flwang1 | can you explain more? is it a very frequent issue? | 09:43 |
| strigazi | It is not agressive for reasonable services. I mean the logs will stay there for long | 09:44 |
| strigazi | if a services produces a lot of logs and they are not rotated the disk fills up. | 09:45 |
| strigazi | or creates preassure | 09:45 |
| brtknr | so do you think this option should be configurable? | 09:46 |
| brtknr | with a default value of 50m? | 09:46 |
| brtknr | or is that overkill? | 09:46 |
| strigazi | it fills like an overkill | 09:46 |
| *** pcaruana has quit IRC | 09:46 | |
| strigazi | the nodes are not a proper place to hold a lot of logs | 09:46 |
| strigazi | this is not an opinion :) | 09:47 |
| strigazi | what do you think? | 09:47 |
| flwang1 | the log is for the pod/container, right? | 09:48 |
| strigazi | and of k8s services in podman | 09:48 |
| strigazi | s/of/for/ | 09:48 |
| flwang1 | ok, max 100pod per node, so 50 * 100 =5G, that makes sense for me | 09:48 |
| flwang1 | and i think it's large enough | 09:49 |
| flwang1 | as the local log | 09:49 |
| *** xinliang has joined #openstack-containers | 09:50 | |
| strigazi | move on? | 09:50 |
| flwang1 | #topic https://review.opendev.org/#/c/712154/ Fedora CoreOS Configurarion | 09:50 |
| *** openstack changes topic to "https://review.opendev.org/#/c/712154/ Fedora CoreOS Configurarion (Meeting topic: magnum)" | 09:50 | |
| flwang1 | strigazi: ^ | 09:50 |
| strigazi | I pushed the ignition user_data in a human readable format | 09:51 |
| strigazi | from that format the user_data.json can be generated | 09:52 |
| flwang1 | cool, looks good, i will review it | 09:52 |
| strigazi | when do we take this? | 09:53 |
| strigazi | before my patch of logging or after? | 09:53 |
| brtknr | in https://review.opendev.org/#/c/712154/3/magnum/drivers/k8s_fedora_coreos_v1/templates/fcct-config.yaml line 167, does $CONTAINER_INFRA_PREFIX come from heat or /etc/environment? | 09:53 |
| strigazi | s/of/for/ this is a new pattern for me now | 09:53 |
| flwang1 | is there any depedency b/w them? | 09:53 |
| brtknr | i think we should take this first then the logging | 09:53 |
| brtknr | flwang1: yes, both update fcct-config.yaml | 09:54 |
| strigazi | brtknr: this means both will be backported | 09:54 |
| brtknr | yes | 09:54 |
| strigazi | ok | 09:54 |
| flwang1 | i agree format first | 09:55 |
| flwang1 | which can make your rebase easier, i gues | 09:55 |
| flwang1 | guess | 09:55 |
| brtknr | i can +2 quickly if you can address my comment | 09:55 |
| strigazi | I will | 09:56 |
| brtknr | Should we add a test to make sure that the user-data.json generated from fcct-config.yaml matches the one in the commit tree? | 09:57 |
| brtknr | :P | 09:57 |
| strigazi | let's test first though :) you never know. And give flwang1 a chance to review | 09:57 |
| flwang1 | thanks :) | 09:57 |
| brtknr | strigazi: what is this cluster resize corner case? | 09:59 |
| strigazi | 1. cluster create | 09:59 |
| brtknr | is this after stein->train upgrade? | 09:59 |
| strigazi | 1.1 with an old kube_version (not kube_tag) in kubecluster.yaml | 09:59 |
| strigazi | 2. create a nodegroup | 10:00 |
| strigazi | 3. resize default ng | 10:00 |
| strigazi | causes change of user_data | 10:00 |
| brtknr | yikes | 10:01 |
| brtknr | do you have a fix? | 10:01 |
| brtknr | does this rebuild the whole cluster? | 10:01 |
| brtknr | or just the nodegroups? | 10:01 |
| brtknr | strigazi: | 10:02 |
| flwang1 | hi team, can i close the meeting? | 10:02 |
| brtknr | yes | 10:02 |
| flwang1 | we can discuss this resize issue offline | 10:03 |
| strigazi | we have | 10:03 |
| flwang1 | #endmeeting | 10:03 |
| *** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting" | 10:03 | |
| openstack | Meeting ended Wed Mar 11 10:03:06 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-03-11-09.00.html | 10:03 |
| flwang1 | thanks | 10:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-03-11-09.00.txt | 10:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-03-11-09.00.log.html | 10:03 |
| brtknr | thanks for hosting flwang1 | 10:03 |
| flwang1 | thank you for joining | 10:03 |
| flwang1 | i have to offline, ttyl | 10:03 |
| strigazi | I'm going offline, I will update gerrit and storyboard | 10:04 |
| brtknr | ok speak soon | 10:07 |
| brtknr | strigazi: do you have a fix for a resize issue? | 10:07 |
| strigazi | I will post when finish with logging. | 10:07 |
| brtknr | we should also try to backport the fix for that | 10:09 |
| brtknr | or is it a much bigger change? | 10:10 |
| brtknr | strigazi: | 10:10 |
| -openstackstatus- NOTICE: The mail server for lists.openstack.org is currently not handling emails. The infra team will investigate and fix during US morning. | 10:25 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 10:28 |
| *** markguz_ has joined #openstack-containers | 10:46 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 10:48 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 10:50 |
| openstackgerrit | Thomas Hartland proposed openstack/magnum master: Add node groups documentation https://review.opendev.org/712340 | 10:54 |
| yankcrime | brtknr: have you noticed on fedora atomic that kube-apiserver.service fails more often than not on bootstrap? | 11:00 |
| yankcrime | ExecStartPre=/usr/bin/podman rm kube-apiserver (code=exited, status=1/FAILURE) | 11:00 |
| yankcrime | if you restart that service then it's fine, but at that point cluster creation has already failed | 11:00 |
| brtknr | yes, kube-apiserver is allowed 10 mins to start, i guess the container download is taking longer than that atm | 11:01 |
| brtknr | it works when you create a cluster with 1 worker and 1 master | 11:01 |
| brtknr | any more workers, the bandwidth appears to struggle | 11:01 |
| yankcrime | it's not bandwidth, unless we're being throttled - and i'd be surprised if we're triggering something from just a couple of hosts.... | 11:04 |
| yankcrime | also this cluster creation failed in under 10 minutes | 11:05 |
| *** udesale_ has quit IRC | 11:17 | |
| *** xinliang has quit IRC | 11:17 | |
| *** xinliang has joined #openstack-containers | 11:17 | |
| brtknr | yankcrime: hmm | 11:21 |
| brtknr | yankcrime: how many workers did you try to create? | 11:23 |
| yankcrime | this failed on the master before any workers | 11:23 |
| *** ianychoi has quit IRC | 11:24 | |
| brtknr | Do we know why it fails? | 11:26 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 11:27 |
| *** xinliang has quit IRC | 11:32 | |
| yankcrime | well i see: | 11:32 |
| yankcrime | + ssh -F /srv/magnum/.ssh/config root@localhost systemctl restart kube-apiserver | 11:32 |
| yankcrime | Job for kube-apiserver.service failed because a timeout was exceeded. | 11:32 |
| yankcrime | See "systemctl status kube-apiserver.service" and "journalctl -xe" for details | 11:32 |
| yankcrime | in the heat logs | 11:32 |
| *** xinliang has joined #openstack-containers | 11:32 | |
| yankcrime | kube-apiserver.service is active / running | 11:33 |
| *** xinliang has quit IRC | 11:33 | |
| yankcrime | but one of the execstartpre commands failed | 11:33 |
| yankcrime | Process: 3371 ExecStartPre=/usr/bin/podman rm kube-apiserver (code=exited, status=1/FAILURE) | 11:33 |
| *** xinliang has joined #openstack-containers | 11:34 | |
| *** mkuf_ has joined #openstack-containers | 11:59 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 12:02 |
| *** mkuf has quit IRC | 12:02 | |
| *** udesale has joined #openstack-containers | 12:02 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 12:03 |
| *** pcaruana has joined #openstack-containers | 12:15 | |
| *** mkuf has joined #openstack-containers | 12:29 | |
| *** mkuf_ has quit IRC | 12:32 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 12:35 |
| brtknr | strigazi: just testing your patch gimme 5 mins | 12:50 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 12:55 |
| *** markguz_ has quit IRC | 12:58 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 13:10 |
| *** irclogbot_2 has quit IRC | 13:22 | |
| *** irclogbot_2 has joined #openstack-containers | 13:22 | |
| *** dave-mccowan has joined #openstack-containers | 13:42 | |
| *** dave-mccowan has quit IRC | 13:46 | |
| *** zigo has quit IRC | 13:49 | |
| *** markguz_ has joined #openstack-containers | 13:52 | |
| markguz_ | brtknr: turns out my problems were selinux related. I manually disabled it on the minions and i'm up and running | 13:53 |
| markguz_ | brtknr: is there a way to disable it at cluster creation time? | 13:53 |
| *** dave-mccowan has joined #openstack-containers | 13:56 | |
| *** vishalmanchanda has quit IRC | 14:00 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add fcct config for coreos user_data https://review.opendev.org/712154 | 14:18 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: fcos-podman: Set max size for logging to 50m https://review.opendev.org/712127 | 14:18 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: atomic-podman: Set log imit to 50m https://review.opendev.org/712153 | 14:18 |
| *** xinliang has quit IRC | 14:21 | |
| *** sapd1_x has joined #openstack-containers | 14:25 | |
| *** vishalmanchanda has joined #openstack-containers | 14:41 | |
| *** ykarel is now known as ykarel|away | 15:13 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s-fedora: Set max-size tp 10m for containers https://review.opendev.org/712475 | 15:18 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: fcos: Disable zincati auto-updates https://review.opendev.org/712476 | 15:18 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s-fedora: Set max-size to 10m for containers https://review.opendev.org/712475 | 15:38 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: fcos: Disable zincati auto-updates https://review.opendev.org/712476 | 15:38 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: fcos: Disable zincati auto-updates https://review.opendev.org/712476 | 15:39 |
| *** sapd1_x has quit IRC | 15:51 | |
| *** zigo has joined #openstack-containers | 15:52 | |
| strigazi | brtknr: so | 15:54 |
| strigazi | what is it going to be? | 15:54 |
| strigazi | with the logs issue | 15:54 |
| strigazi | brtknr: this question: why are we passing this here and not as kubelet arg? | 15:55 |
| brtknr | strigazi: what is the problem you're trying to solve? | 15:56 |
| strigazi | brtknr: docker default | 15:57 |
| strigazi | https://docs.docker.com/config/containers/logging/json-file/ | 15:57 |
| brtknr | if the default for kubelet is already 10m and 5 files, why is it necessary to also apply this to docker? | 15:57 |
| strigazi | max-size | 15:57 |
| strigazi | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). | 15:57 |
| strigazi | Defaults to -1 (unlimited). | 15:57 |
| strigazi | unlimited | 15:57 |
| strigazi | please go to this page: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ | 15:57 |
| strigazi | search for container-log-max-size | 15:58 |
| strigazi | the docs say: his flag can only be used with --container-runtime=remote. | 15:58 |
| strigazi | the docs say: This flag can only be used with --container-runtime=remote. | 15:58 |
| strigazi | I can create a reproducer if you want. | 15:58 |
| brtknr | strigazi: sorry if i misunderstanding | 15:58 |
| strigazi | docker is not a repote runtime | 15:59 |
| brtknr | so containerd is remote? | 15:59 |
| strigazi | yes | 15:59 |
| strigazi | https://github.com/openstack/magnum/blob/fa45002e21ef6de3b4a9da35d590a4c5b3d0d7a4/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh#L274 | 16:00 |
| brtknr | just saw the code | 16:00 |
| brtknr | thanks for the clarification | 16:00 |
| strigazi | for completion: | 16:00 |
| strigazi | People asked the same from containerd here: | 16:01 |
| strigazi | https://github.com/containerd/containerd/issues/3351 | 16:01 |
| strigazi | But since the design for the container runtime interface | 16:01 |
| strigazi | The runtime is not resposible for log rotation | 16:01 |
| strigazi | The orchestrator is | 16:01 |
| strigazi | in our case k8s and kubelet | 16:02 |
| strigazi | docker-daemon is the OG runtime that is not remote | 16:02 |
| brtknr | strigazi: thanks | 16:06 |
| brtknr | btw we should do these changes if container_runtimetime is not containerd right? | 16:07 |
| strigazi | these changes are done /etc/sysconfig/docker | 16:11 |
| strigazi | the line above my addition is changing /etc/sysconfig/docker | 16:11 |
| strigazi | we can do and make backport strange | 16:12 |
| strigazi | it si better to do it safely | 16:12 |
| strigazi | I will add a check for the existence of the file | 16:13 |
| strigazi | It does not make sense to check for containerd at the moment. | 16:13 |
| strigazi | I'm working on the reproducer.. | 16:14 |
| *** flwang has quit IRC | 16:17 | |
| *** ricolin has quit IRC | 16:25 | |
| *** udesale has quit IRC | 16:41 | |
| brtknr | strigazi: gotcha, makes sense | 16:46 |
| strigazi | I left a reply in gerrit | 16:54 |
| strigazi | how to reproduce | 16:55 |
| strigazi | and what to check | 16:55 |
| strigazi | I'm testing now with containerd to post in the patch | 16:55 |
| *** markguz_ has quit IRC | 16:58 | |
| strigazi | brtknr: ^^ | 16:59 |
| brtknr | thanks will check | 17:00 |
| brtknr | but maybe tomorrow as I need to head home now | 17:00 |
| *** ricolin has joined #openstack-containers | 17:01 | |
| brtknr | strigazi: does podman need to be configured in a similar way? | 17:43 |
| *** vishalmanchanda has quit IRC | 18:00 | |
| *** markguz_ has joined #openstack-containers | 18:33 | |
| markguz_ | hey. so is there a label or a setting that allows you disable selinux when spinning up fedora-core based k8s clusters? | 18:33 |
| markguz_ | selinux blocks mounting cinder volumes correctly | 18:34 |
| flwang1 | markguz_: could you please explain more how the selinux blocks the cinder volume mounting? | 18:58 |
| markguz_ | flwang1: so when selinux is set to enforcing in the minion nodes the volumes do not mount correctly to the containers or at all | 18:59 |
| markguz_ | with selinux set to permissive, everything works as expected | 19:00 |
| flwang1 | markguz_: could you please create a story on story board to track this? i don't think we have a label for selinux now https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/disable-selinux.sh | 19:01 |
| markguz_ | the volumes attach ok, i either got a "timeout waiting for volume to attach or mount" or sometimes it did mount but i could not run "chmod" on the mount path to allow non root to write to it | 19:01 |
| flwang1 | so it would be nice if we can figure out what's the root cause, since enable the selinux is a preferable way for security reasons, does that make sense? | 19:02 |
| markguz_ | i also had a problem with nfs mounts, where i was getting "systemctl not found" when the the minion was trying to start statd to mount the nfs. | 19:03 |
| markguz_ | however if i ssh'd in and ran start-statd directly it worked... path seems correct in the start-statd script | 19:04 |
| markguz_ | flwang1: what's the url for the story board? | 19:04 |
| flwang1 | markguz_: https://storyboard.openstack.org/#!/dashboard/stories | 19:05 |
| flwang1 | select Magnum as the project when creating | 19:05 |
| flwang1 | markguz_: are you using stable/train? | 19:05 |
| markguz_ | flwang1: no master | 19:07 |
| flwang1 | ok | 19:07 |
| *** flwang1 has quit IRC | 19:12 | |
| *** iokiwi has quit IRC | 19:20 | |
| *** rcernin has joined #openstack-containers | 21:28 | |
| *** pcaruana has quit IRC | 21:36 | |
| *** dave-mccowan has quit IRC | 22:05 | |
| *** dave-mccowan has joined #openstack-containers | 22:09 | |
| *** tkaprol has joined #openstack-containers | 22:17 | |
| *** tkaprol has quit IRC | 22:22 | |
| *** jrosser has quit IRC | 22:31 | |
| *** andrein has quit IRC | 22:31 | |
| *** jrosser has joined #openstack-containers | 22:36 | |
| *** andrein has joined #openstack-containers | 22:36 | |
| *** guilhermesp has quit IRC | 22:36 | |
| *** mnaser has quit IRC | 22:36 | |
| *** mnaser has joined #openstack-containers | 22:39 | |
| *** guilhermesp has joined #openstack-containers | 22:41 | |
| *** irclogbot_2 has quit IRC | 22:47 | |
| *** markguz_ has quit IRC | 22:47 | |
| *** irclogbot_1 has joined #openstack-containers | 22:47 | |
| *** openstackstatus has quit IRC | 22:48 | |
| *** mnasiadka has quit IRC | 22:50 | |
| *** mnasiadka has joined #openstack-containers | 22:54 | |
| *** lxkong has quit IRC | 22:56 | |
| *** lxkong has joined #openstack-containers | 23:02 | |
| *** ianychoi has joined #openstack-containers | 23:06 | |
| *** dave-mccowan has quit IRC | 23:08 | |
| *** dave-mccowan has joined #openstack-containers | 23:10 | |
| *** KeithMnemonic has quit IRC | 23:51 | |
| *** KeithMnemonic has joined #openstack-containers | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!