Thursday, 2019-10-10

*** goldyfruit has joined #openstack-containers00:43
*** hongbin has joined #openstack-containers00:52
*** threestrands has joined #openstack-containers02:13
*** ricolin has joined #openstack-containers02:52
*** ramishra has joined #openstack-containers02:53
*** hongbin has quit IRC03:23
*** goldyfruit has quit IRC03:48
*** hongbin has joined #openstack-containers03:49
*** udesale has joined #openstack-containers04:00
*** ramishra has quit IRC04:00
*** ykarel has joined #openstack-containers04:13
*** ramishra has joined #openstack-containers04:17
*** hongbin has quit IRC04:41
*** dave-mccowan has quit IRC04:51
*** pcaruana has joined #openstack-containers05:07
*** pcaruana has quit IRC05:35
*** threestrands has quit IRC06:36
*** threestrands has joined #openstack-containers06:36
*** threestrands has quit IRC06:41
*** pcaruana has joined #openstack-containers06:51
*** lsimngar_ has joined #openstack-containers06:57
*** ykarel is now known as ykarel|lunch07:04
*** rcernin has quit IRC07:07
*** ricolin has quit IRC07:10
*** FlorianFa has quit IRC07:24
*** Florian has joined #openstack-containers07:25
*** ttsiouts has joined #openstack-containers07:45
strigaziflwang brtknr, guys take it easy. don't judge things based on names07:55
strigazispc_t means that selinux won't stop the process that has this context from mounting dir and so on07:56
strigazifor flannel we drop all CAPS apart from NET_ADMIN07:56
strigazicalico runs with privileged: true07:56
strigaziso feilong, flannel is more locked down than calico07:57
strigazithe calico devs didn't bother to create a PSP07:57
strigazibut the super privileged permission only for the flannel container, right? This question is very wrong, it implies that flannel is not configured securely. But no, it is the other way round.07:58
*** trident has quit IRC07:58
strigazialso, this discussion is a little pointless because calico and flannel manage the network of everything in the cluster. That is their job. They are privileged anyway.07:59
*** trident has joined #openstack-containers08:01
*** ttsiouts has quit IRC08:04
*** ttsiouts has joined #openstack-containers08:05
*** flwang1 has joined #openstack-containers08:15
flwang1strigazi: around?08:15
strigaziI'm here08:16
strigaziflwang1: I'm here08:17
flwang1strigazi: i can reproduce the issue of heat-container-agent08:18
strigaziok08:18
flwang1the /etc/os-collect-config looks correct08:18
flwang1but the /var/lib/heat-config/heat-config is empty08:19
flwang1seems the /opt/stack/os-config-refresh/configure.d/20-os-apply-config is not working correctly08:19
strigazireally? I could find anything08:20
strigazireally? I could not find anything08:20
strigaziwhat is the issue?08:20
flwang1i don't know yet, but i think i'm closing to the answer08:20
strigaziand the path is wrong right?08:21
strigazilocal-data needs to be a dir, correct?08:21
flwang1if i can't fix it this week, i will propose fix in heat to use the /var/lib/heat-cfntools/08:21
flwang1strigazi: yes, it needs to be dir08:21
strigaziflwang1: at the moment, in our heat deployment /var/lib/cloud is passed by heat08:22
strigazican you check in yours?08:22
flwang1i had a chat with stevebaker, he said it maybe a bug in the local collector08:22
flwang1both /var/lib/heat-cfntools/ and  /var/lib/cloud should work, i tested before08:23
flwang1if i can't find the fix based on local-data, i will roll back to above 2 paths08:23
strigaziI mean, what is being used at Catalyst08:23
brtknro/ hey both, im taking it easy :)08:24
brtknrjust had no idea how selinux works... been reading up on it...08:24
strigazibrtknr: in your heat deployment08:26
flwang1strigazi: i think we're also using /var/lib/cloud, i need to check08:26
strigazibrtknr: which file is being used by the heat agent?08:26
strigaziflwang1: We are big heat users. The development in heat is a bit stale. Why change the current working behaviour?08:27
brtknrstrigazi: do I check heat-container-agent systemd unit?08:27
strigaziflwang1: the arguments about obsolete and deprecated things mean nothing in real life.08:27
flwang1strigazi: we can support both as I proposed in patchset 508:27
strigaziflwang1: that is fine08:28
flwang1as you saw, heat team was suggesting use that local-data, and i thought it's same thing, and i didn't do the final test :(08:28
flwang1that's my fault08:29
strigazibrtknr: check /var/lib/cloud/data/cfn-init-data08:29
brtknrsthttps://seashells.io/p/AabXfFDG08:29
strigaziflwang1: mine too, it is not your fault only. mostly it is heat teams that pushed as for no real reason08:29
brtknrhttps://seashells.io/p/AabXfFDG08:29
strigazibrtknr:08:30
strigazibrtknr: and /var/lib/heat-cfntools/cfn-init-data08:30
brtknrthats my /var/lib/cloud/data/cfn-init-data08:30
brtknrstrigazi: that doesnt exist08:30
strigaziflwang1: so brtknr uses /var/lib/cloud/data/cfn-init-data as well08:31
brtknrstrigazi: is that good? i am not sure what this discussion is abou08:33
strigazibrtknr: yes you using what we use. No, cause heat team wants us to change for "deprecated" reasons.08:34
strigazibrtknr: yes you are using what we use. No, because the heat team wants us to change for "deprecated" reasons.08:34
flwang1strigazi: ok, then how about we support all the 3 directories?08:35
flwang1to get a better backward compatiblity08:36
strigaziflwang1: if it doesn't break things, sgtm08:37
flwang1i don't think it breaks things08:37
flwang1in my devstack env, i can see cfn-init-data at both  /var/lib/heat-cfntools and /var/lib/cloud/data/08:38
flwang1hence why i propose the initial version to have the file for both08:38
flwang1proposed08:39
strigaziwhich makes sense.08:39
brtknrstrigazi: wohoo08:43
brtknri managed to get flannel to run without spc_t08:44
strigazihow?08:44
strigazichcon /run/flannel ?08:44
strigazibrtknr: ?08:44
strigazibrtknr: ?08:45
brtknrsee ^08:45
openstackgerritBharat Kunwar proposed openstack/magnum master: [WIP] Support Fedora CoreOS 30  https://review.opendev.org/67845808:45
brtknrsee ^08:45
brtknrcontainer_runtime_t08:46
strigazican you provide more details on what is the impact of this change08:47
brtknrit allows /run/flannel/subnet.env to be created without permission denied08:48
strigazithat is the goal, we know that08:48
strigaziI try to understand why Jason from RedHat proposed spc instead of container_runtime08:49
strigazialso what this means? "Be aware that this probably disables all SELinux protection for this container."08:51
brtknrstrigazi: where/when did he propose spc instead of container_runtime?08:52
brtknrstrigazi: https://danwalsh.livejournal.com/78312.html < there's also this blog08:52
*** namrata has joined #openstack-containers08:52
strigazihttps://pagure.io/atomic/kubernetes-sig/issue/3 which references https://danwalsh.livejournal.com/74754.html which references https://developers.redhat.com/blog/2014/11/06/introducing-a-super-privileged-container-concept/08:56
strigaziI honestly don't know what is better.08:56
brtknrstrigazi: i suppose flannel is not a "container runtime"08:59
flwang1strigazi: i think i'm very close to the answer now09:00
brtknrbut rather in fact a service running inside a container container09:00
brtknrwhich requires extra privileges09:00
brtknrso maybe spc_t does make more sense09:00
flwang1https://github.com/openstack/os-apply-config/blob/master/os_apply_config/apply_config.py#L7309:01
flwang1this line can't get the deployments when using local-data09:01
strigaziflwang1: I don't get it, why?09:02
flwang1http://paste.openstack.org/show/782635/09:05
flwang1i don't fully understand yet, but seems the data is overwriten09:06
flwang1https://github.com/openstack/os-apply-config/blob/master/os_apply_config/collect_config.py#L7009:07
flwang1i'm trying to add breakpoints here09:07
flwang1something wrong when merge the configs09:10
flwang1https://github.com/openstack/os-apply-config/blob/master/os_apply_config/collect_config.py#L5709:11
flwang1i'm off now, will dig it tomorrow09:13
strigaziflwang1: gn09:13
namrataHi Folks, I have deployed magnum based kubernestes cluster and when I try to access the logs I see this error. has anybody saw this issue and how to solve thisfailed to open log file "/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log": open09:16
namrata/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log: no such file or directory09:16
namrataHi Folks, I have deployed magnum based kubernestes cluster and when I try to access the logs I see this error. has anybody saw this issue and how to solve this`failed to open log file "/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log": open09:16
namrata/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log: no such file or directory`09:16
namrataHi Folks, I have deployed magnum based kubernestes cluster and when I try to access the logs I see this error.`failed to open log file "/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log": open09:17
namrata/var/log/pods/gitlab_gitlab-runner-5f7b586968-sglfb_3b76acb4-dac5-11e9-8744-fa163e1fa925/gitlab-runner/1.log: no such file or directory`. has anybody saw this issue and how to solve this09:17
brtknrflwang1: sleep tight!09:18
*** Florian has quit IRC09:19
*** FlorianFa has joined #openstack-containers09:19
*** namrata89 has joined #openstack-containers09:20
*** namrata89 has left #openstack-containers09:20
*** namrata26 has joined #openstack-containers09:21
*** namrata has quit IRC09:23
*** namrata26 has quit IRC09:24
*** namrata has joined #openstack-containers09:25
*** ykarel|lunch is now known as ykarel09:26
*** ricolin has joined #openstack-containers09:30
*** ttsiouts has quit IRC09:31
*** ttsiouts has joined #openstack-containers09:31
*** ttsiouts_ has joined #openstack-containers09:32
*** ttsiouts has quit IRC09:32
*** udesale has quit IRC09:52
*** udesale has joined #openstack-containers09:53
*** ykarel is now known as ykarel|afk09:54
brtknrstrigazi: what about this:  chcon -t container_file_t -u system_u /var/run/flannel09:58
brtknrthis way, we dont need to specify any security context for the container since it starts with container_t context09:59
*** udesale has quit IRC10:00
*** udesale has joined #openstack-containers10:01
openstackgerritBharat Kunwar proposed openstack/magnum master: [WIP] Support Fedora CoreOS 30  https://review.opendev.org/67845810:04
brtknrstrigazi: as above^10:04
brtknrit is the best of both worlds because its no longer a super privileged container but can write to the path it needs to write to10:05
*** ttsiouts_ has quit IRC10:17
*** ttsiouts has joined #openstack-containers10:18
*** ttsiouts has quit IRC10:22
openstackgerritBharat Kunwar proposed openstack/magnum master: [WIP] Support Fedora CoreOS 30  https://review.opendev.org/67845810:32
*** sapd1 has joined #openstack-containers10:48
*** ttsiouts has joined #openstack-containers11:04
*** udesale has quit IRC11:09
*** sapd1 has quit IRC11:19
strigazibrtknr: I have done this already but we need to make sure the context is kept on reboot.11:25
strigazibrtknr: we use k8s to not have to do this things11:26
strigazibrtknr: /var/run get deleted on reboot.11:26
*** jhesketh has quit IRC11:29
strigazinext time I'll post all alternatives I tried.11:31
*** ykarel|afk is now known as ykarel11:32
lsimngar_hi all, I do not know if here is the best place to ask about this, I think that I found an issue with the flannel driver deploying k8s clusters in magnum..11:38
lsimngar_everything is in place except flannel setup and I get a timeout error from heat11:38
*** jhesketh has joined #openstack-containers11:43
strigazilsimngar_ we can't help with info you provided so far. Please give more details if you can.11:58
*** goldyfruit has joined #openstack-containers12:03
lsimngar_strigazi: ok I have deployed a new k8s cluster with stein magnum everything seems to be fine. I'm using flannel network driver, but from the VM I only see eth0 IP and docker012:10
*** ttsiouts has quit IRC12:10
lsimngar_# cat /run/flannel/subnet.env12:11
lsimngar_cat: /run/flannel/subnet.env: No such file or directory12:11
*** ttsiouts has joined #openstack-containers12:11
lsimngar_# ifconfig flannel012:11
lsimngar_flannel0: error fetching interface information: Device not found12:11
lsimngar_# sudo service flanneld status12:12
lsimngar_Redirecting to /bin/systemctl status flanneld.service12:12
lsimngar_Unit flanneld.service could not be found.12:12
*** ttsiouts has quit IRC12:13
lsimngar_It looks like everything is there but flannel is not set at all. not sure if I'm doing something stupid or if something is missing12:13
*** ttsiouts_ has joined #openstack-containers12:13
lsimngar_# cat /etc/sysconfig/flannel-network.json12:13
lsimngar_cat: /etc/sysconfig/flannel-network.json: No such file or directory12:13
*** goldyfruit has quit IRC12:14
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: ng-13: Support nodegroup upgrade  https://review.opendev.org/68673312:14
lsimngar_on the other hand kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet etcd and docker are up and running12:15
lsimngar_I'm using fedora-atomic-29 btw12:17
strigaziflannel_backend=vxlan ?12:23
strigazimagnnum version?12:23
lsimngar_strigazi: 'flannel_backend': 'host-gw'  and magnum 8.0.0-1.el712:25
lsimngar_strigazi: I didn't try with vxlan but I got the same results with the default upd12:27
strigazilsimngar_: kubectl -n kube-system get po | grep flannel && kubectl -n kube-system  logs ds/kube-flannel-ds-amd6412:30
lsimngar_strigazi: after a while I got "No resources found."12:33
lsimngar_# kubectl get nodes12:34
lsimngar_NAME                                STATUS    ROLES     AGE       VERSION12:34
lsimngar_k8s-cluster-t4jvfntfdes2-master-0   Ready     master    22h       v1.11.612:34
lsimngar_k8s-cluster-t4jvfntfdes2-minion-0   Ready     <none>    22h       v1.11.612:34
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: Docker volume size from nodegroups  https://review.opendev.org/68787912:41
*** namrata has quit IRC12:43
*** udesale has joined #openstack-containers12:55
brtknrstrigazi: lets go with spc_t, seems like the best approach13:02
brtknrstrigazi: but i'd be interested to hear what else you tried13:10
*** trident has quit IRC13:14
*** trident has joined #openstack-containers13:15
*** goldyfruit has joined #openstack-containers13:23
*** goldyfruit has quit IRC13:35
*** goldyfruit has joined #openstack-containers13:37
*** munimeha1 has joined #openstack-containers13:38
*** dave-mccowan has joined #openstack-containers13:41
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: ng-10: Fix cluster template conditions  https://review.opendev.org/68562013:49
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: ng-11: API microversion 1.9  https://review.opendev.org/68608913:49
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: ng-12: Label nodegroup nodes  https://review.opendev.org/68636213:49
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: ng-13: Support nodegroup upgrade  https://review.opendev.org/68673313:49
*** spsurya has joined #openstack-containers13:57
brtknrttsiouts_: nice work!14:14
brtknrttsiouts_: excited to try it! taking it for a spin now14:18
ttsiouts_brtknr: cool! thanks for testing!!14:19
*** iokiwi has quit IRC14:29
*** iokiwi has joined #openstack-containers14:31
openstackgerritSpyros Trigazis proposed openstack/magnum master: [WIP] Support Fedora CoreOS 30  https://review.opendev.org/67845814:36
*** pcaruana has quit IRC14:39
*** dave-mccowan has quit IRC14:41
brtknrttsiouts_: have you tested nodegroups with fedora coreos?14:43
brtknrttsiouts_: or only atomic?14:44
*** jmlowe has quit IRC14:45
ttsiouts_brtknr: only atomic till now..14:56
brtknrttsiouts_: I went ahead of myself and started testing on coreos :P14:56
brtknrunfortunately it failed but its an interesting failure14:57
*** namrata has joined #openstack-containers14:57
ttsiouts_brtknr: can you paste it?14:58
ttsiouts_s/paste/share14:58
brtknr<urlopen error [Errno 2] No such file or directory: '/opt/stack/magnum/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml'>14:59
namrataHi Folks,I would like to know is there any monitoring system available for magnum based kubernetes cluster which will notify when a defined specific alert is triggered. Recently I encountered an issue where kubernetes garabage collector was not working hence the question14:59
brtknrand its true, it doesnt exist14:59
brtknrnamrata: have you tried monitoring_enabled=True,tiller_enabled=True labels?15:00
brtknrit deploys helm based prometheus monitoring15:00
*** jmlowe has joined #openstack-containers15:01
*** pcaruana has joined #openstack-containers15:01
brtknrttsiouts_: looks like your nodegroup changes breaks the reuse of fedora atomic heat templates...15:02
ttsiouts_brtknr: did you rebase on top of ng-13?15:04
brtknrno i checkout out ng-1315:04
namratabrtknr I will check that. Thanks!15:04
brtknrim using SHA: 342449f7011334ef747c0391b63c67d5001c4d5615:04
ttsiouts_brtknr: let me check.15:05
ttsiouts_brtknr: broke on creation?15:06
namratahas anybody has seen the following issue and how to solve this `failed to open log file "/var/log/pods/gitlab_gitlab-runner-5f7b586968-stcfm_63a9c213-eb51-11e9-8744-fa163e1fa925/gitlab-runner/0.log": open /var/log/pods/gitlab_gitlab-runner-5f7b586968-stcfm_63a9c213-eb51-11e9-8744-fa163e1fa925/gitlab-runner/0.log: no such file or directory`15:06
ttsiouts_brtknr: but strigazi's change is not merged yet.15:07
namratawhen I try to see kubectl logs15:08
ttsiouts_brtknr: how do you test it with ng-13?15:08
brtknrttsiouts_: ouch my bad15:10
brtknrlol15:10
ttsiouts_brtknr: lol15:11
brtknrttsiouts_: long day...15:11
*** ykarel is now known as ykarel|afk15:11
brtknrttsiouts_: are there any problems with supplying cluster template as an arg when creating nodegroups?15:13
brtknrttsiouts_: Invalid Cluster Template for upgrade: Nodegroup bharat can be upgraded only to match cluster's template (k8s-calico-fedora-atomic-v1 (HTTP 409) (Request-ID: req-078ecfe9-fc18-486a-b87c-aea8e93034b6)15:18
brtknrI dont understand what this error means15:18
*** lsimngar_ has quit IRC15:18
brtknrttsiouts_: this is what I am trying to run: openstack coe cluster upgrade --nodegroup bharat k8s-calico-fedora-atomic d36cd536-f14b-4e86-bee8-7d85cd4106f615:21
ttsiouts_brtknr: you can only upgrade your nodegroup to match the cluster template that is set in your cluster15:21
ttsiouts_brtknr: so I guess that k8s-calico-fedora-atomic-v1  is the cluster template that the cluster uses15:22
ttsiouts_brtknr: In the meantime I checked coreos15:22
brtknrttsiouts_: no my cluster template is called k8s-calico-fedora-atomic-v1.16.115:23
brtknrttsiouts_: does coreos work?15:24
ttsiouts_brtknr: I have to port ng-10 to the new coreos templates15:24
ttsiouts_brtknr: it should work after that.15:24
brtknrttsiouts_: sounds good15:24
brtknrit is dropping the ".16.1" part from the template some for reason15:24
brtknram I not support to have "." in cluster template name?15:25
ttsiouts_brtknr: you should be able to.15:25
ttsiouts_brtknr: https://review.opendev.org/#/c/686733/3/releasenotes/notes/upgrade_api-1fecc206e5b0ef99.yaml15:25
ttsiouts_brtknr: I tried to explain here some things for upgrade15:26
*** jmlowe has quit IRC15:26
*** pcaruana has quit IRC15:26
ttsiouts_brtknr: I have to run. I will continue tomorrow with coreos15:26
ttsiouts_brtknr: see you tomorrow15:27
*** ttsiouts_ has quit IRC15:27
*** ttsiouts has joined #openstack-containers15:28
brtknrttsiouts_: ttsiouts oh  i see... i was wondering why k8s-calico-fedora-at-bharat-4e3nocuw3fu2-node-0 and not k8s-calico-fedora-atomic -bharat-4e3nocuw3fu2-node-015:32
brtknrlike it was before15:32
*** ttsiouts has quit IRC15:32
*** dave-mccowan has joined #openstack-containers15:41
*** ykarel|afk is now known as ykarel15:41
*** udesale has quit IRC16:19
*** jmlowe has joined #openstack-containers16:35
*** pcaruana has joined #openstack-containers16:40
namratabrtknr any reference link to access the prometheus monitoring system16:47
*** primeministerp has joined #openstack-containers16:58
*** namrata has quit IRC17:05
*** ykarel is now known as ykarel|away17:11
*** ykarel|away has quit IRC17:35
*** ricolin has quit IRC17:39
*** ramishra has quit IRC17:51
*** jmlowe has quit IRC18:22
*** ykarel|away has joined #openstack-containers18:23
primeministerphey all, was wondering if I can get some insight.  I have a aio with magnum deployed with kolla-ansible on aarch64.18:31
primeministerpI'm seeing the following: http://paste.openstack.org/show/782736/18:32
primeministerpWhen deploying a fedora-atomic cluster18:32
*** ykarel|away has quit IRC19:00
*** jmlowe has joined #openstack-containers19:01
*** bline has quit IRC19:33
*** pcaruana has quit IRC20:15
*** spsurya has quit IRC20:28
*** FlorianFa has quit IRC20:49
*** FlorianFa has joined #openstack-containers21:01
*** FlorianFa has quit IRC21:08
*** FlorianFa has joined #openstack-containers21:21
brtknrnamrata: https://lmgtfy.com/?q=monitoring_enabled+magnum+openstack&p=121:38
*** ryn_eq has quit IRC21:44
*** ryn_eq has joined #openstack-containers21:55
*** trident has quit IRC22:03
*** trident has joined #openstack-containers22:05
*** munimeha1 has quit IRC22:26
*** rcernin has joined #openstack-containers22:46
brtknrprimeministerp: which version of magnum? what parameters did you use to create the cluster? so many unknowns...23:22
*** vesper11 has quit IRC23:30
*** vesper11 has joined #openstack-containers23:31
*** goldyfruit has quit IRC23:31

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!