Tuesday, 2019-01-08

*** tobiash has joined #opendev07:51
*** clarkb has joined #opendev20:01
*** ianw has joined #opendev20:01
fungidid the announcement say we were going to have text discussion in here in parallel with the conference call?20:01
corvusi think i mentioned it in the second email20:01
fungithanks, that's what i thought i recalled reading20:02
corvusi don't expect to be able to type a lot, but it might be nice to share links, etc, and have an out-of-band channel20:02
clarkbhrm I said I had it working on android but now I don't20:03
clarkbcomputers how do they work20:03
*** jhesketh has joined #opendev20:03
ianwclarkb: did you just "dial" 6561@pbx.openstack.org" ?\20:04
*** mordred has joined #opendev20:04
fungiclarkb: i use the mute button on my phone to silence my loud keyboard, but i guess that won't help for corvus20:05
clarkbianw: I told mizudroid to connect to pbx.openstack.org as server, username is 'conference', and password is foo20:05
clarkbianw: then it fails and in the phone pad uri thing that comes up I typed in sip:conference@pbx.openstack.org20:05
clarkband followed the prompt to join 656120:06
ianwok i can hear something about renovations20:06
*** Shrews has joined #opendev20:07
ianwok, i've muted to avoid background noise cool20:07
clarkbI'm ok with audio recording20:07
fungiglad this didn't end up being tomorrow. my roof is getting replaced starting at 8am local20:08
corvuswe're recording the audio, and we'll be recording the terminal session (starting shortly) too20:09
fungiit's like corvus is shaking a can of gravel ;)20:09
clarkbnot too loud20:09
ianwhmm, sounds like a 1987 era model M ... light grey, ps/220:10
corvustelnet bridge.openstack.org 888820:10
fungii see a terminal session!20:10
fungiianw: i have that one20:10
jheskethit was very slow for me, but I'm there20:11
clarkbI see the session20:11
jhesketh(I'm assuming my mute is working and nobody can hear me sneezing? :-p)20:12
Shrewsjhesketh: didn't hear you20:13
fungias if anyone would accidentally expose an important password and then have to change it quickly ;)20:13
ianwso when you ssh in, you're actually in a different container to gitea, but with views of most of the shared volumes?20:31
fungithis is the "special" ssh service gitea needs to be able to support git+ssh protocol, right?20:32
corvusfungi: yep20:34
fungijust making sure "ssh in a container" wasn't generally going to be our management pattern for other systems or something20:35
corvusyeah, neither of those containers runs a regular ssh service; the only way into them with a shell is via k8s20:35
ianwahh, right that makes sense20:35
fungipresumably there is an sshd also running in the primary non-container context20:35
corvusfungi, ianw: yes20:36
corvusi think we can demo some things to illustrate this in a sec20:36
fungidid monty suddenly explode?20:37
fungioh, he's still there20:37
jheskethwhy are we using cephfs instead of a cinder volume?20:51
jheskethokay, I follow why we want the shared storage, but it was mentioned earlier that we can attach cinder volumes to a container so I'm unsure why that wouldn't work?20:53
clarkbbefore I forget, on the topic of sorting out logs things the OSH team's test jobs would restart failed conatiners/pods over and over and over and not record the logs from the failed instances. We should see if we can have the log rotation store instances of failed pod/container logs longer than the other stuff20:54
jheskethah okay, I appear to have made a bad assumption that you can attach the same cinder volume to multiple pods/containers20:55
jheskeththanks for the answers :-)20:55
fungicinder multi-attach is only half the solution if you want multiple systems writing20:55
fungiusually that's for multiple readers and then you have a protocol to pass around the "write token" so systems know when it's their turn to flush20:56
* jhesketh nods20:58
clarkbquestions/thoughts as we go through this. Can we restrict access to the k8s api to say just the bridge? is that desireable? If we delete the master node will the minions keep running as is or is that an outage?21:04
fungialso reminds me that i want to understand enough of this to know whether things like https://groups.google.com/forum/#!topic/kubernetes-dev/P0ghX_DViy8 are something we need to worry about21:07
clarkbfungi: potentially access to the db via that method would be bad21:08
corvusback at 21:26 utc21:21
clarkbfor when we get back one of the things that interests me is how our config management through code review would work with such a system. k8s gives you the flexibility to do adhoc edits to your configs or you can reapply templates and it does fancy diffs to figure out what the equivalent adhoc patch to apply is. I kind of figure we'll rely on applying entire yaml templates which can be automated from code21:23
clarkbreview, but then we do also need some way to reconcile or audit any out of band adhoc edits (so that we can capture them or remove them if necessary?) that might be looking way far down the road though21:23
corvusi think we can talk about that as part of the next thing, since i have ansible written already to do the gitea deployment21:24
mordredclarkb: yes - for the most part our interaction has been in teh form of applying entire yaml templates ... and we have ansible to do that for us as well21:24
mordredcorvus: I believe we were typing at the same time :)21:24
corvusand said the ~same thing :)21:25
mordredclarkb: I'd also think that we'd want to consider out-of-band ad-hoc edits in the same way as manual changes to servers at the moment - occasionally necessary, but not a general daily practice and one that needs to be resolved once the need has passed21:26
mordredincidentally, SpamapS made a tool while at godaddy that he uses to get a diff between a yaml he's about to apply and the existing state of the server for previewing things21:27
corvuswe're back!21:27
clarkbmordred: ++ the difference is puppet or ansible will often undo what I did adhoc21:27
mordredI have not yet used it myself, but he says it has been very helful21:27
clarkbwhere as k8s will preserve them aiui through fancy 3 way diffing21:27
mordredclarkb: yeah - well, so will kubectl potentially :)21:27
fungiyeah, back just don't feel like taking my phone off mute ;)21:27
clarkbIm in fungi's boat21:27
mordred./kubectl api-resources --verbs=list --namespaced -o name | xargs -n1 ./kubectl get --show-kind --ignore-not-found -n rook-ceph21:28
fungialso, agree with mordred's reply above about treating manual configuration modification the same way as we do now: avoid them, and communicate when you can't21:29
fungii don't think we can prevent them any more than we can today, and trying to accommodate them seems like a lot of work just to prop up incidental needs21:30
clarkbthe difference is that k8s preserves them forward through the future21:33
clarkbit doesn't overwrite unless you later modify the specific attribute that was changed before21:33
clarkbaiui (it does fancy 3 way diffs and revision control of yaml to do that)21:33
fungiclarkb: got it, so we may want to reapply a complete manifest of configuration every time?21:34
clarkbmaybe? not sure if that is an option.21:34
fungiyeah, the broadcast worked fine on my terminal21:40
clarkbmy clietn seems to have dropepd me from the call21:40
clarkbI'm back21:40
ianwclarkb: that android client ... i seem to have to keep touching it to keep it in the foreground i think21:40
fungithat's one needy client21:41
clarkbianw: ya I'm going to have to find another one after this21:42
clarkbbut this is the first one I've found that will do direct connections like this21:42
ianwcovus: so that inventory file ... the master / minion servers in there you've created?21:42
ianwoh, ok thanks i see now.  sorry was looking at inventory up one level and getting confused21:44
jheskethI've got to drop off unfortunately. I'll catch the rest in the recording. Thanks so much for the demo!22:16
fungiyes, this has been great, thanks again! unfortunately i have to go pick christine up from work now but please do post the link to the audio recording once it's up somewhere22:18
mordredclarkb: https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures22:51
ianwthanks for this overview, been very helpful to understand what's going on22:52
clarkbmordred: reading that it seems like the logs should go into journalctl and then whatever rotation we have for that would be used?23:01
clarkbI think that means we should have everything go to syslog on ubuntu xenial as long as rsyslog is installed too (but kubelet likely won't find the logs tehre if you run kubectl log)23:01
clarkbcorvus: ^ I think by default journactl on ubuntu runs with a ring buffer because it doesn't save to disk. We can change that and maybe make things better logging wise23:02
corvusi think that explains/matches the behavior i've seen23:02
mordredclarkb: https://github.com/infraly/k8s-on-openstack/blob/266f9b08cfab106e67d571adb1d8c59aaa73ce11/roles/kubeadm/tasks/main.yaml#L6523:07
mordredsorry - the one above that23:07
mordredso the current deploy configures the docker to log to journald23:08
clarkbperfect then ya I think we just make jrounald persist its data with some rotation23:08
clarkbwhich is a config flag in /etc/journad.conf or something23:08
mordredclarkb: ++23:14

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!