Saturday, 2020-06-06

*** mlavalle has quit IRC00:00
*** xiaolin has joined #opendev01:35
*** ysandeep|off is now known as ysandeep03:36
*** xiaolin has quit IRC03:37
*** ravsingh has joined #opendev04:58
*** ravsingh has quit IRC06:22
*** xiaolin has joined #opendev06:48
openstackgerritMerged opendev/ master: Revert "Add three more jvbs to DNS"
*** xiaolin has quit IRC07:11
*** avass has quit IRC07:45
*** sgw has quit IRC07:56
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:02
*** xiaolin has joined #opendev08:09
*** slaweq has joined #opendev08:46
*** DSpider has joined #opendev09:21
*** xiaolin has quit IRC09:28
*** elod has quit IRC09:38
*** tosky has joined #opendev09:49
*** slaweq has quit IRC09:53
*** slaweq has joined #opendev10:03
*** slaweq has quit IRC10:35
*** elod has joined #opendev11:14
*** tosky has quit IRC11:18
*** tosky has joined #opendev12:17
*** tosky has quit IRC12:17
*** tosky has joined #opendev12:17
mordredclarkb, corvus: I found myself down a little rabbit hole this morning and read this: as well as found the "no_new_keyring" option in config.json of the OCI spec - which makes me think maybe there's something in there WRT AFS in bubblewrap in a container - like maybe we need to add a seccomp profile that grants the container the ability to create a session12:28
mordredclarkb, corvus: there's also mention of keyctl and request_key in
mordred(which also mentions that the kernel keyring is not namespaced)12:29
mordredcorvus: ooh - --security-opt seccomp=unconfined as an option to docker will run without the seccomp profile - might be a quick and easy  way to see if the seccomp profile is the thing that's causing bubblewrap+AFS to not work right12:30
mordredcorvus: also: might give us some options12:46
corvusmordred: i think that works13:39
corvusthat's the test procedure i used13:39
corvusi ran two instances of that simultaneously, and they did not seem to share credentials.  i believe that was the failure we saw last time -- that all docker containers shared the same tokens.  when i kinit/aklog in the first one, and do nothing in the second, the first has access and the second does not.  when i kinit/aklog in the second with a different principal, the first does not switch to operating13:42
corvuswith the second principal, it continues using the first.13:42
corvusi also verified neither is available in a third container or the host system.13:42
corvus(zuultest in that etherpad is a local image of zuul/zuul-executor with kinit and aklog installed)13:44
corvusmordred: i think that the important thing here for security is that any container where we allow those syscalls needs to trust all other containers where we allow those syscalls on the same host.  since a zuul-executor is the only container we would do that for, i think we can agree to those terms both now and even in the future (if there's a way to do this with k8s, that would probably be fine too).13:48
corvusmordred: i made a custom secconmp profile with add_key, keyctl, and request_key added, and it works13:55
mordredcorvus: ah - cool - so just doing the custome seccomp profile is probably a nice way to do that for us13:56
corvusmordred: yep, i think that's going to be a winner -- double checking a few things real quick before i declare the test complete...13:57
mordredcorvus: I'm guessing that would probably mean in a k8s sense that one would want to add scheduling hints to prevent any non-zuul-executor from running on a host13:57
mordredcorvus: but ... this is a set of things that are only needed for using afs13:57
mordredso it's probably ok to say "if you want to use afs from a zuul-executor in k8s, you need to apply this seccomp profile and these scheduler hints"13:58
corvusmordred: maybe?  if you could specify a seccomp profile for a specific container, then i don't think that would be necessary; otherwise, yeah, that would probably be a good idea13:58
mordredcorvus: oh - good point13:58
fungii concur with the security risk assessment around in-kernel key management13:59
mordredcorvus: doesn't look like it's super-fine-grained in k8s14:00
mordredthe k8s admin can control the list of seccomp profiles that pods can use14:00
mordredbut it doesn't look like there's a way to say "this pod can use this profile, but others can't"14:00
corvusmordred: that seems like a big hole14:01
mordredcorvus: oh - actually14:01
mordredcorvus: "unconfined - Seccomp is not applied to the container processes (this is the default in Kubernetes), if no alternative is provided."14:01
mordredcorvus: am I reading that right that k8s turns off seccomp by default?14:01
corvusmordred: wow.  k8s is all about the just making stuff work out of the box aren't they?14:01
mordredcorvus: so - fwiw, the thing that got me down this rabbit hole is that I was trying to figure out what it would take to make an oci runtime that just used chroot with no other containment technology applied14:03
mordred(because sometimes just using docker as a package manager is really all you want)14:03
mordredcorvus: I did learn that there is a DOCKER_RAMDISK env var that if set will cause docker to use chroot and not pivot_root - but I don't think it disables the other isolation stuff14:04
corvusmordred: hrm, i'm having trouble reproducing what i thought was the original error; so let's not call this done yet14:05
corvusmordred: (also cool)14:05
mordredcorvus: hrm - sad panda14:09
corvusmordred: okay, this is weird.  what i'm seeing is that with the default seccomp profile, all docker containers share the same keyring.  however, zuul-bwrap running within one or more of those containers (i tried both ways) get an isolated keyring.14:23
corvusit almost seems like bwrap is somehow able to still make those syscalls?  maybe privileged allows it to do that?14:24
corvus(this is all really annoyingly hard to test)14:26
corvusi think it may be worth doing this on a real executor and not my workstation -- that way we can reboot to clear out state, etc.14:27
mordredcorvus: yeah. also - I wonder if maybe behavior changed in docker versions and you've upgraded docker since last you tested this14:28
mordredcorvus: cause I think you updated something when we were working on multi-arch a few weeks ago yeah?14:29
corvusmordred: hrm, i don't remember, but that doesn't mean it didn't happen14:30
mordredcorvus: we could go ahead and add the install-docker to the executors so we can test this14:31
corvusmordred: ++14:31
corvusi've got to get started on some saturday stuff now, i'll check back in later14:31
mordredcorvus: kk. I'll get the docker stuff installed14:33
mordredcorvus: it's already done actually14:33
mordredcorvus: I'm going to try your reproducer - without the seccomp stuff first14:34
corvusmordred: you'll need openafs-krb5 installed in the image14:35
corvusmordred: and i'd do this with the executor shut down just so we don't get pollution from any real jobs that use afs14:35
mordredcorvus: nod14:36
mordredcorvus: I would suppose we'd need to do a kinit/aklog in the container to yeah?14:36
mordredcorvus: I'm building a zuultest image with openafs-krb5 in it on ze0114:38
mordredcorvus: alternate strategy:
mordredbut - I like where we're going currently :)14:42
*** ysandeep has quit IRC14:49
*** ysandeep has joined #opendev14:51
mordredcorvus: uhm. I feel like I have lost AFS knowledge. I seem to be able to manipulate things in afs as root on ze01 without any kerberos tickets15:03
mordredcorvus: (although as not-root AFS gives me errors when trying to do that)15:04
mordredcorvus: also - the current config for krb5 seems to be to put ccache into a file, not into keyring15:09
mordredcorvus: so it's defaulting now to FILE:/tmp/krb5cc_%{uid}15:10
mordredcorvus: anywho - I've got a zuultest image on ze01 and the executor shut down - I'm not 100% sure what I should do next to verify15:14
mordredcorvus: I also installed krb5-user and openafs-client in the image so the various cli things would be there15:15
corvusmordred: krb tickets go in the file; afs tokens are in the keyring15:20
corvusmordred: root@ze01 has a token in kerenl memory (run 'tokens')15:22
corvusas long as that's around, root will be able to do afs stuff15:22
corvus'unlog' will remove that15:22
corvusto recap: kinit = get kerberos ticket (save to ccache file in tmp); aklog = turn kerberos ticket into afs token (stored in kernel keyring); unlog = remove token from kernel keyring; kdestroy = delete kerberos credential cache15:23
mordredcorvus: ah15:38
mordredcool. I have unlogged15:39
mordredand I can now confirm that I can't touch the private dir I made in /afs/ even as root15:39
mordredcorvus: fwiw- zuul-bwrap in that container is producing a --ro-bind None None15:58
mordredin the middle of the command line15:58
mordredah - it's ssh_auth_sock - silly me, I skipped the ssh-agent invocation16:01
mordredcorvus: on ze01 I ahve used your reproducer and I have kinit'd and akloged in one bubblewrap16:04
mordredI have started a second container and I do not see tokens16:04
mordrednor can I access AFS resources in the second container that I can in the first16:04
mordred(also I do not see any AFS tokens as root on the host)16:05
mordredcorvus: I'm leaving the executor off on ze01 in case you want to poke16:09
openstackgerritMonty Taylor proposed opendev/system-config master: Run zuul-executor using docker
mordredcorvus: I think that ^^ is what we'll want to do once we're comfortable that this will work16:20
prometheanfireit looks like the daily generate-constraints update failed again16:20
prometheanfireit's running fine manually16:20
*** slaweq has joined #opendev17:12
*** slaweq has quit IRC17:17
*** roman_g has joined #opendev17:38
*** roman_g has quit IRC17:44
*** roman_g has joined #opendev17:45
*** roman_g has quit IRC17:45
*** slaweq has joined #opendev18:05
*** slaweq has quit IRC18:31
*** Dmitrii-Sh has quit IRC20:47
*** Dmitrii-Sh has joined #opendev20:47
*** roman_g has joined #opendev20:54
*** Dmitrii-Sh has quit IRC20:56
*** Dmitrii-Sh has joined #opendev20:56
*** roman_g has quit IRC21:34
*** wrenchyfrenchy has joined #opendev22:02
*** noonedeadpunk has quit IRC22:15
*** noonedeadpunk has joined #opendev22:15
*** dtantsur|afk has quit IRC22:42
*** dtantsur has joined #opendev22:42
*** dtantsur has quit IRC22:49
*** tosky has quit IRC23:35
*** DSpider has quit IRC23:46

Generated by 2.17.2 by Marius Gedminas - find it at!