| *** hongbin has joined #openstack-kuryr | 00:53 | |
| *** hongbin has quit IRC | 01:13 | |
| *** hongbin has joined #openstack-kuryr | 01:13 | |
| *** hongbin has quit IRC | 03:51 | |
| *** openstackgerrit has joined #openstack-kuryr | 04:01 | |
| openstackgerrit | Peng Liu proposed openstack/kuryr-kubernetes master: Implement NPWG multi-vif driver https://review.openstack.org/578009 | 04:01 |
|---|---|---|
| *** maysams has quit IRC | 04:03 | |
| *** maysams has joined #openstack-kuryr | 04:03 | |
| *** janki has joined #openstack-kuryr | 04:34 | |
| ltomasbo | good morning celebdor[m], I'm debugging the last namespace related patch (the one enabling it for openshift gates) and there is still some issues. Does this error rings a bell? http://paste.openstack.org/show/727596/ | 06:24 |
| *** pcaruana has joined #openstack-kuryr | 06:27 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers https://review.openstack.org/580680 | 06:57 |
| *** celebdor has joined #openstack-kuryr | 07:30 | |
| dulek | ltomasbo: "504 GATEWAY TIMEOUT" would mean that daemon timed out waiting for pod to get annotated. | 07:37 |
| dulek | ltomasbo: There might be more in kuryr-daemon logs. | 07:38 |
| dulek | ltomasbo: That's in the gate? | 07:38 |
| dulek | ltomasbo: You can find pod logs in controller/kubernetes/pod_logs. | 07:38 |
| ltomasbo | dulek, is that available for openshift gates too>? | 07:40 |
| dulek | ltomasbo: Should be. Can you point me to the run you're debugging? | 07:40 |
| ltomasbo | dulek, https://review.openstack.org/#/c/580680/ (though I rebase it to retrigger the tests | 07:41 |
| ltomasbo | as some of the failures were due to yum install gate thing | 07:41 |
| ltomasbo | dulek, it is going to fail again for the openshift gate, not sure why the crd objects cannot be found (probably they are not properly created...) http://zuul.openstack.org/stream.html?uuid=0357bea4ffd54429933bf1f03a11302a&logfile=console.log | 07:44 |
| dulek | ltomasbo: http://logs.openstack.org/80/580680/14/experimental/kuryr-kubernetes-tempest-daemon-containerized-openshift-octavia/22b01c6/job-output.txt.gz#_2018-08-07_16_11_41_606779 | 07:44 |
| dulek | ltomasbo: So that's one thing. | 07:45 |
| ltomasbo | yep, I need to understand why that is not working on openshift... | 07:46 |
| ltomasbo | probably I need to add some rbac to be able to write the CRDs?\ | 07:46 |
| dulek | ltomasbo: Hm, different implementation of CRDs? | 07:46 |
| dulek | ltomasbo: It's controller writing CRDs, right? | 07:46 |
| *** pmannidi has quit IRC | 07:46 | |
| celebdor | what happened? | 07:46 |
| ltomasbo | yep | 07:46 |
| ltomasbo | celebdor, I'm testing moving openshift gates to namespace isolation | 07:47 |
| ltomasbo | and it seems CRDs are not being created | 07:47 |
| dulek | ltomasbo: According to controller logs it doesn't have issues with creating CRDs. | 07:50 |
| ltomasbo | umm | 07:50 |
| ltomasbo | then why it is not there! | 07:51 |
| ltomasbo | ahh, perhaps another quota issue? | 07:51 |
| dulek | ltomasbo: You mean it got rolled back? Doesn't look like it… | 07:51 |
| dulek | ltomasbo: Here's the matching log from openshift-master: http://logs.openstack.org/80/580680/14/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/246d4b3/controller/logs/screen-openshift-master.txt.gz#_Aug_07_22_10_45_488104 | 07:52 |
| dulek | ltomasbo: This is time we create the CRDs. | 07:52 |
| ltomasbo | weird thing is taht docker-registry seems to be good | 07:54 |
| ltomasbo | and that is created in the default namespace | 07:54 |
| dulek | ltomasbo: Meanwhile if you want to debug that on the gate you can add your own kubectl commands here: https://github.com/openstack/kuryr-kubernetes/blob/master/tools/gate/copy_k8s_logs.sh#L34 | 07:54 |
| ltomasbo | dulek, ohh, I didn't know that! thanks! | 07:55 |
| ltomasbo | dulek, when do that runs? | 07:56 |
| dulek | ltomasbo: After all the tempest in gates. | 07:56 |
| ltomasbo | ok ok | 07:56 |
| ltomasbo | I'll add one regarding namespaces | 07:56 |
| dulek | ltomasbo: And CRDs! | 08:00 |
| ltomasbo | yes | 08:00 |
| ltomasbo | dulek, as soon as the current run finishes (which should be in 10 min or so) I'll re-spin with that | 08:01 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Implement NP SG create/delete actions https://review.openstack.org/583540 | 08:04 |
| ltomasbo | dulek, I don't see anything useful on the new logs | 08:11 |
| dulek | ltomasbo: This one's a bit odd: http://logs.openstack.org/80/580680/15/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/0357bea/controller/logs/screen-kuryr-kubernetes.txt.gz#_Aug_08_07_36_35_996608 | 08:13 |
| dulek | ltomasbo: Also: http://logs.openstack.org/80/580680/15/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/0357bea/controller/logs/screen-kuryr-kubernetes.txt.gz#_Aug_08_07_35_43_231635 | 08:14 |
| ltomasbo | though now it seem that test_namespace has created the subnet | 08:14 |
| ltomasbo | and the 3 test fails on the get_kuryr_net_crds function | 08:14 |
| dulek | ltomasbo: To debug that you might also want to increase verbose level on openshift-master. | 08:18 |
| dulek | ltomasbo: So we'll see the requests being made. | 08:18 |
| ltomasbo | ok, I'm going to submit a new patch set with the namespace/kuryrnetcrds | 08:19 |
| ltomasbo | and the openshift-master verbose (going to look where to set it...) | 08:19 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers https://review.openstack.org/580680 | 08:27 |
| *** garyloug has joined #openstack-kuryr | 08:47 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-tempest-plugin master: [WIP] Add Network Policy tests https://review.openstack.org/589521 | 09:10 |
| ltomasbo | dulek, strange, there may be some king of race | 09:24 |
| ltomasbo | test_namespace fails on different points everytime | 09:24 |
| ltomasbo | either on check taht the subnets are created | 09:24 |
| ltomasbo | or on the call to get the kuryr_net_crd | 09:25 |
| ltomasbo | http://logs.openstack.org/80/580680/15/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/0357bea/testr_results.html.gz | 09:26 |
| ltomasbo | http://logs.openstack.org/80/580680/15/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/0357bea/testr_results.html.gz | 09:26 |
| dulek | ltomasbo: /apis/openstack.org/v1/ns-default | 09:26 |
| dulek | ltomasbo: Aren't you missing v1 in the code? | 09:26 |
| dulek | ltomasbo: Ah no, you're not, sorry. | 09:26 |
| dulek | ltomasbo: No logs in openshift-master that would match the time of requests failing in kuryr-kubernetes logs… | 09:28 |
| dulek | ltomasbo: Why only OpenShift?! | 09:29 |
| dulek | ltomasbo: Hm, oh wait… | 09:29 |
| ltomasbo | Aug 08 09:02:28.430250 ubuntu-xenial-ovh-bhs1-0001205030 python[22918]: 2018-08-08 09:02:28.429 22918 DEBUG kuryr_kubernetes.controller.handlers.namespace [-] Setting CRD annotations: {'kind': 'KuryrNet', 'spec': {'routerId': 'f2b7b2aa-bea4-408c-92ac-ad02052a3058', 'subnetId': u'3bac6696-9398-4974-a051-8385d03cd994', 'netId': u'88f0242e-3387-45ca-9a12-e883c7054d21', 'sgId': u'6db67f6e-2035-4bcf-9c19-5b79e1d69fd2', 'subnetCIDR': | 09:29 |
| ltomasbo | u'10.1.1.64/26'}, 'apiVersion': 'openstack.org/v1', 'metadata': {'name': u'ns-default', 'annotations': {'namespaceName': u'default'}}} _set_net_crd /opt/stack/kuryr- | 09:29 |
| ltomasbo | but seems the subnet got created and annotated | 09:29 |
| dulek | ltomasbo: We're using OpenShift 3.9 and Kubernetes 1.9 in the gate. | 09:30 |
| dulek | ltomasbo: This matches more or less - 3.9 = 1.9.1. | 09:30 |
| ltomasbo | Creating network resources for namespace: kuryr-namespace-97888954 on_present /opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/namespace.py:49 | 09:30 |
| ltomasbo | umm | 09:32 |
| ltomasbo | http://logs.openstack.org/80/580680/16/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/a116735/controller/logs/screen-kuryr-kubernetes.txt.gz#_Aug_08_09_03_34_044788 | 09:34 |
| ltomasbo | dulek, ^^ | 09:34 |
| ltomasbo | after that, I don se the messages creating the namespaces actually | 09:34 |
| ltomasbo | ahh, yes, there was some other annotations in between | 09:35 |
| dulek | ltomasbo: I have no idea what's happening really… | 09:36 |
| ltomasbo | in the new kuryrnets_crds and namespaces | 09:38 |
| ltomasbo | I can see that the namespace kuryr-namespace-1159... exists | 09:39 |
| ltomasbo | as well as the associated CRD | 09:39 |
| dulek | ltomasbo: But the API returns 404 for it? | 09:41 |
| ltomasbo | well, in this run, it does not even reach there | 09:41 |
| ltomasbo | going to check for the test_namespace_sg_isolation, which reaches that 404 | 09:41 |
| ltomasbo | yep, it is also there | 09:42 |
| ltomasbo | dulek, celebdor: I think I found something | 10:06 |
| ltomasbo | kubectl get clusterrole -n kube-system kuryr-controller -o yaml | 10:06 |
| ltomasbo | this is defined for kuryr-kubernetes but not for openshift | 10:06 |
| dulek | ltomasbo: Uhm? I don't get it. | 10:07 |
| dulek | ltomasbo: But now that I think of this: | 10:07 |
| dulek | ltomasbo: https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/lib/kuryr_kubernetes#L485-L493 | 10:07 |
| ltomasbo | dulek, we need to add customreousrdefinitions to the resources, right? | 10:08 |
| ltomasbo | yep, that is what I think | 10:08 |
| dulek | ltomasbo: But it should error out if there are unsufficient privileges and it doesn't according to logs? | 10:08 |
| ltomasbo | perhaps it does not, I'm not sure | 10:08 |
| ltomasbo | though, the crds are created | 10:08 |
| ltomasbo | and this works for kubernetes | 10:09 |
| ltomasbo | so, it must not be that | 10:09 |
| ltomasbo | I deployed with openshift, and I see it is actually working | 10:11 |
| ltomasbo | creating the subnet, the crds, ..., I'm a bit lost here... | 10:11 |
| ltomasbo | umm, I get this: | 10:14 |
| ltomasbo | error: couldn't get deployment demo-1: replicationcontrollers "demo-1" is forbidden: User "system:serviceaccount:test:deployer" cannot get replicationcontrollers in the namespace "test": User "system:serviceaccount:test:deployer" cannot get replicationcontrollers in project "test" | 10:14 |
| ltomasbo | so, there is something different for openshift and the namespaces... and the roles... | 10:16 |
| ltomasbo | dulek, ^^ | 10:16 |
| dulek | ltomasbo: Okay, that's something, though it's the deployer. Deployer would mean the registry installation may be failing. | 10:16 |
| dulek | It shouldn't be related really. | 10:16 |
| ltomasbo | dulek, I deployed openshift on a local devstack | 10:17 |
| ltomasbo | creating the namespace and such works | 10:17 |
| ltomasbo | but then, oc run --image demo/kuryr demo | 10:17 |
| ltomasbo | works only on the default namespace | 10:17 |
| ltomasbo | and the docker-registry-1-xxx seems to be up and running | 10:18 |
| dulek | ltomasbo: Hm… | 10:19 |
| dulek | ltomasbo: Okay, first of all - tempest doesn't use oc all. | 10:21 |
| dulek | ltomasbo: oc run, I mean. | 10:21 |
| ltomasbo | ahh, true | 10:21 |
| dulek | ltomasbo: Let's put the deployer issues aside and use kubectl. | 10:21 |
| ltomasbo | I'm going to try two things, using kubectl and using oc new-project | 10:21 |
| ltomasbo | ok, that seems to work | 10:23 |
| dulek | ltomasbo: kubectl run? | 10:24 |
| ltomasbo | yep | 10:24 |
| ltomasbo | problem was I mixed commands | 10:25 |
| ltomasbo | oc new-project + oc run works | 10:25 |
| ltomasbo | oc create namespace + oc run does not | 10:25 |
| ltomasbo | anyway, lost again... | 10:26 |
| ltomasbo | dulek, kuryr/controller and cni are 3 adn 8 weeks old on the docker.io, would it be possible to update them? | 10:42 |
| dulek | ltomasbo: Sure, should I just upload newest master? | 10:43 |
| ltomasbo | that would be great | 10:43 |
| ltomasbo | regarding the gate, I'm thinking running in serial to see if that helps | 10:44 |
| dulek | ltomasbo: Hm, not sure really - the issue is with kuryr-controller and we're running just one of them… | 10:45 |
| ltomasbo | for me the issue is that I don't know what the issue is! xD | 10:46 |
| celebdor | ltomasbo: dulek: an I help? | 10:52 |
| celebdor | *can | 10:52 |
| ltomasbo | sure! | 10:52 |
| dulek | celebdor: You can roam the logs of failed run, maybe something stands off there for you. | 10:52 |
| celebdor | dulek: what are the symptoms | 10:53 |
| celebdor | ? | 10:53 |
| dulek | celebdor: But to be honest I doubt it… | 10:53 |
| ltomasbo | celebdor, dulek: I just see that the list of namespaces have some annotation missing for the created namespaces | 10:53 |
| ltomasbo | this is for default namespace: | 10:53 |
| ltomasbo | "annotations":{"openshift.io/sa.initialized-roles":"true","openshift.io/sa.scc.mcs":"s0:c1,c0","openshift.io/sa.scc.supplemental-groups":"1000000000/10000","openshift.io/sa.scc.uid-range":"1000000000/10000","openstack.org/kuryr-net-crd":"ns-default"}} | 10:54 |
| ltomasbo | and this is for the new created namespaces: | 10:54 |
| ltomasbo | "annotations":{"openshift.io/sa.scc.mcs":"s0:c8,c2","openshift.io/sa.scc.supplemental-groups":"1000060000/10000","openshift.io/sa.scc.uid-range":"1000060000/10000"}} | 10:54 |
| dulek | ltomasbo: You're annotating NS's, right? | 10:55 |
| ltomasbo | yep, we are annotating the namespace with the kuryrnet crd object name | 10:55 |
| dulek | ltomasbo: And looks like kuryrnet it's not annotated here? | 10:55 |
| ltomasbo | yep, seems it is missing | 10:55 |
| ltomasbo | though let me see on the new kubernetes logs | 10:55 |
| ltomasbo | in the logs, it has it | 10:56 |
| ltomasbo | annotations: | 10:56 |
| ltomasbo | openshift.io/sa.scc.mcs: s0:c8,c2 | 10:56 |
| ltomasbo | openshift.io/sa.scc.supplemental-groups: 1000060000/10000 | 10:56 |
| ltomasbo | openshift.io/sa.scc.uid-range: 1000060000/10000 | 10:56 |
| ltomasbo | openstack.org/kuryr-net-crd: ns-kuryr-namespace-1159952878 | 10:56 |
| celebdor | ltomasbo: I assume you can't reproduce locally | 10:57 |
| ltomasbo | celebdor, nope, it works for me on local devstack | 11:00 |
| ltomasbo | I actually think it may be a race, that it is trying to check that before the crd object is created | 11:00 |
| ltomasbo | I'm going to try with a dummy sleep | 11:01 |
| celebdor | ltomasbo: what checks it tempest or devstack? | 11:01 |
| celebdor | The former, right? | 11:01 |
| ltomasbo | tempest | 11:01 |
| ltomasbo | ahh, I can try to deploy locally with tempest and try again | 11:01 |
| ltomasbo | I just did it manually, but with tempest enabled it should be reproducible (I hope) | 11:02 |
| celebdor | ok | 11:02 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: DNM: Testing openshift gates https://review.openstack.org/589841 | 11:12 |
| *** rh-jelabarre has quit IRC | 12:26 | |
| openstackgerrit | Itzik Brown proposed openstack/kuryr-tempest-plugin master: Use expected pod output as a constant https://review.openstack.org/589878 | 12:55 |
| celebdor | ltomasbo: dulek: I just tested that on a devstack environment we can test oc new-app | 12:56 |
| celebdor | with the kuryr-demo container :-) | 12:56 |
| celebdor | to test that build actually works | 12:56 |
| celebdor | oc new-app https://git.openstack.org/openstack/kuryr-tempest-plugin --context-dir=test_container | 12:57 |
| celebdor | Let's see if I find how to do that without 'oc' so we can put it in tempest | 12:57 |
| *** garyloug_ has joined #openstack-kuryr | 13:01 | |
| irenab | celebdor, joining the sriov meeting? | 13:02 |
| *** garyloug has quit IRC | 13:04 | |
| celebdor | sure | 13:16 |
| openstackgerrit | Itzik Brown proposed openstack/kuryr-tempest-plugin master: Use expected pod output as a constant https://review.openstack.org/589878 | 13:18 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: DNM: Testing openshift gates https://review.openstack.org/589841 | 13:27 |
| *** garyloug__ has joined #openstack-kuryr | 13:50 | |
| *** garyloug_ has quit IRC | 13:54 | |
| *** rh-jelabarre has joined #openstack-kuryr | 13:58 | |
| *** rh-jelabarre has quit IRC | 13:59 | |
| ltomasbo | dulek, celebdor: so I found the issue with the test | 14:04 |
| dulek | ltomasbo: I'm curious! | 14:04 |
| ltomasbo | not sure why but when creating a namespace, it does not wait enough | 14:05 |
| ltomasbo | and when it checks the created namespaces they are not there yet | 14:05 |
| *** celebdor1 has joined #openstack-kuryr | 14:05 | |
| celebdor1 | ltomasbo: do tell | 14:07 |
| celebdor1 | don't leave me in suspense | 14:07 |
| *** celebdor has quit IRC | 14:07 | |
| ltomasbo | celebdor1, seems etcd is too slow and the kuryrcrd is not yet there | 14:08 |
| ltomasbo | and that is why at the end of the test, when doing the dump of the crds they where actually there... | 14:08 |
| ltomasbo | I'm testing locally with some sleep to see if that helps | 14:08 |
| ltomasbo | dman | 14:16 |
| ltomasbo | damn | 14:16 |
| ltomasbo | whole day with this sit | 14:16 |
| ltomasbo | shit | 14:16 |
| ltomasbo | and it was obvious... | 14:16 |
| ltomasbo | I'll sent a patch in 1 min! | 14:16 |
| ltomasbo | dulek, ^^ | 14:16 |
| dulek | ltomasbo: Like you're adding time.sleep()? | 14:17 |
| ltomasbo | no no, problem is that openshift does some annotations on the namespace that kubernetes does not | 14:18 |
| ltomasbo | and the create_namespace function was waiting until some annotations are done | 14:18 |
| ltomasbo | so, for kubernetes it was waiting as it should, but for openshift, it was returning before the crd was created | 14:18 |
| *** hongbin has joined #openstack-kuryr | 14:18 | |
| dulek | ltomasbo: Oh…! | 14:21 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Ensure create_namespace function waits for CRD annotation https://review.openstack.org/589841 | 14:26 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Ensure create_namespace function waits for CRD annotation https://review.openstack.org/589841 | 14:28 |
| ltomasbo | dulek, https://review.openstack.org/#/c/589841/ | 14:29 |
| ltomasbo | dmellado, celebdor1 ^^ | 14:29 |
| dmellado | ltomasbo: does it finally work? | 14:29 |
| dmellado | you just needed some more time for that, didn't you? | 14:29 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers https://review.openstack.org/580680 | 14:30 |
| ltomasbo | dmellado, not time, but waiting properly for the namespace creation | 14:31 |
| ltomasbo | dmellado, I was waiting until some annotation was done | 14:31 |
| ltomasbo | and openshift does some extra annotations, so it was not waiting | 14:31 |
| ltomasbo | now, I wait for the kuryr-crd annotation, so that should be safe | 14:31 |
| dmellado | great, I shall take a look | 14:32 |
| dulek | ltomasbo: Nice! | 14:37 |
| *** celebdor1 has quit IRC | 15:25 | |
| *** celebdor1 has joined #openstack-kuryr | 15:31 | |
| *** pcaruana has quit IRC | 15:34 | |
| *** aojea has joined #openstack-kuryr | 15:51 | |
| *** celebdor1 has quit IRC | 16:25 | |
| *** garyloug__ has quit IRC | 17:18 | |
| *** janki has quit IRC | 18:37 | |
| *** pcaruana has joined #openstack-kuryr | 21:07 | |
| *** aojea has quit IRC | 21:55 | |
| *** pcaruana has quit IRC | 22:02 | |
| *** aojea has joined #openstack-kuryr | 22:08 | |
| *** aojea has quit IRC | 22:09 | |
| *** aojea has joined #openstack-kuryr | 22:11 | |
| *** aojea has quit IRC | 22:30 | |
| *** hongbin has quit IRC | 22:54 | |
| *** kailun has quit IRC | 23:02 | |
| *** irenab has quit IRC | 23:02 | |
| *** oanson has quit IRC | 23:02 | |
| *** pc_m has quit IRC | 23:02 | |
| *** dulek has quit IRC | 23:02 | |
| *** russellb has quit IRC | 23:02 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!