Thursday, 2019-05-23

*** goldyfruit has quit IRC01:36
*** goldyfruit has joined #openstack-qinling02:06
*** irclogbot_1 has quit IRC02:26
*** irclogbot_3 has joined #openstack-qinling02:29
goldyfruitlxkong, +1 from Zuul https://review.opendev.org/#/c/660789/02:48
goldyfruitWait and see the comments from Kolla core reviewers!02:49
lxkonggoldyfruit: thanks for the contribution02:49
lxkongand thanks for answering questions when i'm offline :-)02:49
goldyfruitYou're welcome!02:50
goldyfruitThe least I can do02:50
lxkongi have an meeting at 15:00, will be back soon02:50
goldyfruit11pm here :)02:50
*** goldyfruit has quit IRC04:06
*** hrw has joined #openstack-qinling13:14
hrwmorning13:14
*** goldyfruit has joined #openstack-qinling13:16
hrwyour README.rst is py3 unfriendly :(13:17
hrwhttp://logs.openstack.org/75/642375/23/check/kolla-build-ubuntu-source/280dece/logs/build/000_FAILED_qinling-base.txt.gz13:18
hrwhttp://lists.openstack.org/pipermail/openstack-discuss/2019-May/006544.html13:28
goldyfruithrw I got the +1 form Zuul, I fixed my Readme13:37
hrwwhere?13:39
goldyfruithttps://review.opendev.org/#/c/66078913:39
goldyfruithrw, sorry I think we are not speaking about the same thing !13:39
hrwgoldyfruit: that job was run with py213:39
hrwgoldyfruit: which gave you +113:39
hrwgoldyfruit: https://review.opendev.org/642375 check13:40
goldyfruityes indeed !13:41
hrw;)13:41
goldyfruitThis is related to tʃinliŋ13:44
hrwyes13:44
goldyfruitI'll try to fix it13:45
hrwthank you13:45
goldyfruitI guess we could go Tchinling13:53
hrwenglished ipa?13:54
goldyfruitI asked to a colleague who speaks mandarin13:57
hrwcool13:58
goldyfruitI hope lxkong will be OK with that :D13:58
hrwbut you can wait for https://review.opendev.org/#/c/56487413:59
hrwas this may solve it globally13:59
goldyfruitLast update, June 201813:59
hrwhttp://lists.openstack.org/pipermail/openstack-discuss/2019-May/006547.html14:00
hrwbbl14:00
openstackgerritGaëtan Trellu proposed openstack/qinling master: README: make it us-ascii  https://review.opendev.org/66100514:03
hrwthanks14:53
hrwcommented and +114:53
goldyfruitOh! hrw = Marcin14:54
hrwyes14:54
goldyfruitDidn't know, we met few times at the summit14:54
goldyfruit<- Ormuco14:54
hrwnever been at summit ;D14:54
hrwmaybe at Snowpenstack in Dublin14:55
goldyfruitOh, wrong Name then :(14:55
goldyfruit(sorry)14:55
hrwhappens ;D14:55
hrwnp14:55
hrwI do not have good face memory14:55
*** goldyfruit has quit IRC17:03
*** hrw has quit IRC17:21
*** goldyfruit has joined #openstack-qinling17:36
nsherry4goldyfruit: I've made some progress since yesterday, but I think I've hit another road block, do you have some time to take a look?18:38
goldyfruiti could try18:39
nsherry4I've been able to create a function, but now when I try to run it, qinling-engine gets a 500 error back18:39
nsherry4which manifests as it not being able to decode a non-existent json payload18:40
goldyfruitJSONDecodeError: Expecting value: line 1 column 1 (char 0)?18:40
nsherry4yup18:40
goldyfruitLike this bug: https://storyboard.openstack.org/#!/story/2005711 ?18:40
goldyfruitAre you using Python 2.7 or Python 3 ?18:41
goldyfruitI got error with Python 2.718:41
goldyfruitWhich is why I opened the bug :p18:41
nsherry4openstackqinling/python3-runtime18:42
goldyfruithum18:42
goldyfruitWhat is your code ?18:42
goldyfruitof the function18:42
nsherry4Just the 'Hello World' function you mentioned18:42
nsherry4I added some debug output to engine, and got "The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application."18:42
goldyfruitCould you please paste the log ?18:43
nsherry4sure thing18:43
nsherry4https://pastebin.com/fkRzi7FS18:44
goldyfruitYou could try to check the pod logs18:45
goldyfruitkubectl get pod -o wide -n qinling18:45
goldyfruitkubectl logs 35bafe75-f5fa-440e-af09-ae2fe97c8e5f-849fcdd9cb-v2dhg worker -n qinlin18:46
goldyfruitDo the same for the 3 pods18:46
nsherry4https://pastebin.com/ZSFqwKWa18:49
goldyfruitFailed to download function package from http://<IP>:7070/v1/functions/eda8a29b-dbc1-47e0-9ed5-a133d2ddac30?download=true18:49
goldyfruitFailed to establish a new connection: [Errno 113] Host is unreachable18:50
goldyfruitWhich means that your worker container doesn't have access to Qinling API18:50
nsherry4Firewall issue?18:50
goldyfruitCould be18:51
goldyfruitAs I mentioned yesterday, I would have disable all the possible failure stuff for the tests18:51
goldyfruitfirewall, selinux18:51
goldyfruitjust for tests18:51
nsherry4Disabled iptables, same result18:53
nsherry4I can get a shell into the worker node with exec, right?18:54
goldyfruitnop18:54
goldyfruitbut you could create a simple pod (not related to qinling)18:55
goldyfruithttps://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/18:55
goldyfruitnginx image has a shell18:55
nsherry4okay, so I _was_ able to shell into the worker node, but it doesn't have ping/telnet/etc. It does have ssh, and with the -v flag, I can see it returning html 400 markup when I try to ssh to port 7070, so it _can_ connect19:00
goldyfruitbut port 7070 is not ssh19:01
goldyfruitIt's HTTP19:01
goldyfruitDo you have curl/wget in the nginx image ?19:02
nsherry4I don't have an nginx image19:02
goldyfruitohh ok19:03
nsherry4I do have curl on the worker, though19:03
nsherry4curl http://10.64.12.232:707019:03
goldyfruityep19:03
nsherry4{"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://10.64.12.232:7070/v1", "target": "v1", "rel": "self"}]}]}19:03
goldyfruitJust in case, could you please list the network policy19:03
goldyfruitkubectl get netpol -n qinling19:04
goldyfruitshould be empty19:04
goldyfruitexcept if qinling-engine has been restarted since yesterday19:04
nsherry4ah, yes, I've restarted it19:04
nsherry4allow-qinling-engine-only   <none>         127m19:04
goldyfruitkubectl delete netpol allow-qinling-engine-only -n qinling19:05
nsherry4I have to delete this with every restart?19:05
goldyfruityeah :(19:05
goldyfruitbecause of the bug I sent you yesterday19:05
goldyfruithttps://storyboard.openstack.org/#!/story/200571019:05
nsherry4Same error19:06
nsherry4wait19:07
nsherry4same error in the engine19:07
nsherry4different error on the worker19:07
goldyfruitAh19:07
nsherry4This time it can't serialize a "Read timed out. (read timeout=5)"19:08
goldyfruitcould you please paste the error19:08
nsherry4https://pastebin.com/t3NPV5Mq19:09
goldyfruitYeah, still the same, unable to reach the Qinling API19:09
nsherry4curling that URL from the container shell gives me a 401 Unauthorized19:11
goldyfruitDo you see the call in qinling-api.log ?19:11
nsherry42 lines, creating execution and downloading function19:12
goldyfruitWhen you create your function, try to specify a bigger timeout19:14
goldyfruit--timeout19:14
nsherry4Wait, the logs seem to show the 'Downloading function' line showing up after the execution fails19:14
nsherry4I'll try the timeout change now19:15
nsherry4Set the timeout to 300s, but got the same result. Didn't take 300s to fail19:22
nsherry4looking at the created/updated times, it took 5 seconds, which is the time in the read timeout error in the worker19:23
goldyfruitI never got this issue :/19:27
goldyfruitCould be related to microk8s19:28
goldyfruiteven with Iptables disable, did you apply the FORWARD19:28
nsherry4I'm seeing what happens when I add a timeout value to the python requests post call  in worker:server.py to fetch the script19:29
goldyfruitkubectl logs 35bafe75-f5fa-440e-af09-ae2fe97c8e5f-849fcdd9cb-v2dhg sidecar -n qinling19:29
goldyfruitCheck the sidecar container19:29
goldyfruitWhich is the one responsible to download the package19:29
nsherry4sidecar shows a number of 5000ms download calls, but I don't see any errors19:33
nsherry4"POST /download => generated 222 bytes in 5018 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)"19:35
nsherry4so the worker node calls the sidecar, which in turn downloads the function package from qinling-api?19:37
goldyfruitIs the engine bind on an IP ?19:38
goldyfruithttps://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/config.py#L86-L12119:38
nsherry4"Failed to request url http://10.64.12.232:31380/execute"19:39
nsherry4I don't have any settings in the [engine] section set in the config19:40
goldyfruitOk, so by default it listens everywhere19:40
goldyfruitI'm running out of ID19:41
goldyfruitidea19:41
nsherry4Does the sidecar have to auth to download the function package?19:41
nsherry4Our auth system in this toy environment is pretty slow19:42
goldyfruitNo the authentication is made by Qinling API19:42
goldyfruitWhen the call goes thru the engine it means that it has already been authenticated19:42
nsherry4got it19:43
goldyfruithttps://github.com/openstack/qinling/blob/master/runtimes/sidecar/sidecar.py19:43
goldyfruitI think I said crap about the auth and the sidecar19:43
goldyfruithttps://github.com/openstack/qinling/blob/master/runtimes/sidecar/sidecar.py#L55-L5619:44
nsherry4line 59: timeout=519:44
goldyfruit:D19:44
nsherry4any way I can change that through kubectl?19:44
goldyfruitYou mean kubectl exec ?19:45
goldyfruitGoing into the sidecar container and change it manually ?19:46
nsherry4a-hah19:46
nsherry4sidecar doesn't have bash, but it does have sh19:46
nsherry4I may have to restart the containers for this to work19:48
goldyfruityep19:48
goldyfruitremove the pyc too before19:48
goldyfruit(if there is)19:48
goldyfruitnsherry4, I just build a new one with 60s as timeout19:51
goldyfruithttps://hub.docker.com/r/goldyfruit/sidecar19:51
goldyfruitIn your qinling.conf19:51
goldyfruitsidecar_image = goldyfruit/sidecar:0.0.319:52
goldyfruitunder the [engine section]19:52
nsherry4restart the engine?19:52
goldyfruityes, and then delete the network policy19:53
goldyfruitYou will have to create a new runtime19:53
goldyfruitruntime create the two containers19:53
goldyfruithttps://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/orchestrator/kubernetes/templates/deployment.j219:54
nsherry4just to keep things simple, I'm going to set the replicas to 119:54
goldyfruit+119:55
nsherry4okay, new runtime is created, creating a new function for it19:57
nsherry4haha, success20:00
goldyfruitYeah !!!20:00
nsherry4thank you so much20:00
goldyfruitYour are welcome20:01
goldyfruitYou*20:01
nsherry4I'm going to play around with this for a while and see what it can do now!20:02
goldyfruitThat's great !20:02
nsherry4Long term, if this timeout issue follows us into a production environment, what would our best bet be?20:06
goldyfruitYou could build your own sidecar with a very high timeout20:07
goldyfruitLike I did20:07
nsherry4and that's just out of the runtimes folder in the git repo?20:08
goldyfruitSorry20:58
goldyfruithttps://paste.api-zulu.com/usorediyej.bash20:59
goldyfruitI created a story about the 5 seconds https://storyboard.openstack.org/#!/story/200576921:07
lxkonggood morning/afternoon/evening, goldyfruit, nsherry421:12
lxkongjust start my day and see your discussions here21:12
goldyfruit:)21:12
goldyfruitMorning21:13
lxkongi will spend some time today to try to fix the bugs you opened, goldyfruit21:13
lxkonggoldyfruit: i have a question, have you done some evaluation for Knative?21:13
goldyfruitDon't know what it is lxkong  :D21:14
lxkonghah21:14
goldyfruitPretty old in the OpenStack world but pretty new un the Kubernetes one21:14
lxkongKnative is the serverless solution in k8s world21:15
goldyfruitOhh21:15
lxkongit's becoming so popular in k8s community, i have plan to integrate with Knative at some point21:15
goldyfruitSo knative is used as standalone ?21:16
goldyfruitno need of k8s cluster ?21:16
goldyfruitor it's running on top of it ?21:16
lxkongon top of it21:16
lxkongand need to use together with Istio for more advanced features21:16
goldyfruitlxkong, what would be the advantage compare to what we currently have ?21:17
lxkongmore popular, more developer, more features :-)21:17
lxkongbut it provides k8s API21:17
goldyfruitSure, could be a very nice to have !21:18
openstackgerritGaëtan Trellu proposed openstack/qinling master: Increase request timeout for sidecar  https://review.opendev.org/66112221:19
openstackgerritGaëtan Trellu proposed openstack/qinling master: Increase request timeout for sidecar  https://review.opendev.org/66112221:20
goldyfruit(forgot the story/task in the commit message...)21:21
lxkonggoldyfruit:  will review that patch today21:21
goldyfruitNo rush !21:22
lxkongand for `tchinling` thing, i need to ask my colleagues for a pronunciation :-)21:22
goldyfruitSure21:22
goldyfruitI did a demo to my CEO21:24
goldyfruitHe loved it :D21:24
lxkongreally?!21:24
lxkongamazing!21:24
goldyfruitTook two devs to show them the potential !21:24
goldyfruitYeah21:24
lxkongif you decide to make some investment on Qinling, i'm very happy to support21:25
goldyfruitOh we will21:26
goldyfruitWhich is why I'm integrating it to Kolla21:26
goldyfruitwhich will help us to maintain/upgrade it in production21:26
lxkongi believe my boss would also love to see there are some other people interested in Qinling after two dev cycles21:26
goldyfruitYeah !21:27
goldyfruitHope the kolla integration will help for the adoption21:27
goldyfruitThe doc needs to be improved too, how to support Magnum, microk8s, etc...21:28
goldyfruitAnd then people will discover the potential21:28
goldyfruitLike nsherry4 :p21:28
lxkongnever heard of microk8s before, but could start with improving the doc related to Magnum21:28
goldyfruitMe too until yesterday :D21:29
openstackgerritMerged openstack/qinling master: README: make it us-ascii  https://review.opendev.org/66100522:34
*** goldyfruit has quit IRC22:38
*** goldyfruit has joined #openstack-qinling22:53
*** goldyfruit has quit IRC23:22
*** goldyfruit has joined #openstack-qinling23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!