*** aojea has joined #openstack-kuryr | 00:02 | |
*** aojea has quit IRC | 00:07 | |
openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: Allow RequiresMACAddress to be configurable https://review.openstack.org/519510 | 00:09 |
---|---|---|
*** hongbin has quit IRC | 00:23 | |
*** salv-orlando has joined #openstack-kuryr | 00:38 | |
*** salv-orlando has quit IRC | 00:43 | |
*** caowei has joined #openstack-kuryr | 01:03 | |
*** kiennt26 has joined #openstack-kuryr | 01:37 | |
*** salv-orlando has joined #openstack-kuryr | 01:39 | |
*** salv-orlando has quit IRC | 01:44 | |
*** kiennt26 has quit IRC | 01:54 | |
*** aojea has joined #openstack-kuryr | 02:03 | |
*** aojea has quit IRC | 02:07 | |
*** kiennt26 has joined #openstack-kuryr | 02:20 | |
*** salv-orlando has joined #openstack-kuryr | 02:40 | |
*** salv-orlando has quit IRC | 02:44 | |
*** hongbin has joined #openstack-kuryr | 02:53 | |
*** salv-orlando has joined #openstack-kuryr | 03:41 | |
*** salv-orlando has quit IRC | 03:45 | |
*** yamamoto has joined #openstack-kuryr | 03:52 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/519094 | 03:56 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 04:00 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 04:00 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 04:00 |
*** aojea has joined #openstack-kuryr | 04:04 | |
*** aojea has quit IRC | 04:11 | |
*** kiennt26 has quit IRC | 04:13 | |
*** salv-orlando has joined #openstack-kuryr | 04:41 | |
*** salv-orlando has quit IRC | 04:46 | |
openstackgerrit | Kien Nguyen proposed openstack/kuryr-libnetwork master: Support shared kuryr subnetpool https://review.openstack.org/516228 | 04:56 |
openstackgerrit | Kien Nguyen proposed openstack/kuryr-libnetwork master: Use specific Exception https://review.openstack.org/519537 | 04:56 |
*** caowei has quit IRC | 04:59 | |
*** hongbin has quit IRC | 05:01 | |
*** yboaron has joined #openstack-kuryr | 05:15 | |
*** caowei has joined #openstack-kuryr | 05:32 | |
*** janki has joined #openstack-kuryr | 05:34 | |
*** salv-orlando has joined #openstack-kuryr | 05:42 | |
*** salv-orlando has quit IRC | 05:47 | |
*** janki has quit IRC | 06:05 | |
*** aojea has joined #openstack-kuryr | 06:06 | |
*** aojea has quit IRC | 06:11 | |
*** aojea has joined #openstack-kuryr | 06:19 | |
*** janki has joined #openstack-kuryr | 06:20 | |
*** aojea has quit IRC | 06:24 | |
*** aojea has joined #openstack-kuryr | 06:28 | |
*** salv-orlando has joined #openstack-kuryr | 06:30 | |
*** aojea has quit IRC | 06:31 | |
*** pmannidi has quit IRC | 07:03 | |
*** janonymous has joined #openstack-kuryr | 07:20 | |
*** salv-orlando has quit IRC | 07:36 | |
*** salv-orlando has joined #openstack-kuryr | 07:36 | |
*** salv-orlando has quit IRC | 07:40 | |
*** salv-orlando has joined #openstack-kuryr | 07:47 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Fix kubelet retries issues https://review.openstack.org/518404 | 07:57 |
*** yamamoto has quit IRC | 08:09 | |
*** pcaruana has joined #openstack-kuryr | 08:10 | |
*** yamamoto has joined #openstack-kuryr | 08:12 | |
*** yamamoto has quit IRC | 08:12 | |
*** salv-orlando has quit IRC | 08:24 | |
*** yamamoto has joined #openstack-kuryr | 08:41 | |
*** salv-orlando has joined #openstack-kuryr | 08:42 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Fix kubelet retries issues https://review.openstack.org/518404 | 08:48 |
*** yamamoto has quit IRC | 08:50 | |
*** garyloug has joined #openstack-kuryr | 08:59 | |
*** salv-orlando has quit IRC | 09:20 | |
*** salv-orlando has joined #openstack-kuryr | 09:22 | |
*** salv-orlando has quit IRC | 09:26 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon https://review.openstack.org/515186 | 09:28 |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized https://review.openstack.org/518024 | 09:29 |
*** yamamoto has joined #openstack-kuryr | 10:01 | |
*** salv-orlando has joined #openstack-kuryr | 10:08 | |
*** salv-orlando has quit IRC | 10:11 | |
*** salv-orlando has joined #openstack-kuryr | 10:11 | |
*** caowei has quit IRC | 10:16 | |
*** yamamoto has quit IRC | 10:36 | |
*** yamamoto has joined #openstack-kuryr | 10:36 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking https://review.openstack.org/517406 | 10:48 |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon https://review.openstack.org/515186 | 10:48 |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add CNI daemon Tempest job https://review.openstack.org/519612 | 10:48 |
openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: Network policy support spec https://review.openstack.org/519239 | 10:48 |
openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod https://review.openstack.org/518502 | 10:51 |
*** gouthamr has joined #openstack-kuryr | 11:10 | |
*** salv-orlando has quit IRC | 11:19 | |
*** salv-orlando has joined #openstack-kuryr | 11:19 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add CNI daemon Tempest job https://review.openstack.org/519612 | 11:29 |
openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod https://review.openstack.org/518502 | 11:34 |
*** wangbo has joined #openstack-kuryr | 11:56 | |
*** gouthamr has quit IRC | 12:05 | |
*** reedip has quit IRC | 12:10 | |
*** gouthamr has joined #openstack-kuryr | 12:20 | |
*** atoth has joined #openstack-kuryr | 12:20 | |
*** reedip has joined #openstack-kuryr | 12:23 | |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking https://review.openstack.org/517406 | 12:28 |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon https://review.openstack.org/515186 | 12:28 |
openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: CNI Daemon documentation https://review.openstack.org/509380 | 12:28 |
*** gouthamr has quit IRC | 12:53 | |
*** garyloug has quit IRC | 13:01 | |
*** gouthamr has joined #openstack-kuryr | 13:02 | |
*** garyloug has joined #openstack-kuryr | 13:03 | |
*** wangbo has quit IRC | 13:10 | |
yboaron | #link https://docs.google.com/document/d/1Z-tRv6nWDLTCLA-uCrrh9aLakGIxxQSRniaS3E4DHIs/edit?usp=sharing | 13:16 |
yboaron | hi all , you can find in attached link a draft design document for openshift-route support in kuryr context | 13:17 |
yboaron | will be glad to get your feedback | 13:17 |
*** gouthamr has quit IRC | 13:38 | |
*** salv-orlando has quit IRC | 13:40 | |
*** apuimedo has joined #openstack-kuryr | 13:41 | |
apuimedo | dulek: what's the deal with the timeouts? | 13:41 |
*** salv-orlando has joined #openstack-kuryr | 13:41 | |
dulek | apuimedo: Hi! :) | 13:41 |
dulek | apuimedo: It's a longer story… | 13:42 |
dulek | So I was testing CNI daemon with running 20-30-50 pods in parallel. I've started to encountered tracebacks in daemon logs. | 13:43 |
dulek | Mostly from pyroute2. | 13:43 |
dulek | And pods that raised tracebacks were unpingable even if kubelet retried the request. | 13:44 |
dulek | So I started to fight with tracebacks - that's why you see those timeouts options added. | 13:44 |
dulek | But eventually me and ltomasbo realized, that tracebacks aren't the clue of the problem - the clue was the fact that for failed requests kubelet may issue DEL after a successful ADD. | 13:45 |
dulek | So I've fixed that, but left the timeouts configurable anyway. | 13:45 |
dulek | apuimedo: What do you think? | 13:46 |
apuimedo | dulek: fine | 13:53 |
apuimedo | sounds good | 13:53 |
apuimedo | irenab: how far along are you in the review for that? | 13:53 |
*** gouthamr has joined #openstack-kuryr | 13:54 | |
apuimedo | dulek: irenab: ltomasbo: https://www.openstack.org/videos/sydney-2017/how-to-build-scalable-reliable-and-stable-kubernetes-cluster-atop-openstack | 13:55 |
dulek | apuimedo: You mean it's worth watching? | 13:56 |
*** yboaron has quit IRC | 14:01 | |
apuimedo | ['[]\| | 14:02 |
dulek | apuimedo: Sorry, I don't get that… | 14:03 |
apuimedo | dulek: that was my daugther messing with the keyboard | 14:05 |
apuimedo | well, they took kuryr-kubernetes and modified it | 14:05 |
apuimedo | so it is worth to watch | 14:05 |
apuimedo | at least that part | 14:05 |
dulek | apuimedo: Nice, I've almost thought it's an emoji. :) | 14:05 |
apuimedo | :-) | 14:06 |
*** gouthamr has quit IRC | 15:13 | |
*** gouthamr has joined #openstack-kuryr | 15:16 | |
*** hongbin has joined #openstack-kuryr | 15:20 | |
*** janki has quit IRC | 15:39 | |
apuimedo | dulek: somehow I expected the readiness to be an http endpoint | 15:51 |
apuimedo | :P | 15:51 |
dulek | apuimedo: Ah, you're talking about ltomasbo's patch. Yeah, readiness can be an HTTP endpoint, /healthz is normally used. | 15:53 |
dulek | apuimedo: But we'd need to set up HTTP server, I'm not sure if that's worth it? | 15:53 |
apuimedo | that's what I had in mind | 15:54 |
apuimedo | but for now | 15:55 |
apuimedo | we can take it | 15:55 |
dulek | apuimedo: I'm not sure if you've checked out our discussion from yesterday's meeting? | 15:57 |
dulek | apuimedo: The problem I've raised is ltomasbo's patch will not stop kuryr-controller from processing requests. | 15:58 |
dulek | apuimedo: It's just an indicator for the administrator. | 15:58 |
apuimedo | I did not | 15:58 |
apuimedo | dulek: yeah. That much I know | 15:59 |
apuimedo | :-) | 15:59 |
apuimedo | I think that's fine | 15:59 |
apuimedo | for now | 15:59 |
apuimedo | we can make it configurable later | 15:59 |
apuimedo | but I think it is quite consistent with the k8s semantics | 15:59 |
dulek | apuimedo: Okay, at least ltomasbo was a bit confused about it and started to work on starting Watcher only after ports are reloaded. | 15:59 |
apuimedo | that pods being ready at t=40 does not mean that they do nothing before | 15:59 |
*** janki has joined #openstack-kuryr | 16:00 | |
dulek | apuimedo: I agree with that. As long as we won't break anything by processing before reload? | 16:00 |
apuimedo | dulek: I know I may be too picky with these things... | 16:02 |
apuimedo | How about using /usr/bin/test instead of cat | 16:02 |
apuimedo | since we have no interest in the content of the file? | 16:02 |
apuimedo | dulek: well, reload should not break existing entries | 16:02 |
apuimedo | if it does, it is a bug | 16:02 |
dulek | apuimedo: I'd say - use whatever is more portable? | 16:03 |
dulek | apuimedo: I just wondered if doing ADD operation won't "steal" already allocated port from the pool if it wasn't reloaded into memory as reserved yet. | 16:03 |
apuimedo | test is everywhere | 16:04 |
apuimedo | even on alpine base image | 16:04 |
apuimedo | ;-) | 16:04 |
dulek | apuimedo: But I haven't deeply looked at port pool code, so I don't know. | 16:04 |
apuimedo | ltomasbo: you here? | 16:04 |
dulek | apuimedo: Then test it is. :) | 16:04 |
apuimedo | dulek: but in any case | 16:04 |
ltomasbo | yep, now I'm here | 16:04 |
apuimedo | follow-up patch for that | 16:04 |
ltomasbo | I can change it with test instead | 16:04 |
apuimedo | don't modify it, dulek | 16:04 |
dulek | apuimedo: Hm? Why should I? | 16:05 |
ltomasbo | apuimedo, dulek: it will not use existing ports if they are not in the pool | 16:05 |
apuimedo | dulek: cause I already triggered the workflow xD | 16:05 |
ltomasbo | so, if a request is sent before the ready, a new port creation will be triggered | 16:05 |
dulek | apuimedo: Oh, okay, sure. :) | 16:06 |
ltomasbo | umm, I think I need to read the backlog... | 16:06 |
dulek | ltomasbo: Ah, so it's fine. I'd leave the readiness probe as is then. | 16:06 |
apuimedo | ltomasbo: the question is whether that may fuck up the recovery | 16:06 |
apuimedo | or the recovery may take over the ports of the new request | 16:06 |
ltomasbo | apuimedo, you mean if we create a pod before the pool is loaded? | 16:07 |
apuimedo | ltomasbo: right | 16:07 |
ltomasbo | apuimedo, it will not still a port, as it can only get existing ports from the pool, otherwise it will call neutron for a new port | 16:08 |
apuimedo | s/still/steal/ ? | 16:08 |
ltomasbo | yep | 16:08 |
apuimedo | ltomasbo: IIRC, the way you did it | 16:08 |
apuimedo | the port pool will not be available until it is all processed | 16:09 |
apuimedo | do I remember well? | 16:09 |
*** salv-orlando has quit IRC | 16:09 | |
ltomasbo | no, I don't think so, it is including the ports sequentially | 16:10 |
ltomasbo | so, for the nested case, it will populate the ports pool by pool, i.e., trunk by trunk | 16:10 |
apuimedo | ltomasbo: that means, then, that a request may succeed while the ports are being recovered | 16:10 |
ltomasbo | a request will succeed anyway, the only different is whether it will get a port from the pool, or create a new one | 16:11 |
dulek | ltomasbo: I can imagine a race condition here: | 16:12 |
dulek | ltomasbo: 1. Recovery starts. | 16:12 |
dulek | ltomasbo: Okay, even better. | 16:12 |
ltomasbo | the recovery action is a different eventlet thread | 16:12 |
dulek | ltomasbo: 1. ADD request comes, Neutron port gets created and starts to get annotated into pod. | 16:12 |
dulek | ltomasbo: 2. Recovery starts. Fetches all the ports from Neutron. | 16:13 |
ltomasbo | https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/vif_pool.py#L110-L115 | 16:13 |
dulek | ltomasbo: 3. Port from point 1 gets added into pool as unreserved. | 16:13 |
ltomasbo | dulek, umm, let me think about that | 16:15 |
ltomasbo | not sure if that case is possible (and it should not be if you wait for the readiness flag...) | 16:15 |
ltomasbo | so, it should not be an issue anyway | 16:16 |
ltomasbo | but, we also check from the kubernetes side, if the port id is annotated in an existing pod | 16:16 |
ltomasbo | before recovering the ports | 16:16 |
dulek | ltomasbo: I don't really know, I would need to read your code more carefully. But if there's a race condition we should start Watcher only after recovery is finished. | 16:19 |
ltomasbo | dulek, agreed | 16:21 |
apuimedo | this case is possible | 16:22 |
apuimedo | for sure | 16:22 |
*** lihi has quit IRC | 16:22 | |
apuimedo | because creating the port and it being annotated have a significant time difference | 16:23 |
*** lihi has joined #openstack-kuryr | 16:24 | |
apuimedo | ltomasbo: dulek ^^ | 16:27 |
dulek | apuimedo, ltomasbo: Okay, so we know there's a potential bug in there. | 16:28 |
apuimedo | yup | 16:29 |
*** salv-orlando has joined #openstack-kuryr | 16:31 | |
ltomasbo | got it! | 16:31 |
openstackgerrit | Merged openstack/kuryr-kubernetes master: Add readiness probe to kuryr-controller pod https://review.openstack.org/518502 | 16:31 |
apuimedo | dulek: ltomasbo: we should have health probes for the daemonset that keep track of the health of the requests connection to the k8s api | 16:34 |
apuimedo | I'll add a bug for that | 16:34 |
*** pcaruana has quit IRC | 16:48 | |
openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add oslo_cache to default_subnet driver https://review.openstack.org/519704 | 17:04 |
openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add oslo_cache to default_subnet driver https://review.openstack.org/519704 | 17:06 |
*** dougbtv__ has joined #openstack-kuryr | 17:08 | |
*** janki has quit IRC | 17:10 | |
*** yamamoto has quit IRC | 17:15 | |
*** salv-orl_ has joined #openstack-kuryr | 17:20 | |
*** yamamoto has joined #openstack-kuryr | 17:21 | |
*** salv-orlando has quit IRC | 17:22 | |
*** yamamoto has quit IRC | 17:26 | |
*** yamamoto has joined #openstack-kuryr | 17:36 | |
*** dougbtv__ has quit IRC | 17:39 | |
*** yamamoto_ has joined #openstack-kuryr | 17:40 | |
*** yamamoto has quit IRC | 17:41 | |
*** salv-orl_ has quit IRC | 17:44 | |
*** dougbtv__ has joined #openstack-kuryr | 17:45 | |
*** yamamoto_ has quit IRC | 17:45 | |
*** dougbtv_ has quit IRC | 17:48 | |
*** salv-orlando has joined #openstack-kuryr | 17:52 | |
*** yamamoto has joined #openstack-kuryr | 17:52 | |
*** salv-orlando has quit IRC | 17:54 | |
*** salv-orlando has joined #openstack-kuryr | 17:55 | |
*** salv-orl_ has joined #openstack-kuryr | 17:58 | |
*** janonymous has quit IRC | 18:00 | |
*** salv-orlando has quit IRC | 18:01 | |
*** yamamoto has quit IRC | 18:03 | |
*** dougbtv__ has quit IRC | 18:12 | |
*** salv-orl_ has quit IRC | 18:24 | |
*** salv-orlando has joined #openstack-kuryr | 18:24 | |
*** salv-orlando has quit IRC | 18:28 | |
*** yamamoto has joined #openstack-kuryr | 18:29 | |
*** garyloug has quit IRC | 18:33 | |
*** yamamoto has quit IRC | 18:35 | |
*** yamamoto has joined #openstack-kuryr | 18:36 | |
*** dougbtv__ has joined #openstack-kuryr | 18:36 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/519094 | 18:45 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 18:50 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 18:50 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 18:50 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/519094 | 18:56 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 19:04 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 19:04 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 19:05 |
*** mestery has quit IRC | 19:10 | |
*** mestery_ has joined #openstack-kuryr | 19:10 | |
*** mestery_ is now known as mestery | 19:10 | |
*** yamamoto has quit IRC | 19:11 | |
*** yboaron has joined #openstack-kuryr | 19:14 | |
*** aojea has joined #openstack-kuryr | 20:08 | |
*** yamamoto has joined #openstack-kuryr | 20:12 | |
*** pcaruana has joined #openstack-kuryr | 20:12 | |
*** yamamoto has quit IRC | 20:23 | |
*** gouthamr has quit IRC | 20:53 | |
*** salv-orlando has joined #openstack-kuryr | 21:18 | |
*** gouthamr has joined #openstack-kuryr | 21:23 | |
*** aojea has quit IRC | 21:24 | |
*** aojea has joined #openstack-kuryr | 21:24 | |
*** aojea has quit IRC | 21:29 | |
*** gouthamr has quit IRC | 21:49 | |
*** pcaruana has quit IRC | 22:02 | |
*** yboaron has quit IRC | 22:04 | |
*** pcaruana has joined #openstack-kuryr | 22:06 | |
*** pcaruana has quit IRC | 22:07 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/519094 | 22:19 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 22:22 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 22:22 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 22:22 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/519782 | 22:22 |
*** aojea has joined #openstack-kuryr | 22:25 | |
*** gouthamr has joined #openstack-kuryr | 22:26 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/519094 | 22:26 |
*** aojea has quit IRC | 22:29 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/516936 | 22:30 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/516937 | 22:30 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/509781 | 22:30 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/519782 | 22:30 |
*** pmannidi has joined #openstack-kuryr | 23:08 | |
*** aojea has joined #openstack-kuryr | 23:26 | |
*** aojea has quit IRC | 23:31 | |
*** gouthamr has quit IRC | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!