Wednesday, 2019-05-15

*** hamzy has joined #oooq00:00
*** chem has quit IRC00:03
weshayrlandy|bbl hrm.. not recreating locally w/ quickstart.sh00:13
* weshay fires up the repro00:13
rlandy|bblI saw it in the browbeat twice00:14
rlandy|bbland then on rdocloud00:14
weshayrlandy|bbl ya.. I see it all over00:14
weshaylook at 658358,4 in zuul00:14
rlandy|bblweshay; on the upside - first success with baseos00:14
weshayrlandy|bbl has to be related to python00:14
weshayf28 is working.. so python3 is not affected00:14
rlandy|bblyeah - totally a python problem00:15
* rlandy|bbl on way to gym - back later to check in - I checked tq/tqe/tripleo-ci for recent merges00:15
rlandy|bblnothing showed00:15
rlandy|bbland the product hasn't started yet00:16
rlandy|bblso it's on our side00:16
rlandy|bblor zuul00:16
weshayrlandy|bbl I think it's a pip module00:16
weshaynot sure which one00:16
rlandy|bblok - will check in when I get back - we know when it started st least00:18
rlandy|bblha - sec - let me compare two jobs00:19
weshayok.. I'm on nodepool node00:25
rlandy|bblhttps://stackoverflow.com/questions/4484872/why-doesnt-exec-work-in-a-function-with-a-subfunction00:27
rlandy|bblLoading callback plugin debug of type stdout, v2.0 from /home/zuul/workspace/.quickstart/lib/python2.7/site-packages/ansible/plugins/callback/debug.pyc00:30
rlandy|bblhttps://sf.hosted.upshift.rdu2.redhat.com/logs/31/169131/8/check/tripleo-ci-base-baremetal-microcloud-ha-2/9185c42/logs/quickstart_collect_logs.log00:30
rlandy|bblweshay: ^^00:30
weshayhrm00:31
weshayperfect, reporting bug and disabling00:32
weshayhttps://bugs.launchpad.net/tripleo/+bug/182908800:35
openstackLaunchpad bug 1829088 in tripleo "all upstream jobs failing on ansible debug, ansible/plugins/callback/debug.py" [Critical,Triaged]00:35
weshayhttps://review.opendev.org/65919100:36
weshaysee what happens w/ that00:36
rlandy|bbl2019-05-14 15:42:30.155828 | primary |   Downloading http://mirror.regionone.rdo-cloud.rdoproject.org/pypifiles/packages/18/79/84f02539cc181cdbf5ff5a41b9f52cae870b6f632767e43ba6ac70132e92/Werkzeug-0.15.2-py2.py3-none-any.whl (328kB)00:37
rlandy|bbl2019-05-14 22:52:58.124645 | primary |   Downloading http://mirror.regionone.rdo-cloud.rdoproject.org/pypifiles/packages/3d/bf/79101bd1d6a2b3fe0888e8d6e039800b173f26b7388308fc4bcc45de8d0a/Werkzeug-0.15.3-py2.py3-none-any.whl (327kB)00:37
rlandy|bblweshay: ^^ diff00:37
weshayrepro is failing silently00:38
weshay2019-05-15 00:37:39.838734 | primary | ok: 20 changed: 10 unreachable: 0 failed: 100:38
weshayprobably on 2019-05-15 00:37:39.838734 | primary | ok: 20 changed: 10 unreachable: 0 failed: 100:38
weshayoops00:38
rlandy|bblcheck versions above00:38
weshay2019-05-15 00:36:51.605099 | TASK [prepare-workspace : Synchronize src repos to workspace directory.]00:38
weshay2019-05-15 00:37:39.814876 | primary | Output suppressed because no_log was given00:38
rlandy|bblthat is diff between pasing log and failing log00:39
* rlandy|bbl really goes now00:40
weshayrlandy|bbl thanks.. ttyl00:40
weshayrlandy|bbl https://pypi.org/project/Werkzeug/#history00:52
weshayrlandy|bbl fixed it01:18
weshayhttp://zuul.openstack.org/stream/7ef3208196944770ac6105d19a05c43e?logfile=console.log01:18
*** sshnaidm|afk has quit IRC01:42
*** apetrich has quit IRC01:57
*** rlandy|bbl is now known as rlandy02:12
rlandyweshay: cool - happy we have a fix02:12
*** dsneddon has quit IRC02:14
rlandyretrying perf jobs with that dep02:15
*** dsneddon has joined #oooq02:22
*** dsneddon has quit IRC02:28
*** dsneddon has joined #oooq02:35
*** dsneddon has quit IRC02:40
*** rlandy has quit IRC02:40
*** dsneddon has joined #oooq02:44
*** dsneddon has quit IRC02:49
*** dsneddon has joined #oooq02:50
*** ykarel|away has joined #oooq03:04
*** ykarel|away is now known as ykarel03:21
*** dsneddon has quit IRC03:49
*** ykarel is now known as ykarel|afk03:54
*** udesale has joined #oooq03:58
*** ykarel|afk has quit IRC03:58
*** ykarel|afk has joined #oooq04:15
*** dsneddon has joined #oooq04:16
*** dsneddon has quit IRC04:21
*** ykarel|afk is now known as ykarel04:27
*** dsneddon has joined #oooq04:48
*** dsneddon has quit IRC04:55
*** dsneddon has joined #oooq05:24
*** ratailor has joined #oooq05:27
*** dsneddon has quit IRC05:29
*** dsneddon has joined #oooq05:30
*** gkadam has joined #oooq05:36
*** marios has joined #oooq05:40
*** jtomasek has joined #oooq05:52
*** skramaja has joined #oooq06:02
*** holser_ has joined #oooq06:27
*** dsneddon has quit IRC06:34
*** jaosorior has quit IRC06:42
*** apetrich has joined #oooq06:47
*** jaosorior has joined #oooq06:51
*** dsneddon has joined #oooq07:02
*** dsneddon has quit IRC07:07
*** tesseract has joined #oooq07:13
*** dsneddon has joined #oooq07:15
*** tosky has joined #oooq07:27
*** bogdando has joined #oooq07:27
*** dsneddon has quit IRC07:47
*** dsneddon has joined #oooq07:48
*** dsneddon has quit IRC07:54
*** ykarel is now known as ykarel|lunch07:55
*** kopecmartin|off is now known as kopecmartin08:01
*** chem has joined #oooq08:06
*** dsneddon has joined #oooq08:28
*** dtantsur|afk is now known as dtantsur08:35
*** ykarel|lunch is now known as ykarel08:54
*** derekh has joined #oooq08:57
arxcruzhey guys, I need to get back to the hospital, my daughter still with fever09:15
*** sshnaidm has joined #oooq09:21
mariosarxcruz: ack :( sorry to hear it dude hope she is better soon09:24
*** dsneddon has quit IRC09:28
pandaarxcruz: ouch again.09:29
*** sshnaidm is now known as sshnaidm|afk09:49
*** gkadam has quit IRC09:57
*** gkadam has joined #oooq09:58
pandamarios: can you rereview https://code.engineering.redhat.com/gerrit/167729  ?09:59
mariospanda: ack in bit10:01
*** dsneddon has joined #oooq10:01
*** dsneddon has quit IRC10:05
*** holser_ is now known as holser|lunch10:10
*** tesseract has quit IRC10:12
*** tesseract has joined #oooq10:12
*** skramaja has quit IRC10:32
*** dsneddon has joined #oooq10:35
*** yolanda_ has joined #oooq10:39
*** dsneddon has quit IRC10:41
mariospanda: rfolco|ruck so lets merge that then please https://review.rdoproject.org/r/#/c/20241/10:51
mariosrfolco|ruck: panda posted new review which depends-on that ^^^ @ https://review.rdoproject.org/r/#/c/20717/10:51
*** udesale has quit IRC11:08
*** udesale has joined #oooq11:09
*** holser|lunch is now known as holser_11:12
*** dsneddon has joined #oooq11:13
*** dsneddon has quit IRC11:18
*** dsneddon has joined #oooq11:49
*** dsneddon has quit IRC11:56
*** jjoyce has quit IRC11:57
*** jjoyce has joined #oooq12:00
*** panda is now known as panda|launch12:01
*** holser_ has quit IRC12:11
mariosrfolco|ruck: can you check that when you have time again please thanks https://review.rdoproject.org/r/#/c/20241/ updated per your comments. there is a new patch https://review.rdoproject.org/r/#/c/20717/ which depends on it thanks12:31
*** holser_ has joined #oooq12:32
*** dsneddon has joined #oooq12:32
*** rlandy has joined #oooq12:36
*** dsneddon has quit IRC12:36
*** gkadam has quit IRC12:43
mariosrfolco|ruck: we don't have unisprint9 filter for trello retrospective12:51
mariosthis is terribruu12:51
weshayrfolco|ruck can you prep the retro board12:54
mariosrfolco|ruck: created https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint912:55
mariosweshay: ^12:55
*** gkadam has joined #oooq12:55
mariosretro now?12:59
mariosweshay: rfolco|ruck at 5878458097 or i'm wrong place13:00
marios(thats what the calendar says anyway)13:00
mariospanda|launch: rlandy /13:00
mariosarxcruz:13:00
*** dsneddon has joined #oooq13:05
marios15:54 < weshay> rfolco|ruck can you prep the retro board13:05
marios15:55 < marios> rfolco|ruck: created https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint913:05
marios15:55 < marios> weshay: ^13:05
*** panda|launch is now known as panda13:05
*** dsneddon has quit IRC13:09
rlandyarxcruz: https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/create-zuul-based-reproducer/templates13:19
*** vinaykns has joined #oooq13:20
*** Goneri has joined #oooq13:30
*** ratailor has quit IRC13:39
*** dsneddon has joined #oooq13:39
*** aakarsh has quit IRC13:40
*** dsneddon has quit IRC13:44
*** aakarsh has joined #oooq14:02
mariosrfolco|ruck: please check that when you hav time thanks cc rlandy panda https://review.rdoproject.org/r/#/c/20241/14:03
marioser panda you already voted thanks14:03
pandamarios: yes but I'm invisible14:04
marioswho said that /me looks around14:05
*** dsneddon has joined #oooq14:12
*** ykarel is now known as ykarel|away14:12
*** jbadiapa has quit IRC14:14
*** dsneddon has quit IRC14:16
rlandymarios: 20241 merged - two questions for you (after meeting) before merging the next ...14:30
rlandyis this added to promotion crietria14:30
rlandydo we have a test run of these?14:30
*** Vorrtex has joined #oooq14:33
mariosrlandy: not added to promotion criteria and14:44
mariosrlandy: test run - they ran in check if that is what you mean, in v5 https://review.rdoproject.org/r/#/c/20241/514:45
mariosrlandy: see http://logs.rdoproject.org/41/20241/5/check/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/9486b82/ and http://logs.rdoproject.org/41/20241/5/check/periodic-tripleo-ci-centos-7-standalone-upgrade-master/88ee2f6/14:45
mariosrlandy: is that sufficient ?14:46
rlandymarios: testing is sufficient ... do you want these in criteria?14:47
rlandyif so, we are missing a review14:47
*** dsneddon has joined #oooq14:48
*** ykarel|away is now known as ykarel14:48
mariosrlandy: that can be follow up review no?14:48
mariosrlandy: maybe next sprint14:48
rlandymarios: ack - diff repo14:48
mariosrlandy: we can track adding it into promotion critiera14:48
mariosrlandy: its enough for now that we added those jobs into promotion pipeline so they're running14:48
mariosrlandy: but yeah, we probably want to add at least the stein one14:48
mariosrlandy: into critiria14:49
marioscriteria14:49
rlandymarios: fine - I can add the criteria review14:49
mariosrlandy: cool thanks add me if you do i'll vote on it in the morning14:49
rlandymarios: ack ... https://review.rdoproject.org/r/#/c/20717/ w+14:49
mariosty14:49
ykarelIs the zuul based reproducer working? anyone used recently?14:51
ykarelme tried to reproducer fs039 ovb job and it failed in pre run14:52
ykarelsaying some roles can't be found14:52
*** dsneddon has quit IRC14:52
ykarelmarios, rlandy weshay ^^ any idea?14:52
weshayykarel I saw the same last night14:53
ykarelERROR! the role 'emit-job-header' was not found in /tmp/tmp_ocmxjeh/5c353604cec943f78ecd4e2c208adb24/trusted/project_0/gerrit/zuul-config/playbooks/base/roles:/tmp/tmp_ocmxjeh/5c353604cec943f78ecd4e2c208adb24/ansible/pre_playbook_0/role_0/zuul-config/roles:/tmp/tmp_ocmxjeh/5c353604cec943f78ecd4e2c208adb24/ansible/pre_playbook_0/role_1/rdo-jobs/roles:/tmp/tmp_ocmxjeh/5c353604cec943f78ecd4e2c208adb24/trusted/project_0/gerrit/zuul-config/playbooks14:53
ykarel/base14:53
weshaywas reloading my box to confirm the issue14:53
ykarelweshay, Thanks for confirming14:53
weshayykarel do you have a sec to chat actually?14:53
weshay3min?14:53
ykarelweshay, yes14:53
weshayhttps://bluejeans.com/u/whayutin/14:54
rlandyykarel: looks familiar - have seen that before14:54
mariosykarel: not seen that but last fri last run14:55
ykarelrlandy, marios ack15:03
rlandymarios: https://review.rdoproject.org/r/#/c/20720/15:09
rlandypls review15:09
rlandymarios: suggestion - not to merge this until we see the jobs run successfully twice in the pipelines15:10
rlandyhttps://tree.taiga.io/project/tripleo-ci-board/task/990?kanban-status=1447276 updated15:11
*** marios has quit IRC15:14
weshaysshnaidm|afk for tomorrow can you try a fresh run of the repro15:15
*** tosky has quit IRC15:16
weshayykarel and I are seeing it's busted atm15:16
weshayperhaps we need to add a few jobs to the periodic pipeline and add the results to the cockpit as well15:17
*** tosky has joined #oooq15:18
*** dsneddon has joined #oooq15:19
weshayrfolco|ruck who is my new ruck/rover?15:19
weshayGabriele and Marios15:20
rfolco|ruckweshay, panda (and marios)15:20
*** marios has joined #oooq15:21
weshaypanda please renick and please update the cockpit to point to the new sprint etherpad15:22
rfolco|ruckweshay, I am creating a poll for taiga. Would you like a poll for ruck/rover in the terms proposed in the mtg ?15:22
*** rfolco|ruck is now known as rfolco15:22
weshaypanda I can show you how if you need help15:22
weshayrfolco share it w/ me first please15:22
rfolcoweshay, sure15:22
rfolcoweshay, I can show panda how to hack cockpit15:22
weshayrfolco panda make the txt a url this time please15:23
mariosrfolco: rlandy chkumar|pto reviews merged done? https://tree.taiga.io/project/tripleo-ci-board/task/990 (thanks rlandy for final push!)15:23
*** dsneddon has quit IRC15:23
weshaymarios thanks for your help yesterday.. this is what we needed https://review.opendev.org/#/c/659121/15:23
*** aakarsh has quit IRC15:24
mariosweshay: ack15:24
weshayrfolco can you please set up the etherpad for tomorrow15:25
rfolcoweshay, will do in a bit15:25
* marios shutdown sequence 15:26
*** panda is now known as panda|ruck15:26
*** ykarel is now known as ykarel|away15:28
weshaymarios I sent the ack on the invoice15:30
mariosweshay: ack thx15:30
mariosdoing15:30
sshnaidm|afkweshay, I'll try to run it with podman too.. just testing15:33
*** tesseract has quit IRC15:35
*** ykarel|away has quit IRC15:36
*** bogdando has quit IRC15:51
*** marios has quit IRC15:54
*** ccamacho has quit IRC15:56
*** dsneddon has joined #oooq15:57
*** dsneddon has quit IRC16:02
*** kopecmartin is now known as kopecmartin|off16:05
weshayarxcruz you still around?16:08
arxcruzweshay:yes16:10
weshayarxcruz seems to me before asking the compute team for help, we should get the qemu-guest-agent deployed w/ standalone re: https://bugs.launchpad.net/tripleo/+bug/182431516:10
openstackLaunchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell)16:10
arxcruzi though this was already fixed16:11
arxcruzweshay:well, not on os-tempest side for sure16:12
arxcruzso, on standalone deployment16:12
arxcruzweshay: did we hit this again ?16:13
*** tosky has quit IRC16:13
weshayit was never fixed16:14
weshayafaik16:14
arxcruzbut we weren't hitting it anymore16:15
arxcruzi wasn't able to reproduce locally also16:15
weshayit's not consistent, you know that :)  I added a fresh failure16:15
weshayI think all we need to do is add the rpm16:15
arxcruzi hope so :)16:19
arxcruzthat would be in the nova-api container right ? or nova-compute container16:20
*** tesseract has joined #oooq16:24
weshayarxcruz would it be installed on the container?16:34
weshayor bm16:34
arxcruzcontainer16:34
arxcruzbecause it's where the vm is get spawned right ?16:34
weshayhttps://github.com/openstack/tripleo-common/blob/master/container-images/tripleo_kolla_template_overrides.j2#L5316:36
*** dsneddon has joined #oooq16:36
*** dsneddon has quit IRC16:41
*** dsneddon has joined #oooq16:56
weshayarxcruz discussing this in #openstack-nova16:57
weshayarxcruz context http://paste.openstack.org/show/751432/16:59
weshayarxcruz sounds like the error may be a red herring17:00
*** dtantsur is now known as dtantsur|afk17:01
arxcruzweshay: i see17:01
*** derekh has quit IRC17:02
weshayarxcruz I've been suspicious of https://github.com/openstack/tripleo-quickstart-extras/blob/a466cc6a2018f64e118024f9f8bd1bb1997a0310/playbooks/multinode-standalone-upgrade.yml#L5717:03
weshayer wait17:03
weshayarxcruz https://github.com/openstack/tripleo-quickstart-extras/blob/a466cc6a2018f64e118024f9f8bd1bb1997a0310/playbooks/multinode-standalone.yml#L3417:03
arxcruzweshay: the first one we change due other issue right ?17:04
arxcruzweshay: we changed for qemu because it was requiring qemu-somethingthaticantremember17:04
arxcruzthat was failing17:04
weshayhttps://github.com/openstack/tripleo-quickstart-extras/blob/08e183aa15f703aece72a3a469e39e1e8e8b3fc6/roles/standalone/defaults/main.yml#L3617:06
weshayhowever I don't think we set use it17:07
weshayhttp://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/undercloud/var/log/libvirt/qemu/instance-0000000a.log.txt.gz17:08
arxcruzi see17:09
weshayarxcruz http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/undercloud/etc/libvirt/qemu.conf.txt.gz17:12
weshayarxcruz although the module is loaded http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/undercloud/var/log/extra/lsmod.txt.gz17:14
weshayso maybe it is using kvm17:15
arxcruzweshay: i don't know, the module loaded doesn't means it's using right?17:15
arxcruzalso, the qemu.conf is comented17:16
weshayarxcruz well.. I didn't think so but I'm trolling the doc and I don't see that you have to update etc/libvirt anywhere17:16
arxcruzall17:16
arxcruzweshay: i can ask my friend who works on qemu/kvm tomorrow17:16
*** udesale has quit IRC17:21
*** ykarel|away has joined #oooq17:24
weshayarxcruz f.. it closing the bug17:24
*** holser_ has quit IRC17:24
weshayrlandy afaict.. ovb jobs are starting to run long17:24
weshay2hr 55 min17:25
rlandyweshay:they have 3.5 hrs. no?17:25
rlandylooking at undercloud deploy failures now though17:25
rlandyin promotion17:25
rlandyI see them in bm17:25
rlandymaster17:25
weshayya.. I'm just saying.. something has changed causing jobs to run longer17:25
arxcruzweshay: i'll take a look at it later :)17:26
rlandyweshay: you were right about the image download17:26
rlandythat was killing my job time17:26
weshayah k17:26
rlandybut I am living on the edge17:26
weshayrlandy k.. the nice thing about baseos is that you never have to redownload that until centos updates and we update the config17:26
rlandycorrect17:26
rlandywe will expect the first job in each env to timeout17:27
rlandynot ideal to increase max time if we don't really need it17:27
weshayk17:27
rlandyweshay: 1-on-1?17:30
weshayrlandy aye17:31
rlandyjoined bj17:31
*** ykarel|away has quit IRC17:32
*** aakarsh has joined #oooq17:53
*** tesseract has quit IRC18:01
rlandyneed to drop something off at post office - brb18:12
*** rlandy is now known as rlandy|brb18:12
*** Goneri has quit IRC18:14
arxcruzit seems we have a limit for ssh review.opendev.org18:43
arxcruzReceived disconnect from 104.130.246.32 port 29418:12: Too many concurrent connections (64) - max. allowed: 6418:43
*** tosky has joined #oooq18:43
*** rlandy|brb is now known as rlandy18:45
weshayrlandy fyi https://bugs.launchpad.net/tripleo/+bug/182928218:47
openstackLaunchpad bug 1829282 in tripleo "glance-api broken on wsme json import change" [Critical,Triaged] - Assigned to wes hayutin (weshayutin)18:47
rlandyweshay: thanks18:50
*** dsneddon has quit IRC18:54
*** Goneri has joined #oooq19:18
*** dsneddon has joined #oooq19:19
*** dsneddon has quit IRC19:32
*** dsneddon has joined #oooq19:43
rlandyweshay: do we want to revert this per ykarel's comment: https://review.opendev.org/#/c/659191?19:44
weshayrlandy ya.. let's put up a revert and see if it works19:45
*** holser_ has joined #oooq19:48
weshayhttps://review.opendev.org/#/c/659367/19:48
rlandyack19:50
rlandylet's see what version the tests get19:51
*** dsneddon has quit IRC19:53
rlandyweshay: to bug you again ... opinion pls ... closing out review comments on baseos ... https://review.opendev.org/#/c/648519/27/playbooks/baremetal-prep-virthost.yml - what do you think of the role name? change it per marios comments or not? I prefer to keep the playbook name unchanged as not to impact ovb tests19:57
* weshay looks19:57
weshayrlandy I would use the original name for the playbook... to_build_or_not_to_build19:59
weshayrlandy that way no one can blame you for the name :)19:59
rlandyweshay: I am keeping the playbook name and just naming the role20:00
weshayrlandy you can do that kind of thing in a follow patch as well20:00
rlandywhich I added 'check-' to20:00
weshayif you want/need to be done w/ that patch20:00
rlandyfine20:00
rlandywith apologies to Shakespeare20:00
weshayheh20:00
*** Vorrtex has quit IRC20:05
weshayrlandy master is blocked until the updated package gets released to consistent20:12
rlandymarios: pls revote ... https://review.opendev.org/#/c/648519/20:12
rlandyweshay: understand - if I know the error is the same as ovb, I'll just continue to merge BM-related patches20:13
*** rlandy has quit IRC20:15
*** rlandy has joined #oooq20:16
*** dsneddon has joined #oooq20:19
*** dsneddon has quit IRC20:25
rlandyweshay: testing rocky with baseos ... https://code.engineering.redhat.com/gerrit/#/c/165815/- to compare run times20:53
weshayk20:53
*** dsneddon has joined #oooq20:54
*** dsneddon has quit IRC20:58
*** dtrainor has quit IRC21:02
*** dtrainor has joined #oooq21:02
*** Goneri has quit IRC21:07
*** Goneri has joined #oooq21:07
*** panda|ruck has quit IRC21:10
*** panda has joined #oooq21:14
*** dsneddon has joined #oooq21:15
*** vinaykns has quit IRC21:22
*** holser_ has quit IRC21:50
*** Goneri has quit IRC22:14
*** rlandy is now known as rlandy|bbl22:18
*** dsneddon has quit IRC22:25
*** holser_ has joined #oooq22:45
*** dsneddon has joined #oooq22:52
*** dsneddon has quit IRC22:57
*** tosky has quit IRC23:00
*** holser_ has quit IRC23:12
*** panda has quit IRC23:30
*** dsneddon has joined #oooq23:31
*** panda has joined #oooq23:31
*** dsneddon has quit IRC23:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!