*** ansmith has joined #openstack-infra | 00:01 | |
ianw | clarkb: it does super() from the openstacksdk version? -> https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/task_manager.py#n50 | 00:01 |
---|---|---|
clarkb | only for init, not post_run_task or submit_task etc | 00:02 |
clarkb | submit_task is the one that matters here I think? | 00:02 |
corvus | ianw: +2 on the ansible changes (with a comment that override-checkout shouldn't be necessary since https://review.openstack.org/616314 merged) | 00:03 |
ianw | clarkb: ohhh, yes i see what you're saying, the nodepool submit_task is it's own thing and has overridden it | 00:03 |
corvus | ianw: i didn't +3 due to eod; feel free to self-approve | 00:03 |
clarkb | ianw: yup | 00:03 |
ianw | clarkb: first thought is i wonder if that needs to be overridden | 00:05 |
ianw | oh, it rasies a slightly different exception | 00:06 |
clarkb | ianw: functionally though both put the task on the queue and wait for it to complete then return the result | 00:07 |
*** dhellmann has joined #openstack-infra | 00:07 | |
clarkb | it is possible that we could use submit_task from sdk I think | 00:07 |
clarkb | both inits manage self.queue which is weird but should be a non issue by the time submit_ask is called | 00:08 |
ianw | yep, possibly the sdk should take a new argument which is the exception to raise | 00:08 |
clarkb | or update nodepool to handle the sdk exception? | 00:09 |
ianw | or, we could wrap nodepool and re-raise the right one ... that might be better | 00:09 |
clarkb | ya that would be one way of doing that | 00:09 |
ianw | ok, will try that | 00:11 |
clarkb | ok not sure if I should be happy that log server stopped having problems are sad that as soon as I feel like I'm somewhat set up to watch it real time it decides to be done | 00:11 |
clarkb | fungi: did you see the etherpad for your forum session? | 00:12 |
clarkb | fungi: any other prep work you think I/we should be doing for that one? | 00:12 |
*** sthussey has quit IRC | 00:30 | |
dmsimard | huh, no other spikes | 00:31 |
clarkb | dmsimard: ya i think we watch it and look at swift after the summit | 00:32 |
clarkb | if it isn't getting worse then its pretty happy right this moment | 00:32 |
*** dhill_ has quit IRC | 00:41 | |
*** longkb has joined #openstack-infra | 00:46 | |
*** calebb has quit IRC | 00:50 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task https://review.openstack.org/616387 | 00:51 |
*** agopi is now known as agopi|brb | 00:54 | |
*** gyee has quit IRC | 00:55 | |
*** agopi|brb has quit IRC | 00:59 | |
fungi | clarkb: thanks! to be honest, i forgot "i" had a forum session ;) | 01:07 |
fungi | happy to moderate it | 01:07 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: pin minikube version by default https://review.openstack.org/616388 | 01:12 |
*** bobh has joined #openstack-infra | 01:12 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: DNM: test older minikube version https://review.openstack.org/616389 | 01:13 |
fungi | clarkb: i proposed that session by proxy, to be honest | 01:16 |
fungi | though i did flesh out the description a bit | 01:16 |
fungi | not aware of any prep we need to be doing | 01:17 |
fungi | just wanted to be sure opendev had a session reserved to get feedback | 01:17 |
fungi | we can really use it however we want | 01:17 |
*** felipemonteiro has joined #openstack-infra | 01:19 | |
*** Swami has quit IRC | 01:24 | |
*** bobh has quit IRC | 01:28 | |
*** yamamoto has joined #openstack-infra | 01:45 | |
*** mriedem has quit IRC | 01:50 | |
*** agopi|brb has joined #openstack-infra | 01:52 | |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: [wip] add docker package pin option https://review.openstack.org/616391 | 01:52 |
*** diablo_rojo has quit IRC | 01:53 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning https://review.openstack.org/616388 | 01:56 |
*** rkukura has quit IRC | 01:56 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role https://review.openstack.org/605585 | 01:58 |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role https://review.openstack.org/605585 | 02:06 |
*** hongbin has joined #openstack-infra | 02:13 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role https://review.openstack.org/605585 | 02:29 |
*** fuentess has quit IRC | 02:33 | |
*** felipemonteiro has quit IRC | 02:33 | |
*** mrsoul has joined #openstack-infra | 02:44 | |
*** markvoelker has joined #openstack-infra | 02:45 | |
*** markvoelker has quit IRC | 02:45 | |
*** markvoelker has joined #openstack-infra | 02:45 | |
*** yamamoto has quit IRC | 02:49 | |
*** bobh has joined #openstack-infra | 02:49 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dmn] test pinning docker version https://review.openstack.org/616398 | 02:50 |
*** markvoelker has quit IRC | 02:50 | |
*** felipemonteiro has joined #openstack-infra | 02:55 | |
*** markvoelker has joined #openstack-infra | 02:55 | |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: [wip] add docker package pin option https://review.openstack.org/616391 | 02:56 |
*** anteaya has quit IRC | 03:02 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [dmn] test pinning docker version https://review.openstack.org/616398 | 03:04 |
*** fungi has quit IRC | 03:06 | |
*** fungi has joined #openstack-infra | 03:09 | |
*** fungi has quit IRC | 03:10 | |
*** larainema has joined #openstack-infra | 03:12 | |
*** armax has quit IRC | 03:18 | |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: install-docker: add package version option https://review.openstack.org/616391 | 03:18 |
*** mrhillsman has joined #openstack-infra | 03:19 | |
*** armax has joined #openstack-infra | 03:19 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Pin docker for k8s test https://review.openstack.org/616398 | 03:21 |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task https://review.openstack.org/616387 | 03:21 |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test" https://review.openstack.org/616404 | 03:21 |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Pin docker for k8s test https://review.openstack.org/616398 | 03:24 |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task https://review.openstack.org/616387 | 03:24 |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test" https://review.openstack.org/616404 | 03:24 |
*** armax has quit IRC | 03:25 | |
*** felipemonteiro has quit IRC | 03:35 | |
*** felipemonteiro has joined #openstack-infra | 03:36 | |
*** armax has joined #openstack-infra | 03:38 | |
*** fungi has joined #openstack-infra | 03:40 | |
*** fungi has quit IRC | 03:41 | |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: install-docker: add package version option https://review.openstack.org/616391 | 03:43 |
*** fungi has joined #openstack-infra | 03:45 | |
*** bobh has quit IRC | 03:55 | |
*** armax has quit IRC | 03:59 | |
*** rlandy|bbl is now known as rlandy | 03:59 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Remove unused nodepool-k8s-functional role https://review.openstack.org/616409 | 04:05 |
*** bobh has joined #openstack-infra | 04:08 | |
*** bobh has quit IRC | 04:10 | |
*** sambetts|afk has quit IRC | 04:11 | |
*** felipemonteiro has quit IRC | 04:11 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Remove nodepool-k8s-functional and install-nodepool roles https://review.openstack.org/616409 | 04:24 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 04:29 |
*** janki has joined #openstack-infra | 04:36 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Import install-docker role https://review.openstack.org/605585 | 04:39 |
ianw | clarkb / mordred / corvus : ^ this was the role i mentioned i needed to loop back on in the meeting | 04:39 |
*** apetrich has quit IRC | 04:45 | |
*** apetrich has joined #openstack-infra | 04:46 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 04:49 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 04:52 |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Import install-docker role https://review.openstack.org/605585 | 05:29 |
*** yamamoto has joined #openstack-infra | 05:45 | |
*** hongbin has quit IRC | 06:02 | |
*** kjackal has joined #openstack-infra | 06:03 | |
*** pcrews has quit IRC | 06:20 | |
*** diablo_rojo has joined #openstack-infra | 06:20 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 06:26 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 06:29 |
*** pcrews has joined #openstack-infra | 06:33 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats https://review.openstack.org/613680 | 06:35 |
*** pcrews has quit IRC | 06:57 | |
*** roman_g has joined #openstack-infra | 06:57 | |
*** ifat_afek has joined #openstack-infra | 06:59 | |
*** quiquell|off is now known as quiquell | 07:03 | |
*** dpawlik has joined #openstack-infra | 07:04 | |
openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: install-docker: add package version option https://review.openstack.org/616391 | 07:10 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning https://review.openstack.org/616388 | 07:11 |
*** gfidente has joined #openstack-infra | 07:13 | |
*** aojea has quit IRC | 07:15 | |
AJaeger | ianw: Just fixed some nits ^ | 07:16 |
*** aojea has joined #openstack-infra | 07:20 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 07:23 |
*** pcaruana has joined #openstack-infra | 07:34 | |
*** slaweq has joined #openstack-infra | 07:39 | |
*** yamamoto has quit IRC | 07:46 | |
*** yamamoto has joined #openstack-infra | 07:46 | |
*** yamamoto has quit IRC | 07:47 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 07:47 |
ianw | AJaeger: thanks :) | 07:48 |
*** quiquell is now known as quiquell|brb | 07:49 | |
*** ginopc has joined #openstack-infra | 07:58 | |
*** bnemec has quit IRC | 08:00 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 08:02 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 08:02 |
*** carl_cai has joined #openstack-infra | 08:05 | |
openstackgerrit | Luka Peschke proposed openstack-infra/irc-meetings master: Creating an IRC meeting for CloudKitty https://review.openstack.org/616205 | 08:06 |
*** diablo_rojo has quit IRC | 08:08 | |
*** florianf|afk is now known as florianf | 08:10 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode https://review.openstack.org/616450 | 08:14 |
*** ralonsoh has joined #openstack-infra | 08:17 | |
*** quiquell|brb is now known as quiquell | 08:17 | |
*** jtomasek has joined #openstack-infra | 08:18 | |
*** bnemec has joined #openstack-infra | 08:26 | |
*** shrasool has joined #openstack-infra | 08:30 | |
*** yamamoto has joined #openstack-infra | 08:34 | |
*** jpena|off is now known as jpena | 08:36 | |
*** noama has joined #openstack-infra | 08:38 | |
*** yamamoto has quit IRC | 08:40 | |
openstackgerrit | Merged openstack-infra/irc-meetings master: Add ralonsoh to chairing neutron-qos meeting https://review.openstack.org/616208 | 08:42 |
*** yamamoto has joined #openstack-infra | 08:46 | |
*** roman_g has quit IRC | 08:46 | |
*** yamamoto has quit IRC | 08:47 | |
openstackgerrit | Merged openstack-infra/irc-meetings master: Creating an IRC meeting for CloudKitty https://review.openstack.org/616205 | 08:47 |
openstackgerrit | Merged openstack-infra/irc-meetings master: add day_specifier from recurrence https://review.openstack.org/615270 | 08:47 |
*** yamamoto has joined #openstack-infra | 08:49 | |
*** sshnaidm|afk is now known as sshnaidm|rover | 08:54 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode https://review.openstack.org/616450 | 09:07 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add extra safety belt when reusing a node https://review.openstack.org/616465 | 09:07 |
*** roman_g has joined #openstack-infra | 09:10 | |
*** shrasool has quit IRC | 09:11 | |
*** yamamoto has quit IRC | 09:13 | |
*** jpich has joined #openstack-infra | 09:14 | |
*** rkukura has joined #openstack-infra | 09:22 | |
*** shardy has joined #openstack-infra | 09:22 | |
*** chem has quit IRC | 09:24 | |
*** ifat_afek has quit IRC | 09:28 | |
*** derekh has joined #openstack-infra | 09:32 | |
*** lucasagomes has joined #openstack-infra | 09:35 | |
*** CrayZee has joined #openstack-infra | 09:48 | |
*** CrayZee is now known as Guest31212 | 09:48 | |
*** Guest31212 has quit IRC | 09:49 | |
*** e0ne has joined #openstack-infra | 09:50 | |
*** ssbarnea has joined #openstack-infra | 09:50 | |
*** snapiri- has joined #openstack-infra | 09:52 | |
*** yamamoto has joined #openstack-infra | 09:54 | |
*** dtantsur|afk is now known as dtantsur | 09:55 | |
openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: wip: Try optimizing node lock check https://review.openstack.org/616484 | 09:58 |
*** CrayZee- has joined #openstack-infra | 09:59 | |
*** CrayZee- has quit IRC | 09:59 | |
*** yamamoto has quit IRC | 10:00 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 10:01 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode https://review.openstack.org/616450 | 10:01 |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add extra safety belt when reusing a node https://review.openstack.org/616465 | 10:01 |
*** yamamoto has joined #openstack-infra | 10:04 | |
*** shardy has quit IRC | 10:07 | |
*** shardy has joined #openstack-infra | 10:09 | |
*** electrofelix has joined #openstack-infra | 10:10 | |
*** longkb has quit IRC | 10:12 | |
*** jrist has quit IRC | 10:14 | |
*** shardy has quit IRC | 10:14 | |
*** jrist has joined #openstack-infra | 10:16 | |
*** shardy has joined #openstack-infra | 10:19 | |
*** snapiri- has quit IRC | 10:19 | |
*** snapiri- has joined #openstack-infra | 10:20 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: install-docker: add package version option https://review.openstack.org/616391 | 10:23 |
*** CrayZee has joined #openstack-infra | 10:29 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning https://review.openstack.org/616388 | 10:30 |
*** panda|off is now known as panda | 10:31 | |
*** Emine has joined #openstack-infra | 10:32 | |
*** snapiri- has quit IRC | 10:32 | |
*** shardy has quit IRC | 10:38 | |
*** efried has quit IRC | 10:38 | |
*** efried has joined #openstack-infra | 10:38 | |
*** shardy has joined #openstack-infra | 10:39 | |
*** ginopc has quit IRC | 10:43 | |
*** ginopc has joined #openstack-infra | 10:43 | |
*** shardy has quit IRC | 10:44 | |
*** CrayZee_ has joined #openstack-infra | 11:00 | |
*** CrayZee has quit IRC | 11:02 | |
*** ifat_afek has joined #openstack-infra | 11:02 | |
*** larainema has quit IRC | 11:05 | |
*** yamamoto has quit IRC | 11:10 | |
*** yamamoto has joined #openstack-infra | 11:14 | |
*** CrayZee_ has quit IRC | 11:31 | |
*** carl_cai has quit IRC | 11:35 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats https://review.openstack.org/613680 | 11:38 |
*** ChosSimbaOne has quit IRC | 11:49 | |
quiquell | AJaeger, ianw: At zuul centos-7 images when do we install python-pip ? | 11:54 |
quiquell | AJaeger, ianw: Do the images already have it ? | 11:55 |
*** xek__ is now known as xek | 12:02 | |
*** jpena is now known as jpena|lunch | 12:03 | |
*** emine__ has joined #openstack-infra | 12:08 | |
*** pbourke has quit IRC | 12:11 | |
*** Emine has quit IRC | 12:11 | |
*** pbourke has joined #openstack-infra | 12:11 | |
*** jcoufal has joined #openstack-infra | 12:13 | |
*** kjackal has quit IRC | 12:13 | |
*** kjackal has joined #openstack-infra | 12:19 | |
*** janki has quit IRC | 12:20 | |
*** quiquell is now known as quiquell|lunch | 12:24 | |
*** ansmith has quit IRC | 12:31 | |
*** dtantsur is now known as dtantsur|brb | 12:32 | |
*** jcoufal has quit IRC | 12:35 | |
*** dhill_ has joined #openstack-infra | 12:36 | |
openstackgerrit | Merged openstack-infra/os-loganalyze master: Adds libmagic on plaforms missing it https://review.openstack.org/616266 | 12:40 |
openstackgerrit | Merged openstack-infra/os-loganalyze master: Move zuul job definition inside the respository https://review.openstack.org/616273 | 12:40 |
*** rkukura has quit IRC | 12:47 | |
*** rlandy has joined #openstack-infra | 12:51 | |
*** AJaeger_ has joined #openstack-infra | 12:52 | |
*** ccamacho has quit IRC | 12:53 | |
*** AJaeger has quit IRC | 12:55 | |
*** jcoufal has joined #openstack-infra | 12:55 | |
openstackgerrit | Merged openstack-infra/project-config master: Move os-loganalize job definition inside the project https://review.openstack.org/616275 | 12:56 |
*** fuentess has joined #openstack-infra | 12:56 | |
openstackgerrit | Merged openstack-infra/nodepool master: Only set basepython once https://review.openstack.org/615942 | 12:57 |
*** boden has joined #openstack-infra | 12:58 | |
*** ifat_afek has quit IRC | 12:58 | |
*** panda is now known as panda|off | 12:59 | |
*** weshay is now known as weshay_ruck | 12:59 | |
*** ifat_afek has joined #openstack-infra | 13:00 | |
*** emine__ has quit IRC | 13:00 | |
*** emine__ has joined #openstack-infra | 13:00 | |
*** e0ne has quit IRC | 13:03 | |
*** yamamoto has quit IRC | 13:07 | |
*** yamamoto has joined #openstack-infra | 13:09 | |
*** dpawlik has quit IRC | 13:11 | |
*** yamamoto has quit IRC | 13:14 | |
fungi | quiquell|lunch: we have a role for installing them, but at the moment it mostly does nothing because we preinstall them in our images via dib (it would be nice to change that soon) | 13:17 |
*** ccamacho has joined #openstack-infra | 13:18 | |
*** trown|outtypewww is now known as trown | 13:20 | |
*** jpena|lunch is now known as jpena | 13:24 | |
*** ifat_afek has quit IRC | 13:24 | |
fungi | infra-root: just a heads up, i don't anticipate being around much today. have a few pre-travel errands to which i must attend before i begin my road movie to berlin in the morning | 13:26 |
fungi | (can't drive out the way we drove in!) | 13:26 |
*** dpawlik has joined #openstack-infra | 13:26 | |
*** eharney has joined #openstack-infra | 13:27 | |
*** ifat_afek has joined #openstack-infra | 13:28 | |
*** janki has joined #openstack-infra | 13:29 | |
openstackgerrit | Andriy Shevchenko proposed openstack/ansible-role-cloud-launcher master: Update min tox version to 2.0 https://review.openstack.org/611745 | 13:29 |
*** ifat_afek has quit IRC | 13:31 | |
*** dpawlik has quit IRC | 13:31 | |
*** e0ne has joined #openstack-infra | 13:37 | |
*** kgiusti has joined #openstack-infra | 13:40 | |
*** quiquell|lunch is now known as quiquell | 13:41 | |
ssbarnea | who can help me adding ssbarnea rfolco chandan to elastic-recehck https://review.openstack.org/#/admin/groups/218,members so we can take care of unclassified failures? | 13:49 |
*** rh-jelabarre has joined #openstack-infra | 13:51 | |
*** ansmith has joined #openstack-infra | 13:51 | |
ssbarnea | fungi: AJaeger_ ^ can you help? | 13:51 |
*** ccamacho has quit IRC | 13:52 | |
AJaeger_ | ssbarnea: I don't have those permissions - better ask clarkb as PTL | 13:52 |
*** ccamacho has joined #openstack-infra | 13:53 | |
ssbarnea | AJaeger_: i almost forgot you are not on infra-core :) | 13:54 |
*** AJaeger_ has quit IRC | 13:54 | |
*** jcoufal has quit IRC | 13:55 | |
ssbarnea | quiquell: regarding pip availability: if I remember well pip is not installed from rpm, is installed using easy_install. Still, you should write code that works with any flavor of pip, regardless on how it ended up installed ;) | 13:55 |
quiquell | ssbarnea: Just want to know if it's cooked in the image or not, and looks like it's cooke with dib | 13:57 |
*** AJaeger has joined #openstack-infra | 13:57 | |
*** dpawlik has joined #openstack-infra | 13:58 | |
*** yamamoto has joined #openstack-infra | 13:58 | |
*** dpawlik has quit IRC | 13:59 | |
fungi | ssbarnea: i recommend mriedem, mtreinish or clarkb work with you as they've been the ones active on classifying failures in elastic-recheck lately. i'm not super familiar with the workflow so not sure how to confirm you are either | 13:59 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove setup.py check from check-python-release role https://review.openstack.org/616550 | 13:59 |
dhellmann | AJaeger : ^^ | 14:00 |
fungi | ssbarnea: i take it you're seeking membership in that group so that you can approve proposed logstash queries? | 14:00 |
fungi | (after all, anyone can propose queries to improve the classification rate, they just need a reviewer to approve them once proposed) | 14:01 |
*** ccamacho has quit IRC | 14:02 | |
*** pbourke has quit IRC | 14:02 | |
*** ssbarnea has quit IRC | 14:02 | |
*** derekh has quit IRC | 14:02 | |
*** jpich has quit IRC | 14:02 | |
*** roman_g has quit IRC | 14:02 | |
*** bnemec has quit IRC | 14:02 | |
*** jtomasek has quit IRC | 14:02 | |
*** aojea has quit IRC | 14:02 | |
*** kota_ has quit IRC | 14:02 | |
*** kukacz has quit IRC | 14:02 | |
*** rascasoft has quit IRC | 14:02 | |
*** auristor has quit IRC | 14:02 | |
*** niska has quit IRC | 14:02 | |
*** tdasilva has quit IRC | 14:02 | |
*** trown has quit IRC | 14:02 | |
*** jaosorior has quit IRC | 14:02 | |
*** bdodd has quit IRC | 14:02 | |
*** bstinson has quit IRC | 14:02 | |
*** pleia2 has quit IRC | 14:02 | |
*** tonyb has quit IRC | 14:02 | |
*** scas has quit IRC | 14:02 | |
*** dmellado has quit IRC | 14:02 | |
*** hwoarang has quit IRC | 14:02 | |
*** eandersson has quit IRC | 14:02 | |
*** mmethot has quit IRC | 14:02 | |
*** dabukalam has quit IRC | 14:02 | |
*** mattoliverau has quit IRC | 14:02 | |
*** tobiash has quit IRC | 14:02 | |
*** rook has quit IRC | 14:02 | |
*** pbourke has joined #openstack-infra | 14:02 | |
*** hwoarang has joined #openstack-infra | 14:02 | |
*** roman_g has joined #openstack-infra | 14:02 | |
*** bstinson_ has joined #openstack-infra | 14:02 | |
*** yamamoto has quit IRC | 14:03 | |
*** aojea has joined #openstack-infra | 14:03 | |
*** jpich has joined #openstack-infra | 14:03 | |
*** dmellado has joined #openstack-infra | 14:04 | |
*** jtomasek has joined #openstack-infra | 14:04 | |
*** dpawlik has joined #openstack-infra | 14:04 | |
*** kukacz has joined #openstack-infra | 14:04 | |
*** mmethot has joined #openstack-infra | 14:04 | |
*** bdodd has joined #openstack-infra | 14:04 | |
*** aojea has quit IRC | 14:04 | |
*** kota_ has joined #openstack-infra | 14:04 | |
*** dpawlik has quit IRC | 14:04 | |
*** rascasoft has joined #openstack-infra | 14:04 | |
*** aojea has joined #openstack-infra | 14:04 | |
*** dpawlik has joined #openstack-infra | 14:04 | |
*** tobiash has joined #openstack-infra | 14:05 | |
*** trown has joined #openstack-infra | 14:06 | |
*** pleia2 has joined #openstack-infra | 14:07 | |
*** tonyb has joined #openstack-infra | 14:07 | |
*** ssbarnea has joined #openstack-infra | 14:07 | |
*** bnemec has joined #openstack-infra | 14:07 | |
*** jaosorior has joined #openstack-infra | 14:08 | |
*** dabukalam has joined #openstack-infra | 14:10 | |
*** mriedem has joined #openstack-infra | 14:13 | |
*** cgoncalves has joined #openstack-infra | 14:15 | |
*** bobh has joined #openstack-infra | 14:16 | |
cgoncalves | is http://status.openstack.org/openstack-health displaying jobs for anyone? | 14:16 |
*** aojea has quit IRC | 14:20 | |
*** agopi|brb has quit IRC | 14:21 | |
fungi | Script timed out before returning headers: openstack-health | 14:23 |
fungi | in the apache error log for the health api server | 14:23 |
fungi | it's able to connect to the trove instance though | 14:25 |
fungi | cgoncalves: mtreinish: restarting apache on health.openstack.org seems to have gotten http://status.openstack.org/openstack-health/ working again. not sure why it was hanging but i wonder if it's getting overloaded | 14:28 |
cgoncalves | fungi, yep, working now. thanks! | 14:28 |
fungi | yeah, cacti graphs suggest something may have been hammering it | 14:29 |
fungi | memory usage, load average, cpu utilization were all really high | 14:29 |
fungi | i wonder if my restarting apache killed a long-running query (or many of them) | 14:29 |
*** janki has quit IRC | 14:30 | |
fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2860&rra_id=all | 14:30 |
fungi | active memory jumped by a factor of something like 6x starting around 13:10z | 14:31 |
cgoncalves | fungi, one other question, I can't seem to find (periodic) job neutron-lbaas-to-octavia-migration there. e.g. http://status.openstack.org/openstack-health/#/g/project/openstack~2Fneutron-lbaas?duration=P3M&searchJob=migration | 14:36 |
fungi | cgoncalves: openstack-health relies on the subunit2sql database to get information. does the neutron-lbaas-to-octavia-migration job produce subunit data for indexing? | 14:37 |
*** auristor has joined #openstack-infra | 14:40 | |
cgoncalves | fungi, uh, no. would there be some other way of checking latest run results? | 14:41 |
fungi | cgoncalves: http://zuul.openstack.org/builds?job_name=neutron-lbaas-to-octavia-migration | 14:42 |
fungi | (queried from the "builds" tab of the zuul status dashboard) | 14:42 |
cgoncalves | fungi, aaah, thanks! bookmarking that page :) | 14:42 |
fungi | np | 14:43 |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Report tenant and project specific resource usage stats https://review.openstack.org/616306 | 14:45 |
*** agopi|brb has joined #openstack-infra | 14:46 | |
*** agopi|brb is now known as agopi | 14:46 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add resource metadata to nodes https://review.openstack.org/616262 | 14:47 |
*** bstinson_ is now known as bstinson | 14:52 | |
*** emine__ has quit IRC | 14:53 | |
openstackgerrit | 98k proposed openstack-infra/project-config master: remove job settings for oslo repositories https://review.openstack.org/616576 | 14:56 |
*** aojea has joined #openstack-infra | 14:57 | |
*** Emine has joined #openstack-infra | 15:02 | |
*** dtantsur|brb is now known as dtantsur | 15:02 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Preparatory fixes for py3 https://review.openstack.org/616578 | 15:07 |
*** shardy has joined #openstack-infra | 15:08 | |
*** ccamacho has joined #openstack-infra | 15:09 | |
*** shardy has quit IRC | 15:09 | |
*** shardy has joined #openstack-infra | 15:09 | |
*** shardy has quit IRC | 15:09 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/os-loganalyze master: Removing pypy https://review.openstack.org/468762 | 15:10 |
*** shardy has joined #openstack-infra | 15:11 | |
*** tdasilva has joined #openstack-infra | 15:11 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/os-loganalyze master: Stop ignoring H904 hacking rule in tox https://review.openstack.org/468761 | 15:12 |
mtreinish | fungi: I haven't been tracking health.o.o closely | 15:16 |
mtreinish | it could have been a bunch of slow queries, but slow queries aren't anything new | 15:17 |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Move all zull jobs definitions inside repo https://review.openstack.org/616582 | 15:17 |
mtreinish | and there shouldn't be that much local cpu load. It's mostly just averaging and resampling of time series in the api process | 15:17 |
fungi | thanks, good to know | 15:18 |
fungi | whatever was causing it, the apache restart seems to have interrupted it and restored the service to a sane state | 15:18 |
fungi | guess we'll keep an eye out for similar future behavior | 15:18 |
mtreinish | oh, I guess maybe if there is a big payload the json serialization could take some number crunching too | 15:18 |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Move elastic-recheck zull jobs definitions inside project repo https://review.openstack.org/616583 | 15:18 |
fungi | it looked like it basically pegged one cpu on the vm | 15:19 |
mtreinish | yeah, looking at the graph one core was definitely sitting there actively thinking about something | 15:21 |
mtreinish | not sure about what though | 15:21 |
dhellmann | gerrit admins: There are no members of the cross-project-spec-liaissons-chair group, and so we cannot approve any patches in the openstack-specs repo. Could someone please add me to that group so we can unblock some of the CI-related changes? https://review.openstack.org/#/admin/groups/1371,members | 15:28 |
*** armax has joined #openstack-infra | 15:35 | |
*** hamzy has quit IRC | 15:36 | |
ssbarnea | clarkb: when you have few minutes please check the elastic-receck reviews, starting with the question on https://review.openstack.org/#/c/608968/ - thanks. | 15:37 |
*** shardy has quit IRC | 15:40 | |
frickler | dhellmann: it looks like any of the liaisons should be able to do it, see https://review.openstack.org/#/c/593702/ . doesn't need a +2 like for other repos. | 15:44 |
frickler | dhellmann: mike perez left the chair position half a year ago, not sure how the formal procedures to replace him would look like, so if the other solution works, I'd prefer that for now | 15:45 |
dhellmann | frickler : ah, I didn't read closely enough | 15:45 |
dhellmann | that repo isn't used, I just don't want it to be out of date if we decide we need to resurrect it | 15:45 |
*** derekh has joined #openstack-infra | 15:52 | |
dhellmann | frickler : jroll is a member of that group but not able to add anyone to it. Can you tell who has that permission? | 15:54 |
jroll | dhellmann: I believe the "included groups" usually defines editing permissions, and there is none here | 15:54 |
frickler | dhellmann: yeah, I was just checking that, and the owner is the chair again ... so regarding the fact that you are TC chair anyway, now you have another one ;) | 15:55 |
dhellmann | frickler : thanks | 15:55 |
frickler | dhellmann: jroll: it is the "owners" list here https://review.openstack.org/#/admin/groups/1372,info | 15:56 |
jroll | frickler: ah right, thanks | 15:56 |
openstackgerrit | Merged openstack-infra/git-review master: As suggested by pep8 don't compare boolean values or empty sequences https://review.openstack.org/221172 | 15:57 |
*** noama has quit IRC | 16:06 | |
*** pcaruana has quit IRC | 16:14 | |
*** sthussey has joined #openstack-infra | 16:16 | |
*** dklyle has quit IRC | 16:17 | |
*** imacdonn has quit IRC | 16:17 | |
*** imacdonn has joined #openstack-infra | 16:18 | |
*** gyee has joined #openstack-infra | 16:18 | |
*** panda|off is now known as panda|rover | 16:19 | |
*** jcoufal has joined #openstack-infra | 16:20 | |
*** jcoufal has quit IRC | 16:20 | |
*** jcoufal has joined #openstack-infra | 16:21 | |
*** hamzy has joined #openstack-infra | 16:22 | |
*** eernst has joined #openstack-infra | 16:22 | |
*** dklyle has joined #openstack-infra | 16:23 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Fix commit-msg hook installation for worktrees https://review.openstack.org/397206 | 16:23 |
fungi | corvus: earlier today i ran `sudo MAILMAN_SITE_DIR=/srv/mailman/openstack /usr/lib/mailman/bin/rmlist openstack-internships` and it kept the archive at http://lists.openstack.org/pipermail/openstack-internships/ but removed the list from the list index. does that seem like a sane choice for retiring other lists in the future? | 16:27 |
clarkb | morning | 16:29 |
*** irdr has joined #openstack-infra | 16:30 | |
*** irdr has quit IRC | 16:31 | |
* fungi is doing a terrible job of not being here | 16:33 | |
*** irdr has joined #openstack-infra | 16:36 | |
*** irdr has quit IRC | 16:37 | |
*** eernst has quit IRC | 16:38 | |
*** quiquell is now known as quiquell|off | 16:39 | |
*** yamamoto has joined #openstack-infra | 16:40 | |
corvus | fungi: hrm, i'm not sure that was effective. it still shows up in http://lists.openstack.org/cgi-bin/mailman/listinfo and http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-internships still exists, and i can submit a subscription request on that form and get a response. | 16:40 |
*** slaweq has quit IRC | 16:40 | |
clarkb | ssbarnea: left a comment | 16:40 |
*** agopi is now known as agopi|food | 16:41 | |
ssbarnea | clarkb: thanks! I was thinking about the same approach with suffix and prefix. going to update it now. | 16:43 |
*** yamamoto has quit IRC | 16:44 | |
*** sshnaidm|rover is now known as sshnaidm|notrove | 16:46 | |
*** e0ne has quit IRC | 16:48 | |
*** xarses has joined #openstack-infra | 16:48 | |
fungi | corvus: weird. oh! | 16:49 |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Add check for failed to connect to mirror.*.openstack.org https://review.openstack.org/608968 | 16:49 |
fungi | corvus: i think i needed to delete it from puppet | 16:49 |
fungi | i'll push up a patch | 16:49 |
clarkb | ssbarnea: the other thing you can do is test the query at http://logstash.openstack.org | 16:49 |
fungi | pretty sure it got recreated by puppet | 16:49 |
corvus | fungi: haha :( | 16:50 |
*** sshnaidm|notrove is now known as sshnaidm|wknd | 16:53 | |
openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Retire the openstack-internships mailing list https://review.openstack.org/616628 | 16:54 |
fungi | corvus: ^ | 16:54 |
*** slaweq has joined #openstack-infra | 16:58 | |
*** xarses has quit IRC | 16:59 | |
*** rkukura has joined #openstack-infra | 17:00 | |
*** panda|rover is now known as panda|rover|off | 17:02 | |
ssbarnea | clarkb: i am playing with logstash but I get very weird behavior, most of the time I get no answer at all, so even for the same query I may get results or not.... | 17:05 |
clarkb | ssbarnea: in the top right is the time window. Perhaps that is shifting away from when there are records? | 17:06 |
clarkb | ssbarnea: one thing I like to do is start with a more broad search then refine it, that way I know there are results and if they go away I know I've broken my query | 17:07 |
*** nicolasbock_ has joined #openstack-infra | 17:09 | |
openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 17:09 |
*** agopi|food is now known as agopi | 17:10 | |
*** trown is now known as trown|lunch | 17:11 | |
*** derekh has quit IRC | 17:14 | |
*** derekh has joined #openstack-infra | 17:15 | |
*** gfidente has quit IRC | 17:19 | |
*** irclogbot_1 has quit IRC | 17:31 | |
*** ccamacho has quit IRC | 17:32 | |
fuentess | clarkb, hi, can you help us hold a VM from the kata-containers CI ? | 17:34 |
*** jpich has quit IRC | 17:34 | |
clarkb | fuentess: yup, kata-runsh job on the proxy repo? | 17:34 |
fuentess | yes please | 17:34 |
clarkb | fuentess: next failure should be held | 17:36 |
*** gtmanfred is now known as yaml | 17:36 | |
*** yaml is now known as gtmanfred | 17:36 | |
fuentess | clarkb, thanks | 17:37 |
openstackgerrit | Merged openstack-infra/nodepool master: Pin docker for k8s test https://review.openstack.org/616398 | 17:38 |
*** sshnaidm|wknd is now known as sshnaidm|off | 17:42 | |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Convert kubernetes config docs to zuul-sphinx https://review.openstack.org/616643 | 17:45 |
*** diablo_rojo has joined #openstack-infra | 17:48 | |
*** jpena is now known as jpena|off | 17:51 | |
ssbarnea | is anyone aware of twine check failing with exception on docutils like http://logs.openstack.org/74/615974/3/check/test-release-openstack/5682347/job-output.txt.gz#_2018-11-08_17_22_19_298514 | 17:52 |
clarkb | ssbarnea: no, that is interesting. bug in twine maybe? or maybe missing data field it assumes will be there and the error message is not good? | 17:54 |
clarkb | ssbarnea: the purpose of that check is to have twine determine if the package contents are valid for pypi | 17:54 |
agopi | clarkb, the same twine check ran perfectly fine locally for me. | 17:55 |
ssbarnea | yep, running twine locally works perfect, only on zuul I seen that weird error. | 17:57 |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Alter the metasyntactic variable in driver docs https://review.openstack.org/616646 | 17:58 |
*** irclogbot_1 has joined #openstack-infra | 18:00 | |
*** derekh has quit IRC | 18:01 | |
openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Normalize sidebar in docs https://review.openstack.org/616649 | 18:01 |
clarkb | ssbarnea: agopi I notice that the job installs docutils and pygments into one virtualenv then runs twine out of another. That may be related? I'm currently trying to reproduce it locally | 18:02 |
clarkb | ssbarnea: agopi I am able to reproduce it | 18:02 |
*** irclogbot_1 has quit IRC | 18:05 | |
ssbarnea | clarkb: waiting for details.... | 18:05 |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 18:07 |
clarkb | ssbarnea: well I don't have many yet. Other than I get the same traceback if I follow the steps of the job. run `python3 setup.py sdist bdist_wheel --universal` then make python2 venv and install twine there then run twince check dist/* | 18:07 |
clarkb | ssbarnea: produces the same traceback for me | 18:07 |
clarkb | looks like it is trying to render the description field which has a value of UNKNOWN in the PKG-INFO | 18:07 |
clarkb | maybe it needs to be known? | 18:07 |
*** pcaruana has joined #openstack-infra | 18:10 | |
fuentess | clarkb, opened a PR in the proxy repo but I cannot see the the zuul job, any idea? I already checked and the zuul config seems ok on our side... | 18:10 |
fuentess | clarkb: https://github.com/kata-containers/proxy/pull/127 | 18:11 |
clarkb | fuentess: that is curious. THe app is still enabled on your side? | 18:11 |
agopi | okay that probably might be the cause, I was using python2 to build the package. will check again thanks clarkb :) | 18:11 |
fuentess | yes, still enabled | 18:11 |
ssbarnea | clarkb: i was to replicate it too: it happens only when we build using *python3*, the wheel is valid but the sdist is not. | 18:11 |
ssbarnea | mc | 18:12 |
openstackgerrit | Merged openstack-infra/system-config master: Retire the openstack-internships mailing list https://review.openstack.org/616628 | 18:12 |
clarkb | fuentess: I wonder if the depends on confused it? | 18:13 |
clarkb | corvus: can you look at https://github.com/kata-containers/proxy/pull/127 the depends on format there might be sufficiently confusing to zuul such that the job won't run? specifically its not a full url? | 18:14 |
ssbarnea | comparing the archive I found that on py3: Description is UNKNOWN but on py2 is False - this being the only difference. | 18:14 |
fuentess | clarkb, ahh ok, let me remove it | 18:14 |
*** trown|lunch is now known as trown | 18:14 | |
clarkb | fuentess: should be Depends-On: https://github.com/stuff | 18:15 |
fuentess | clarkb, ok, thanks, although I actually dont need it, I have removed it.. lets see how it goes | 18:15 |
ssbarnea | isn't sdist part of python distro? | 18:16 |
clarkb | sdist is the source distribution of python packges | 18:16 |
*** kgiusti has left #openstack-infra | 18:22 | |
*** irclogbot_1 has joined #openstack-infra | 18:25 | |
*** armax has quit IRC | 18:25 | |
*** bobh has quit IRC | 18:29 | |
corvus | clarkb: looking. this will take a few minutes. | 18:31 |
*** irclogbot_1 has quit IRC | 18:32 | |
*** irclogbot_1 has joined #openstack-infra | 18:34 | |
ssbarnea | clarkb: thanks for helping with twine issue, is clearly caused by py3 but avoidable. | 18:35 |
clarkb | ssbarnea: any idea why python3 is causing the problem? | 18:35 |
clarkb | I wonder if running twine under python3 fixes it? | 18:35 |
ssbarnea | clarkb: running undep python2 would avoid it, this happens only when you build the package using python3. and when the package has no "Description" section in setup.cfg | 18:36 |
clarkb | ah | 18:37 |
*** kgiusti has joined #openstack-infra | 18:38 | |
*** Emine has quit IRC | 18:40 | |
clarkb | dhellmann: re ^ is the intent to run twine under python3 too? | 18:44 |
clarkb | wondering if that was missed in the transition to python3 by default | 18:44 |
ssbarnea | clarkb: i do not find this bug critical, only confusing. i do expect any project to have a description, probably that is why it was missed. | 18:45 |
ssbarnea | still seem like a something we should raise a a bug, not sure why if that is python bug or is in setuptools or something else. | 18:46 |
ssbarnea | i do expect to see others hitting it again as now, especially with newer projects. | 18:46 |
agopi | clarkb, ssbarnea it's still a bit confusing given that building package happens in one env and then twine check in another | 18:48 |
dhellmann | clarkb : yeah, did I mess that up? | 18:48 |
agopi | not sure if that's the expected behavior. | 18:49 |
clarkb | dhellmann: http://logs.openstack.org/74/615974/3/check/test-release-openstack/5682347/ara-report/result/b995038f-05ce-44b7-bb2a-d7fee9474304/ shows twine running under python2 but the sdist was run under python3 | 18:49 |
clarkb | dhellmann: the pip install of twine may need to do pip3 ? | 18:49 |
dhellmann | likely | 18:49 |
dhellmann | let me see if I can fix that quickly | 18:49 |
ssbarnea | python3 -m twine | 18:50 |
clarkb | agopi: ssbarnea it is possibly a bug in twine's handling of that case | 18:50 |
clarkb | and the error message should be fixed there? | 18:50 |
corvus | clarkb: i'm wondering if the githubeventconnector stopped | 18:51 |
openstackgerrit | Dmitry Tyzhnenko proposed openstack-infra/git-review master: Add reviewers by group alias on upload https://review.openstack.org/195043 | 18:51 |
clarkb | corvus: is that a thread in zuul-web? | 18:51 |
corvus | clarkb: in scheduler | 18:51 |
corvus | i'm grepping for it now; i'll look for errors around the last instance when i find it. i may sigusr2 in a bit | 18:52 |
dhellmann | clarkb : ensure-twine is defined in zuul-jobs, so that fix is going to require multiple steps | 18:52 |
clarkb | corvus: ok let me know if I can help | 18:53 |
clarkb | dhellmann: maybe we add ensure-twine3 or make it a config flag for backward compat? | 18:53 |
dhellmann | the pattern we've used elsewhere is to specify a python executable name in a variable | 18:53 |
dhellmann | so we can do {{whatevervarforpython}} -m pip install | 18:54 |
dhellmann | although that assumes pip is present for that python too | 18:54 |
dhellmann | sigh | 18:54 |
clarkb | dhellmann: I think {{ whatevervarforpython }} /usr/bin/pip will work though | 18:55 |
*** electrofelix has quit IRC | 18:55 | |
dhellmann | do you have any idea where we install pip? | 18:57 |
clarkb | I think the images may bake that in | 18:58 |
clarkb | (because we need virtualenv?) | 18:58 |
dhellmann | ok | 18:58 |
ssbarnea | clarkb: how about calling python3 -m pip -- this works for sure, no need to guess location. | 18:58 |
dhellmann | ssbarnea : that only works if pip is already installed for python3, though | 18:59 |
dhellmann | that's the bit I can't figure out how to verify by reading roles | 18:59 |
*** dpawlik has quit IRC | 19:00 | |
clarkb | dhellmann: instead of using -m pip just run the pip executable under the python version directly I think that works | 19:00 |
*** dpawlik has joined #openstack-infra | 19:00 | |
*** armax has joined #openstack-infra | 19:00 | |
dhellmann | ok | 19:01 |
clarkb | `python2 $(which pip)` and `python3 $(which pip)`? | 19:01 |
clarkb | I do this with pbr a lot | 19:01 |
ssbarnea | clarkb: no need to to as this logic would mess pip, what is wrong in using the module calling? | 19:02 |
ssbarnea | here is the quickstart.sh ensure_pip version (bash) https://github.com/openstack/tripleo-quickstart/blob/master/quickstart.sh#L154 | 19:03 |
clarkb | ssbarnea: -m only looks in your module path, if pip isn't in the module path for that version of python it won't work | 19:04 |
*** Emine has joined #openstack-infra | 19:04 | |
dhellmann | clarkb : where would I look to find a list of things being installed into the images? | 19:04 |
ssbarnea | clarkb: if pip is not in your module path.... you don't have pip. pip is not a standalone script, it needs to be installed as a module. | 19:04 |
dhellmann | yeah, I don't think running it without the -m is going to work | 19:05 |
clarkb | dhellmann: https://nb01.openstack.org/ubuntu-bionic-0000000018.log is the full build log for a recent bionic build. You can navigate the root there to see logs for other distros | 19:06 |
*** ifat_afek has joined #openstack-infra | 19:06 | |
ssbarnea | dhellmann: the same applies to other tools, like virtualenv for example. the only safe way to call them is as modules. (virtualenv is called virtualenv-3 on python3) | 19:06 |
dhellmann | clarkb : is there a manifest or something that I could look at? | 19:07 |
*** ifat_afek has quit IRC | 19:07 | |
openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: enable setting python version for ensure-twine https://review.openstack.org/616673 | 19:08 |
dhellmann | clarkb : step 1 ^ | 19:08 |
clarkb | dhellmann: if you grep for dpkg-query --show there is a dpkg listing | 19:08 |
dhellmann | do we install pip from system packages? | 19:08 |
clarkb | no, and we don't appear to have a pip freeze output | 19:08 |
clarkb | (we stopped listing all files in the image because it made the image builds take forever stating all the things iirc) | 19:09 |
dhellmann | I don't really want to wade through tons of output if I can look at the instructions used to build the image | 19:09 |
openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: ensure twine is installed with python 3 https://review.openstack.org/616676 | 19:11 |
dhellmann | step 2 ^ | 19:11 |
clarkb | https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv that is what we use to install pip and virtualenv | 19:11 |
dhellmann | ok, the readme there says it installed pip3 | 19:12 |
*** diablo_rojo has quit IRC | 19:12 | |
dhellmann | so running -m pip should work fine | 19:12 |
clarkb | I think as long as the chosen distro has python2 and python3 it tries to accomodate both? | 19:13 |
clarkb | but will do one or the other if the distro doesn't have both | 19:13 |
clarkb | (those jobs run on ubuntu and should have both) | 19:13 |
agopi | thanks dhellmann clarkb, looks like dhellmann's commits helps clear up the air. | 19:14 |
*** diablo_rojo has joined #openstack-infra | 19:15 | |
dhellmann | the only thing this would break would be that check job and actual releases, so the impact will be minimal. if we aren't confident of the approach based on the readmes and whatever else, then maybe we should go ahead with and then I can run a test using the release-test repo (we can't test speculatively, unfortunately) | 19:15 |
clarkb | I think we can go ahead and make the changes and then test it. It is easy enough to revert 616676 if necessary | 19:16 |
dhellmann | yeah | 19:16 |
dhellmann | I made the project-config change on top of https://review.openstack.org/#/c/616550/ because we want to remove that call of setup.py too | 19:16 |
clarkb | ssbarnea has a comment on the zuul-jobs change if you want to make it consistent with other roles | 19:16 |
dhellmann | ssbarnea , clarkb : we have a release_python variable in the release role, so I followed that pattern | 19:17 |
dhellmann | is that not the right pattern to follow? | 19:17 |
ssbarnea | i hope that you may find that "pattern" useful, i would be really happy if you also adopt it. we have in many roles in tripleo-quickstart now aftet f28 enablement work. | 19:17 |
clarkb | oh I thought that was for consistency with other zuul-jobs roles | 19:18 |
*** Emine has quit IRC | 19:18 | |
clarkb | (I don't really care) | 19:18 |
dhellmann | I suppose I could reuse release_python since we want them set the same, but that's managed by a different role | 19:18 |
dhellmann | that would result in one less patch, since we wouldn't need the one in project-config | 19:19 |
clarkb | ya I think python_cmd is nice in that we can use it everywhere without too much ambiguity | 19:19 |
clarkb | but release_python would be weird there I think | 19:19 |
dhellmann | zuul-jobs has a release_python and sphinx_python | 19:19 |
dhellmann | so I followed that pattern | 19:19 |
dhellmann | I don't see any roles setting python_cmd | 19:20 |
clarkb | ya probably fine to stick to that pattern for now then change to python_cmd if we want to in the future and apply it globally in zuul-jobs | 19:20 |
dhellmann | presumably because we want the flexibility of using different pythons in different roles? | 19:20 |
dhellmann | ok | 19:20 |
ssbarnea | i personally think that we don't really need one variable for each tool, seems .... bit of maintenance nightmare. i volunteer to help migrating to one standard solution. | 19:21 |
*** auristor has quit IRC | 19:22 | |
dhellmann | I don't know the best practice for using the same variable across roles that may not even be defined in the same git repo | 19:22 |
dhellmann | I need to step out. I'll check back in a bit. | 19:23 |
clarkb | dhellmann: it comes down to whether or not you want to select things globally (so can set it at a host level) or if you want fine grained control to say run twine under python2 and sphinx under 3 and release under 3 | 19:23 |
clarkb | (there is precedence but understanding that is confusing enough that using explicit values is nice) | 19:24 |
ssbarnea | dhellmann: the nice part is that you don't need to define it, unless you want to override its implicit value. we copied those two tasks in each role that needed python_cmd. | 19:24 |
*** ralonsoh has quit IRC | 19:24 | |
*** auristor has joined #openstack-infra | 19:30 | |
*** mriedem has quit IRC | 19:32 | |
*** auristor has quit IRC | 19:35 | |
*** auristor has joined #openstack-infra | 19:37 | |
ssbarnea | clarkb: need bit of help for browbeat team: the core-release group is empty : https://review.openstack.org/#/admin/projects/openstack/browbeat,access | 19:38 |
clarkb | ssbarnea: looks like Joe Talerico added the acl. | 19:40 |
ssbarnea | clarkb: can you please add agopi and jtaleric to release group? | 19:40 |
clarkb | I've added jtaleric as the person who updated the acl file. They can add others as they like | 19:41 |
agopi | oh okay that'd work clarkb | 19:41 |
agopi | thanks clarkb and ssbarnea :) | 19:41 |
ssbarnea | clarkb: perfect! thanks. they can sort it themselves now. | 19:41 |
ssbarnea | clarkb: i was curious about the release process in general. to my knowledge the only way to release is to push a tag. this means that gerrit cannot really be used to review a release (tagging) - like having multiple people blessing the release. right? | 19:44 |
*** aojea has quit IRC | 19:44 | |
clarkb | ssbarnea: sort of. The process openstack has built is to have a release control repository into whcih you put release data, it gets reviewed adn approved and those changes merging push tags that kick off the process | 19:44 |
clarkb | ssbarnea: if you are not part of that system you are correct. The pushing of tags yourself will immediately kick off the release process so if you want to review things you need to do it before pushing to gerrit | 19:45 |
clarkb | ssbarnea: usually what the infra team does is just informally poll the cores and ask for input on the project, version, sha1 being tagged to make sure we are in rough aggreement | 19:45 |
ssbarnea | clarkb: thanks. | 19:46 |
clarkb | ssbarnea: agopi other things to keep in mind are that deleting or moving a tag later doesn't update dowsntreams that have already pulled the old tag | 19:46 |
clarkb | so in general we don't delete or move tags, instead we highly encourage you to push a subsequent release if there was an error in the last one | 19:47 |
openstackgerrit | Dmitry Tyzhnenko proposed openstack-infra/git-review master: Add reviewers by group alias on upload https://review.openstack.org/195043 | 19:49 |
ssbarnea | i told him that once you push a tag, you don't edit it, you only create a new one if the prev one was buggy. | 19:49 |
agopi | thanks ssbarnea clarkb :) | 19:52 |
*** dave-mccowan has joined #openstack-infra | 19:56 | |
ianw | dhellmann: did you get all your answers on the pip install in images? | 19:56 |
*** rlandy is now known as rlandy|brb | 19:57 | |
*** ifat_afek has joined #openstack-infra | 19:57 | |
*** mriedem has joined #openstack-infra | 19:59 | |
*** jtomasek has quit IRC | 20:05 | |
*** dpawlik has quit IRC | 20:10 | |
dhellmann | ianw : I think we have enough confidence that we will have pip3 that we can approve the changes and then test them with some inconsequential release (like release-test) | 20:16 |
*** e0ne has joined #openstack-infra | 20:19 | |
*** rlandy|brb is now known as rlandy | 20:19 | |
*** e0ne has quit IRC | 20:22 | |
ianw | dhellmann: on the infra nodes, yes the idea is that "pip" and "virtualenv" are both python2 installers, while "pip3" and "virtualenv3" are python3 | 20:23 |
ianw | that's what the prior referenced dib element jumps through hoops to provide | 20:24 |
*** eharney has quit IRC | 20:24 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Use six for cross python compatibility https://review.openstack.org/616688 | 20:25 |
ianw | i think the "virtualenv3" might an ... anti-pattern ... or something, but we had things using it. it seems "python3 -m venv" is the way to go, maybe | 20:25 |
AJaeger | config-core, could you review https://review.openstack.org/616550 , please? I think we should remove the role completely - but like somebody to confirm... | 20:25 |
*** dpawlik has joined #openstack-infra | 20:26 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task https://review.openstack.org/616387 | 20:29 |
*** eharney has joined #openstack-infra | 20:29 | |
openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test" https://review.openstack.org/616404 | 20:30 |
*** dpawlik has quit IRC | 20:30 | |
ssbarnea | AJaeger: not a core but by reading it the body of the role I would say role removal is what we want. | 20:30 |
*** dave-mccowan has quit IRC | 20:33 | |
clarkb | AJaeger: dhellmann I think twine installs docutils and pygments now? I'm checking | 20:35 |
AJaeger | clarkb: it does not - just checked | 20:36 |
clarkb | docutils==0.14 and Pygments==2.2.0 installed in the venv where I installed twine and I did not install them myself | 20:36 |
clarkb | twine==1.12.1 | 20:36 |
* AJaeger needs better glasses ;( | 20:36 | |
AJaeger | clarkb: you're right... | 20:36 |
* AJaeger sends a patch... | 20:36 | |
clarkb | I think they are deps of readme_renderer which is a dep of twine | 20:37 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove check-python-release role https://review.openstack.org/616689 | 20:39 |
ianw | AJaeger: so now the role is just a virtualenv & package installs? | 20:39 |
AJaeger | ianw: that would be the result of 616550 - and I think we don't need that, so proposed 616689 as alternative | 20:40 |
AJaeger | dhellmann: are you sure that twine does the README check? Or is that not done anymore? | 20:41 |
*** mriedem has quit IRC | 20:41 | |
ianw | i appreciate the "get it working" angle :) but if we can clean it up now and not forget, that sounds good ... around if any help needed triggering anything to test | 20:42 |
AJaeger | Just checked - twine does the README check, so we can remove it.... | 20:43 |
AJaeger | config-core, dhellmann, I suggest to go with 616689 instead of 616550 - please review | 20:43 |
dhellmann | AJaeger : http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/pti-python-tarball/check.yaml#n7 | 20:45 |
*** dklyle has quit IRC | 20:45 | |
*** dklyle has joined #openstack-infra | 20:45 | |
AJaeger | dhellmann: yes, I remove that line - nothing needs this | 20:45 |
*** mriedem has joined #openstack-infra | 20:45 | |
AJaeger | dhellmann: ignore my comment. Yes, that is fine. | 20:45 |
*** dklyle has quit IRC | 20:46 | |
dhellmann | AJaeger : line 8 runs "twine check" and that needs docutils and pygments installed | 20:46 |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 20:46 |
AJaeger | dhellmann: and that is done by ensure-twine, clarkb and myself just tested | 20:46 |
*** mftech has joined #openstack-infra | 20:46 | |
dhellmann | ok | 20:46 |
dhellmann | I guess some time next week I'll try to find time to rebase the rest of my changes on top of this | 20:47 |
AJaeger | dhellmann: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/ensure-twine/tasks/main.yaml#n14 | 20:47 |
dhellmann | I wish we could separate fixing things from cleaning things up | 20:47 |
*** rh-jelabarre has quit IRC | 20:48 | |
AJaeger | dhellmann: this is not proper cleaning IMHO - and the fix was easy. the role had setup and execution, you removed the execution - and then the setup should really go as well. | 20:49 |
dhellmann | it was not clear, and still is not, that the cleanup isn't needed | 20:49 |
AJaeger | dhellmann: perhaps your commit message was not clear - but I did not see that anything here is broken that needs a half-fix | 20:50 |
dhellmann | I guess you're saying the readme rendering thing is what causes those dependencies to be installed? | 20:50 |
AJaeger | dhellmann: yes | 20:50 |
dhellmann | my change was "stop doing something redundant" | 20:50 |
dhellmann | it's fine, I don't care | 20:50 |
AJaeger | clarkb: want to +2A https://review.openstack.org/616689 , please? Then I rebase dhellmann's change on top of it... | 20:52 |
*** ifat_afek has quit IRC | 20:53 | |
clarkb | AJaeger: you mean rebase the python3 fix? | 20:55 |
clarkb | AJaeger: thats done | 20:56 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: ensure twine is installed with python 3 https://review.openstack.org/616676 | 20:57 |
openstackgerrit | Merged openstack-infra/nodepool master: Convert kubernetes config docs to zuul-sphinx https://review.openstack.org/616643 | 20:57 |
openstackgerrit | Merged openstack-infra/nodepool master: Alter the metasyntactic variable in driver docs https://review.openstack.org/616646 | 20:57 |
AJaeger | clarkb: yeah - I just did the rebase of 616676 | 20:58 |
openstackgerrit | Merged openstack-infra/nodepool master: Normalize sidebar in docs https://review.openstack.org/616649 | 20:58 |
AJaeger | clarkb: in case you or ianw want to +2 616676 as well - I'll +2A now the zuul-jobs one... | 20:59 |
clarkb | I've actually got to pop out for a bit now. Lunch time | 20:59 |
AJaeger | clarkb: enjoy! | 21:00 |
openstackgerrit | Merged openstack-infra/project-config master: Remove check-python-release role https://review.openstack.org/616689 | 21:03 |
ssbarnea | maybe we can also merge https://review.openstack.org/#/c/471078/ ? | 21:07 |
*** kgiusti has left #openstack-infra | 21:10 | |
AJaeger | ssbarnea: better ask ianw and clarkb for system-config ^ | 21:10 |
ssbarnea | sure, anytime time to go to bed... that change is more than an year old, it probably can wait anotehr day or two | 21:11 |
* AJaeger takes ssbarnea's advice and calls it a day ;) | 21:13 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: enable setting python version for ensure-twine https://review.openstack.org/616673 | 21:16 |
*** mriedem has left #openstack-infra | 21:18 | |
*** nicolasbock_ has quit IRC | 21:32 | |
*** Emine has joined #openstack-infra | 21:36 | |
*** eharney has quit IRC | 21:37 | |
*** ansmith has quit IRC | 21:37 | |
*** hamzy has quit IRC | 21:43 | |
*** diablo_rojo has quit IRC | 21:50 | |
*** florianf is now known as florianf|afk | 21:51 | |
*** kjackal has quit IRC | 21:56 | |
*** kjackal has joined #openstack-infra | 21:57 | |
*** trown is now known as trown|outtypewww | 22:00 | |
*** manjeets has quit IRC | 22:00 | |
corvus | clarkb: i don't see a smoking gun in the logs, just that the last time the GithubEventConnector did anything was 03 nov | 22:02 |
corvus | clarkb: i'm going to sigusr2 | 22:02 |
*** diablo_rojo has joined #openstack-infra | 22:03 | |
*** agopi is now known as agopi|brb | 22:04 | |
mordred | ianw: re: the nodepool stats patch above - ifyou're in the brainspace for it, I'm working on an update looking towards using the keystoneauth patch that will allow us to get rid of the taskmanager | 22:06 |
mordred | ianw: https://review.openstack.org/#/c/614834/ - would love some eyes on it - but it's WIP because I need to write docs | 22:06 |
mordred | ianw: it might be worth looking at https://review.openstack.org/#/c/604926 and https://review.openstack.org/#/c/605043/ for context | 22:06 |
mordred | ianw: BUT - if you don't want to and have other things on your plate - totally no worries | 22:07 |
*** mattoliverau has joined #openstack-infra | 22:07 | |
clarkb | corvus: ok, back from lunch now | 22:08 |
*** agopi|brb has quit IRC | 22:09 | |
ianw | mordred: i'll take a look :) | 22:10 |
*** mftech has quit IRC | 22:11 | |
clarkb | the gerrit bug tracker tells me that gerrit upstream may be doing work to fix the relationship between changes data presentation | 22:12 |
clarkb | this is exciting | 22:12 |
*** mftech has joined #openstack-infra | 22:12 | |
mordred | oooh | 22:13 |
mordred | that is exciting | 22:13 |
clarkb | Labels: -UX-Incoming UX-Acked was the update | 22:14 |
clarkb | hoping that -UX-Incoming means we are working on it now | 22:14 |
*** Emine has quit IRC | 22:14 | |
clarkb | an update on the logs.o.o thing. I don't see any more recent errors for wsgi timeouts | 22:19 |
clarkb | I think we should go ahead and get dmsimard's change to add processes in | 22:19 |
*** eernst has joined #openstack-infra | 22:19 | |
clarkb | we also had a large spike in data transfer around 1800UTC today so it is working when put under load | 22:19 |
clarkb | https://review.openstack.org/#/c/616297/ is the chagne from dmsmimard if anyone else can review it (system-config) | 22:23 |
logan- | has anyone seen xenial DIB builds failing with "Operation failed: Too many levels of symbolic links" during post-install.d/80-enable-sshd-service recently? I had a working build on 11/1 and then today they are failing consistently this way. http://paste.openstack.org/raw/734447/ | 22:27 |
ianw | logan:- hrm, not aware of that. there probably has been a dib release in between but not sure we touched anything related to that | 22:28 |
clarkb | logan-: let me grep our logs | 22:28 |
logan- | was just looking at https://nb01.openstack.org/ubuntu-xenial-0000000018.log and it seems like nb01 was at least able to complete a build today | 22:29 |
clarkb | logan-: https://nb01.openstack.org/ubuntu-xenial-0000000018.log ya that log looks ok | 22:29 |
clarkb | logan-: are you using ubuntu or ubuntu-minimal element? | 22:29 |
clarkb | (we use ubuntu-minimal) | 22:29 |
logan- | ubuntu-minimal | 22:30 |
logan- | ianw: good tip, i confirmed my 11/1 build was on DIB 2.17, today is 2.18 | 22:30 |
logan- | let me try a build on 2.17 and see how that owkrs | 22:31 |
ianw | logan-: can you run with "-x" ? | 22:31 |
logan- | yep | 22:31 |
logan- | on it | 22:31 |
logan- | thanks | 22:31 |
ianw | looks like https://review.openstack.org/#/c/503844/ touched that file, but only for the openrc path? | 22:32 |
*** mriedem has joined #openstack-infra | 22:36 | |
clarkb | thats a systemctl error message? | 22:37 |
logan- | light googling led me to think that yeah | 22:37 |
logan- | wondering if its more an ubuntu package update problem than dib | 22:37 |
*** kjackal has quit IRC | 22:39 | |
*** Emine has joined #openstack-infra | 22:41 | |
openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/os-loganalyze master: Add support for Magnum https://review.openstack.org/616705 | 22:42 |
*** otherwiseguy has joined #openstack-infra | 22:45 | |
*** mriedem has quit IRC | 22:46 | |
*** eharney has joined #openstack-infra | 22:47 | |
*** manjeets has joined #openstack-infra | 22:55 | |
clarkb | diablo_rojo: you wanted stickers right? /me attempts to remember to pack them | 22:56 |
*** Emine has quit IRC | 23:06 | |
*** mftech has quit IRC | 23:12 | |
*** Tim_ok has quit IRC | 23:12 | |
*** mftech has joined #openstack-infra | 23:12 | |
*** mftech has quit IRC | 23:13 | |
*** agopi|brb has joined #openstack-infra | 23:14 | |
*** kjackal has joined #openstack-infra | 23:15 | |
*** slaweq has quit IRC | 23:15 | |
*** kjackal has quit IRC | 23:25 | |
* logan- tries to find a paste site that will accept this DIB log | 23:28 | |
*** jcoufal has quit IRC | 23:28 | |
logan- | 2.17 completed, 2.18 failed.. going to upload debug logs for both | 23:28 |
clarkb | logan-: gist is usually good for giant logs | 23:28 |
logan- | thanks | 23:29 |
diablo_rojo | clarkb, yes please! the small ants :) | 23:29 |
ianw | logan-: it's usually on the last bit that's useful ... but the real problem can be a few hundred lines back | 23:30 |
ianw | as everything unwinds | 23:30 |
logan- | working 2.17 - https://gist.githubusercontent.com/logan2211/3b4312ff2f77f7aec73fb36caac45a87/raw/f5839c411f3fb3998dd3867ce379d5cfaf366606/DIB%25202.17%2520build | 23:31 |
logan- | failed 2.18 - https://gist.githubusercontent.com/logan2211/3b4312ff2f77f7aec73fb36caac45a87/raw/f5839c411f3fb3998dd3867ce379d5cfaf366606/DIB%25202.18%2520build | 23:31 |
logan- | afk a while -- but ill try to figure it out later this evening if it is still unsolved. thanks clarkb & ianw | 23:32 |
*** fuentess has quit IRC | 23:33 | |
clarkb | fwiw I'm slowly fading out myself. I keep getting distracted by the list of things I need to do around the house and making it hard to focus on work. I should probably just go drain the that pipe then I don't have to worry about it freezing while I am gone | 23:33 |
ianw | hrm, appears to be the same openssh version openssh-server amd64 1:7.2p2-4ubuntu2.6 [335 kB] | 23:33 |
ianw | clarkb: so weird, i have never once worried about a pipe freezing in all my years :) | 23:34 |
clarkb | ianw: this one runs through the garage wall which isn't a heated space so its got a plug I can take out upstream with a valve there allowing me to drain everything in the unheated space | 23:35 |
clarkb | next time I need plumbing work done I am going to have them replace that valve with a ball valve so that closign it is easy | 23:36 |
dmsimard | ianw: does it even get close to freezing temperature where you are ? haha | 23:37 |
ianw | you'll sometimes get a morning frost on the ground in Sydney, but nothing that will freeze a pipe | 23:38 |
clarkb | its not forecast to freeze here really and should be fine. But I know I won't have to worry about it later (or when in berlin) if I just take care of it now | 23:38 |
*** boden has quit IRC | 23:39 | |
*** boden has joined #openstack-infra | 23:40 | |
*** dklyle has joined #openstack-infra | 23:44 | |
*** boden has quit IRC | 23:45 | |
corvus | clarkb: ball valves are great -- but (and i have recently seen this is not theoretical) they can leak slightly, and if they do, you can't tighten them more. so they're great for things that aren't open-ended (ie, in the middle of plumbing). for a spigot, or something like it, a regular globe valve may be better. | 23:51 |
clarkb | huh | 23:53 |
corvus | (i just got ball valves for my washing machine, and they're great, except one of them has like a one drop/minute leak, which was a minor inconvenience when i had no washing machine. now that i do, and the hoses are all hooked up, it's fine.) | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!