Thursday, 2018-11-08

*** ansmith has joined #openstack-infra00:01
ianwclarkb: it does super() from the openstacksdk version? -> https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/task_manager.py#n5000:01
clarkbonly for init, not post_run_task or submit_task etc00:02
clarkbsubmit_task is the one that matters here I think?00:02
corvusianw: +2 on the ansible changes (with a comment that override-checkout shouldn't be necessary since https://review.openstack.org/616314 merged)00:03
ianwclarkb: ohhh, yes i see what you're saying, the nodepool submit_task is it's own thing and has overridden it00:03
corvusianw: i didn't +3 due to eod; feel free to self-approve00:03
clarkbianw: yup00:03
ianwclarkb: first thought is i wonder if that needs to be overridden00:05
ianwoh, it rasies a slightly different exception00:06
clarkbianw: functionally though both put the task on the queue and wait for it to complete then return the result00:07
*** dhellmann has joined #openstack-infra00:07
clarkbit is possible that we could use submit_task from sdk I think00:07
clarkbboth inits manage self.queue which is weird but should be a non issue by the time submit_ask is called00:08
ianwyep, possibly the sdk should take a new argument which is the exception to raise00:08
clarkbor update nodepool to handle the sdk exception?00:09
ianwor, we could wrap nodepool and re-raise the right one ... that might be better00:09
clarkbya that would be one way of doing that00:09
ianwok, will try that00:11
clarkbok not sure if I should be happy that log server stopped having problems are sad that as soon as I feel like I'm somewhat set up to watch it real time it decides to be done00:11
clarkbfungi: did you see the etherpad for your forum session?00:12
clarkbfungi: any other prep work you think I/we should be doing for that one?00:12
*** sthussey has quit IRC00:30
dmsimardhuh, no other spikes00:31
clarkbdmsimard: ya i think we watch it and look at swift after the summit00:32
clarkbif it isn't getting worse then its pretty happy right this moment00:32
*** dhill_ has quit IRC00:41
*** longkb has joined #openstack-infra00:46
*** calebb has quit IRC00:50
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task  https://review.openstack.org/61638700:51
*** agopi is now known as agopi|brb00:54
*** gyee has quit IRC00:55
*** agopi|brb has quit IRC00:59
fungiclarkb: thanks! to be honest, i forgot "i" had a forum session ;)01:07
fungihappy to moderate it01:07
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: pin minikube version by default  https://review.openstack.org/61638801:12
*** bobh has joined #openstack-infra01:12
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: DNM: test older minikube version  https://review.openstack.org/61638901:13
fungiclarkb: i proposed that session by proxy, to be honest01:16
fungithough i did flesh out the description a bit01:16
funginot aware of any prep we need to be doing01:17
fungijust wanted to be sure opendev had a session reserved to get feedback01:17
fungiwe can really use it however we want01:17
*** felipemonteiro has joined #openstack-infra01:19
*** Swami has quit IRC01:24
*** bobh has quit IRC01:28
*** yamamoto has joined #openstack-infra01:45
*** mriedem has quit IRC01:50
*** agopi|brb has joined #openstack-infra01:52
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] add docker package pin option  https://review.openstack.org/61639101:52
*** diablo_rojo has quit IRC01:53
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning  https://review.openstack.org/61638801:56
*** rkukura has quit IRC01:56
openstackgerritIan Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role  https://review.openstack.org/60558501:58
openstackgerritIan Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role  https://review.openstack.org/60558502:06
*** hongbin has joined #openstack-infra02:13
openstackgerritIan Wienand proposed openstack-infra/system-config master: [wip] initial port of install-docker role  https://review.openstack.org/60558502:29
*** fuentess has quit IRC02:33
*** felipemonteiro has quit IRC02:33
*** mrsoul has joined #openstack-infra02:44
*** markvoelker has joined #openstack-infra02:45
*** markvoelker has quit IRC02:45
*** markvoelker has joined #openstack-infra02:45
*** yamamoto has quit IRC02:49
*** bobh has joined #openstack-infra02:49
openstackgerritIan Wienand proposed openstack-infra/nodepool master: [dmn] test pinning docker version  https://review.openstack.org/61639802:50
*** markvoelker has quit IRC02:50
*** felipemonteiro has joined #openstack-infra02:55
*** markvoelker has joined #openstack-infra02:55
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: [wip] add docker package pin option  https://review.openstack.org/61639102:56
*** anteaya has quit IRC03:02
openstackgerritIan Wienand proposed openstack-infra/nodepool master: [dmn] test pinning docker version  https://review.openstack.org/61639803:04
*** fungi has quit IRC03:06
*** fungi has joined #openstack-infra03:09
*** fungi has quit IRC03:10
*** larainema has joined #openstack-infra03:12
*** armax has quit IRC03:18
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: install-docker: add package version option  https://review.openstack.org/61639103:18
*** mrhillsman has joined #openstack-infra03:19
*** armax has joined #openstack-infra03:19
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Pin docker for k8s test  https://review.openstack.org/61639803:21
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task  https://review.openstack.org/61638703:21
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test"  https://review.openstack.org/61640403:21
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Pin docker for k8s test  https://review.openstack.org/61639803:24
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task  https://review.openstack.org/61638703:24
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test"  https://review.openstack.org/61640403:24
*** armax has quit IRC03:25
*** felipemonteiro has quit IRC03:35
*** felipemonteiro has joined #openstack-infra03:36
*** armax has joined #openstack-infra03:38
*** fungi has joined #openstack-infra03:40
*** fungi has quit IRC03:41
openstackgerritIan Wienand proposed openstack-infra/zuul-jobs master: install-docker: add package version option  https://review.openstack.org/61639103:43
*** fungi has joined #openstack-infra03:45
*** bobh has quit IRC03:55
*** armax has quit IRC03:59
*** rlandy|bbl is now known as rlandy03:59
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Remove unused nodepool-k8s-functional role  https://review.openstack.org/61640904:05
*** bobh has joined #openstack-infra04:08
*** bobh has quit IRC04:10
*** sambetts|afk has quit IRC04:11
*** felipemonteiro has quit IRC04:11
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Remove nodepool-k8s-functional and install-nodepool roles  https://review.openstack.org/61640904:24
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider  https://review.openstack.org/57066704:29
*** janki has joined #openstack-infra04:36
openstackgerritIan Wienand proposed openstack-infra/system-config master: Import install-docker role  https://review.openstack.org/60558504:39
ianwclarkb / mordred / corvus : ^ this was the role i mentioned i needed to loop back on in the meeting04:39
*** apetrich has quit IRC04:45
*** apetrich has joined #openstack-infra04:46
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider  https://review.openstack.org/57066704:49
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider  https://review.openstack.org/59033504:52
openstackgerritIan Wienand proposed openstack-infra/system-config master: Import install-docker role  https://review.openstack.org/60558505:29
*** yamamoto has joined #openstack-infra05:45
*** hongbin has quit IRC06:02
*** kjackal has joined #openstack-infra06:03
*** pcrews has quit IRC06:20
*** diablo_rojo has joined #openstack-infra06:20
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator  https://review.openstack.org/60464806:26
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator  https://review.openstack.org/60464806:29
*** pcrews has joined #openstack-infra06:33
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats  https://review.openstack.org/61368006:35
*** pcrews has quit IRC06:57
*** roman_g has joined #openstack-infra06:57
*** ifat_afek has joined #openstack-infra06:59
*** quiquell|off is now known as quiquell07:03
*** dpawlik has joined #openstack-infra07:04
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: install-docker: add package version option  https://review.openstack.org/61639107:10
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning  https://review.openstack.org/61638807:11
*** gfidente has joined #openstack-infra07:13
*** aojea has quit IRC07:15
AJaegerianw: Just fixed some nits ^07:16
*** aojea has joined #openstack-infra07:20
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver  https://review.openstack.org/53555607:23
*** pcaruana has joined #openstack-infra07:34
*** slaweq has joined #openstack-infra07:39
*** yamamoto has quit IRC07:46
*** yamamoto has joined #openstack-infra07:46
*** yamamoto has quit IRC07:47
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver  https://review.openstack.org/53555607:47
ianwAJaeger: thanks :)07:48
*** quiquell is now known as quiquell|brb07:49
*** ginopc has joined #openstack-infra07:58
*** bnemec has quit IRC08:00
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider  https://review.openstack.org/57066708:02
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider  https://review.openstack.org/59033508:02
*** carl_cai has joined #openstack-infra08:05
openstackgerritLuka Peschke proposed openstack-infra/irc-meetings master: Creating an IRC meeting for CloudKitty  https://review.openstack.org/61620508:06
*** diablo_rojo has quit IRC08:08
*** florianf|afk is now known as florianf08:10
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode  https://review.openstack.org/61645008:14
*** ralonsoh has joined #openstack-infra08:17
*** quiquell|brb is now known as quiquell08:17
*** jtomasek has joined #openstack-infra08:18
*** bnemec has joined #openstack-infra08:26
*** shrasool has joined #openstack-infra08:30
*** yamamoto has joined #openstack-infra08:34
*** jpena|off is now known as jpena08:36
*** noama has joined #openstack-infra08:38
*** yamamoto has quit IRC08:40
openstackgerritMerged openstack-infra/irc-meetings master: Add ralonsoh to chairing neutron-qos meeting  https://review.openstack.org/61620808:42
*** yamamoto has joined #openstack-infra08:46
*** roman_g has quit IRC08:46
*** yamamoto has quit IRC08:47
openstackgerritMerged openstack-infra/irc-meetings master: Creating an IRC meeting for CloudKitty  https://review.openstack.org/61620508:47
openstackgerritMerged openstack-infra/irc-meetings master: add day_specifier from recurrence  https://review.openstack.org/61527008:47
*** yamamoto has joined #openstack-infra08:49
*** sshnaidm|afk is now known as sshnaidm|rover08:54
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode  https://review.openstack.org/61645009:07
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Add extra safety belt when reusing a node  https://review.openstack.org/61646509:07
*** roman_g has joined #openstack-infra09:10
*** shrasool has quit IRC09:11
*** yamamoto has quit IRC09:13
*** jpich has joined #openstack-infra09:14
*** rkukura has joined #openstack-infra09:22
*** shardy has joined #openstack-infra09:22
*** chem has quit IRC09:24
*** ifat_afek has quit IRC09:28
*** derekh has joined #openstack-infra09:32
*** lucasagomes has joined #openstack-infra09:35
*** CrayZee has joined #openstack-infra09:48
*** CrayZee is now known as Guest3121209:48
*** Guest31212 has quit IRC09:49
*** e0ne has joined #openstack-infra09:50
*** ssbarnea has joined #openstack-infra09:50
*** snapiri- has joined #openstack-infra09:52
*** yamamoto has joined #openstack-infra09:54
*** dtantsur|afk is now known as dtantsur09:55
openstackgerritSimon Westphahl proposed openstack-infra/nodepool master: wip: Try optimizing node lock check  https://review.openstack.org/61648409:58
*** CrayZee- has joined #openstack-infra09:59
*** CrayZee- has quit IRC09:59
*** yamamoto has quit IRC10:00
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator  https://review.openstack.org/60464810:01
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Update node during lockNode  https://review.openstack.org/61645010:01
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Add extra safety belt when reusing a node  https://review.openstack.org/61646510:01
*** yamamoto has joined #openstack-infra10:04
*** shardy has quit IRC10:07
*** shardy has joined #openstack-infra10:09
*** electrofelix has joined #openstack-infra10:10
*** longkb has quit IRC10:12
*** jrist has quit IRC10:14
*** shardy has quit IRC10:14
*** jrist has joined #openstack-infra10:16
*** shardy has joined #openstack-infra10:19
*** snapiri- has quit IRC10:19
*** snapiri- has joined #openstack-infra10:20
openstackgerritMerged openstack-infra/zuul-jobs master: install-docker: add package version option  https://review.openstack.org/61639110:23
*** CrayZee has joined #openstack-infra10:29
openstackgerritMerged openstack-infra/zuul-jobs master: install-kubernetes: enable minikube version pinning  https://review.openstack.org/61638810:30
*** panda|off is now known as panda10:31
*** Emine has joined #openstack-infra10:32
*** snapiri- has quit IRC10:32
*** shardy has quit IRC10:38
*** efried has quit IRC10:38
*** efried has joined #openstack-infra10:38
*** shardy has joined #openstack-infra10:39
*** ginopc has quit IRC10:43
*** ginopc has joined #openstack-infra10:43
*** shardy has quit IRC10:44
*** CrayZee_ has joined #openstack-infra11:00
*** CrayZee has quit IRC11:02
*** ifat_afek has joined #openstack-infra11:02
*** larainema has quit IRC11:05
*** yamamoto has quit IRC11:10
*** yamamoto has joined #openstack-infra11:14
*** CrayZee_ has quit IRC11:31
*** carl_cai has quit IRC11:35
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats  https://review.openstack.org/61368011:38
*** ChosSimbaOne has quit IRC11:49
quiquellAJaeger, ianw: At zuul centos-7 images when do we install python-pip ?11:54
quiquellAJaeger, ianw: Do the images already have it ?11:55
*** xek__ is now known as xek12:02
*** jpena is now known as jpena|lunch12:03
*** emine__ has joined #openstack-infra12:08
*** pbourke has quit IRC12:11
*** Emine has quit IRC12:11
*** pbourke has joined #openstack-infra12:11
*** jcoufal has joined #openstack-infra12:13
*** kjackal has quit IRC12:13
*** kjackal has joined #openstack-infra12:19
*** janki has quit IRC12:20
*** quiquell is now known as quiquell|lunch12:24
*** ansmith has quit IRC12:31
*** dtantsur is now known as dtantsur|brb12:32
*** jcoufal has quit IRC12:35
*** dhill_ has joined #openstack-infra12:36
openstackgerritMerged openstack-infra/os-loganalyze master: Adds libmagic on plaforms missing it  https://review.openstack.org/61626612:40
openstackgerritMerged openstack-infra/os-loganalyze master: Move zuul job definition inside the respository  https://review.openstack.org/61627312:40
*** rkukura has quit IRC12:47
*** rlandy has joined #openstack-infra12:51
*** AJaeger_ has joined #openstack-infra12:52
*** ccamacho has quit IRC12:53
*** AJaeger has quit IRC12:55
*** jcoufal has joined #openstack-infra12:55
openstackgerritMerged openstack-infra/project-config master: Move os-loganalize job definition inside the project  https://review.openstack.org/61627512:56
*** fuentess has joined #openstack-infra12:56
openstackgerritMerged openstack-infra/nodepool master: Only set basepython once  https://review.openstack.org/61594212:57
*** boden has joined #openstack-infra12:58
*** ifat_afek has quit IRC12:58
*** panda is now known as panda|off12:59
*** weshay is now known as weshay_ruck12:59
*** ifat_afek has joined #openstack-infra13:00
*** emine__ has quit IRC13:00
*** emine__ has joined #openstack-infra13:00
*** e0ne has quit IRC13:03
*** yamamoto has quit IRC13:07
*** yamamoto has joined #openstack-infra13:09
*** dpawlik has quit IRC13:11
*** yamamoto has quit IRC13:14
fungiquiquell|lunch: we have a role for installing them, but at the moment it mostly does nothing because we preinstall them in our images via dib (it would be nice to change that soon)13:17
*** ccamacho has joined #openstack-infra13:18
*** trown|outtypewww is now known as trown13:20
*** jpena|lunch is now known as jpena13:24
*** ifat_afek has quit IRC13:24
fungiinfra-root: just a heads up, i don't anticipate being around much today. have a few pre-travel errands to which i must attend before i begin my road movie to berlin in the morning13:26
fungi(can't drive out the way we drove in!)13:26
*** dpawlik has joined #openstack-infra13:26
*** eharney has joined #openstack-infra13:27
*** ifat_afek has joined #openstack-infra13:28
*** janki has joined #openstack-infra13:29
openstackgerritAndriy Shevchenko proposed openstack/ansible-role-cloud-launcher master: Update min tox version to 2.0  https://review.openstack.org/61174513:29
*** ifat_afek has quit IRC13:31
*** dpawlik has quit IRC13:31
*** e0ne has joined #openstack-infra13:37
*** kgiusti has joined #openstack-infra13:40
*** quiquell|lunch is now known as quiquell13:41
ssbarneawho can help me adding ssbarnea rfolco chandan to elastic-recehck https://review.openstack.org/#/admin/groups/218,members  so we can take care of unclassified failures?13:49
*** rh-jelabarre has joined #openstack-infra13:51
*** ansmith has joined #openstack-infra13:51
ssbarneafungi: AJaeger_  ^ can you help?13:51
*** ccamacho has quit IRC13:52
AJaeger_ssbarnea: I don't have those permissions - better ask clarkb as PTL13:52
*** ccamacho has joined #openstack-infra13:53
ssbarneaAJaeger_: i almost forgot you are not on infra-core :)13:54
*** AJaeger_ has quit IRC13:54
*** jcoufal has quit IRC13:55
ssbarneaquiquell: regarding pip availability: if I remember well pip is not installed from rpm, is installed using easy_install. Still, you should write code that works with any flavor of pip, regardless on how it ended up installed ;)13:55
quiquellssbarnea: Just want to know if it's cooked in the image or not, and looks like it's cooke with dib13:57
*** AJaeger has joined #openstack-infra13:57
*** dpawlik has joined #openstack-infra13:58
*** yamamoto has joined #openstack-infra13:58
*** dpawlik has quit IRC13:59
fungissbarnea: i recommend mriedem, mtreinish or clarkb work with you as they've been the ones active on classifying failures in elastic-recheck lately. i'm not super familiar with the workflow so not sure how to confirm you are either13:59
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: remove setup.py check from check-python-release role  https://review.openstack.org/61655013:59
dhellmannAJaeger : ^^14:00
fungissbarnea: i take it you're seeking membership in that group so that you can approve proposed logstash queries?14:00
fungi(after all, anyone can propose queries to improve the classification rate, they just need a reviewer to approve them once proposed)14:01
*** ccamacho has quit IRC14:02
*** pbourke has quit IRC14:02
*** ssbarnea has quit IRC14:02
*** derekh has quit IRC14:02
*** jpich has quit IRC14:02
*** roman_g has quit IRC14:02
*** bnemec has quit IRC14:02
*** jtomasek has quit IRC14:02
*** aojea has quit IRC14:02
*** kota_ has quit IRC14:02
*** kukacz has quit IRC14:02
*** rascasoft has quit IRC14:02
*** auristor has quit IRC14:02
*** niska has quit IRC14:02
*** tdasilva has quit IRC14:02
*** trown has quit IRC14:02
*** jaosorior has quit IRC14:02
*** bdodd has quit IRC14:02
*** bstinson has quit IRC14:02
*** pleia2 has quit IRC14:02
*** tonyb has quit IRC14:02
*** scas has quit IRC14:02
*** dmellado has quit IRC14:02
*** hwoarang has quit IRC14:02
*** eandersson has quit IRC14:02
*** mmethot has quit IRC14:02
*** dabukalam has quit IRC14:02
*** mattoliverau has quit IRC14:02
*** tobiash has quit IRC14:02
*** rook has quit IRC14:02
*** pbourke has joined #openstack-infra14:02
*** hwoarang has joined #openstack-infra14:02
*** roman_g has joined #openstack-infra14:02
*** bstinson_ has joined #openstack-infra14:02
*** yamamoto has quit IRC14:03
*** aojea has joined #openstack-infra14:03
*** jpich has joined #openstack-infra14:03
*** dmellado has joined #openstack-infra14:04
*** jtomasek has joined #openstack-infra14:04
*** dpawlik has joined #openstack-infra14:04
*** kukacz has joined #openstack-infra14:04
*** mmethot has joined #openstack-infra14:04
*** bdodd has joined #openstack-infra14:04
*** aojea has quit IRC14:04
*** kota_ has joined #openstack-infra14:04
*** dpawlik has quit IRC14:04
*** rascasoft has joined #openstack-infra14:04
*** aojea has joined #openstack-infra14:04
*** dpawlik has joined #openstack-infra14:04
*** tobiash has joined #openstack-infra14:05
*** trown has joined #openstack-infra14:06
*** pleia2 has joined #openstack-infra14:07
*** tonyb has joined #openstack-infra14:07
*** ssbarnea has joined #openstack-infra14:07
*** bnemec has joined #openstack-infra14:07
*** jaosorior has joined #openstack-infra14:08
*** dabukalam has joined #openstack-infra14:10
*** mriedem has joined #openstack-infra14:13
*** cgoncalves has joined #openstack-infra14:15
*** bobh has joined #openstack-infra14:16
cgoncalvesis http://status.openstack.org/openstack-health displaying jobs for anyone?14:16
*** aojea has quit IRC14:20
*** agopi|brb has quit IRC14:21
fungiScript timed out before returning headers: openstack-health14:23
fungiin the apache error log for the health api server14:23
fungiit's able to connect to the trove instance though14:25
fungicgoncalves: mtreinish: restarting apache on health.openstack.org seems to have gotten http://status.openstack.org/openstack-health/ working again. not sure why it was hanging but i wonder if it's getting overloaded14:28
cgoncalvesfungi, yep, working now. thanks!14:28
fungiyeah, cacti graphs suggest something may have been hammering it14:29
fungimemory usage, load average, cpu utilization were all really high14:29
fungii wonder if my restarting apache killed a long-running query (or many of them)14:29
*** janki has quit IRC14:30
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2860&rra_id=all14:30
fungiactive memory jumped by a factor of something like 6x starting around 13:10z14:31
cgoncalvesfungi, one other question, I can't seem to find (periodic) job neutron-lbaas-to-octavia-migration there. e.g. http://status.openstack.org/openstack-health/#/g/project/openstack~2Fneutron-lbaas?duration=P3M&searchJob=migration14:36
fungicgoncalves: openstack-health relies on the subunit2sql database to get information. does the neutron-lbaas-to-octavia-migration job produce subunit data for indexing?14:37
*** auristor has joined #openstack-infra14:40
cgoncalvesfungi, uh, no. would there be some other way of checking latest run results?14:41
fungicgoncalves: http://zuul.openstack.org/builds?job_name=neutron-lbaas-to-octavia-migration14:42
fungi(queried from the "builds" tab of the zuul status dashboard)14:42
cgoncalvesfungi, aaah, thanks! bookmarking that page :)14:42
funginp14:43
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Report tenant and project specific resource usage stats  https://review.openstack.org/61630614:45
*** agopi|brb has joined #openstack-infra14:46
*** agopi|brb is now known as agopi14:46
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Add resource metadata to nodes  https://review.openstack.org/61626214:47
*** bstinson_ is now known as bstinson14:52
*** emine__ has quit IRC14:53
openstackgerrit98k proposed openstack-infra/project-config master: remove job settings for oslo repositories  https://review.openstack.org/61657614:56
*** aojea has joined #openstack-infra14:57
*** Emine has joined #openstack-infra15:02
*** dtantsur|brb is now known as dtantsur15:02
openstackgerritSorin Sbarnea proposed openstack-infra/elastic-recheck master: Preparatory fixes for py3  https://review.openstack.org/61657815:07
*** shardy has joined #openstack-infra15:08
*** ccamacho has joined #openstack-infra15:09
*** shardy has quit IRC15:09
*** shardy has joined #openstack-infra15:09
*** shardy has quit IRC15:09
openstackgerritSorin Sbarnea proposed openstack-infra/os-loganalyze master: Removing pypy  https://review.openstack.org/46876215:10
*** shardy has joined #openstack-infra15:11
*** tdasilva has joined #openstack-infra15:11
openstackgerritSorin Sbarnea proposed openstack-infra/os-loganalyze master: Stop ignoring H904 hacking rule in tox  https://review.openstack.org/46876115:12
mtreinishfungi: I haven't been tracking health.o.o closely15:16
mtreinishit could have been a bunch of slow queries, but slow queries aren't anything new15:17
openstackgerritSorin Sbarnea proposed openstack-infra/elastic-recheck master: Move all zull jobs definitions inside repo  https://review.openstack.org/61658215:17
mtreinishand there shouldn't be that much local cpu load. It's mostly just averaging and resampling of time series in the api process15:17
fungithanks, good to know15:18
fungiwhatever was causing it, the apache restart seems to have interrupted it and restored the service to a sane state15:18
fungiguess we'll keep an eye out for similar future behavior15:18
mtreinishoh, I guess maybe if there is a big payload the json serialization could take some number crunching too15:18
openstackgerritSorin Sbarnea proposed openstack-infra/project-config master: Move elastic-recheck zull jobs definitions inside project repo  https://review.openstack.org/61658315:18
fungiit looked like it basically pegged one cpu on the vm15:19
mtreinishyeah, looking at the graph one core was definitely sitting there actively thinking about something15:21
mtreinishnot sure about what though15:21
dhellmanngerrit admins: There are no members of the cross-project-spec-liaissons-chair group, and so we cannot approve any patches in the openstack-specs repo. Could someone please add me to that group so we can unblock some of the CI-related changes? https://review.openstack.org/#/admin/groups/1371,members15:28
*** armax has joined #openstack-infra15:35
*** hamzy has quit IRC15:36
ssbarneaclarkb: when you have few minutes please check the elastic-receck reviews, starting with the question on https://review.openstack.org/#/c/608968/ - thanks.15:37
*** shardy has quit IRC15:40
fricklerdhellmann: it looks like any of the liaisons should be able to do it, see https://review.openstack.org/#/c/593702/ . doesn't need a +2 like for other repos.15:44
fricklerdhellmann: mike perez left the chair position half a year ago, not sure how the formal procedures to replace him would look like, so if the other solution works, I'd prefer that for now15:45
dhellmannfrickler : ah, I didn't read closely enough15:45
dhellmannthat repo isn't used, I just don't want it to be out of date if we decide we need to resurrect it15:45
*** derekh has joined #openstack-infra15:52
dhellmannfrickler : jroll is a member of that group but not able to add anyone to it. Can you tell who has that permission?15:54
jrolldhellmann: I believe the "included groups" usually defines editing permissions, and there is none here15:54
fricklerdhellmann: yeah, I was just checking that, and the owner is the chair again ... so regarding the fact that you are TC chair anyway, now you have another one ;)15:55
dhellmannfrickler : thanks15:55
fricklerdhellmann: jroll: it is the "owners" list here https://review.openstack.org/#/admin/groups/1372,info15:56
jrollfrickler: ah right, thanks15:56
openstackgerritMerged openstack-infra/git-review master: As suggested by pep8 don't compare boolean values or empty sequences  https://review.openstack.org/22117215:57
*** noama has quit IRC16:06
*** pcaruana has quit IRC16:14
*** sthussey has joined #openstack-infra16:16
*** dklyle has quit IRC16:17
*** imacdonn has quit IRC16:17
*** imacdonn has joined #openstack-infra16:18
*** gyee has joined #openstack-infra16:18
*** panda|off is now known as panda|rover16:19
*** jcoufal has joined #openstack-infra16:20
*** jcoufal has quit IRC16:20
*** jcoufal has joined #openstack-infra16:21
*** hamzy has joined #openstack-infra16:22
*** eernst has joined #openstack-infra16:22
*** dklyle has joined #openstack-infra16:23
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Fix commit-msg hook installation for worktrees  https://review.openstack.org/39720616:23
fungicorvus: earlier today i ran `sudo MAILMAN_SITE_DIR=/srv/mailman/openstack /usr/lib/mailman/bin/rmlist openstack-internships` and it kept the archive at http://lists.openstack.org/pipermail/openstack-internships/ but removed the list from the list index. does that seem like a sane choice for retiring other lists in the future?16:27
clarkbmorning16:29
*** irdr has joined #openstack-infra16:30
*** irdr has quit IRC16:31
* fungi is doing a terrible job of not being here16:33
*** irdr has joined #openstack-infra16:36
*** irdr has quit IRC16:37
*** eernst has quit IRC16:38
*** quiquell is now known as quiquell|off16:39
*** yamamoto has joined #openstack-infra16:40
corvusfungi: hrm, i'm not sure that was effective.  it still shows up in http://lists.openstack.org/cgi-bin/mailman/listinfo and http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-internships still exists, and i can submit a subscription request on that form and get a response.16:40
*** slaweq has quit IRC16:40
clarkbssbarnea: left a comment16:40
*** agopi is now known as agopi|food16:41
ssbarneaclarkb: thanks! I was thinking about the same approach with suffix and prefix. going to update it now.16:43
*** yamamoto has quit IRC16:44
*** sshnaidm|rover is now known as sshnaidm|notrove16:46
*** e0ne has quit IRC16:48
*** xarses has joined #openstack-infra16:48
fungicorvus: weird. oh!16:49
openstackgerritSorin Sbarnea proposed openstack-infra/elastic-recheck master: Add check for failed to connect to mirror.*.openstack.org  https://review.openstack.org/60896816:49
fungicorvus: i think i needed to delete it from puppet16:49
fungii'll push up a patch16:49
clarkbssbarnea: the other thing you can do is test the query at http://logstash.openstack.org16:49
fungipretty sure it got recreated by puppet16:49
corvusfungi: haha :(16:50
*** sshnaidm|notrove is now known as sshnaidm|wknd16:53
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Retire the openstack-internships mailing list  https://review.openstack.org/61662816:54
fungicorvus: ^16:54
*** slaweq has joined #openstack-infra16:58
*** xarses has quit IRC16:59
*** rkukura has joined #openstack-infra17:00
*** panda|rover is now known as panda|rover|off17:02
ssbarneaclarkb: i am playing with logstash but I get very weird behavior, most of the time I get no answer at all, so even for the same query I may get results or not....17:05
clarkbssbarnea: in the top right is the time window. Perhaps that is shifting away from when there are records?17:06
clarkbssbarnea: one thing I like to do is start with a more broad search then refine it, that way I know there are results and if they go away I know I've broken my query17:07
*** nicolasbock_ has joined #openstack-infra17:09
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver  https://review.openstack.org/60440417:09
*** agopi|food is now known as agopi17:10
*** trown is now known as trown|lunch17:11
*** derekh has quit IRC17:14
*** derekh has joined #openstack-infra17:15
*** gfidente has quit IRC17:19
*** irclogbot_1 has quit IRC17:31
*** ccamacho has quit IRC17:32
fuentessclarkb, hi, can you help us hold a VM from the kata-containers CI ?17:34
*** jpich has quit IRC17:34
clarkbfuentess: yup, kata-runsh job on the proxy repo?17:34
fuentessyes please17:34
clarkbfuentess: next failure should be held17:36
*** gtmanfred is now known as yaml17:36
*** yaml is now known as gtmanfred17:36
fuentessclarkb, thanks17:37
openstackgerritMerged openstack-infra/nodepool master: Pin docker for k8s test  https://review.openstack.org/61639817:38
*** sshnaidm|wknd is now known as sshnaidm|off17:42
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Convert kubernetes config docs to zuul-sphinx  https://review.openstack.org/61664317:45
*** diablo_rojo has joined #openstack-infra17:48
*** jpena is now known as jpena|off17:51
ssbarneais anyone aware of twine check failing with exception on docutils like http://logs.openstack.org/74/615974/3/check/test-release-openstack/5682347/job-output.txt.gz#_2018-11-08_17_22_19_29851417:52
clarkbssbarnea: no, that is interesting. bug in twine maybe? or maybe missing data field it assumes will be there and the error message is not good?17:54
clarkbssbarnea: the purpose of that check is to have twine determine if the package contents are valid for pypi17:54
agopiclarkb, the same twine check ran perfectly fine locally for me.17:55
ssbarneayep, running twine locally works perfect, only on zuul I seen that weird error.17:57
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Alter the metasyntactic variable in driver docs  https://review.openstack.org/61664617:58
*** irclogbot_1 has joined #openstack-infra18:00
*** derekh has quit IRC18:01
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Normalize sidebar in docs  https://review.openstack.org/61664918:01
clarkbssbarnea: agopi I notice that the job installs docutils and pygments into one virtualenv then runs twine out of another. That may be related? I'm currently trying to reproduce it locally18:02
clarkbssbarnea: agopi I am able to reproduce it18:02
*** irclogbot_1 has quit IRC18:05
ssbarneaclarkb: waiting for details....18:05
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2  https://review.openstack.org/58353518:07
clarkbssbarnea: well I don't have many yet. Other than I get the same traceback if I follow the steps of the job. run `python3 setup.py sdist bdist_wheel --universal` then make python2 venv and install twine there then run twince check dist/*18:07
clarkbssbarnea: produces the same traceback for me18:07
clarkblooks like it is trying to render the description field which has a value of UNKNOWN in the PKG-INFO18:07
clarkbmaybe it needs to be known?18:07
*** pcaruana has joined #openstack-infra18:10
fuentessclarkb,  opened a PR in the proxy repo but I cannot see the the zuul job, any idea? I already checked and the zuul config seems ok on our side...18:10
fuentessclarkb: https://github.com/kata-containers/proxy/pull/12718:11
clarkbfuentess: that is curious. THe app is still enabled on your side?18:11
agopiokay that probably might be the cause, I was using python2 to build the package. will check again thanks clarkb :)18:11
fuentessyes, still enabled18:11
ssbarneaclarkb: i was to replicate it too: it happens only when we build using *python3*, the wheel is valid but the sdist is not.18:11
ssbarneamc18:12
openstackgerritMerged openstack-infra/system-config master: Retire the openstack-internships mailing list  https://review.openstack.org/61662818:12
clarkbfuentess: I wonder if the depends on confused it?18:13
clarkbcorvus: can you look at https://github.com/kata-containers/proxy/pull/127 the depends on format there might be sufficiently confusing to zuul such that the job won't run? specifically its not a full url?18:14
ssbarneacomparing the archive I found that on py3: Description is UNKNOWN but on py2 is False - this being the only difference.18:14
fuentessclarkb, ahh ok, let me remove it18:14
*** trown|lunch is now known as trown18:14
clarkbfuentess: should be Depends-On: https://github.com/stuff18:15
fuentessclarkb, ok, thanks, although I actually dont need it, I have removed it.. lets see how it goes18:15
ssbarneaisn't sdist part of python distro?18:16
clarkbsdist is the source distribution of python packges18:16
*** kgiusti has left #openstack-infra18:22
*** irclogbot_1 has joined #openstack-infra18:25
*** armax has quit IRC18:25
*** bobh has quit IRC18:29
corvusclarkb: looking.  this will take a few minutes.18:31
*** irclogbot_1 has quit IRC18:32
*** irclogbot_1 has joined #openstack-infra18:34
ssbarneaclarkb: thanks for helping with twine issue, is clearly caused by py3 but avoidable.18:35
clarkbssbarnea: any idea why python3 is causing the problem?18:35
clarkbI wonder if running twine under python3 fixes it?18:35
ssbarneaclarkb: running undep python2 would avoid it, this happens only when you build the package using python3. and when the package has no "Description" section in setup.cfg18:36
clarkbah18:37
*** kgiusti has joined #openstack-infra18:38
*** Emine has quit IRC18:40
clarkbdhellmann: re ^ is the intent to run twine under python3 too?18:44
clarkbwondering if that was missed in the transition to python3 by default18:44
ssbarneaclarkb: i do not find this bug critical, only confusing. i do expect any project to have a description, probably that is why it was missed.18:45
ssbarneastill seem like a something we should raise a a bug, not sure why if that is python bug or is in setuptools or something else.18:46
ssbarneai do expect to see others hitting it again as now, especially with newer projects.18:46
agopiclarkb, ssbarnea it's still a bit confusing given that building package happens in one env and then twine check in another18:48
dhellmannclarkb : yeah, did I mess that up?18:48
agopinot sure if that's the expected behavior.18:49
clarkbdhellmann: http://logs.openstack.org/74/615974/3/check/test-release-openstack/5682347/ara-report/result/b995038f-05ce-44b7-bb2a-d7fee9474304/ shows twine running under python2 but the sdist was run under python318:49
clarkbdhellmann: the pip install of twine may need to do pip3 ?18:49
dhellmannlikely18:49
dhellmannlet me see if I can fix that quickly18:49
ssbarneapython3 -m twine18:50
clarkbagopi: ssbarnea it is possibly a bug in twine's handling of that case18:50
clarkband the error message should be fixed there?18:50
corvusclarkb: i'm wondering if the githubeventconnector stopped18:51
openstackgerritDmitry Tyzhnenko proposed openstack-infra/git-review master: Add reviewers by group alias on upload  https://review.openstack.org/19504318:51
clarkbcorvus: is that a thread in zuul-web?18:51
corvusclarkb: in scheduler18:51
corvusi'm grepping for it now; i'll look for errors around the last instance when i find it.  i may sigusr2 in a bit18:52
dhellmannclarkb : ensure-twine is defined in zuul-jobs, so that fix is going to require multiple steps18:52
clarkbcorvus: ok let me know if I can help18:53
clarkbdhellmann: maybe we add ensure-twine3 or make it a config flag for backward compat?18:53
dhellmannthe pattern we've used elsewhere is to specify a python executable name in a variable18:53
dhellmannso we can do {{whatevervarforpython}} -m pip install18:54
dhellmannalthough that assumes pip is present for that python too18:54
dhellmannsigh18:54
clarkbdhellmann: I think {{ whatevervarforpython }} /usr/bin/pip will work though18:55
*** electrofelix has quit IRC18:55
dhellmanndo you have any idea where we install pip?18:57
clarkbI think the images may bake that in18:58
clarkb(because we need virtualenv?)18:58
dhellmannok18:58
ssbarneaclarkb: how about calling python3 -m pip -- this works for sure, no need to guess location.18:58
dhellmannssbarnea : that only works if pip is already installed for python3, though18:59
dhellmannthat's the bit I can't figure out how to verify by reading roles18:59
*** dpawlik has quit IRC19:00
clarkbdhellmann: instead of using -m pip just run the pip executable under the python version directly I think that works19:00
*** dpawlik has joined #openstack-infra19:00
*** armax has joined #openstack-infra19:00
dhellmannok19:01
clarkb`python2 $(which pip)` and `python3 $(which pip)`?19:01
clarkbI do this with pbr a lot19:01
ssbarneaclarkb: no need to to as this logic would mess pip, what is wrong in using the module calling?19:02
ssbarneahere is the quickstart.sh ensure_pip version (bash) https://github.com/openstack/tripleo-quickstart/blob/master/quickstart.sh#L15419:03
clarkbssbarnea: -m only looks in your module path, if pip isn't in the module path for that version of python it won't work19:04
*** Emine has joined #openstack-infra19:04
dhellmannclarkb : where would I look to find a list of things being installed into the images?19:04
ssbarneaclarkb: if pip is not in your module path.... you don't have pip. pip is not a standalone script, it needs to be installed as a module.19:04
dhellmannyeah, I don't think running it without the -m is going to work19:05
clarkbdhellmann: https://nb01.openstack.org/ubuntu-bionic-0000000018.log is the full build log for a recent bionic build. You can navigate the root there to see logs for other distros19:06
*** ifat_afek has joined #openstack-infra19:06
ssbarneadhellmann: the same applies to other tools, like virtualenv for example. the only safe way to call them is as modules. (virtualenv is called virtualenv-3 on python3)19:06
dhellmannclarkb : is there a manifest or something that I could look at?19:07
*** ifat_afek has quit IRC19:07
openstackgerritDoug Hellmann proposed openstack-infra/zuul-jobs master: enable setting python version for ensure-twine  https://review.openstack.org/61667319:08
dhellmannclarkb : step 1 ^19:08
clarkbdhellmann: if you grep for dpkg-query --show there is a dpkg listing19:08
dhellmanndo we install pip from system packages?19:08
clarkbno, and we don't appear to have a pip freeze output19:08
clarkb(we stopped listing all files in the image because it made the image builds take forever stating all the things iirc)19:09
dhellmannI don't really want to wade through tons of output if I can look at the instructions used to build the image19:09
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: ensure twine is installed with python 3  https://review.openstack.org/61667619:11
dhellmannstep 2 ^19:11
clarkbhttps://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv that is what we use to install pip and virtualenv19:11
dhellmannok, the readme there says it installed pip319:12
*** diablo_rojo has quit IRC19:12
dhellmannso running -m pip should work fine19:12
clarkbI think as long as the chosen distro has python2 and python3 it tries to accomodate both?19:13
clarkbbut will do one or the other if the distro doesn't have both19:13
clarkb(those jobs run on ubuntu and should have both)19:13
agopithanks dhellmann clarkb, looks like dhellmann's commits helps clear up the air.19:14
*** diablo_rojo has joined #openstack-infra19:15
dhellmannthe only thing this would break would be that check job and actual releases, so the impact will be minimal. if we aren't confident of the approach based on the readmes and whatever else, then maybe we should go ahead with and then I can run a test using the release-test repo (we can't test speculatively, unfortunately)19:15
clarkbI think we can go ahead and make the changes and then test it. It is easy enough to revert 616676 if necessary19:16
dhellmannyeah19:16
dhellmannI made the project-config change on top of https://review.openstack.org/#/c/616550/ because we want to remove that call of setup.py too19:16
clarkbssbarnea has a comment on the zuul-jobs change if you want to make it consistent with other roles19:16
dhellmannssbarnea , clarkb : we have a release_python variable in the release role, so I followed that pattern19:17
dhellmannis that not the right pattern to follow?19:17
ssbarneai hope that you may find that "pattern" useful, i would be really happy if you also adopt it. we have in many roles in tripleo-quickstart now aftet f28 enablement work.19:17
clarkboh I thought that was for consistency with other zuul-jobs roles19:18
*** Emine has quit IRC19:18
clarkb(I don't really care)19:18
dhellmannI suppose I could reuse release_python since we want them set the same, but that's managed by a different role19:18
dhellmannthat would result in one less patch, since we wouldn't need the one in project-config19:19
clarkbya I think python_cmd is nice in that we can use it everywhere without too much ambiguity19:19
clarkbbut release_python would be weird there I think19:19
dhellmannzuul-jobs has a release_python and sphinx_python19:19
dhellmannso I followed that pattern19:19
dhellmannI don't see any roles setting python_cmd19:20
clarkbya probably fine to stick to that pattern for now then change to python_cmd if we want to in the future and apply it globally in zuul-jobs19:20
dhellmannpresumably because we want the flexibility of using different pythons in different roles?19:20
dhellmannok19:20
ssbarneai personally think that we don't really need one variable for each tool, seems .... bit of maintenance nightmare. i volunteer to help migrating to one standard solution.19:21
*** auristor has quit IRC19:22
dhellmannI don't know the best practice for using the same variable across roles that may not even be defined in the same git repo19:22
dhellmannI need to step out. I'll check back in a bit.19:23
clarkbdhellmann: it comes down to whether or not you want to select things globally (so can set it at a host level) or if you want fine grained control to say run twine under python2 and sphinx under 3 and release under 319:23
clarkb(there is precedence but understanding that is confusing enough that using explicit values is nice)19:24
ssbarneadhellmann: the nice part is that you don't need to define it, unless you want to override its implicit value. we copied those two tasks in each role that needed python_cmd.19:24
*** ralonsoh has quit IRC19:24
*** auristor has joined #openstack-infra19:30
*** mriedem has quit IRC19:32
*** auristor has quit IRC19:35
*** auristor has joined #openstack-infra19:37
ssbarneaclarkb: need bit of help for browbeat team: the core-release group is empty : https://review.openstack.org/#/admin/projects/openstack/browbeat,access19:38
clarkbssbarnea: looks like Joe Talerico added the acl.19:40
ssbarneaclarkb: can you please add agopi and jtaleric to release group?19:40
clarkbI've added jtaleric as the person who updated the acl file. They can add others as they like19:41
agopioh okay that'd work clarkb19:41
agopithanks clarkb and ssbarnea :)19:41
ssbarneaclarkb: perfect! thanks. they can sort it themselves now.19:41
ssbarneaclarkb: i was curious about the release process in general. to my knowledge the only way to release is to push a tag. this means that gerrit cannot really be used to review a release (tagging) - like having multiple people blessing the release. right?19:44
*** aojea has quit IRC19:44
clarkbssbarnea: sort of. The process openstack has built is to have a release control repository into whcih you put release data, it gets reviewed adn approved and those changes merging push tags that kick off the process19:44
clarkbssbarnea: if you are not part of that system you are correct. The pushing of tags yourself will immediately kick off the release process so if you want to review things you need to do it before pushing to gerrit19:45
clarkbssbarnea: usually what the infra team does is just informally poll the cores and ask for input on the project, version, sha1 being tagged to make sure we are in rough aggreement19:45
ssbarneaclarkb: thanks.19:46
clarkbssbarnea: agopi other things to keep in mind are that deleting or moving a tag later doesn't update dowsntreams that have already pulled the old tag19:46
clarkbso in general we don't delete or move tags, instead we highly encourage you to push a subsequent release if there was an error in the last one19:47
openstackgerritDmitry Tyzhnenko proposed openstack-infra/git-review master: Add reviewers by group alias on upload  https://review.openstack.org/19504319:49
ssbarneai told him that once you push a tag, you don't edit it, you only create a new one if the prev one was buggy.19:49
agopithanks ssbarnea clarkb :)19:52
*** dave-mccowan has joined #openstack-infra19:56
ianwdhellmann: did you get all your answers on the pip install in images?19:56
*** rlandy is now known as rlandy|brb19:57
*** ifat_afek has joined #openstack-infra19:57
*** mriedem has joined #openstack-infra19:59
*** jtomasek has quit IRC20:05
*** dpawlik has quit IRC20:10
dhellmannianw : I think we have enough confidence that we will have pip3 that we can approve the changes and then test them with some inconsequential release (like release-test)20:16
*** e0ne has joined #openstack-infra20:19
*** rlandy|brb is now known as rlandy20:19
*** e0ne has quit IRC20:22
ianwdhellmann: on the infra nodes, yes the idea is that "pip" and "virtualenv" are both python2 installers, while "pip3" and "virtualenv3" are python320:23
ianwthat's what the prior referenced dib element jumps through hoops to provide20:24
*** eharney has quit IRC20:24
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Use six for cross python compatibility  https://review.openstack.org/61668820:25
ianwi think the "virtualenv3" might an ... anti-pattern ... or something, but we had things using it.  it seems "python3 -m venv" is the way to go, maybe20:25
AJaegerconfig-core, could you review https://review.openstack.org/616550  , please? I think we should remove the role completely - but like somebody to confirm...20:25
*** dpawlik has joined #openstack-infra20:26
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Use openstacksdk submit_task  https://review.openstack.org/61638720:29
*** eharney has joined #openstack-infra20:29
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Revert "Pin docker for k8s test"  https://review.openstack.org/61640420:30
*** dpawlik has quit IRC20:30
ssbarneaAJaeger: not a core but by reading it the body of the role I would say role removal is what we want.20:30
*** dave-mccowan has quit IRC20:33
clarkbAJaeger: dhellmann I think twine installs docutils and pygments now? I'm checking20:35
AJaegerclarkb: it does not - just checked20:36
clarkbdocutils==0.14 and Pygments==2.2.0 installed in the venv where I installed twine and I did not install them myself20:36
clarkbtwine==1.12.120:36
* AJaeger needs better glasses ;(20:36
AJaegerclarkb: you're right...20:36
* AJaeger sends a patch...20:36
clarkbI think they are deps of readme_renderer which is a dep of twine20:37
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove check-python-release role  https://review.openstack.org/61668920:39
ianwAJaeger: so now the role is just a virtualenv & package installs?20:39
AJaegerianw: that would be the result of 616550  - and I think we don't need that, so proposed 616689 as alternative20:40
AJaegerdhellmann: are you sure that twine does the README check? Or is that not done anymore?20:41
*** mriedem has quit IRC20:41
ianwi appreciate the "get it working" angle :)  but if we can clean it up now and not forget, that sounds good ... around if any help needed triggering anything to test20:42
AJaegerJust checked - twine does the README check, so we can remove it....20:43
AJaegerconfig-core, dhellmann, I suggest to go with 616689 instead of 616550 - please review20:43
dhellmannAJaeger : http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/pti-python-tarball/check.yaml#n720:45
*** dklyle has quit IRC20:45
*** dklyle has joined #openstack-infra20:45
AJaegerdhellmann: yes, I remove that line - nothing needs this20:45
*** mriedem has joined #openstack-infra20:45
AJaegerdhellmann: ignore my comment. Yes, that is fine.20:45
*** dklyle has quit IRC20:46
dhellmannAJaeger : line 8 runs "twine check" and that needs docutils and pygments installed20:46
openstackgerritSorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2  https://review.openstack.org/58353520:46
AJaegerdhellmann: and that is done by ensure-twine, clarkb and myself just tested20:46
*** mftech has joined #openstack-infra20:46
dhellmannok20:46
dhellmannI guess some time next week I'll try to find time to rebase the rest of my changes on top of this20:47
AJaegerdhellmann: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/ensure-twine/tasks/main.yaml#n1420:47
dhellmannI wish we could separate fixing things from cleaning things up20:47
*** rh-jelabarre has quit IRC20:48
AJaegerdhellmann: this is not proper cleaning IMHO - and the fix was easy. the role had setup and execution, you removed the execution - and then the setup should really go as well.20:49
dhellmannit was not clear, and still is not, that the cleanup isn't needed20:49
AJaegerdhellmann: perhaps your commit message was not clear - but I did not see that anything here is broken that needs a half-fix20:50
dhellmannI guess you're saying the readme rendering thing is what causes those dependencies to be installed?20:50
AJaegerdhellmann: yes20:50
dhellmannmy change was "stop doing something redundant"20:50
dhellmannit's fine, I don't care20:50
AJaegerclarkb: want to +2A https://review.openstack.org/616689 , please? Then I rebase dhellmann's change on top of it...20:52
*** ifat_afek has quit IRC20:53
clarkbAJaeger: you mean rebase the python3 fix?20:55
clarkbAJaeger: thats done20:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: ensure twine is installed with python 3  https://review.openstack.org/61667620:57
openstackgerritMerged openstack-infra/nodepool master: Convert kubernetes config docs to zuul-sphinx  https://review.openstack.org/61664320:57
openstackgerritMerged openstack-infra/nodepool master: Alter the metasyntactic variable in driver docs  https://review.openstack.org/61664620:57
AJaegerclarkb: yeah - I just did the rebase of 61667620:58
openstackgerritMerged openstack-infra/nodepool master: Normalize sidebar in docs  https://review.openstack.org/61664920:58
AJaegerclarkb: in case you or ianw want to +2 616676 as well - I'll +2A now the zuul-jobs one...20:59
clarkbI've actually got to pop out for a bit now. Lunch time20:59
AJaegerclarkb: enjoy!21:00
openstackgerritMerged openstack-infra/project-config master: Remove check-python-release role  https://review.openstack.org/61668921:03
ssbarneamaybe we can also merge https://review.openstack.org/#/c/471078/ ?21:07
*** kgiusti has left #openstack-infra21:10
AJaegerssbarnea: better ask ianw and clarkb for system-config ^21:10
ssbarneasure, anytime time to go to bed...  that change is more than an year old, it probably can wait anotehr day or two21:11
* AJaeger takes ssbarnea's advice and calls it a day ;)21:13
openstackgerritMerged openstack-infra/zuul-jobs master: enable setting python version for ensure-twine  https://review.openstack.org/61667321:16
*** mriedem has left #openstack-infra21:18
*** nicolasbock_ has quit IRC21:32
*** Emine has joined #openstack-infra21:36
*** eharney has quit IRC21:37
*** ansmith has quit IRC21:37
*** hamzy has quit IRC21:43
*** diablo_rojo has quit IRC21:50
*** florianf is now known as florianf|afk21:51
*** kjackal has quit IRC21:56
*** kjackal has joined #openstack-infra21:57
*** trown is now known as trown|outtypewww22:00
*** manjeets has quit IRC22:00
corvusclarkb: i don't see a smoking gun in the logs, just that the last time the GithubEventConnector did anything was 03 nov22:02
corvusclarkb: i'm going to sigusr222:02
*** diablo_rojo has joined #openstack-infra22:03
*** agopi is now known as agopi|brb22:04
mordredianw: re: the nodepool stats patch above - ifyou're in the brainspace for it, I'm working on an update looking towards using the keystoneauth patch that will allow us to get rid of the taskmanager22:06
mordredianw: https://review.openstack.org/#/c/614834/ - would love some eyes on it - but it's WIP because I need to write docs22:06
mordredianw: it might be worth looking at https://review.openstack.org/#/c/604926 and https://review.openstack.org/#/c/605043/ for context22:06
mordredianw: BUT - if you don't want to and have other things on your plate - totally no worries22:07
*** mattoliverau has joined #openstack-infra22:07
clarkbcorvus: ok, back from lunch now22:08
*** agopi|brb has quit IRC22:09
ianwmordred: i'll take a look :)22:10
*** mftech has quit IRC22:11
clarkbthe gerrit bug tracker tells me that gerrit upstream may be doing work to fix the relationship between changes data presentation22:12
clarkbthis is exciting22:12
*** mftech has joined #openstack-infra22:12
mordredoooh22:13
mordredthat is exciting22:13
clarkbLabels: -UX-Incoming UX-Acked was the update22:14
clarkbhoping that -UX-Incoming means we are working on it now22:14
*** Emine has quit IRC22:14
clarkban update on the logs.o.o thing. I don't see any more recent errors for wsgi timeouts22:19
clarkbI think we should go ahead and get dmsimard's change to add processes in22:19
*** eernst has joined #openstack-infra22:19
clarkbwe also had a large spike in data transfer around 1800UTC today so it is working when put under load22:19
clarkbhttps://review.openstack.org/#/c/616297/ is the chagne from dmsmimard if anyone else can review it (system-config)22:23
logan-has anyone seen xenial DIB builds failing with "Operation failed: Too many levels of symbolic links" during post-install.d/80-enable-sshd-service recently? I had a working build on 11/1 and then today they are failing consistently this way. http://paste.openstack.org/raw/734447/22:27
ianwlogan:- hrm, not aware of that.  there probably has been a dib release in between but not sure we touched anything related to that22:28
clarkblogan-: let me grep our logs22:28
logan-was just looking at https://nb01.openstack.org/ubuntu-xenial-0000000018.log and it seems like nb01 was at least able to complete a build today22:29
clarkblogan-: https://nb01.openstack.org/ubuntu-xenial-0000000018.log ya that log looks ok22:29
clarkblogan-: are you using ubuntu or ubuntu-minimal element?22:29
clarkb(we use ubuntu-minimal)22:29
logan-ubuntu-minimal22:30
logan-ianw: good tip, i confirmed my 11/1 build was on DIB 2.17, today is 2.1822:30
logan-let me try a build on 2.17 and see how that owkrs22:31
ianwlogan-: can you run with "-x" ?22:31
logan-yep22:31
logan-on it22:31
logan-thanks22:31
ianwlooks like https://review.openstack.org/#/c/503844/ touched that file, but only for the openrc path?22:32
*** mriedem has joined #openstack-infra22:36
clarkbthats a systemctl error message?22:37
logan-light googling led me to think that yeah22:37
logan-wondering if its more an ubuntu package update problem than dib22:37
*** kjackal has quit IRC22:39
*** Emine has joined #openstack-infra22:41
openstackgerritErik Olof Gunnar Andersson proposed openstack-infra/os-loganalyze master: Add support for Magnum  https://review.openstack.org/61670522:42
*** otherwiseguy has joined #openstack-infra22:45
*** mriedem has quit IRC22:46
*** eharney has joined #openstack-infra22:47
*** manjeets has joined #openstack-infra22:55
clarkbdiablo_rojo: you wanted stickers right? /me attempts to remember to pack them22:56
*** Emine has quit IRC23:06
*** mftech has quit IRC23:12
*** Tim_ok has quit IRC23:12
*** mftech has joined #openstack-infra23:12
*** mftech has quit IRC23:13
*** agopi|brb has joined #openstack-infra23:14
*** kjackal has joined #openstack-infra23:15
*** slaweq has quit IRC23:15
*** kjackal has quit IRC23:25
* logan- tries to find a paste site that will accept this DIB log23:28
*** jcoufal has quit IRC23:28
logan-2.17 completed, 2.18 failed.. going to upload debug logs for both23:28
clarkblogan-: gist is usually good for giant logs23:28
logan-thanks23:29
diablo_rojoclarkb, yes please! the small ants :)23:29
ianwlogan-: it's usually on the last bit that's useful ... but the real problem can be a few hundred lines back23:30
ianwas everything unwinds23:30
logan-working 2.17 - https://gist.githubusercontent.com/logan2211/3b4312ff2f77f7aec73fb36caac45a87/raw/f5839c411f3fb3998dd3867ce379d5cfaf366606/DIB%25202.17%2520build23:31
logan-failed 2.18 - https://gist.githubusercontent.com/logan2211/3b4312ff2f77f7aec73fb36caac45a87/raw/f5839c411f3fb3998dd3867ce379d5cfaf366606/DIB%25202.18%2520build23:31
logan-afk a while -- but ill try to figure it out later this evening if it is still unsolved. thanks clarkb & ianw23:32
*** fuentess has quit IRC23:33
clarkbfwiw I'm slowly fading out myself. I keep getting distracted by the list of things I need to do around the house and making it hard to focus on work. I should probably just go drain the that pipe then I don't have to worry about it freezing while I am gone23:33
ianwhrm, appears to be the same openssh version openssh-server amd64 1:7.2p2-4ubuntu2.6 [335 kB]23:33
ianwclarkb: so weird, i have never once worried about a pipe freezing in all my years :)23:34
clarkbianw: this one runs through the garage wall which isn't a heated space so its got a plug I can take out upstream with a valve there allowing me to drain everything in the unheated space23:35
clarkbnext time I need plumbing work done I am going to have them replace that valve with a ball valve so that closign it is easy23:36
dmsimardianw: does it even get close to freezing temperature where you are ? haha23:37
ianwyou'll sometimes get a morning frost on the ground in Sydney, but nothing that will freeze a pipe23:38
clarkbits not forecast to freeze here really and should be fine. But I know I won't have to worry about it later (or when in berlin) if I just take care of it now23:38
*** boden has quit IRC23:39
*** boden has joined #openstack-infra23:40
*** dklyle has joined #openstack-infra23:44
*** boden has quit IRC23:45
corvusclarkb: ball valves are great -- but (and i have recently seen this is not theoretical) they can leak slightly, and if they do, you can't tighten them more.  so they're great for things that aren't open-ended (ie, in the middle of plumbing).  for a spigot, or something like it, a regular globe valve may be better.23:51
clarkbhuh23:53
corvus(i just got ball valves for my washing machine, and they're great, except one of them has like a one drop/minute leak, which was a minor inconvenience when i had no washing machine.  now that i do, and the hoses are all hooked up, it's fine.)23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!