| corvus | andreaf: or, if what you're concerned about is the inventory prep, and you know what you want the local.conf file to look like, you could make a patch which removes all jobs from the repo, and just generates a local.conf. that would iterate quickly. | 00:00 |
|---|---|---|
| clarkb | ianw: I approved that then realized, don't you have to define linaro_username and linaro_password? | 00:01 |
| clarkb | ianw: looks like we do that in site.pp under the nodepool.o.o nl and nb servers? | 00:02 |
| clarkb | but I don't see it for linaro yet | 00:02 |
| ianw | ohhh, hmm i added it to hiera | 00:02 |
| clarkb | I guess if its verbatim linaro_username in hiera it may just work? | 00:02 |
| clarkb | ianw: look at citycloud_username for example in nl's site.pp entry for example of what I am wondering about | 00:02 |
| clarkb | we do a probably unnecessary explict retrieval from hiera then pass that through | 00:03 |
| *** yamamoto has joined #openstack-infra | 00:03 | |
| *** slaweq has joined #openstack-infra | 00:04 | |
| pabelanger | ah, I missed that too | 00:04 |
| clarkb | ianw: I think the idea is to separate the nodepool credentials from the control plane credentials that go in the other clouds.yaml files? | 00:04 |
| ianw | $citycloud_username = hiera('nodepool_citycloud_username', 'username') | 00:04 |
| ianw | what's the second argument? | 00:04 |
| clarkb | ianw: its a default I think to make our noop apply happy | 00:05 |
| clarkb | so when we noop apply we use the 'username' value for that variable | 00:05 |
| ianw | ahh, then we don't have that on the passwords? | 00:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml https://review.openstack.org/546834 | 00:06 |
| clarkb | ya thts weird | 00:06 |
| clarkb | its possible we don't need any of that default stuff? | 00:06 |
| *** daidv has quit IRC | 00:06 | |
| pabelanger | we used too, but it was removed I think. Something about missing hiera data in production adding test password into live data I think | 00:06 |
| *** daidv has joined #openstack-infra | 00:06 | |
| andreaf | corvus how do I make a patch that removes all jobs? I cannot remove jobs that are defined somewhere else... oh I guess I could use depends-on for that | 00:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml https://review.openstack.org/546834 | 00:07 |
| ianw | ^ better i hope | 00:07 |
| *** slaweq has quit IRC | 00:09 | |
| *** yamamoto has quit IRC | 00:09 | |
| corvus | andreaf: yes, that doesn't work as well if you have jobs attached to the project in project-config. depends-on won't help with that. | 00:10 |
| clarkb | corvus: andreaf speaking of removing jobs, not sure if you saw mriedem's questions about how to remoe grenade from last supported branch without having to update all the repos | 00:11 |
| andreaf | upz I missed that | 00:11 |
| clarkb | basically if we put the job listing in the tree branches then you have to go to the last branch and remove grenade whenever the previous one is EOL'd | 00:11 |
| clarkb | or we can manage that centrally | 00:11 |
| clarkb | and use a branch exclusion on the job directly | 00:12 |
| clarkb | neither option seemed great | 00:12 |
| andreaf | clarkb why centrally is not ok? | 00:13 |
| andreaf | clarkb I was thinking have zuul v3 grenade job defined in grenade | 00:13 |
| clarkb | andreaf: I think mostly because its what we've done for years and its always been a pain | 00:13 |
| clarkb | but its doable | 00:13 |
| corvus | why isn't updating the repos ok? :) | 00:13 |
| clarkb | corvus: I think because there are many of them so its seen as a lot of work | 00:15 |
| clarkb | you'd have to updatea ll the repos that are cogated on grenade. Probably 20-30 of them? | 00:15 |
| *** VW has joined #openstack-infra | 00:15 | |
| corvus | okay, i'm happy to talk about solutions, but we'll need to all get together to find out what requirements we value | 00:17 |
| *** cmurphy has quit IRC | 00:17 | |
| andreaf | clarkb corvus grenade has branches can't we just go on the version of the job on the last stable branch and put a branch filter in there? | 00:17 |
| *** cmurphy has joined #openstack-infra | 00:17 | |
| corvus | sorry, i can't design this now, i have too much to do before leaving for dublin | 00:18 |
| clarkb | andreaf: and then update the last supported branch to not run? I think mriedem didn't really like that because its weird to have a job then not run it. But ya I too am running out of time to get packing and errands done if I spend too much time doing othe rthings | 00:18 |
| clarkb | but I have a dishwasher again so thats a major item off the list \o/ | 00:19 |
| andreaf | :) | 00:20 |
| andreaf | yeah we can try to sort this out in Dublin | 00:20 |
| *** VW has quit IRC | 00:20 | |
| *** xarses has quit IRC | 00:21 | |
| *** stakeda has joined #openstack-infra | 00:22 | |
| *** lbragstad has quit IRC | 00:23 | |
| *** eernst has quit IRC | 00:23 | |
| *** slaweq has joined #openstack-infra | 00:25 | |
| *** dhill_ has quit IRC | 00:26 | |
| *** abelur_ has joined #openstack-infra | 00:28 | |
| *** slaweq has quit IRC | 00:29 | |
| *** olaph has joined #openstack-infra | 00:30 | |
| *** rossella_s has quit IRC | 00:30 | |
| *** abelur_ has quit IRC | 00:30 | |
| *** olaph1 has quit IRC | 00:31 | |
| *** edmondsw has quit IRC | 00:31 | |
| *** rossella_s has joined #openstack-infra | 00:32 | |
| *** agopi has quit IRC | 00:37 | |
| *** r-daneel has quit IRC | 00:43 | |
| *** threestrands has joined #openstack-infra | 00:44 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch astara release jobs to not use pypi https://review.openstack.org/547226 | 00:45 |
| *** eernst has joined #openstack-infra | 00:45 | |
| *** slaweq has joined #openstack-infra | 00:46 | |
| *** cuongnv has joined #openstack-infra | 00:47 | |
| *** VW has joined #openstack-infra | 00:47 | |
| *** cuongnv has quit IRC | 00:47 | |
| *** cuongnv has joined #openstack-infra | 00:48 | |
| *** slaweq has quit IRC | 00:50 | |
| *** VW has quit IRC | 00:52 | |
| *** rossella_s has quit IRC | 00:52 | |
| *** rossella_s has joined #openstack-infra | 00:53 | |
| *** rlandy|bbl is now known as rlandy|ruck | 00:56 | |
| *** claudiub has joined #openstack-infra | 00:58 | |
| *** caphrim007_ has quit IRC | 01:03 | |
| *** yamamoto has joined #openstack-infra | 01:05 | |
| *** slaweq has joined #openstack-infra | 01:06 | |
| *** aviau has quit IRC | 01:07 | |
| *** aviau has joined #openstack-infra | 01:07 | |
| *** caphrim007 has joined #openstack-infra | 01:07 | |
| *** caphrim007_ has joined #openstack-infra | 01:08 | |
| *** yamamoto has quit IRC | 01:11 | |
| *** slaweq has quit IRC | 01:11 | |
| *** caphrim007 has quit IRC | 01:11 | |
| *** caphrim007_ has quit IRC | 01:12 | |
| *** wolverineav has quit IRC | 01:16 | |
| *** rossella_s has quit IRC | 01:18 | |
| *** rossella_s has joined #openstack-infra | 01:19 | |
| *** VW has joined #openstack-infra | 01:19 | |
| *** VW has quit IRC | 01:24 | |
| *** slaweq has joined #openstack-infra | 01:27 | |
| *** iyamahat_ has quit IRC | 01:29 | |
| *** claudiub has quit IRC | 01:30 | |
| *** xarses has joined #openstack-infra | 01:31 | |
| *** xarses has quit IRC | 01:31 | |
| *** slaweq has quit IRC | 01:32 | |
| *** xarses has joined #openstack-infra | 01:32 | |
| *** yamahata has quit IRC | 01:32 | |
| *** dangers_away has quit IRC | 01:37 | |
| *** dangers_away has joined #openstack-infra | 01:42 | |
| *** eernst has quit IRC | 01:45 | |
| *** eernst has joined #openstack-infra | 01:45 | |
| *** slaweq has joined #openstack-infra | 01:48 | |
| *** xarses has quit IRC | 01:52 | |
| *** eernst has quit IRC | 01:53 | |
| *** slaweq has quit IRC | 01:53 | |
| *** VW has joined #openstack-infra | 01:55 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint update track by summit https://review.openstack.org/547234 | 01:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 testing to ozj https://review.openstack.org/546847 | 01:57 |
| *** VW has quit IRC | 02:00 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint update track by summit https://review.openstack.org/547234 | 02:00 |
| *** rossella_s has quit IRC | 02:00 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 02:00 | |
| *** rossella_s has joined #openstack-infra | 02:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 testing to ozj https://review.openstack.org/546847 | 02:03 |
| mriedem | i will yell at you all about grenade in dublin | 02:04 |
| *** mriedem has quit IRC | 02:06 | |
| *** yamamoto has joined #openstack-infra | 02:07 | |
| *** slaweq has joined #openstack-infra | 02:09 | |
| *** slaweq_ has joined #openstack-infra | 02:09 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** slaweq_ has quit IRC | 02:14 | |
| *** slaweq has quit IRC | 02:14 | |
| *** verdurin has quit IRC | 02:15 | |
| *** ykarel|away has joined #openstack-infra | 02:17 | |
| *** edmondsw has joined #openstack-infra | 02:17 | |
| *** verdurin has joined #openstack-infra | 02:20 | |
| *** kiennt26 has joined #openstack-infra | 02:21 | |
| *** edmondsw has quit IRC | 02:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/gear master: Add zuul.yaml file https://review.openstack.org/547237 | 02:22 |
| *** bhujay has joined #openstack-infra | 02:24 | |
| *** zhenguo has joined #openstack-infra | 02:27 | |
| *** caphrim007 has joined #openstack-infra | 02:27 | |
| *** rlandy|ruck has quit IRC | 02:30 | |
| *** slaweq has joined #openstack-infra | 02:30 | |
| *** xarses has joined #openstack-infra | 02:30 | |
| *** andreww has joined #openstack-infra | 02:31 | |
| *** yamahata has joined #openstack-infra | 02:32 | |
| *** caphrim007 has quit IRC | 02:32 | |
| *** mattoliverau_ is now known as mattoliverau | 02:35 | |
| *** xarses has quit IRC | 02:35 | |
| *** slaweq has quit IRC | 02:35 | |
| *** ykarel|away has quit IRC | 02:36 | |
| ianw | it's alive! 211.148.24.197 ... nb03 built that image, uploaded it, i booted it manually but that's a full arm64 infra node!!! | 02:45 |
| prometheanfire | :D:D | 02:45 |
| prometheanfire | ianw: that is awesome | 02:45 |
| prometheanfire | ianw: I've been keywording arm64 for a while, nice to see it get better testing | 02:46 |
| *** slaweq has joined #openstack-infra | 02:46 | |
| *** salv-orl_ has joined #openstack-infra | 02:46 | |
| ianw | yep, it will soon be just a "-arm64" suffix to node requests away :) | 02:47 |
| prometheanfire | :D | 02:48 |
| ianw | with "only" room for 8 nodes we'll need to keep it small | 02:48 |
| ianw | that said, it's an extremely generous allocation ... just we could flood that very quickly :) | 02:48 |
| prometheanfire | packet.net? | 02:49 |
| *** salv-orlando has quit IRC | 02:49 | |
| prometheanfire | some arm64 people gentoo side have been bugging me for an arm64 image | 02:49 |
| *** annp has joined #openstack-infra | 02:49 | |
| prometheanfire | and we have a donation from them | 02:50 |
| prometheanfire | might want to look into that too | 02:50 |
| prometheanfire | iirc they are openstack | 02:50 |
| prometheanfire | maybe :| | 02:51 |
| *** hongbin has joined #openstack-infra | 02:51 | |
| *** slaweq has quit IRC | 02:51 | |
| *** slaweq has joined #openstack-infra | 02:51 | |
| ianw | if they want a stress test, they've come to the right place | 02:53 |
| clarkb | ianw: nice! | 02:54 |
| prometheanfire | does infra do clouds other than openstack ones? | 02:54 |
| clarkb | prometheanfire: currently no. Nodepool has provider driver support now so in theory we could. Then there is layer8 to sort out like do we test openstack on not openstack etc | 02:55 |
| *** slaweq has quit IRC | 02:55 | |
| prometheanfire | sure | 02:56 |
| clarkb | ianw did the puppet for clouds.yaml get sorted? | 02:57 |
| ianw | clarkb: reviews on https://review.openstack.org/546834 and https://review.openstack.org/547216/ should allow us to launch a node via nl03 | 02:58 |
| clarkb | ianw: on thr first ine thats applying to nodepool.o.o not nl0x | 02:59 |
| clarkb | we might want to trim the puppetry there to make that clearer | 03:00 |
| clarkb | I should get to real computer and leave some real comments. This is really exciting though | 03:01 |
| clarkb | dib can have proper multiarch gates potentially | 03:01 |
| *** tinwood has quit IRC | 03:05 | |
| clarkb | ianw: left comments, hopefully that helps | 03:05 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml https://review.openstack.org/546834 | 03:05 |
| ianw | clarkb: ahh, good call! should have kept searching | 03:05 |
| ianw | that should do builder and launcher; i'm not sure why nodepool.o.o needs any of that, but anyway | 03:06 |
| *** tinwood has joined #openstack-infra | 03:06 | |
| *** rossella_s has quit IRC | 03:06 | |
| clarkb | ya nodepool.o.o doesn't need it anymore at all | 03:06 |
| ianw | i'm sure the interested parties will thrash out something over the config-file, and splitting it etc etc; at that point we can re-do the puppet to be a bit more sane | 03:06 |
| clarkb | I think a good followup would be to remove all the nodepool.o.o config that isn't for zookeeper | 03:07 |
| *** rossella_s has joined #openstack-infra | 03:07 | |
| clarkb | in site.pp | 03:07 |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| pabelanger | clarkb: ianw: left comment on 547216. Pretty sure it will work as setup | 03:09 |
| clarkb | oh right we generate proper requests for min ready | 03:10 |
| *** slaweq has joined #openstack-infra | 03:10 | |
| *** bhujay has quit IRC | 03:10 | |
| clarkb | for some reason I thought the provider handlers just polled and booted as necessary. I'll update my comment | 03:10 |
| ianw | pabelanger: yeah, your comment was my understanding | 03:10 |
| *** Faster-Fanboi_ has quit IRC | 03:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Cleanup nodepool.openstack.org to be a zk server https://review.openstack.org/547248 | 03:10 |
| clarkb | updated | 03:11 |
| clarkb | +2 to both now | 03:12 |
| *** slaweq_ has joined #openstack-infra | 03:12 | |
| ianw | thanks, just ducking out, bib | 03:12 |
| *** Faster-Fanboi has joined #openstack-infra | 03:12 | |
| *** rossella_s has quit IRC | 03:12 | |
| clarkb | ya I've got dinner now can check in after ot make sure there isn't anything else I can help with | 03:12 |
| clarkb | also zuul is looking much more happy after a restart | 03:13 |
| *** yamamoto has quit IRC | 03:14 | |
| *** rossella_s has joined #openstack-infra | 03:14 | |
| *** slaweq has quit IRC | 03:15 | |
| *** slaweq_ has quit IRC | 03:16 | |
| *** xarses has joined #openstack-infra | 03:18 | |
| *** olaph1 has joined #openstack-infra | 03:19 | |
| *** olaph has quit IRC | 03:20 | |
| *** andreww has quit IRC | 03:21 | |
| *** xarses has quit IRC | 03:23 | |
| *** caphrim007 has joined #openstack-infra | 03:26 | |
| *** yamamoto has joined #openstack-infra | 03:27 | |
| *** slaweq has joined #openstack-infra | 03:28 | |
| *** slaweq has quit IRC | 03:32 | |
| *** olaph has joined #openstack-infra | 03:33 | |
| *** olaph1 has quit IRC | 03:33 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml https://review.openstack.org/546834 | 03:37 |
| *** slaweq has joined #openstack-infra | 03:43 | |
| *** slaweq has quit IRC | 03:48 | |
| *** cshastri has joined #openstack-infra | 03:53 | |
| *** vivsoni__ has joined #openstack-infra | 03:55 | |
| *** gyee has quit IRC | 03:56 | |
| *** vivsoni_ has quit IRC | 03:58 | |
| *** slaweq has joined #openstack-infra | 03:59 | |
| *** slaweq has quit IRC | 04:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add arm64 nodes to launcher https://review.openstack.org/547216 | 04:05 |
| *** edmondsw has joined #openstack-infra | 04:06 | |
| *** janki has joined #openstack-infra | 04:06 | |
| *** ykarel|away has joined #openstack-infra | 04:08 | |
| openstackgerrit | Anshul Jain proposed openstack/diskimage-builder master: Fix for passing user defined value for satellite cert for rhel-common. https://review.openstack.org/547255 | 04:08 |
| *** edmondsw has quit IRC | 04:11 | |
| *** psachin has joined #openstack-infra | 04:20 | |
| *** links has joined #openstack-infra | 04:21 | |
| *** slaweq has joined #openstack-infra | 04:23 | |
| *** udesale has joined #openstack-infra | 04:25 | |
| *** slaweq has quit IRC | 04:29 | |
| *** slaweq has joined #openstack-infra | 04:31 | |
| *** links has quit IRC | 04:33 | |
| ianw | 2018-02-23 04:30:29,369 INFO nodepool.PoolWorker.linaro-cn1-main: Assigning node request <NodeRequest {'nodes': [], 'stat': ZnodeStat(czxid=1744993377, mzxid=1744993377, ctime=1519360228481, mtime=1519360228481, version=0, cversion=0, aversion=0, ephemeralOwner=99428362472260057, dataLength=161, numChildren=0, pzxid=1744993377), 'declined_by': [], 'requestor': 'zuul01.openstack.org', 'state_time': 1519360228.4781013, 'state': 'requested', | 04:33 |
| ianw | 'node_types': ['ubuntu-xenial'], 'id': '200-0002846867', 'reuse': True}> | 04:33 |
| ianw | oh dear, why is nodepool thinking linaro can handle anything but arm64 images :/ | 04:34 |
| *** slaweq has quit IRC | 04:35 | |
| eglute | hello Infra, any idea why this would not be merged? https://review.openstack.org/#/c/529836/ Stuck in "Needs Verified Label" status, though it appears that Zuul has verified it and no obvious errors? I did recheck a few times. | 04:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Fix ubuntu-xenial-arm64 label https://review.openstack.org/547257 | 04:36 |
| *** zhenguo has quit IRC | 04:37 | |
| ianw | eglute: it depends-on a unmerged change | 04:37 |
| eglute | ianw thanks, didn't notice that. | 04:38 |
| *** sree_ has joined #openstack-infra | 04:40 | |
| *** andreas_s has joined #openstack-infra | 04:40 | |
| *** sree_ is now known as Guest16541 | 04:40 | |
| sc` | any infra folks around? i'm branching repos and i inadvertently branched one on the wrong commit id | 04:40 |
| sc` | never mind. i can cherry-pick. it's new enough | 04:42 |
| *** olaph1 has joined #openstack-infra | 04:43 | |
| *** olaph has quit IRC | 04:43 | |
| *** andreas_s has quit IRC | 04:44 | |
| *** Faster-Fanboi has quit IRC | 04:46 | |
| *** slaweq has joined #openstack-infra | 04:47 | |
| *** links has joined #openstack-infra | 04:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Fix ubuntu-xenial-arm64 label https://review.openstack.org/547257 | 04:51 |
| *** slaweq has quit IRC | 04:51 | |
| *** Faster-Fanboi has joined #openstack-infra | 04:52 | |
| *** hongbin has quit IRC | 04:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix ubuntu-xenial-arm64 label https://review.openstack.org/547257 | 05:02 |
| *** slaweq has joined #openstack-infra | 05:02 | |
| *** baoli has quit IRC | 05:03 | |
| *** baoli has joined #openstack-infra | 05:03 | |
| *** cuongnv has quit IRC | 05:05 | |
| *** pgadiya has joined #openstack-infra | 05:06 | |
| *** slaweq has quit IRC | 05:07 | |
| *** Faster-Fanboi has quit IRC | 05:14 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add arm64 min-ready to nl03 https://review.openstack.org/547271 | 05:16 |
| *** liuyulong has joined #openstack-infra | 05:16 | |
| ianw | clarkb: ^ pabelanger -- you were right, nl01 won't launch the min-ready nodes | 05:17 |
| *** kiennt26 has quit IRC | 05:17 | |
| *** aeng has quit IRC | 05:18 | |
| *** slaweq has joined #openstack-infra | 05:18 | |
| ianw | | 0002683657 | linaro-cn1 | ubuntu-xenial-arm64 | c7bb6da6-52e5-4aab-88f1-ec0f1b392a0c | 211.148.24.200 | | ready | 00:00:03:43 | unlocked | | 05:19 |
| ianw | and that is a ready arm64 node! | 05:19 |
| *** VW has joined #openstack-infra | 05:19 | |
| *** Faster-Fanboi has joined #openstack-infra | 05:20 | |
| liuyulong | hello guys, is there any way to enable one neutron-agent-extension during the tempest zuul job? Such as L3 agent extension `fip_qos`. | 05:21 |
| liuyulong | #link https://github.com/openstack/neutron/blob/master/setup.cfg#L119 | 05:21 |
| *** slaweq has quit IRC | 05:23 | |
| *** VW has quit IRC | 05:24 | |
| *** baoli has quit IRC | 05:25 | |
| liuyulong | I have one patch: https://review.openstack.org/#/c/529784/, which needs the `fip_qos` L3 agent extension enabled. So, how to enable the job `neutron-tempest-plugin-dvr-multinode-scenario` and `neutron-tempest-plugin-scenario-linuxbridge` L3 agent extension? | 05:25 |
| *** caphrim007 has quit IRC | 05:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add arm64 min-ready to nl03 https://review.openstack.org/547271 | 05:31 |
| *** niska has quit IRC | 05:31 | |
| *** slaweq has joined #openstack-infra | 05:34 | |
| *** slaweq has quit IRC | 05:39 | |
| *** slaweq has joined #openstack-infra | 05:41 | |
| *** niska has joined #openstack-infra | 05:42 | |
| *** slaweq has quit IRC | 05:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rework base-publish-static jobs using protected https://review.openstack.org/545234 | 05:47 |
| *** slaweq has joined #openstack-infra | 05:50 | |
| *** claudiub has joined #openstack-infra | 05:50 | |
| *** slaweq has quit IRC | 05:54 | |
| *** edmondsw has joined #openstack-infra | 05:55 | |
| *** VW has joined #openstack-infra | 05:55 | |
| *** ykarel|away is now known as ykarel | 05:56 | |
| *** pcichy has joined #openstack-infra | 05:56 | |
| *** udesale_ has joined #openstack-infra | 05:58 | |
| *** ykarel_ has joined #openstack-infra | 05:58 | |
| *** eernst has joined #openstack-infra | 05:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch astara release jobs to not use pypi https://review.openstack.org/547226 | 05:59 |
| *** edmondsw has quit IRC | 05:59 | |
| *** VW has quit IRC | 06:00 | |
| *** udesale has quit IRC | 06:00 | |
| *** ykarel has quit IRC | 06:01 | |
| *** kjackal has joined #openstack-infra | 06:03 | |
| *** slaweq has joined #openstack-infra | 06:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/547282 | 06:09 |
| *** slaweq has quit IRC | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** namnh has joined #openstack-infra | 06:12 | |
| *** threestrands has quit IRC | 06:13 | |
| *** namnh has quit IRC | 06:16 | |
| *** annp has quit IRC | 06:16 | |
| *** slaweq has quit IRC | 06:16 | |
| *** namnh has joined #openstack-infra | 06:16 | |
| *** annp has joined #openstack-infra | 06:16 | |
| *** udesale_ is now known as udesale | 06:18 | |
| *** cuongnv has joined #openstack-infra | 06:21 | |
| *** slaweq has joined #openstack-infra | 06:21 | |
| *** ramishra has joined #openstack-infra | 06:22 | |
| *** slaweq has quit IRC | 06:26 | |
| openstackgerrit | Anshul Jain proposed openstack/diskimage-builder master: Fix for passing user defined value for satellite cert for rhel-common. https://review.openstack.org/547255 | 06:26 |
| *** slaweq has joined #openstack-infra | 06:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-devstack-dsvm-py36-updown-fedora-26 https://review.openstack.org/547147 | 06:38 |
| *** olaph has joined #openstack-infra | 06:41 | |
| *** olaph1 has quit IRC | 06:41 | |
| *** slaweq has quit IRC | 06:42 | |
| *** andreas_s has joined #openstack-infra | 06:42 | |
| *** eernst has quit IRC | 06:43 | |
| *** andreas_s has quit IRC | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:47 | |
| *** slaweq has quit IRC | 06:52 | |
| *** spligak has quit IRC | 06:52 | |
| *** pcaruana has joined #openstack-infra | 06:56 | |
| *** olaph1 has joined #openstack-infra | 06:56 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove devstack-dsvm-py36-updown-fedora-26 job https://review.openstack.org/547152 | 06:56 |
| *** olaph has quit IRC | 06:58 | |
| *** andreas_s has joined #openstack-infra | 07:00 | |
| *** pcaruana has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-infra | 07:02 | |
| *** rossella_s has quit IRC | 07:08 | |
| *** rossella_s has joined #openstack-infra | 07:09 | |
| *** ramishra has quit IRC | 07:09 | |
| *** VW has joined #openstack-infra | 07:10 | |
| *** ramishra has joined #openstack-infra | 07:10 | |
| *** udesale_ has joined #openstack-infra | 07:12 | |
| *** pblaho has quit IRC | 07:12 | |
| *** ykarel__ has joined #openstack-infra | 07:14 | |
| *** VW has quit IRC | 07:14 | |
| *** udesale has quit IRC | 07:15 | |
| *** snapiri has quit IRC | 07:16 | |
| *** ykarel_ has quit IRC | 07:16 | |
| *** ykarel__ is now known as ykarel|lunch | 07:18 | |
| *** dhajare has joined #openstack-infra | 07:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Skip autohold if no autohold was requested https://review.openstack.org/547302 | 07:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/547282 | 07:20 |
| *** numans_ has joined #openstack-infra | 07:21 | |
| *** panda|sick has quit IRC | 07:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Skip autohold if no autohold was requested https://review.openstack.org/547302 | 07:22 |
| *** slaweq has joined #openstack-infra | 07:23 | |
| *** numans has quit IRC | 07:24 | |
| *** rossella_s has quit IRC | 07:24 | |
| *** panda has joined #openstack-infra | 07:27 | |
| *** slaweq has quit IRC | 07:27 | |
| *** rossella_s has joined #openstack-infra | 07:28 | |
| *** olaph1 has quit IRC | 07:30 | |
| *** eventingmonkey has quit IRC | 07:30 | |
| *** bstinson has quit IRC | 07:30 | |
| *** zxiiro has quit IRC | 07:30 | |
| *** GregHouse has quit IRC | 07:30 | |
| *** berendt has quit IRC | 07:30 | |
| *** Zara has quit IRC | 07:30 | |
| *** ilpianista has quit IRC | 07:30 | |
| *** Miouge has quit IRC | 07:30 | |
| *** AJaeger has quit IRC | 07:30 | |
| *** vdrok has quit IRC | 07:30 | |
| *** electrical has quit IRC | 07:30 | |
| *** khappone has quit IRC | 07:30 | |
| *** hwoarang has quit IRC | 07:30 | |
| *** tobberydberg has quit IRC | 07:30 | |
| *** aviau has quit IRC | 07:31 | |
| *** clayg has quit IRC | 07:31 | |
| *** ying_zuo has quit IRC | 07:31 | |
| *** jlabarre has quit IRC | 07:31 | |
| *** inc0 has quit IRC | 07:31 | |
| *** sileht has quit IRC | 07:31 | |
| *** ilpianista_ has quit IRC | 07:31 | |
| *** gpocentek has quit IRC | 07:31 | |
| *** Shrews has quit IRC | 07:31 | |
| *** dgonzalez has quit IRC | 07:31 | |
| *** Keitaro has quit IRC | 07:31 | |
| *** e0ne has joined #openstack-infra | 07:31 | |
| *** Keitaro has joined #openstack-infra | 07:32 | |
| *** rossella_s has quit IRC | 07:32 | |
| *** rossella_s has joined #openstack-infra | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:37 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Normalize hostname for statsd https://review.openstack.org/547309 | 07:38 |
| *** jtomasek has joined #openstack-infra | 07:38 | |
| *** rossella_s has quit IRC | 07:41 | |
| *** rossella_s has joined #openstack-infra | 07:42 | |
| *** slaweq has quit IRC | 07:42 | |
| *** slaweq has joined #openstack-infra | 07:42 | |
| *** edmondsw has joined #openstack-infra | 07:43 | |
| *** slaweq has quit IRC | 07:44 | |
| *** tosky has joined #openstack-infra | 07:44 | |
| *** slaweq has joined #openstack-infra | 07:44 | |
| *** VW has joined #openstack-infra | 07:44 | |
| *** edmondsw has quit IRC | 07:48 | |
| *** VW has quit IRC | 07:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add python35 jobs to osc-placement https://review.openstack.org/547315 | 07:51 |
| *** hwoarang has joined #openstack-infra | 07:52 | |
| *** aviau has joined #openstack-infra | 07:53 | |
| *** clayg has joined #openstack-infra | 07:53 | |
| *** ying_zuo has joined #openstack-infra | 07:53 | |
| *** jlabarre has joined #openstack-infra | 07:53 | |
| *** inc0 has joined #openstack-infra | 07:53 | |
| *** sileht has joined #openstack-infra | 07:53 | |
| *** ilpianista_ has joined #openstack-infra | 07:53 | |
| *** gpocentek has joined #openstack-infra | 07:53 | |
| *** Shrews has joined #openstack-infra | 07:53 | |
| *** dgonzalez has joined #openstack-infra | 07:53 | |
| *** olaph1 has joined #openstack-infra | 07:57 | |
| *** eventingmonkey has joined #openstack-infra | 07:57 | |
| *** bstinson has joined #openstack-infra | 07:57 | |
| *** GregHouse has joined #openstack-infra | 07:57 | |
| *** zxiiro has joined #openstack-infra | 07:57 | |
| *** berendt has joined #openstack-infra | 07:57 | |
| *** Zara has joined #openstack-infra | 07:57 | |
| *** ilpianista has joined #openstack-infra | 07:57 | |
| *** Miouge has joined #openstack-infra | 07:57 | |
| *** AJaeger has joined #openstack-infra | 07:57 | |
| *** vdrok has joined #openstack-infra | 07:57 | |
| *** electrical has joined #openstack-infra | 07:57 | |
| *** khappone has joined #openstack-infra | 07:57 | |
| *** tobberydberg has joined #openstack-infra | 07:57 | |
| *** olaph has joined #openstack-infra | 07:59 | |
| *** njohnston has quit IRC | 07:59 | |
| *** it-praktyk[m] has quit IRC | 08:00 | |
| *** ilpianista_ has quit IRC | 08:00 | |
| *** aspiers[m] has quit IRC | 08:00 | |
| *** masayukig[m] has quit IRC | 08:00 | |
| *** olaph1 has quit IRC | 08:00 | |
| *** njohnston has joined #openstack-infra | 08:02 | |
| *** vivsoni__ has quit IRC | 08:05 | |
| *** claudiub has quit IRC | 08:07 | |
| *** claudiub|2 has joined #openstack-infra | 08:08 | |
| *** vivsoni has joined #openstack-infra | 08:08 | |
| *** claudiub has joined #openstack-infra | 08:10 | |
| *** claudiub|2 has quit IRC | 08:11 | |
| *** eventingmonkey has quit IRC | 08:11 | |
| *** bstinson has quit IRC | 08:11 | |
| *** zxiiro has quit IRC | 08:11 | |
| *** GregHouse has quit IRC | 08:11 | |
| *** berendt has quit IRC | 08:11 | |
| *** Zara has quit IRC | 08:11 | |
| *** ilpianista has quit IRC | 08:11 | |
| *** Miouge has quit IRC | 08:11 | |
| *** AJaeger has quit IRC | 08:11 | |
| *** vdrok has quit IRC | 08:11 | |
| *** electrical has quit IRC | 08:11 | |
| *** khappone has quit IRC | 08:11 | |
| *** tobberydberg has quit IRC | 08:11 | |
| *** aviau has quit IRC | 08:11 | |
| *** clayg has quit IRC | 08:11 | |
| *** ying_zuo has quit IRC | 08:11 | |
| *** jlabarre has quit IRC | 08:11 | |
| *** inc0 has quit IRC | 08:11 | |
| *** sileht has quit IRC | 08:11 | |
| *** gpocentek has quit IRC | 08:11 | |
| *** Shrews has quit IRC | 08:11 | |
| *** dgonzalez has quit IRC | 08:11 | |
| *** hashar has joined #openstack-infra | 08:12 | |
| *** amoralej|off is now known as amoralej | 08:14 | |
| *** Zara has joined #openstack-infra | 08:14 | |
| *** jpena|off is now known as jpena | 08:16 | |
| *** zoli is now known as zoli|wfh | 08:16 | |
| *** zoli|wfh is now known as zoli | 08:16 | |
| *** rcernin has quit IRC | 08:16 | |
| *** VW has joined #openstack-infra | 08:18 | |
| *** tesseract has joined #openstack-infra | 08:19 | |
| *** armaan has joined #openstack-infra | 08:19 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** numans has joined #openstack-infra | 08:21 | |
| *** numans_ has quit IRC | 08:22 | |
| *** dgonzalez has joined #openstack-infra | 08:22 | |
| *** ying_zuo has joined #openstack-infra | 08:22 | |
| *** gpocentek has joined #openstack-infra | 08:22 | |
| *** Shrews has joined #openstack-infra | 08:22 | |
| *** inc0 has joined #openstack-infra | 08:22 | |
| *** jlabarre has joined #openstack-infra | 08:22 | |
| *** aviau has joined #openstack-infra | 08:22 | |
| *** VW has quit IRC | 08:23 | |
| *** clayg has joined #openstack-infra | 08:23 | |
| *** sileht has joined #openstack-infra | 08:23 | |
| *** edmondsw has joined #openstack-infra | 08:28 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:30 | |
| *** shu-mutou-AWAY is now known as shu-mutou-PTG | 08:30 | |
| *** berendt has joined #openstack-infra | 08:33 | |
| *** tobberydberg has joined #openstack-infra | 08:33 | |
| *** vdrok has joined #openstack-infra | 08:33 | |
| *** GregHouse has joined #openstack-infra | 08:33 | |
| *** ilpianista has joined #openstack-infra | 08:33 | |
| *** Miouge has joined #openstack-infra | 08:33 | |
| *** AJaeger has joined #openstack-infra | 08:33 | |
| *** zxiiro has joined #openstack-infra | 08:33 | |
| *** electrical has joined #openstack-infra | 08:33 | |
| *** eventingmonkey has joined #openstack-infra | 08:33 | |
| *** bstinson has joined #openstack-infra | 08:33 | |
| *** khappone has joined #openstack-infra | 08:34 | |
| *** danpawlik has joined #openstack-infra | 08:38 | |
| *** slaweq_ has joined #openstack-infra | 08:48 | |
| *** spligak has joined #openstack-infra | 08:50 | |
| *** VW has joined #openstack-infra | 08:50 | |
| *** bhavik1 has joined #openstack-infra | 08:53 | |
| *** slaweq_ has quit IRC | 08:53 | |
| *** udesale_ is now known as udesale | 08:53 | |
| *** VW has quit IRC | 08:55 | |
| *** jpich has joined #openstack-infra | 08:55 | |
| *** slaweq_ has joined #openstack-infra | 08:58 | |
| *** masayukig[m] has joined #openstack-infra | 09:00 | |
| *** liuyulong has quit IRC | 09:01 | |
| *** kiennt26 has joined #openstack-infra | 09:01 | |
| *** yamahata has quit IRC | 09:03 | |
| *** bhavik1 has quit IRC | 09:03 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove newton job from tempest pipeline https://review.openstack.org/547328 | 09:04 |
| *** gfidente has joined #openstack-infra | 09:07 | |
| *** gfidente has joined #openstack-infra | 09:07 | |
| *** rcernin has joined #openstack-infra | 09:09 | |
| *** rossella_s has quit IRC | 09:09 | |
| *** slaweq_ has quit IRC | 09:09 | |
| *** slaweq_ has joined #openstack-infra | 09:10 | |
| *** rossella_s has joined #openstack-infra | 09:13 | |
| *** pcichy has quit IRC | 09:13 | |
| *** slaweq_ has quit IRC | 09:14 | |
| *** lucas-pto is now known as lucasagomes | 09:15 | |
| *** AJaeger has quit IRC | 09:15 | |
| *** AJaeger has joined #openstack-infra | 09:17 | |
| *** yamamoto_ has joined #openstack-infra | 09:21 | |
| *** VW has joined #openstack-infra | 09:22 | |
| *** AJaeger has quit IRC | 09:24 | |
| *** yamamoto has quit IRC | 09:24 | |
| *** AJaeger has joined #openstack-infra | 09:25 | |
| *** VW has quit IRC | 09:26 | |
| *** ykarel is now known as ykarel|afk | 09:28 | |
| *** rubasov has joined #openstack-infra | 09:30 | |
| *** vivsoni has quit IRC | 09:31 | |
| *** ilpianista_ has joined #openstack-infra | 09:31 | |
| *** aspiers[m] has joined #openstack-infra | 09:31 | |
| *** it-praktyk[m] has joined #openstack-infra | 09:32 | |
| *** liuyulong has joined #openstack-infra | 09:35 | |
| *** vivsoni has joined #openstack-infra | 09:39 | |
| *** spligak has quit IRC | 09:40 | |
| *** spligak has joined #openstack-infra | 09:41 | |
| *** efoley has joined #openstack-infra | 09:41 | |
| *** AJaeger has quit IRC | 09:44 | |
| *** mgoddard_ has joined #openstack-infra | 09:44 | |
| *** prometheanfire has quit IRC | 09:45 | |
| *** liuyulong has quit IRC | 09:47 | |
| *** spligak has quit IRC | 09:48 | |
| *** spligak has joined #openstack-infra | 09:49 | |
| *** derekh has joined #openstack-infra | 09:53 | |
| *** ykarel|afk is now known as ykarel | 09:54 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack-infra/puppet-openstackci master: Remove empty file https://review.openstack.org/547343 | 09:55 |
| *** VW has joined #openstack-infra | 09:57 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script https://review.openstack.org/535878 | 09:58 |
| *** rcernin has quit IRC | 09:59 | |
| *** pcichy has joined #openstack-infra | 09:59 | |
| *** masber has quit IRC | 10:01 | |
| *** VW has quit IRC | 10:01 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 10:02 |
| *** bauzas is now known as bauwser | 10:04 | |
| *** AJaeger has joined #openstack-infra | 10:06 | |
| *** liuyulong has joined #openstack-infra | 10:06 | |
| *** kiennt26 has quit IRC | 10:06 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: Adds publish-to-pypi to oswin-tempest-plugin https://review.openstack.org/547348 | 10:06 |
| *** Guest16541 has quit IRC | 10:13 | |
| *** cuongnv has quit IRC | 10:14 | |
| *** ispp has quit IRC | 10:15 | |
| *** olaph1 has joined #openstack-infra | 10:17 | |
| *** olaph has quit IRC | 10:17 | |
| *** pblaho has joined #openstack-infra | 10:17 | |
| AJaeger | jamespage: the post jobs for charm-swarm-proxy and charm-swift-storage fail - looking at http://logs.openstack.org/35/359f541d71e9e4fff38ec29d7d6adb7dff9a8a5e/post/publish-openstack-python-branch-tarball/475f819/job-output.txt.gz#_2018-02-23_09_28_44_016397 I don't think we should run the branch-tarball at all for this. Could you investigate and either fix or remove jobs, please? | 10:18 |
| AJaeger | jamespage: other log: http://logs.openstack.org/9b/9baae6cb89a64bac3c628d5716b1566e56c0a299/post/publish-openstack-python-branch-tarball/00c5613/job-output.txt.gz#_2018-02-23_09_26_14_298761 | 10:18 |
| *** vivsoni has quit IRC | 10:19 | |
| *** vivsoni has joined #openstack-infra | 10:19 | |
| *** pcichy has quit IRC | 10:22 | |
| *** sree_ has joined #openstack-infra | 10:24 | |
| *** sree_ is now known as Guest82581 | 10:25 | |
| jamespage | AJaeger: looking | 10:25 |
| jamespage | not sure where we have that configured tbh | 10:25 |
| *** e0ne has quit IRC | 10:27 | |
| *** stakeda has quit IRC | 10:27 | |
| *** udesale_ has joined #openstack-infra | 10:28 | |
| *** Guest82581 has quit IRC | 10:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove tools/zuul.conf-sample https://review.openstack.org/547361 | 10:29 |
| *** udesale has quit IRC | 10:29 | |
| AJaeger | jamespage: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n3594 and http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n789 | 10:30 |
| jamespage | sorry where/why | 10:30 |
| AJaeger | can't help with that one ;) | 10:30 |
| *** bnemec-pto has quit IRC | 10:32 | |
| jamespage | AJaeger: copy/paste I suspect | 10:33 |
| openstackgerrit | James Page proposed openstack-infra/openstack-zuul-jobs master: Drop tarball publishing for charms https://review.openstack.org/547362 | 10:33 |
| AJaeger | jamespage: thanks, +2 | 10:34 |
| *** janki has quit IRC | 10:35 | |
| *** VW has joined #openstack-infra | 10:36 | |
| *** e0ne has joined #openstack-infra | 10:38 | |
| *** VW has quit IRC | 10:40 | |
| *** namnh has quit IRC | 10:44 | |
| *** e0ne has quit IRC | 10:47 | |
| *** pbourke_ has joined #openstack-infra | 10:53 | |
| *** daidv has quit IRC | 10:58 | |
| *** tosky has quit IRC | 10:58 | |
| *** dbecker has joined #openstack-infra | 11:01 | |
| *** olaph1 has quit IRC | 11:04 | |
| *** olaph has joined #openstack-infra | 11:04 | |
| *** jlabarre has quit IRC | 11:05 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 11:07 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 11:07 |
| *** gouthamr has joined #openstack-infra | 11:08 | |
| openstackgerrit | Javier Peña proposed openstack-infra/system-config master: Move AFS mirror code to puppet-openstackci https://review.openstack.org/529032 | 11:10 |
| *** annp has quit IRC | 11:11 | |
| *** VW has joined #openstack-infra | 11:11 | |
| *** udesale__ has joined #openstack-infra | 11:15 | |
| *** VW has quit IRC | 11:16 | |
| *** udesale_ has quit IRC | 11:18 | |
| *** armaan has quit IRC | 11:26 | |
| *** armaan has joined #openstack-infra | 11:27 | |
| *** janki has joined #openstack-infra | 11:28 | |
| *** rossella_s has quit IRC | 11:29 | |
| *** shoogz has joined #openstack-infra | 11:33 | |
| *** rossella_s has joined #openstack-infra | 11:33 | |
| *** shoogz has joined #openstack-infra | 11:33 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** shoogz has joined #openstack-infra | 11:34 | |
| *** shoogz has joined #openstack-infra | 11:34 | |
| *** pcichy has joined #openstack-infra | 11:34 | |
| *** shoogz has joined #openstack-infra | 11:35 | |
| *** shoogz has joined #openstack-infra | 11:35 | |
| *** udesale__ has quit IRC | 11:36 | |
| *** shoogz has joined #openstack-infra | 11:36 | |
| *** shoogz has joined #openstack-infra | 11:39 | |
| *** dtantsur|afk is now known as dtantsur | 11:40 | |
| *** rossella_s has quit IRC | 11:40 | |
| *** shoogz has joined #openstack-infra | 11:41 | |
| *** liuyulong has quit IRC | 11:41 | |
| *** shoogz has joined #openstack-infra | 11:42 | |
| *** links has quit IRC | 11:42 | |
| *** shoogz has joined #openstack-infra | 11:43 | |
| *** rossella_s has joined #openstack-infra | 11:43 | |
| *** tosky has joined #openstack-infra | 11:44 | |
| *** VW has joined #openstack-infra | 11:49 | |
| *** olaph1 has joined #openstack-infra | 11:51 | |
| *** d0ugal has quit IRC | 11:52 | |
| *** olaph has quit IRC | 11:53 | |
| *** VW has quit IRC | 11:54 | |
| *** links has joined #openstack-infra | 11:56 | |
| *** d0ugal has joined #openstack-infra | 11:58 | |
| *** kjackal has quit IRC | 12:02 | |
| *** kjackal has joined #openstack-infra | 12:02 | |
| *** rossella_s has quit IRC | 12:04 | |
| *** rossella_s has joined #openstack-infra | 12:05 | |
| andreaf | AJaeger tobiash corvus mordred ianw this change is handy to rename log files staged as folders https://review.openstack.org/#/c/544606/ | 12:07 |
| andreaf | Specifically it helps with being able to see service config files from the browser :) | 12:08 |
| *** edmondsw has quit IRC | 12:18 | |
| *** pcichy has quit IRC | 12:18 | |
| *** zoli is now known as zoli|lunch | 12:19 | |
| *** zoli|lunch is now known as zoli | 12:19 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Add publish-to-pypi job for monasca-tempest-plugin https://review.openstack.org/547406 | 12:19 |
| *** ramishra has quit IRC | 12:21 | |
| *** ldnunes has quit IRC | 12:23 | |
| *** lucasagomes is now known as lucas-hungry | 12:24 | |
| *** VW has joined #openstack-infra | 12:25 | |
| *** VW has quit IRC | 12:30 | |
| *** links has quit IRC | 12:30 | |
| *** ldnunes has joined #openstack-infra | 12:37 | |
| *** olaph1 has quit IRC | 12:38 | |
| *** masber has joined #openstack-infra | 12:38 | |
| *** olaph has joined #openstack-infra | 12:39 | |
| *** tosky has quit IRC | 12:42 | |
| *** links has joined #openstack-infra | 12:44 | |
| *** dprince has joined #openstack-infra | 12:45 | |
| *** eernst has joined #openstack-infra | 12:46 | |
| *** cshastri has quit IRC | 12:46 | |
| *** zoli is now known as zoli|wfh | 12:49 | |
| *** zoli|wfh is now known as zoli | 12:49 | |
| *** efoley has quit IRC | 12:52 | |
| *** jpena is now known as jpena|lunch | 12:55 | |
| *** rossella_s has quit IRC | 13:00 | |
| *** efoley has joined #openstack-infra | 13:01 | |
| *** eernst has quit IRC | 13:02 | |
| *** rossella_s has joined #openstack-infra | 13:02 | |
| *** VW has joined #openstack-infra | 13:02 | |
| *** gouthamr has quit IRC | 13:04 | |
| *** gouthamr has joined #openstack-infra | 13:04 | |
| *** tosky has joined #openstack-infra | 13:05 | |
| *** VW has quit IRC | 13:07 | |
| *** rossella_s has quit IRC | 13:09 | |
| *** edmondsw has joined #openstack-infra | 13:10 | |
| *** adarazs is now known as adarazs_brb | 13:11 | |
| *** rossella_s has joined #openstack-infra | 13:12 | |
| *** jcoufal has joined #openstack-infra | 13:13 | |
| *** adarazs_brb is now known as adarazs | 13:14 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Improve file renaming in stage-output https://review.openstack.org/544606 | 13:17 |
| *** armaan has quit IRC | 13:20 | |
| *** armaan has joined #openstack-infra | 13:21 | |
| *** lucas-hungry is now known as lucasagomes | 13:21 | |
| *** VW has joined #openstack-infra | 13:22 | |
| *** udesale has joined #openstack-infra | 13:22 | |
| *** jlabarre has joined #openstack-infra | 13:27 | |
| *** rlandy has joined #openstack-infra | 13:31 | |
| *** rlandy is now known as rlandy|ruck | 13:35 | |
| *** camunoz has joined #openstack-infra | 13:36 | |
| *** pgadiya has quit IRC | 13:36 | |
| *** Goneri has joined #openstack-infra | 13:42 | |
| *** ykarel is now known as ykarel|afk | 13:44 | |
| *** pbourke_ has quit IRC | 13:45 | |
| *** gouthamr has quit IRC | 13:50 | |
| *** ykarel|afk has quit IRC | 13:52 | |
| *** armaan has quit IRC | 13:53 | |
| *** armaan has joined #openstack-infra | 13:53 | |
| *** jpena|lunch is now known as jpena | 13:54 | |
| *** links has quit IRC | 13:56 | |
| *** kgiusti has joined #openstack-infra | 14:02 | |
| *** psachin has quit IRC | 14:04 | |
| *** dklyle has quit IRC | 14:04 | |
| *** imacdonn has quit IRC | 14:06 | |
| *** imacdonn has joined #openstack-infra | 14:07 | |
| *** eharney has joined #openstack-infra | 14:08 | |
| *** dhajare has quit IRC | 14:13 | |
| *** pcichy has joined #openstack-infra | 14:14 | |
| *** baoli has joined #openstack-infra | 14:14 | |
| *** dhill_ has joined #openstack-infra | 14:17 | |
| *** Guest60045 is now known as zeus | 14:18 | |
| *** zeus has joined #openstack-infra | 14:18 | |
| *** esberglu has joined #openstack-infra | 14:21 | |
| *** efoley has quit IRC | 14:21 | |
| *** kashyap has joined #openstack-infra | 14:23 | |
| kashyap | clarkb: Hi, are Monday Infra helproom slots available? https://ethercalc.openstack.org/cvro305izog2 | 14:23 |
| kashyap | I just one topic that takes about 15 minutes or so | 14:23 |
| clarkb | kashyap: yes unless someone granbed them all, Monday should still have free slots go ahead and take one | 14:27 |
| kashyap | clarkb: Yep, I just entered one for 14:30 | 14:27 |
| *** rossella_s has quit IRC | 14:28 | |
| *** udesale_ has joined #openstack-infra | 14:28 | |
| *** udesale has quit IRC | 14:29 | |
| *** rossella_s has joined #openstack-infra | 14:29 | |
| kashyap | Thanks; that ethercalc idea is very convenient. /me was about to write an email | 14:29 |
| kashyap | But I recalled your email to the list | 14:29 |
| *** eernst has joined #openstack-infra | 14:30 | |
| AJaeger | config-core, can we avoid dpulicate job runs on sahara - please review https://review.openstack.org/546695 | 14:33 |
| *** rossella_s has quit IRC | 14:37 | |
| *** lbragstad has joined #openstack-infra | 14:37 | |
| *** rossella_s has joined #openstack-infra | 14:37 | |
| *** yamahata has joined #openstack-infra | 14:41 | |
| *** amoralej is now known as amoralej|lunch | 14:42 | |
| *** armaan has quit IRC | 14:44 | |
| *** armaan has joined #openstack-infra | 14:45 | |
| *** efoley has joined #openstack-infra | 14:48 | |
| *** r-daneel has joined #openstack-infra | 14:54 | |
| *** ldnunes has quit IRC | 14:57 | |
| *** ldnunes has joined #openstack-infra | 14:58 | |
| *** VW has quit IRC | 14:58 | |
| *** VW has joined #openstack-infra | 14:59 | |
| *** mriedem has joined #openstack-infra | 14:59 | |
| *** edmondsw has quit IRC | 15:00 | |
| *** edmondsw has joined #openstack-infra | 15:00 | |
| *** rossella_s has quit IRC | 15:00 | |
| * fungi is increasingly glad we decided to try hosting an ethercalc service | 15:02 | |
| pabelanger | ianw: great work! | 15:03 |
| *** olaph1 has joined #openstack-infra | 15:03 | |
| *** zhipeng has joined #openstack-infra | 15:03 | |
| *** e0ne has joined #openstack-infra | 15:03 | |
| mordred | fungi: ++ | 15:03 |
| *** rossella_s has joined #openstack-infra | 15:03 | |
| *** panda is now known as panda|sick | 15:04 | |
| *** edmondsw has quit IRC | 15:05 | |
| *** masber has quit IRC | 15:05 | |
| *** olaph has quit IRC | 15:05 | |
| AJaeger | config-core, two changes to add abstract attributes to our base jobs are https://review.openstack.org/545603 and https://review.openstack.org/545442 - reviews welcome, please | 15:08 |
| mordred | infra-root: I put an autohold in place on neutron-tempest-plugin-scenario-linuxbridge for slaweq to debug a hard-to-reproduce issue that's happening | 15:08 |
| mordred | infra-root: nevermind. I've removed it - miscommunication on timing | 15:09 |
| mordred | Shrews: heh. I have now discovered that there is no autohold-remove command :) | 15:10 |
| slaweq | thx mordred, but can we do it on monday? | 15:10 |
| mordred | slaweq: yup. no problem | 15:10 |
| corvus | mordred: set the count to 0 | 15:11 |
| tosky | frickler: uh, re support for Ocata https://review.openstack.org/#/c/546695/ | 15:12 |
| mordred | corvus: just repeat the same autohold command with 0 ? | 15:12 |
| mordred | cool | 15:12 |
| *** rossella_s has quit IRC | 15:12 | |
| tosky | frickler: how likely is going to happen, and how much time? | 15:12 |
| dmsimard | infra-root: Asking out of curiosity: We started getting a SSH crytpo exchange issue yesterday with review.rdoproject.org's gerrit yesterday, stack trace looks like this: http://paste.openstack.org/raw/682961/ | 15:13 |
| mordred | corvus: heh. count=0 still requires a reason argument | 15:13 |
| dmsimard | We're not quite up to date on gerrit yet and at a quick glance review.openstack.org doesn't have any replication errors right now.. does that issue ring a bell to anyone ? | 15:13 |
| frickler | tosky: not sure how likely, been trying without success to find the reference I remember having seen. I think they were talking about a couple of months possibly | 15:13 |
| pabelanger | dmsimard: github removed weak SSH keys recently | 15:14 |
| dmsimard | pabelanger: source ? | 15:14 |
| pabelanger | dmsimard: guess you might be still using them? | 15:14 |
| *** pblaho has quit IRC | 15:14 | |
| pabelanger | let me fine | 15:14 |
| pabelanger | https://github.com/blog/2507-weak-cryptographic-standards-removed | 15:14 |
| corvus | mordred: i think there's a patch to add a remove command too :) | 15:14 |
| dmsimard | I could try and see if we can force specific ciphers through .ssh/config | 15:14 |
| *** rossella_s has joined #openstack-infra | 15:15 | |
| mordred | corvus: neat | 15:15 |
| dmsimard | pabelanger: thanks for that, I haven't seen it in my feed | 15:15 |
| tosky | frickler: and I guess that knowning who are "they" depends also on recalling where the reference is? | 15:15 |
| pabelanger | dmsimard: https://githubengineering.com/crypto-removal-notice/ | 15:15 |
| frickler | tosky: yeah, my memory pointed to dev-ml, but no hit in my searching there | 15:16 |
| pabelanger | are you running centos <7.2? | 15:16 |
| pabelanger | dmsimard: if so, you likely need to update git client | 15:16 |
| dmsimard | pabelanger: no, 7.4 | 15:16 |
| dmsimard | pabelanger: the java lib that does the replication might have an older implementation though, I dunno | 15:16 |
| dmsimard | pabelanger: there's a brief mention of it here: https://github.com/is/jsch/issues/2 | 15:17 |
| pabelanger | I don't think our gerrit is affected, but can check quickly | 15:18 |
| dmsimard | pabelanger: it's not, I looked | 15:18 |
| pabelanger | dmsimard: k, thanks | 15:18 |
| fungi | dmsimard: what gerrit version are you on? | 15:18 |
| *** jcoufal_ has joined #openstack-infra | 15:19 | |
| dmsimard | fungi: 2.11.10 :( | 15:19 |
| fungi | dmsimard: i also just remembered i could pull up the webui since it's public | 15:19 |
| *** jcoufal has quit IRC | 15:20 | |
| fungi | but yeah, perhaps the implementation in 2.11 is running afoul of that | 15:20 |
| *** jcoufal has joined #openstack-infra | 15:21 | |
| *** david-lyle has joined #openstack-infra | 15:21 | |
| *** armaan has quit IRC | 15:21 | |
| *** ldnunes has quit IRC | 15:21 | |
| *** armaan has joined #openstack-infra | 15:22 | |
| *** dave-mccowan has joined #openstack-infra | 15:23 | |
| *** prometheanfire has joined #openstack-infra | 15:23 | |
| *** jcoufal_ has quit IRC | 15:24 | |
| *** kashyap has left #openstack-infra | 15:24 | |
| *** rossella_s has quit IRC | 15:25 | |
| *** rossella_s has joined #openstack-infra | 15:27 | |
| *** dave-mccowan has quit IRC | 15:27 | |
| frickler | tosky: o.k., still nothing found. so if you prefer, I can +3 your patch now and we can revert or do some special stuff for ocata if you later decide to still need to support it | 15:28 |
| *** amoralej|lunch is now known as amoralej | 15:28 | |
| *** dave-mccowan has joined #openstack-infra | 15:29 | |
| tosky | frickler: or alternatively, who could know about this? TC? Release team? | 15:33 |
| tosky | or better, the question is "who can I nag?" - which is not the question which has a lot of possibility for a successfull answer on Friday before the PTG... | 15:35 |
| frickler | tosky: yeah, not sure, and its EOD now for me. so I'll just do a +2 now and you can ask someone else to just +W once you decide | 15:37 |
| tosky | ack | 15:37 |
| clarkb | dmsimard: jgit has a command line client that may help you debug better | 15:37 |
| clarkb | dmsimard: basically check if jgit cli fails too and might give more verbose errors. My hunch is it does the sha1 diffie hellman key exchange | 15:38 |
| *** jpena is now known as jpena|brb | 15:39 | |
| dmsimard | clarkb: we ended up updating the jsch lib in-place and that resolved it for us | 15:39 |
| *** witek has quit IRC | 15:40 | |
| *** olaph has joined #openstack-infra | 15:47 | |
| *** olaph1 has quit IRC | 15:47 | |
| *** ykarel|afk has joined #openstack-infra | 15:49 | |
| *** ldnunes has joined #openstack-infra | 15:50 | |
| *** dklyle has joined #openstack-infra | 15:50 | |
| *** jcoufal has quit IRC | 15:53 | |
| *** david-lyle has quit IRC | 15:53 | |
| *** jcoufal has joined #openstack-infra | 15:54 | |
| *** slaweq has quit IRC | 15:54 | |
| *** ykarel|afk has quit IRC | 15:54 | |
| *** slaweq has joined #openstack-infra | 15:55 | |
| *** VW has quit IRC | 15:57 | |
| *** slaweq_ has joined #openstack-infra | 15:57 | |
| *** slaweq_ has quit IRC | 15:57 | |
| pabelanger | dirk: AJaeger: clarkb: could use some help, which repo does tumbleweed pull from for updates? | 15:58 |
| *** slaweq_ has joined #openstack-infra | 15:58 | |
| pabelanger | I am guessing http://download.opensuse.org/tumbleweed/repo/oss/ ? | 15:58 |
| dirk | pabelanger: yes | 15:58 |
| dirk | pabelanger: there is no separate update channel | 15:58 |
| dirk | if you need one, there is a fake one that is empty | 15:58 |
| dirk | pabelanger: sorry I didn't have time to look at that yet, maybe over the weekend | 15:58 |
| pabelanger | okay, cool. Let me propose some patches to our mirror scripts and you can review | 15:59 |
| clarkb | http://download.opensuse.org/update/tumbleweed/ is my update channel | 15:59 |
| *** slaweq_ has quit IRC | 15:59 | |
| *** slaweq_ has joined #openstack-infra | 16:00 | |
| *** slaweq_ has quit IRC | 16:00 | |
| *** slaweq_ has joined #openstack-infra | 16:00 | |
| *** jcoufal_ has joined #openstack-infra | 16:01 | |
| pabelanger | clarkb: that doesn't seem to exist in the root directory: http://download.opensuse.org/update/ | 16:02 |
| clarkb | pabelanger: looks like its a redirect | 16:03 |
| pabelanger | yah | 16:03 |
| clarkb | to http://download.opensuse.org/repositories/openSUSE:/Factory:/Update/standard/ | 16:03 |
| *** jcoufal has quit IRC | 16:03 | |
| *** jaosorior has quit IRC | 16:05 | |
| *** slaweq_ has quit IRC | 16:05 | |
| *** jcoufal has joined #openstack-infra | 16:07 | |
| *** jcoufal_ has quit IRC | 16:08 | |
| AJaeger | any config-core to review stable/queens notificatoins for gerritbot using the new regex? https://review.openstack.org/546686 and https://review.openstack.org/546432 and https://review.openstack.org/546682 | 16:10 |
| *** mgoddard_ has quit IRC | 16:11 | |
| *** armaan has quit IRC | 16:12 | |
| corvus | i'm heading into the metal tubes | 16:12 |
| pabelanger | good luck! | 16:13 |
| *** armaan has joined #openstack-infra | 16:13 | |
| pabelanger | I'll be heading out in a few hours myself | 16:13 |
| clarkb | I need to start packing, I head out early tomorrow morning | 16:13 |
| clarkb | AJaeger: looking at https://review.openstack.org/#/c/545603/2/zuul.yaml unittest and multinode clearly need to be extended by child jobs but nodejs-npm does have a run playbook, couldn't it run standalone? | 16:16 |
| clarkb | AJaeger: same with openstack-infra-multinode-integration in https://review.openstack.org/#/c/545442/1/zuul.d/jobs.yaml should that be abstract too? it has a run playbook? | 16:17 |
| AJaeger | clarkb: indeed the npm one could - we just don't do it... | 16:18 |
| *** yamamoto_ has quit IRC | 16:18 | |
| AJaeger | clarkb: openstack-infra-multinode-integration is our own integration test - we do not want anybody else to run those. So, I would leave it that way. | 16:19 |
| clarkb | AJaeger: protected is fine but should it be abstract? | 16:19 |
| *** jcoufal_ has joined #openstack-infra | 16:20 | |
| AJaeger | clarkb: infra is the only user - and we tread it as base job that we don't run right now. | 16:20 |
| *** jcoufal has quit IRC | 16:20 | |
| AJaeger | clarkb: In this case, i would leave as is - and if needed change later (depending on use case) | 16:20 |
| AJaeger | but if you feel strongly, I can remove now as well... | 16:20 |
| * AJaeger needs to catch his son, will be back soonish | 16:21 | |
| *** danpawlik has quit IRC | 16:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add stable branch notification for openstack-cinder https://review.openstack.org/546686 | 16:22 |
| *** yamamoto has joined #openstack-infra | 16:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update bot for puppet-openstack to include Queens https://review.openstack.org/546432 | 16:22 |
| clarkb | I don't feel super strongly about it, it just feels slighyly weird to have runnable abstract jobs. I've +2'd that change. The change to zuul-jobs should probably be updated though as its more public facing | 16:22 |
| corvus | you can't run an abstract job... | 16:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Regex gerritbot notifications: #fuel-tracker https://review.openstack.org/546682 | 16:23 |
| *** jpena|brb is now known as jpena | 16:23 | |
| clarkb | corvus: yes, so in this case the job I'm asking about are potentially runnable on their own | 16:23 |
| pabelanger | clarkb: have time to look at glean patch for systemd? https://review.openstack.org/485748/ | 16:24 |
| clarkb | nodejs-npm and the openstack infra multinode integration. I think we need to be more careful with nodejs-npm since people could be using it at this point (and reading it it may be useful without inheriting from it) | 16:24 |
| *** zhipeng has quit IRC | 16:24 | |
| corvus | clarkb: well, abstract is a great way to prevent a potentially runnable job from being run. but yeah, if we think it's *okay* for them to be run on their own, abstract may be too much. we don't want to make it useless. we should probably use it sparingly. | 16:25 |
| corvus | i don't have time to dig in further. hth. :) | 16:25 |
| pabelanger | infra-root: any objection of tagging a new release of grafyaml today? Include new functionality needed by user looking at presenting at grafanacon next week | 16:25 |
| clarkb | pabelanger: any idea why we don't seem to have a problem with glean on xenial (re the glean patch)? | 16:26 |
| *** e0ne has quit IRC | 16:26 | |
| clarkb | I guess its detecting a loop but working anyways? | 16:26 |
| *** rossella_s has quit IRC | 16:26 | |
| *** numans has quit IRC | 16:27 | |
| clarkb | re glean I don't really want to make changes to that just before we all hop on airplanes (we won't consume it until we push a tag but seems like maybe its better to pick this up on monday?) | 16:28 |
| *** bauwser is now known as bauzas | 16:28 | |
| *** rossella_s has joined #openstack-infra | 16:28 | |
| pabelanger | clarkb: yah, loop is detected but works. However, debian-jessie doesn't. I am guessing because of a systemd difference between debian / ububntu | 16:31 |
| pabelanger | clarkb: yah, monday works. Just wanted to get some eyes on it | 16:31 |
| clarkb | pabelanger: does the nodepool + glean integration test fail without that patch on debian? | 16:31 |
| clarkb | looks like it passes with that patch so that would inidicate fixing of something | 16:31 |
| clarkb | fungi: my terminal's font doesn't have a glyph for stein | 16:32 |
| clarkb | :) | 16:32 |
| clarkb | er :( | 16:32 |
| *** numans has joined #openstack-infra | 16:32 | |
| *** openstackgerrit has quit IRC | 16:33 | |
| pabelanger | clarkb: yah, debian-jessie is currently failing in nodepool integration testing | 16:35 |
| *** claudiub|2 has joined #openstack-infra | 16:35 | |
| clarkb | pabelanger: ok maybe we can pick this up as a quiet time monday morning task so that we can have eyeballs on glean when we fix it | 16:35 |
| pabelanger | wgm | 16:36 |
| pabelanger | wfm* | 16:36 |
| *** openstackgerrit has joined #openstack-infra | 16:38 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script https://review.openstack.org/535878 | 16:38 |
| *** claudiub has quit IRC | 16:38 | |
| *** rossella_s has quit IRC | 16:40 | |
| *** rossella_s has joined #openstack-infra | 16:43 | |
| *** agopi has joined #openstack-infra | 16:48 | |
| *** slaweq_ has joined #openstack-infra | 16:48 | |
| *** udesale__ has joined #openstack-infra | 16:52 | |
| *** udesale_ has quit IRC | 16:52 | |
| *** caphrim007 has joined #openstack-infra | 16:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Add abstract attribute to base jobs https://review.openstack.org/545603 | 16:54 |
| AJaeger | clarkb: removed the abstract from nodejs-npm ^ | 16:54 |
| *** slaweq_ has quit IRC | 16:56 | |
| *** yamamoto has quit IRC | 16:57 | |
| dirk | clarkb: pabelanger : exactly - its a fake update channel, normally it is totally empty (only used to unbreak the update stack in case of a real fsckup) | 16:58 |
| dirk | we don't need to mirror it or embed it in the image | 16:58 |
| clarkb | dirk: got it thanks | 17:00 |
| *** yamamoto_ has joined #openstack-infra | 17:01 | |
| *** hashar is now known as hasharAway | 17:03 | |
| pabelanger | ack | 17:04 |
| fungi | clarkb: neither does my terminal, but my web browser seems to | 17:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove max-ready-age from nodepool https://review.openstack.org/547520 | 17:09 |
| *** lucasagomes is now known as lucas-afk | 17:10 | |
| *** gyee has joined #openstack-infra | 17:10 | |
| *** yamamoto_ has quit IRC | 17:12 | |
| *** efried has joined #openstack-infra | 17:13 | |
| *** jpena is now known as jpena|off | 17:15 | |
| *** dave-mccowan has quit IRC | 17:15 | |
| *** dave-mccowan has joined #openstack-infra | 17:16 | |
| *** rossella_s has quit IRC | 17:17 | |
| *** dklyle has quit IRC | 17:17 | |
| *** wolverineav has joined #openstack-infra | 17:17 | |
| *** edmondsw has joined #openstack-infra | 17:18 | |
| *** gmmaha has left #openstack-infra | 17:18 | |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| *** shardy is now known as shardy_afk | 17:20 | |
| *** sree_ has joined #openstack-infra | 17:21 | |
| *** sree_ is now known as Guest93643 | 17:22 | |
| Shrews | pabelanger: the commit message in 547520 isn't quite correct. we added max-ready-age to avoid the provider wedge scenario | 17:22 |
| Shrews | pabelanger: fixed by https://review.openstack.org/539316 | 17:23 |
| *** Guest93643 has quit IRC | 17:25 | |
| pabelanger | Shrews: oh, right. let me update commit and reference 539316 | 17:26 |
| Shrews | not sure what "min-ready logic issue" you're refering to | 17:27 |
| pabelanger | yah | 17:27 |
| *** yamamoto has joined #openstack-infra | 17:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove max-ready-age from nodepool https://review.openstack.org/547520 | 17:28 |
| pabelanger | Shrews: hows that?^ | 17:29 |
| Shrews | lgtm | 17:29 |
| pabelanger | thanks | 17:29 |
| *** slaweq_ has joined #openstack-infra | 17:30 | |
| *** rossella_s has quit IRC | 17:30 | |
| *** hongbin has joined #openstack-infra | 17:32 | |
| *** yamamoto has quit IRC | 17:32 | |
| *** jpich has quit IRC | 17:33 | |
| AJaeger | infra-root, we're seeing timeouts syncing to AFS, e.g. http://logs.openstack.org/b7/b7075038fa51935937353602c33a1da62f81fd06/post/publish-openstack-manuals/243ae93/job-output.txt.gz#_2018-02-22_17_28_24_882617 | 17:33 |
| AJaeger | also, http://logs.openstack.org/8c/8c71dc5946e1d930c55a157877f9eae706586fbb/post/publish-openstack-manuals/40d2b20/job-output.txt.gz#_2018-02-20_16_05_05_516743 | 17:34 |
| *** rossella_s has joined #openstack-infra | 17:34 | |
| clarkb | AJaeger: do you know if it needs more than half an hour nirmally or is this abnormal? | 17:34 |
| clarkb | if it just needs more time we can increase the post-timeout on those jobs | 17:34 |
| *** gfidente has quit IRC | 17:34 | |
| clarkb | project-config has post-timeout examples on base jobs and it is dovumented too | 17:35 |
| AJaeger | 21 mins here http://logs.openstack.org/3e/3edc5322ab23421a00197178beeecac97813c0f5/post/publish-openstack-manuals/af6557e/job-output.txt.gz#_2018-02-23_17_06_57_987510 | 17:35 |
| clarkb | if that is abnormal we probably have to debug the afs rw volume | 17:35 |
| *** efoley has quit IRC | 17:35 | |
| AJaeger | so, yes, we should probably increase timeout... | 17:35 |
| *** amoralej is now known as amoralej|off | 17:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Increase openstack-manuals publish timeout https://review.openstack.org/547525 | 17:38 |
| *** vaidy has quit IRC | 17:40 | |
| *** isviridov_away has quit IRC | 17:40 | |
| *** yamamoto has joined #openstack-infra | 17:42 | |
| *** yamamoto has quit IRC | 17:42 | |
| pabelanger | AJaeger: clarkb: I don't see any issue with afs losing access to cell. Perhaps a bandwidth issue? | 17:43 |
| clarkb | ya could be cross atlantic bw issue? | 17:44 |
| clarkb | I think bumping timeout is good first step | 17:44 |
| *** danpawlik has joined #openstack-infra | 17:45 | |
| *** d0ugal has quit IRC | 17:46 | |
| *** yamahata has quit IRC | 17:47 | |
| *** udesale_ has joined #openstack-infra | 17:47 | |
| *** lpetrut has joined #openstack-infra | 17:48 | |
| *** udesale__ has quit IRC | 17:50 | |
| *** danpawlik has quit IRC | 17:51 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Protect some jobs https://review.openstack.org/545442 | 17:53 |
| *** d0ugal has joined #openstack-infra | 17:54 | |
| *** zoli is now known as zoli|gone | 17:55 | |
| *** zoli|gone is now known as zoli | 17:55 | |
| *** efried is now known as fried_rice_jfk | 17:56 | |
| *** dtantsur is now known as dtantsur|afk | 17:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add abstract attribute to base jobs https://review.openstack.org/545603 | 18:00 |
| *** hongbin has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:03 | |
| pabelanger | some excitement over in npm land: https://github.com/npm/npm/issues/19883 | 18:05 |
| pabelanger | I guess sudo npm borked file system permissions some how | 18:05 |
| clarkb | ya I was reading that this morning | 18:05 |
| clarkb | pabelanger: they made a prerelease that npm would upgrade itself too because it looked like a regular release | 18:05 |
| clarkb | and being a prerelease it hadn't been properly tested or something | 18:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Increase openstack-manuals publish timeout https://review.openstack.org/547525 | 18:06 |
| clarkb | then a bunch of people ended up upgrading to it automatically when telling npm to update | 18:06 |
| *** rm_work has quit IRC | 18:06 | |
| pabelanger | ouch | 18:06 |
| clarkb | they've since pushed a new release to supercede that one that doesn't have the bug | 18:06 |
| *** VW has joined #openstack-infra | 18:06 | |
| clarkb | there is a lot of talk about whether or not they are doing semver properly :) | 18:07 |
| *** dansmith has quit IRC | 18:07 | |
| *** SpamapS has quit IRC | 18:07 | |
| pabelanger | yah, can see that | 18:07 |
| *** bradm has quit IRC | 18:07 | |
| *** zeus has quit IRC | 18:07 | |
| *** StevenK has quit IRC | 18:07 | |
| clarkb | for those of us that don't live in canada heads up that long term dublin forecast shows snow for some of next week | 18:08 |
| clarkb | pack warm | 18:08 |
| clarkb | for the canadians you'll probably be fine in tshirts and shorts because its warm snow :) | 18:08 |
| pabelanger | :) | 18:08 |
| *** bhavik1 has joined #openstack-infra | 18:14 | |
| fungi | i'm good with warm snow, but christine is insisting that i pack long pants and bring a jacket | 18:15 |
| * fungi makes "whatever" emoticon | 18:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 18:15 |
| *** bhavik1 has quit IRC | 18:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Inject url endpoint information https://review.openstack.org/538130 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: angular controller for status and stream page https://review.openstack.org/545755 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorgnize the output bundle split https://review.openstack.org/545706 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 18:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make Info.endpoint a config override https://review.openstack.org/547576 | 18:15 |
| * mordred replies to the theory of 'warm snow' with a giant stack of javascript patches | 18:15 | |
| clarkb | by warm I mean weahter is like -1 to 1 C | 18:17 |
| *** salv-orl_ has quit IRC | 18:18 | |
| *** salv-orlando has joined #openstack-infra | 18:18 | |
| *** tesseract has quit IRC | 18:19 | |
| mordred | clarkb: you have a different definition of the word warm than I do | 18:19 |
| clarkb | mordred: it was ~ -7C here last night | 18:20 |
| *** camunoz has quit IRC | 18:20 | |
| clarkb | but ya its not warm to me but I expect if you live in canada near a lake that is balmy in the winter | 18:20 |
| *** xarses has joined #openstack-infra | 18:20 | |
| *** yamahata has joined #openstack-infra | 18:20 | |
| mordred | clarkb: this is a very good point | 18:21 |
| *** salv-orlando has quit IRC | 18:22 | |
| *** bnemec has joined #openstack-infra | 18:23 | |
| *** olaph1 has joined #openstack-infra | 18:24 | |
| *** olaph has quit IRC | 18:24 | |
| *** r-daneel has quit IRC | 18:24 | |
| *** dbecker has quit IRC | 18:28 | |
| *** bnemec is now known as bnemec-pto | 18:30 | |
| *** dprince has quit IRC | 18:30 | |
| *** lbragstad has quit IRC | 18:31 | |
| *** esberglu has quit IRC | 18:31 | |
| ssbarnea | https://review.openstack.org/#/c/205637/ -- any chance of getting this python-jenkins change in? already got a +2. | 18:33 |
| *** pcaruana has quit IRC | 18:33 | |
| *** dansmith has joined #openstack-infra | 18:33 | |
| *** dansmith is now known as Guest17026 | 18:33 | |
| *** danpawlik has joined #openstack-infra | 18:33 | |
| mnaser | hi infra | 18:35 |
| *** olaph has joined #openstack-infra | 18:35 | |
| mnaser | do we have something to make sure we have enough entropy in the workers | 18:35 |
| *** zeus has joined #openstack-infra | 18:35 | |
| mnaser | i am running into an issue where services take a really long time to start up and some googling got me here - https://www.linuxquestions.org/questions/slackware-14/libvirt-libvirtd-often-takes-~45-seconds-to-start-4175598850/ | 18:36 |
| *** udesale_ has quit IRC | 18:36 | |
| *** zeus is now known as Guest89134 | 18:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Mirror tumbleweed for opensuse https://review.openstack.org/547646 | 18:37 |
| *** rajinir_ has joined #openstack-infra | 18:37 | |
| *** jcoufal has joined #openstack-infra | 18:37 | |
| *** berendt_ has joined #openstack-infra | 18:37 | |
| *** seongsoocho_ has joined #openstack-infra | 18:37 | |
| *** xyang_ has joined #openstack-infra | 18:37 | |
| *** robcresswell_ has joined #openstack-infra | 18:37 | |
| *** Dmitrii-Sh_ has joined #openstack-infra | 18:37 | |
| *** vdrok_ has joined #openstack-infra | 18:37 | |
| *** neith_ has joined #openstack-infra | 18:37 | |
| *** hogepodge_ has joined #openstack-infra | 18:37 | |
| *** olaph1 has quit IRC | 18:37 | |
| *** mwhahaha_ has joined #openstack-infra | 18:37 | |
| pabelanger | mordred: dirk: AJaeger: clarkb: ^should logic to start mirroring tumbleweed | 18:37 |
| *** coreycb_ has joined #openstack-infra | 18:37 | |
| *** dprince has joined #openstack-infra | 18:37 | |
| *** khappone_ has joined #openstack-infra | 18:37 | |
| *** electrical_ has joined #openstack-infra | 18:37 | |
| mnaser | ok i see the 'initialize-urandom' element being used | 18:38 |
| pabelanger | mnaser: we have a pump that corvus wrote.... yah that | 18:38 |
| pabelanger | should only apply to xenial | 18:38 |
| *** gus_ has joined #openstack-infra | 18:38 | |
| *** wendar_ has joined #openstack-infra | 18:38 | |
| mnaser | i wonder if a recent change may have caused it not to work lately or something (just a theory to why this happens) | 18:38 |
| *** jcoufal_ has quit IRC | 18:39 | |
| pabelanger | mnaser: in our case, it was preventing SSH key from generating fast enough. | 18:39 |
| pabelanger | where are you seeing the issue? | 18:39 |
| *** cmurphy_ has joined #openstack-infra | 18:40 | |
| mnaser | pabelanger: right now libvirt takes almost ~45s to start up in stable/pike gate (seems to have started recently) | 18:40 |
| mnaser | this means some operations which wait for the service to go up (with a 60s timeout) actually end up timing out and the job fails | 18:40 |
| *** shardy_a1k has joined #openstack-infra | 18:40 | |
| pabelanger | mnaser: for devstack? | 18:41 |
| *** rossella_s has quit IRC | 18:41 | |
| mnaser | pabelanger: yes | 18:41 |
| AJaeger | pabelanger: thanks! | 18:41 |
| mnaser | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22No%20host-to-cell%20mapping%20found%20for%20selected%20host%5C%22%20AND%20tags:%5C%22screen-n-cond.txt%5C%22%20AND%20voting:1&from=7d | 18:41 |
| mnaser | it does look that libvirt uses /dev/random instead of /dev/urandom | 18:42 |
| pabelanger | mnaser: is there a way to change it to use /dev/urandom? | 18:43 |
| *** therve_ has joined #openstack-infra | 18:43 | |
| *** andreykurilin_ has joined #openstack-infra | 18:43 | |
| *** yamamoto has joined #openstack-infra | 18:43 | |
| *** jlabarre has quit IRC | 18:43 | |
| *** rossella_s has joined #openstack-infra | 18:43 | |
| mnaser | pabelanger: checking, not sure | 18:43 |
| *** jamielennox_ has joined #openstack-infra | 18:43 | |
| *** ruhe2 has joined #openstack-infra | 18:43 | |
| pabelanger | that will already be primed | 18:43 |
| *** logan_ has joined #openstack-infra | 18:43 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 18:44 | |
| *** johnthetubaguy_ has joined #openstack-infra | 18:44 | |
| *** xarses has quit IRC | 18:44 | |
| pabelanger | oh, actually. it looks like in our pump should do both | 18:44 |
| *** khappone has quit IRC | 18:44 | |
| *** electrical has quit IRC | 18:44 | |
| *** vdrok has quit IRC | 18:44 | |
| *** tobberydberg has quit IRC | 18:44 | |
| *** berendt has quit IRC | 18:44 | |
| *** verdurin has quit IRC | 18:44 | |
| *** cmurphy has quit IRC | 18:44 | |
| *** robcresswell has quit IRC | 18:44 | |
| *** mwhahaha has quit IRC | 18:44 | |
| *** shardy_afk has quit IRC | 18:44 | |
| *** wendar has quit IRC | 18:44 | |
| *** rajinir has quit IRC | 18:44 | |
| *** logan- has quit IRC | 18:44 | |
| *** Dmitrii-Sh has quit IRC | 18:44 | |
| *** andreykurilin has quit IRC | 18:44 | |
| *** neith has quit IRC | 18:44 | |
| *** coreycb has quit IRC | 18:44 | |
| *** seongsoocho has quit IRC | 18:44 | |
| *** hogepodge has quit IRC | 18:44 | |
| *** xyang has quit IRC | 18:44 | |
| *** nibalizer has quit IRC | 18:44 | |
| *** gus has quit IRC | 18:44 | |
| *** SergeyLukjanov has quit IRC | 18:44 | |
| *** ruhe has quit IRC | 18:44 | |
| *** jamielennox has quit IRC | 18:44 | |
| *** therve has quit IRC | 18:44 | |
| *** johnthetubaguy has quit IRC | 18:44 | |
| *** vdrok_ is now known as vdrok | 18:44 | |
| *** electrical_ is now known as electrical | 18:44 | |
| *** berendt_ is now known as berendt | 18:44 | |
| *** rajinir_ is now known as rajinir | 18:44 | |
| *** mwhahaha_ is now known as mwhahaha | 18:44 | |
| *** neith_ is now known as neith | 18:44 | |
| *** jamielennox_ is now known as jamielennox | 18:44 | |
| *** thorre has quit IRC | 18:44 | |
| *** Dmitrii-Sh_ is now known as Dmitrii-Sh | 18:44 | |
| *** robcresswell_ is now known as robcresswell | 18:44 | |
| *** seongsoocho_ is now known as seongsoocho | 18:44 | |
| *** xyang_ is now known as xyang | 18:44 | |
| *** hogepodge_ is now known as hogepodge | 18:44 | |
| *** coreycb_ is now known as coreycb | 18:45 | |
| *** ruhe2 is now known as ruhe | 18:45 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 18:45 | |
| *** thorre has joined #openstack-infra | 18:45 | |
| pabelanger | my mistake, we read from /dev/random | 18:45 |
| mnaser | internet is melting down | 18:45 |
| mnaser | digging in the code so far .. https://github.com/libvirt/libvirt/blob/master/src/util/virrandom.c#L93-L127 | 18:46 |
| *** jcoufal_ has joined #openstack-infra | 18:46 | |
| *** danpawlik has quit IRC | 18:47 | |
| *** logan_ is now known as logan- | 18:47 | |
| *** Qiming_ has quit IRC | 18:47 | |
| mnaser | im not sure where the random() func gets that stuff from | 18:48 |
| *** tobberydberg has joined #openstack-infra | 18:48 | |
| *** wolverineav has quit IRC | 18:49 | |
| *** yamamoto has quit IRC | 18:49 | |
| johnsom | Is zuul status down? | 18:49 |
| *** Qiming has joined #openstack-infra | 18:49 | |
| *** wolverineav has joined #openstack-infra | 18:49 | |
| *** jcoufal has quit IRC | 18:50 | |
| mordred | johnsom: it does not seem to be loading for me | 18:50 |
| mnaser | same | 18:50 |
| pabelanger | something is going on | 18:50 |
| pabelanger | http://grafana.openstack.org/dashboard/db/zuul-status | 18:50 |
| mordred | the server is reachable over ssh and the processes are running | 18:50 |
| *** nibalizer has joined #openstack-infra | 18:50 | |
| mnaser | i wonder if it has to do with the big netsplit | 18:51 |
| *** rm_work has joined #openstack-infra | 18:51 | |
| *** rm_work has quit IRC | 18:51 | |
| *** rm_work has joined #openstack-infra | 18:51 | |
| pabelanger | wow | 18:51 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 18:51 |
| pabelanger | large spike in memory | 18:52 |
| pabelanger | we are swapping | 18:52 |
| mnaser | merger queue | 18:52 |
| mordred | pabelanger: yah - that really spiked | 18:52 |
| *** verdurin has joined #openstack-infra | 18:52 | |
| mnaser | peaked at 1.1k? | 18:52 |
| clarkb | I think release team has a change series exercising theergers | 18:52 |
| clarkb | but ahouldnt be doing zuul updates accorsing to them | 18:53 |
| *** dhajare has joined #openstack-infra | 18:53 | |
| AJaeger | swapping again ;( | 18:54 |
| *** wolverineav has quit IRC | 18:54 | |
| pabelanger | yah, looks like dhellmann pushed up a stack of changes http://eavesdrop.openstack.org/irclogs/%23openstack-release/latest.log.html#t2018-02-23T18:09:08 then large spike after that | 18:54 |
| AJaeger | according to http://grafana.openstack.org/dashboard/db/zuul-status only 5 changes in check, and 9 in gate - that looks wrong | 18:54 |
| pabelanger | we'll have to dump queues and restart zuul to stop swapping | 18:55 |
| pabelanger | I have to finish packing for ptg myself | 18:56 |
| persia | pabelanger: the large spike correlates with changes from "OpenStack Release Bot" for me. Looks like 2-3 changes per repo were reported in the channels I watch, although that was ~45 minutes ago (and should be clear by now) | 18:56 |
| *** rossella_s has quit IRC | 18:56 | |
| *** lbragstad has joined #openstack-infra | 18:57 | |
| *** fried_rice_jfk has quit IRC | 18:57 | |
| *** rossella_s has joined #openstack-infra | 18:57 | |
| pabelanger | yah, https://review.openstack.org/#/q/owner:infra-root%2540openstack.org show a lot of new patches | 18:59 |
| AJaeger | pabelanger: feel free to +A https://review.openstack.org/#/c/547520 anytime | 18:59 |
| AJaeger | pabelanger: topic:create-newton ??? that would explain some things... | 19:00 |
| *** olaph1 has joined #openstack-infra | 19:00 | |
| pabelanger | yah | 19:00 |
| *** olaph has quit IRC | 19:00 | |
| clarkb | new branches with new config all at once? | 19:00 |
| pabelanger | I see at least 75 of them | 19:01 |
| pabelanger | I guess we'll need dhellmann or smcginnis to confirm what is going on there | 19:02 |
| *** jcoufal has joined #openstack-infra | 19:02 | |
| smcginnis | Looks like some clean up went awry and recreated a bunch of stable/newton branches. | 19:03 |
| *** agopi is now known as agopi|brb | 19:03 | |
| smcginnis | How do we get those deleted again? | 19:03 |
| pabelanger | if we created new branches, i think that means zuul now starts to try and parse configuration from it. Leading to increased memory usage? | 19:03 |
| *** jcoufal__ has joined #openstack-infra | 19:04 | |
| pabelanger | mordred: clarkb: I have to step away and finish packing. I'll try to return shortly. | 19:05 |
| *** jcoufal_ has quit IRC | 19:05 | |
| AJaeger | pabelanger: 75+ repos - new branches to parse, 75+ new changes - nice DoS ;( | 19:05 |
| clarkb | ya so what we probably want to do is get any necessary cleanup done then restart zuul | 19:06 |
| clarkb | smcginnis: can you work out what cleanup is necessary then infra can help as needed to do that? | 19:06 |
| *** jcoufal has quit IRC | 19:07 | |
| smcginnis | clarkb: Can we delete those branches without open patches out there? | 19:07 |
| smcginnis | Or do we need to get all patches abandoned first, then delete branches? | 19:08 |
| clarkb | must abandon first | 19:08 |
| smcginnis | dhellmann: ^^ | 19:08 |
| smcginnis | OK, I am working on that where I have rights and I have sent out a post to the ML. | 19:08 |
| clarkb | gerrit wont delete a branch if there are changes proposed to the branch | 19:08 |
| dhellmann | o/ | 19:08 |
| smcginnis | Unfortunately bad timing with people starting to travel. | 19:08 |
| dhellmann | ugh, ok | 19:08 |
| clarkb | infra can help abandon and delete | 19:08 |
| AJaeger | clarkb: want to hand out some rights? I can help with abandon... | 19:08 |
| clarkb | yup | 19:09 |
| clarkb | if we make a list of what is required I can delegate access | 19:09 |
| clarkb | (I just want us ti make sure we know what we are deleting before we delete it) | 19:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: have tag-releases skip closed series https://review.openstack.org/546801 | 19:10 |
| dhellmann | clarkb , AJaeger : can we land ^^ please? that should prevent this from happening again | 19:10 |
| dhellmann | and then we can clean up the mess | 19:10 |
| dhellmann | did my patch series cause the swapping issue? | 19:10 |
| AJaeger | clarkb: https://review.openstack.org/#/q/owner:infra-root%2540openstack.org+topic:create-newton and topic:reno-newton | 19:10 |
| clarkb | dhellmann: the creation if new branches with new zuul config did I think | 19:10 |
| dhellmann | I'm working on a script to find the spurious branches but maybe smcginnis is ahead of me | 19:10 |
| dhellmann | ok | 19:10 |
| AJaeger | dhellmann: or both together | 19:11 |
| dhellmann | is it just newton? | 19:11 |
| dhellmann | wow, that's a bunch | 19:11 |
| mordred | dhellmann: +2 | 19:11 |
| smcginnis | I don't have alist yet. Just going through the open patches and abandoning. | 19:12 |
| dhellmann | smcginnis : ok. do you want to use the patch list to make the branch list first? | 19:12 |
| * mordred needs a gerrty config that has all of the openstack projects in it ... | 19:12 | |
| clarkb | re that change I guess we might need to force merge it? | 19:13 |
| dhellmann | smcginnis : alternatively I can script something to look at repos that have eol tags and branches with the same name | 19:13 |
| AJaeger | dhellmann: also some mitaka ones - but those have been abandoned already. | 19:13 |
| smcginnis | dhellmann: That might be quickest. | 19:13 |
| AJaeger | Question is whether https://review.openstack.org/#/q/owner:infra-root%2540openstack.org is all we need to check? Or whether there's something else? | 19:13 |
| dhellmann | smcginnis : which? | 19:13 |
| smcginnis | dhellmann: Your script. | 19:13 |
| dhellmann | ok, I'll work on that while you abandon patches | 19:14 |
| mordred | AJaeger: I think : https://review.openstack.org/#/q/owner:infra-root%2540openstack.org+(topic:create-newton+OR+topic:reno-newton)+status:open | 19:14 |
| *** myoung is now known as myoung|food | 19:14 | |
| mordred | gets the list of them | 19:14 |
| smcginnis | Should probably just be https://review.openstack.org/#/q/status:open+branch:stable/newton | 19:16 |
| mnaser | smcginnis: fyi some projects still carry stable/newton branches | 19:16 |
| smcginnis | I've -1 the ones where I don't have rights to abandon. | 19:16 |
| smcginnis | mnaser: Yeah, I'm watching out for any that pop up, but none so far. | 19:16 |
| mnaser | ok cool | 19:17 |
| mnaser | feel free to ping for assistance :> | 19:17 |
| * mordred has discovered the "O" keyboard shortcut is useful here | 19:18 | |
| *** andreas_s has quit IRC | 19:18 | |
| clarkb | mnaser: ya thats why I want us to start from a set list so its clear what needs to be abandoned and deleted | 19:18 |
| *** salv-orlando has joined #openstack-infra | 19:19 | |
| smcginnis | OK, I think they are all gone. | 19:19 |
| mordred | smcginnis: yah - I believe they're all abandoned now | 19:20 |
| smcginnis | So now we just need the list of newly created branches and to get those deleted. | 19:20 |
| mordred | ++ | 19:21 |
| *** salv-orlando has quit IRC | 19:23 | |
| dhellmann | here's a script to find the branches, please check my logic : http://paste.openstack.org/show/683144/ | 19:24 |
| dhellmann | I'm still updating the local copies of those repos | 19:24 |
| smcginnis | +1 | 19:24 |
| dhellmann | what's the git syntax to close a branch, I can just have my script produce the commands to run | 19:27 |
| *** jcoufal has joined #openstack-infra | 19:27 | |
| dhellmann | git push origin :$branch right? | 19:27 |
| fungi | `git push gerrit :stable/newton` i think? | 19:27 |
| fungi | assuming review.o.o is your "gerrit" remote | 19:28 |
| dhellmann | oh, yeah, and it has to run in that directory | 19:28 |
| dhellmann | hrm | 19:28 |
| fungi | or use --git-dir=... | 19:28 |
| dhellmann | either way, it assumes all of these things are checked out | 19:29 |
| fungi | but yes you at least need clones of them all | 19:29 |
| *** SpamapS has joined #openstack-infra | 19:29 | |
| *** SpamapS has quit IRC | 19:29 | |
| *** SpamapS has joined #openstack-infra | 19:29 | |
| *** jcoufal__ has quit IRC | 19:29 | |
| smcginnis | I don't suppose there is a way to query gerrit for that. | 19:29 |
| dhellmann | we could query cgit but if we need local copies anyway... | 19:30 |
| dhellmann | I'm happy to either run the commands or give someone the output to run | 19:30 |
| dhellmann | 49 branches, that's impressive | 19:31 |
| dhellmann | I'm still updating some of the projects that come later in the alphabet though | 19:31 |
| *** SpamapS has quit IRC | 19:31 | |
| *** SpamapS has joined #openstack-infra | 19:31 | |
| persia | smcginnis: https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html suggets it is possible to construct a query to get a list of branches for a project, which may be sufficient. | 19:31 |
| fungi | so we've at least proved that the current flavor for the zuul scheduler takes roughly 50 concurrent config proposals to knock it over | 19:32 |
| persia | can branches be closed without a local checkout? | 19:32 |
| smcginnis | persia: Thanks | 19:32 |
| dhellmann | here we go: http://paste.openstack.org/show/683158/ | 19:32 |
| dhellmann | oh, not quite, missed the remote name | 19:33 |
| dhellmann | http://paste.openstack.org/show/683159/ | 19:33 |
| fungi | persia: in theory i think you could use just a git init and then repeatedly tweak the remote (or use explicit remote urls on the command line) | 19:33 |
| smcginnis | Ah, probably not. That must be why dhellmann said you need local copies. | 19:33 |
| dhellmann | closing the branch is a push operation, so you have to have a local repo to push from | 19:33 |
| dhellmann | the querying could all be done remotely, but you still need copies of everything to change it | 19:34 |
| persia | fungi: Indeed. An empty repo and hacking .git/ in the script should work, assuming one is only pushing null (which may be sufficient to close things) | 19:34 |
| dhellmann | so I built that list by running list-deliverables -r --series newton and --series mitaka then running the find script over the list of repositories | 19:34 |
| fungi | since it's just pushing an object id of 0 it could _probably_ be done with any local git repo, not necessarily a clone of the actual repo, but that would need testing | 19:35 |
| dhellmann | that should give us all of the repos involved in mitaka or newton that have branches that also have eol tags | 19:35 |
| dhellmann | should I go back further than mitaka? | 19:35 |
| smcginnis | Feels right. 49 branches. | 19:35 |
| smcginnis | I didn't see anything older than mitaka. | 19:35 |
| fungi | persia: anything settable in .git/config can also just be set as command-line options too, so no actual need to tweak configuration on each pass | 19:35 |
| persia | Even better. Do you have a sample command line that would close a branch in a repo from a bare `git init` result? | 19:36 |
| dhellmann | I already have copies of every repo, so it's no hardship for me to use those local directories | 19:37 |
| fungi | something like `git push ssh://fungi@review.openstack.org:29418/openstack-infra/bindep.git :stable/newton` should do | 19:37 |
| persia | Oh, heh, false optimisation. | 19:37 |
| fungi | but as i said, i've not actually tested that to confirm | 19:37 |
| dhellmann | including liberty does not give me any additional branches | 19:38 |
| dhellmann | do we trust the script output? | 19:39 |
| clarkb | fungi: you need repo ownership to do that though right? | 19:40 |
| *** danpawlik has joined #openstack-infra | 19:40 | |
| clarkb | alternatively we can just clikc the buttons in the web ui | 19:40 |
| mordred | clarkb: I think project-bootstrappers can totally do that | 19:40 |
| clarkb | mordred: ya thats true, it can do just about anything | 19:40 |
| fungi | clarkb: yeah, whoever is going to do the deletes can just be add to bootstrappeers | 19:41 |
| fungi | i think there's also a branch deletion rest api method now in 2.13 | 19:41 |
| dhellmann | I'm comfortable using the push method this time around | 19:41 |
| fungi | though explicit permission for that separate from owner needs 2.14 | 19:42 |
| dhellmann | although if we have an API for doing it we could finally update the release automation to use that | 19:42 |
| mordred | dhellmann: your script looks fine to me | 19:42 |
| * mordred would be happy to give dhellmann project-bootstrappers and let him run that script | 19:42 | |
| mordred | dhellmann: yah - once we've upgraded to 2.14 you should be good to go | 19:43 |
| fungi | dhellmann: yeah, we planned to do it with the api once we don't need to give the release automation owner permissions | 19:43 |
| clarkb | I'm not really able to debug or help much right this moment so will defer to others who can | 19:43 |
| dhellmann | fungi : right, I forgot there was still a permission granularity thing | 19:43 |
| fungi | so needs 2.14 or later to add the delete ref acl | 19:43 |
| *** dhajare has quit IRC | 19:43 | |
| dhellmann | ok, if someone wants to do the needful with the acls I will test with the aodh repo before running the whole script | 19:43 |
| mordred | fungi, clarkb: I vote for giving dhellmann project-bootstrappers,but I can also work on clicking thingsin the web ui if y'all aren't comfortable with that | 19:43 |
| dhellmann | sure, that works too, whatever folks think is safer | 19:44 |
| clarkb | we've given tonyb similar access before to run script that eol's branches | 19:44 |
| clarkb | so I'm fine with it from that perspective. Maybe test script aginst single project first before doing the rest? | 19:44 |
| mordred | ++ | 19:44 |
| dhellmann | clarkb : yep, definitely | 19:44 |
| fungi | yeah, i'm cool with temporary privilege escalation for that | 19:44 |
| mordred | dhellmann: you have access - wanna try the aodh one? | 19:44 |
| dhellmann | I have all but the aodh line commented out in my local copy | 19:45 |
| dhellmann | ok, running now | 19:45 |
| *** rossella_s has quit IRC | 19:45 | |
| dhellmann | sigh, bug in the output, stand by | 19:46 |
| fungi | i would help but am wrists deep in pizza dough | 19:46 |
| mordred | yay testing! | 19:46 |
| dhellmann | git review -s not git remote -s | 19:46 |
| mordred | dhellmann: :) | 19:46 |
| mordred | fungi: are you baking your hands into a pizza? | 19:46 |
| *** e0ne has joined #openstack-infra | 19:46 | |
| fungi | yes. totally | 19:46 |
| *** yamamoto has joined #openstack-infra | 19:47 | |
| dhellmann | ok aodh seems to have worked | 19:47 |
| mordred | I concur | 19:47 |
| *** rossella_s has joined #openstack-infra | 19:47 | |
| smcginnis | Confirmed | 19:47 |
| dhellmann | I'll run a few more | 19:48 |
| *** danpawlik has quit IRC | 19:48 | |
| dhellmann | seems to be working ok | 19:49 |
| dhellmann | shall I go ahead and run all of the rest? | 19:49 |
| mordred | dhellmann: go for it! | 19:49 |
| pabelanger | and back | 19:49 |
| pabelanger | reading backlog | 19:49 |
| *** jcoufal has quit IRC | 19:50 | |
| dhellmann | chugging along | 19:50 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Mirror tumbleweed for opensuse https://review.openstack.org/547646 | 19:50 |
| clarkb | once dhellmann is done next step is to restart zuul I think | 19:51 |
| clarkb | as I don't think it will recover on its own will it? | 19:51 |
| *** yamamoto has quit IRC | 19:52 | |
| AJaeger | clarkb: it did recover somehow... | 19:52 |
| AJaeger | clarkb: I think it's still swapping but website is reachable again | 19:52 |
| clarkb | oh interesting | 19:52 |
| AJaeger | clarkb: so, yes - let's restart | 19:52 |
| dhellmann | ok the script is done | 19:53 |
| dhellmann | re-running my find script locally after updating the repos with -p shows no branches to clean up | 19:54 |
| mordred | woot | 19:55 |
| mordred | dhellmann: I have removed you from project bootstrappers | 19:56 |
| dhellmann | good, thank you | 19:56 |
| smcginnis | \o/ | 19:56 |
| dhellmann | sorry for the screw-up and thank you all for helping to fix it | 19:56 |
| smcginnis | ++ | 19:57 |
| clarkb | mordred: are you able to do the zuul restart too? process is make sure nothing important in the release pipeline. Grab check and gate pipelines. Stop all three zuul services on zuul01 (scheduler, web, fingergw). Check that they actually stop, scheduler can be slow and in some cases won't stop until you kill it. Then start services and reenqueue | 19:57 |
| *** sree__ has joined #openstack-infra | 19:57 | |
| mordred | clarkb: do we have helper commands for grabbing the pipelines and re-enqueing now? | 19:58 |
| clarkb | mordred: we do, if you check roots history on zuul01 and grep for check.sh and gate.sh you'll see them | 19:58 |
| clarkb | its a script running out of /opt/zuul/somethingorother | 19:58 |
| mordred | clarkb: I do indeed see that - yup, I can do it | 19:59 |
| clarkb | it looks at status.json and generates enqueing scripts | 19:59 |
| mordred | clarkb: you wanna send a notice first? | 19:59 |
| clarkb | we can | 19:59 |
| clarkb | I think today is kind of slow with being friday and a travel day for many | 20:00 |
| mordred | clarkb: of course - checking the release pipeline and/or fetching check and gate is ... slow | 20:00 |
| clarkb | ya :( | 20:00 |
| mordred | I'm going to go ahead and start the 'please to fetching me the pipelines' commands | 20:00 |
| *** sree__ has quit IRC | 20:02 | |
| fungi | dhellmann: no need to apologize for things you fix! ;) | 20:02 |
| fungi | that should be one of our primary polices around here | 20:03 |
| clarkb | and thank you for helping to fix things :) | 20:03 |
| * fungi breaks stuff constantly, considers a "good day" to be any where he fixes more things than he breaks | 20:05 | |
| *** agopi|brb is now known as agopi | 20:06 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Mirror tumbleweed for opensuse https://review.openstack.org/547646 | 20:07 |
| *** rossella_s has quit IRC | 20:08 | |
| mordred | clarkb: urllib2.HTTPError: HTTP Error 502: Proxy Error | 20:09 |
| mordred | clarkb, fungi: actually - zuul seems to have recovered | 20:10 |
| mordred | so we may not need to do the restart? | 20:10 |
| fungi | if it's still sucked up all the available ram, we probably need to restart it anyway | 20:12 |
| *** lpetrut has quit IRC | 20:12 | |
| *** rossella_s has joined #openstack-infra | 20:12 | |
| fungi | yeah, memory usage and iowait on cpu still look no bueno | 20:13 |
| pabelanger | mordred: what I do, it cp /var/lib/zuul/www/backup/status.json.<latest hash> into /tmp | 20:16 |
| pabelanger | then use file:///tmp for python script url | 20:16 |
| fungi | i expect python gobbled up all available ram plus some extra, and now the kernel is swapping other active system tasks in and out | 20:16 |
| *** wolverineav has joined #openstack-infra | 20:17 | |
| fungi | yeah, we're around 16gb into swap right now | 20:18 |
| *** salv-orlando has joined #openstack-infra | 20:19 | |
| *** wolverineav has quit IRC | 20:22 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/zuul-jobs master: Setup repositories for openSUSE Tumbleweed https://review.openstack.org/547660 | 20:23 |
| *** salv-orlando has quit IRC | 20:24 | |
| *** weshay is now known as weshay_bbin30 | 20:25 | |
| *** jcoufal has joined #openstack-infra | 20:25 | |
| *** david-lyle has joined #openstack-infra | 20:25 | |
| *** xarses has joined #openstack-infra | 20:25 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Use new names for jobs in neutron periodic dashboard https://review.openstack.org/547661 | 20:25 |
| *** caphrim007_ has joined #openstack-infra | 20:25 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/openstack-zuul-jobs master: DNM https://review.openstack.org/546846 | 20:25 |
| *** andreww has joined #openstack-infra | 20:26 | |
| *** caphrim007_ has quit IRC | 20:26 | |
| *** dprince has quit IRC | 20:27 | |
| *** myoung|food is now known as myoung | 20:28 | |
| *** weshay_bbin30 is now known as weshay | 20:28 | |
| *** hongbin has joined #openstack-infra | 20:28 | |
| *** rossella_s has quit IRC | 20:28 | |
| *** caphrim007 has quit IRC | 20:29 | |
| *** xarses has quit IRC | 20:30 | |
| *** rossella_s has joined #openstack-infra | 20:33 | |
| *** caphrim007 has joined #openstack-infra | 20:33 | |
| *** caphrim007 has quit IRC | 20:34 | |
| *** pramodrj07 has joined #openstack-infra | 20:34 | |
| *** VW has quit IRC | 20:41 | |
| dirk | is zuul.o.o still supposed to do something? I'm trying to trace a running build but the site is unresponsive | 20:41 |
| *** danpawlik has joined #openstack-infra | 20:43 | |
| *** rossella_s has quit IRC | 20:44 | |
| AJaeger | dirk: known issue - mordred is restarting.. | 20:45 |
| *** sree__ has joined #openstack-infra | 20:45 | |
| *** e0ne has quit IRC | 20:46 | |
| *** rossella_s has joined #openstack-infra | 20:46 | |
| *** yamamoto has joined #openstack-infra | 20:48 | |
| *** sree__ has quit IRC | 20:49 | |
| *** slaweq_ has quit IRC | 20:50 | |
| *** slaweq_ has joined #openstack-infra | 20:51 | |
| *** wolverineav has joined #openstack-infra | 20:52 | |
| *** yamamoto has quit IRC | 20:53 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/statusbot master: WIP: Add optional MQTT notification support https://review.openstack.org/547667 | 20:55 |
| *** slaweq_ has quit IRC | 20:55 | |
| fungi | that's ^ a neat idea! | 20:57 |
| *** wolverineav has quit IRC | 20:57 | |
| fungi | though the bot centralization plan suggests maybe we don't need to do the same to every bot if there's insufficient rev/dev bandwidth | 20:58 |
| *** rossella_s has quit IRC | 20:58 | |
| *** kgiusti has left #openstack-infra | 20:59 | |
| *** r-daneel has joined #openstack-infra | 21:00 | |
| *** rossella_s has joined #openstack-infra | 21:01 | |
| dirk | pabelanger: we still need one change to zuul-jobs, I've updated your DNM patch but I can not find the job in the zuul queue and I'm giving up for today, its getting late here | 21:01 |
| *** olaph has joined #openstack-infra | 21:01 | |
| dirk | AJaeger: thanks | 21:02 |
| mordred | ok. I'm gonna do the restart now | 21:02 |
| mordred | I use pabelanger's idea and have queues saved | 21:02 |
| *** pramodrj07 has quit IRC | 21:02 | |
| *** olaph1 has quit IRC | 21:03 | |
| *** weshay is now known as weshay_bbiab | 21:04 | |
| *** jtomasek has quit IRC | 21:05 | |
| mordred | services are restarted | 21:05 |
| *** wolverineav has joined #openstack-infra | 21:05 | |
| fungi | thanks mordred! | 21:08 |
| mordred | the config is being re-read atm - will re-enqueue as soon as that's done | 21:08 |
| mordred | uhm | 21:09 |
| mordred | 2018-02-23 21:08:52,674 ERROR zuul.RPCListener: Exception while running job | 21:09 |
| mordred | have we seen that before? | 21:09 |
| mordred | for job_name in sorted(tenant.layout.jobs): | 21:09 |
| mordred | AttributeError: 'NoneType' object has no attribute 'layout' | 21:09 |
| mordred | I think that's maybe just the rpc listener being sad before the tenant is ready | 21:10 |
| mordred | re-enqueuing now | 21:11 |
| dmsimard | mordred: does it spam that ? I've seen that before and yeah, that's because the tenant isn't ready yet | 21:12 |
| dmsimard | sorry if I haven't been active today, fought a tire fire of my own | 21:12 |
| *** cmurphy_ is now known as cmurphy | 21:13 | |
| mriedem | it would be swell if we could unblock ocata https://review.openstack.org/#/c/546850/ | 21:14 |
| mriedem | needs a 2nd +2 | 21:14 |
| *** janki has quit IRC | 21:14 | |
| mordred | mriedem: ocata. that's so last year | 21:18 |
| mordred | mriedem: (done) | 21:18 |
| *** jcoufal_ has joined #openstack-infra | 21:18 | |
| smcginnis | Seems longer ago than last year. | 21:18 |
| AJaeger | mordred: exactly that's the problem ;) | 21:18 |
| *** ldnunes has quit IRC | 21:18 | |
| fungi | put ocata on a pike for the queens | 21:18 |
| smcginnis | And wash it down with a stein? :) | 21:19 |
| fungi | mmm | 21:19 |
| dmsimard | fungi: that escalated quickly | 21:20 |
| *** salv-orlando has joined #openstack-infra | 21:20 | |
| *** jcoufal has quit IRC | 21:20 | |
| *** salv-orlando has quit IRC | 21:22 | |
| *** olaph1 has joined #openstack-infra | 21:24 | |
| *** salv-orlando has joined #openstack-infra | 21:25 | |
| mriedem | thanks | 21:26 |
| *** olaph has quit IRC | 21:26 | |
| mnaser | frickler: around buy any chance? | 21:27 |
| mnaser | if so, can you have a look at https://review.openstack.org/#/c/547431/ ? stable/pike is blocked :< | 21:28 |
| *** olaph has joined #openstack-infra | 21:28 | |
| mnaser | need that and then something on top in stable/ocata and stable/pike to bump it up | 21:28 |
| *** olaph1 has quit IRC | 21:28 | |
| *** armaan has quit IRC | 21:30 | |
| *** armaan has joined #openstack-infra | 21:30 | |
| *** andreww has quit IRC | 21:33 | |
| *** dave-mcc_ has joined #openstack-infra | 21:34 | |
| *** dave-mccowan has quit IRC | 21:36 | |
| *** jcoufal has joined #openstack-infra | 21:39 | |
| *** andreww has joined #openstack-infra | 21:40 | |
| *** edmondsw has quit IRC | 21:40 | |
| *** jcoufal_ has quit IRC | 21:41 | |
| *** armaan has quit IRC | 21:42 | |
| *** armaan has joined #openstack-infra | 21:42 | |
| *** xarses has joined #openstack-infra | 21:42 | |
| tobiash | mordred: I think I've seen a patch doing something with sorted and layout and jobs | 21:44 |
| *** andreww has quit IRC | 21:45 | |
| *** mriedem has quit IRC | 21:49 | |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| *** VW has joined #openstack-infra | 21:52 | |
| *** hasharAway has quit IRC | 21:54 | |
| *** yamamoto has quit IRC | 21:55 | |
| *** hongbin has quit IRC | 21:56 | |
| *** hongbin has joined #openstack-infra | 21:57 | |
| *** danpawlik has quit IRC | 22:00 | |
| *** danpawlik has joined #openstack-infra | 22:01 | |
| *** danpawlik has quit IRC | 22:02 | |
| *** danpawlik has joined #openstack-infra | 22:02 | |
| sc` | zuul.o.o is unresponsive to me. is it just my awesome dsl? | 22:06 |
| *** caphrim007 has joined #openstack-infra | 22:08 | |
| dmsimard | mordred, clarkb, fungi, corvus: ^ zuul01 went straight back to full RAM and swap http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=0&view_type=tree&graph_start=1519337276&graph_end=1519423676 | 22:09 |
| dmsimard | pabelanger: ^ | 22:09 |
| *** slaweq has quit IRC | 22:09 | |
| *** danpawlik has quit IRC | 22:10 | |
| fungi | maybe there were some other changes we renqueued which were actually the cause? | 22:10 |
| fungi | er, reenqueued | 22:10 |
| *** VW has quit IRC | 22:10 | |
| dmsimard | the zuul logs are eerily quiet | 22:11 |
| *** caphrim007 has quit IRC | 22:11 | |
| *** caphrim007 has joined #openstack-infra | 22:11 | |
| dmsimard | The last line is 2018-02-23 21:54:45,234 INFO zuul.GerritConnection: Updating <Change 0x7fa2b8eb9208 547673,None> | 22:11 |
| dmsimard | Which is a good 20 minutes ago, it would seem odd that there's a change with a "None" patchset | 22:11 |
| dmsimard | mnaser: nothing special about that change I suppose? https://review.openstack.org/#/c/547673/ | 22:12 |
| fungi | dmsimard: that seems to be a common pattern | 22:13 |
| mnaser | I did push and repush quickly after | 22:13 |
| fungi | grep ",None>" | 22:13 |
| *** bradm has joined #openstack-infra | 22:13 | |
| fungi | i imagine that's zuul updating a change when it doesn't (yet) know the current patchset number | 22:14 |
| fungi | plenty of them in older logs too | 22:14 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/statusbot master: Add optional MQTT notification support https://review.openstack.org/547667 | 22:14 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-statusbot master: Add support for configuring mqtt in statusbot https://review.openstack.org/547684 | 22:14 |
| *** StevenK has joined #openstack-infra | 22:15 | |
| clarkb | last time I did a restart it failed to stop because of a weird sync issue with a change | 22:15 |
| clarkb | I wonder if this is a similar issue | 22:16 |
| fungi | scheduler.pid is last modified Feb 23 21:04 | 22:16 |
| fungi | and the only zuul-scheduler processes running have a similar start time | 22:17 |
| *** caphrim007_ has joined #openstack-infra | 22:17 | |
| dmsimard | strace on the scheduler shows nothing | 22:17 |
| fungi | occam's razor: it wasn't the branch creation events which spiked memory, and we reenqueued whatever caused it after restart | 22:17 |
| dmsimard | (well, nothing useful) | 22:17 |
| *** fultonj has quit IRC | 22:18 | |
| *** weshay_bbiab is now known as weshay | 22:18 | |
| *** yamamoto has joined #openstack-infra | 22:18 | |
| *** jcoufal has quit IRC | 22:19 | |
| *** hongbin has quit IRC | 22:20 | |
| dmsimard | I'll comb the logs to see if I can find anything useful but logs are so verbose | 22:21 |
| *** salv-orlando has quit IRC | 22:21 | |
| *** caphrim007 has quit IRC | 22:21 | |
| *** salv-orlando has joined #openstack-infra | 22:21 | |
| *** rlandy|ruck has quit IRC | 22:23 | |
| dmsimard | 2018-02-23 21:50:49,200 WARNING kazoo.client: Connection dropped: socket connection error: Permission denied ? | 22:23 |
| dmsimard | 2018-02-23 21:19:31,916 ERROR gear.Client.b'Zuul Executor Client': Exception in poll loop: | 22:24 |
| dmsimard | There's a lot of kazoo.exceptions.NoNodeError | 22:24 |
| fungi | we seem to see that when memory spikes hard, and i suspect it's a symptom rather than a cause | 22:25 |
| *** e0ne has joined #openstack-infra | 22:25 | |
| dmsimard | #freenode is aware of the spamwave fyi | 22:25 |
| fungi | i figure | 22:25 |
| *** salv-orlando has quit IRC | 22:25 | |
| fungi | d | 22:25 |
| *** e0ne has quit IRC | 22:26 | |
| dmsimard | How do we tell if Zookeeper is healthy on nodepool.o.o ? /var/log/zookeeper logs date back to october | 22:27 |
| *** masber has joined #openstack-infra | 22:27 | |
| *** olaph1 has joined #openstack-infra | 22:27 | |
| *** olaph has quit IRC | 22:27 | |
| sc` | if those NoNodeError are followed by a success, that would be an indicator of good health | 22:28 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-statusbot master: Add support for configuring mqtt in statusbot https://review.openstack.org/547684 | 22:28 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable mqtt statusbot events on eavesdrop.o.o https://review.openstack.org/547685 | 22:28 |
| mtreinish | fungi, clarkb, pabelanger: ^^^ I think I got that right | 22:28 |
| fungi | dmsimard: anyway, given that immediately after restart memory utilization returned to almost exactly the same levels as before restart suggests it's something (or perhaps lots of somethings) we reenqueued | 22:29 |
| *** masber has quit IRC | 22:30 | |
| dmsimard | fungi: ah, I wasn't there for the better part of the day | 22:30 |
| *** masber has joined #openstack-infra | 22:30 | |
| fungi | in the span of a few minutes it ate 32gb of ram and 16gb of swap | 22:30 |
| dmsimard | it's crazy | 22:30 |
| fungi | i'm just looking at the cacti graph | 22:31 |
| mtreinish | fungi: you just need more ram :) | 22:31 |
| fungi | the original spike up to that happened between 18:30 and 19:00 | 22:32 |
| dmsimard | fungi: gerrit allows us to search with time ranges, there must not be that many changes uploaded within that span of time ? | 22:33 |
| fungi | right, that's where i was headed too | 22:33 |
| dmsimard | my advanced query-fu is not great but I'll try my hand at it | 22:34 |
| *** dave-mcc_ has quit IRC | 22:34 | |
| fungi | https://review.openstack.org/#/q/after:%222018-02-23+18:30:00%22+before:%222018-02-23+19:00:00%22 | 22:35 |
| dmsimard | fungi: the changes have been abandoned but have the branches been deleted ? | 22:35 |
| fungi | i wonder if we reenqueued them | 22:35 |
| dmsimard | I'll look in the last status.json we have backed up | 22:36 |
| fungi | seems like they would have all been immediately ejected though since they were already abandoned | 22:36 |
| fungi | i can load the status page right now | 22:38 |
| fungi | and there's really not a _lot_ queued | 22:38 |
| dmsimard | fungi: wait that's weird | 22:39 |
| dmsimard | there's a huge chain of changes | 22:39 |
| dmsimard | for openstack/releases | 22:39 |
| dmsimard | fungi: https://review.openstack.org/#/c/547210/ | 22:39 |
| dmsimard | is part of the chain | 22:39 |
| dmsimard | https://review.openstack.org/#/c/547562/ is the tip | 22:40 |
| fungi | the one for the releases repo, yeah. we've not previously seen long-ish (this was around 30?) stacks of changes eat that much memory | 22:40 |
| dmsimard | there's a large chain for zuul as well | 22:40 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't run neutron-grenade job on stable/ocata changes https://review.openstack.org/546850 | 22:41 |
| dmsimard | I'm not saying this is it but it feels unusual | 22:41 |
| *** salv-orlando has joined #openstack-infra | 22:43 | |
| clarkb | the large oen for releases repo was pushed yesterday and it was fine iirc | 22:44 |
| clarkb | because they don't touch zuul.yaml | 22:44 |
| *** dave-mccowan has joined #openstack-infra | 22:44 | |
| dmsimard | hmm I had a gerrit search yesterday to pull up all open changes with zuul files, let me get that | 22:45 |
| fungi | swap use has started falling sharply now too | 22:46 |
| dmsimard | file:"^(zuul.d|.zuul.d|zuul.yaml|.zuul.yaml).*" | 22:47 |
| dmsimard | fungi: the "after" field, is that since last update or creation of the change ? | 22:48 |
| dmsimard | Changes modified after the given 'TIME', inclusive. | 22:48 |
| fungi | after means it's been "modified (comment, new patchset, merged, something)" since then | 22:50 |
| dmsimard | fungi: swap is maxed again | 22:50 |
| fungi | so any activity between the after and before times counts | 22:51 |
| dmsimard | it's surprising oomkiller hasn't done anything | 22:51 |
| fungi | anyway https://review.openstack.org/#/q/after:%222018-02-23+17:00:00%22+before:%222018-02-23+20:00:00%22+file:%22%255E(zuul.d%257C.zuul.d%257Czuul.yaml%257C.zuul.yaml).*%22 doesn't indicate many changes to configuration | 22:51 |
| fungi | and yeah, swap is spiking back up again | 22:51 |
| dmsimard | it might not be related to zuul.yaml :/ | 22:52 |
| clarkb | 23:31:10* corvus | it's stuck because it's not actually removing 547073,5 from the queue for some reason <- from corvus on what the issue was last time we saw it spinning on a pipeline | 22:54 |
| dmsimard | This merged about 5 hours ago, could it be related ? http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/commit/?id=317eda746db3b7e5e1ea34f3e1133b01543a6a14 | 22:55 |
| clarkb | its possible that is buggy | 22:56 |
| dmsimard | The first spike started happening at around 18:20 ? According to cacti | 22:57 |
| dmsimard | The merge happened at 17:53 | 22:57 |
| *** yamamoto has quit IRC | 23:00 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools master: add missing lib files for presentation https://review.openstack.org/547689 | 23:03 |
| dmsimard | so I confirm that at 17:53, right after the merge, there was a reconfiguration (bunch of "zuul.TenantParser: Loading configuration from ...") | 23:04 |
| dmsimard | combing through that general area... | 23:04 |
| *** masuberu has joined #openstack-infra | 23:07 | |
| dmsimard | it seems to start at 2018-02-23 17:53:13,142 and end at 2018-02-23 17:53:55,451 | 23:08 |
| *** masber has quit IRC | 23:09 | |
| *** dave-mccowan has quit IRC | 23:11 | |
| dmsimard | I'd like to take a wild guess at submitting a revert for https://review.openstack.org/#/c/545442/, we can always re-submit it later if it's not that | 23:12 |
| dmsimard | I really don't have any other lead | 23:12 |
| *** yamamoto has joined #openstack-infra | 23:13 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Revert "Protect some jobs" https://review.openstack.org/547691 | 23:13 |
| dmsimard | infra-root: ^ I've had a rough day and taking a break for a bit, I don't have any other idas | 23:14 |
| dmsimard | ideas* | 23:14 |
| *** yamamoto has quit IRC | 23:14 | |
| *** e0ne has joined #openstack-infra | 23:14 | |
| clarkb | we can certainly try it and restart zuul again | 23:14 |
| *** e0ne has quit IRC | 23:15 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Revert "Protect some jobs" https://review.openstack.org/547691 | 23:15 |
| dmsimard | Maybe abstract and protected jobs are buggy or expensive? I have no clue | 23:15 |
| dmsimard | I looked at codesearch and there's a few instances of both but it seems like the first time we're adding that many at once | 23:16 |
| dmsimard | really just a wild stab in the dark | 23:16 |
| dmsimard | stepping away, I'll check in later | 23:16 |
| *** olaph1 has quit IRC | 23:20 | |
| *** olaph has joined #openstack-infra | 23:21 | |
| *** vaidy has joined #openstack-infra | 23:23 | |
| *** isviridov_away has joined #openstack-infra | 23:24 | |
| clarkb | another idea is we could restart zuul again and not requeue | 23:27 |
| clarkb | just let it sit for 20 minutes and see if memory skyrockets again. That implies the problem is merged already. If it is stable when idle that implies its something being tested on the fly | 23:27 |
| *** r-daneel has quit IRC | 23:27 | |
| *** slaweq has joined #openstack-infra | 23:30 | |
| *** eernst has quit IRC | 23:31 | |
| *** eernst has joined #openstack-infra | 23:31 | |
| *** slaweq has quit IRC | 23:35 | |
| dmsimard | clarkb: that's not a terrible idea | 23:35 |
| dmsimard | I'm cooking before I die of hunger but you have my full support :D | 23:36 |
| *** rossella_s has quit IRC | 23:36 | |
| andreaf | fungi dmsimard clarkb so are you guys planning a zuul restart? | 23:37 |
| *** rossella_s has joined #openstack-infra | 23:37 | |
| clarkb | andreaf: maybe? I'm not entirely sure we have a better option | 23:38 |
| clarkb | is any other infra-root around at this point? | 23:38 |
| *** salv-orlando has quit IRC | 23:39 | |
| *** salv-orlando has joined #openstack-infra | 23:39 | |
| clarkb | I'm checking to see if I can save queues | 23:40 |
| dmsimard | clarkb: mordred used the backup files earlier iirc | 23:40 |
| clarkb | ok if this doesn't come around in a few more minutes I'll look at doing that too | 23:41 |
| *** rossella_s has quit IRC | 23:42 | |
| *** salv-orlando has quit IRC | 23:44 | |
| *** rossella_s has joined #openstack-infra | 23:45 | |
| clarkb | hey it eventually came back | 23:47 |
| clarkb | doing gate now then I'll restart zuul I guess adn not immediately requeue anything | 23:47 |
| *** rossella_s has quit IRC | 23:47 | |
| *** rossella_s has joined #openstack-infra | 23:48 | |
| clarkb | ok I've got them trying restart now | 23:49 |
| clarkb | andreaf: ^ so yes going to give it a go | 23:49 |
| clarkb | I'll let it try to stop "gracefully" for a few minutse but kill it if it doesn't | 23:50 |
| *** eernst has quit IRC | 23:53 | |
| *** eernst has joined #openstack-infra | 23:54 | |
| clarkb | scheduler hasn't logged anything for about 5 minutes but is still running | 23:59 |
| clarkb | wondering if that is a good indication I'll need to kill it | 23:59 |
| clarkb | give it a couple more minutes then I'm going to do it | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!