| *** lukebrowning has quit IRC | 00:00 | |
| *** esberglu has joined #openstack-infra | 00:03 | |
| *** yamamoto has joined #openstack-infra | 00:04 | |
| *** lukebrowning has joined #openstack-infra | 00:06 | |
| *** esberglu has quit IRC | 00:07 | |
| jlvillal | clarkb, mordred: Just an FYI, I still see that POST_FAILURE issue on: https://review.openstack.org/#/c/508287/ | 00:08 |
|---|---|---|
| *** yamamoto has quit IRC | 00:09 | |
| *** markvoelker has quit IRC | 00:10 | |
| *** lukebrowning has quit IRC | 00:10 | |
| *** lnxnut has joined #openstack-infra | 00:11 | |
| clarkb | jlvillal: http://logs.openstack.org/87/508287/1/check/openstack-tox-pep8/a0a87e1/job-output.txt.gz#_2017-09-30_23_33_34_471809 thats really confusing | 00:11 |
| clarkb | it says its there and the perms are 775 | 00:11 |
| clarkb | its not a symlink | 00:11 |
| *** lukebrowning has joined #openstack-infra | 00:12 | |
| clarkb | but why is it using env dist when the job target is pep8? | 00:13 |
| clarkb | http://logs.openstack.org/87/508287/1/check/openstack-tox-pep8/a0a87e1/job-output.txt.gz#_2017-09-30_23_33_19_320707 | 00:13 |
| *** lrossetti_ has quit IRC | 00:13 | |
| clarkb | hrm that must be where tox stores the sdist? | 00:15 |
| *** lukebrowning has quit IRC | 00:16 | |
| *** lukebrowning has joined #openstack-infra | 00:18 | |
| clarkb | jlvillal: ok I think the reason it happens to oslo.serializations is that it doesn't skipsdist | 00:19 |
| clarkb | (still a bug in the job but explains why this project is special) | 00:19 |
| clarkb | ok I think I know what needs ot be done to fix it, now just a matter of implementing that in ansible | 00:20 |
| dmsimard | clarkb: you gave me a link about firewall not being set up properly but that link is ephemeral | 00:20 |
| dmsimard | did it end up being an issue ? | 00:21 |
| clarkb | dmsimard: ya sorry, I actually think it was fine and I was misreading the logs | 00:21 |
| clarkb | dmsimard: there were spaces in the log that split the output up and I just needed to keep reading to see where itdid the rest of it | 00:21 |
| dmsimard | clarkb: okay, fwiw the multi node things are fairly well integration tested in the o-z-j patchsets of zuulv3-multinode | 00:21 |
| dmsimard | but it's possible I missed something | 00:22 |
| *** lukebrowning has quit IRC | 00:22 | |
| *** larainema has joined #openstack-infra | 00:23 | |
| *** edmondsw has joined #openstack-infra | 00:23 | |
| clarkb | it seems to be working against the legacy too | 00:23 |
| clarkb | dmsimard: ok I have an ansible question for you, let me grab a link and describe what I want to do | 00:24 |
| mordred | clarkb, AJaeger: woot the infra-manual is publishing once more :) | 00:24 |
| *** lukebrowning has joined #openstack-infra | 00:24 | |
| clarkb | dmsimard: mordred https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/fetch-tox-output/tasks/main.yaml#n26 in that task we only want to copy that dir if the log dir exists as tox may put other dirs in .tox which don't have logs dirs in them | 00:24 |
| clarkb | hrm maybe I can edit the find task at the top of the file | 00:25 |
| clarkb | aha yup thats it | 00:26 |
| dmsimard | clarkb: let me see | 00:26 |
| *** kfarr has quit IRC | 00:26 | |
| *** sree has joined #openstack-infra | 00:26 | |
| dmsimard | oh you have your answer then ? | 00:26 |
| mordred | dmsimard: did we land the integration tests? | 00:26 |
| dmsimard | mordred: nope, the can't land until the tip of zuul-jobs lands, there's a depends-on | 00:27 |
| dmsimard | mordred: https://review.openstack.org/#/q/topic:zuulv3-multinode | 00:27 |
| clarkb | dmsimard: I think I do if you are able to review in a minute that would be great | 00:27 |
| mordred | clarkb: wait - did we not land my patch to that role? | 00:27 |
| clarkb | mordred: ^ you too | 00:27 |
| clarkb | mordred: we did, but its not sufficient | 00:27 |
| clarkb | mordred: you'll see | 00:27 |
| *** edmondsw has quit IRC | 00:27 | |
| mordred | clarkb: I will - but also, the patch isn't landed | 00:27 |
| mordred | clarkb: https://review.openstack.org/#/c/508563/ | 00:28 |
| mordred | clarkb: so make sure you start with that ^^ | 00:28 |
| mordred | clarkb: hrm. why did that not merge :( | 00:28 |
| *** lukebrowning has quit IRC | 00:29 | |
| mordred | clarkb: it depends on an old versoin of the parent patch - I think I should rebase it | 00:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make fetch-tox-output more resilient https://review.openstack.org/508563 | 00:30 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Only copy tox logs if the logs dir exists https://review.openstack.org/508712 | 00:30 |
| clarkb | mordred: oh actually your fix is probably better to start wit then | 00:30 |
| dmsimard | hmm, so it looks like the next tripleo hurdle is /opt/stack/new/tripleo-ci/toci_gate_test.sh: line 154: NODEPOOL_PROVIDER: unbound variable | 00:30 |
| *** lukebrowning has joined #openstack-infra | 00:30 | |
| *** sree has quit IRC | 00:31 | |
| dmsimard | NODEPOOL_PROVIDER is provided by /etc/ci/ci_info.sh I guess.. we added that, right ? | 00:31 |
| clarkb | dmsimard: before we get to that. You will likely need the all -> primary change in tripleo ci multinode jobs and you will likely need the firewall update too | 00:31 |
| mordred | clarkb: I rebased and re-aproved it - its jobs don't take long, so hopefully it'll land soon | 00:31 |
| clarkb | mordred: cool, I think yours makes mine unnecessary, forsome reason I thought I was operating on your change already | 00:32 |
| clarkb | I'll abandon | 00:32 |
| dmsimard | clarkb: yeah I reviewed those already but I can't really +2 :) | 00:32 |
| dmsimard | maybe I'll get it one day, but today is not that day | 00:32 |
| clarkb | dmsimard: ya just pointing out tripleo will face the same problems | 00:32 |
| mordred | dmsimard: I dont see a depends-on in https://review.openstack.org/#/q/topic:zuulv3-multinode | 00:32 |
| clarkb | dmsimard: so you need to port it to those jobs too | 00:32 |
| dmsimard | mordred: this one: https://review.openstack.org/#/c/504787/ | 00:33 |
| mordred | AH - I was looking at wrong thing | 00:33 |
| dmsimard | clarkb: yeah it's okay, I've just been fixing issues one after the other | 00:33 |
| dmsimard | thanks for figuring that one out | 00:33 |
| mordred | oh - that zuul-jobs should have landed days ago - re-kicking | 00:33 |
| clarkb | as for ci_info I have no idea | 00:33 |
| dmsimard | All of this we're hitting is kind of why I wanted to test deployment projects before doing the cutover :( | 00:34 |
| dmsimard | I knew they were the most vulnerable | 00:34 |
| *** bobh has joined #openstack-infra | 00:34 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused shebang from legacy jobs https://review.openstack.org/508627 | 00:34 |
| *** lnxnut has quit IRC | 00:34 | |
| clarkb | eh | 00:35 |
| mordred | dmsimard: OHHH - I get it all now | 00:35 |
| clarkb | puppet managed to completely switch out their jobs | 00:35 |
| *** lukebrowning has quit IRC | 00:35 | |
| clarkb | so yes, but also its not the end of the world I don't think | 00:35 |
| dmsimard | clarkb: yeah they got a head start and we found different issues just by submitting https://review.openstack.org/#/c/505921/ ahead of time | 00:36 |
| clarkb | dmsimard: but also devstack-gate has had many of these same problems | 00:36 |
| dmsimard | mordred: not sure what you're getting but I'm happy for you :D | 00:36 |
| clarkb | and we are iterating through them there as well | 00:36 |
| dmsimard | clarkb: surprisingly, right ? I thought devstack-gate should've been solid | 00:37 |
| clarkb | dmsimard: do you know why tripleo cares what the provider is? it really shouldn't | 00:37 |
| *** lukebrowning has joined #openstack-infra | 00:37 | |
| dmsimard | with all the work that went into it | 00:37 |
| clarkb | dmsimard: ya I just spent all day today getting devstack-gate working | 00:37 |
| clarkb | which then will allow devstack to get working next | 00:37 |
| mordred | clarkb: https://review.openstack.org/#/c/504554 ... I +2'd it which should clear that stack and let us land the integration tests - you wanna review it? | 00:37 |
| clarkb | which will then allow novaclient etc to work | 00:37 |
| dmsimard | clarkb: tripleo does a lot of funky things, I'm hunting it down now.. not super familiar with it | 00:37 |
| mordred | dmsimard: sorry - I understand why that whole stack isn't landed now :) | 00:37 |
| clarkb | dmsimard: my guess is its some abuse of some system that should be more directly addressed if we can track it down | 00:38 |
| dmsimard | clarkb: are we setting up /etc/nodepool/provider in v3 ? | 00:38 |
| clarkb | dmsimard: eg use the described mirror instead of constructing it | 00:38 |
| clarkb | dmsimard: no | 00:38 |
| clarkb | dmsimard: at least I don't think so | 00:38 |
| dmsimard | clarkb: then that's the issue.. not sure why they don't use /etc/ci https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test-oooq.sh#L19 | 00:38 |
| mordred | whatever we set is set in playbooks/legacy/pre.yaml | 00:38 |
| clarkb | mordred: ya I don't think anything is really using it yet and I seem to recall its a port of the other code soshould be fine | 00:38 |
| clarkb | mordred: so approve awawy | 00:39 |
| mordred | clarkb: kk. I'll just +A | 00:39 |
| *** bobh has quit IRC | 00:39 | |
| clarkb | https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test-oooq.sh#L49 that makes me sad | 00:39 |
| clarkb | dmsimard: but ya thta is consuming mirror info | 00:40 |
| clarkb | dmsimard: can probably better consume that from the other location | 00:40 |
| dmsimard | clarkb: the real reason why they need it is around here https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test-oooq.sh#L154 | 00:40 |
| dmsimard | clarkb: not going to pretend that's a clean way to do it or anything | 00:40 |
| dmsimard | but I understand why | 00:41 |
| dmsimard | tl;dr, different ovb and whatever config they might end up needing to set up depending if they're using rh1 or rdo-cloud (where we're migrating rh1) | 00:41 |
| *** lukebrowning has quit IRC | 00:41 | |
| clarkb | ugh | 00:41 |
| clarkb | that should be a job configuration item | 00:41 |
| dmsimard | I think that's in /etc/ci though, let me see | 00:42 |
| clarkb | then only run the job in the env thta it works in | 00:42 |
| clarkb | dmsimard: it might be | 00:42 |
| dmsimard | yeah, double checking | 00:42 |
| *** lukebrowning has joined #openstack-infra | 00:43 | |
| clarkb | its also in the inventory, could write out that file in the tripleo jobs (the provider file) | 00:44 |
| dmsimard | nope, it's not :/ | 00:44 |
| dmsimard | yeah, trying to think of the easiest way to fix that | 00:44 |
| dmsimard | so messy | 00:44 |
| clarkb | I would write out that file in a new role and probably just apply that to tripleo? | 00:44 |
| dmsimard | I already set all their dsvm jobs with a parent job to set the right required projects, I'll just add a playbook to template that file out | 00:45 |
| dmsimard | and run it in pre | 00:45 |
| clarkb | soundsgood | 00:45 |
| dmsimard | this is legacy stuff, not super interested in writing roles and stuff | 00:46 |
| dmsimard | we'll get things working again and they can figure out how they want to clean things up later | 00:46 |
| mordred | dmsimard: exactly | 00:46 |
| *** thorst has joined #openstack-infra | 00:48 | |
| *** lukebrowning has quit IRC | 00:48 | |
| clarkb | mordred: looking at current failures on the status page, the legacy-releasenotes job is going to be an important one to get sorted so that openstack can get back to business as usual | 00:50 |
| *** lukebrowning has joined #openstack-infra | 00:50 | |
| clarkb | I think you mentioned writing a native version of that job? also dhellmann has fixes to reno iirc (just pointing it out so we know its an important one) | 00:50 |
| mordred | clarkb: yes - I need to finish the native one - and write a publish-releasenotes | 00:51 |
| *** bobh has joined #openstack-infra | 00:51 | |
| mordred | clarkb: I want to rename the new releasenotes job to "build-releasenotes" as well before we start using it | 00:51 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 00:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Rename new releasenotes job https://review.openstack.org/508713 | 00:53 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 00:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 00:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Make fetch-tox-output more resilient https://review.openstack.org/508563 | 00:54 |
| * clarkb is going to recheck jlvillal's change now | 00:54 | |
| *** lukebrowning has quit IRC | 00:55 | |
| *** lukebrowning has joined #openstack-infra | 00:57 | |
| mordred | clarkb: there's a rename - I'll try to get them both done first thing in the morning | 00:57 |
| mordred | \o/ and there's the tox fix | 00:58 |
| clarkb | I think if we get release notes fixed and multinode fixed then projects like nova and neutron should be set | 00:58 |
| *** kaisers1 has joined #openstack-infra | 00:58 | |
| mordred | clarkb: I don't know if you've looked at https://review.openstack.org/#/c/504789/ yet ... but it adds integration tests for the multinode roles - including the one you started using in legacy jobs | 00:59 |
| *** lbragstad has quit IRC | 00:59 | |
| mordred | (it's a 4-patch stack) | 00:59 |
| mordred | but it's clear to land now | 00:59 |
| clarkb | I haven't I'll try to get to it tomorrow | 00:59 |
| *** kaisers has quit IRC | 01:00 | |
| clarkb | jlvillal: looks like it is working now that mordreds fix finally got in | 01:00 |
| * mordred does a dance ... and then goes to eat dinners | 01:00 | |
| *** lukebrowning has quit IRC | 01:01 | |
| *** lnxnut has joined #openstack-infra | 01:02 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add backwards-compat playbook for /etc/nodepool/provider https://review.openstack.org/508714 | 01:04 |
| dmsimard | clarkb, mordred ^ I think that should do it. | 01:04 |
| *** yamamoto has joined #openstack-infra | 01:05 | |
| *** lukebrowning has joined #openstack-infra | 01:05 | |
| dmsimard | er, zuul isn't happy with that one. | 01:06 |
| dmsimard | ah, doh, the key is pre-run, not pre | 01:07 |
| * dmsimard saturday evening commits | 01:07 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add backwards-compat playbook for /etc/nodepool/provider https://review.openstack.org/508714 | 01:08 |
| *** abelur_ has quit IRC | 01:08 | |
| *** lukebrowning has quit IRC | 01:10 | |
| *** yamamoto has quit IRC | 01:10 | |
| dmsimard | clarkb: in unrelated news, there's lot of good discussions around the upcoming things in firefox (performance improvement things with fancy names like quantum etc.) | 01:10 |
| dmsimard | might just win me back over, not that I'm especially attached to chrome | 01:11 |
| clarkb | ya I'm running beta :) | 01:12 |
| *** lukebrowning has joined #openstack-infra | 01:12 | |
| clarkb | noticeably faster | 01:12 |
| clarkb | but aome annoying new behavior around tabs | 01:12 |
| dmsimard | the new container thing ? | 01:12 |
| clarkb | no if I open a tab from a pinned tab new tab is at front of list. if I ^T new tab is at end of list | 01:13 |
| clarkb | I just want conaistency | 01:13 |
| dmsimard | oh | 01:14 |
| dmsimard | chrome actually does that | 01:15 |
| jeblair | clarkb, fungi, mordred: cacti is now collecting data on zuulv3 swap usage. also, for any new hosts, cacti will handle 1tb swap partitions. :) | 01:15 |
| dmsimard | if I middle click a link, it opens the tab to the right of the current tab | 01:15 |
| clarkb | ya they chromified it | 01:15 |
| dmsimard | ctrl+t will open on the far right | 01:15 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63981&rra_id=all | 01:15 |
| clarkb | which us annoyinf | 01:15 |
| dmsimard | jeblair: wow that's a lot of swap | 01:16 |
| dmsimard | jeblair: how come ? | 01:16 |
| *** lukebrowning has quit IRC | 01:16 | |
| dmsimard | it's usually bad when you're swapping at all, let alone 100+GB of swap lol | 01:16 |
| jeblair | dmsimard: zeros are free? i just added some zeroes to the end of the max value cacti accepts | 01:17 |
| jeblair | it was 100GB before | 01:17 |
| dmsimard | jeblair: oh, sure, but I meant that the zuulv3 node has 160GB of swap ? | 01:17 |
| jeblair | dmsimard: oh, 'cause we weren't using it for anything else | 01:17 |
| dmsimard | ¯\_(ツ)_/¯ | 01:17 |
| jeblair | and we wanted to let it get bigger than 10GB if it wasn't actively swapping so we could find out if it's idle or active memory | 01:18 |
| *** lukebrowning has joined #openstack-infra | 01:18 | |
| jeblair | obviously if it's actively swapping, it's not going to get that big before we restart anyway | 01:18 |
| dmsimard | right | 01:19 |
| dmsimard | jeblair: btw we missed another backwards compat thing, I sent https://review.openstack.org/#/c/508714/ | 01:19 |
| jeblair | dmsimard: thx | 01:19 |
| clarkb | jeblair: if you have time reviewing topic:fix-multinode would be great | 01:19 |
| dmsimard | I'm off for now, ttyl | 01:20 |
| *** thorst has quit IRC | 01:20 | |
| jeblair | clarkb: i have some holes in my house i need to close up before night creatures arrive; i may not get a chance soon. | 01:20 |
| *** thorst has joined #openstack-infra | 01:21 | |
| *** thorst has quit IRC | 01:21 | |
| *** lnxnut has quit IRC | 01:21 | |
| *** cshastri has joined #openstack-infra | 01:22 | |
| *** lukebrowning has quit IRC | 01:22 | |
| *** lukebrowning has joined #openstack-infra | 01:24 | |
| *** Apoorva has joined #openstack-infra | 01:25 | |
| *** lukebrowning has quit IRC | 01:29 | |
| *** LindaWang has quit IRC | 01:29 | |
| *** LindaWang has joined #openstack-infra | 01:29 | |
| *** caphrim007 has joined #openstack-infra | 01:32 | |
| *** lukebrowning has joined #openstack-infra | 01:33 | |
| *** caphrim007 has quit IRC | 01:37 | |
| *** lukebrowning has quit IRC | 01:38 | |
| *** Apoorva has quit IRC | 01:41 | |
| *** lukebrowning has joined #openstack-infra | 01:41 | |
| *** thorst has joined #openstack-infra | 01:41 | |
| *** lukebrowning has quit IRC | 01:45 | |
| *** thorst has quit IRC | 01:47 | |
| *** LindaWang has quit IRC | 01:47 | |
| *** lukebrowning has joined #openstack-infra | 01:47 | |
| *** esberglu has joined #openstack-infra | 01:50 | |
| *** lukebrowning has quit IRC | 01:52 | |
| *** lukebrowning has joined #openstack-infra | 01:53 | |
| *** esberglu has quit IRC | 01:55 | |
| *** lbragstad has joined #openstack-infra | 01:57 | |
| *** lukebrowning has quit IRC | 01:58 | |
| *** thorst has joined #openstack-infra | 02:03 | |
| *** yamamoto has joined #openstack-infra | 02:06 | |
| *** abelur has quit IRC | 02:07 | |
| *** markvoelker has joined #openstack-infra | 02:07 | |
| *** thorst has quit IRC | 02:08 | |
| *** abelur has joined #openstack-infra | 02:08 | |
| *** jascott1_ has joined #openstack-infra | 02:11 | |
| *** jascott1 has quit IRC | 02:11 | |
| *** edmondsw has joined #openstack-infra | 02:11 | |
| *** Goneri has quit IRC | 02:12 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** edmondsw has quit IRC | 02:15 | |
| jlvillal | clarkb, Thanks. Yes no more POST_FAILURE :) | 02:17 |
| *** lbragstad has quit IRC | 02:30 | |
| *** calebb has quit IRC | 02:35 | |
| *** slaweq has joined #openstack-infra | 02:39 | |
| *** sree has joined #openstack-infra | 02:40 | |
| *** markvoelker has quit IRC | 02:40 | |
| *** slaweq has quit IRC | 02:43 | |
| *** sree has quit IRC | 02:45 | |
| *** esberglu has joined #openstack-infra | 02:45 | |
| *** mat128 has quit IRC | 02:46 | |
| *** esberglu has quit IRC | 02:49 | |
| *** calebb has joined #openstack-infra | 02:49 | |
| *** lnxnut has joined #openstack-infra | 02:53 | |
| *** dave-mccowan has joined #openstack-infra | 02:53 | |
| *** lukebrowning has joined #openstack-infra | 02:54 | |
| *** baoli has quit IRC | 03:01 | |
| *** lukebrowning has quit IRC | 03:05 | |
| *** yamamoto has joined #openstack-infra | 03:08 | |
| *** lukebrowning has joined #openstack-infra | 03:09 | |
| *** isaacb has joined #openstack-infra | 03:10 | |
| *** lukebrowning has quit IRC | 03:13 | |
| *** yamamoto has quit IRC | 03:13 | |
| *** dave-mccowan has quit IRC | 03:14 | |
| *** lnxnut has quit IRC | 03:14 | |
| *** isaacb has quit IRC | 03:15 | |
| *** lukebrowning has joined #openstack-infra | 03:15 | |
| *** bobh has quit IRC | 03:15 | |
| *** dave-mccowan has joined #openstack-infra | 03:17 | |
| *** lukebrowning has quit IRC | 03:19 | |
| *** lukebrowning has joined #openstack-infra | 03:21 | |
| *** lukebrowning has quit IRC | 03:26 | |
| *** dave-mccowan has quit IRC | 03:26 | |
| *** dave-mccowan has joined #openstack-infra | 03:27 | |
| *** lukebrowning has joined #openstack-infra | 03:27 | |
| *** lukebrowning has quit IRC | 03:31 | |
| *** lukebrowning has joined #openstack-infra | 03:32 | |
| *** caphrim007 has joined #openstack-infra | 03:33 | |
| *** lukebrowning has quit IRC | 03:37 | |
| *** caphrim007 has quit IRC | 03:37 | |
| *** lukebrowning has joined #openstack-infra | 03:38 | |
| *** esberglu has joined #openstack-infra | 03:39 | |
| *** lukebrowning_ has joined #openstack-infra | 03:41 | |
| *** dave-mccowan has quit IRC | 03:41 | |
| *** lukebrowning has quit IRC | 03:42 | |
| *** esberglu has quit IRC | 03:43 | |
| *** lukebrowning_ has quit IRC | 03:46 | |
| *** thorst has joined #openstack-infra | 03:46 | |
| *** lukebrowning has joined #openstack-infra | 03:52 | |
| *** lukebrowning has quit IRC | 03:56 | |
| *** edmondsw has joined #openstack-infra | 03:59 | |
| *** edmondsw has quit IRC | 04:04 | |
| *** lukebrowning has joined #openstack-infra | 04:05 | |
| *** boris_42_ has joined #openstack-infra | 04:06 | |
| boris_42_ | anybody here?) | 04:06 |
| *** lukebrowning has quit IRC | 04:09 | |
| *** yamamoto has joined #openstack-infra | 04:10 | |
| *** lukebrowning has joined #openstack-infra | 04:11 | |
| *** dave-mccowan has joined #openstack-infra | 04:11 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** ykarel has joined #openstack-infra | 04:15 | |
| *** lukebrowning has quit IRC | 04:15 | |
| *** SumitNaiksatam has quit IRC | 04:16 | |
| *** bobh has joined #openstack-infra | 04:16 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:17 | |
| *** lukebrowning has joined #openstack-infra | 04:17 | |
| *** bobh has quit IRC | 04:21 | |
| *** lukebrowning has quit IRC | 04:22 | |
| *** lukebrowning has joined #openstack-infra | 04:23 | |
| *** lukebrowning has quit IRC | 04:28 | |
| *** dave-mccowan has quit IRC | 04:28 | |
| *** lukebrowning has joined #openstack-infra | 04:30 | |
| *** esberglu has joined #openstack-infra | 04:33 | |
| *** lukebrowning has quit IRC | 04:34 | |
| *** lukebrowning has joined #openstack-infra | 04:35 | |
| *** esberglu has quit IRC | 04:37 | |
| *** markvoelker has joined #openstack-infra | 04:38 | |
| *** lukebrowning has quit IRC | 04:40 | |
| *** lukebrowning has joined #openstack-infra | 04:41 | |
| *** lnxnut has joined #openstack-infra | 04:41 | |
| *** Goneri has joined #openstack-infra | 04:43 | |
| *** lukebrowning_ has joined #openstack-infra | 04:43 | |
| *** lukebrowning has quit IRC | 04:45 | |
| *** lukebrowning has joined #openstack-infra | 04:46 | |
| *** lukebrow_ has joined #openstack-infra | 04:48 | |
| *** lukebrowning_ has quit IRC | 04:48 | |
| *** lukebrowning has quit IRC | 04:51 | |
| *** lukebrow_ has quit IRC | 04:52 | |
| *** lukebrowning has joined #openstack-infra | 04:54 | |
| *** lukebrowning has quit IRC | 04:59 | |
| *** eranrom_ has quit IRC | 05:03 | |
| *** lukebrowning has joined #openstack-infra | 05:03 | |
| *** xarses has joined #openstack-infra | 05:04 | |
| *** xarses has quit IRC | 05:05 | |
| *** lnxnut has quit IRC | 05:05 | |
| *** xarses has joined #openstack-infra | 05:05 | |
| *** andreww has quit IRC | 05:07 | |
| *** xarses_ has joined #openstack-infra | 05:08 | |
| *** lukebrowning has quit IRC | 05:08 | |
| *** yamamoto has joined #openstack-infra | 05:11 | |
| *** lukebrowning has joined #openstack-infra | 05:11 | |
| *** xarses has quit IRC | 05:11 | |
| *** markvoelker has quit IRC | 05:11 | |
| *** lukebrowning has quit IRC | 05:15 | |
| *** lukebrowning has joined #openstack-infra | 05:16 | |
| *** yamamoto has quit IRC | 05:16 | |
| *** bobh has joined #openstack-infra | 05:17 | |
| *** lukebrowning has quit IRC | 05:21 | |
| *** lukebrowning has joined #openstack-infra | 05:22 | |
| *** bobh has quit IRC | 05:22 | |
| *** bstinson has quit IRC | 05:23 | |
| *** lukebrowning has quit IRC | 05:27 | |
| *** esberglu has joined #openstack-infra | 05:27 | |
| *** esberglu has quit IRC | 05:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/shade master: Consume publish-openstack-sphinx-docs https://review.openstack.org/508722 | 05:32 |
| AJaeger | mordred: https://review.openstack.org/508722 is needed for your sphinx publishing changes in shade, once that is in, we can merge https://review.openstack.org/#/c/508689/ | 05:32 |
| *** Goneri has quit IRC | 05:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove old publish-openstack-python-docs jobs https://review.openstack.org/508689 | 05:33 |
| *** lukebrowning has joined #openstack-infra | 05:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix typo in comment https://review.openstack.org/508697 | 05:33 |
| *** caphrim007 has joined #openstack-infra | 05:34 | |
| *** bstinson has joined #openstack-infra | 05:36 | |
| *** lukebrowning_ has joined #openstack-infra | 05:37 | |
| *** lukebrowning has quit IRC | 05:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't run d-g against all hosts in multinode jobs https://review.openstack.org/508711 | 05:38 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set up firewall for legacy multinode jobs https://review.openstack.org/508710 | 05:38 |
| *** caphrim007 has quit IRC | 05:38 | |
| *** edmondsw has joined #openstack-infra | 05:39 | |
| AJaeger | mordred, fungi, jeblair, clarkb : Yeah, infra-manual is publishing again! Woot! | 05:40 |
| *** lukebrowning has joined #openstack-infra | 05:40 | |
| *** lukebrowning_ has quit IRC | 05:42 | |
| *** edmondsw has quit IRC | 05:44 | |
| *** askb has quit IRC | 05:44 | |
| AJaeger | could I get a +2A on https://review.openstack.org/#/c/508489/ , please? | 05:46 |
| *** bstinson has quit IRC | 05:48 | |
| *** lukebrowning has quit IRC | 05:49 | |
| *** bstinson has joined #openstack-infra | 05:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-puppet-openstack-integration jobs https://review.openstack.org/508388 | 05:52 |
| *** lukebrowning has joined #openstack-infra | 05:52 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add backwards-compat playbook for /etc/nodepool/provider https://review.openstack.org/508714 | 05:54 |
| *** lukebrowning has quit IRC | 05:57 | |
| *** lukebrowning has joined #openstack-infra | 05:58 | |
| *** askb has joined #openstack-infra | 06:00 | |
| *** lukebrowning has quit IRC | 06:03 | |
| *** lukebrowning has joined #openstack-infra | 06:04 | |
| *** markvoelker has joined #openstack-infra | 06:08 | |
| *** lukebrowning has quit IRC | 06:08 | |
| *** lukebrowning has joined #openstack-infra | 06:09 | |
| *** yamamoto has joined #openstack-infra | 06:12 | |
| *** lukebrowning_ has joined #openstack-infra | 06:13 | |
| *** xarses_ has quit IRC | 06:14 | |
| *** lukebrowning has quit IRC | 06:14 | |
| *** xarses_ has joined #openstack-infra | 06:14 | |
| *** lukebrowning_ has quit IRC | 06:17 | |
| *** yamamoto has quit IRC | 06:18 | |
| *** bobh has joined #openstack-infra | 06:18 | |
| *** lukebrowning has joined #openstack-infra | 06:19 | |
| *** esberglu has joined #openstack-infra | 06:21 | |
| *** bobh has quit IRC | 06:23 | |
| *** lukebrowning has quit IRC | 06:24 | |
| *** esberglu has quit IRC | 06:26 | |
| *** lukebrowning has joined #openstack-infra | 06:26 | |
| *** lukebrowning has quit IRC | 06:31 | |
| *** lukebrowning has joined #openstack-infra | 06:32 | |
| *** lukebrowning_ has joined #openstack-infra | 06:34 | |
| *** lukebrowning has quit IRC | 06:36 | |
| *** lnxnut has joined #openstack-infra | 06:37 | |
| *** lukebrowning_ has quit IRC | 06:39 | |
| boris_42_ | AJaeger: hi there | 06:40 |
| boris_42_ | AJaeger: could you please take a look https://review.openstack.org/#/c/508630/ this should potentially unblock rally gates.. | 06:41 |
| *** markvoelker has quit IRC | 06:41 | |
| *** lukebrowning has joined #openstack-infra | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| *** lukebrowning_ has joined #openstack-infra | 06:45 | |
| *** lukebrowning has quit IRC | 06:47 | |
| *** lukebrowning has joined #openstack-infra | 06:49 | |
| *** slaweq has quit IRC | 06:49 | |
| *** lukebrowning_ has quit IRC | 06:50 | |
| *** lukebrowning_ has joined #openstack-infra | 06:51 | |
| *** lukebrowning has quit IRC | 06:53 | |
| *** bhavik1 has joined #openstack-infra | 06:54 | |
| *** lukebrowning_ has quit IRC | 06:56 | |
| *** lnxnut has quit IRC | 06:58 | |
| *** bhavik1 has quit IRC | 06:59 | |
| *** lukebrowning has joined #openstack-infra | 07:04 | |
| *** lukebrowning has quit IRC | 07:08 | |
| *** lukebrowning has joined #openstack-infra | 07:10 | |
| *** lukebrowning_ has joined #openstack-infra | 07:13 | |
| *** lukebrowning has quit IRC | 07:14 | |
| *** yamamoto has joined #openstack-infra | 07:15 | |
| *** esberglu has joined #openstack-infra | 07:16 | |
| *** lukebrowning_ has quit IRC | 07:18 | |
| *** lukebrowning has joined #openstack-infra | 07:19 | |
| *** yamamoto has quit IRC | 07:20 | |
| *** esberglu has quit IRC | 07:20 | |
| *** lukebrowning_ has joined #openstack-infra | 07:22 | |
| *** lukebrowning has quit IRC | 07:24 | |
| *** lukebrowning_ has quit IRC | 07:26 | |
| *** edmondsw has joined #openstack-infra | 07:27 | |
| *** isaacb has joined #openstack-infra | 07:31 | |
| *** edmondsw has quit IRC | 07:32 | |
| *** lukebrowning has joined #openstack-infra | 07:35 | |
| *** caphrim007 has joined #openstack-infra | 07:35 | |
| *** markvoelker has joined #openstack-infra | 07:39 | |
| *** caphrim007 has quit IRC | 07:39 | |
| *** kiennt26 has joined #openstack-infra | 07:43 | |
| *** kiennt26 has quit IRC | 07:43 | |
| *** lukebrowning has quit IRC | 07:45 | |
| *** lukebrowning has joined #openstack-infra | 07:48 | |
| *** lukebrowning_ has joined #openstack-infra | 07:52 | |
| *** lukebrowning has quit IRC | 07:53 | |
| *** lukebrowning has joined #openstack-infra | 07:55 | |
| *** lukebrowning_ has quit IRC | 07:57 | |
| *** lukebrowning has quit IRC | 07:59 | |
| *** lukebrowning has joined #openstack-infra | 08:01 | |
| *** yamamoto has joined #openstack-infra | 08:01 | |
| *** lukebrowning has quit IRC | 08:06 | |
| *** yamamoto has quit IRC | 08:06 | |
| *** lukebrowning has joined #openstack-infra | 08:07 | |
| frickler | infra-root: starting to see timeouts and slowness from review.o.o, seems its time for a gerrit restart again | 08:09 |
| *** esberglu has joined #openstack-infra | 08:09 | |
| *** yamamoto has joined #openstack-infra | 08:11 | |
| *** lukebrowning has quit IRC | 08:12 | |
| *** markvoelker has quit IRC | 08:12 | |
| *** yamamoto has quit IRC | 08:12 | |
| *** lukebrowning has joined #openstack-infra | 08:14 | |
| *** esberglu has quit IRC | 08:14 | |
| *** slaweq has joined #openstack-infra | 08:17 | |
| *** lukebrowning has quit IRC | 08:18 | |
| *** kiennt26 has joined #openstack-infra | 08:19 | |
| *** bobh has joined #openstack-infra | 08:20 | |
| *** bobh has quit IRC | 08:24 | |
| *** lnxnut has joined #openstack-infra | 08:25 | |
| *** lukebrowning has joined #openstack-infra | 08:31 | |
| *** lukebrowning has quit IRC | 08:35 | |
| *** lukebrowning has joined #openstack-infra | 08:37 | |
| *** lukebrowning has quit IRC | 08:41 | |
| *** yamamoto has joined #openstack-infra | 08:43 | |
| *** lukebrowning has joined #openstack-infra | 08:43 | |
| *** lukebrowning has quit IRC | 08:47 | |
| *** yamamoto has quit IRC | 08:47 | |
| *** lukebrowning has joined #openstack-infra | 08:49 | |
| *** lnxnut has quit IRC | 08:51 | |
| *** rtjure has quit IRC | 08:52 | |
| *** lukebrowning has quit IRC | 08:54 | |
| *** lukebrowning has joined #openstack-infra | 08:54 | |
| *** lukebrowning has quit IRC | 08:58 | |
| dimak | hey | 09:01 |
| dimak | gerrit is super slow today, anyone reported any issues? | 09:02 |
| *** lukebrowning has joined #openstack-infra | 09:03 | |
| *** esberglu has joined #openstack-infra | 09:04 | |
| *** lukebrowning has quit IRC | 09:07 | |
| *** esberglu has quit IRC | 09:08 | |
| *** lukebrowning has joined #openstack-infra | 09:09 | |
| *** lukebrowning has quit IRC | 09:14 | |
| *** askb has quit IRC | 09:14 | |
| *** rtjure has joined #openstack-infra | 09:15 | |
| *** edmondsw has joined #openstack-infra | 09:15 | |
| *** lukebrowning has joined #openstack-infra | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:16 | |
| *** lukebrowning_ has joined #openstack-infra | 09:18 | |
| *** edmondsw has quit IRC | 09:20 | |
| *** yee379 has quit IRC | 09:20 | |
| *** yamamoto has quit IRC | 09:20 | |
| *** lukebrowning has quit IRC | 09:20 | |
| *** yee379 has joined #openstack-infra | 09:20 | |
| *** bobh has joined #openstack-infra | 09:20 | |
| *** lukebrowning has joined #openstack-infra | 09:21 | |
| *** lukebrowning_ has quit IRC | 09:22 | |
| *** yamamoto has joined #openstack-infra | 09:23 | |
| *** lukebrowning_ has joined #openstack-infra | 09:24 | |
| *** bstinson has quit IRC | 09:24 | |
| *** bobh has quit IRC | 09:25 | |
| *** lukebrowning has quit IRC | 09:25 | |
| *** ykarel has quit IRC | 09:26 | |
| *** bstinson has joined #openstack-infra | 09:27 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** lukebrowning_ has quit IRC | 09:28 | |
| *** dizquierdo has joined #openstack-infra | 09:29 | |
| *** lukebrowning has joined #openstack-infra | 09:30 | |
| *** gongysh has joined #openstack-infra | 09:33 | |
| *** lukebrowning_ has joined #openstack-infra | 09:33 | |
| *** gongysh has quit IRC | 09:33 | |
| *** tosky has joined #openstack-infra | 09:33 | |
| *** lukebrowning has quit IRC | 09:34 | |
| *** gongysh has joined #openstack-infra | 09:35 | |
| *** gongysh has quit IRC | 09:35 | |
| *** caphrim007 has joined #openstack-infra | 09:35 | |
| *** lukebrowning_ has quit IRC | 09:38 | |
| *** slaweq has quit IRC | 09:38 | |
| *** gongysh has joined #openstack-infra | 09:39 | |
| *** lukebrowning has joined #openstack-infra | 09:40 | |
| *** caphrim007 has quit IRC | 09:40 | |
| *** lukebrowning has quit IRC | 09:44 | |
| *** psachin has joined #openstack-infra | 09:45 | |
| *** lukebrowning has joined #openstack-infra | 09:46 | |
| *** gridinv has joined #openstack-infra | 09:48 | |
| *** lukebrowning has quit IRC | 09:50 | |
| *** lukebrowning_ has joined #openstack-infra | 09:50 | |
| *** gridinv has left #openstack-infra | 09:51 | |
| *** lukebrowning_ has quit IRC | 09:55 | |
| *** lukebrowning has joined #openstack-infra | 09:56 | |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** esberglu has joined #openstack-infra | 09:58 | |
| *** lukebrowning has quit IRC | 10:00 | |
| *** esberglu has quit IRC | 10:02 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** lukebrowning has joined #openstack-infra | 10:03 | |
| *** esberglu has joined #openstack-infra | 10:03 | |
| *** lukebrowning has quit IRC | 10:07 | |
| *** esberglu has quit IRC | 10:08 | |
| *** markvoelker has joined #openstack-infra | 10:09 | |
| *** lukebrowning has joined #openstack-infra | 10:09 | |
| *** cshastri has quit IRC | 10:10 | |
| *** lukebrowning has quit IRC | 10:14 | |
| *** dizquierdo has quit IRC | 10:14 | |
| *** lukebrowning has joined #openstack-infra | 10:14 | |
| *** lukebrowning has quit IRC | 10:18 | |
| *** bobh has joined #openstack-infra | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:21 | |
| *** lnxnut has joined #openstack-infra | 10:22 | |
| openstackgerrit | Jan Zerebecki proposed openstack-infra/infra-manual master: Correct link of playbooks/legacy https://review.openstack.org/508726 | 10:23 |
| *** bobh has quit IRC | 10:26 | |
| *** lukebrowning has joined #openstack-infra | 10:27 | |
| *** yamamoto has quit IRC | 10:29 | |
| *** lukebrowning has quit IRC | 10:32 | |
| *** lukebrowning has joined #openstack-infra | 10:39 | |
| *** claudiub has joined #openstack-infra | 10:39 | |
| *** psachin has quit IRC | 10:39 | |
| *** lukebrowning_ has joined #openstack-infra | 10:42 | |
| *** inc0 has quit IRC | 10:42 | |
| *** markvoelker has quit IRC | 10:42 | |
| *** inc0 has joined #openstack-infra | 10:42 | |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** yamamoto has quit IRC | 10:42 | |
| *** lukebrowning has quit IRC | 10:43 | |
| *** lukebrowning_ has quit IRC | 10:44 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Correct link of playbooks/legacy https://review.openstack.org/508726 | 10:44 |
| *** lnxnut has quit IRC | 10:45 | |
| *** lukebrowning has joined #openstack-infra | 10:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Switch to legacy puppet check jobs https://review.openstack.org/508373 | 10:45 |
| *** yamamoto has joined #openstack-infra | 10:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove python-aodhclient as g-r.txt has aodhclient https://review.openstack.org/508626 | 10:46 |
| *** pbourke has quit IRC | 10:49 | |
| *** lukebrowning has quit IRC | 10:49 | |
| *** yamamoto has quit IRC | 10:50 | |
| *** pbourke has joined #openstack-infra | 10:50 | |
| *** lukebrowning has joined #openstack-infra | 10:51 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove some pypy jobs that don't work https://review.openstack.org/504748 | 10:53 |
| *** lukebrowning_ has joined #openstack-infra | 10:56 | |
| *** lukebrowning has quit IRC | 10:56 | |
| *** kiennt26 has quit IRC | 10:56 | |
| *** psachin has joined #openstack-infra | 10:57 | |
| *** esberglu has joined #openstack-infra | 10:58 | |
| *** lukebrowning_ has quit IRC | 11:00 | |
| *** lukebrowning has joined #openstack-infra | 11:00 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Only run tox-siblings on required projects https://review.openstack.org/507955 | 11:01 |
| *** esberglu has quit IRC | 11:02 | |
| *** edmondsw has joined #openstack-infra | 11:03 | |
| *** yamamoto has quit IRC | 11:04 | |
| *** slaweq has joined #openstack-infra | 11:05 | |
| *** lukebrowning has quit IRC | 11:05 | |
| *** lukebrowning has joined #openstack-infra | 11:05 | |
| *** edmondsw has quit IRC | 11:08 | |
| *** lukebrowning_ has joined #openstack-infra | 11:08 | |
| *** lukebrowning has quit IRC | 11:09 | |
| *** lukebrowning has joined #openstack-infra | 11:11 | |
| *** lnxnut has joined #openstack-infra | 11:12 | |
| *** lukebrowning_ has quit IRC | 11:12 | |
| *** lukebrowning_ has joined #openstack-infra | 11:15 | |
| *** slaweq has quit IRC | 11:15 | |
| *** lukebrowning has quit IRC | 11:15 | |
| *** lnxnut has quit IRC | 11:16 | |
| *** lukebrowning_ has quit IRC | 11:19 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** lukebrowning has joined #openstack-infra | 11:21 | |
| *** bobh has joined #openstack-infra | 11:22 | |
| *** lukebrowning has quit IRC | 11:26 | |
| *** bobh has quit IRC | 11:27 | |
| *** lukebrowning has joined #openstack-infra | 11:27 | |
| *** lukebrowning_ has joined #openstack-infra | 11:29 | |
| *** lukebrow_ has joined #openstack-infra | 11:31 | |
| *** lukebrowning has quit IRC | 11:31 | |
| *** lukebrowning_ has quit IRC | 11:34 | |
| *** lukebrow_ has quit IRC | 11:36 | |
| *** caphrim007 has joined #openstack-infra | 11:37 | |
| *** lukebrowning has joined #openstack-infra | 11:38 | |
| *** markvoelker has joined #openstack-infra | 11:39 | |
| *** lukebrowning_ has joined #openstack-infra | 11:40 | |
| *** slaweq has joined #openstack-infra | 11:41 | |
| *** caphrim007 has quit IRC | 11:41 | |
| *** lukebrowning has quit IRC | 11:42 | |
| *** slaweq has quit IRC | 11:45 | |
| *** lukebrowning_ has quit IRC | 11:45 | |
| *** lukebrowning has joined #openstack-infra | 11:46 | |
| *** lukebrowning has quit IRC | 11:51 | |
| *** lukebrowning has joined #openstack-infra | 11:53 | |
| clarkb | there was definitely a garbage collection event on review.o.o but it appeas to be over now (unlike in the past when we just leaked indefinitely) | 11:55 |
| clarkb | the service is working for me now. I'd like to keep it up as long as possible to see if the service self corrects | 11:55 |
| *** lukebrowning has quit IRC | 11:58 | |
| *** lukebrowning has joined #openstack-infra | 11:58 | |
| *** rtjure has quit IRC | 11:59 | |
| *** edmondsw has joined #openstack-infra | 11:59 | |
| *** rtjure has joined #openstack-infra | 12:02 | |
| *** lukebrowning has quit IRC | 12:03 | |
| *** edmondsw has quit IRC | 12:04 | |
| *** slaweq has joined #openstack-infra | 12:04 | |
| *** lukebrowning has joined #openstack-infra | 12:05 | |
| *** lukebrowning has quit IRC | 12:10 | |
| *** slaweq has quit IRC | 12:11 | |
| *** markvoelker has quit IRC | 12:12 | |
| *** lukebrowning has joined #openstack-infra | 12:12 | |
| *** lnxnut has joined #openstack-infra | 12:15 | |
| *** lukebrowning has quit IRC | 12:16 | |
| *** lukebrowning has joined #openstack-infra | 12:18 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-releases-python35 jobs https://review.openstack.org/508730 | 12:18 |
| clarkb | smcginnis: I think there is a non legacy version of ^ that projects should be using? Curious where you find that legacy job | 12:19 |
| smcginnis | clarkb: I saw it fail on here: https://review.openstack.org/#/c/508482/ | 12:19 |
| *** yamamoto has quit IRC | 12:20 | |
| clarkb | openstack-tox-py35 is the non legacy job and runs against nova at least | 12:20 |
| clarkb | I wonder if the releases job is specialer | 12:20 |
| clarkb | like doesn't run tox? | 12:20 |
| smcginnis | clarkb: I wouldn't think so. Does that job just run tox on teh repo? | 12:20 |
| clarkb | ya openstack-tox-py35 does basically tox -e py35 on the repo with requirements | 12:21 |
| clarkb | smcginnis: I'll approve ^ but may want to look at openstack-tox-py35 too depending on what your job does | 12:21 |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| smcginnis | clarkb: What's the plan with all these legacy jobs? | 12:21 |
| *** yamamoto has quit IRC | 12:22 | |
| clarkb | smcginnis: projects should migrate away from them over time. Likely a mix of infra, qa, and projects themselves doing the migration | 12:22 |
| *** lrossetti has joined #openstack-infra | 12:22 | |
| *** lukebrowning has quit IRC | 12:23 | |
| *** nicolasbock has joined #openstack-infra | 12:23 | |
| smcginnis | clarkb: And what's really the difference between the "legacy" ones and these new ones? | 12:23 |
| *** bobh has joined #openstack-infra | 12:23 | |
| clarkb | smcginnis: the legacy jobs are direct ports of the jenkins job builder configs. So we tell ansible to run shell and check the return code for hte most part | 12:24 |
| *** lukebrowning has joined #openstack-infra | 12:24 | |
| clarkb | non legacy jobs can/should in many cases live in the project itself and be part of the project repo rather than something infra oversees | 12:24 |
| *** nicolasbock_ has joined #openstack-infra | 12:24 | |
| smcginnis | clarkb: Ah, ok. So the non-legacy ones are pure ansible? | 12:24 |
| clarkb | smcginnis: not necessarily pure ansible but far more ansible aware (I am sure we still run shell in them in places) | 12:25 |
| smcginnis | clarkb: Any examples of any legacy jobs that have been migrated yet? | 12:25 |
| clarkb | smcginnis: yes! mnaser has apparnetly done all of the puppet openstack jobs | 12:25 |
| clarkb | and then infra did the openstack-tox-* jobs | 12:25 |
| smcginnis | Cool, I'll look for those for examples. | 12:26 |
| clarkb | smcginnis: really where it will make a noticeable difference is in multinode testing reaplcement of the "base" tempest/devstack jobs is still a work in progress but you can see the starts of that in devstack-gate | 12:26 |
| *** bobh has quit IRC | 12:28 | |
| *** nicolasbock has quit IRC | 12:28 | |
| *** lukebrowning has quit IRC | 12:29 | |
| *** lukebrowning has joined #openstack-infra | 12:30 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 12:31 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-releases-python35 jobs https://review.openstack.org/508730 | 12:32 |
| *** lukebrowning has quit IRC | 12:35 | |
| *** lukebrowning has joined #openstack-infra | 12:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Fix cinder mn-sub mulitnode jobs https://review.openstack.org/508731 | 12:36 |
| clarkb | fungi: thanks for the approve on the d-g branch fix | 12:37 |
| clarkb | change above should fix some cinderthings | 12:37 |
| *** lukebrowning_ has joined #openstack-infra | 12:37 | |
| *** lnxnut has quit IRC | 12:38 | |
| * clarkb works up a tripleo fix like that too | 12:38 | |
| *** lukebrowning has quit IRC | 12:39 | |
| *** lukebrowning_ has quit IRC | 12:41 | |
| *** esberglu has joined #openstack-infra | 12:46 | |
| *** lukebrowning has joined #openstack-infra | 12:50 | |
| *** esberglu has quit IRC | 12:50 | |
| mnaser | https://review.openstack.org/#/c/508390/ | 12:50 |
| mnaser | does anyone know why that keeps getting in a merge conflict | 12:51 |
| mnaser | i rebased off master and therer is no changes | 12:53 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Fix tripleo multinode job defs https://review.openstack.org/508732 | 12:53 |
| clarkb | mnaser: I was goin to say maybe the depends on but gerrit says its a conflict too | 12:53 |
| clarkb | and hte depends on has merged | 12:53 |
| *** psachin has quit IRC | 12:54 | |
| mnaser | yeah... maybe try dropping the depends on and see what happens? | 12:54 |
| *** lukebrowning has quit IRC | 12:54 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Remove all legacy puppet openstack integration jobs https://review.openstack.org/508390 | 12:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: legacy-horizon-selenium-headless add requirements https://review.openstack.org/508734 | 12:55 |
| clarkb | mnaser: the parent commit is actually relatively old with all the churn going on. Quite possible it is a valid conflict. Keep in mind gerrit uses jgit which has much more conservative ideas about merging things than C git. Zuul attempts to approximate the jgit behavior using C git with particular options | 12:55 |
| *** lukebrowning has joined #openstack-infra | 12:56 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config master: Remove karborclient LIBS_FROM_GIT from Karbor gate https://review.openstack.org/508735 | 12:57 |
| *** edmondsw has joined #openstack-infra | 12:57 | |
| *** lukebrowning_ has joined #openstack-infra | 12:59 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch to new puppet-openstack-check jobs https://review.openstack.org/508736 | 12:59 |
| *** lukebrowning has quit IRC | 13:01 | |
| *** edmondsw has quit IRC | 13:02 | |
| *** lukebrowning_ has quit IRC | 13:04 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Remove all legacy puppet openstack integration jobs https://review.openstack.org/508390 | 13:04 |
| mnaser | ok, resolved them, it was my bad | 13:04 |
| *** isaacb has quit IRC | 13:04 | |
| *** isaacb has joined #openstack-infra | 13:05 | |
| *** esberglu has joined #openstack-infra | 13:07 | |
| *** markvoelker has joined #openstack-infra | 13:09 | |
| *** esberglu has quit IRC | 13:11 | |
| *** kiennt26 has joined #openstack-infra | 13:13 | |
| *** baoli has joined #openstack-infra | 13:15 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: legacy-horizon-selenium-headless add requirements https://review.openstack.org/508734 | 13:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix cinder mn-sub mulitnode jobs https://review.openstack.org/508731 | 13:19 |
| *** yamamoto has joined #openstack-infra | 13:22 | |
| *** bobh has joined #openstack-infra | 13:24 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Switch puppet-openstack-libraries to legacy parent https://review.openstack.org/508739 | 13:24 |
| *** yamamoto has quit IRC | 13:24 | |
| *** yamamoto has joined #openstack-infra | 13:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Cleanup kolla-ansible multinode jobs https://review.openstack.org/508740 | 13:26 |
| clarkb | fungi: ^ more cleanup | 13:26 |
| *** yamamoto has quit IRC | 13:27 | |
| mnaser | i know its sunday morning but https://review.openstack.org/#/c/508390/ (removes un-used integration jobs, trivial) and https://review.openstack.org/#/c/508736/ (reduces the number of jobs ran by 2 to waste less resources, removing old jobs) | 13:28 |
| *** bobh has quit IRC | 13:28 | |
| mnaser | this one doesn't have a +1 yet but also changes the parent of a job that was missed https://review.openstack.org/#/c/508739/ | 13:29 |
| *** dave-mccowan has joined #openstack-infra | 13:30 | |
| *** lrossetti_ has joined #openstack-infra | 13:30 | |
| *** lrossetti has quit IRC | 13:33 | |
| frickler | the links to commits on post jobs seem broken https://review.openstack.org/gitweb?p=openstack-infra/openstack-zuul-jobs.git;a=commitdiff;h=bce1964ca88b0c55d1ba9da6be117d47ff153645 | 13:37 |
| frickler | also got an error here http://logs.openstack.org/79/791fef4c3c6c6a99d50aa941cc8d0d80bf4e615a/post/publish-openstack-sphinx-docs-infra/c931175/ara/ | 13:37 |
| *** caphrim007 has joined #openstack-infra | 13:37 | |
| *** dave-mccowan has quit IRC | 13:38 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Switch release-note-jobs project-template to use new jobs https://review.openstack.org/508742 | 13:38 |
| mnaser | ERROR: toxini file 'tox.ini' not found | 13:39 |
| clarkb | frickler: we switched to cgit with the gerrit upgrade, likely the cause of the post job commit links being broken | 13:40 |
| *** isaacb has quit IRC | 13:40 | |
| clarkb | frickler: mnaser ya vmware nsx ran into that too (tox.ini missing) I think the dir must be wrong on that task | 13:40 |
| clarkb | (but am trying to finish up my train of thought on multinode things before digging into that) | 13:41 |
| mnaser | yeah im not sure i just found that error and thats as much as i know :p | 13:41 |
| *** esberglu has joined #openstack-infra | 13:42 | |
| *** caphrim007 has quit IRC | 13:42 | |
| *** markvoelker has quit IRC | 13:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Fix use of hostvars for writing provider file https://review.openstack.org/508743 | 13:45 |
| clarkb | dmsimard: ^ fyi that fixes a small issue in the /etc/nodepool/provider file generation | 13:45 |
| clarkb | ( I think ) | 13:45 |
| *** esberglu has quit IRC | 13:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 13:46 |
| mnaser | if i wanted to create a 'rake' role similar to the tox one, would zuul-jobs be the ideal spot (given its much more generic?) or should i stick to openstack-zuul-jobs | 13:46 |
| clarkb | mnaser: I think zuul-jobs makes sense, we just have to be careful not to bake openstackisms in there | 13:47 |
| mnaser | yeah i'll look at the tox one and adapt to it | 13:47 |
| clarkb | mordred: is it possible the console log streaming is at least partially to blame for the memory usage? | 13:50 |
| clarkb | thinking that maybe putting the redirect in placemight be what precipitated the path to OOM? I'd have to go examine timelines better though | 13:50 |
| *** isaacb has joined #openstack-infra | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:56 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Add rake role to run specific rake tasks https://review.openstack.org/508745 | 13:57 |
| fungi | clarkb: wouldn't that affect the zuul-web service rather than zuul-scheduler? | 14:00 |
| clarkb | oh right | 14:01 |
| clarkb | they are on the same host but zuul-web didn't get targeted | 14:01 |
| fungi | [Fri Sep 29 22:07:20 2017] Killed process 23406 (zuul-scheduler) total-vm:18389268kB, anon-rss:14652884kB, file-rss:0kB | 14:01 |
| fungi | i think that means zuul-web wasn't using much (if any) | 14:02 |
| clarkb | ya, definitely the scheduler that was targeted for using all the memory | 14:02 |
| fungi | [Fri Sep 29 22:07:20 2017] [30870] 3000 30870 188987 16255 107 4 0 0 zuul-web | 14:02 |
| fungi | field order there is: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name | 14:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 14:03 |
| fungi | so memory utilization by zuul-web was mostly negligible at the time of the oom | 14:04 |
| *** lnxnut has joined #openstack-infra | 14:05 | |
| clarkb | https://review.openstack.org/#/c/507071/ shows we need to clear out jenkins votes with zuul too | 14:05 |
| *** mriedem has joined #openstack-infra | 14:05 | |
| mnaser | but does zuul-scheduler perhaps buffer the content of the console in memory? | 14:05 |
| mnaser | or does it store it on disk | 14:06 |
| *** leyal has joined #openstack-infra | 14:06 | |
| fungi | clarkb: just -2 right? we can probably just query for all open changes with a verify -2 and unset those votes en masse | 14:06 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/elastic-recheck master: Fix check for console log https://review.openstack.org/508746 | 14:06 |
| clarkb | oh hrm I thought I could have zuul do it trivially but it assumes its own user on the voting in pipelines | 14:06 |
| clarkb | fungi: so ya that may be the easiest option | 14:06 |
| fungi | clarkb: oh! or... | 14:08 |
| fungi | remember our discussion about removing jenkins from the ci group in gerrit? | 14:08 |
| fungi | and how we might want to avoid that because it would hide all its verify votes on open changes? | 14:09 |
| clarkb | does hiding them prevent Jenkins from preventing merges? | 14:09 |
| fungi | i believe so, yes | 14:09 |
| clarkb | though really we only want to get rid of the -2s I think. The other votes are potentially useful | 14:10 |
| fungi | aiui, gerrit will ignore them in the db and filter them from webui/api results | 14:10 |
| clarkb | this requires thinking beyond what I am capable of at 7am after getting up at 4am to smoke brisket | 14:10 |
| fungi | we _could_ move jenkins to a group which is allowed global verify -1..+1 but not -2 | 14:10 |
| fungi | and yeah, i too need to go tend to the grill | 14:11 |
| clarkb | http://zuulv3.openstack.org/static/stream.html?uuid=ca7aeb25166c40febcbcd15295c9b7a9&logfile=console.log is my check if all the tripleo fixes make things happy job | 14:11 |
| fungi | though no brisket, today is strip steak | 14:11 |
| fungi | mordred: for the zuul-cloner error message you added, we wanted it to show the project name instead of the path to it on disk, right? http://logs.openstack.org/18/498018/20/check/legacy-horizon-selenium-headless/ff26a1c/job-output.txt.gz#_2017-10-01_06_30_10_711909 | 14:13 |
| fungi | i can work up a patch, i think the error as it stands is still mildly confusing | 14:13 |
| mordred | fungi: ah - yes. whoops | 14:13 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Add rake role to run specific rake tasks https://review.openstack.org/508745 | 14:14 |
| mnaser | gah | 14:14 |
| mnaser | sorry for spam forgot to git add | 14:14 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Add rake role to run specific rake tasks https://review.openstack.org/508745 | 14:14 |
| *** lukebrowning has joined #openstack-infra | 14:15 | |
| *** ykarel has joined #openstack-infra | 14:16 | |
| *** kiennt26 has quit IRC | 14:19 | |
| *** lukebrowning_ has joined #openstack-infra | 14:19 | |
| *** lukebrowning has quit IRC | 14:20 | |
| *** lukebrowning has joined #openstack-infra | 14:21 | |
| *** atarakt_ has joined #openstack-infra | 14:22 | |
| *** atarakt_ has quit IRC | 14:22 | |
| *** lukebrow_ has joined #openstack-infra | 14:23 | |
| *** lukebrowning_ has quit IRC | 14:23 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Correct cloner shim missing project error https://review.openstack.org/508747 | 14:24 |
| *** bobh has joined #openstack-infra | 14:25 | |
| fungi | mordred: ^ that should hopefully dtrt | 14:25 |
| *** lukebrowning has quit IRC | 14:25 | |
| jeblair | clarkb, smcginnis: another important aspect of the conversion from legacy to v3 is awareness of zuul features -- for instance, using the prepared repos on the worker, and removing branch specifc jobs. for instance, we shouldn't need any dedicated periodic jobs anymore because any job can now be used as a periodic job because of the way zuul triggers it. | 14:26 |
| openstackgerrit | Merged openstack-infra/shade master: Consume publish-openstack-sphinx-docs https://review.openstack.org/508722 | 14:27 |
| jeblair | clarkb, smcginnis: so, for instance, we collapsed something like 6,000 python27 + periodic-python27 jobs into one: openstack-tox-py27. | 14:27 |
| *** lukebrowning has joined #openstack-infra | 14:27 | |
| *** lukebrow_ has quit IRC | 14:27 | |
| mordred | clarkb, smcginnis: also - the releases job does do more than just run tox - there's a big pile of logic around translations | 14:28 |
| jeblair | clarkb: reviewed "topic:fix-multinode" mordred if you have a sec, there are 2 changes there that could use your +3 | 14:29 |
| clarkb | openstackgerrit: aha | 14:29 |
| *** bobh has quit IRC | 14:29 | |
| clarkb | er | 14:29 |
| clarkb | mordred: ^ | 14:29 |
| clarkb | jeblair: mordred also https://review.openstack.org/508743 | 14:29 |
| clarkb | this one seems to work and fix the problem but tripleo jobs now running into the /etc/nodepool/sub_nodes file missing | 14:30 |
| *** lukebrowning_ has joined #openstack-infra | 14:30 | |
| *** lukebrowning has quit IRC | 14:32 | |
| jeblair | clarkb: oh, -1 on that one | 14:32 |
| clarkb | fungi: I went ahead and removed the jenkins -2 from that devstack change and rechecked it. That seems like a simple fix for thatspecific case for now | 14:32 |
| *** lnxnut has quit IRC | 14:32 | |
| jeblair | mordred: double check me on the -1 on 508743 pls | 14:32 |
| clarkb | jeblair: so drop use of hostvars entirely? | 14:33 |
| mordred | jeblair, clarkb: yes! what jeblair said | 14:33 |
| fungi | clarkb: agreed, but it does involve one of us (gerrit admins) doing that every time so may still want a mass solution | 14:33 |
| jeblair | clarkb: ya, i think as long as you're asking for the vars for *this* host you don't need it. that form is necessary only for vars for *another* host | 14:33 |
| mordred | yes. that is correct | 14:33 |
| clarkb | gotcha | 14:34 |
| *** lukebrowning has joined #openstack-infra | 14:34 | |
| * clarkb pdates | 14:34 | |
| *** lukebrowning_ has quit IRC | 14:34 | |
| mordred | I have found an error in generated project config that we haven't run in to yet but I believe will screw us ... | 14:34 |
| mordred | - publish-openstack-python-tarball: | 14:34 |
| mordred | dependencies: | 14:34 |
| mordred | - publish-openstack-python-tarball | 14:34 |
| mordred | whoops | 14:34 |
| jeblair | ya. that should throw an exception during job freezing and cause no jobs to run when the item is enqueued. won't be detected until then though. | 14:36 |
| *** esberglu has joined #openstack-infra | 14:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Fix use of hostvars for writing provider file https://review.openstack.org/508743 | 14:36 |
| *** lukebrowning_ has joined #openstack-infra | 14:37 | |
| mordred | there's a whole big pile of mess in release pipelines related to that - which can be completely replaced with tempalte: publish-to-pypi | 14:37 |
| mordred | I'll try to work up a script to fix it / patch | 14:37 |
| *** lukebrowning has quit IRC | 14:38 | |
| clarkb | jeblair: mordred is /etc/nodepool/id_rsa still a thing? | 14:39 |
| clarkb | I'm looking at fixing more tripleo things | 14:39 |
| clarkb | and unsure if we put that file in place | 14:39 |
| *** markvoelker has joined #openstack-infra | 14:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove legacy jobs from shade https://review.openstack.org/508749 | 14:40 |
| *** esberglu has quit IRC | 14:40 | |
| jeblair | clarkb: yeah, should be done by the compat stuff we did for devstack | 14:41 |
| *** lukebrowning_ has quit IRC | 14:41 | |
| mnaser | if i make a depends-on patch for https://review.openstack.org/#/c/508745/ .. will it be able to inherit those tasks? | 14:42 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Cleanup kolla-ansible multinode jobs https://review.openstack.org/508740 | 14:43 |
| jeblair | mnaser: yes | 14:43 |
| mnaser | ok so the answer for every zuul question is yes | 14:43 |
| jeblair | mnaser: "yes; except for project-config" :) | 14:43 |
| mnaser | (the only time i tried something without asking, it wasn't a yes, which was depends-on project-config) | 14:43 |
| mnaser | haha | 14:43 |
| *** lukebrowning has joined #openstack-infra | 14:43 | |
| *** lukebrowning_ has joined #openstack-infra | 14:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Switch to using publish-openstack-sphinx-docs-infra https://review.openstack.org/508751 | 14:46 |
| mordred | jeblair, clarkb: mind +Aing that ^^ ? | 14:46 |
| *** atarakt_ has joined #openstack-infra | 14:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/tripleo-ci master: Use sub_nodes_private https://review.openstack.org/508752 | 14:48 |
| *** lukebrowning has quit IRC | 14:48 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix tripleo multinode job defs https://review.openstack.org/508732 | 14:48 |
| clarkb | mordred: +2'd will let jeblair take a look if interested | 14:49 |
| clarkb | mordred: jeblair can you rereview https://review.openstack.org/508743 | 14:49 |
| *** lukebrowning has joined #openstack-infra | 14:49 | |
| *** lukebrowning_ has quit IRC | 14:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add objgraph to SIGUSR2 https://review.openstack.org/508753 | 14:51 |
| jeblair | clarkb, mordred: ^ would like to have that in place before next restart | 14:51 |
| mordred | jeblair: +2 | 14:52 |
| jeblair | (there is a way to do that with a running process with gdb, but i'm pretty sure if we did that now, we'd kick zuul off of zookeeper and we'd make a big mess, so maybe not worth it) | 14:53 |
| clarkb | jeblair: you'll just separately install that dep? | 14:53 |
| jeblair | clarkb: ya | 14:53 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 14:54 |
| jeblair | clarkb: https://review.openstack.org/508743 +2 with question | 14:54 |
| *** lukebrowning_ has joined #openstack-infra | 14:54 | |
| *** lukebrowning has quit IRC | 14:54 | |
| clarkb | jeblair: I thought about it but the hostvar is still the generic method to get the data if someone sees that in their job and want to update their jobs. | 14:55 |
| clarkb | (But I'm not sure what the best way to describe these things in an ansibly manner is) | 14:55 |
| *** lukebrowning_ has quit IRC | 14:59 | |
| *** lrossetti_ has quit IRC | 15:00 | |
| *** lukebrowning has joined #openstack-infra | 15:01 | |
| mnaser | question, is the merge-check template still necessary to have for jobs? | 15:01 |
| clarkb | mnaser: probably best to leave it for now as we may reenable that pipeline at some point | 15:02 |
| *** yamamoto has quit IRC | 15:02 | |
| mnaser | ok cool | 15:02 |
| clarkb | (and can do a bulk removal if we commit to not running that ina zuul any longer) | 15:02 |
| mnaser | thing is we're gonna have per-repo very basic zuul file with a project-template from puppet-openstack-integration | 15:02 |
| mnaser | so like im trying to see what it should be doing if/when we bring it back, it might be easier to keep it in our main repo rather than have it as a template in every repo | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 15:03 |
| jeblair | mnaser: you could leave it in the project defn in project-config for now | 15:03 |
| jeblair | mnaser: then if we decide to keep it, move it if you want, and if we remove it, we'll be able to take care of it easily | 15:04 |
| mnaser | oh good idea | 15:04 |
| AJaeger | mordred: regarding https://review.openstack.org/#/c/508749/1 - do you have a change for openstack-zuul-jobs to remove the zuul changes from there? | 15:04 |
| mnaser | thanks! | 15:04 |
| mordred | AJaeger: coming right up | 15:04 |
| AJaeger | mordred: great | 15:05 |
| *** lukebrowning has quit IRC | 15:05 | |
| *** lukebrowning has joined #openstack-infra | 15:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove auto-converted legacy shade jobs https://review.openstack.org/508757 | 15:06 |
| mordred | AJaeger: ^^ thereyago | 15:06 |
| AJaeger | fine! | 15:06 |
| AJaeger | then let's ask for a +2A on 508749... | 15:07 |
| AJaeger | Also, could I get a +2A on https://review.openstack.org/#/c/508708/ , please? mordred already gave a +2 | 15:07 |
| dmsimard | clarkb: I guess those hostvars are equivalent ? But cleaner indeed, thanks. | 15:08 |
| *** lukebrowning_ has joined #openstack-infra | 15:09 | |
| AJaeger | mnaser: can we remove puppet-check-jobs from openstack-zuul-jobs once https://review.openstack.org/#/c/508736/1/zuul.d/projects.yaml merges? | 15:09 |
| clarkb | dmsimard: it didn't work in the old version because the hostname wasn't there to index the hostvars | 15:09 |
| mnaser | AJaeger yes, but i wanted to do that once i ran a "recheck" on my dnm change and make sure that the appropriate jobs are indeed gone and the good ones are still there | 15:10 |
| *** lukebrowning has quit IRC | 15:10 | |
| AJaeger | mnaser: ok, then I'll +2A and wait for your change.. | 15:10 |
| *** psachin has joined #openstack-infra | 15:11 | |
| mnaser | AJaeger will do, i plan to constantly audit the puppet module template/jobs to ensure we dont leave any cruft in project-config or openstack-zuul-jobs :) | 15:11 |
| dmsimard | clarkb: doh | 15:11 |
| mnaser | till we've moved stuff out | 15:11 |
| clarkb | dmsimard: https://review.openstack.org/508752 is current tip of my tripleo fixing | 15:11 |
| AJaeger | mordred: did you see the error message in https://review.openstack.org/508689 ? This needs some more work... | 15:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Export missing projects in Rally legacy jobs https://review.openstack.org/508630 | 15:12 |
| *** claudiub has quit IRC | 15:12 | |
| dmsimard | clarkb: awesome, thanks for the help. I'll take a look in a bit. | 15:12 |
| *** markvoelker has quit IRC | 15:13 | |
| *** lukebrowning_ has quit IRC | 15:13 | |
| AJaeger | mordred: looking at https://review.openstack.org/#/c/508542 - I think we still need that, don't we? That's the broken pep8 change. Let me recheck since it has no votes | 15:15 |
| *** oanson has quit IRC | 15:18 | |
| *** oanson has joined #openstack-infra | 15:18 | |
| *** lukebrowning has joined #openstack-infra | 15:20 | |
| *** lukebrowning has quit IRC | 15:24 | |
| *** bobh has joined #openstack-infra | 15:25 | |
| *** lukebrowning has joined #openstack-infra | 15:26 | |
| clarkb | I left a comment about checking before using cp -l but then managed to convince myself it would be fine for raisins | 15:27 |
| clarkb | well http://logs.openstack.org/52/508752/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet/433da97/job-output.txt.gz#_2017-10-01_15_18_06_383893 shows that I should never make assumptions about what jobs can or will do :) | 15:28 |
| clarkb | dmsimard: so I think next step is updating the z-c shim to handle cross filesystem copies | 15:28 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Check for branches in zuulv3 git repos properly https://review.openstack.org/508702 | 15:30 |
| *** bobh has quit IRC | 15:30 | |
| *** esberglu has joined #openstack-infra | 15:30 | |
| *** lukebrowning has quit IRC | 15:31 | |
| AJaeger | clarkb, mordred, fungi, if you have time to review, here's a list of changes that we should merge IMHO: https://review.openstack.org/508708/ https://review.openstack.org/508542/ https://review.openstack.org/508609/ https://review.openstack.org/508706 https://review.openstack.org/508489/ https://review.openstack.org/508749 https://review.openstack.org/504748 https://review.openstack.org/508556/ | 15:31 |
| AJaeger | https://review.openstack.org/508519/ https://review.openstack.org/508664/ Reviews welcome! | 15:31 |
| *** lbragstad has joined #openstack-infra | 15:32 | |
| *** lukebrowning has joined #openstack-infra | 15:33 | |
| *** esberglu has quit IRC | 15:34 | |
| jeblair | AJaeger, mordred: re 508542 -- will cinder still run pep8? | 15:35 |
| jeblair | (ie, is that just a local variant on the pep8 which it gets via openstack-python-jobs ?) | 15:35 |
| *** ykarel has quit IRC | 15:36 | |
| *** lukebrowning_ has joined #openstack-infra | 15:36 | |
| *** dave-mccowan has joined #openstack-infra | 15:36 | |
| *** lukebrowning has quit IRC | 15:37 | |
| AJaeger | jeblair: yes, exactly - it will run but now on all branches. But better run it on driverfixes and block those for now - alternative is running it nowhere I think. So, this needs more work. mordred, correct? | 15:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix use of hostvars for writing provider file https://review.openstack.org/508743 | 15:38 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove all legacy puppet openstack integration jobs https://review.openstack.org/508390 | 15:38 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch puppet-openstack-libraries to legacy parent https://review.openstack.org/508739 | 15:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to new puppet-openstack-check jobs https://review.openstack.org/508736 | 15:38 |
| AJaeger | thanks for review love! | 15:38 |
| *** caphrim007 has joined #openstack-infra | 15:38 | |
| *** dave-mccowan has quit IRC | 15:40 | |
| *** lukebrowning_ has quit IRC | 15:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Update retire instructions for Zuul 3 https://review.openstack.org/508706 | 15:42 |
| *** lukebrowning has joined #openstack-infra | 15:42 | |
| AJaeger | jeblair: addressed your comment -thanks ^ | 15:42 |
| *** caphrim007 has quit IRC | 15:43 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Correct cloner shim missing project error https://review.openstack.org/508747 | 15:44 |
| leyal | hi - question - if i use a common template for run my testing with tox(for example: openstack-python-jobs) - where should i add "required-projects" field ? (according to https://docs.openstack.org/infra/manual/zuulv3.html in "Changes to OpenStack tox jobs") | 15:45 |
| clarkb | leyal: you would create a job variant for your project and apply the required-projects there I think | 15:45 |
| clarkb | jeblair: ^ that sound right? | 15:45 |
| jeblair | AJaeger: see comment on 508519 (we discussed this on friday, but i forgot to put a note on the change) | 15:46 |
| *** lukebrowning has quit IRC | 15:46 | |
| AJaeger | jeblair: I see... | 15:47 |
| jeblair | clarkb, leyal: yes -- and that variant can actually be in the project-pipeline for your project. i will make an example | 15:47 |
| * SpamapS has been wondering if there could be a local commandline client we could make that would ask a Zuul to print out what the frozen tree would be given a local git repo. | 15:48 | |
| jeblair | clarkb, leyal: https://etherpad.openstack.org/p/WYZjjcI1GW | 15:48 |
| leyal | jeblair , clarkb - thanks :) | 15:49 |
| SpamapS | like 'zuul-show-tree https://zuulv3.openstack.org/' from inside a git repo with a remote that matches something in the zuul config and zuul would tell print out a simplified {pipeline: [job1, job2, job3]} | 15:49 |
| jeblair | leyal: what project is this for? | 15:49 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Run release note jobs only when they change https://review.openstack.org/508762 | 15:49 |
| leyal | jeblair - dragonflow | 15:50 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch jobs to use new release notes job https://review.openstack.org/508763 | 15:50 |
| jeblair | leyal: thx | 15:50 |
| jeblair | we may want to add that to the faq / migration doc | 15:50 |
| jeblair | but right now, i have to go open up holes in my house | 15:50 |
| AJaeger | leyal: want to send a patch for infra-manual - file zuulv3.rst? | 15:51 |
| leyal | AJaeger , well as i am new with the openstack infra , i prefer to check that i manged with that in dragonflow, and if so i will happy to send a patch :) | 15:52 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Switch release-note-jobs project-template to use new jobs https://review.openstack.org/508742 | 15:53 |
| mnaser | ok i promise im done with the spam :< | 15:53 |
| AJaeger | leyal: sure, let's get the change merged and then document for the benefit of others ;) | 15:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add releasenotes publication job https://review.openstack.org/508764 | 15:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Extract releasenotes build into a role https://review.openstack.org/508765 | 15:53 |
| mordred | AJaeger, clarkb: ^^ I believe that should get us both build and publish jobs that work for releasenotes | 15:54 |
| *** lukebrowning has joined #openstack-infra | 15:54 | |
| leyal | AJaeger , sure .. | 15:55 |
| AJaeger | mordred: found one small problem - releasenotes life in releasenotes dir, not in doc. Gave a -1 on both | 15:56 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Remove un-used legacy-releasenotes job https://review.openstack.org/508766 | 15:58 |
| mordred | AJaeger: yes - I actually have a mv at the bottom of the build script | 15:58 |
| AJaeger | mnaser, mordred, you both work on releasenotes jobs - hope those don't conflict. Should we push one set first? | 15:59 |
| mnaser | AJaeger i rebased off monty's job rename :) | 15:59 |
| AJaeger | mordred: I'll check again... | 15:59 |
| *** lukebrowning has quit IRC | 15:59 | |
| mnaser | so im writing all my stuff based on the newly renamed job | 15:59 |
| mordred | mnaser: yah- so - your patch looks good - but you may want to rebase on my latest with the publish- job in the project template | 16:00 |
| AJaeger | mordred: found it... | 16:00 |
| *** lnxnut has joined #openstack-infra | 16:00 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Remove un-used legacy-releasenotes job https://review.openstack.org/508766 | 16:01 |
| mordred | AJaeger: adding file exclusion real quick | 16:01 |
| *** hongbin has joined #openstack-infra | 16:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add releasenotes publication job https://review.openstack.org/508764 | 16:01 |
| AJaeger | mordred: will file exclusion work on post job now? | 16:02 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch jobs to use new release notes job https://review.openstack.org/508763 | 16:02 |
| mordred | AJaeger: it should do - yah | 16:02 |
| *** yamamoto has joined #openstack-infra | 16:02 | |
| mordred | AJaeger: I will admit I have not tried however - maybe I shoudl remove that, change it to a followup - and ask jeblair about it | 16:03 |
| *** isaacb has quit IRC | 16:04 | |
| AJaeger | mordred: might be safer ;) | 16:04 |
| mnaser | does someone know what would be the best/right step to rebase a set of commits on top of another? | 16:05 |
| mnaser | if it was one, i'd just cherry pick on top but | 16:05 |
| mnaser | i have a few in this case | 16:05 |
| SpamapS | mnaser: git rebase branch-with-those-commits | 16:06 |
| *** lukebrowning has joined #openstack-infra | 16:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add releasenotes publication job https://review.openstack.org/508764 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Only publish releasenotes if we need to https://review.openstack.org/508767 | 16:06 |
| clarkb | one hack is you can edit the shas in the rebase editor if you use -i | 16:06 |
| *** florianf has joined #openstack-infra | 16:06 | |
| mordred | jeblair: when you're back - could you look at https://review.openstack.org/508767 to make sure that'll work? | 16:06 |
| *** yamamoto has quit IRC | 16:08 | |
| *** ykarel has joined #openstack-infra | 16:08 | |
| *** lukebrowning_ has joined #openstack-infra | 16:08 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Run release note jobs only when they change https://review.openstack.org/508762 | 16:09 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Switch release-note-jobs project-template to use new jobs https://review.openstack.org/508742 | 16:09 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Remove un-used legacy-releasenotes job https://review.openstack.org/508766 | 16:09 |
| mnaser | mordred i added my stuff on top of yours if you want to add that locally (i think i did it right?) | 16:09 |
| *** markvoelker has joined #openstack-infra | 16:10 | |
| *** lukebrowning has quit IRC | 16:10 | |
| mordred | mnaser: cool ... I'm working on a patch to project-config now to take care of the ones not using the template | 16:10 |
| SpamapS | mordred: btw... did you have a plan to add streaming output to the pip module? | 16:10 |
| mnaser | mordred i think i already did that? | 16:11 |
| mordred | mnaser: you did? | 16:11 |
| mnaser | mordred https://review.openstack.org/#/c/508763/ this? | 16:11 |
| fungi | mnaser: if i don't have edits pending and just want to arrange commits in a stack, i'll use git review -d to grab the bottom one followed by git review -x for each of the children in order, or use the checkout and cherry-pick options in gertty. also git-restack is useful if you just want to reorder commits without rebasing the bottom one on the branch | 16:11 |
| mordred | mnaser: oh! awesome - thanks! | 16:11 |
| mordred | mnaser: I'll build on top of that one to extract those to using the template | 16:12 |
| fungi | (git restack is mostly syntactic sugar wrapping git rebase) | 16:12 |
| mnaser | fungi so much to learn.. i just recently started getting more comfortable with stacks of changes | 16:12 |
| clarkb | ok there is apparently not an easy way to get the filesystem of a file via python stdlib | 16:12 |
| mnaser | before i would do endless amounts of branches :) | 16:12 |
| clarkb | I guess that means we parse mount output? | 16:12 |
| *** lukebrowning has joined #openstack-infra | 16:12 | |
| clarkb | fungi: is there a different trick for determining if cp -l will work? | 16:12 |
| mnaser | it's how i did my first openstack commit (based from the contrib docs with git branch ..; git commit; git review) back in 2011 :P | 16:12 |
| clarkb | I guess I can try making a hardlink first? | 16:12 |
| mnaser | by the way.. either zuul has been reconfiguring for a long time .. or nothing has gotten into its queue for 30 minutes | 16:13 |
| *** lukebrowning_ has quit IRC | 16:13 | |
| fungi | clarkb: hrm... you can get os.path to canonicalize the path to a file by following all symlink nodes in the given path, i forget the function off the top of my head though. as for mapping that to a mountpoint... not sure | 16:13 |
| mnaser | (as i say this, i bet it'll snap out of it) | 16:13 |
| mordred | SpamapS: I do not have any plans to do that at the moment - adding streaming output to shell is a *massive* hack, adding it to other modules without significant plumbing in ansible itself would be pretty hard | 16:13 |
| fungi | clarkb: but yeah, lbyl is likely to be painful in this situation, probably easier to ask for forgiveness | 16:14 |
| mordred | SpamapS: I do intend, though, to work on adding streaming output possibilities to ansible core and have talked some with the core team about it | 16:14 |
| *** florianf has quit IRC | 16:15 | |
| *** lukebrowning has quit IRC | 16:16 | |
| *** ykarel has quit IRC | 16:18 | |
| clarkb | looks like tripleo may be copying everything off hosts again http://logs.openstack.org/52/508752/1/check/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/f746ec7/logs/subnode-2/etc/ | 16:19 |
| dansmith | clarkb: so we merged some stuff friday night but I think we're stuck again on some multinode setup issues lately | 16:19 |
| clarkb | dansmith: yup, I think the fixes for that just merged | 16:19 |
| dansmith | clarkb: ah okay so I should recheck? | 16:20 |
| clarkb | dansmith: the last remaining issue itting nova(client) is the libs from git thing I think | 16:20 |
| clarkb | dansmith: ya I think its worth a recheck | 16:20 |
| dansmith | cool, thanks | 16:20 |
| *** lukebrowning has joined #openstack-infra | 16:22 | |
| *** lukebrowning has quit IRC | 16:26 | |
| *** bobh has joined #openstack-infra | 16:27 | |
| *** atarakt_ has quit IRC | 16:27 | |
| *** lnxnut has quit IRC | 16:27 | |
| *** lukebrowning has joined #openstack-infra | 16:28 | |
| mnaser | nothings queued up for close to 50 minutes now.. i think there's something wrong :x | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Collapse releasenotes jobs to using project template https://review.openstack.org/508769 | 16:29 |
| mordred | AJaeger, mnaser: ^^ ok - that should get us back on release-notes-jobs | 16:31 |
| AJaeger | mordred: great - looking forward to have those all in. Right now waiting for Zuul to move forward | 16:31 |
| AJaeger | mnaser: I just noticed that queue length changed - from 379 events, 198 results to 202 results. So, it's processing events slowly | 16:32 |
| *** lukebrowning has quit IRC | 16:32 | |
| AJaeger | bbl | 16:33 |
| mnaser | im not sure.. :X | 16:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Switch release-note-jobs project-template to use new jobs https://review.openstack.org/508742 | 16:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove un-used legacy-releasenotes job https://review.openstack.org/508766 | 16:33 |
| *** edmondsw has joined #openstack-infra | 16:33 | |
| mordred | mnaser: I also updated your patch https://review.openstack.org/508742 to use the publish job | 16:33 |
| mnaser | oh good call | 16:34 |
| *** bobh has quit IRC | 16:34 | |
| *** lukebrowning has joined #openstack-infra | 16:34 | |
| mordred | clarkb: got a sec for https://review.openstack.org/#/c/508765 ? | 16:34 |
| mordred | clarkb: and https://review.openstack.org/#/c/508713 | 16:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Switch to using publish-openstack-sphinx-docs-infra https://review.openstack.org/508751 | 16:35 |
| mnaser | yay finally zuul is flowing | 16:37 |
| *** edmondsw has quit IRC | 16:38 | |
| *** lukebrowning has quit IRC | 16:38 | |
| *** lukebrowning has joined #openstack-infra | 16:39 | |
| fungi | mnaser: this is probably another example of the slow dynamic reconfiguration situation we want to address | 16:41 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add shade functional libs tests to check as non-voting https://review.openstack.org/508770 | 16:42 |
| *** markvoelker has quit IRC | 16:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Handle z-c shim copies across filesystems https://review.openstack.org/508772 | 16:43 |
| clarkb | ok ^ is hacky and only slightly tested | 16:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove legacy shade jobs from os-client-config https://review.openstack.org/508773 | 16:43 |
| clarkb | but I think that is basically what we need to make z-c + tripleo happier | 16:44 |
| *** lukebrowning has quit IRC | 16:44 | |
| *** lukebrowning has joined #openstack-infra | 16:44 | |
| mordred | clarkb: wfm | 16:45 |
| clarkb | I should be able to depends on that I think | 16:46 |
| clarkb | so will do that | 16:46 |
| *** lukebrowning_ has joined #openstack-infra | 16:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/tripleo-ci master: Use sub_nodes_private https://review.openstack.org/508752 | 16:48 |
| *** lukebrowning has quit IRC | 16:49 | |
| *** lukebrow_ has joined #openstack-infra | 16:49 | |
| *** hongbin has quit IRC | 16:49 | |
| *** hongbin has joined #openstack-infra | 16:50 | |
| *** lukebrowning_ has quit IRC | 16:51 | |
| clarkb | jeblair: the memory increases in zuul seem to coincide with these long pauses of queue processing | 16:52 |
| *** lukebrow_ has quit IRC | 16:53 | |
| clarkb | there are two recent steps up in memory use according to the cacti graph. First one seems to line up with the issue mnaser pointed out of the world pausing, then once it got through and did somethings it quickly went into a similar state with a corresponding memory increase | 16:53 |
| *** lukebrowning has joined #openstack-infra | 16:54 | |
| *** hongbin_ has joined #openstack-infra | 16:54 | |
| frickler | can I add a required-project to a different project globally in p-c/zuul.d/projects.yaml? neutron-dynamic-routing needs neutron everywhere it seems, see https://review.openstack.org/503935 | 16:55 |
| clarkb | frickler: yes you have to make a job variant, https://etherpad.openstack.org/p/WYZjjcI1GW is the example jeblair posted earlier | 16:56 |
| *** hongbin has quit IRC | 16:56 | |
| *** lukebrowning_ has joined #openstack-infra | 16:56 | |
| *** lukebrowning has quit IRC | 16:58 | |
| frickler | clarkb: oh, so do a variant for each job, o.k., thx | 17:00 |
| SpamapS | clarkb: stat -c %m a/file/in/linux can show you the mount point | 17:00 |
| SpamapS | clarkb: so in python you can at least exec that | 17:00 |
| *** lukebrowning_ has quit IRC | 17:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add noop job description https://review.openstack.org/508774 | 17:01 |
| frickler | clarkb: ah, wait, those jobs are coming from templates. | 17:02 |
| mordred | frickler: which jobs? | 17:02 |
| *** yamamoto has joined #openstack-infra | 17:04 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Add neutron to required-projects for neutron-dynamic-routing https://review.openstack.org/508775 | 17:05 |
| frickler | mordred: nevermind, this should be it methinks ^^ | 17:05 |
| *** lukebrowning has joined #openstack-infra | 17:08 | |
| *** yamamoto has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add objgraph to SIGUSR2 https://review.openstack.org/508753 | 17:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Use default releasenotes jobs https://review.openstack.org/508708 | 17:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from shade https://review.openstack.org/508749 | 17:11 |
| SpamapS | mordred: count me in for helping drive streaming into core. THe people here at GoDaddy were blown away by the log streaming in Zuul and basically want all their ansible run that way yesterday | 17:11 |
| *** lukebrowning has quit IRC | 17:12 | |
| mordred | SpamapS: cool - I'll try to write down the state of my brain on it this week so that we can work together on it | 17:14 |
| clarkb | ya zuulv3 is heading towards oom now I think | 17:15 |
| SpamapS | mordred: I'd love to see it incorporated into ARA | 17:15 |
| clarkb | and it definitely seemed to coincide with whatever caused zuul to pause | 17:15 |
| *** lukebrowning has joined #openstack-infra | 17:15 | |
| SpamapS | clarkb: I wonder if it has to do with things piling up waiting on gearman jobs to respond. | 17:16 |
| SpamapS | I forget if we do background or foreground jobs | 17:16 |
| clarkb | I think foreground | 17:16 |
| SpamapS | foreground yeah | 17:17 |
| SpamapS | couldn't use the work complete stuff otherwise | 17:17 |
| SpamapS | so that means there's a thread for every job that we've tried to execute in the scheduler | 17:17 |
| *** lukebrowning_ has joined #openstack-infra | 17:18 | |
| SpamapS | ah no | 17:19 |
| SpamapS | we just have the one thread | 17:19 |
| SpamapS | good | 17:19 |
| *** lukebrowning has quit IRC | 17:20 | |
| SpamapS | clarkb: have we looked at what gets logged just before/after the pause/memory spike? | 17:20 |
| SpamapS | anyway, time to Sunday | 17:20 |
| mordred | SpamapS: I don't think we've seen anything specifically - although we just landed "Add objgraph to SIGUSR2" - so hopefully we'll be able to analyze a bit rolling forward | 17:21 |
| clarkb | SpamapS: I was looking at logs but nothing stands out to me | 17:21 |
| *** lukebrowning_ has quit IRC | 17:22 | |
| *** lbragstad has quit IRC | 17:24 | |
| *** lbragstad has joined #openstack-infra | 17:28 | |
| *** bobh has joined #openstack-infra | 17:30 | |
| *** eroux has joined #openstack-infra | 17:30 | |
| *** bobh has quit IRC | 17:34 | |
| fungi | SpamapS: oh, if stat gives it to you, i think there's an os.stat or similar module which gets it from the kernel too | 17:35 |
| fungi | (unix/linux-only obviously) | 17:35 |
| fungi | worth noting, memory utilization has picked up and zuul is dipping ~3gib into swap now | 17:37 |
| fungi | latest climb seems to have started around 16:40z | 17:38 |
| fungi | judging from cacti | 17:38 |
| *** caphrim007 has joined #openstack-infra | 17:39 | |
| mordred | fungi: yah - to be fair on unix/linux-only, the z-c shim is only intended to be around for a short time - so not working on non-linux sohuldn't be an issue | 17:40 |
| fungi | and 508751 (Switch to using publish-openstack-sphinx-docs-infra) merged at 16:35z | 17:40 |
| *** markvoelker has joined #openstack-infra | 17:40 | |
| mordred | fungi: my money is on the memory spikes being tied to reconfiguration events | 17:40 |
| fungi | i wonder if these jumps in memory utilization are related to reconfig | 17:40 |
| fungi | yeah, thaht | 17:40 |
| *** lbragstad has quit IRC | 17:41 | |
| mordred | yah - especially ones that affect project-config and/or a large swath of repos | 17:41 |
| *** dave-mccowan has joined #openstack-infra | 17:41 | |
| fungi | jeblair: ^ anecdotal but possibly helpful datapoint | 17:41 |
| *** lukebrowning has joined #openstack-infra | 17:42 | |
| *** caphrim007 has quit IRC | 17:44 | |
| mnaser | mordred https://review.openstack.org/#/c/508742/ - i believe the publish-openstack-releasenotes job is not a thing | 17:45 |
| mordred | mnaser: https://review.openstack.org/#/c/508764/3/zuul.d/jobs.yaml | 17:46 |
| mordred | mnaser: it's just that depends-on doesn't work witih project-config | 17:46 |
| mnaser | oopsies | 17:46 |
| mnaser | sorry | 17:46 |
| *** lukebrowning has quit IRC | 17:46 | |
| mnaser | this whole working with 3 or 4 sets of job repos hopefully will become easier :p | 17:46 |
| mnaser | in other fun news tho, puppet openstack integration jobs are 100% zuulv3 and i think i can get the libraries 100% zuulv3'd today | 17:47 |
| *** lukebrowning has joined #openstack-infra | 17:48 | |
| AJaeger | woot! | 17:48 |
| fungi | mnaser: the goal is that you eventually only end up working with job configuration in your repos and adding projects to project-config... the rest ought to stabilize over time | 17:49 |
| mnaser | yeah, i've been trying to leave useful abstracted stuff back up to openstack-zuul-jobs and zuul-jobs | 17:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy-rpm-packaging-tox-lint https://review.openstack.org/508610 | 17:51 |
| mnaser | i think zuul has oom'd | 17:52 |
| *** lukebrowning has quit IRC | 17:52 | |
| AJaeger | mnaser: indeed http://zuulv3.openstack.org/ has no real content | 17:53 |
| AJaeger | infra-root, we need a doctor for zuul ^ | 17:53 |
| AJaeger | mnaser: no, it's back... | 17:53 |
| mnaser | ha, tricked us. | 17:54 |
| AJaeger | mnaser: maybe heavy swapping? let's check cacti... | 17:54 |
| mnaser | i think thats the case | 17:54 |
| mnaser | 7g swap used | 17:55 |
| *** lnxnut has joined #openstack-infra | 17:55 | |
| mnaser | it def looks like a memory leak pattern | 17:55 |
| AJaeger | mnaser: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63981&rra_id=all shows 520 MB | 17:56 |
| AJaeger | oh, current - indeed 7G | 17:57 |
| AJaeger | and growing - 7.86 now | 17:57 |
| AJaeger | queue length on zuulv3.o.o is increasing, so something works | 17:58 |
| mnaser | AJaeger i guess the advantage in this case is that while its still alive folks can try debugging it | 17:58 |
| mnaser | and seeing whats taking up all the memory | 17:58 |
| *** hongbin_ has quit IRC | 17:58 | |
| *** lukebrowning has joined #openstack-infra | 17:59 | |
| *** eroux has quit IRC | 17:59 | |
| AJaeger | mnaser: ;) | 17:59 |
| AJaeger | I'll be back later... | 17:59 |
| openstackgerrit | garyk proposed openstack-infra/project-config master: Add missing projects to vmware-nsxlib https://review.openstack.org/508779 | 18:01 |
| pabelanger | AJaeger: not sure, I can see something is using 8GB of swap on zuulv3.o.o | 18:03 |
| pabelanger | 2017-10-01 17:56:45,634 DEBUG zuul.IndependentPipelineManager: Loading dynamic layout (phase 1) | 18:03 |
| pabelanger | that is currently the last line in debug.log | 18:04 |
| *** lukebrowning has quit IRC | 18:04 | |
| pabelanger | I am guessing zuul is doing something, but slow due to swapping | 18:04 |
| pabelanger | Ya, and nodepool is idle | 18:05 |
| *** yamamoto has joined #openstack-infra | 18:05 | |
| pabelanger | seems statsd isn't setup on nl01 and nl02 too | 18:06 |
| pabelanger | checking to see why | 18:06 |
| *** eranrom has joined #openstack-infra | 18:09 | |
| *** dave-mccowan has quit IRC | 18:10 | |
| *** yamamoto has quit IRC | 18:11 | |
| *** dave-mccowan has joined #openstack-infra | 18:11 | |
| *** markvoelker has quit IRC | 18:14 | |
| *** dave-mccowan has quit IRC | 18:17 | |
| *** ijw has joined #openstack-infra | 18:18 | |
| mnaser | its up to 10G of swap :< | 18:18 |
| *** edmondsw has joined #openstack-infra | 18:19 | |
| *** ykarel has joined #openstack-infra | 18:21 | |
| *** lnxnut has quit IRC | 18:22 | |
| *** edmondsw has quit IRC | 18:24 | |
| mordred | swap use is dropping | 18:25 |
| *** thorst has quit IRC | 18:25 | |
| *** thorst has joined #openstack-infra | 18:25 | |
| *** ijw has quit IRC | 18:25 | |
| *** thorst has quit IRC | 18:26 | |
| mordred | and rising again | 18:29 |
| *** slaweq has joined #openstack-infra | 18:30 | |
| *** bobh has joined #openstack-infra | 18:31 | |
| mnaser | that sigusr2 patch would be nice if it landed | 18:32 |
| fungi | yeah, but won't help until after a restart | 18:33 |
| SamYaple | can I publish to tarball.o.o from a job I create in an unprotected repo? or is that a change i would have to land in project-config? | 18:34 |
| *** bobh has quit IRC | 18:35 | |
| pabelanger | mordred: re: 508563 I don't see how tox -l will work. Since there is no tox.ini in that directory: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/fetch-tox-output/tasks/main.yaml#n22 we'd need a chdir: into src directory. However, looking at the patch, I think it would be better just to pass into the role the environments we want, over having it run tox directly. | 18:37 |
| pabelanger | you can see the failure here: http://logs.openstack.org/e5/e5a2189276396201ad88a6c47360c90447c91589/release/publish-openstack-python-tarball/2bdd521/job-output.txt.gz#_2017-10-01_07_35_10_920772 | 18:37 |
| *** slaweq has quit IRC | 18:38 | |
| fungi | SamYaple: jobs which publish to tarballs.o.o need access to the auth secret for the site, so have to go in project-config where those secrets are defined (this is a security measure in zuul itself) | 18:38 |
| pabelanger | SamYaple: no, you'll need to parent to the publish-artifact jobs in project-config | 18:38 |
| fungi | pabelanger: can an untrusted project actually do that? is it just the role with access to the secret which needs to be in project-config? | 18:39 |
| SamYaple | ok let me see if i can find an example of this | 18:40 |
| jeblair | fungi: we have a job in p-c that publishes things to the $project/ directory under tarballs | 18:40 |
| jeblair | so any job can parent to it, and thereby publish thre | 18:40 |
| pabelanger | fungi: yup, untrusted jobs can be child to trusted. | 18:40 |
| jeblair | there | 18:40 |
| pabelanger | keep in might, we only publish on post pipelines too | 18:40 |
| jeblair | they can't override the publish location though | 18:41 |
| SamYaple | im ok with that | 18:41 |
| pabelanger | yah | 18:41 |
| fungi | okay, so it's just if they need to modify the role with access to the secret that it ends up needing to be in project-config | 18:41 |
| jeblair | fungi: it's the playbook that's key here -- because the parent job in p-c has a post playbook that uses the secret | 18:41 |
| SamYaple | do you have an example of this inheritance somewhere? | 18:41 |
| jeblair | the playbook is what is run with perms to use that | 18:42 |
| jeblair | it's basically just like how log publication works | 18:42 |
| pabelanger | SamYaple: publish-openstack-python-tarball for example | 18:42 |
| SamYaple | looking, thanks :) | 18:42 |
| fungi | ahh, okay | 18:42 |
| pabelanger | that is an untrusted job in openstack-zuul-jobs that will parent to the trusted | 18:42 |
| fungi | i'm still mildly fuzzy on the difference between ansible roles and playbooks tbh | 18:43 |
| pabelanger | managed to read the highlights of the cutover, all and all, seems to have gone well | 18:43 |
| pabelanger | fungi: playbooks use a role, which ansible-playbook invokes. Roles cannot be run directly themself, they always need a playbook. Where playbooks can be run with / without a role | 18:44 |
| frickler | doesn't look like zuul has been making progress in the last 2h, new patches missing, existing jobs stuck | 18:45 |
| jeblair | and roles don't need to be in the same repo. so it doesn't matter where the publish tarball role is defined, only that the playbook which uses it is in project-config. that's what lets it use the secret. | 18:45 |
| pabelanger | and right now, we are placing all our shared logic into roles, so we can share them between jobs | 18:45 |
| jeblair | frickler: thanks; i have to run right now, hopefully someone can look at the logs and see what's up | 18:46 |
| pabelanger | I think zuul is just slow because of swapping. Right now scheduler is at 100% cpu | 18:46 |
| SamYaple | so something like this should work? https://review.openstack.org/#/c/508782/1/.zuul.yaml and in the post job just move the tarball to "{{ zuul.executor.work_root }}/artifacts/" ? | 18:48 |
| pabelanger | also seeing some connection drop to zookeeper and an exception in gearman: http://paste.openstack.org/show/622439/ | 18:48 |
| frickler | https://review.openstack.org/508775 has not been showing up on http://zuulv3.openstack.org/ for almost 2h now | 18:48 |
| pabelanger | SamYaple: did you miss adding playbooks to the review? | 18:49 |
| pabelanger | SamYaple: but ya, adding files into artifacts folder on node is how we collect files | 18:50 |
| SamYaple | pabelanger: yea that was just to make sure the job was described right | 18:50 |
| pabelanger | SamYaple: Yup, looks right so far | 18:50 |
| SamYaple | will you publish arbitary files to tarball.o.o? or does the parent job wrap all of those files into another tarball to publish? | 18:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Change in to work dir before executing tox https://review.openstack.org/508783 | 18:51 |
| mordred | pabelanger: thanks ^^ that should fix the fetch-tox-output error | 18:51 |
| mordred | fungi: roles are a collection of tasks and variables ... playbooks are associations between tasks and what host to run them on | 18:52 |
| pabelanger | SamYaple: Ya, there is no filter, we just sync what ever is in the folder. Long term, we likely should ensure they are gziped things | 18:52 |
| SamYaple | in my case they will be, but cool. thats exactly what im looking for. thanks pabelanger | 18:52 |
| pabelanger | np | 18:53 |
| pabelanger | mordred: +2 | 18:53 |
| *** hashar has joined #openstack-infra | 18:53 | |
| mordred | fungi: so a playbook can contain tasks directly, or it can contain roles - or it can contain both - but ultimatley a playbook is how you tell ansible where to run things - and roles are a way to bundle up a collection of things to run | 18:53 |
| pabelanger | will bbiab, going to enjoy weather outside | 18:53 |
| SamYaple | i dont go outside until the sunsets | 18:53 |
| mordred | we've made it to phase 1 of the most recent layout change | 18:55 |
| mordred | and are back down to 5.9G of swap, fwiw | 18:55 |
| mnaser | 2017-10-01 18:23:03,754 INFO kazoo.client: Zookeeper session lost, state: EXPIRED_SESSION | 18:56 |
| mnaser | i think that's just literally zuul not responding for so long that it gets dropped by zookeeper, and probably not an issue (.. keyword: i think) | 18:56 |
| *** lukebrowning has joined #openstack-infra | 19:00 | |
| *** rkukura_ has joined #openstack-infra | 19:06 | |
| *** yamamoto has joined #openstack-infra | 19:07 | |
| *** rkukura has quit IRC | 19:09 | |
| *** rkukura_ is now known as rkukura | 19:09 | |
| *** lukebrowning_ has joined #openstack-infra | 19:10 | |
| *** slaweq has joined #openstack-infra | 19:11 | |
| *** lukebrowning has quit IRC | 19:11 | |
| *** markvoelker has joined #openstack-infra | 19:11 | |
| *** yamamoto has quit IRC | 19:12 | |
| *** lukebrowning_ has quit IRC | 19:15 | |
| *** lukebrowning has joined #openstack-infra | 19:17 | |
| AJaeger | infra-root, translations jobs are broken - post jobs fail, see http://logs.openstack.org/d8/d82508e96f3d586669ed30a72482cf66706b37fe/post/legacy-upstream-translation-update/5d3dd52/ - and the periodic jobs do not seem to run at all | 19:18 |
| AJaeger | that's a nova post job - but http://logs.openstack.org/d8/d82508e96f3d586669ed30a72482cf66706b37fe/post/legacy-upstream-translation-update/5d3dd52/job-output.txt.gz#_2017-09-30_18_10_16_053992 shows me that nova is missing in required-projects. Shouldn't that one exist automatically? | 19:19 |
| *** lukebrowning has quit IRC | 19:21 | |
| mnaser | AJaeger z-c bug afaik | 19:21 |
| mnaser | https://review.openstack.org/#/c/508747/ | 19:21 |
| mnaser | acutally sorry | 19:22 |
| mnaser | im wrong | 19:22 |
| mnaser | it looks like you can even see it clone things /home/zuul/src/git.openstack.org/openstack/nova | 19:22 |
| *** ijw has joined #openstack-infra | 19:22 | |
| *** dave-mccowan has joined #openstack-infra | 19:23 | |
| *** lukebrowning has joined #openstack-infra | 19:23 | |
| AJaeger | mnaser: it needs requiremetns as well - which is not there... | 19:24 |
| mnaser | maybe wrong error | 19:24 |
| AJaeger | yeah, 508747 should fix that... | 19:25 |
| *** thorst has joined #openstack-infra | 19:26 | |
| *** hashar has quit IRC | 19:26 | |
| *** lukebrowning has quit IRC | 19:27 | |
| *** ijw has quit IRC | 19:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add requirements for translation jobs https://review.openstack.org/508784 | 19:28 |
| AJaeger | mnaser: this should fix it ^ | 19:28 |
| AJaeger | (at least the post job - the periodic jobs seem to not to be run at all) | 19:29 |
| *** lukebrowning has joined #openstack-infra | 19:29 | |
| *** claudiub has joined #openstack-infra | 19:30 | |
| *** bobh has joined #openstack-infra | 19:31 | |
| *** lukebrowning has quit IRC | 19:33 | |
| *** lukebrowning has joined #openstack-infra | 19:35 | |
| *** bobh has quit IRC | 19:36 | |
| *** dave-mcc_ has joined #openstack-infra | 19:39 | |
| *** slaweq has quit IRC | 19:40 | |
| *** lukebrowning has quit IRC | 19:40 | |
| *** dave-mccowan has quit IRC | 19:40 | |
| *** slaweq has joined #openstack-infra | 19:40 | |
| *** caphrim007 has joined #openstack-infra | 19:40 | |
| *** lukebrowning has joined #openstack-infra | 19:41 | |
| *** ykarel has quit IRC | 19:42 | |
| *** markvoelker has quit IRC | 19:44 | |
| *** slaweq has quit IRC | 19:44 | |
| *** caphrim007 has quit IRC | 19:45 | |
| *** efried_thbagh has quit IRC | 19:45 | |
| *** slaweq has joined #openstack-infra | 19:45 | |
| *** lukebrowning has quit IRC | 19:46 | |
| *** rkukura has quit IRC | 19:46 | |
| *** lukebrowning has joined #openstack-infra | 19:47 | |
| *** lnxnut has joined #openstack-infra | 19:50 | |
| *** lukebrowning has quit IRC | 19:51 | |
| *** lukebrowning has joined #openstack-infra | 19:53 | |
| *** slaweq has quit IRC | 19:55 | |
| *** efried_thbagh has joined #openstack-infra | 19:55 | |
| SamYaple | I have noticed my jobs (loci) also appear in the check-triple-o queue.... but that shouldn't happen right? | 19:57 |
| SamYaple | it actually looks like *everything* ends up in the check AND check-tripleo queues | 19:58 |
| *** lukebrowning has quit IRC | 19:58 | |
| *** lukebrowning has joined #openstack-infra | 20:00 | |
| jeblair | SamYaple: not your *jobs*, just your *changes*, right? | 20:01 |
| *** esberglu has joined #openstack-infra | 20:01 | |
| SamYaple | sorry, yes | 20:01 |
| SamYaple | changes | 20:01 |
| jeblair | SamYaple: that's zuul figuring out whether the change has any jobs to run there. everything's dynamic now :) | 20:01 |
| SamYaple | so its not actually going to spin up a vm and start running tests in that queue? | 20:02 |
| jeblair | SamYaple: not if you don't see any jobs show up | 20:02 |
| SamYaple | cool. ok | 20:02 |
| SamYaple | i see. that explains the lag between the change appearing and jobs showing up | 20:03 |
| *** lukebrowning has quit IRC | 20:05 | |
| *** esberglu has quit IRC | 20:05 | |
| *** slaweq has joined #openstack-infra | 20:05 | |
| jeblair | AJaeger: the nova repo should have been there. it does appear in the inventory variables for the job. | 20:06 |
| *** lukebrowning has joined #openstack-infra | 20:06 | |
| *** slaweq has quit IRC | 20:07 | |
| *** edmondsw has joined #openstack-infra | 20:07 | |
| *** yamamoto has joined #openstack-infra | 20:08 | |
| *** lukebrowning has quit IRC | 20:10 | |
| jeblair | AJaeger, mordred: here's the problem: http://logs.openstack.org/d8/d82508e96f3d586669ed30a72482cf66706b37fe/post/legacy-upstream-translation-update/5d3dd52/job-output.txt.gz#_2017-09-30_18_10_14_703044 | 20:10 |
| jeblair | AJaeger, mordred: unfortunately you can't actually see that shell command | 20:11 |
| jeblair | AJaeger, mordred: and i can't deep-link into ara | 20:11 |
| jeblair | AJaeger, mordred: but, if you go clicking around yourself, you'll find "rm -rf `dirname $WORKSPACE`/*" | 20:11 |
| *** mriedem has quit IRC | 20:12 | |
| *** edmondsw has quit IRC | 20:12 | |
| mnaser | jeblair you may be happy to know about this https://review.openstack.org/#/c/508704/ if this is the reason you cant deep link | 20:12 |
| jeblair | AJaeger, mordred: which seems to delete everything above $WORKSPACE; that would be /home/zuul, which includes the src dir | 20:12 |
| *** yamamoto has quit IRC | 20:13 | |
| jeblair | mnaser: indeed! i saw the output of that the other day, and eagerly await a release :) | 20:13 |
| *** lukebrowning has joined #openstack-infra | 20:13 | |
| *** lukebrowning has quit IRC | 20:17 | |
| *** lukebrowning has joined #openstack-infra | 20:17 | |
| *** lnxnut has quit IRC | 20:18 | |
| *** lukebrowning_ has joined #openstack-infra | 20:19 | |
| *** lukebrowning has quit IRC | 20:22 | |
| *** ijw has joined #openstack-infra | 20:23 | |
| *** lukebrowning_ has quit IRC | 20:24 | |
| *** ijw has quit IRC | 20:28 | |
| *** lukebrowning has joined #openstack-infra | 20:28 | |
| *** yamahata has joined #openstack-infra | 20:29 | |
| *** bobh has joined #openstack-infra | 20:32 | |
| *** lukebrowning has quit IRC | 20:33 | |
| *** lukebrowning has joined #openstack-infra | 20:34 | |
| *** bobh has quit IRC | 20:36 | |
| *** thorst has quit IRC | 20:38 | |
| *** thorst has joined #openstack-infra | 20:38 | |
| *** lukebrowning has quit IRC | 20:39 | |
| *** esberglu has joined #openstack-infra | 20:39 | |
| *** lukebrowning has joined #openstack-infra | 20:41 | |
| *** markvoelker has joined #openstack-infra | 20:41 | |
| *** dave-mcc_ has quit IRC | 20:42 | |
| *** thorst has quit IRC | 20:42 | |
| openstackgerrit | Eyal Leshem proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 20:44 |
| *** lukebrowning has quit IRC | 20:45 | |
| *** lukebrowning has joined #openstack-infra | 20:47 | |
| *** lukebrowning has quit IRC | 20:52 | |
| *** thorst has joined #openstack-infra | 20:57 | |
| *** lukebrowning has joined #openstack-infra | 20:58 | |
| SpamapS | So, if the zuul scheduler is causing swapping... is it time to move it to a bigger VM? | 20:58 |
| SpamapS | Especially since it sounds like memory actually does walk back a bit after reconfigures.. so maybe it's not a leak situation. | 21:00 |
| *** thorst has quit IRC | 21:01 | |
| *** lukebrowning has quit IRC | 21:01 | |
| *** lukebrowning has joined #openstack-infra | 21:02 | |
| jeblair | infra-root: i think i'd like to go ahead and restart zuul to get a head start on collecting data with objgraph | 21:05 |
| SamYaple | +1 | 21:05 |
| *** yamahata has quit IRC | 21:06 | |
| *** lukebrowning has quit IRC | 21:07 | |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| jeblair | okay, i installed objgraph and am restarting the scheduler | 21:11 |
| jeblair | zuul was swapping while creating a dynamic config. i suspect part of the idle memory it swapped out earlier contained the cached config, which it needs on dynamic reconfig | 21:12 |
| *** markvoelker has quit IRC | 21:14 | |
| *** yamamoto has quit IRC | 21:14 | |
| jeblair | and i sent it a sigusr2 so we have a baseline shortly after start (though due to queued events, it already wasn't idle) | 21:16 |
| *** thorst has joined #openstack-infra | 21:17 | |
| *** lukebrowning has joined #openstack-infra | 21:21 | |
| *** thorst has quit IRC | 21:22 | |
| jeblair | i ran it again, and i didn't get an yappi or objgraph output | 21:23 |
| jeblair | i think there may be something wrong; i'll add exception handlers | 21:24 |
| *** lukebrowning has quit IRC | 21:26 | |
| *** lukebrowning has joined #openstack-infra | 21:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add debug messages and exception handling to stack dump handler https://review.openstack.org/508786 | 21:28 |
| jeblair | i'm going to restart with that manually applied | 21:31 |
| *** lukebrowning has quit IRC | 21:32 | |
| jeblair | ya, it's a python3 error with the yappi handler | 21:33 |
| jeblair | i'm going to uninstall yappi for now so we don't have to worry about it impacting this | 21:34 |
| *** slaweq has joined #openstack-infra | 21:35 | |
| SpamapS | yappi claims to support <= 3.4 | 21:36 |
| SpamapS | so maybe they have some borken stuff | 21:36 |
| jeblair | SpamapS: oh, sorry, it's in our use of it | 21:37 |
| SpamapS | ahh | 21:37 |
| *** esberglu has quit IRC | 21:38 | |
| jeblair | or maybe not? | 21:38 |
| jeblair | http://paste.openstack.org/show/622441/ | 21:38 |
| jeblair | yeah, i think we just want to change our bytesio to a stringio | 21:38 |
| *** slaweq has quit IRC | 21:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change yappi bytesio to stringio https://review.openstack.org/508787 | 21:39 |
| jeblair | i haven't tested that, but maybe that's the fix | 21:40 |
| *** lukebrowning has joined #openstack-infra | 21:40 | |
| *** caphrim007 has joined #openstack-infra | 21:41 | |
| *** lnxnut has joined #openstack-infra | 21:45 | |
| *** lukebrowning has quit IRC | 21:45 | |
| *** caphrim007 has quit IRC | 21:46 | |
| *** lukebrowning has joined #openstack-infra | 21:46 | |
| *** lukebrowning has quit IRC | 21:51 | |
| *** edmondsw has joined #openstack-infra | 21:55 | |
| *** Sukhdev has joined #openstack-infra | 21:56 | |
| *** robled has quit IRC | 21:57 | |
| *** robled has joined #openstack-infra | 21:58 | |
| *** robled has quit IRC | 21:58 | |
| *** robled has joined #openstack-infra | 21:58 | |
| *** lukebrowning has joined #openstack-infra | 21:59 | |
| *** edmondsw has quit IRC | 22:00 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Export missing projects in Rally legacy jobs https://review.openstack.org/508630 | 22:01 |
| *** lukebrowning_ has joined #openstack-infra | 22:01 | |
| *** robled has quit IRC | 22:02 | |
| *** atarakt_ has joined #openstack-infra | 22:03 | |
| *** robled has joined #openstack-infra | 22:03 | |
| *** lukebrow_ has joined #openstack-infra | 22:03 | |
| *** lukebrowning has quit IRC | 22:04 | |
| *** lukebrowning_ has quit IRC | 22:05 | |
| *** esberglu has joined #openstack-infra | 22:06 | |
| *** lukebrow_ has quit IRC | 22:08 | |
| *** esberglu has quit IRC | 22:10 | |
| *** yamamoto has joined #openstack-infra | 22:10 | |
| *** lnxnut has quit IRC | 22:13 | |
| *** yamamoto has quit IRC | 22:15 | |
| *** bobh has joined #openstack-infra | 22:29 | |
| *** ekcs has joined #openstack-infra | 22:29 | |
| SamYaple | jeblair: out.write(LINESEP.encode('utf-8')) would be the fix, no? | 22:32 |
| *** atarakt_ has quit IRC | 22:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: alien-list: use provider name https://review.openstack.org/508788 | 22:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-requirements-python34 job https://review.openstack.org/508489 | 22:40 |
| *** markvoelker has joined #openstack-infra | 22:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove some pypy jobs that don't work https://review.openstack.org/504748 | 22:46 |
| tonyb | The election publish job failed with: http://logs.openstack.org/a1/a1411cd50c74234692e0fd64c7df365663d37067/post/legacy-static-election-publish/e02d4e2/job-output.txt.gz#_2017-10-01_22_33_45_968313 I wont be around to help debug it but I was hoping I could ask y'all to take a look | 22:49 |
| tonyb | fungi, clarkb ^^ | 22:49 |
| *** thorst has joined #openstack-infra | 22:53 | |
| *** thorst has quit IRC | 22:53 | |
| *** jascott1_ has quit IRC | 22:59 | |
| *** jascott1 has joined #openstack-infra | 23:00 | |
| mnaser | jeblair https://review.openstack.org/#/c/508713 -- could be *really* interesting to look at this sort of failure... "Job base not defined" ? :\ | 23:00 |
| *** jascott1 has quit IRC | 23:04 | |
| *** thorst has joined #openstack-infra | 23:06 | |
| *** tosky has quit IRC | 23:11 | |
| *** yamamoto has joined #openstack-infra | 23:12 | |
| *** thorst has quit IRC | 23:14 | |
| SpamapS | mnaser: yeah that is quite interesting. | 23:15 |
| *** markvoelker has quit IRC | 23:15 | |
| SpamapS | since obviously base is defined in project-config | 23:16 |
| *** yamamoto has quit IRC | 23:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make legacy-swift-tox-xfs-tmp-func-ec voting https://review.openstack.org/508589 | 23:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove broken openstack-tox-pep8 variant https://review.openstack.org/508542 | 23:32 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Rename new releasenotes job https://review.openstack.org/508713 | 23:40 |
| *** lnxnut has joined #openstack-infra | 23:40 | |
| *** caphrim007 has joined #openstack-infra | 23:42 | |
| *** edmondsw has joined #openstack-infra | 23:43 | |
| *** Sukhdev has quit IRC | 23:44 | |
| *** caphrim007 has quit IRC | 23:47 | |
| *** edmondsw has quit IRC | 23:48 | |
| *** adisky has joined #openstack-infra | 23:52 | |
| jeblair | mnaser: that change was re-enqueued during a reconfiguration while it was still waiting for its own configuration to be read from the git repo. i'm not sure what would cause that, but when we have a moment, that's probably where i'd start looking | 23:55 |
| jeblair | i'm going to do some stuff in gdb now; that's probably going to cause all of the jobs to abort, sorry. | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!