| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Aggregate OpenStack-Ansible required projects in a single base job https://review.openstack.org/512126 | 00:00 |
|---|---|---|
| pabelanger | mnaser: wait | 00:00 |
| pabelanger | why are you using base-test? | 00:00 |
| dmsimard | odyssey4me, cloudnull, evrardjp, logan- ^ this probably resolves broken OSA gate jobs but I'd like OSA folks to chime in | 00:01 |
| *** felipemonteiro has quit IRC | 00:02 | |
| mnaser | pabelanger that was an oopsie which i later fixed in a follow-up change on master (which i was going to backport to stable branches afterwards) | 00:02 |
| *** bobh has joined #openstack-infra | 00:02 | |
| mnaser | maybe that could be related | 00:03 |
| mnaser | let me fix that in the backport as well | 00:03 |
| mnaser | and see what gives | 00:03 |
| pabelanger | yah, I'm going to stop looking until jeblair returns. I'm just guessing at things now | 00:04 |
| pabelanger | mnaser: if you haven't done so, please update zuulv3-issues on etherpad | 00:05 |
| pabelanger | so we can properly track this | 00:05 |
| jeblair | back | 00:05 |
| pabelanger | yay | 00:05 |
| mnaser | pabelanger im going to push them back up with base-test removed | 00:07 |
| sc` | chef jobs seem to work. it just. won't. die. | 00:08 |
| pabelanger | sc`: that's good right? | 00:08 |
| sc` | apparently! | 00:08 |
| mnaser | welp, still happened | 00:10 |
| mnaser | ill add it to zuul issues | 00:10 |
| jeblair | mnaser: link to change without base-test? | 00:10 |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=e6ee304be4974c3da305370b5ee988cd&logfile=console.log | 00:10 |
| mnaser | if you can click super quick | 00:11 |
| mnaser | otherwise gonna have to wait for it to hit retry_limit | 00:11 |
| jeblair | mnaser: well, i meant the change | 00:11 |
| mnaser | oh, sure one sec | 00:11 |
| mnaser | jeblair: https://review.openstack.org/#/c/509202/ https://review.openstack.org/#/c/509204/ https://review.openstack.org/#/c/509205/ | 00:11 |
| *** baoli has joined #openstack-infra | 00:11 | |
| mnaser | (all same issue) | 00:11 |
| *** felipemonteiro_ has quit IRC | 00:14 | |
| *** baoli has quit IRC | 00:16 | |
| pabelanger | http://logs.openstack.org/27/512027/1/gate/legacy-grenade-dsvm-neutron/c131ea1/ just failed with POST_FAILURE looking | 00:16 |
| jeblair | i'm going to dig into the multi-pre thing | 00:17 |
| pabelanger | remove-build-sshkey : Remove the build SSH key from all nodes - unreachable | 00:17 |
| pabelanger | that was citycloud | 00:17 |
| pabelanger | which is odd | 00:19 |
| pabelanger | becaue post-logs playbook worked | 00:19 |
| pabelanger | because* | 00:19 |
| pabelanger | I wonder if we should consider adding retries to ansible.cfg | 00:20 |
| pabelanger | http://docs.ansible.com/ansible/latest/intro_configuration.html#retries | 00:20 |
| ianw | i'm going to move the hardlink issues from needs triage to in progress | 00:21 |
| ianw | just to keep that clear-ish | 00:23 |
| pabelanger | mordred: odyssey4me: cloudnull: do you remember where we added the SSH tweaks for ansible went? | 00:23 |
| pabelanger | mordred: odyssey4me: cloudnull: I think there was some recommendations based on cloud1 | 00:23 |
| *** slaweq has quit IRC | 00:23 | |
| *** slaweq has joined #openstack-infra | 00:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Enabled ssh retries for ansible https://review.openstack.org/512130 | 00:30 |
| *** baoli has joined #openstack-infra | 00:32 | |
| *** gildub has joined #openstack-infra | 00:32 | |
| *** psachin has joined #openstack-infra | 00:34 | |
| *** Goneri has joined #openstack-infra | 00:38 | |
| jeblair | mnaser: i've reproduced the issue in a test case; i'm assuming it's a zuul bug; will continue to dig | 00:38 |
| mnaser | jeblair cool, i guess that's good in some ways :) | 00:39 |
| jeblair | yeah, i'm pretty happy as long as i can make a test case :) | 00:40 |
| *** huanxie has joined #openstack-infra | 00:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add statsd_host to zuulv3.o.o, zuul-mergers and zuul-executors https://review.openstack.org/512131 | 00:41 |
| pabelanger | jeblair: not to distract, but I believe we also need ^ for statsd on zuulv3 things | 00:41 |
| pabelanger | I just check zuul.conf on zuulv3.o.o and statsd wasn't configured yet | 00:42 |
| jeblair | pabelanger: oh i thought that was set, thanks | 00:45 |
| pabelanger | I think I see an issue with neutron releasenotes jobs, looking at that now | 00:46 |
| *** ijw has joined #openstack-infra | 00:48 | |
| *** sdague has quit IRC | 00:50 | |
| fungi | okay, i'm done deleting the ~60 verify -2 votes from "jenkins" on open changes now | 00:50 |
| fungi | gonna knock off for the evening and prepare for an exciting day of zuul v3 tomorrow | 00:51 |
| fungi | night all! great work on the rollout | 00:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix root-marker for releasenotes publish job https://review.openstack.org/512132 | 00:52 |
| * clarkb had to put dinner in the oven | 00:53 | |
| *** ijw_ has joined #openstack-infra | 00:54 | |
| pabelanger | okay, ^ should fix releasenotes publishing jobs | 00:54 |
| pabelanger | added to zuulv3-issues | 00:54 |
| *** ijw__ has joined #openstack-infra | 00:55 | |
| *** baoli has quit IRC | 00:56 | |
| *** ijw has quit IRC | 00:56 | |
| *** bobh has quit IRC | 00:56 | |
| *** armax has joined #openstack-infra | 00:57 | |
| *** ijw_ has quit IRC | 00:58 | |
| *** kiennt26 has joined #openstack-infra | 00:59 | |
| *** slaweq has quit IRC | 01:00 | |
| pabelanger | Hmm | 01:02 |
| pabelanger | SSH Error: data could not be sent to remote host "146.20.110.99". Make sure this host can be reached over ssh | 01:02 |
| pabelanger | that is another job | 01:02 |
| pabelanger | http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/ | 01:02 |
| *** baoli has joined #openstack-infra | 01:02 | |
| pabelanger | I'm not sure why we are seeing an increase | 01:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix root-marker for releasenotes publish job https://review.openstack.org/512132 | 01:04 |
| *** slaweq has joined #openstack-infra | 01:04 | |
| *** LindaWang has joined #openstack-infra | 01:04 | |
| mnaser | pabelanger i have an idea about the possibility of a cause | 01:05 |
| pabelanger | could be networking coming out of rackspace | 01:05 |
| mnaser | related to unbound issue, timing out ssh-ing because DNS resolution? | 01:05 |
| clarkb | pabelanger: the host key changed in that log and we have strict host key checking | 01:05 |
| pabelanger | clarkb: oh, really? | 01:05 |
| clarkb | pabelanger: ya its in the log you linked | 01:05 |
| pabelanger | Oh, wow | 01:06 |
| clarkb | http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/job-output.txt.gz#_2017-10-16_00_59_22_003572 | 01:06 |
| pabelanger | I see that now | 01:06 |
| mnaser | "Resolving apt.puppetlabs.com (apt.puppetlabs.com)... failed: Temporary failure in name resolution." boo :< | 01:06 |
| pabelanger | clarkb: but we did copy logs off in a later task | 01:06 |
| pabelanger | mnaser: log? | 01:06 |
| mnaser | http://logs.openstack.org/10/511910/2/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/d97bdb2/ | 01:07 |
| clarkb | the task before it did package updates | 01:07 |
| pabelanger | clarkb: maybe this is the issue of a node coming back online for a moment, then disappering | 01:07 |
| mnaser | looks like it ran on our cloud, but still strange | 01:07 |
| clarkb | pabelanger: or multiple nodes with the same IP? | 01:07 |
| clarkb | or package updates restarting services? | 01:07 |
| pabelanger | clarkb: it is rax-iad | 01:07 |
| clarkb | so maybe the ssh service wasn'tthere and it got a "null" host key? | 01:07 |
| pabelanger | clarkb: no, because http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/job-output.txt.gz#_2017-10-16_00_57_06_235134 would fail | 01:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ignore parents on job variants https://review.openstack.org/512134 | 01:08 |
| *** baoli has quit IRC | 01:08 | |
| jeblair | mnaser: ^ | 01:08 |
| clarkb | Adding debian:TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem that is an awesome cert name btw | 01:08 |
| jeblair | wow that pretty much hosed my irc client | 01:09 |
| pabelanger | mnaser: so that failure is in vexxhost, and I did see an DNS issue today from them. | 01:09 |
| clarkb | pabelanger: the package updates happen afterwards though | 01:09 |
| clarkb | jeblair: sorry (but that makes it even more awesome I think | 01:09 |
| *** bobh has joined #openstack-infra | 01:09 | |
| *** thorst has quit IRC | 01:09 | |
| mnaser | pabelanger weird. i can resolve that with no problems on another vm here (granted it doesn't run unbound to do the recursive resolve) | 01:10 |
| jeblair | mnaser, pabelanger: ^ there's the fix for the multiple pre thing | 01:10 |
| pabelanger | clarkb: yah, but we do run another pre playbook after the updates | 01:10 |
| * mnaser looks | 01:10 | |
| clarkb | pabelanger: that is the one that failed | 01:10 |
| jeblair | mnaser: you can get the correct behavior now by dropping the 'parent' lines from the jobs you're adding to the stable branch | 01:10 |
| pabelanger | mnaser: it is possible related to unbound, ianw and dmsimard have been working on that | 01:10 |
| pabelanger | clarkb: no, run playbook fails I think | 01:11 |
| jeblair | mnaser: (when that fix lands, they'll be ignored anyway, and eventually, we'll figure out how to get zuul to emit warnings suggesting you remove them) | 01:11 |
| clarkb | pabelanger: http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/job-output.txt.gz#_2017-10-16_00_59_19_546353 its bindep then error | 01:11 |
| *** cuongnv has joined #openstack-infra | 01:11 | |
| ianw | pabelanger: we can *probably* put that setup back in ... depends if it's more risky than the problem at hand :) | 01:12 |
| pabelanger | clarkb: wow, I think that log is truncated | 01:13 |
| mnaser | jeblair does that mean that if we wanted a base per branch, we'd need to have a different name of parent in each branch? i was hoping each branch would maintain its own fully self-contained zuul.yaml so when we branch out a release, it keeps its own jobs without us possibly breaking them inadvertently by making a change to a base | 01:13 |
| pabelanger | clarkb: see: http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/ara/ | 01:13 |
| pabelanger | clarkb: oh wait | 01:13 |
| pabelanger | clarkb: Now I understand | 01:13 |
| jeblair | mnaser: no, you can still have the same config -- the branch | 01:13 |
| jeblair | blah | 01:13 |
| clarkb | pabelanger: ya its run.yaml's bindep install that breaks | 01:14 |
| jeblair | mnaser: you can still have the same jobs on all the branches -- the ones on the branches just become "variants" | 01:14 |
| clarkb | or rather the next one after that? | 01:14 |
| pabelanger | clarkb: very odd | 01:14 |
| clarkb | I wouldn't be too surprised if it is part of package updates racing somehow | 01:14 |
| jeblair | mnaser: it is intended to work the way you expect; i just didn't have a complex enough test case to catch the issue with the variants having parents | 01:15 |
| clarkb | though openssh-server is not in the package list | 01:15 |
| jeblair | mnaser: what you won't be able to do is change the inheritance structure across branches. if you want to do that, you'll need to make new jobs. | 01:15 |
| pabelanger | clarkb: it' like we lose the node during bindep, like you said | 01:15 |
| pabelanger | clarkb: I do see openssl | 01:16 |
| pabelanger | but, should be an issue | 01:16 |
| pabelanger | shouldn't* | 01:16 |
| jeblair | pabelanger, clarkb: some kinds of package updates restart sshd | 01:17 |
| jeblair | ie glibc | 01:17 |
| clarkb | jeblair: ya I'm guessing it is something like that, maybe openssl (thought I thought openssh was completely its own implementation) | 01:17 |
| ianw | do we have any best practices around base-test, or just submit a [DNM] something | 01:17 |
| jeblair | ianw: that's the best practice | 01:17 |
| pabelanger | jeblair: clarkb: yah, so possible we might be seeing more until new DIBs are uploaded | 01:18 |
| clarkb | existing connections should remain though, I thought we used the connection manager pool thing? | 01:18 |
| pabelanger | time will tell I think | 01:18 |
| clarkb | eg why is ansible making a new connectio nat all? | 01:18 |
| pabelanger | clarkb: right, that is what I am confused about too | 01:18 |
| jeblair | clarkb: yes, we use control persist | 01:18 |
| mnaser | jeblair makes sense, that's fine by me in that case, makes complete sense, thank you! | 01:18 |
| pabelanger | but even if sshd restarted, why would we get new hostkey for 1 task, then work again for another | 01:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [dnm] parent integration test to base-test https://review.openstack.org/512135 | 01:21 |
| clarkb | rereading it it says data could not be sent to the remote | 01:21 |
| *** yamamoto has joined #openstack-infra | 01:21 | |
| clarkb | then the next thing is the host key thing | 01:21 |
| clarkb | as if the connection actually is closed by the remote side, then we retry to connect and that fails | 01:21 |
| pabelanger | yah | 01:21 |
| *** ijw__ has quit IRC | 01:23 | |
| clarkb | it does Running hooks in /etc/ca-certificates/update.d... then the bindep listing for test | 01:23 |
| clarkb | I wonder if it is a package in test breaking it? and we don't see it because th econnection borks before we can read that data | 01:24 |
| clarkb | probably the thing to do is run that command on a booted rax-iad instance and see what happens to ssh | 01:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Set SETUP_PIP=false for tools/prep-apply.sh https://review.openstack.org/512137 | 01:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Document base job change procedure https://review.openstack.org/512138 | 01:25 |
| jeblair | ianw: ^ | 01:25 |
| *** wewe0901 has joined #openstack-infra | 01:25 | |
| *** baoli has joined #openstack-infra | 01:26 | |
| * jeblair -> dinner | 01:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [dnm] parent integration test to base-test https://review.openstack.org/512135 | 01:27 |
| ianw | jeblair: cool ^ is sort of testing a role-back, so we could mention the integration tests/match as being good targets, but probably not necessary for new commits | 01:30 |
| ianw | roll back even | 01:31 |
| clarkb | pabelanger: maybe you know the anwer, which job has a variant in the test for the multi pre fix change? | 01:31 |
| *** Goneri has quit IRC | 01:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Set SETUP_PIP=false for tools/prep-apply.sh https://review.openstack.org/512137 | 01:31 |
| clarkb | oh is it implicit because of the branch? | 01:31 |
| clarkb | that is quite magical | 01:32 |
| clarkb | ianw: how goes es07? | 01:32 |
| *** daidv has joined #openstack-infra | 01:33 | |
| ianw | volume is 69.1% moved | 01:33 |
| ianw | as discussed, when done i'll remove the ssd, reboot for good measure and then bring it up | 01:33 |
| clarkb | cool thanks | 01:34 |
| pabelanger | clarkb: in puppet-integrations, I believe name: puppet-module-base is the job that tests the problem | 01:34 |
| ianw | pabelanger: ^^^ should you check for zuul vars? | 01:34 |
| pabelanger | ianw: zuul vars? | 01:35 |
| ianw | pabelanger: sorry, before setting it to not install pip, just in case it's used externally? | 01:35 |
| pabelanger | ianw: oh, yah. Maybe we should | 01:35 |
| ianw | i dunno, google seems to suggest that's an overblown concern | 01:36 |
| pabelanger | So far, zuulv3 holding up well | 01:37 |
| pabelanger | jobs generally appear to be green too | 01:37 |
| pabelanger | Hmm | 01:37 |
| pabelanger | mnaser: puppet-cinder and puppet-tripleo should be in same change pipeline in gate right? | 01:37 |
| mnaser | pabelanger yeah, i was looking at that going "hmm" | 01:38 |
| mnaser | pretty much most of puppet-* gates with puppet-tripleo and tripleo-heat-templates and networking-bagpipe and a few others | 01:38 |
| *** slaweq has quit IRC | 01:38 | |
| pabelanger | yah, I think are missing queue: tripleo some place | 01:39 |
| pabelanger | let me see if I can find a good job | 01:39 |
| pabelanger | likely tripleo-ci is one | 01:40 |
| *** baoli has quit IRC | 01:41 | |
| pabelanger | mnaser: where did puppet-openstack-integration project-templates go? | 01:43 |
| mnaser | pabelanger moved to .zuul.yaml in-repo | 01:43 |
| *** sbezverk has quit IRC | 01:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create tripleo change queue for gate pipeline https://review.openstack.org/512140 | 01:44 |
| *** sbezverk has joined #openstack-infra | 01:44 | |
| pabelanger | clarkb: mnaser: jeblair: okay, so that should be the first step to create tripleo change queue for gate. | 01:44 |
| *** dhinesh has joined #openstack-infra | 01:45 | |
| pabelanger | mnaser: I think the next step would be to add queue: tripleo to your openstack-puppet-integration template for gate pipeline too | 01:45 |
| mnaser | pabelanger ack, will do that now | 01:45 |
| *** slaweq has joined #openstack-infra | 01:45 | |
| pabelanger | I say, I think because I'd like jeblair or clarkb to comment | 01:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Ignore parents on job variants https://review.openstack.org/512134 | 01:46 |
| mnaser | pabelanger but the thing is i didnt migrate any of the puppet-cinder jobs for example | 01:46 |
| mnaser | well, some of them did get migrated but not all | 01:46 |
| pabelanger | mnaser: right, but they should be using the project-template? | 01:46 |
| *** fanzhang has left #openstack-infra | 01:49 | |
| mnaser | pabelanger remote: https://review.openstack.org/512141 move jobs to tripleo gate queue | 01:50 |
| mnaser | and ... | 01:50 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: move puppet jobs to tripleo queue https://review.openstack.org/512142 | 01:50 |
| openstackgerrit | Merged openstack-infra/system-config master: Add statsd_host to zuulv3.o.o, zuul-mergers and zuul-executors https://review.openstack.org/512131 | 01:55 |
| pabelanger | mnaser: I don't think you need to add linters, but shouldn't hurt anything | 01:55 |
| pabelanger | lets see what others say | 01:55 |
| pabelanger | mnaser: do you mind updating zuulv3-issues etherpad | 01:55 |
| mnaser | pabelanger sure np one moment! | 01:55 |
| mnaser | pabelanger triaged or? | 01:56 |
| mnaser | i'll go for issues with jobs | 01:56 |
| mnaser | pabelanger added | 01:58 |
| *** baoli has joined #openstack-infra | 02:01 | |
| pabelanger | and releasenotes publishing fixed | 02:02 |
| pabelanger | https://docs.openstack.org/releasenotes/puppet-tripleo/.root-marker | 02:02 |
| *** slaweq has quit IRC | 02:17 | |
| *** boden has quit IRC | 02:20 | |
| *** namnh has joined #openstack-infra | 02:26 | |
| *** gouthamr has quit IRC | 02:27 | |
| *** slaweq has joined #openstack-infra | 02:28 | |
| *** dave-mccowan has quit IRC | 02:35 | |
| *** yamamoto_ has joined #openstack-infra | 02:36 | |
| *** baoli has quit IRC | 02:37 | |
| *** adreznec has quit IRC | 02:37 | |
| *** felipemonteiro_ has joined #openstack-infra | 02:38 | |
| *** adreznec has joined #openstack-infra | 02:39 | |
| *** yamamoto has quit IRC | 02:39 | |
| pabelanger | another SSH Error | 02:40 |
| pabelanger | http://logs.openstack.org/15/512115/4/check/openstack-tox-linters/0d6e0a6/job-output.txt.gz#_2017-10-16_02_32_38_909191 | 02:40 |
| pabelanger | this time ssh host key didn't change | 02:40 |
| pabelanger | is there a way we can confirm our controlpersist is still working? | 02:41 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Trove https://review.openstack.org/511149 | 02:42 |
| *** esberglu has quit IRC | 02:43 | |
| clarkb | pabelanger ps -elf and look for control persist option on ssh commands? | 02:44 |
| clarkb | that will at least show if the option is set | 02:45 |
| pabelanger | yah, I do ee | 02:46 |
| pabelanger | ssh -o ControlMaster=auto -o ControlPersist=60s | 02:46 |
| clarkb | I think the sshd must be crashing/stopping in a way that existing connectiona notice | 02:47 |
| clarkb | then if it reconnects fast enough we may see the host key issue but I bet thats mostly a race and not 100% based on how fast it reconnects | 02:47 |
| pabelanger | if that is the case, maybe https://review.openstack.org/512130/ will help | 02:49 |
| pabelanger | but surprising we are seeing it happen more with zuulv3 | 02:49 |
| pabelanger | maybe because we are running more ansible-playbook invokes per job | 02:49 |
| clarkb | ya I think that may be it | 02:50 |
| pabelanger | there also is some SSH settings that cloud1 ran for ansible, would be intersting to find them again and maybe test | 02:50 |
| *** gcb has joined #openstack-infra | 02:51 | |
| jeblair | controlpersist doesn't span playbook invocations | 02:52 |
| jeblair | are we seeing errors when crossing a playbook boundary, or within the same playbook? | 02:52 |
| pabelanger | no, playbook boundary is more | 02:53 |
| pabelanger | I've seen about 5 since going live | 02:53 |
| pabelanger | I've added a few links to zuulv3-issues | 02:53 |
| jeblair | pabelanger: i do not understand the sentence "no, playbook boundary is more" | 02:53 |
| pabelanger | and just posted one above^ | 02:53 |
| pabelanger | I am seeing error more when we cross playbook boundary | 02:54 |
| pabelanger | then in same playbook | 02:54 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Remove v3 devstack job https://review.openstack.org/512027 | 02:54 |
| jeblair | pabelanger: any within the same playbook? | 02:54 |
| pabelanger | jeblair: just 1 so far | 02:54 |
| pabelanger | let me get link | 02:54 |
| pabelanger | http://logs.openstack.org/37/509337/2/check/legacy-interop-tox-doc8/dbe1920/job-output.txt.gz#_2017-10-16_00_59_19_817729 | 02:55 |
| pabelanger | that was the ssh host key changed issue | 02:55 |
| *** bobh has quit IRC | 02:57 | |
| pabelanger | jeblair: what would we need to change to have controlpersist span playbooks? | 02:57 |
| jeblair | pabelanger: https://storyboard.openstack.org/#!/story/2001072 | 02:57 |
| pabelanger | jeblair: Ah, thanks. Now I remember | 02:58 |
| *** slaweq has quit IRC | 02:58 | |
| *** gongysh has joined #openstack-infra | 02:58 | |
| jeblair | pabelanger: are you still suspecting that the error you liked above might be due to a package install, or have you eliminated that possibility? | 02:59 |
| *** slaweq has joined #openstack-infra | 02:59 | |
| pabelanger | jeblair: I'm not sure myself, I haven't really digged into it yet. | 02:59 |
| *** udesale has joined #openstack-infra | 03:00 | |
| jeblair | pabelanger: okay, so it's possible there are two problems | 03:00 |
| *** mrhillsman has quit IRC | 03:00 | |
| jeblair | and that spending time on making controlpersist span playbooks may help with one of them | 03:01 |
| *** mrhillsman has joined #openstack-infra | 03:01 | |
| pabelanger | yes, I think clarkb is suggesting legacy-interop-tox-doc8 issue was connection was closed on host side for some reason, which could be package update, but I am not sure how host key could change | 03:02 |
| clarkb | I'm not sure the host key actually changed could just be it wasnt read as expeceted due to the service restarta | 03:02 |
| jeblair | clarkb: sounds plausible | 03:03 |
| *** gildub has quit IRC | 03:03 | |
| jeblair | i'll poke at persistent controlpersist for a few mins | 03:03 |
| pabelanger | great | 03:05 |
| ianw | #status log elasticsearch07.o.o rebooted & elasticsearch started. data was migrated from SSD storage and "main" vg contains only one block device now | 03:05 |
| openstackstatus | ianw: finished logging | 03:05 |
| ianw | clarkb: ^ | 03:05 |
| ianw | i've detached & removed the other volume | 03:06 |
| clarkb | tyty | 03:06 |
| clarkb | did it come up quickly after reboot? | 03:06 |
| ianw | status is still "red" but it's pretty busy | 03:06 |
| clarkb | not slow with timeouts like last one? | 03:06 |
| ianw | yep, came up normally | 03:06 |
| clarkb | ya I expect it will be red until it notices the old indexes it had have since been deleted | 03:06 |
| clarkb | so that isnt entirely surprising | 03:07 |
| *** thorst has joined #openstack-infra | 03:10 | |
| *** thorst has quit IRC | 03:15 | |
| *** masber has joined #openstack-infra | 03:20 | |
| *** felipemonteiro_ has quit IRC | 03:22 | |
| dmsimard | clarkb, ianw: fyi I successfully tested a middleware for loading sqlite databases on the fly https://review.openstack.org/#/c/511992/ | 03:26 |
| *** esberglu has joined #openstack-infra | 03:26 | |
| dmsimard | I'm just waiting on an OSA gate job to finish to retrieve it's database (which is a good benchmark) to put numbers and perspective | 03:26 |
| *** chlong has joined #openstack-infra | 03:29 | |
| *** esberglu has quit IRC | 03:30 | |
| *** bobh has joined #openstack-infra | 03:34 | |
| *** wewe0901 has quit IRC | 03:35 | |
| ianw | i do get the heeby-jeebies when i see os.path.join on a provided URI :) | 03:36 |
| dmsimard | ianw: it's not like I took that from your downloader patch or anything :p | 03:38 |
| dmsimard | happy to run it through some security improvements, this is just a proof of concept to show -- hey, we can do this and it works, let's do it properly though | 03:39 |
| dmsimard | ianw: it's heavily inspired by https://review.openstack.org/#/c/122615/ :D | 03:40 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Revert "Add configure-unbound to base job"" https://review.openstack.org/512153 | 03:40 |
| dmsimard | you get all the credit :) | 03:41 |
| ianw | 2014 seems to be both so long ago, and actually not that long ago, both at the same time! | 03:42 |
| dmsimard | ianw: ah yeah you have https://review.openstack.org/#/c/122615/2/os_loganalyze/downloader.py@83 to avoid people poking out | 03:43 |
| *** bobh has quit IRC | 03:44 | |
| *** udesale has quit IRC | 03:46 | |
| ianw | pabelanger: ^^^ that's the unbound restore, i'll add some notes in there, but I'll also mark it WIP to avoid merge. just thought we should have it primed in case these ssh issues turn out to be related | 03:46 |
| *** bobh has joined #openstack-infra | 03:48 | |
| *** ykarel has joined #openstack-infra | 03:52 | |
| ianw | for something like -- http://logs.openstack.org/70/504870/5/infra-check/project-config-irc-access/f3625f9/ -- we haven't really logged anywhere what node it ran on, or what provider? i'd really like that as a debug statement right at the top | 03:52 |
| *** bobh has quit IRC | 03:53 | |
| *** edmondsw has joined #openstack-infra | 03:53 | |
| *** felipemonteiro_ has joined #openstack-infra | 03:55 | |
| *** coolsvap has joined #openstack-infra | 03:57 | |
| dmsimard | ianw: WIP | 03:57 |
| *** edmondsw has quit IRC | 03:57 | |
| dmsimard | ianw: see https://review.openstack.org/#/c/511948/ and https://review.openstack.org/#/c/511821/ | 03:57 |
| dmsimard | (and the child of https://review.openstack.org/#/c/511821/) | 03:58 |
| ianw | ++ that's exactly what i wanted :) | 03:59 |
| dmsimard | ianw: it doesn't give the cloud provider, maybe we can add that as follow up. | 03:59 |
| *** udesale has joined #openstack-infra | 03:59 | |
| ianw | hostname is enough to figure that out | 04:00 |
| *** udesale has quit IRC | 04:00 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| dmsimard | ianw: not in v3 anymore ? nodes can be given any name | 04:03 |
| *** dhinesh has quit IRC | 04:04 | |
| *** slaweq has quit IRC | 04:05 | |
| clarkb | thats an inventory thing but hostname should be the nodepool name | 04:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Start ssh persistent connections outside bwrap https://review.openstack.org/512158 | 04:09 |
| jeblair | oy | 04:09 |
| jeblair | pabelanger, clarkb, ianw: ^ i *think* that will work. | 04:10 |
| jeblair | jlk, SpamapS, mordred: ^ | 04:10 |
| *** claudiub has joined #openstack-infra | 04:10 | |
| *** dhinesh has joined #openstack-infra | 04:10 | |
| jeblair | that has depleted my brain for the evening. good night! | 04:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Start ssh persistent connections outside bwrap https://review.openstack.org/512158 | 04:12 |
| *** felipemonteiro__ has joined #openstack-infra | 04:15 | |
| *** dhinesh has quit IRC | 04:15 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Disable sql_reporter alembic downgrades https://review.openstack.org/512159 | 04:16 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Clarify how the sql database is managed https://review.openstack.org/512160 | 04:16 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Move alembic_reporter back to alembic https://review.openstack.org/512161 | 04:16 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Allow the pool_recycle to be configured https://review.openstack.org/512162 | 04:16 |
| dmsimard | ianw: I added some numbers for perspective on sqlite middleware: https://review.openstack.org/#/c/511992/ | 04:16 |
| dmsimard | bed time for me, catch ya later | 04:17 |
| *** slaweq has joined #openstack-infra | 04:17 | |
| *** felipemonteiro_ has quit IRC | 04:18 | |
| *** huanxie has quit IRC | 04:18 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Freezer https://review.openstack.org/511151 | 04:18 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Freezer https://review.openstack.org/511152 | 04:18 |
| *** huanxie has joined #openstack-infra | 04:24 | |
| *** hongbin_ has quit IRC | 04:24 | |
| *** felipemonteiro__ has quit IRC | 04:27 | |
| SamYaple | is there anyway to get the post logs from a commit after its merged (not in real time) | 04:28 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Clarify how the sql database is managed https://review.openstack.org/512160 | 04:32 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Move alembic_reporter back to alembic https://review.openstack.org/512161 | 04:32 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Allow the pool_recycle to be configured https://review.openstack.org/512162 | 04:32 |
| clarkb | SamYaple: http://logs.openstack.org/firsttwoofsha1/sha1 | 04:33 |
| SamYaple | clarkb: and the sha1 is the git commit id as seen by the "commit" field on the review? | 04:35 |
| clarkb | I think it may be the merge commit if one was genwrated | 04:37 |
| clarkb | but otherwise yes | 04:38 |
| *** jascott1 has quit IRC | 04:41 | |
| SamYaple | perfect. i can figure it out from one of those. thanks! | 04:42 |
| *** Dinesh_Bhor has joined #openstack-infra | 04:42 | |
| SamYaple | my first dockerhub-pushing POST job is running. wish me luck | 04:45 |
| cloudnull | pabelanger: IDK if you had already found this but we use the following SSH tweaks within OSA https://github.com/openstack/openstack-ansible/blob/master/group_vars/all/all.yml#L107-L117 | 04:47 |
| *** slaweq has quit IRC | 04:48 | |
| *** sree has joined #openstack-infra | 04:48 | |
| *** armaan has joined #openstack-infra | 04:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Import legacy playbooks https://review.openstack.org/512164 | 04:52 |
| ykarel | Hi, is there a way to reuse the values from a argument from base job in zuulv3, For example i want to reuse irrelevant-files from base job and add additional in child jobs. | 04:53 |
| *** kiennt26 has quit IRC | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Remove dib legacy playbooks https://review.openstack.org/512166 | 04:54 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul feature/zuulv3: Allow the pool_recycle to be configured https://review.openstack.org/512162 | 04:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove diskimage-builder functional test jobs https://review.openstack.org/512169 | 04:57 |
| *** links has joined #openstack-infra | 04:57 | |
| *** markvoelker has quit IRC | 04:58 | |
| *** markvoelker has joined #openstack-infra | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Remove dib legacy playbooks https://review.openstack.org/512166 | 04:58 |
| *** jascott1 has joined #openstack-infra | 04:59 | |
| *** slaweq has joined #openstack-infra | 04:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move bashate jobs in-tree https://review.openstack.org/512118 | 05:00 |
| ykarel | Hi Team, one more why check and gate job defined in .zuul.yml in magnum repo didn't run in https://review.openstack.org/#/c/488581/ (stable/newton) patch, can someone guide where to look | 05:00 |
| ykarel | is jobs defined in master branch will not work for newton? | 05:01 |
| SamYaple | hmmm. its seems my secret is not accessible via the post job http://logs.openstack.org/1d/1d7787eff61c898051833a14a4f59dd88a1c3ab9/post/publish-loci-keystone/c5d4128/job-output.txt.gz#_2017-10-16_04_52_26_262861 | 05:01 |
| SamYaple | it is declared here https://github.com/openstack/loci/blob/master/.zuul.yaml#L174 | 05:01 |
| SamYaple | and used here https://github.com/openstack/loci/blob/master/playbooks/post.yaml#L38 | 05:01 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-windmill jobs https://review.openstack.org/512110 | 05:02 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: move puppet jobs to tripleo queue https://review.openstack.org/512142 | 05:03 |
| SamYaple | is the secret not available in the post-run part of the job? | 05:03 |
| *** armaan has quit IRC | 05:05 | |
| *** armax has quit IRC | 05:07 | |
| *** armax has joined #openstack-infra | 05:07 | |
| *** armax has quit IRC | 05:07 | |
| *** armax has joined #openstack-infra | 05:08 | |
| *** armax has quit IRC | 05:08 | |
| *** gongysh has quit IRC | 05:09 | |
| *** armax has joined #openstack-infra | 05:09 | |
| *** armax has quit IRC | 05:09 | |
| *** armax has joined #openstack-infra | 05:10 | |
| *** armax has quit IRC | 05:10 | |
| *** dhajare has joined #openstack-infra | 05:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove diskimage-builder functional test jobs https://review.openstack.org/512169 | 05:10 |
| *** armax has joined #openstack-infra | 05:10 | |
| *** armax has quit IRC | 05:11 | |
| *** dhinesh has joined #openstack-infra | 05:11 | |
| *** armax has joined #openstack-infra | 05:11 | |
| *** armax has quit IRC | 05:11 | |
| *** thorst has joined #openstack-infra | 05:12 | |
| *** armax has joined #openstack-infra | 05:12 | |
| *** armax has quit IRC | 05:12 | |
| *** armax has joined #openstack-infra | 05:13 | |
| *** armax has quit IRC | 05:13 | |
| *** armax has joined #openstack-infra | 05:14 | |
| *** armax has quit IRC | 05:14 | |
| *** armax has joined #openstack-infra | 05:14 | |
| *** armax has quit IRC | 05:15 | |
| *** dhinesh has quit IRC | 05:15 | |
| *** thorst has quit IRC | 05:16 | |
| snapiri | Hi Infra, I have some issue with different builds running on zuul. When the build runs on a machine with specific filesystem, the build fails with zuul-cloner issues (e.g. http://logs.openstack.org/96/512096/1/check/legacy-dragonflow-dsvm-fullstack/5670008/job-output.txt.gz#_2017-10-15_20_51_17_069037). When the filesystem is different (I believe it is a different hypervisor) it succeeds and passes the zuul-cloner stage with no | 05:24 |
| snapiri | issue. | 05:24 |
| snapiri | When the filesystem is as follows (http://logs.openstack.org/96/512096/1/check/legacy-dragonflow-dsvm-rally/fe8384c/logs/df.txt.gz) it seems to pass the zuul-cloner stage with no problem | 05:25 |
| ianw | snapiri: i think this is more likely related to http://logs.openstack.org/96/512096/1/check/legacy-dragonflow-dsvm-fullstack/5670008/job-output.txt.gz#_2017-10-15_20_51_12_604631 | 05:27 |
| *** gildub has joined #openstack-infra | 05:27 | |
| ianw | you're running that as "stack" but those repos will have been cloned by zuul user | 05:27 |
| ianw | does it really need to be stack? | 05:28 |
| ianw | as a first step, you can probably try replacing that with $USER | 05:28 |
| snapiri | ianw: thanks, will try | 05:28 |
| *** huanxie has quit IRC | 05:32 | |
| *** slaweq has quit IRC | 05:32 | |
| *** slaweq has joined #openstack-infra | 05:34 | |
| *** ramishra has joined #openstack-infra | 05:35 | |
| *** kaisers1 has joined #openstack-infra | 05:36 | |
| *** gongysh has joined #openstack-infra | 05:37 | |
| openstackgerrit | zhangfei gao proposed openstack/diskimage-builder master: Fix grub2 dependency on arm64 https://review.openstack.org/512174 | 05:40 |
| openstackgerrit | zhangfei gao proposed openstack/diskimage-builder master: Add debian minimal requirement for arm64 https://review.openstack.org/512175 | 05:40 |
| *** edmondsw has joined #openstack-infra | 05:41 | |
| *** edmondsw has quit IRC | 05:45 | |
| *** markvoelker has quit IRC | 05:46 | |
| *** armaan has joined #openstack-infra | 05:47 | |
| snapiri | ianw: will that not be a problem as the /opt/stack is owned by stack... see http://logs.openstack.org/96/512096/1/check/legacy-dragonflow-dsvm-fullstack/5670008/job-output.txt.gz#_2017-10-15_20_22_56_260035 | 05:53 |
| snapiri | shouldn't everything run as the user stack? | 05:54 |
| *** huanxie has joined #openstack-infra | 06:01 | |
| *** slaweq has quit IRC | 06:07 | |
| *** Guest5873 has quit IRC | 06:08 | |
| *** pgadiya has joined #openstack-infra | 06:09 | |
| *** pgadiya has quit IRC | 06:09 | |
| *** slaweq has joined #openstack-infra | 06:10 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make statsd import failure raise exception https://review.openstack.org/511978 | 06:11 |
| *** dhinesh has joined #openstack-infra | 06:12 | |
| *** dhinesh has quit IRC | 06:16 | |
| *** yamahata has joined #openstack-infra | 06:22 | |
| *** martinkopec has joined #openstack-infra | 06:23 | |
| *** larainema has quit IRC | 06:24 | |
| *** electrical has quit IRC | 06:24 | |
| *** simonmcc has quit IRC | 06:24 | |
| *** berendt has quit IRC | 06:24 | |
| *** serverascode has quit IRC | 06:24 | |
| *** ameade has quit IRC | 06:24 | |
| *** cargonza has quit IRC | 06:24 | |
| *** tomhambleton_ has quit IRC | 06:24 | |
| *** NobodyCam has quit IRC | 06:24 | |
| *** zhongjun has quit IRC | 06:24 | |
| *** zul has quit IRC | 06:24 | |
| *** betherly has quit IRC | 06:24 | |
| *** clayg has quit IRC | 06:24 | |
| *** hrybacki has quit IRC | 06:24 | |
| *** sweston has quit IRC | 06:24 | |
| *** aarefiev has quit IRC | 06:24 | |
| *** csatari has quit IRC | 06:24 | |
| *** kmalloc has quit IRC | 06:24 | |
| *** hogepodge has quit IRC | 06:24 | |
| *** gcb has quit IRC | 06:24 | |
| *** ggherdov- has quit IRC | 06:24 | |
| *** DuncanT has quit IRC | 06:25 | |
| *** TheJulia has quit IRC | 06:25 | |
| *** zxiiro has quit IRC | 06:25 | |
| *** samueldmq has quit IRC | 06:25 | |
| *** fenar has quit IRC | 06:25 | |
| *** vdrok has quit IRC | 06:25 | |
| *** gmann has quit IRC | 06:25 | |
| *** jamespage has quit IRC | 06:25 | |
| *** fmccrthy has quit IRC | 06:25 | |
| *** srwilkers has quit IRC | 06:25 | |
| *** evgenyl has quit IRC | 06:25 | |
| *** mrmartin has quit IRC | 06:25 | |
| *** fmccrthy has joined #openstack-infra | 06:26 | |
| *** srwilkers has joined #openstack-infra | 06:26 | |
| *** clayg has joined #openstack-infra | 06:26 | |
| *** NobodyCam has joined #openstack-infra | 06:26 | |
| *** evgenyl has joined #openstack-infra | 06:26 | |
| *** kmalloc has joined #openstack-infra | 06:26 | |
| *** vdrok has joined #openstack-infra | 06:26 | |
| *** berendt has joined #openstack-infra | 06:26 | |
| *** jamespage has joined #openstack-infra | 06:26 | |
| *** samueldmq has joined #openstack-infra | 06:26 | |
| *** sweston has joined #openstack-infra | 06:26 | |
| *** hogepodge has joined #openstack-infra | 06:26 | |
| *** ameade has joined #openstack-infra | 06:26 | |
| *** cargonza has joined #openstack-infra | 06:26 | |
| *** csatari has joined #openstack-infra | 06:26 | |
| *** ryotagami has quit IRC | 06:26 | |
| *** hrybacki has joined #openstack-infra | 06:26 | |
| *** serverascode has joined #openstack-infra | 06:26 | |
| *** calbers has quit IRC | 06:26 | |
| *** ying_zuo has quit IRC | 06:26 | |
| *** electrical has joined #openstack-infra | 06:26 | |
| *** fenar has joined #openstack-infra | 06:27 | |
| *** DuncanT has joined #openstack-infra | 06:27 | |
| *** simonmcc has joined #openstack-infra | 06:27 | |
| *** aarefiev has joined #openstack-infra | 06:27 | |
| *** TheJulia has joined #openstack-infra | 06:27 | |
| *** betherly has joined #openstack-infra | 06:27 | |
| *** zxiiro has joined #openstack-infra | 06:27 | |
| *** openstackgerrit has quit IRC | 06:28 | |
| *** zoli has quit IRC | 06:28 | |
| *** Jeffrey4l has quit IRC | 06:28 | |
| *** gcb has joined #openstack-infra | 06:28 | |
| *** calbers has joined #openstack-infra | 06:29 | |
| *** ryotagami has joined #openstack-infra | 06:29 | |
| *** Jeffrey4l has joined #openstack-infra | 06:29 | |
| *** psachin has quit IRC | 06:30 | |
| *** zoli has joined #openstack-infra | 06:31 | |
| *** kiennt26 has joined #openstack-infra | 06:36 | |
| *** openstackgerrit has joined #openstack-infra | 06:36 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Rally https://review.openstack.org/512182 | 06:36 |
| *** esberglu has joined #openstack-infra | 06:38 | |
| *** priya_ has joined #openstack-infra | 06:38 | |
| *** ccamacho has joined #openstack-infra | 06:40 | |
| *** slaweq has quit IRC | 06:42 | |
| *** kjackal_ has joined #openstack-infra | 06:43 | |
| *** esberglu has quit IRC | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:45 | |
| *** psachin has joined #openstack-infra | 06:46 | |
| *** psachin has joined #openstack-infra | 06:46 | |
| *** liusheng has quit IRC | 06:51 | |
| *** liusheng has joined #openstack-infra | 06:52 | |
| ianw | snapiri: going to have to think about it, let me add something to the issues page | 06:52 |
| *** gongysh has quit IRC | 06:54 | |
| *** gongysh has joined #openstack-infra | 06:56 | |
| ianw | see https://etherpad.openstack.org/p/zuulv3-issues | 06:56 |
| *** gridinv has quit IRC | 06:56 | |
| *** ykarel_ has joined #openstack-infra | 06:57 | |
| *** mrmartin has joined #openstack-infra | 06:57 | |
| *** s-shiono has joined #openstack-infra | 06:57 | |
| priya_ | hi , Could somebody review the commit - https://review.openstack.org/#/c/508533/ | 06:58 |
| *** pgadiya has joined #openstack-infra | 06:58 | |
| *** liusheng has quit IRC | 06:59 | |
| *** liusheng has joined #openstack-infra | 06:59 | |
| *** ykarel has quit IRC | 06:59 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Rally legacy jobs https://review.openstack.org/512186 | 07:00 |
| ianw | priya_: reviewers are unlikely to see things here, try #openstack-nova | 07:01 |
| *** claudiub has quit IRC | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove diskimage-builder functional test jobs https://review.openstack.org/512169 | 07:03 |
| *** priya_ has quit IRC | 07:03 | |
| *** florianf has joined #openstack-infra | 07:07 | |
| snapiri | ianw: thanks | 07:07 |
| *** jtomasek has joined #openstack-infra | 07:10 | |
| *** jtomasek has quit IRC | 07:11 | |
| *** jtomasek has joined #openstack-infra | 07:11 | |
| *** gildub has quit IRC | 07:14 | |
| *** claudiub has joined #openstack-infra | 07:14 | |
| *** Hal has joined #openstack-infra | 07:17 | |
| *** Hal is now known as Guest71201 | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:18 | |
| *** markvoelker has quit IRC | 07:18 | |
| *** shardy has joined #openstack-infra | 07:18 | |
| *** slaweq has quit IRC | 07:19 | |
| *** markvoelker has joined #openstack-infra | 07:20 | |
| *** markvoelker has quit IRC | 07:20 | |
| *** markvoelker has joined #openstack-infra | 07:20 | |
| *** gongysh has quit IRC | 07:21 | |
| *** gongysh has joined #openstack-infra | 07:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner-shim: don't link if owners different https://review.openstack.org/512194 | 07:23 |
| ianw | snapiri: i think ^ is the solution, maybe. I don't have time to test it fully right now, but watch out and others can post their ideas on the review/etherpad | 07:23 |
| *** slaweq has joined #openstack-infra | 07:23 | |
| *** edmondsw has joined #openstack-infra | 07:29 | |
| *** yamahata has quit IRC | 07:30 | |
| snapiri | ianw: looks like this should work :) | 07:30 |
| *** toabctl has joined #openstack-infra | 07:32 | |
| *** efried has quit IRC | 07:32 | |
| *** edmondsw has quit IRC | 07:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move bashate jobs in-tree https://review.openstack.org/512118 | 07:35 |
| *** yamamoto_ has quit IRC | 07:35 | |
| *** xinliang has joined #openstack-infra | 07:37 | |
| *** xinliang has quit IRC | 07:37 | |
| *** xinliang has joined #openstack-infra | 07:37 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Networking-cisco https://review.openstack.org/512197 | 07:38 |
| openstackgerrit | Numan Siddique proposed openstack-infra/openstack-zuul-jobs master: Add files section in the legact tripleo ci scenario007 jobs https://review.openstack.org/512007 | 07:38 |
| *** jpich has joined #openstack-infra | 07:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create tripleo change queue for gate pipeline https://review.openstack.org/512140 | 07:39 |
| snapiri | ianw: so I am a bit confused, should I continue to use the zuul-cloner or not? In the migration guide it ways to stop using it. I understand that the required projects are cloned by zuul now, but I should install them, so if zuul brings them, will I still have the same issue when I try to install them as user stack? | 07:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "neutron: Make grenade-neutron-dvr-multinode job non-voting" https://review.openstack.org/505318 | 07:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding check/publish jobs for contributor guide https://review.openstack.org/511946 | 07:41 |
| *** slaweq_ has joined #openstack-infra | 07:44 | |
| *** efried has joined #openstack-infra | 07:44 | |
| *** egonzalez has joined #openstack-infra | 07:47 | |
| *** jascott1 has quit IRC | 07:47 | |
| *** jpena|off is now known as jpena | 07:48 | |
| *** jascott1 has joined #openstack-infra | 07:48 | |
| *** jascott1 has quit IRC | 07:52 | |
| *** bobh has joined #openstack-infra | 07:54 | |
| *** slaweq has quit IRC | 07:57 | |
| *** bobh has quit IRC | 07:58 | |
| *** slaweq has joined #openstack-infra | 08:01 | |
| *** d0ugal_ has quit IRC | 08:01 | |
| *** d0ugal has joined #openstack-infra | 08:01 | |
| *** d0ugal has quit IRC | 08:01 | |
| *** d0ugal has joined #openstack-infra | 08:01 | |
| *** cuongnv_ has joined #openstack-infra | 08:07 | |
| *** namnh_ has joined #openstack-infra | 08:07 | |
| *** cuongnv has quit IRC | 08:07 | |
| *** namnh has quit IRC | 08:08 | |
| *** armaan has quit IRC | 08:08 | |
| *** armaan has joined #openstack-infra | 08:08 | |
| *** panda|rover|off has quit IRC | 08:10 | |
| SamYaple | hmmm. im not able to access my secret. i defined the secret in .zuul.yaml and try to access it in a playbook and it is not defined. any help here? | 08:12 |
| SamYaple | does the secret need to be defined in project-config? and the job? or what? im confused | 08:12 |
| *** derekh has joined #openstack-infra | 08:13 | |
| SamYaple | can secrets be used in untrusted playbooks? | 08:19 |
| SamYaple | it appears not | 08:19 |
| *** dbecker has joined #openstack-infra | 08:19 | |
| *** lucas-afk is now known as lucasagomes | 08:21 | |
| *** AJaeger has joined #openstack-infra | 08:21 | |
| *** tomhambleton_ has joined #openstack-infra | 08:22 | |
| *** gmann has joined #openstack-infra | 08:22 | |
| *** larainema has joined #openstack-infra | 08:22 | |
| *** zhongjun has joined #openstack-infra | 08:22 | |
| AJaeger | SamYaple: project-config is a trusted repository and secrets are only available there. Best discuss with rest of the team later, my understanding is that your post job needs to get defined in project-config. | 08:22 |
| *** ggherdov- has joined #openstack-infra | 08:22 | |
| *** zul has joined #openstack-infra | 08:22 | |
| SamYaple | AJaeger: thats contrary to the zuul docs which say secrets can be defined and used in untrusted jobs | 08:25 |
| SamYaple | (if the secrets are defined in the untrusted repository) | 08:25 |
| SamYaple | i think i found the missing bit that i needed though https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-pipeline.post-review | 08:26 |
| *** esberglu has joined #openstack-infra | 08:26 | |
| *** ying_zuo has joined #openstack-infra | 08:26 | |
| SamYaple | maybe not. it is really late for me | 08:27 |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** s-shiono has quit IRC | 08:27 | |
| *** armaan has quit IRC | 08:28 | |
| *** armaan has joined #openstack-infra | 08:28 | |
| SamYaple | ohhh i think i see it "It is possible to use secrets for jobs defined in config projects as well as untrusted projects," | 08:28 |
| SamYaple | i just need to define the secret in project-config | 08:28 |
| SamYaple | i can consume it in my untrusted job *if* the pipeline has post-review=true | 08:29 |
| SamYaple | such as the post pipeline | 08:29 |
| SamYaple | if that is true, thats wierd. i can see no security benefit from defining a secret in project-config vs the untrusted repository | 08:30 |
| *** esberglu has quit IRC | 08:30 | |
| *** markvoelker has quit IRC | 08:31 | |
| *** slaweq has quit IRC | 08:34 | |
| *** yamamoto has joined #openstack-infra | 08:36 | |
| *** slaweq has joined #openstack-infra | 08:36 | |
| AJaeger | SamYaple: I'm not the expert yet on this, best to ask jeblair later today | 08:36 |
| *** mpeterson has joined #openstack-infra | 08:36 | |
| SamYaple | indeed. still, thanks for the help | 08:37 |
| *** pgadiya has quit IRC | 08:40 | |
| *** yamamoto has quit IRC | 08:42 | |
| *** ccamacho1 has joined #openstack-infra | 08:45 | |
| *** ccamacho has quit IRC | 08:45 | |
| *** markvoelker has joined #openstack-infra | 08:45 | |
| *** ykarel_ is now known as ykarel | 08:46 | |
| *** ccamacho1 has quit IRC | 08:48 | |
| *** ccamacho has joined #openstack-infra | 08:48 | |
| kazsh | AJaeger: thanks for your +2 on https://review.openstack.org/#/c/509119/ Should I ask someone to review this or wait for a while ? | 08:49 |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** bobh has joined #openstack-infra | 08:55 | |
| *** bobh has quit IRC | 08:59 | |
| *** tosky has joined #openstack-infra | 08:59 | |
| *** ykarel is now known as ykarel|lunch | 09:00 | |
| *** yamamoto has joined #openstack-infra | 09:04 | |
| *** kashyap has joined #openstack-infra | 09:04 | |
| *** e0ne has joined #openstack-infra | 09:08 | |
| *** slaweq has quit IRC | 09:08 | |
| *** yamamoto has quit IRC | 09:09 | |
| kashyap | How can one do a 'recheck' on a Zuul v3 job? | 09:09 |
| tosky | kashyap: I don't think that it changed | 09:10 |
| kashyap | tosky: So a 'recheck' should Just Work? | 09:10 |
| tosky | kashyap: yes | 09:10 |
| kashyap | Thanks, asking for a colleague | 09:11 |
| *** slaweq has joined #openstack-infra | 09:13 | |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| AJaeger | kazsh: just wait - we're all busy with Zuul v3 migration, it will get reviewed... | 09:19 |
| kazsh | AJarger: Noted with thanks! | 09:20 |
| AJaeger | yolanda: do you want to put 509119 on your review queue? | 09:20 |
| *** esberglu has joined #openstack-infra | 09:20 | |
| *** esberglu has quit IRC | 09:20 | |
| *** esberglu has joined #openstack-infra | 09:20 | |
| *** esberglu has quit IRC | 09:21 | |
| *** askb has quit IRC | 09:21 | |
| kazsh | Oops sorry s/AJarger/AJaeger | 09:21 |
| openstackgerrit | Tetsuro Nakamura proposed openstack-infra/project-config master: Add new project networking-spp https://review.openstack.org/512223 | 09:22 |
| *** sambetts|afk is now known as sambetts | 09:27 | |
| sambetts | AJaeger: thank you for adding the networking-cisc projects :D | 09:27 |
| andreaf | clarkb, jeblair: ok thanks for the follow up devstack change | 09:29 |
| *** kjackal_ has quit IRC | 09:30 | |
| andreaf | clarkb, jeblair: since now the job definition is spread across a mix of branchless and branched repos, we may have to start backporting jobs parts every now and then | 09:30 |
| andreaf | clarkb, jeblair: for the devstack job/roles we may want to wait until it's ~complete and then backport the whole ansible bits + .zuul.yaml at once? | 09:32 |
| *** dhajare has quit IRC | 09:37 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Networking-cisco legacy jobs https://review.openstack.org/512235 | 09:38 |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: networking-odl: Add required-projects for tox jobs https://review.openstack.org/512238 | 09:39 |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: networking-odl: Add required-projects for functional tests https://review.openstack.org/512238 | 09:42 |
| *** e0ne has quit IRC | 09:45 | |
| *** askb has joined #openstack-infra | 09:45 | |
| *** slaweq has quit IRC | 09:46 | |
| *** kota_ has joined #openstack-infra | 09:47 | |
| kota_ | hello folks. Could i confirm what's happen in zuul v3 migration on the gerrit? | 09:48 |
| kota_ | it looks like to change the gate jobs as sort of 'legacy-' in the commit, https://github.com/openstack-infra/project-config/commit/e87b20f719f691f61e35be1054ccb4fbcf65e433 | 09:49 |
| kota_ | but it seems to cause gate failure in our project | 09:49 |
| kota_ | e.g. https://review.openstack.org/#/c/483798/ | 09:49 |
| *** jascott1 has joined #openstack-infra | 09:50 | |
| *** kiennt26 has quit IRC | 09:50 | |
| kota_ | I'd like to get how i can resolve the failure and gate jobs. | 09:50 |
| kota_ | perhaps, just fix the gate job name to be same with jenkins jobs? | 09:51 |
| *** ykarel|lunch is now known as ykarel | 09:52 | |
| *** Shrews has quit IRC | 09:53 | |
| *** slaweq has joined #openstack-infra | 09:54 | |
| *** Shrews has joined #openstack-infra | 09:55 | |
| tosky | kota_: there were few emails about this topic, the main suggestion is read this: https://docs.openstack.org/infra/manual/zuulv3.html | 09:55 |
| kota_ | tosky: thx | 09:55 |
| *** bobh has joined #openstack-infra | 09:55 | |
| openstackgerrit | Bernard Cafarelli proposed openstack-infra/project-config master: Fix build-openstack-sphinx-docs for networking-sfc https://review.openstack.org/512244 | 09:55 |
| tosky | well, not "few" emails, but many emails | 09:55 |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Allow EOL branches to use the test-matrix.py script https://review.openstack.org/510946 | 09:56 |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Add ability to override project branch to a tag https://review.openstack.org/512245 | 09:56 |
| AJaeger | andreaf: you can backport .zuul.yaml if you want different config on different branches. But you don't have to. Your call - best to discuss later... | 09:56 |
| andreaf | AJaeger: uhm ok - so zuul will always take it from master? | 09:57 |
| andreaf | AJaeger: what about roles and playbooks? Are those taken from master as well? Even for jobs that run on stable branches? | 09:57 |
| AJaeger | andreaf: If you have .zuul.yaml on a branch, it uses that one. IF there's none, it takes the one from master | 09:57 |
| AJaeger | andreaf: best discuss with jeblair and mordred later... Just wanted to point out that you have options ;) | 09:57 |
| andreaf | AJaeger: thanks :) the fallback on master is handy indeed | 09:58 |
| *** derekh has quit IRC | 09:58 | |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: networking-odl: Add required-projects for functional tests https://review.openstack.org/512238 | 09:58 |
| *** namnh_ has quit IRC | 09:58 | |
| kota_ | ah, ok. It looks the legacy one starts to work well, just timed out for too long job. | 09:58 |
| *** Shrews has quit IRC | 09:59 | |
| *** derekh has joined #openstack-infra | 09:59 | |
| *** bobh has quit IRC | 10:00 | |
| *** kjackal_ has joined #openstack-infra | 10:00 | |
| *** Shrews has joined #openstack-infra | 10:01 | |
| *** LindaWang has quit IRC | 10:03 | |
| yolanda | AJaeger, looking | 10:08 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Rework features doc https://review.openstack.org/512248 | 10:08 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Minor rework of usage doc https://review.openstack.org/512249 | 10:08 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Use publish-openstack-sphinx-docs-neutron https://review.openstack.org/512250 | 10:09 |
| *** cuongnv_ has quit IRC | 10:10 | |
| *** esberglu has joined #openstack-infra | 10:14 | |
| *** derekh has quit IRC | 10:15 | |
| openstackgerrit | Bernard Cafarelli proposed openstack-infra/project-config master: Use -neutron template variants for networking-sfc jobs https://review.openstack.org/512244 | 10:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 10:16 |
| *** sdague has joined #openstack-infra | 10:17 | |
| *** esberglu has quit IRC | 10:18 | |
| *** derekh has joined #openstack-infra | 10:20 | |
| *** fanzhang has joined #openstack-infra | 10:21 | |
| *** pbourke has quit IRC | 10:23 | |
| openstackgerrit | Michel Peterson proposed openstack-infra/openstack-zuul-jobs master: networking-odl: Add required-projects for functional tests https://review.openstack.org/512254 | 10:24 |
| *** pbourke has joined #openstack-infra | 10:25 | |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: networking-odl: Add required-projects for functional tests https://review.openstack.org/512238 | 10:25 |
| *** psachin has quit IRC | 10:31 | |
| *** LindaWang has joined #openstack-infra | 10:31 | |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** daidv has quit IRC | 10:38 | |
| *** rbergeron has joined #openstack-infra | 10:38 | |
| *** openstackgerrit has joined #openstack-infra | 10:43 | |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder master: dib-lint: test elements have README.rst file https://review.openstack.org/177832 | 10:43 |
| *** armaan has quit IRC | 10:43 | |
| *** armaan has joined #openstack-infra | 10:44 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Use release-notes-jobs-neutron https://review.openstack.org/512260 | 10:54 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: publish-to-pypi-neutron https://review.openstack.org/512261 | 10:54 |
| *** udesale has quit IRC | 10:54 | |
| *** udesale has joined #openstack-infra | 10:54 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Use publish-to-pypi-neutron https://review.openstack.org/512261 | 10:55 |
| *** bobh has joined #openstack-infra | 10:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Document base job change procedure https://review.openstack.org/512138 | 10:58 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Use publish-to-pypi-neutron https://review.openstack.org/512261 | 10:59 |
| *** bobh has quit IRC | 11:01 | |
| ykarel | <AJaeger> andreaf: If you have .zuul.yaml on a branch, it uses that one. IF there's none, it takes the one from master, AJaeger any idea why jobs defined in master not executed in stable branches: For example: https://review.openstack.org/#/c/512172/ | 11:03 |
| *** dtantsur|afk is now known as dtantsur | 11:04 | |
| AJaeger | ykarel: I don't understand your problem on that one - the jobs run, didn't they? | 11:05 |
| *** edmondsw has joined #openstack-infra | 11:05 | |
| ykarel | AJaeger, no jobs defined in .zuul.yaml in master are not running in stable branches | 11:05 |
| AJaeger | ykarel: what did you expect - and didn't get exactly? | 11:05 |
| AJaeger | ykarel: example, please | 11:05 |
| ykarel | AJaeger, https://review.openstack.org/#/c/512172/ stable pike patch | 11:06 |
| ykarel | jobs defined at https://github.com/openstack/magnum/blob/master/.zuul.yaml#L218-L225 didn't ran | 11:06 |
| pabelanger | morning | 11:07 |
| *** udesale has quit IRC | 11:07 | |
| dmellado | AJaeger: pabelanger I've a question, what to do with a patch that still has both zuul and jenkins | 11:07 |
| AJaeger | morning, pabelanger | 11:07 |
| dmellado | and when jenkins is blocking zuul | 11:07 |
| dmellado | https://review.openstack.org/#/c/510157/ | 11:07 |
| dmellado | ^^ | 11:07 |
| dmellado | I submitted a patch for removing legacy jenkins jobs | 11:07 |
| *** rook-afk is now known as rook | 11:07 | |
| dmellado | but landed after the first jenkins -1 | 11:07 |
| dmellado | should I abandon+recover? | 11:07 |
| AJaeger | dmellado: you should be able to merge this without problems | 11:08 |
| dmellado | AJaeger: so jenkins wouldn't matter? | 11:08 |
| AJaeger | dmellado: yes, it wouldn't matter | 11:08 |
| dmellado | ack, thanks andreas | 11:08 |
| AJaeger | ykarel: this needs some debugging, let's add to the etherpad. Will you do? | 11:08 |
| pabelanger | dmellado: yah, what AJaeger said | 11:08 |
| ykarel | AJaeger, Ok i can | 11:09 |
| dmellado | thanks too, pabelanger, btw, hope you're done with the moving by now! | 11:09 |
| *** edmondsw has quit IRC | 11:09 | |
| *** dave-mccowan has joined #openstack-infra | 11:09 | |
| pabelanger | dmellado: yah, we are in a cottage on Lake Huron (great lakes) for the next 2 months. We are still looking for rental, but we've moved to the correct town | 11:10 |
| *** Shrews has quit IRC | 11:10 | |
| pabelanger | dmellado: we had a massive wind storm roll through yesterday, it was awesome watching the waves on the Lake | 11:11 |
| AJaeger | ykarel: put it under "to triage", please | 11:11 |
| *** armaan has quit IRC | 11:11 | |
| dmellado | pabelanger: that looks awesome, take some pics! :) | 11:11 |
| *** armaan has joined #openstack-infra | 11:11 | |
| ykarel | AJaeger, ok updating https://etherpad.openstack.org/p/zuulv3-issues | 11:11 |
| pabelanger | AJaeger: just looking at backscroll, we appear to be in good shape this morning | 11:12 |
| *** Shrews has joined #openstack-infra | 11:12 | |
| *** armaan has quit IRC | 11:12 | |
| *** armaan has joined #openstack-infra | 11:13 | |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Add ability to override project branch to a tag https://review.openstack.org/512245 | 11:14 |
| *** smatzek has joined #openstack-infra | 11:14 | |
| AJaeger | pabelanger: we do basically. All proposal jobs are broken (see line 90 in etherpad), publish-api-ref is broken (line 78) and the post upstream-translation job just gives an error. | 11:15 |
| AJaeger | mordred: ^ | 11:15 |
| AJaeger | but regular ones look okish - and tehre are a couple of changes to add required-repos if you want to +2A a few... | 11:15 |
| *** jkilpatr_ has joined #openstack-infra | 11:16 | |
| dmellado | I was checking some issues that showed up on some lbaas/octavia gates | 11:16 |
| dmellado | but that might be just related to octavia itself and not zuul | 11:16 |
| *** masber has quit IRC | 11:16 | |
| dmellado | it seems that octavia's no longer being installed | 11:16 |
| dmellado | http://logs.openstack.org/57/510157/2/check/tempest-kuryr-kubernetes-octavia/2705827/ | 11:16 |
| dmellado | in case you'd like to have a look | 11:17 |
| pabelanger | dmellado: if you think there is an issue, best to add it to https://etherpad.openstack.org/p/zuulv3-issues so we can properly triage it | 11:18 |
| dmellado | pabelanger: I'm checking locally and I'll add it there if it works from my side, thanks | 11:18 |
| openstackgerrit | Michel Peterson proposed openstack-infra/openstack-zuul-jobs master: networking-odl: Add required-projects for functional tests https://review.openstack.org/512254 | 11:20 |
| openstackgerrit | Michel Peterson proposed openstack-infra/project-config master: networking-odl: Add required-projects missing https://review.openstack.org/512238 | 11:20 |
| AJaeger | dmellado: I see "sudo stack" - there's something for that already on the etherpad | 11:20 |
| *** shardy is now known as shardy_lunch | 11:20 | |
| dmellado | AJaeger: oh, I see | 11:21 |
| dmellado | thanks! | 11:21 |
| *** ociuhandu has quit IRC | 11:22 | |
| ykarel | AJaeger, updated the etherpad | 11:23 |
| AJaeger | thanks, ykarel | 11:23 |
| cmurphy | seeing a POST_FAILURE here https://review.openstack.org/#/c/508411/, is that something i should add to the zuulv3-issues etherpad? or just ignore and try again? | 11:23 |
| *** pblaho has joined #openstack-infra | 11:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Fix syntax errors with set-service-type-data-fact role https://review.openstack.org/512267 | 11:26 |
| pabelanger | rsync: recv_generator: mkdir "/var/lib/zuul/builds/5a11ee7f05e64391806dcafb1afdfe3f/work/logs/logs" failed: No space left on device (28) | 11:26 |
| pabelanger | cmurphy: ^ yes, please add to etherpad | 11:26 |
| pabelanger | looks like executor has no more disk | 11:26 |
| *** Shrews has quit IRC | 11:27 | |
| AJaeger | argh ;( | 11:27 |
| cmurphy | pabelanger: kk | 11:27 |
| pabelanger | AJaeger: ^will also fix api-ref jobs | 11:27 |
| pabelanger | looking at HDD issues on ze03.o.o now | 11:28 |
| AJaeger | pabelanger: thanks! | 11:28 |
| *** Shrews has joined #openstack-infra | 11:29 | |
| pabelanger | /dev/xvde2 72G 34G 34G 51% /var/lib/zuul | 11:29 |
| *** slaweq has quit IRC | 11:29 | |
| pabelanger | that is ze03.o.o currently | 11:29 |
| AJaeger | inodes? | 11:29 |
| pabelanger | /dev/xvde2 4.6M 3.8M 799K 83% /var/lib/zuul | 11:29 |
| pabelanger | yah, that is more likley | 11:29 |
| *** Swimingly has quit IRC | 11:30 | |
| pabelanger | AJaeger: it is likely number of concurrent jobs on the executor | 11:30 |
| pabelanger | trying to see if I can calculate that | 11:30 |
| *** huanxie has quit IRC | 11:30 | |
| pabelanger | Yah, I can see in logs it has happened a few times on ze03.o.o | 11:32 |
| pabelanger | 2017-10-16 09:01:39,787 ERROR zuul.AnsibleJob: [build: fa8098c3ef55417884306eeeb5649806] Unable to initialize repo for ssh://zuul@review.openstack.org:29418/openstack/horizon | 11:32 |
| pabelanger | that is because we ran out of HDDs | 11:32 |
| pabelanger | or inodes | 11:33 |
| *** tpsilva has joined #openstack-infra | 11:33 | |
| pabelanger | this is likely of the spike in merge_failure we are seeing too | 11:34 |
| AJaeger | can we unmount and add more inodes? | 11:35 |
| *** nicolasbock has joined #openstack-infra | 11:36 | |
| AJaeger | pabelanger: could you check whether this is specific to ze03 or happens on others as well, please? | 11:36 |
| AJaeger | pabelanger: I suggest to wait until others are around and discuss options - and collect more information for now. | 11:37 |
| pabelanger | AJaeger: we'd have to stop the executor first, but yah, it should be possible | 11:37 |
| pabelanger | AJaeger: Yah, collecting info now and adding to zuulv3-issues | 11:37 |
| *** slaweq has joined #openstack-infra | 11:38 | |
| *** yamamoto has quit IRC | 11:38 | |
| *** Swimingly has joined #openstack-infra | 11:39 | |
| *** Swimingly has quit IRC | 11:39 | |
| *** Swimingly has joined #openstack-infra | 11:39 | |
| *** rhallisey has joined #openstack-infra | 11:39 | |
| *** lucasagomes is now known as lucas-hungry | 11:39 | |
| AJaeger | pabelanger: do we monitor inode and disk usage? What about a simple loop that watches df -i and df on that filesystem once a minute? That way we can corrolate back to the next failure where space was needed | 11:41 |
| pabelanger | AJaeger: we should have disk space in cacti | 11:41 |
| pabelanger | AJaeger: and I think dmsimard is working on a patch for inodes in cacti too | 11:42 |
| pabelanger | AJaeger: okay, so far only ze03 and ze09 have ran out of disk space this morning | 11:42 |
| AJaeger | pabelanger: indeed, disk space is there - but quite static. So, might be inodes | 11:43 |
| AJaeger | thanks, pabelanger. | 11:43 |
| * AJaeger will be back online later | 11:43 | |
| *** tmorin has joined #openstack-infra | 11:46 | |
| tmorin | hi infra-root : we've legacy-networking-bgpvpn-dsvm-functional failing on "cannot create hard link" ... is there a known solution ? should I track that in storyboard ? | 11:47 |
| pabelanger | tmorin: knownn issue, see https://etherpad.openstack.org/p/zuulv3-issues | 11:47 |
| pabelanger | that is where we are tracking this currently | 11:47 |
| *** udesale has joined #openstack-infra | 11:48 | |
| tmorin | pabelanger: ok | 11:48 |
| pabelanger | tmorin: which review are you getting the error on? | 11:49 |
| tmorin | pabelanger: 494213 | 11:49 |
| *** ociuhandu has joined #openstack-infra | 11:49 | |
| tmorin | pabelanger: "sudo -H -u stack tox -e dsvm-functional" ... | 11:49 |
| tmorin | pabelanger: matches the description of what https://review.openstack.org/512194 tries to solve | 11:50 |
| pabelanger | yah | 11:50 |
| pabelanger | I'm going to try seeing if that fixes it | 11:50 |
| *** udesale__ has joined #openstack-infra | 11:50 | |
| pabelanger | remote: https://review.openstack.org/512270 DNM - testing 512194 | 11:51 |
| *** udesale has quit IRC | 11:51 | |
| pabelanger | tmorin: ^lets see what happens | 11:51 |
| *** thorst has joined #openstack-infra | 11:54 | |
| *** thorst has quit IRC | 11:54 | |
| *** thorst has joined #openstack-infra | 11:56 | |
| *** udesale__ has quit IRC | 11:56 | |
| pabelanger | fungi: clarkb: unrelated, but what do you think about deleting devstack_cache (git repos) on nodepool-builders. We haven't done that in a while and see us still coping stable/newton things when setting up repos | 11:56 |
| pabelanger | we talked about that a few weeks ago | 11:57 |
| *** bobh has joined #openstack-infra | 11:57 | |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** bobh has quit IRC | 11:58 | |
| *** bobh has joined #openstack-infra | 11:59 | |
| *** gcb has quit IRC | 12:01 | |
| *** jpena is now known as jpena|lunch | 12:04 | |
| *** mat128 has joined #openstack-infra | 12:07 | |
| *** huanxie has joined #openstack-infra | 12:08 | |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** edmondsw has joined #openstack-infra | 12:09 | |
| *** chlong has quit IRC | 12:10 | |
| *** stakeda has quit IRC | 12:10 | |
| *** mat128 has quit IRC | 12:10 | |
| *** mat128 has joined #openstack-infra | 12:11 | |
| *** slaweq has quit IRC | 12:11 | |
| *** Guest71201 has quit IRC | 12:14 | |
| *** sree has quit IRC | 12:16 | |
| *** sree has joined #openstack-infra | 12:16 | |
| *** gongysh has quit IRC | 12:17 | |
| *** tmorin has quit IRC | 12:18 | |
| *** slaweq has joined #openstack-infra | 12:18 | |
| *** tmorin has joined #openstack-infra | 12:18 | |
| *** sree has quit IRC | 12:20 | |
| *** yamamoto has quit IRC | 12:21 | |
| *** lucas-hungry is now known as lucasagomes | 12:23 | |
| *** dave-mccowan has quit IRC | 12:23 | |
| *** florianf has quit IRC | 12:23 | |
| *** florianf has joined #openstack-infra | 12:23 | |
| *** shardy_lunch is now known as shardy | 12:25 | |
| *** coolsvap has quit IRC | 12:26 | |
| *** rlandy has joined #openstack-infra | 12:27 | |
| *** bobh has quit IRC | 12:30 | |
| *** Hal has joined #openstack-infra | 12:31 | |
| *** Hal is now known as Guest78255 | 12:32 | |
| *** bobh has joined #openstack-infra | 12:34 | |
| *** wolverineav has joined #openstack-infra | 12:35 | |
| mordred | pabelanger: morning - purging cache sounds great | 12:35 |
| *** dprince has joined #openstack-infra | 12:37 | |
| *** esberglu has joined #openstack-infra | 12:38 | |
| *** huanxie has quit IRC | 12:39 | |
| *** felipemonteiro has joined #openstack-infra | 12:39 | |
| *** felipemonteiro_ has joined #openstack-infra | 12:41 | |
| pabelanger | kk, I'll do that now | 12:42 |
| tmorin | pabelanger: https://review.openstack.org/#/c/512270 => didn't work, zuul-cloner still trying to create hard links | 12:43 |
| *** wolverineav has quit IRC | 12:43 | |
| pabelanger | tmorin: yah, i think we need to first land it | 12:43 |
| tmorin | pabelanger: yes, the Depends-On magic must have some limits I guess :) | 12:43 |
| mordred | pabelanger: that should work fine with depends-on | 12:43 |
| pabelanger | cool, then in that case, something didn't work | 12:44 |
| pabelanger | need to see why | 12:44 |
| mordred | nevermind | 12:44 |
| mordred | nope | 12:44 |
| mordred | I'm totally wrong | 12:44 |
| mordred | you were right | 12:44 |
| pabelanger | :) | 12:44 |
| mordred | must land first :) | 12:44 |
| dmsimard | pabelanger: I'm priorizing decreasing the disk space and inode consumption of ara over adding them into cacti right now | 12:45 |
| mordred | pabelanger: also - I had an idea yesterday of just chmodding the repos to be world writable | 12:45 |
| *** dave-mccowan has joined #openstack-infra | 12:45 | |
| *** felipemonteiro has quit IRC | 12:45 | |
| dmsimard | if someone else wants to take a stab at inodes in cacti, feel free | 12:45 |
| pabelanger | mordred: yah, maybe that is the easiest path forward | 12:45 |
| *** nhicher has joined #openstack-infra | 12:46 | |
| mordred | pabelanger: well - the ianw patch doesn't look bad - it'll just have more copying for folks ... gimme a sec though, lemme see about a quick patch | 12:46 |
| *** ykarel is now known as ykarel|away | 12:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make git repos writable so that hard links work https://review.openstack.org/512285 | 12:50 |
| mordred | pabelanger: ^^ somethign like that | 12:50 |
| dmsimard | ianw: replied to your comments on https://review.openstack.org/#/c/504789/ | 12:50 |
| flaper87 | hey folks, sorry if this is an already-answered question but, has the process for adding new repos changed in any way after the roll-out of zuul v3? | 12:51 |
| *** slaweq has quit IRC | 12:51 | |
| mwhahaha | hey so i've noticed that zuul is overwriting the results in gerrit for tripleo (example https://review.openstack.org/#/c/512219/) the ovb results cleared out the normal check results | 12:51 |
| dmsimard | flaper87: gerrit, gerritbot and such are the same, what's different is the job configuration and layout | 12:51 |
| *** ykarel|away has quit IRC | 12:51 | |
| flaper87 | dmsimard: gotcha, thank you | 12:51 |
| mordred | flaper87: the manual has been update | 12:51 |
| flaper87 | mordred: that's what I was hoping for, thanks | 12:52 |
| flaper87 | I'll read the manual then | 12:52 |
| * flaper87 hasn't done this in a while | 12:52 | |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| dmsimard | mwhahaha: hm, because you have jobs in check and check-tripleo pipeline I guess ? | 12:52 |
| mwhahaha | yea | 12:52 |
| dmsimard | mordred: ^ that's likely a legit bug | 12:52 |
| mordred | flaper87: https://docs.openstack.org/infra/manual/creators.html - let us know if any of it sucks | 12:52 |
| dmsimard | adding to zuulv3 issues | 12:52 |
| flaper87 | mordred: will do by proposing new patches if needed :) | 12:53 |
| pabelanger | mordred: do we plan on using mirror-workspace-git-repos on none legacy jobs? If so, maybe we should move it some other place. I can think of some places where we might not want that on non zuul-cloner jobs | 12:53 |
| mordred | pabelanger: ah. good call - we could do it in the copy shim role instead | 12:54 |
| mordred | flaper87: \o/ | 12:54 |
| *** armaan has quit IRC | 12:54 | |
| pabelanger | mordred: yah, shim role might be better | 12:55 |
| *** armaan has joined #openstack-infra | 12:55 | |
| mordred | dmsimard, mwhahaha: hrm. ok. so check-tripleo is just supposed to comment, not to vote, right? | 12:56 |
| *** Shrews has quit IRC | 12:56 | |
| dmsimard | mwhahaha: not sure what to say for the time being other than to be careful, it wasn't a known issue but I just added it to our tracker. | 12:56 |
| mwhahaha | mordred: correct | 12:56 |
| *** dbecker has quit IRC | 12:56 | |
| dmsimard | mordred: It votes, I put two examples on the etherpad, in v2 both pipelines had distinct results | 12:56 |
| dmsimard | mwhahaha: it doesn't vote ?? | 12:56 |
| * dmsimard confused | 12:56 | |
| mwhahaha | the ovb jobs didn't vote | 12:56 |
| mordred | it's not supposed to vote | 12:56 |
| mordred | it's supposed to just leave a comment | 12:57 |
| mordred | success: | 12:57 |
| mordred | gerrit: {} | 12:57 |
| dmsimard | mwhahaha: I don't see any 'non-voting' messages in the results of https://review.openstack.org/#/c/512219/ | 12:57 |
| dmsimard | mwhahaha: other than some individual jobs | 12:57 |
| mwhahaha | dmsimard: the check-tripleo ci doesnt' Verifiied+1 | 12:58 |
| mwhahaha | in the past that just ends up being a different block in the gerrit UI | 12:58 |
| mordred | dmsimard, mwhahaha: it looks like that's doing the right thing - the javascript to show the results in the table up top might not be doing the right thing | 12:58 |
| mwhahaha | yea | 12:58 |
| mordred | yah - I think the bug is in presentation - not in the votes being wrong | 12:58 |
| mwhahaha | that's the problem you have to view the toggle CI to see the results | 12:58 |
| dmsimard | oh I see what you mean | 12:58 |
| mordred | (which is still a bug - but incorrect ui is slightly less disaster than incorrect voting) | 12:58 |
| mwhahaha | correct it's a presentation issue so i don't know if that's in teh way it's posted or whatever | 12:58 |
| *** huanxie has joined #openstack-infra | 12:59 | |
| dmsimard | I got mistaken because the -V vote had been moved but it's because it was enqueued to gate jobs | 12:59 |
| mordred | mwhahaha: it's ... the way that table is created is some amazing hacky javascript :) | 12:59 |
| mwhahaha | mmmm hacky javascript my favorit :D | 12:59 |
| *** apetrich has quit IRC | 12:59 | |
| *** sree has joined #openstack-infra | 12:59 | |
| *** Shrews has joined #openstack-infra | 12:59 | |
| mordred | mwhahaha: when we upgrade gerrit to 2.14 there's an actual API that's been added to gerrit that will allow us to register build results directly - which will be GREAT (thanks zaro) | 13:00 |
| *** apetrich has joined #openstack-infra | 13:00 | |
| mordred | mwhahaha: but for now it's the hacky javascript - so we'll need to figure out what is confusing it | 13:00 |
| dmsimard | hey if the 2.14 upgrade goes as smooth as the 2.13 one, it'll be awesome | 13:00 |
| mwhahaha | mordred: the name | 13:00 |
| mwhahaha | mordred: see https://review.openstack.org/#/c/506186/4 | 13:00 |
| mwhahaha | mordred: it used to be 'jenkins' and 'jenkins check' | 13:00 |
| mwhahaha | we need a zuul and zuul check | 13:00 |
| mordred | mwhahaha: so - that's all actually driven by scraping the html in the comments ... | 13:01 |
| *** slaweq has joined #openstack-infra | 13:02 | |
| mordred | so lookingto see what's different about the two jenkins comments vs. the two zuul comments | 13:02 |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| *** mriedem has joined #openstack-infra | 13:02 | |
| cmurphy | is MERGER_FAILURE another incarnation of out of inodes? https://review.openstack.org/#/c/500005/ | 13:02 |
| *** jpena|lunch is now known as jpena | 13:03 | |
| jkilpatr_ | any idea why I'm getting permission denied pubkey when doing this | 13:04 |
| *** Shrews has quit IRC | 13:04 | |
| jkilpatr_ | 13:03:06 + git remote add gerrit https://review.openstack.org/openstack/browbeat | 13:04 |
| jkilpatr_ | 13:03:06 + git fetch --all | 13:04 |
| *** Guest78255 has quit IRC | 13:05 | |
| jkilpatr_ | wondering if it's an upstream issue or if I should start debugging the downstream jenkins. | 13:05 |
| dmsimard | cmurphy: merger failure would not occur on the server we've been having inode issues on | 13:05 |
| cmurphy | dmsimard: okay, is it related to something else you know about or should i add it to the etherpad? | 13:05 |
| *** dave-mccowan has quit IRC | 13:06 | |
| *** tmorin has quit IRC | 13:06 | |
| *** Shrews has joined #openstack-infra | 13:06 | |
| cmurphy | looks like you're adding it, ty | 13:06 |
| dmsimard | cmurphy: nothing I know about but I already added it to the etherpad after skimming to see if it wasn't already there.. iirc jeblair did some fixes but maybe there is something else | 13:07 |
| dmsimard | or maybe it hasn't merged yet, let me look | 13:07 |
| AJaeger | mordred: morning! All proposal jobs are broken (see line 90 in etherpad), publish-api-ref is broken (line 78 - fixed with https://review.openstack.org/#/c/512267/) and the post upstream-translation job just gives an error - in case you want to review/dig into either of these... | 13:08 |
| mordred | AJaeger: yay! | 13:08 |
| *** bobh has quit IRC | 13:08 | |
| *** tmorin has joined #openstack-infra | 13:08 | |
| dmsimard | cmurphy: yeah, the change merged ( https://review.openstack.org/#/c/511957/ ) which jeblair mentioned was a fix for mergers but it's merged and very likely loaded at this point. It is maybe a new one, feel free to recheck for the time being. | 13:09 |
| cmurphy | dmsimard: the failure only happened an hour ago so if the fix was merged three days ago i think it would be something different | 13:10 |
| dmsimard | yup. | 13:10 |
| *** links has quit IRC | 13:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make git repos writable so that hard links work https://review.openstack.org/512285 | 13:12 |
| *** bauwser is now known as bauzas | 13:12 | |
| mordred | pabelanger: ^^ how about that? | 13:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Fix set-service-type-data-fact role https://review.openstack.org/512292 | 13:14 |
| mordred | oh - whoops - pabelanger had that one already | 13:14 |
| AJaeger | mordred: yeah - just +2A https://review.openstack.org/512267 ;) | 13:15 |
| pabelanger | looking | 13:15 |
| *** dave-mccowan has joined #openstack-infra | 13:16 | |
| mordred | yup. done | 13:16 |
| pabelanger | jkilpatr_: where are you seeing the permisson denied pubkey? Have a log? | 13:16 |
| pabelanger | mordred: +2 | 13:17 |
| *** dbecker has joined #openstack-infra | 13:18 | |
| mordred | tmorin: you should be able to depends-on with https://review.openstack.org/512285 | 13:18 |
| *** armaan has quit IRC | 13:19 | |
| *** Shrews has quit IRC | 13:19 | |
| *** armaan has joined #openstack-infra | 13:20 | |
| *** mat128_ has joined #openstack-infra | 13:20 | |
| *** rosmaita has joined #openstack-infra | 13:20 | |
| jkilpatr_ | pabelanger, a downstream jenkins, playing around with the issue more it seems to be a local settings issue. I'll ping the right people. | 13:20 |
| pabelanger | AJaeger: mind adding https://review.openstack.org/512114 to your review pipeline | 13:20 |
| *** mat128_ has quit IRC | 13:20 | |
| *** wolverineav has joined #openstack-infra | 13:20 | |
| *** Shrews has joined #openstack-infra | 13:21 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix syntax errors with set-service-type-data-fact role https://review.openstack.org/512267 | 13:21 |
| tmorin | mordred: thanks, trying this right now (https://review.openstack.org/512294) | 13:22 |
| AJaeger | pabelanger: +2A | 13:22 |
| *** mat128 has quit IRC | 13:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: It is pre_tasks, not pre-tasks https://review.openstack.org/512295 | 13:22 |
| pabelanger | AJaeger: now we wait for post-api-ref job^ | 13:23 |
| *** dbecker has quit IRC | 13:23 | |
| *** smatzek has quit IRC | 13:23 | |
| mordred | AJaeger: ^^ that should fix the proposal job issue | 13:23 |
| pabelanger | +3 | 13:23 |
| *** kiennt26 has joined #openstack-infra | 13:23 | |
| AJaeger | mordred: that's all? ansible should be intelligent enough to self-fix ;) Thanks! | 13:24 |
| pabelanger | yah, onc we get linters job working, should be able to catch a lot of that | 13:24 |
| pabelanger | I'm going to try hacking on that this week | 13:24 |
| *** gcb has joined #openstack-infra | 13:25 | |
| AJaeger | pabelanger: jlk has a linters job up as well that includes a few fixes | 13:25 |
| AJaeger | pabelanger: https://review.openstack.org/503751 | 13:26 |
| *** Shrews has quit IRC | 13:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove windmill from zuul-legacy-project-templates https://review.openstack.org/512114 | 13:29 |
| *** kashyap has left #openstack-infra | 13:29 | |
| *** huanxie has quit IRC | 13:29 | |
| *** ihrachys has joined #openstack-infra | 13:30 | |
| *** erlon has joined #openstack-infra | 13:30 | |
| mordred | pabelanger, tmorin: https://review.openstack.org/512270 DNM - testing 512285 ... I re-submitted that one witha depends-on the chmod patch | 13:32 |
| pabelanger | ack | 13:32 |
| mordred | since that happens in the zuul-cloner copy role, Im pretty sure it's depends-on-able, which'll be nice if it fixes it | 13:33 |
| *** rkukura has joined #openstack-infra | 13:33 | |
| pabelanger | yah, watching stream now | 13:33 |
| *** slaweq has quit IRC | 13:34 | |
| *** rkukura has quit IRC | 13:34 | |
| *** dbecker has joined #openstack-infra | 13:35 | |
| *** baoli has joined #openstack-infra | 13:36 | |
| mpeterson | AJaeger: hey! I was wondering if we can merge these? https://review.openstack.org/#/c/512238/ and https://review.openstack.org/#/c/512254/ | 13:36 |
| *** gongysh has joined #openstack-infra | 13:36 | |
| *** gongysh has quit IRC | 13:37 | |
| *** kiennt26 has quit IRC | 13:37 | |
| *** slaweq has joined #openstack-infra | 13:37 | |
| *** lbragstad has joined #openstack-infra | 13:37 | |
| AJaeger | mpeterson: all chagnes need a second +2... and the second needs a recheck once the first is in. | 13:38 |
| mordred | mpeterson, AJaeger: first one approved | 13:40 |
| dmsimard | mordred, pabelanger: https://review.openstack.org/#/c/512295 would very likely have been caught by an 'ansible-playbook --syntax-check' or by ansible-lint, does it not run ? | 13:40 |
| mpeterson | AJaeger: gotcha. | 13:40 |
| mpeterson | mordred: thanks | 13:40 |
| *** hongbin has joined #openstack-infra | 13:41 | |
| *** kgiusti has joined #openstack-infra | 13:41 | |
| AJaeger | dmsimard: no, we need one - see my discussion with pabelanger earlier and check https://review.openstack.org/503751 and https://review.openstack.org/495463 | 13:42 |
| *** ramishra has quit IRC | 13:42 | |
| dmsimard | AJaeger: ack | 13:43 |
| openstackgerrit | Merged openstack-infra/project-config master: It is pre_tasks, not pre-tasks https://review.openstack.org/512295 | 13:43 |
| *** mriedem1 has joined #openstack-infra | 13:43 | |
| *** Hal has joined #openstack-infra | 13:44 | |
| *** Hal is now known as Guest68844 | 13:44 | |
| pabelanger | AJaeger: dmsimard: Yah at PTG talked with mordred about created a linter jobs to pull in all playbooks and roles to be shared with project-config, zuul-jobs, openstack-zuul-jobs. Then we could get a good coverage going across a three repos. | 13:44 |
| dmsimard | I added it to zuulv3-issues in response to the pre-tasks vs pre_tasks | 13:45 |
| pabelanger | I'm flying back up to Ottawa on Wednesday, will have some free time to hack on it | 13:45 |
| dmsimard | pabelanger: oh that's right, OpenStack Canada thursday.. need to leave from Montreal around like 6AM T_T | 13:45 |
| AJaeger | dmsimard: don't add it to the "Bug fixes section" - we clean that up regularly, it's solely for tracking... | 13:46 |
| *** dtantsur_ has joined #openstack-infra | 13:46 | |
| dmsimard | sure | 13:46 |
| mordred | pabelanger: looks like https://review.openstack.org/512270 worked - let's land 512285 | 13:46 |
| pabelanger | mordred: 512285 worked: http://logs.openstack.org/70/512270/2/check/legacy-networking-bgpvpn-dsvm-functional/3897d47/ | 13:47 |
| pabelanger | haha | 13:47 |
| *** _bmjen has joined #openstack-infra | 13:47 | |
| mordred | :) | 13:47 |
| mordred | AJaeger: wanna +3 https://review.openstack.org/#/c/512285/ ? | 13:47 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-odl: Add required-projects missing https://review.openstack.org/512238 | 13:47 |
| *** mriedem has quit IRC | 13:48 | |
| *** liusheng has quit IRC | 13:48 | |
| *** bmjen has quit IRC | 13:48 | |
| *** dtantsur has quit IRC | 13:48 | |
| *** dtantsur_ is now known as dtantsur | 13:48 | |
| *** liusheng has joined #openstack-infra | 13:48 | |
| AJaeger | mordred: done | 13:48 |
| *** bobh has joined #openstack-infra | 13:49 | |
| *** mriedem1 is now known as mriedem | 13:49 | |
| *** ryotagami has quit IRC | 13:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Collapse job header lines into a single statement https://review.openstack.org/511821 | 13:51 |
| *** psachin has joined #openstack-infra | 13:53 | |
| mpeterson | I have a question regarding migrating. Once everything is migrated then the project no longer needs to exist on `project-config/zuul.d/projects.yaml`, right? As it will be on the project repository | 13:54 |
| AJaeger | mpeterson: Wrong ;) | 13:54 |
| mordred | dmsimard, pabelanger: ^^ I have fixed dmsimard's review comments on https://review.openstack.org/511821 - pabelanger, are you ok with the followup patch version? | 13:54 |
| AJaeger | mpeterson: did you read the zuulv3 document? | 13:54 |
| *** gouthamr has joined #openstack-infra | 13:54 | |
| pabelanger | AJaeger: did you want to make project-config-frozen-files voting now in project-config? | 13:54 |
| AJaeger | mpeterson: https://docs.openstack.org/infra/manual/zuulv3.html#moving-legacy-jobs-to-projects | 13:54 |
| *** chlong has joined #openstack-infra | 13:54 | |
| pabelanger | mordred: looking | 13:54 |
| AJaeger | pabelanger: we could - right now I'm fine taking in changes that touch those files in addition to v3 files. But we can change | 13:55 |
| mpeterson | AJaeger: yes, and that bit wasn't clear to me :) because 3.1 of those steps says to remove them from the pipeline definition | 13:55 |
| mordred | pabelanger: (mostly want to either land those or abandon them to get them off teh list) | 13:55 |
| pabelanger | mordred: I'd like to see ansible_hostname if possible, that is helpful information to quickly tell the type of node in nodepool (eg: centos in vexxhost) | 13:55 |
| pabelanger | mordred: but that could be a follow up | 13:56 |
| AJaeger | mpeterson: did you see "what not convert"? | 13:56 |
| *** armax has joined #openstack-infra | 13:56 | |
| AJaeger | mpeterson: you remove the legacy jobs, leve the standard ones in | 13:56 |
| mordred | mpeterson, AJaeger ah - I think we could add the word "legacy" to that sentence in 3.1 ... | 13:56 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Ensure that build_names are unique per project https://review.openstack.org/508258 | 13:57 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Strip legacy prefix if it's present https://review.openstack.org/508272 | 13:57 |
| pabelanger | AJaeger: okay, your call | 13:57 |
| *** ryotagami has joined #openstack-infra | 13:57 | |
| *** kiennt26 has joined #openstack-infra | 13:57 | |
| mpeterson | AJaeger: wow, I had actually read that.. Just didn't stick to my mind... | 13:58 |
| mtreinish | infra-root: ^^^ we might need to manually restart the subunit workers to take those | 13:58 |
| pabelanger | mtreinish: can you add that to zuulv3-issues etherpad | 13:59 |
| pabelanger | so we don't lose them | 13:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Clarify job removal section https://review.openstack.org/512304 | 13:59 |
| mordred | AJaeger: ^^^ like that | 13:59 |
| mordred | mpeterson: also, would that have helped? | 14:00 |
| *** dbecker has quit IRC | 14:00 | |
| mordred | pabelanger: kk. I can add followup with that | 14:00 |
| AJaeger | mpeterson: if you have ideas for better wording, feel free to send changes | 14:00 |
| mtreinish | pabelanger: sure | 14:00 |
| AJaeger | bbl | 14:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Make git repos writable so that hard links work https://review.openstack.org/512285 | 14:01 |
| *** jcoufal_ has joined #openstack-infra | 14:01 | |
| mpeterson | mordred: I think it would have raised a flag on my mind, yes. I would probably add a (See "What Not to Convert?") | 14:01 |
| mpeterson | AJaeger: will keep that in mind | 14:01 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: deprecations: Deprecate support for '-py{N}' requirements https://review.openstack.org/512306 | 14:02 |
| jeblair | andreaf: i think for devstack, and likely tempest too, we should have the job definition in multiple branches. when you do that, they become variants, and override settings on the main job. I think that will be a good way to manage configuration changes across branches as devstack evolves. | 14:03 |
| mpeterson | so I have another doubt now... it says for example to leave openstack-python-jobs in project-config... but in our specific case we need a lot more requirements than for example the one given by openstack-python-jobs-neutron, so I defined a project-template that includes those requirements. In that case, in project-config openstack-python-jobs-neutron should remain and in the project repo we add | 14:04 |
| mpeterson | the project-template defined in the repo? | 14:04 |
| *** yamahata has joined #openstack-infra | 14:04 | |
| andreaf | jeblair: well tempest is branchless | 14:04 |
| *** jcoufal has quit IRC | 14:04 | |
| jeblair | andreaf: however, it's also possible to do all of that on the master branch if we want. i just think that putting it on different branches may be more intuitiv. | 14:04 |
| jeblair | andreaf: ah right, let's not do that for tempest then. :) we can still define all the branches on master. | 14:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add tripleo-common to tripleo change queue for gate https://review.openstack.org/512308 | 14:05 |
| mordred | pabelanger: actually -we can't emit hostname -that job header runs on localhost (so that we don't print job info more than one time) | 14:05 |
| andreaf | jeblair: is it possible to do that at role granularity? i.e. if override a single role in a branch for instance? | 14:05 |
| *** dbecker has joined #openstack-infra | 14:05 | |
| pabelanger | mordred: boo | 14:05 |
| pabelanger | mordred: but makes sense | 14:06 |
| pabelanger | mordred: wonder if we should write it into inventory along with other nodepool variables. I can look into that in a bit | 14:06 |
| * fungi is attempting to skim the ~700 lines of scrollback which have accumulated since he went to sleep last night. may take a while | 14:06 | |
| mordred | pabelanger: well, it's in the host-info file: http://logs.openstack.org/85/512285/2/check/tox-linters/318980d/zuul-info/host-info.ubuntu-xenial.yaml | 14:06 |
| *** mat128 has joined #openstack-infra | 14:07 | |
| mordred | pabelanger: and, for that matter, the zuul-info file too: http://logs.openstack.org/85/512285/2/check/tox-linters/318980d/zuul-info/zuul-info.ubuntu-xenial.txt | 14:07 |
| *** yamamoto has quit IRC | 14:07 | |
| mordred | jeblair: there is an issue related to the discussion above with andreaf ... | 14:07 |
| pabelanger | AJaeger: with 512308, it might be time for tripleo to create something long the lines of 'integrated-gate' project-template, but 'tripleo-integrated-gate'. Otherwise, I see a lot of place where we are adding the queue keywork into jobs | 14:07 |
| jeblair | andreaf: not for a single role, though you can specify a branch for the project the role is in | 14:07 |
| mordred | jeblair: on the https://etherpad.openstack.org/p/zuulv3-issues line 62 | 14:08 |
| pabelanger | mordred: yah, just helpful to print it in realtime too. But, that's what I end up looking at | 14:08 |
| jeblair | mordred: ack, thanks | 14:08 |
| mordred | pabelanger: agree. maybe we should add a emit-host-header that's similar but per-host | 14:09 |
| dtantsur | folks, any known problems with our SIP conferencing? we're trying to have a virtual meetup, but apparently only one person can connect of all | 14:09 |
| mordred | pabelanger: it's mostly the most useful when I'm watching a live stream and the other files don't yet exist | 14:09 |
| dtantsur | well, 2 | 14:09 |
| pabelanger | dtantsur: I can look | 14:10 |
| *** slaweq has quit IRC | 14:10 | |
| dtantsur | pabelanger: thanks! it's strange. Registrar-less accounts work for half of people. People using Red Hat SIP accounts can connect but do not hear anyone | 14:10 |
| pabelanger | dtantsur: what conference room | 14:11 |
| dtantsur | pabelanger: 6005 | 14:11 |
| pabelanger | dtantsur: yah, I see 9 users. Let me call in real quick | 14:11 |
| pabelanger | dtantsur: you can try testing with sip:echo@pbx.openstack.org | 14:12 |
| *** wolverineav has quit IRC | 14:12 | |
| dtantsur | sure | 14:12 |
| *** hashar is now known as hasharAway | 14:12 | |
| pabelanger | dtantsur: that should allow users to test audio | 14:12 |
| *** masber has joined #openstack-infra | 14:13 | |
| *** dhinesh has joined #openstack-infra | 14:13 | |
| *** iyamahat has joined #openstack-infra | 14:13 | |
| dtantsur | pabelanger: the same result | 14:14 |
| pabelanger | dtantsur: no audio? or 1 way audio | 14:14 |
| dtantsur | pabelanger: I hear nothing. the mic level in jitsi seems to move when I speak | 14:14 |
| *** david-lyle has joined #openstack-infra | 14:14 | |
| pabelanger | dtantsur: can you try again | 14:15 |
| dtantsur | this is with RH SIP, with registrar-less I still cannot connect | 14:15 |
| dtantsur | still nothing | 14:15 |
| pabelanger | dtantsur: well, all traffic in to pbx.o.o will be not register. | 14:15 |
| pabelanger | dtantsur: again please | 14:16 |
| pabelanger | collecting sip trace | 14:16 |
| dtantsur | pabelanger: which account to use? | 14:16 |
| *** rkukura has joined #openstack-infra | 14:16 | |
| *** rbrndt has joined #openstack-infra | 14:16 | |
| dtantsur | trying with a registrar-less one now | 14:17 |
| *** slaweq has joined #openstack-infra | 14:17 | |
| pabelanger | dtantsur: there is no account, we just allow anonymous | 14:17 |
| pabelanger | dtantsur: do you have ekiga installed? | 14:17 |
| dtantsur | well, for me the difference is big | 14:17 |
| dtantsur | pabelanger: no, but I can install it | 14:17 |
| *** dhinesh has quit IRC | 14:17 | |
| *** masber has quit IRC | 14:17 | |
| *** dbecker has quit IRC | 14:17 | |
| pabelanger | It's likely something with NAT on redhat side, SIP trace looks to be correct | 14:18 |
| pabelanger | I should be able to test that in a bit with my corp account, but not right now | 14:18 |
| pabelanger | lets see how ekiga works | 14:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Start ssh persistent connections outside bwrap https://review.openstack.org/512158 | 14:18 |
| *** dizquierdo has joined #openstack-infra | 14:18 | |
| *** wolverineav has joined #openstack-infra | 14:19 | |
| *** milan_ has joined #openstack-infra | 14:19 | |
| dtantsur | pabelanger: trying ekiga now with echo | 14:19 |
| dtantsur | pabelanger: "remove host is offline" | 14:20 |
| *** smarcet has joined #openstack-infra | 14:20 | |
| pabelanger | dtantsur: I didn't see attempt | 14:20 |
| dtantsur | I suspect you only see the attempts from my RH SIP | 14:21 |
| dtantsur | I wonder if my provider just blocks SIP in the end.. | 14:21 |
| dmsimard | mordred, pabelanger: btw ianw said he'd like to see the nodepool provider in the job header but he says it was also available in the inventory if need be. I don't have a strong opinion, maybe you have one :) | 14:21 |
| jeblair | dmsimard: yeah, though by adding it to the job log output, a user can see it before the job completes | 14:22 |
| dmsimard | oh, there's a nice thought | 14:22 |
| dmsimard | indeed | 14:22 |
| dmsimard | Do you want to add it on https://review.openstack.org/#/c/511821/ mordred ? | 14:23 |
| mordred | dmsimard: I'll do a followup ... | 14:23 |
| dmsimard | wfm | 14:23 |
| *** slaweq_ has quit IRC | 14:24 | |
| *** slaweq_ has joined #openstack-infra | 14:24 | |
| pabelanger | dtantsur: best I can see, we are getting Retransmission timeout from RH account | 14:25 |
| dtantsur | meh.. | 14:25 |
| pabelanger | which likey is routing issues | 14:26 |
| dtantsur | thanks pabelanger! we'll probably have to try bluejeans or something like that | 14:26 |
| AJaeger | mordred, jeblair, jlk, upstream-translation-update fails without a log file - it just gives "error". I put this on the etherpad | 14:26 |
| *** smarcet has quit IRC | 14:26 | |
| pabelanger | dtantsur: okay, sorry we couldn't get it working | 14:27 |
| dtantsur | np | 14:28 |
| *** slaweq_ has quit IRC | 14:29 | |
| jeblair | mordred: is the issue described under "Still seeing the hardlink copy failures between /home/zuul/... and /tmp in OSC functional jobs (dtroyer/ianw)" the same as the one described under "Running tox as alternative users" ? | 14:29 |
| *** david-lyle has quit IRC | 14:29 | |
| mordred | jeblair: yes | 14:29 |
| * jeblair rejiggers etherpad | 14:30 | |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| mordred | jeblair: at least, that is my current belief | 14:30 |
| *** camunoz has joined #openstack-infra | 14:30 | |
| jeblair | 512285 merged, i'll move the combined entry to fixed | 14:31 |
| jeblair | dtroyer: can you retry the osc jobs that failed with perms issues? | 14:32 |
| pabelanger | okay, I see build-openstack-releasenotes failing in gate. Possible issue with tox-siblings http://logs.openstack.org/30/511630/1/gate/build-openstack-releasenotes/06eeaac/ara/result/eb67d5f9-3800-4565-adc0-e3ce284da7eb/ | 14:33 |
| pabelanger | going to check if known issue | 14:33 |
| dtroyer | jeblair: https://review.openstack.org/512125 is ianw's fix, passed check and gate, https://review.openstack.org/509015 is the original, passed chack, gate failure appears to be one of our occasional racy failures in the functional tests, unrelated to zuulv3 | 14:34 |
| dtroyer | thanks | 14:35 |
| jeblair | dtroyer: cool, thx | 14:35 |
| pabelanger | okay, I've added build-openstack-releasenotes MODULE_FAILURE to zuulv3-issue | 14:35 |
| jeblair | pabelanger: can you move that to issues with jobs? | 14:35 |
| pabelanger | something not correct in tox-siblings | 14:35 |
| jeblair | pabelanger: you have triaged it -- it's a job bug not a zuul bug | 14:36 |
| pabelanger | sure, and done | 14:36 |
| *** sree has quit IRC | 14:37 | |
| *** camunoz has quit IRC | 14:38 | |
| dmsimard | jeblair: btw there was a MERGER_FAILURE earlier, I posted it in triage | 14:38 |
| jeblair | dmsimard: ack, thx | 14:39 |
| *** dbecker has joined #openstack-infra | 14:39 | |
| * clarkb attempts to catch up on scrollback like fungi | 14:39 | |
| clarkb | pabelanger: so we think inodes for /var/lib/zuul filling? | 14:40 |
| mordred | pabelanger: didn't we land a patch forever ago to make sure MODULE FAILURE was printe to the consolelog? | 14:40 |
| clarkb | we should be able to stop executors one at a time and reformat that partition right? | 14:40 |
| pabelanger | clarkb: I am not sure, it was something AJaeger suggested we look at | 14:41 |
| pabelanger | mordred: cannot remember, but I'll look on executor now | 14:41 |
| pabelanger | mordred: I guess not | 14:42 |
| *** esberglu has quit IRC | 14:46 | |
| jeblair | mordred: regarding the magnum stable jobs not running issue -- i think there is perhaps a subtlety to https://review.openstack.org/508658 that we missed. if a project adds an in-repo project stanza, they either need to add it to all branches, or specify all the branches on the jobs if they only add it to master. | 14:46 |
| jeblair | mordred: i think we need a infra-manual zuulv3 section update for that | 14:46 |
| *** d0ugal has quit IRC | 14:46 | |
| *** yamahata has quit IRC | 14:49 | |
| *** yamahata has joined #openstack-infra | 14:50 | |
| jeblair | (we could consider not doing implied branch matchers on project stanzas in the master branch, but then if you add a job to master, it will also run on stable, so you'd have to do explicit branch exclusions, which partly defeats the purpose of having implied branch matchers on non-master branch project stanzas) | 14:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix wrong paths for api-guide/api-ref https://review.openstack.org/512319 | 14:52 |
| AJaeger | this fixes publishing of guides, reviews welcome ^ | 14:52 |
| *** camunoz has joined #openstack-infra | 14:53 | |
| *** gcb has quit IRC | 14:54 | |
| Jeffrey4l | does /var/lib/zuul/builds folder really exist? why i can not find it on the ci node? | 14:54 |
| Jeffrey4l | in zuulv3 | 14:54 |
| *** xarses has joined #openstack-infra | 14:55 | |
| clarkb | Jeffrey4l: that is on the executor not the test node(s) | 14:55 |
| pabelanger | Jeffrey4l: that exists on our zuul-executors, so host: localhost | 14:55 |
| odyssey4me | is it normal to not see the console log - for example all I see is 'end of stream' here: http://zuulv3.openstack.org/static/stream.html?uuid=4d46810442244f549a25c1cf70b4c4fe&logfile=console.log | 14:55 |
| *** wolverineav has quit IRC | 14:56 | |
| Jeffrey4l | so where is the executor node? is it a static node? | 14:56 |
| pabelanger | odyssey4me: which patchset is that? | 14:56 |
| *** wolverineav has joined #openstack-infra | 14:56 | |
| *** slaweq_ has joined #openstack-infra | 14:56 | |
| *** sree has joined #openstack-infra | 14:56 | |
| pabelanger | Jeffrey4l: Yah, they exist in our control plane, ze01.o.o to ze10.o.o | 14:56 |
| odyssey4me | pabelanger https://review.openstack.org/508509 | 14:56 |
| Jeffrey4l | roger. thanks | 14:57 |
| odyssey4me | pabelanger actually, sorry it's https://review.openstack.org/511912 | 14:57 |
| odyssey4me | pabelanger another issue is that the legacy-ansible-func job is non-voting for some reason - where do I correct it? | 14:57 |
| pabelanger | odyssey4me: do you know whick job the console stream was? could it be possible the job was actually finished? | 14:58 |
| odyssey4me | pabelanger I tried at the beginning of the job, and during it - no difference | 14:59 |
| *** slaweq has quit IRC | 14:59 | |
| *** slaweq has joined #openstack-infra | 14:59 | |
| odyssey4me | pabelanger it's still running now - openstack-ansible-tests | 14:59 |
| clarkb | odyssey4me: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml to fix the non voting thing | 14:59 |
| pabelanger | odyssey4me: you can look in project-config/zuul.d/project.yaml | 14:59 |
| *** beekneemech is now known as bnemec | 15:00 | |
| odyssey4me | clarkb pabelanger thanks - patch incoming | 15:00 |
| *** d0ugal has joined #openstack-infra | 15:00 | |
| *** wolverineav has quit IRC | 15:00 | |
| *** slaweq_ has quit IRC | 15:00 | |
| *** slaweq has quit IRC | 15:00 | |
| pabelanger | odyssey4me: both streams for openstack-ansible-tests are working for me currently | 15:01 |
| clarkb | pabelanger: looks like it is inodes /dev/xvde2 4743168 4743168 0 100% /var/lib/zuul <- just now from ze03 | 15:01 |
| pabelanger | clarkb: kk | 15:01 |
| *** dbecker has quit IRC | 15:01 | |
| pabelanger | nice work | 15:01 |
| *** Shrews has joined #openstack-infra | 15:01 | |
| odyssey4me | pabelanger hmm, maybe it's a firewall thing - but it's never seemed to work for me, whether at the office or home | 15:01 |
| pabelanger | odyssey4me: but, not streaming should be working | 15:01 |
| clarkb | so we probably do want to shut off each executor one by one and reformat that partition with significantly more inodes | 15:01 |
| clarkb | we have 4 million now, maybe go up to 100 million? | 15:01 |
| odyssey4me | I'll try again from home tomorrow and look more closely at browser things. | 15:01 |
| odyssey4me | Thanks for checking pabelanger . | 15:01 |
| pabelanger | odyssey4me: k, we do have finger protocol too | 15:02 |
| fungi | clarkb: well, you set it as a ratio of inodes to filesystem size | 15:02 |
| pabelanger | but still need to add links to status page | 15:02 |
| clarkb | (I've got a meeting right now but can annotate the etherpad shortly then help with that work) | 15:02 |
| clarkb | fungi: mkfs.ext4 takes a -N flag to just set it directly | 15:03 |
| clarkb | but we can also bump the ration instead, either way as long as we end up with more :) | 15:03 |
| pabelanger | odyssey4me: if you want to add your issue to etherpad.openstack.org/p/zuulv3-issues we can try and debug once you are at home | 15:03 |
| clarkb | I also wonder if ze03 has leaked inodes? | 15:04 |
| fungi | clarkb: ahh, right, i guess you can do it either way (it gets stored as a ratio on the fs though) | 15:04 |
| clarkb | eg do we delete all the things on shutdown? | 15:04 |
| clarkb | I'll have to dig in more after meeting but that gives us a good place to start | 15:04 |
| jeblair | clarkb: what's the disk usage that goes along with those inodes? | 15:06 |
| clarkb | jeblair: /dev/xvde2 72G 40G 29G 59% /var/lib/zuul was taken at roughly the same time as the inode listing | 15:07 |
| *** ijw has joined #openstack-infra | 15:07 | |
| *** sdague has quit IRC | 15:07 | |
| clarkb | 60% | 15:07 |
| mordred | pabelanger: ok - I see the bug with ironic-inspector and build-openstack-releasenotes | 15:07 |
| *** sdague has joined #openstack-infra | 15:07 | |
| *** yamamoto has joined #openstack-infra | 15:08 | |
| pabelanger | mordred: great | 15:08 |
| mordred | [testenv:releasenotes] | 15:08 |
| mordred | envdir = {toxworkdir}/venv | 15:08 |
| mordred | they redefine envdir for that venv in their tox.ini | 15:08 |
| clarkb | mordred: maybe in that case we stat if the dir exists where we want it then ignore copying those logs otherwise? | 15:08 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-cookiecutter https://review.openstack.org/512323 | 15:10 |
| mordred | clarkb: yah - I think for today we can skip if it doesn't exist - and I thinkn we can make it better for tomorrow | 15:10 |
| *** Shrews has quit IRC | 15:12 | |
| clarkb | we can likely just copy anything out of .tox/**/log | 15:12 |
| clarkb | or is it .tox/**/logs | 15:12 |
| clarkb | (to use the jenkins globbing method) | 15:12 |
| pabelanger | it was like that original but think we changed it | 15:12 |
| *** dave-mccowan has quit IRC | 15:13 | |
| AJaeger | team, please put https://review.openstack.org/512319 on your review queue - to fix a wrong path for api-ref/api-guide publishing | 15:13 |
| *** yamamoto has quit IRC | 15:14 | |
| *** armax has quit IRC | 15:14 | |
| pabelanger | +2 | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Work around non-standard envdir locations https://review.openstack.org/512325 | 15:14 |
| AJaeger | thanks | 15:14 |
| *** kjackal_ has quit IRC | 15:15 | |
| clarkb | pabelanger: I think the old method was grab anything under .tox and assume it had a log dir which is why it failed. There is at least one dir that doesn't collect logs that tox uses | 15:15 |
| clarkb | pabelanger: so we just have to glob it properly I Think | 15:15 |
| pabelanger | clarkb: ah, possible | 15:16 |
| mordred | pabelanger, clarkb, AJaeger: ^^ https://review.openstack.org/512325 should work around the releasenotes job for ironic-inspector | 15:16 |
| mordred | and I mention a follow up we can write later | 15:16 |
| mordred | we also need to do a similar thing for fetching logs | 15:16 |
| clarkb | du --inodes says half a million inodes in executor-git then 3.9 million in builds | 15:17 |
| *** e0ne has joined #openstack-infra | 15:17 | |
| dmsimard | Do we have/use python3 on logs.o.o (for os-loganalyze) ? | 15:18 |
| clarkb | dmsimard: I think it is python2 currently but it should be able to python3.4 | 15:18 |
| pabelanger | yah, logs.o.o is trusty still | 15:18 |
| * clarkb is digging in more to see if there are exceptional builds or if this is normal ish, but we copy the git repos in there don't we? guessing its copies of git repos that may be a problem especially if not hardlinked | 15:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Ignore missing tox log directories https://review.openstack.org/512327 | 15:19 |
| dmsimard | clarkb: ok, I'll write something compatible with both just in case.. py3 has something convenient for a problem I have | 15:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Zuulv3: Add note about in-repo stable config https://review.openstack.org/512328 | 15:20 |
| jeblair | mordred, AJaeger: ^ does that make sense? | 15:20 |
| *** trown is now known as trown|brb | 15:20 | |
| odyssey4me | pabelanger clarkb - I've added an issue with our jobs into 'issues needing triage' in the etherpad... not too crucial at this time, but when there's a gap we'd like to get it resolved | 15:21 |
| ilpianista_ | hi, what's needed to accept this change? https://review.openstack.org/#/c/507531/ | 15:21 |
| mordred | jeblair: yes - with one wording nit (expand a pronoun) | 15:21 |
| ilpianista_ | I see no updates since 3 weaks... | 15:21 |
| clarkb | #status log elasticsearch cluster is now green after triggering index curator early to clear out old indexes "lost" on es07 | 15:21 |
| openstackstatus | clarkb: finished logging | 15:21 |
| ilpianista_ | weeks* | 15:21 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-cookiecutter https://review.openstack.org/512323 | 15:22 |
| AJaeger | jeblair: this backporting is worth an email to openstack-dev IMHO | 15:22 |
| *** LindaWang has quit IRC | 15:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Zuulv3: Add note about in-repo stable config https://review.openstack.org/512328 | 15:23 |
| jeblair | mordred: good call | 15:23 |
| clarkb | ilpianista_: JJB is mostly reviewed by electrofelix, zaro, and zxiiro at this point. It is possible that we may need more volunteers around that tools particularly since we don't consume it anymore | 15:23 |
| clarkb | or maybe go to single core approvals so you aren't waiting for a second reviewer | 15:23 |
| jeblair | there's also a jjb channel | 15:23 |
| zxiiro | ilpianista_: we need another core to review it. I already have so cannot review it again. Also the jjb channel is now #openstack-jjb | 15:24 |
| *** trown|brb is now known as trown | 15:24 | |
| *** gmann is now known as gmann_afk | 15:25 | |
| jeblair | AJaeger: agreed. what topic flags should we use? | 15:25 |
| tmorin | hi infraroot: I have a template producing both a check and a gate job, I need the check job becoming non-voting, and (consistently) want to plainly disable the gate job ... how can I do that ? | 15:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix wrong paths for api-guide/api-ref https://review.openstack.org/512319 | 15:25 |
| AJaeger | jeblair: just choose one ;) | 15:26 |
| * AJaeger needs to go offline now, will be back later | 15:26 | |
| jeblair | mordred: do you remember who reported the magnum issue? | 15:26 |
| jeblair | tmorin: can you point me at the template? | 15:27 |
| *** kiennt26 has quit IRC | 15:27 | |
| tmorin | jeblair: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n1200 | 15:28 |
| jeblair | tmorin: delete the gate section entirely, then under 'branches' in the check job say "voting: false" | 15:28 |
| tmorin | jeblair: I was hopping that there would be something like "voting: false", but for completely disabling | 15:28 |
| jeblair | tmorin: well, removing it is completely disabling it | 15:29 |
| tmorin | jeblair: I don't/can't change the template definition, I just want the gate job disabled in one place where the template is used | 15:29 |
| *** Shrews has joined #openstack-infra | 15:29 | |
| *** Shrews has quit IRC | 15:29 | |
| *** Shrews has joined #openstack-infra | 15:29 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Add ansible-role-k8s-cookiecutter https://review.openstack.org/512323 | 15:29 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: ansible-role-k8s-cookiecutter to zuul.d/projects https://review.openstack.org/512330 | 15:29 |
| pabelanger | yah, creating a new project-template for non-voting is how we usually had it | 15:29 |
| tmorin | jeblair: sorry, I realise I wa a bit elliptic | 15:29 |
| mwhahaha | tmorin: jeblair: we think we spotted, it we're getting git.o.o instead of review.o.o for the changes somewhere | 15:30 |
| pabelanger | handle* | 15:30 |
| jeblair | tmorin: maybe don't use the template and just manually add the check job? | 15:30 |
| *** slaweq has joined #openstack-infra | 15:30 | |
| *** sbezverk has quit IRC | 15:31 | |
| tmorin | jeblair: ah, yes, sure ( no risk that I might get out-of-sync with whatever the template would tweak in the job later ?) | 15:31 |
| *** bramleya_ has joined #openstack-infra | 15:31 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: ansible-role-k8s-cookiecutter to zuul.d/projects https://review.openstack.org/512330 | 15:31 |
| *** dizquierdo has quit IRC | 15:32 | |
| tmorin | mwhahaha: nice if you have a fix, but given past week experience, I'd prefer the job to be non-voting, for some time at least, and then see how this goes | 15:32 |
| mwhahaha | tmorin: sure | 15:32 |
| clarkb | mordred: is http://logs.openstack.org/25/512325/1/check/tox-py35-on-zuul/8fc7774/tox/ self testing (this is change that udpates how we collect tox logs)? | 15:33 |
| tmorin | mwhahaha: we will notify if we see it breaking too often, if you need to, but I guess you must have a tool for that, don't you ? | 15:33 |
| jeblair | tmorin: oh definitely, but you're already out of sync. you could define 2 templates, one that's non-voting. but a template for a single job in a single pipeline seems unecessary. | 15:33 |
| mwhahaha | tmorin: we do but it's beneficial to have some communication from the other projects (ie feedback) | 15:33 |
| tmorin | mwhahaha: understood & agreed | 15:34 |
| *** dbecker has joined #openstack-infra | 15:34 | |
| *** slaweq has quit IRC | 15:34 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: n8g-bgpvpn: make triple-o job check/nv https://review.openstack.org/512332 | 15:35 |
| *** jogo has quit IRC | 15:36 | |
| clarkb | digging into inode usage http://paste.openstack.org/show/623745/ it does seem to largely be git repos | 15:36 |
| jeblair | odyssey4me: legacy-ansible-func itself is defined as non-voting: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n7 | 15:36 |
| *** thorst has quit IRC | 15:37 | |
| odyssey4me | jeblair that would explain it - any idea why? | 15:37 |
| *** gongysh has joined #openstack-infra | 15:37 | |
| *** udesale has joined #openstack-infra | 15:38 | |
| jeblair | odyssey4me: perhaps a bug in the migration script? | 15:38 |
| odyssey4me | jeblair pabelanger assuming it's alright with you, I'll push up a review to remove that | 15:38 |
| jeblair | odyssey4me: sounds good | 15:38 |
| pabelanger | ++ | 15:39 |
| Shrews | pabelanger: there are 2 held nodes from the multinode-integration-fedora-26 job hold. probably hold overs from before we had the bug fix in. can i delete those? | 15:39 |
| pabelanger | Shrews: yes | 15:39 |
| jeblair | mordred: +3 512328 ? | 15:39 |
| Shrews | pabelanger: done | 15:40 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Switch legacy-ansible-func to voting https://review.openstack.org/512334 | 15:41 |
| clarkb | I've updated the etherpad with info collected re inodes | 15:41 |
| pabelanger | Shrews: danke | 15:41 |
| *** ifat_afek has joined #openstack-infra | 15:41 | |
| mhayden | how should zuulv3 jobs be handled if they need sudo? the revoke-sudo role appears to remove it | 15:42 |
| *** e0ne_ has joined #openstack-infra | 15:42 | |
| jeblair | mhayden: don't use that role | 15:42 |
| mhayden | jeblair: hah! i guess i should have my own 'pre' playbook to resolve that? :) | 15:43 |
| jeblair | mhayden: iirc, we don't do it in all jobs, just unit test jobs | 15:43 |
| *** slaweq has joined #openstack-infra | 15:44 | |
| jeblair | mhayden: what kind of job are you building, and what were you going to use as a parent? | 15:44 |
| *** e0ne_ has quit IRC | 15:44 | |
| mhayden | jeblair: it's a tox job that runs ansible playbooks, and i'm using openstack-tox as a the parent | 15:44 |
| *** jpena is now known as jpena|brb | 15:45 | |
| pabelanger | Did we write an openstack-tox-with-sudo jobs? | 15:45 |
| fungi | and tox is firing scripts which invoke sudo? | 15:45 |
| *** e0ne has quit IRC | 15:45 | |
| pabelanger | yah, we have gate-{name}-tox-with-sudo-{envlist} today in JJB for that | 15:45 |
| jeblair | there are a lot of 'functional' test jobs which use sudo | 15:45 |
| jeblair | however, openstack-tox -> tox -> unittests. and unittests uses revoke-sudo | 15:46 |
| *** caphrim007 has quit IRC | 15:46 | |
| jeblair | i think there's a slight mismatch here. we should probably get mordred in on this too. | 15:46 |
| jeblair | it seems like folks should be able to easily say "this is a functional test job that uses tox" | 15:47 |
| *** markvoelker has quit IRC | 15:47 | |
| jeblair | (which therefore requires sudo) | 15:47 |
| *** jascott1 has quit IRC | 15:48 | |
| *** slaweq has quit IRC | 15:48 | |
| jeblair | maybe we need an openstack-tox-functional which parents to base, and has a pre-playbook that uses the roles in tox and unittests, but omits revoke-sudo. | 15:49 |
| SpamapS | jeblair: I like that plan. | 15:50 |
| *** wolverineav has joined #openstack-infra | 15:51 | |
| *** e0ne has joined #openstack-infra | 15:51 | |
| pabelanger | could somebody remind me where we are setting up ZUUL_CHANGES on legacy zuulv3 jobs? | 15:51 |
| ifat_afek | Hi, can someone please help me? I’m trying to understand why vitrage release notes no longer exists: https://docs.openstack.org/releasenotes/vitrage/pike.html | 15:52 |
| ifat_afek | As far as I know, nobody has changed anything in Vitrage related to the releasenotes | 15:52 |
| pabelanger | ifat_afek: https://docs.openstack.org/releasenotes/vitrage/ is empty for some reason | 15:53 |
| *** caphrim007 has joined #openstack-infra | 15:53 | |
| pabelanger | ifat_afek: can you update zuulv3-issue on etherpad.o.o in triage section | 15:53 |
| pabelanger | https://etherpad.openstack.org/p/zuulv3-issues | 15:53 |
| fungi | pabelanger: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3#n43 | 15:53 |
| jeblair | pabelanger: is that a zuulv3 issue? | 15:54 |
| clarkb | fungi: digging in we have ~18 million blocks so if we set bytes per inode to fs block size we should get a mostly maxmimum number of useable inodes without wasting space (and that will be ~18 million inodes) | 15:54 |
| fungi | clarkb: and what's the current ratio? | 15:54 |
| pabelanger | jeblair: unknown would need to look more. I can once I figure out an issue with tripleo jobs | 15:54 |
| ifat_afek | pabelanger: sure, I’ll do that, but do you think this is relevant to the latest zuul changes? this link used to exist before, it was somehow deleted | 15:54 |
| pabelanger | fungi: thank you | 15:55 |
| jeblair | dhellmann: can you look into ifat_afek's question? | 15:55 |
| ifat_afek | pabelanger: when I run ‘tox -e releasenotes’ the release notes are generated fine | 15:55 |
| clarkb | fungi: we have 4 million so about 4x larger? | 15:55 |
| clarkb | fungi: blocks size is 4096, that puts it in the 16k range of bytes per inode I think | 15:55 |
| pabelanger | ifat_afek: okay, looking now | 15:55 |
| *** martinkopec has quit IRC | 15:56 | |
| ifat_afek | pabelanger: thanks! At the moment it prevents me from releasing a new Pike version, so I’ll be really happy to have it fixed | 15:57 |
| clarkb | also df's output is in kB not B that was only slightly confusing | 15:57 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Ceilometer: disable Tempest tests for master and Queens https://review.openstack.org/512342 | 15:57 |
| *** bandini has quit IRC | 15:57 | |
| fungi | clarkb: sounds reasonable | 15:58 |
| *** bandini has joined #openstack-infra | 15:58 | |
| clarkb | that is interesting, btrfs doesn't list inodes? (locally on a btrfs fs its lists 0 inodes available and free when running df) | 15:59 |
| fungi | clarkb: i thought btrfs had adaptable inode handling (like reiserfs) | 16:00 |
| *** nikhil has joined #openstack-infra | 16:00 | |
| clarkb | fungi: ah maybe that is it. I mostly just have it over here in a semi corner because I was curious and it hasn't exploded yet | 16:01 |
| jeblair | clarkb: i am not using it because it exploded. | 16:01 |
| clarkb | that and suse seems to really like it and I installed suse | 16:01 |
| clarkb | jeblair: ya I only trust it on the battery backed up static host. Definitely not on my latptop | 16:01 |
| clarkb | for certain values of trust | 16:01 |
| fungi | yeah, i had btrfs explode on me once and was entirely unrecoverable (had to restore from nightly backups) | 16:01 |
| mhayden | jeblair: should i open a bug for the sudo + tox issue? | 16:02 |
| fungi | but that was also back before there was a fsck or any recovery tools | 16:02 |
| jeblair | fungi: yes, btrfs was also part of my restore test strategy. | 16:02 |
| pabelanger | ifat_afek: okay, I don't see any logs for your most recent post jobs yet | 16:02 |
| jeblair | mhayden: well, i was hoping that we could discuss solutions to that, but mordred isn't around and he was heavily involved in setting that stuff up originally | 16:03 |
| *** dave-mccowan has joined #openstack-infra | 16:03 | |
| pabelanger | ifat_afek: however, i did confirm the project should be building releasenotes | 16:03 |
| jeblair | mhayden: i guess why don't you put it on the zuulv3 issues etherpad under the "Issues with Jobs" section | 16:03 |
| clarkb | infra-root I think we should do seomthing like this: for each executor: gracefully stop zuul executor, unmount /var/lib/zuul, mkfs.ext -i 4096, mount -a, start zuul executor | 16:03 |
| clarkb | er mkfs.ext4 -i 4096 /dev/xvde2 | 16:04 |
| jeblair | mhayden: aside from just doing it, that's probably the best way to avoid losing track of it | 16:04 |
| pabelanger | ifat_afek: is there anything ready to merge now? | 16:04 |
| clarkb | jeblair: mhayden fwiw my initial impression is that isn't a bug, instead I think we need something that is tox but also not unittests | 16:04 |
| ifat_afek | pabelanger: yes, should I do workflow+1 so we can check? there is one change that should be approved | 16:05 |
| clarkb | jeblair: so maybe we have tox-unittests and tox-vanilla? | 16:05 |
| clarkb | I guess the bug would be in the organization | 16:05 |
| *** slaweq has joined #openstack-infra | 16:05 | |
| pabelanger | ifat_afek: sure, I just need to know the patch you are going to +A | 16:05 |
| jeblair | clarkb: ya | 16:05 |
| fungi | jeblair: it was part of mine too, but luckily my strategy involved making a btrfs snapshot and then backing that up to another system over the network for extra insurance | 16:05 |
| ifat_afek | pabelanger: that would be https://review.openstack.org/#/c/512206/ | 16:06 |
| jeblair | pabelanger: were you doing something with tox-siblings before? | 16:06 |
| jeblair | ah i found it on the etherpad | 16:07 |
| SamYaple | im not able to access my secret. i defined the secret in .zuul.yaml and try to access it in a playbook and it is not defined. | 16:07 |
| jeblair | a new tox-siblings error appeared under triage, i'm moving it to job issues | 16:07 |
| SamYaple | does the secret needto be defined in project-config? | 16:07 |
| *** udesale has quit IRC | 16:07 | |
| fungi | jeblair: thanks, i just spotted the entry mordred had for what looks like probably the same problem | 16:07 |
| pabelanger | jeblair: yah, I just reported the error, think mordred is working on fix | 16:08 |
| dhellmann | pabelanger, ifat_afek : I'm just catching up here. Let me know if I can help at all. | 16:08 |
| *** d0ugal has quit IRC | 16:08 | |
| jeblair | fungi: i think that may have been you; see line 107 now | 16:08 |
| pabelanger | ifat_afek: thanks | 16:08 |
| jeblair | fungi: ya | 16:08 |
| mhayden | jeblair / clarkb: can do, thanks for looking into it | 16:08 |
| jeblair | SamYaple: no, a secret can be defined anywhere | 16:08 |
| clarkb | SamYaple: the consumption of the secret has to be merged before it will work | 16:08 |
| fungi | jeblair: yep, i saw you move it--thanks! | 16:08 |
| pabelanger | ifat_afek: ready here | 16:08 |
| pabelanger | dhellmann: sure, hope to know more in a few moments | 16:08 |
| jeblair | SamYaple: point me at patches if you got 'em | 16:09 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch legacy-ansible-func to voting https://review.openstack.org/512334 | 16:09 |
| clarkb | jeblair: reading the zuul-executor init script it isn't clear to me that there is a way to gracefully stop that process. Is that something we are able to do? | 16:09 |
| jeblair | clarkb: nope. implement graceful is a "# TODO" | 16:09 |
| *** amoralej has joined #openstack-infra | 16:09 | |
| *** tmorin has quit IRC | 16:09 | |
| fungi | clarkb: i think the changes merged to make running jobs reenqueue on abort from an executor restart at least? | 16:09 |
| jeblair | fungi, clarkb: yes | 16:10 |
| *** yamamoto has joined #openstack-infra | 16:10 | |
| jeblair | clarkb, pabelanger: take a look at https://review.openstack.org/512158 and let me know if you want to merge that before we do an executor restart | 16:10 |
| clarkb | infra-root knowing ^ should I just go ahead and stop an executor now (probably 03 since that is where I've been working) and reformat the xvde2 partition? | 16:10 |
| * clarkb looks | 16:10 | |
| fungi | clarkb: that seems prudent | 16:10 |
| pabelanger | clarkb: looking | 16:11 |
| odyssey4me | I've popped another issue - not urgent into the 'needs triage' queue on the etherpad. | 16:11 |
| SamYaple | jeblair: secret declared: https://github.com/openstack/loci/blob/master/.zuul.yaml#L49 | 16:11 |
| jeblair | i can also see the inode issue being too urgent to slow down with that patch; i just don't have a relative sense for how often we're seeing inode failures vs ssh failures and how to prioritize | 16:11 |
| SamYaple | jeblair: secret in job: https://github.com/openstack/loci/blob/master/.zuul.yaml#L174 | 16:11 |
| clarkb | jeblair: the 60 second hold on the jobdir will consume indoes that we know are currently more precious than we'd like | 16:11 |
| SamYaple | jeblair: secret used: https://github.com/openstack/loci/blob/master/playbooks/loci-builder.yaml#L73 | 16:11 |
| SamYaple | playbook doesnt run that task because secret is not defiend | 16:12 |
| jeblair | clarkb: yes, but we won't run that change before your inode reformat | 16:12 |
| ifat_afek | pabelanger: I approved https://review.openstack.org/#/c/512206/ | 16:12 |
| *** trown is now known as trown|lunch | 16:13 | |
| pabelanger | jeblair: clarkb: I haven't see any POST_FAILURES this morning. But haven't been looking too much. | 16:13 |
| *** mat128 has quit IRC | 16:13 | |
| pabelanger | but, happy to try the patch if we want | 16:13 |
| jeblair | pabelanger, clarkb: if things aren't on fire due to ssh connection issues, let's defer it until the next restart and only do inodes for now. | 16:13 |
| *** dhinesh has joined #openstack-infra | 16:14 | |
| pabelanger | sure, I'm okay with that | 16:14 |
| jeblair | SamYaple: what job are you looking at? | 16:14 |
| clarkb | jeblair: ok that works for me | 16:14 |
| SamYaple | jeblair: any of the publish jobs | 16:15 |
| SamYaple | jeblair: see http://logs.openstack.org/74/743fdb397cf9d0e80ba00bd1b4ef85d1bd41e2ec/post/publish-loci-cinder/43fa65d/job-output.txt.gz#_2017-10-16_07_57_51_062339 | 16:15 |
| dansmith | is the v2->v3 redirect for the status page not in place? | 16:15 |
| SamYaple | the when conditional in this case is "is secret defined" | 16:15 |
| *** yamamoto has quit IRC | 16:15 | |
| clarkb | jeblair: before I start reformating things one question about the ssh wrapper. Isn't a single bwrap used per job? which means it could start the control process and persist across all of the playbook invocations? | 16:15 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config master: Ceilometer: disable Tempest tests for master and Queens https://review.openstack.org/512342 | 16:15 |
| clarkb | dansmith: it is, but I had to run a hard refresh to get it | 16:15 |
| dansmith | clarkb: I don't seem to be getting status.json from it like I did last time | 16:16 |
| *** armax has joined #openstack-infra | 16:16 | |
| dansmith | clarkb: no caching going on there, of course | 16:16 |
| clarkb | dansmith: it redirects you to zuulv3 | 16:16 |
| clarkb | dansmith: so you should get a 301 | 16:16 |
| dhellmann | is the "MERGER_FAILURE" error condition defined somewhere? | 16:16 |
| SamYaple | dansmith: i had to update the script to get it to work | 16:17 |
| dansmith | clarkb: hard refresh gives me the redirect in a browser, but not status.json.. this worked in the first v3 switch | 16:17 |
| dansmith | SamYaple: update how? maybe this is related to the https thing? | 16:17 |
| *** dbecker has quit IRC | 16:17 | |
| SamYaple | i pointed it at zuulv3 directly | 16:17 |
| pabelanger | dhellmann: yah, see zuulv3-issues etherpad. TL;DR: we think it is because we ran out of inodes on zuul-executor. clarkb is working on it | 16:17 |
| dhellmann | ah | 16:17 |
| dansmith | SamYaple: okay last time the redirect worked so we don't have to change again | 16:18 |
| *** dhinesh has quit IRC | 16:18 | |
| clarkb | oh before I stop ze03's executor, do I need to worry about caching the git repos? | 16:18 |
| clarkb | jeblair: ^ | 16:18 |
| SamYaple | dansmith: yea i know. dont know whats up. it might be related to https? | 16:19 |
| SamYaple | dansmith: you can try removing it and see if it follows links after that | 16:19 |
| SamYaple | dansmith: its probably the fact that the 301 redirect is only setup on port 80 | 16:19 |
| dansmith | ah, clarkb^ ? | 16:19 |
| fungi | odyssey4me: looking at the command task, using == for string comparison is not posixly-correct. as to why this seems to be getting executed under dash instead of bash, i'm less sure | 16:20 |
| jeblair | SamYaple: ah, i think i see the problem. secrets bind tightly to playbooks in job definitions. so you have to have the playbook that uses the secret in the same job definition that uses the secret. for example: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n88 uses the tarballs secret in its post-run playbook. then other jobs inherit from that. | 16:20 |
| dhellmann | pabelanger : thanks | 16:20 |
| SamYaple | dansmith: clarkb **im guessing. i havent confirmed the 301 is only on port 80 | 16:20 |
| *** markvoelker has joined #openstack-infra | 16:20 | |
| jeblair | clarkb: one bwrap per playbook, many per job. | 16:20 |
| odyssey4me | fungi any shell task should ideally specify the environment under which it should run | 16:21 |
| *** jpena|brb is now known as jpena | 16:21 | |
| *** lucasagomes is now known as lucas-afk | 16:21 | |
| clarkb | jeblair: ah that explains my confusion. | 16:21 |
| fungi | odyssey4me: agreed, and i thought we usually set it to bash | 16:21 |
| *** rloo has joined #openstack-infra | 16:21 | |
| clarkb | SamYaple: dansmith fungi put the redirect in place and may know more | 16:21 |
| odyssey4me | fungi one would hope so ;) | 16:21 |
| pabelanger | SamYaple: I'm not sure we have https on status.o.o, I've always used http | 16:21 |
| SamYaple | jeblair: oh. so all i would need to do is redeclare the playbook in that job definition. let me try that | 16:21 |
| SamYaple | pabelanger: you do | 16:21 |
| SamYaple | signed cert and all | 16:22 |
| SamYaple | well for zuul.o.o, not zuulv3 yet | 16:22 |
| fungi | SamYaple: dansmith: oh, possible i missed the https vhost block. will check | 16:22 |
| *** d0ugal has joined #openstack-infra | 16:22 | |
| robcresswell | o/ I don't suppose anyone has any insight into why https://review.openstack.org/#/c/512333 is incorrect? Thought I'd try my hand at porting some things to v3 config files. | 16:22 |
| jeblair | clarkb: saving /var/lib/zuul would be nice | 16:22 |
| jeblair | clarkb: should have enough space on / to hold it once executor is stopped | 16:22 |
| *** e0ne has quit IRC | 16:23 | |
| openstackgerrit | Terry Wilson proposed openstack-infra/project-config master: Fix networking-ovn openstack-tox-cover job https://review.openstack.org/512347 | 16:23 |
| *** iyamahat has quit IRC | 16:23 | |
| clarkb | jeblair: ok | 16:23 |
| *** egonzalez has quit IRC | 16:23 | |
| jeblair | robcresswell: i'll take a look | 16:23 |
| * clarkb updates etherpad with an rsync back and forth | 16:23 | |
| jeblair | clarkb: or a tar? | 16:23 |
| robcresswell | jeblair: Much obliged :) | 16:23 |
| pabelanger | SamYaple: https://review.openstack.org/511067/ was the revert | 16:23 |
| fungi | odyssey4me: in that playbook, it looks like the first cmd task does specify "executable: /bin/bash" but the other 3 do not | 16:23 |
| *** e0ne has joined #openstack-infra | 16:24 | |
| odyssey4me | fungi which playbook is that? I can fix it up quick | 16:24 |
| pabelanger | SamYaple: should be able to see in system-config if we had redirect on 443 before | 16:24 |
| *** e0ne has quit IRC | 16:24 | |
| fungi | odyssey4me: openstack-infra/openstack-zuul-jobs/playbooks/legacy/ansible-func/run.yaml | 16:24 |
| SamYaple | pabelanger: the issue is a script that dan wrote we updated in this timeframe to us https over http, so the redirect worked lasttime and not this time | 16:24 |
| clarkb | jeblair: rsync is easy I thinK? I dunno its what I've written down in the etherpad, canyou see if that looks sane to you? | 16:25 |
| *** dizquierdo has joined #openstack-infra | 16:25 | |
| *** e0ne has joined #openstack-infra | 16:25 | |
| tosky | uhm, if I use the devstack job (or a job which inherits it), and I specify vars.devstack_localrc, should I rewrite the entire content or the base job, or will the content be merged with the content from the base job? | 16:25 |
| jeblair | clarkb: i think you would make the rsync authors so happy if you told them it was easier than tar. :) | 16:26 |
| *** e0ne has quit IRC | 16:26 | |
| clarkb | jeblair: ha | 16:26 |
| *** yamahata has quit IRC | 16:26 | |
| pabelanger | SamYaple: which URL did you say https worked on? | 16:27 |
| *** e0ne has joined #openstack-infra | 16:27 | |
| fungi | SamYaple: dansmith: should be redirecting correctly now | 16:27 |
| dansmith | confirmed | 16:27 |
| fungi | i had indeed only added it for http and not https | 16:27 |
| SamYaple | pabelanger: zuul.o.o or zuulv3.o.o (though only zuul.o.o has a valid cert) | 16:27 |
| pabelanger | Oh | 16:27 |
| SamYaple | i have predicted a thing! | 16:27 |
| *** e0ne has quit IRC | 16:27 | |
| *** slaweq has quit IRC | 16:27 | |
| SamYaple | thanks fungi | 16:27 |
| pabelanger | yah, i think fungi just manually changed them | 16:27 |
| pabelanger | I was looking on status.o.o | 16:27 |
| SamYaple | ah | 16:27 |
| fungi | yeah, i'm redirecting from https://zuul.openstack.org/ to http://zuulv3.openstack.org/ in the apache config there now | 16:27 |
| *** slaweq has joined #openstack-infra | 16:27 | |
| *** gongysh has quit IRC | 16:28 | |
| jeblair | clarkb: i made some updates | 16:28 |
| clarkb | jeblair: I have bad memories of tar in subshells piped through sockets to do what essentialy rsync -a does | 16:28 |
| clarkb | jeblair: I see them thanks | 16:28 |
| jeblair | clarkb: erm i wasn't suggesting anything complicated | 16:28 |
| SamYaple | dansmith: we might as well revert the ssl patch if we are just redirecting from https to http | 16:28 |
| clarkb | jeblair: I know, its just hwy I tend to go for rsync in these cases regardless of complexity | 16:28 |
| *** e0ne has joined #openstack-infra | 16:28 | |
| *** ociuhandu has quit IRC | 16:28 | |
| jeblair | clarkb: just tar cvf /root/zuul.tar /var/lib/zuul | 16:28 |
| *** e0ne has quit IRC | 16:29 | |
| SamYaple | jeblair: https://review.openstack.org/#/c/512348/1/.zuul.yaml i am consuming the secret in the post playbook, so this looks right, correct? | 16:29 |
| jeblair | or since it's going to be a lot of files: tar cf /root/zuul.tar /var/lib/zuul | 16:29 |
| *** e0ne has joined #openstack-infra | 16:29 | |
| fungi | SamYaple: i _could_ redirect from https to https, but sending people to a server with a temporary cert seems messy | 16:29 |
| *** chlong has quit IRC | 16:29 | |
| *** e0ne has quit IRC | 16:29 | |
| clarkb | infra-root I think I am ready to begin the filesystem work on ze03, I will be stopping zuul executor there nowish unless you have further input (steps are on the etherpad) | 16:30 |
| SamYaple | fungi: reasonable. i just updated my the script im running to go to zuulv3 directly, i dont want to have the overhead of a redirect each time personally | 16:30 |
| *** xarses has quit IRC | 16:30 | |
| SamYaple | will zuulv3 eventually be just zuul.o.o ? or will it always redirect? | 16:30 |
| fungi | SamYaple: sure, the redirect was intended more as a convenience so that people who were consuming the status.json didn't get an unpleasant surprise | 16:30 |
| *** cdent has joined #openstack-infra | 16:30 | |
| fungi | SamYaple: zuulv3 will (likely somewhat soon) be renamed to zuul.openstack.org | 16:31 |
| jeblair | robcresswell: left comment pointing out the error | 16:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Ensure func->functional toe env mapping works https://review.openstack.org/512351 | 16:31 |
| SamYaple | cool. then ill just ride out the storm :) | 16:31 |
| cdent | Hello openstack infra stalwarts, I don’t know if this is a thing you want to know about, but just saw a POST_FAILURE on https://review.openstack.org/#/c/510626/ | 16:31 |
| fungi | SamYaple: the zuulv3.openstack.org subdomain name is not meant as a long-term stable interface | 16:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Ensure func->functional tox env mapping works https://review.openstack.org/512351 | 16:31 |
| clarkb | ok not hearing objections so here goes | 16:31 |
| SamYaple | got it | 16:31 |
| fungi | cdent: it absolutely is | 16:32 |
| *** sbezverk has joined #openstack-infra | 16:32 | |
| fungi | looking into it now to see if it's a known issue | 16:32 |
| cdent | fungi: well then, I’m glad I mentioned it | 16:32 |
| fungi | yes, thanks! | 16:32 |
| pabelanger | cdent: ++ | 16:32 |
| robcresswell | jeblair: Ah dammit, sorry, I should've caught that. Thanks for your time. | 16:32 |
| pabelanger | clarkb: ++ | 16:32 |
| pabelanger | cdent: sorry | 16:32 |
| clarkb | ok process isn't actually stopping, I think jeblair said last night that was because things keep the socket for console logs open? | 16:33 |
| clarkb | jeblair: do I manually kill the child zuul-execturo in that case? | 16:33 |
| *** jogo has joined #openstack-infra | 16:33 | |
| *** jogo has joined #openstack-infra | 16:33 | |
| jeblair | clarkb: how did you stop it? | 16:33 |
| clarkb | jeblair: service zuul-executor stop | 16:33 |
| clarkb | so via the init script | 16:33 |
| cdent | fungi: I was hoping to get to use finger there (because that would be fun), but sadly no data | 16:33 |
| jeblair | clarkb: on 03? | 16:33 |
| clarkb | jeblair: yes | 16:33 |
| cdent | pabelanger: no worries, clarkb is practically my brother | 16:33 |
| fungi | cdent: aha, the executor for that build was struggling with inode capacity in its scratch space | 16:33 |
| fungi | cdent: we're presently working to remedy that specific case | 16:34 |
| * cdent shakes fist at inodes | 16:34 | |
| cdent | cool, glad to hear it is somewhat under control | 16:34 |
| pabelanger | yah, inodes would be my guess also | 16:34 |
| fungi | cdent: so basically it didn't have room to retrieve the logs from the job node | 16:34 |
| * cdent nods | 16:34 | |
| fungi | pabelanger: i looked in the logs on ze03 to confirm that one | 16:34 |
| jeblair | SamYaple: left comment on 512348 | 16:34 |
| pabelanger | fungi: ++ | 16:35 |
| *** slaweq has quit IRC | 16:35 | |
| *** tesseract has quit IRC | 16:35 | |
| jeblair | clarkb: the finger streaming issue should only leave us with some simple, idle zuul-executor proceseses. but it looks like we still have jobs running | 16:35 |
| clarkb | jeblair: ya I'm noticing disk usage hasn't fallen dramatically | 16:36 |
| fungi | pabelanger: cdent: 2017-10-16 14:40:08,645 DEBUG zuul.AnsibleJob: [build: 6875450d24054b4ab5b44ef331a31bde] msg: 'Unable to write to compressed file: [Errno 28] No space left on device: | 16:36 |
| clarkb | it is falling though | 16:36 |
| jeblair | clarkb: i see 'ABORTED' lines in the log | 16:36 |
| jeblair | clarkb: i think we're just seeing the executor stop | 16:36 |
| jeblair | which, under load, is not immediate. :) | 16:36 |
| clarkb | jeblair: ok I shall practice patience then, thank you for looking | 16:36 |
| *** ifat_afek has left #openstack-infra | 16:36 | |
| *** ifat_afek has joined #openstack-infra | 16:36 | |
| SamYaple | jeblair: ok. otherwise the patchset looks good? | 16:36 |
| *** jpich has quit IRC | 16:37 | |
| *** xarses has joined #openstack-infra | 16:37 | |
| jeblair | SamYaple: i haven't really internalized the whole thing to say whether that's the best approach, but i believe aside from that, it will work. :) | 16:37 |
| *** sambetts is now known as sambetts|afk | 16:37 | |
| *** shardy has quit IRC | 16:39 | |
| jeblair | clarkb: looks like it stopped | 16:39 |
| *** dbecker has joined #openstack-infra | 16:39 | |
| SamYaple | will give it a shot | 16:39 |
| SamYaple | its always easier to iterate from something thats working :) | 16:39 |
| clarkb | jeblair: ya but still using 25GB of disk? I'm checking what is using that really quickly | 16:40 |
| clarkb | jeblair: also 2.2 million inodes. If we've leaked that it could potentialy contribute to the problem we are seeing | 16:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Update CI links to use zuul user https://review.openstack.org/512358 | 16:41 |
| *** milan_ has quit IRC | 16:42 | |
| openstackgerrit | Javier Peña proposed openstack-infra/zuul feature/zuulv3: Fix ref calculation for ZUUL_CHANGES legacy var https://review.openstack.org/512359 | 16:42 |
| clarkb | jeblair: /var/lib/zuul/builds has 16GB left in it | 16:42 |
| pabelanger | just noticed that too | 16:42 |
| *** ijw has quit IRC | 16:42 | |
| fungi | clarkb: the only processes i see with open files on that fs now are your shell session | 16:43 |
| clarkb | executor-git is 8.3GB but I think we expect that as its the cache right? | 16:43 |
| fungi | (su and du) | 16:43 |
| clarkb | we have builds from september 29th and october 2 and 15 | 16:44 |
| jeblair | there are a lot of builds | 16:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: New client command for printing autohold requests https://review.openstack.org/512360 | 16:44 |
| jeblair | some of those may be left over from someone setting 'keep' | 16:44 |
| pabelanger | is it possible we had keep-jobs at some point? | 16:44 |
| clarkb | basically half of our inodes are consumed by these builds | 16:45 |
| jeblair | the ones from oct 15/16 seem likely to be bugs. but they only total 4. | 16:45 |
| jeblair | the rest are probably left over from keep. | 16:45 |
| jpena | pabelanger, dmsimard: I think https://review.openstack.org/512359 should fix the issue with the legacy tripleo jobs in https://review.openstack.org/512270 | 16:45 |
| clarkb | jeblair: does that means I can delete them? | 16:45 |
| pabelanger | clarkb: we should consider checking ze09 too | 16:45 |
| pabelanger | that was the other server to have issues | 16:46 |
| clarkb | pabelanger: plan is to do this for all the executors just one at a time so we don't have an outage | 16:46 |
| clarkb | I can start with 09 next | 16:46 |
| jeblair | clarkb: yep; delete them all | 16:46 |
| pabelanger | jpena: please add it to https://etherpad.openstack.org/p/zuulv3-issues so we can get more eyes on the issue | 16:46 |
| clarkb | jeblair: ok am deleting, then will backup | 16:46 |
| * clarkb updates etherpad first | 16:46 | |
| *** iyamahat has joined #openstack-infra | 16:46 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Switch cistatus page to zuul v3 https://review.openstack.org/508546 | 16:47 |
| jeblair | jpena: oops, thanks | 16:47 |
| *** andreww has joined #openstack-infra | 16:48 | |
| jeblair | jpena, pabelanger: that's going to require an executor restart | 16:48 |
| jpena | pabelanger: it's there already | 16:48 |
| jeblair | the good news is that clarkb is doing those right now | 16:48 |
| *** andreww has quit IRC | 16:48 | |
| clarkb | first one is slow but I expect subsequent restarst to go much quicker :) | 16:48 |
| jeblair | if we can merge that very quickly, we might even be able to sneak it into the restart of ze03 | 16:48 |
| pabelanger | KK, let me +3 then :) | 16:48 |
| *** xarses has quit IRC | 16:49 | |
| jeblair | how about i directly enqueue that into gate? | 16:49 |
| clarkb | jeblair: the other weird thing I noticed is /var/lib/zuul/executor.socket should that be cleaned up when the executor stops? | 16:49 |
| pabelanger | jeblair: efm | 16:50 |
| pabelanger | wfm* | 16:50 |
| jeblair | clarkb: ideally, it make suffer from the same perm issue that causes it to be unable to delete its pid | 16:50 |
| *** andreww has joined #openstack-infra | 16:50 | |
| jeblair | s/make/may/ | 16:50 |
| jeblair | 512359 is in gate | 16:50 |
| clarkb | jeblair: should I delete it too for cleanliness? | 16:51 |
| jeblair | clarkb: shouldn't matter | 16:51 |
| clarkb | ok | 16:51 |
| *** links has joined #openstack-infra | 16:51 | |
| *** wolverineav has quit IRC | 16:52 | |
| pabelanger | going to see why cacti.o.o isn't showing 2nd HDD for ze01 (all executors actually) | 16:52 |
| clarkb | backing up /var/lib/zuul on ze03 now | 16:54 |
| *** markmcclain has quit IRC | 16:55 | |
| * clarkb grabs tea while waiting | 16:55 | |
| *** markmcclain has joined #openstack-infra | 16:56 | |
| *** slaweq has joined #openstack-infra | 16:57 | |
| *** derekh has quit IRC | 16:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix ref calculation for ZUUL_CHANGES legacy var https://review.openstack.org/512359 | 17:00 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Zuulv3: Add note about in-repo stable config https://review.openstack.org/512328 | 17:01 |
| jeblair | clarkb: i'm running kick.sh on ze03 | 17:02 |
| dansmith | clarkb: fungi: sorry for the curt response, I was on a call.. but, status feed is working via https again, thanks a lot for whatever you did | 17:03 |
| *** slaweq has quit IRC | 17:04 | |
| jeblair | clarkb: it's complete; zuul is upgraded on ze03 | 17:04 |
| clarkb | jeblair: thanks will continue with fs there shortly | 17:05 |
| *** cdent has left #openstack-infra | 17:05 | |
| *** trown|lunch is now known as trown | 17:06 | |
| *** yamahata has joined #openstack-infra | 17:06 | |
| jeblair | running kick on all the other ze's now | 17:06 |
| *** sree has quit IRC | 17:07 | |
| *** gordc has joined #openstack-infra | 17:08 | |
| chandankumar | clarkb: hello | 17:08 |
| *** sree has joined #openstack-infra | 17:08 | |
| chandankumar | clarkb: i logged out early on friday | 17:08 |
| clarkb | ok var lib zuul backup is done | 17:10 |
| clarkb | I am unmounting and reformating now | 17:10 |
| clarkb | chandankumar: did you see the questions I had? We will need to get answers for those to see what the best way forward is | 17:10 |
| clarkb | chandankumar: basically what account id do you have logging in and what ssh username do you use to push code | 17:10 |
| chandankumar | clarkb: ssh username chkumar246 and Account ID 12393 | 17:11 |
| *** chlong has joined #openstack-infra | 17:11 | |
| *** sree has quit IRC | 17:11 | |
| *** sree has joined #openstack-infra | 17:12 | |
| *** yamamoto has joined #openstack-infra | 17:12 | |
| clarkb | mkfs resulted in 17751479 blocks with 18972672 inodes so we have about 1 million too many inodes | 17:12 |
| clarkb | infra-root ^ do we want to try tuning that or call it good and move on? | 17:13 |
| clarkb | I could do -N 18000000 ? | 17:13 |
| clarkb | chandankumar: ok once a gerrit admin has a moment we will have to see how those two accounts map onto those items and see what we can do to fix the accounts | 17:14 |
| clarkb | chandankumar: currently in the middle of filesystem reformatting for zuul | 17:14 |
| chandankumar | clarkb: no problem thanks, :-) take your time :-) | 17:14 |
| clarkb | it made the fs 65GB too | 17:15 |
| clarkb | down from 72GB | 17:15 |
| clarkb | er I guess 68GB down from 72GB | 17:16 |
| clarkb | fungi: jeblair ^ any thoughts on if we want to tune that better? | 17:16 |
| andreaf | mordred: is there a way with os_* ansible modules to take a snapshot of a VM? http://docs.ansible.com/ansible/latest/list_of_cloud_modules.html | 17:16 |
| jeblair | clarkb: what lead you to 4096 bytes per inode? | 17:16 |
| clarkb | jeblair: 4096 is the block size so we can only have as many files/inodes as blocks | 17:17 |
| *** mat128 has joined #openstack-infra | 17:17 | |
| clarkb | jeblair: unfortauntely this doesn't account for the overhead of the inodes themselves on the filesystem so we end up with more inodes tahn useable blocks | 17:17 |
| jeblair | clarkb: yeah, but 4096 isn't a typical size for us | 17:17 |
| gordc | hi, just curious, is there any way to support OVERRIDE_<projectname>_PROJECT_BRANCH functionality in zuulv3? | 17:17 |
| *** yamamoto has quit IRC | 17:18 | |
| jeblair | gordc: yes: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.override-branch | 17:18 |
| clarkb | jeblair: we had 4.7 million or so inodes previously or about 1/4 the total on the current fs | 17:18 |
| jeblair | gordc: and https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.required-projects.override-branch | 17:18 |
| clarkb | jeblair: since we were using about half the disk at full inode I could set it up to double the number of inodes rather than 4x them | 17:19 |
| jeblair | clarkb: and that was with the fs 59% full, so we really only need to double... right that | 17:19 |
| clarkb | I think that means we'd roughly run out of disk and inodes at the same time | 17:19 |
| clarkb | ok let me rerun with -i 8192 | 17:19 |
| clarkb | etherpad has been updated | 17:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add field name check to add_ds_graph https://review.openstack.org/512367 | 17:19 |
| *** jpena is now known as jpena|off | 17:20 | |
| *** Apoorva has joined #openstack-infra | 17:20 | |
| clarkb | 18344375 blocks and 9486336 inodes now | 17:20 |
| clarkb | that looks better | 17:20 |
| clarkb | and down to 70GB from 72GB whcih looks about right based on how we have been halving things. Ok moving forward with this | 17:21 |
| jeblair | clarkb: ++ | 17:21 |
| clarkb | backup is restoring | 17:22 |
| *** sree has quit IRC | 17:23 | |
| *** amoralej is now known as amoralej|off | 17:24 | |
| *** camunoz has quit IRC | 17:24 | |
| gordc | jeblair: cool cool. i'll take a look. | 17:24 |
| gordc | thanks | 17:24 |
| tosky | what is a "MERGER_FAILURE" error? ( https://review.openstack.org/#/c/507080/ ) | 17:24 |
| *** sree has joined #openstack-infra | 17:25 | |
| jeblair | tosky: at the moment, likely the issue clarkb is fixing -- our executors are running out of inodes | 17:25 |
| tosky | jeblair: oh, ok, I didn't check the backlog | 17:25 |
| tosky | I will wait then | 17:25 |
| clarkb | backup restore is complete | 17:26 |
| clarkb | jeblair: anything else you want me to do before restarting zuul-executor? | 17:26 |
| jeblair | clarkb: nope | 17:26 |
| clarkb | ok starting now | 17:26 |
| jeblair | tosky: feel free to go ahead and recheck, the odds are still pretty good (and increasing!) that you won't hit it. | 17:26 |
| tosky | jeblair: okidoki, rechecking, thanks! | 17:27 |
| *** slaweq has joined #openstack-infra | 17:28 | |
| clarkb | 2017-10-16 17:27:56,955 DEBUG zuul.AnsibleJob: [build: b7671b4998b54836abc8fbf872789fd7] Ansible complete, result RESULT_NORMAL code 0 | 17:28 |
| clarkb | I think that means it is working | 17:28 |
| jeblair | clarkb: ++ | 17:28 |
| clarkb | I'm going to move on to 09 now if anyone else wants to monitor 03 for problems | 17:28 |
| jeblair | clarkb: i'll tail logs there for a bit | 17:28 |
| clarkb | thanks | 17:28 |
| *** ijw has joined #openstack-infra | 17:29 | |
| *** tosky has quit IRC | 17:29 | |
| jeblair | wow it's busy :) | 17:29 |
| *** sree has quit IRC | 17:29 | |
| clarkb | zuul-executor is stopping on 09 now | 17:30 |
| *** slaweq has quit IRC | 17:32 | |
| *** ijw has quit IRC | 17:33 | |
| EmilienM | project-config/zuul/layout.yaml is only used by zuulv3, correct? | 17:35 |
| EmilienM | err | 17:35 |
| EmilienM | by zuul v2 I mean | 17:35 |
| clarkb | yes | 17:35 |
| clarkb | to v2 | 17:35 |
| EmilienM | ok thx | 17:35 |
| fungi | EmilienM: also known as "unused" at this stage | 17:36 |
| fungi | (zuul v2 is not presently running at all) | 17:36 |
| EmilienM | fungi: wait, aren't we running jenkins? I'm confused | 17:36 |
| * EmilienM monday jokes | 17:36 | |
| fungi | EmilienM: happy monday! | 17:36 |
| AJaeger | jeblair: regarding releasenotes: Did we push out some releasenotes without root-marker? That might explain it ifat_afek' problem. We fixed root-marker today (?). So, why not push up a new change and see whether it appears again? | 17:37 |
| pabelanger | jeblair: when you have time for non zuulv3 things, I'd most welcome a review on https://review.openstack.org/512367 for smaller cacti emails for crontab | 17:37 |
| jeblair | AJaeger: pabelanger was looking into that; may have more info | 17:38 |
| inc0 | good morning ... need your help folks https://review.openstack.org/#/c/508661/ | 17:38 |
| inc0 | ahh | 17:38 |
| inc0 | disregard that ^ | 17:38 |
| inc0 | I was looking at wrong change:( | 17:38 |
| pabelanger | AJaeger: we did update releasenotes job root-marker yesterday, it is possible I did it incorrectly. That was in project-config | 17:39 |
| pabelanger | AJaeger: if you could help find the post jobs logs for https://review.openstack.org/#/c/512206/, that would be awesome | 17:41 |
| *** links has quit IRC | 17:42 | |
| *** dhinesh has joined #openstack-infra | 17:42 | |
| *** dave-mccowan has quit IRC | 17:42 | |
| odyssey4me | pabelanger do you have any objections to me switching up zuul.d/zuul-legacy-jobs.yaml to make all legacy-ansible-func-.* jobs use the same set of playbooks, instead of the current mechanism which has one per distro? | 17:43 |
| pabelanger | odyssey4me: well, you likely want to just copy them inrepo and start modifying them | 17:44 |
| pabelanger | odyssey4me: then we can delete the legacy jobs | 17:44 |
| odyssey4me | pabelanger alright, I guess we can do that too | 17:44 |
| AJaeger | pabelanger: looking | 17:45 |
| *** esberglu has joined #openstack-infra | 17:45 | |
| odyssey4me | pabelanger I was hoping to get into a working state before doing that | 17:45 |
| odyssey4me | much of the brokenness there now is due to the auto-conversion | 17:45 |
| *** smatzek has joined #openstack-infra | 17:45 | |
| pabelanger | odyssey4me: what is the issue you are having? | 17:45 |
| AJaeger | question for the team: Is this knowon "build-tox-manuals-checkbuild build-tox-manuals-checkbuild : ERROR Unable to find playbook /var/lib/zuul/builds/" | 17:45 |
| pabelanger | AJaeger: where did you see that? | 17:45 |
| AJaeger | pabelanger: http://logs.openstack.org/29/29388b98f9999e4f98aaa4f421c090fb580aab10/post/publish-openstack-releasenotes/045cdd0/ is the post log | 17:46 |
| AJaeger | pabelanger: https://review.openstack.org/512370 is the error | 17:46 |
| odyssey4me | pabelanger none of the functional tests are working quite right - turns out I'll end up having to do https://review.openstack.org/#/c/512351/2/playbooks/legacy/ansible-func/run.yaml across many, many jobs | 17:46 |
| ifat_afek | pabelanger: https://review.openstack.org/#/c/512206/ was merged, and now https://docs.openstack.org/releasenotes/vitrage/pike.html is ok | 17:46 |
| AJaeger | pabelanger: sorry, need to step out quickly... | 17:46 |
| odyssey4me | but they all work the same, so I can also just reduce them down to a smaller set | 17:46 |
| pabelanger | AJaeger: Yah, that likely means the post-run path in zuul.yaml is not correct | 17:47 |
| pabelanger | odyssey4me: yah, I'd just make the same change across all run playbooks for now. We don't really want to refactor legacy jobs longer then needed | 17:48 |
| odyssey4me | pabelanger okie dokey | 17:48 |
| *** slaweq has joined #openstack-infra | 17:48 | |
| pabelanger | odyssey4me: did you want to update that patch or just do a follow up? | 17:48 |
| odyssey4me | pabelanger I'll do a follow up | 17:48 |
| pabelanger | ifat_afek: Yay, but still trying to see why that is | 17:49 |
| pabelanger | odyssey4me: kk, +3 | 17:49 |
| odyssey4me | ta | 17:49 |
| inc0 | ok I'll need help after all - NODEPOOL_TARBALLS_MIRROR <- how to get url to local tarballs in zuulv3? | 17:52 |
| inc0 | zuul_site_mirror_fqdn | 17:52 |
| inc0 | ? | 17:52 |
| *** slaweq has quit IRC | 17:53 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Ensure func->functional tox env mapping works https://review.openstack.org/512351 | 17:53 |
| dtantsur | so, is the release notes problem known? our gate (not check?) jobs seem to fail with RETRY_LIMIT | 17:54 |
| dtantsur | or was it a check job.. anyway, a lot of these recently | 17:54 |
| pabelanger | okay, I finally figure out why cacti is having a syntax error in our emails | 17:55 |
| clarkb | 09 is starting again | 17:55 |
| jeblair | AJaeger: sent email to -dev about stable branch project stanza backports | 17:55 |
| fungi | dtantsur: yeah, for projects using nonstandard tox configuration there's an issue with the tox-siblings pre task causnig the job to get repeatedly retried. it's listed under "issues with jobs" (currently bullet #6 i think?) at https://etherpad.openstack.org/p/zuulv3-issues | 17:55 |
| pabelanger | when we try to add graphs for puppetmaster.o.o, it is also matching on ci-puppetmaster.o.o and old.puppetmaster.o.o. So, I thin we can just delete them | 17:55 |
| dtantsur | ack, thanks fungi | 17:55 |
| jeblair | fungi: looks like we should merge those 2 changes | 17:56 |
| fungi | jeblair: yeah, i was just pulling them up now | 17:56 |
| pabelanger | or make the matching logic better | 17:56 |
| jeblair | dtantsur: can you Depends-On: 512325 and 512327 and let us know if it fixes things for you? | 17:58 |
| fungi | a depends-on the latter should be sufficient since its parent is the former | 17:59 |
| clarkb | 2017-10-16 17:58:54,090 DEBUG zuul.AnsibleJob: [build: 418f826adacf4f9086cde51dbd885aa5] Ansible complete, result RESULT_NORMAL code 0 | 17:59 |
| clarkb | with that I'm moving on to 01-10 sequentially | 17:59 |
| fungi | clarkb: looks like good news! | 17:59 |
| clarkb | tracking the completion status on the etherpad | 17:59 |
| fungi | result code 0 is the best result code | 17:59 |
| dtantsur | jeblair: I think both recent patches have W+1 already, so people may not like me modifying them.. | 17:59 |
| *** dave-mccowan has joined #openstack-infra | 17:59 | |
| *** psachin has quit IRC | 18:00 | |
| inc0 | anyone? how can I get url to local cached tarballs in zuulv3?:/ | 18:00 |
| fungi | inc0: i think you want to add this role: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/mirror-info | 18:01 |
| jeblair | dtantsur: okay, if you don't want to wait for a merge/recheck cycle, you could push up a new copy of a change. or you could wait and see. :) | 18:01 |
| dtantsur | jlvillal, rloo, re gate ^^^ | 18:02 |
| jeblair | dtantsur: (just have to remove the change-id so a new one gets generated) | 18:02 |
| inc0 | thanks fungi yeah | 18:02 |
| dtantsur | I'll leave it up for these folks, as I have to run now.. thanks jeblair | 18:02 |
| fungi | inc0: or access the mirror_fqdn variable and then create your own | 18:02 |
| fungi | inc0: (look at the script template in that role for an example) | 18:02 |
| *** dtantsur is now known as dtantsur|afk | 18:02 | |
| *** ldnunes has quit IRC | 18:03 | |
| inc0 | yeah, I see it, thanks fungi | 18:03 |
| SamYaple | are the variables zuul_site_mirror_fqdn and mirror_fqdn different? | 18:03 |
| SamYaple | ive been using zuul_site_mirror_fqdn | 18:03 |
| inc0 | SamYaple: zuul_site.. is good one | 18:04 |
| andreykurilin | hi folks! I got something strange - `ERROR Project openstack/manila does not have the default branch master` . Should I file it at etherpad or just ignore and make recheck? | 18:04 |
| fungi | andreykurilin: a url for the log where you saw that would be great | 18:04 |
| andreykurilin | fungi: https://review.openstack.org/#/c/512246/1 | 18:05 |
| andreykurilin | there is no job log | 18:05 |
| clarkb | 01 executor is stopping now | 18:06 |
| jeblair | andreykurilin: if you can add that to etherpad please, that would be great | 18:06 |
| fungi | andreykurilin: that's a failure mode i haven't seen before, thanks! | 18:06 |
| fungi | looks like the same job worked on that change in the check pipeline ~1.5 hours prior | 18:07 |
| andreykurilin | jeblair: np, will add :) | 18:07 |
| andreykurilin | fungi: yes, everything was ok at first check | 18:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Match HOST_ID using tab character too https://review.openstack.org/512376 | 18:07 |
| jeblair | unfortunately, zuul did not output a finger url, so we don't have the build uuid | 18:08 |
| jeblair | that will make it slightly harder to track down | 18:08 |
| pabelanger | fungi: jeblair: clarkb: ^non-zuulv3 related, but will help fix cacti email spam issue for me. When you have a free moment | 18:08 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Work around non-standard envdir locations https://review.openstack.org/512325 | 18:10 |
| *** baoli has quit IRC | 18:10 | |
| *** tosky has joined #openstack-infra | 18:11 | |
| AJaeger | even a recheck does not help with https://review.openstack.org/#/c/512371/1 - "build-tox-manuals-checkbuild build-tox-manuals-checkbuild : ERROR Unable to find playbook /var/lib/zuul/builds/" | 18:13 |
| AJaeger | jeblair: that is a backport of .zuul.yaml to a stable branch | 18:13 |
| andreykurilin | jeblair: I put it into issues with jobs section | 18:13 |
| AJaeger | adding to the etherpad now | 18:13 |
| clarkb | jeblair: do you think zuul would be ok with me taking two exectuors out of the rotation at the same time? 01 is stopping very slowly... | 18:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add tripleo-quickstart jobs into tripleo change queue https://review.openstack.org/512378 | 18:13 |
| pabelanger | AJaeger: ^another tripleo change queue patch for gate | 18:14 |
| *** yamamoto has joined #openstack-infra | 18:14 | |
| pabelanger | added both to zuulv3-issues etherpad | 18:14 |
| jeblair | clarkb: maybe? probably the load governor will keep us from hitting inode problems more than normal. | 18:14 |
| pabelanger | AJaeger: if you have an idea how to better group jobs, I'd be happy to review too | 18:15 |
| jeblair | andreykurilin, fungi: no space left on device on ze03. likely the inode issue which has since been corrected on that host. | 18:16 |
| fungi | jeblair: yep, that was my conclusion as well | 18:16 |
| *** markvoelker has quit IRC | 18:16 | |
| andreykurilin | but the error message looks quite strange:) | 18:16 |
| pabelanger | mwhahaha: EmilienM: just a heads up, tripleo changes in gate are not in the same change queue right now, so it is possible for broken code to slip through. If you don't mind keeping an eye out on zuulv3.o.o gate pipeline and make sure your specific jobs are in the same shared queue would be helpful | 18:16 |
| *** markvoelker has joined #openstack-infra | 18:17 | |
| *** ldnunes has joined #openstack-infra | 18:17 | |
| jeblair | andreykurilin: yes indeed. :) | 18:17 |
| clarkb | on 03 inode usage percentage remains under disk usage | 18:17 |
| fungi | the inode exhaustion problem on the executors seems to have manifested in a number of different reporting behaviors | 18:18 |
| clarkb | ya | 18:18 |
| *** jcoufal has joined #openstack-infra | 18:18 | |
| pabelanger | clarkb: did you by chance confirm 512359 was installed when you restarted ze03.o.o? | 18:18 |
| clarkb | pabelanger: jeblair said it was and I trusted him :) | 18:18 |
| pabelanger | great! | 18:18 |
| pabelanger | cleaning up etherpad then | 18:18 |
| AJaeger | pabelanger: no idea right now. Do we really need to group them explicitely now? | 18:18 |
| clarkb | pabelanger: well its not on all executors yet | 18:18 |
| clarkb | pabelanger: just 03 and 09 | 18:18 |
| clarkb | working on 01 now | 18:18 |
| *** yamamoto has quit IRC | 18:19 | |
| jeblair | AJaeger: re 512371 -- is that job defined in that repo? if so, you may need to backport the playbooks too | 18:19 |
| fungi | clarkb: speaking of inode utilization, i'm still monitoring progress of the logs.o.o cleanup, and we're now using a lower percentage of inodes than blocks on that filesystem | 18:19 |
| pabelanger | AJaeger: yah, we do that with integrated today. We likely need to do something the same with tripleo moving forward | 18:19 |
| clarkb | fungi: woo | 18:19 |
| jeblair | fungi: fascinating | 18:19 |
| pabelanger | fungi: yah, I see that this morning, great news | 18:20 |
| AJaeger | jeblair: yes, there are playbooks with that job in the repo | 18:20 |
| fungi | 64% space in use, 56% inode utilization | 18:20 |
| AJaeger | jeblair: ok, will try... | 18:20 |
| pabelanger | \o/ | 18:20 |
| EmilienM | pabelanger: I see tripleo jobs in zuul v3 gate pipeline | 18:21 |
| *** jcoufal_ has quit IRC | 18:21 | |
| EmilienM | pabelanger: but in different queues | 18:21 |
| pabelanger | EmilienM: yes, but not in same change queue. look at quickstart and tripleo-heat-templates | 18:21 |
| EmilienM | yeah, I just saw it | 18:21 |
| *** markvoelker has quit IRC | 18:21 | |
| pabelanger | EmilienM: yah, thats the issue. have 2 patches up see etherpad | 18:21 |
| pabelanger | we might need more | 18:21 |
| EmilienM | let me see | 18:22 |
| EmilienM | pabelanger: ok I got it, I'll be careful | 18:22 |
| pabelanger | EmilienM: we likey need to create a project-template called tripleo-change-queue (or something) which has enough common jobs to be added to all tripleo / puppet projects. | 18:23 |
| pabelanger | unless there is something today you can think off | 18:23 |
| clarkb | pabelanger: I've +2'd your cacti script fix but not approved it in case you wanted jeblair to see it. But feel free to approve it if you like | 18:24 |
| jeblair | clarkb: you wrote " Use hardlinks for the git repos (is this possible? I think we are on the same fs as the source repos)" can you elaborate? | 18:24 |
| clarkb | jeblair: when we copy all the git repos can we use hardlink copies? since the cost seems largely tied up in all the git repos in the builds | 18:25 |
| jeblair | clarkb: you mean when we create the git repos inside the jobdir work/ directories? | 18:25 |
| clarkb | jeblair: I think that if you use git clone with the hardlink option you can treat the local repo as different than the source even with hardlinks (it just won't be a perfect copy as you change it) | 18:25 |
| clarkb | jeblair: yes | 18:25 |
| jeblair | clarkb: we already do that | 18:25 |
| clarkb | ah ok so this is already just all the unique bits and changes for each job? | 18:25 |
| jeblair | clarkb: yep | 18:25 |
| fungi | if it's a git clone of a file path git should already hardlink when it sees that is possible | 18:26 |
| pabelanger | clarkb: kk | 18:26 |
| clarkb | in that case increasing the number of inodes is lamost definitely the best answer | 18:26 |
| pabelanger | team, https://review.openstack.org/512308/ and https://review.openstack.org/512378/ could use a +3 to help tripleo jobs form a change queue in gate | 18:27 |
| jeblair | clarkb: some jobs still use a couple hundred MB of data for git repos, even though they hard-linked (for instance, a full devstack legacy repo job) | 18:27 |
| clarkb | jeblair: the paste.o.o link in the etherpad shows what inode usage looks like for a specific build | 18:28 |
| clarkb | but ya | 18:28 |
| jeblair | clarkb: du will count those once | 18:28 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Ignore missing tox log directories https://review.openstack.org/512327 | 18:29 |
| clarkb | jeblair: ya I think du --inodes does that right thing? | 18:29 |
| jeblair | clarkb: basically, to figure out the delta, you need to du two copies of the same repo, then subtract the du for one copy | 18:29 |
| jeblair | clarkb: that will give you "incremental usage of things not covered by hard links" | 18:30 |
| AJaeger | jeblair: yes, that was it - backporting playbooks fixes it... | 18:30 |
| jeblair | (hard linked copies, obviously) | 18:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: mention playbooks for backporting https://review.openstack.org/512381 | 18:31 |
| clarkb | the other thing that may potentially help here is repacking to reduce the number of small fiels per repo | 18:31 |
| jeblair | clarkb: oh, erm, we're talking about inodes. inodes *are* the hard links. :) | 18:31 |
| clarkb | jeblair: yes | 18:31 |
| AJaeger | jeblair: documented in 512381 | 18:31 |
| clarkb | jeblair: I'm pretty sure du --inodes will do the right thing when run at the top level of the builds dir for example liek in my paste | 18:31 |
| jeblair | clarkb: so yeah, we're not goiing to be able to use any fewer of those things. | 18:32 |
| clarkb | jeblair: ya I think repacking is likely the only thing ot possibly use fewer | 18:32 |
| clarkb | and maybe gc'ing if branches get deleted | 18:32 |
| jeblair | clarkb: well, regardless, the accounting issue only applies to block usage. you still need just as many inodes for a hard-linked copy of a git repo | 18:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-quickstart jobs into tripleo change queue https://review.openstack.org/512378 | 18:33 |
| clarkb | oh the way we do branch deletions is to make a tag so likely won't help to gc | 18:33 |
| clarkb | 01 is still running :/ | 18:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Clarify job removal section https://review.openstack.org/512304 | 18:33 |
| jeblair | AJaeger: comments on 512381 | 18:34 |
| clarkb | ps -elf | grep zuul | wc -l is trending down so I guess I just need to keep waiting | 18:34 |
| pabelanger | EmilienM: okay, we have tripleo change queue now. So, should be able to start adding projcts to it. We likely need to do that in project-config/zuul.d/projects.yaml for now | 18:35 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: mention playbooks for backporting https://review.openstack.org/512381 | 18:35 |
| jeblair | clarkb: i wonder if we're killing them one at a time | 18:35 |
| pabelanger | clarkb: yah, on a busy executor it too a while to stop for me too | 18:35 |
| AJaeger | jeblair: fixed, thanks | 18:35 |
| pabelanger | IIRC: pre abort, it was because post playbooks needed to run | 18:36 |
| *** MasterOfBugs has joined #openstack-infra | 18:36 | |
| *** efried has left #openstack-infra | 18:39 | |
| *** pramodrj07 has joined #openstack-infra | 18:39 | |
| *** ociuhandu has joined #openstack-infra | 18:40 | |
| AJaeger | team, if you later find time: upstream-translation-jobs report "error" without any logs - mentioned on etherpad. | 18:41 |
| * AJaeger calls it a day now | 18:41 | |
| clarkb | would there be objections if I started to do two exectuors concurrently? still waiting on 01 and wanting to wait less :) | 18:42 |
| pabelanger | no issue here, we seem to be handing jobs well right now | 18:43 |
| clarkb | ok I'm going to start on 02 as well | 18:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Executor: stop jobs in parallel on shutdown https://review.openstack.org/512383 | 18:45 |
| jeblair | clarkb: speedification of your current task ^ | 18:46 |
| jeblair | something to review while waiting :| | 18:46 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config master: install using pip https://review.openstack.org/512384 | 18:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: mention playbooks for backporting https://review.openstack.org/512381 | 18:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-common to tripleo change queue for gate https://review.openstack.org/512308 | 18:48 |
| clarkb | jeblair: thanks | 18:48 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config master: install gnocchi using pip https://review.openstack.org/512384 | 18:48 |
| jeblair | i'm afk for an hour or so for lunch | 18:49 |
| clarkb | maybe I should do that while I wait | 18:49 |
| *** slaweq has joined #openstack-infra | 18:50 | |
| pabelanger | gordc: fyi, we're migrated to zuulv3, so change to JJB are frozen. You should be making the changes the playbooks for your jobs. | 18:51 |
| * gordc thinks about how to pretend like i read the zuulv3 emails. | 18:52 | |
| gordc | ok. i guess i'll go read them :P | 18:52 |
| *** slaweq has quit IRC | 18:55 | |
| *** sree has joined #openstack-infra | 18:57 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 18:57 |
| jlvillal | mordred: So the RETRY_LIMIT is fixed for the ironic-inspector releasenotes job. | 18:59 |
| jlvillal | mordred: But. Now I see a POST_FAILURE | 18:59 |
| jlvillal | mordred: Looking at: http://zuulv3.openstack.org/ | 18:59 |
| pabelanger | have a log? | 18:59 |
| jlvillal | For https://review.openstack.org/511634 | 19:00 |
| jlvillal | pabelanger: It hasn't finished. And if it has POST_FAILURE, doesn't that mean no log? :( | 19:00 |
| pabelanger | jlvillal: no, there might be a log | 19:00 |
| jlvillal | pabelanger: Ah: http://logs.openstack.org/34/511634/1/check/build-openstack-releasenotes/8c93973/ | 19:00 |
| pabelanger | looking | 19:01 |
| jlvillal | pabelanger: Related patch: https://review.openstack.org/#/c/512327/ | 19:01 |
| fungi | if the post failure is for the log uploading task then there often won't be logs, but there are other post tasks which can fail and we'd still get logs | 19:01 |
| jlvillal | pabelanger: I think that got rid of the retry_limit | 19:01 |
| jlvillal | pabelanger: fungi: Thanks | 19:01 |
| pabelanger | rsync: change_dir "/home/zuul//src/git.openstack.org/openstack/ironic-inspector/.tox/releasenotes/log" failed: No such file or directory (2) | 19:01 |
| pabelanger | Hmm, I think we might have a patch to fix that | 19:02 |
| pabelanger | checking | 19:02 |
| fungi | jlvillal: we just merged a couple of suspected fixes for the ironic-inspector (and similar) releasenotes jobs | 19:02 |
| *** sree has quit IRC | 19:02 | |
| jlvillal | pabelanger: https://review.openstack.org/#/c/512327/ was kind of like that. It merged. | 19:02 |
| fungi | pabelanger: jlvillal: that particular problem should be addressed by https://review.openstack.org/512327 | 19:03 |
| fungi | which merged at 18:29 utc, a little over half an hour ago | 19:03 |
| jlvillal | fungi: I did the check after that | 19:03 |
| jlvillal | fungi: I think it fixed the RETRY_LIMIT | 19:03 |
| pabelanger | fungi: jlvillal: no, we are running that patch I think | 19:03 |
| pabelanger | http://logs.openstack.org/34/511634/1/check/build-openstack-releasenotes/8c93973/ara/file/4c9220a4-5726-44db-86c0-bfcbe79a97c5/#line-34 | 19:03 |
| *** edmondsw has quit IRC | 19:04 | |
| fungi | there was a patch just before that one which addressed the tox-siblings issue resulting in the retry_limit failures for ironic-inspector (and similar) release notes jobs | 19:04 |
| *** edmondsw has joined #openstack-infra | 19:04 | |
| fungi | https://review.openstack.org/512325 which merged at 18:10 utc | 19:04 |
| fungi | pabelanger: okay, so telling rsync to include --ignore-missing-args is insufficient i guess | 19:05 |
| pabelanger | yah | 19:05 |
| pabelanger | i think /home/zuul//src/git.openstack.org/openstack/ironic-inspector/.tox/releasenotes/log is missing | 19:05 |
| pabelanger | and ignore-missing-args is for inside the directory | 19:06 |
| pabelanger | I think we need to stat the folder first or use the glob match clarkb suggested this morning | 19:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 19:07 |
| fungi | oh, glob instead of iterating sounds safe enough | 19:07 |
| jlvillal | pabelanger: fungi: I'm proposing a change to the tox.ini | 19:07 |
| pabelanger | or that | 19:08 |
| inc0 | another question, how do I publish logs in v3? we had post job in v2 | 19:08 |
| *** edmondsw has quit IRC | 19:09 | |
| fungi | oh, also, why is that job running (and failing) the same post playbook twice? | 19:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Set v3 nodepool citycloud timeout to 600 https://review.openstack.org/511656 | 19:09 |
| inc0 | basically I have all logs in dir, how do I make them visible in http://logs.openstack.org/59/508759/55/check/kolla-ansible-centos-source/161c76b/ ? | 19:09 |
| pabelanger | inc0: you'd sync then back to executor into the zuul.executor.log_src folder | 19:11 |
| pabelanger | we should have something in devstack-gate that does this today | 19:11 |
| inc0 | rsync module? | 19:11 |
| SamYaple | W00T!!! a successful dockerhub push using secrets in zuulv3 | 19:11 |
| SamYaple | thanks everyone for your help | 19:11 |
| inc0 | cool SamYaple :) | 19:11 |
| pabelanger | inc0: see https://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/fetch-devstack-log-dir for an example | 19:12 |
| inc0 | thank you Paul | 19:12 |
| pabelanger | inc0: there is also a proposal from mordred to make this a little easier | 19:12 |
| pabelanger | inc0: see opentack-infra ML for more on the topic | 19:13 |
| pabelanger | SamYaple: cool, url? | 19:13 |
| inc0 | I will, although this seems straightforward to me | 19:13 |
| SamYaple | pabelanger: to the images? https://hub.docker.com/r/openstackloci/requirements/tags/ (master-ubuntu and master-centos tags) | 19:13 |
| pabelanger | SamYaple: neat, have a log file handy? | 19:14 |
| SamYaple | pabelanger: indeed i do http://logs.openstack.org/59/59e8266aa0856a5462ec44820401cec52c8731f7/post/publish-loci-requirements/c7e9047/job-output.txt.gz#_2017-10-16_19_05_50_275939 | 19:14 |
| pabelanger | I like the warning onthe next line :) | 19:14 |
| *** yamamoto has joined #openstack-infra | 19:15 | |
| SamYaple | pabelanger: as long as it doesnt expose the password im fine with it :) | 19:15 |
| pabelanger | SamYaple: where are you keeping your secret / playbook for that? | 19:17 |
| inc0 | speaking of which, infra hosted registry, once you guys feel comfortable with zuulv3 I'd love to get back ot it | 19:17 |
| pabelanger | I see it now | 19:17 |
| SamYaple | pabelanger: https://github.com/openstack/loci/blob/master/.zuul.yaml | 19:17 |
| clarkb | 01 and 02 still running but process count continues to trend down | 19:17 |
| pabelanger | SamYaple: yah, no_log should help with that | 19:18 |
| SamYaple | pabelanger: im running no_log :) | 19:19 |
| SamYaple | pabelanger: https://github.com/openstack/loci/blob/master/playbooks/push.yaml#L8 | 19:20 |
| *** yamamoto has quit IRC | 19:20 | |
| pabelanger | SamYaple: cool, now to see if they rate limit your uploads | 19:20 |
| *** hemna_ has quit IRC | 19:20 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Start ssh persistent connections outside bwrap https://review.openstack.org/512158 | 19:21 |
| SamYaple | pabelanger: our next step is adding a post job for cinder to build loci images. so well see. | 19:21 |
| SamYaple | we just need to figure out how to do that now | 19:22 |
| pabelanger | shouldn't be hard to do that | 19:23 |
| *** rbrndt has quit IRC | 19:23 | |
| SamYaple | would having cinder post pipeline run a loci job mean that the loci secrets/playbooks/job definitions would need to move to project-config? | 19:24 |
| pabelanger | i don't think so, jobs are currently global | 19:25 |
| pabelanger | so, any project should be able to reference that | 19:25 |
| *** bramleya_ has quit IRC | 19:25 | |
| pabelanger | yah, I think that is right | 19:26 |
| jlk | So I've been out of it for a bit, where do things stand with zuul v3 and infra production? | 19:26 |
| SamYaple | but the job would be running as cinder, so i would have to adjust the allowed-projects on the secrets if im recalling correctly | 19:26 |
| * mordred waves to everyone from the air | 19:26 | |
| pabelanger | jlk: https://etherpad.openstack.org/p/zuulv3-issues is the current issues we are working on | 19:26 |
| pabelanger | jlk: but alive and working | 19:26 |
| pabelanger | mordred: ^5 | 19:27 |
| jlk | is it gating for everything now, or still a minimal set? | 19:27 |
| pabelanger | jlk: everything, we rolled out 22:00UTC yesterday | 19:27 |
| SamYaple | jlk: everything | 19:27 |
| jlk | nice | 19:27 |
| pabelanger | only a few bumps so far | 19:27 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use weakref for change cache https://review.openstack.org/511355 | 19:27 |
| pabelanger | but all and all, looking real good! | 19:28 |
| jlk | alrighty. I'll try to get some reviewing done today. | 19:28 |
| jlvillal | pabelanger: fungi: As a note, for ironic-python-agent, we also set the envdir for 'docs', but that job passes. Not sure how the sphinx-docs job differs from the releasenotes job. | 19:28 |
| SamYaple | if all job definitions are global, does that mean that from the cinder repo they could install a job in the nova pipelines? | 19:28 |
| jeblair | mordred: o/ if you have time to think about a tox base job that doesn't use sudo, that'd be great. see "Need a openstack-tox job that doesn't revoke sudo" in etherpad | 19:28 |
| jlvillal | pabelanger: fungi: Of course we can't look at the logfiles for those jobs unless the job fails. | 19:29 |
| jlk | SamYaple: "install a job in the nova pipelines" waht does that mean? | 19:29 |
| * jeblair continues lunching | 19:30 | |
| SamYaple | jlk: define a job in the cinder .zuul.yaml that says "run this job in the nova check pipeline" | 19:30 |
| jlk | hrm. | 19:30 |
| jlk | I know the job definition could live in cinder, I'm not sure if you can make additions to the nova project definition. | 19:31 |
| *** mordred has quit IRC | 19:31 | |
| tosky | SamYaple: I don't think you can do that; which syntax would you use for that? | 19:31 |
| SamYaple | jlk: but its all global... | 19:31 |
| jlk | yes, global. But not necessarily /merged/ | 19:32 |
| SamYaple | tosky: https://review.openstack.org/#/c/512394/ | 19:32 |
| SamYaple | jlk: thats not what ive been lead to believe | 19:32 |
| jlk | If you try to define a project with a specific name in two different locations you may get a conflict | 19:32 |
| SamYaple | jlk: thats why we need uqniuely named jobs still, because it _is_ merged | 19:32 |
| jlk | just like if you define a job in two different places you get conflicts | 19:32 |
| tosky | SamYaple: and in fact zuul refused it | 19:33 |
| jlk | Maybe we're talking past each other | 19:33 |
| tosky | you can't use another unprivileged queue | 19:33 |
| *** hemna_ has joined #openstack-infra | 19:34 | |
| SamYaple | right. so my original question was to do this, i must put this in project-config | 19:34 |
| jlk | When I say "merged" i mean zuul would interleave the dictionary content of the two definitions, which is ... difficult | 19:34 |
| jlk | By global, I mean they all exist in the same namespace, but duplications aren't necessarily allowed | 19:34 |
| fungi | SamYaple: project A can use a job defined in project B, but project B cannot force project A to use a job it defines | 19:34 |
| jlk | SamYaple: you could put it in nova's repo | 19:34 |
| jlk | if the project is defined in nova's repo, you would make the addition there | 19:34 |
| fungi | (excepting for the project-config repo) | 19:35 |
| jlk | Define the job in loci, make use of it in the nova repo, and the cinder repo | 19:35 |
| SamYaple | our end goal is for every merged commit to cinder to trigger a post job publish-loci-cinder. i will accept advice on how to do this | 19:35 |
| *** slaweq has joined #openstack-infra | 19:36 | |
| SamYaple | is the best way to A) put this in project-config or B) put this in cinder | 19:36 |
| SamYaple | secrets from openstack/loci are involved here | 19:36 |
| jlk | Add it to the place where the cinder project is defined. | 19:36 |
| jlk | Sorry, by "it" I mean the configuration of the job to run in the post pipeline | 19:36 |
| fungi | SamYaple: cinder would need to add that job to run in the post pipeline, but it can use the job defined in loci | 19:36 |
| jlk | Somewhere there is a - project: name: openstack/cinder block | 19:37 |
| jlk | that location is where you'd add the new job | 19:37 |
| SamYaple | jlk: you mean i need to copy around all the playbooks in openstack/loci to openstack/cinder to do this? | 19:37 |
| jlk | no | 19:37 |
| jlk | you reference the job name in openstack/cinder | 19:37 |
| jlk | the content can all live in openstack/loci | 19:37 |
| SamYaple | ok im following you | 19:37 |
| SamYaple | what about the secrets? | 19:37 |
| SamYaple | i believe i recall an allowed-projects list for secrets for cross-project secrets? am i mistaken? | 19:38 |
| SamYaple | or will the fact taht the job is defined in the repo with teh secret all it to use the secret? | 19:38 |
| pabelanger | cinder won't have access to the secret | 19:38 |
| pabelanger | only your job in loci will | 19:38 |
| jlk | "A Secret may only be used by jobs defined within the same project." | 19:38 |
| jlk | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#secret | 19:38 |
| pabelanger | secrets are tied to playbooks | 19:38 |
| pabelanger | yah, what jlk said | 19:39 |
| jlk | To run the job, Zuul will check out the job content from the place it's defined | 19:39 |
| jlk | and have access to the secret to run that playbook | 19:39 |
| SamYaple | so calling the job will let cinder run the job, and the job has access to the secret, but at no point cinder had access to the secret, correct? | 19:39 |
| fungi | so the job can run triggered in cinder's post pipeline, and as long as the secret and job are both defined in loci that should work fine | 19:39 |
| SamYaple | sorry, defining the job | 19:39 |
| pabelanger | SamYaple: right | 19:39 |
| jlk | SamYaple: nobody could write a job in cinder's repo to make use of the secret in your repo | 19:39 |
| SamYaple | got it. im on the same page now | 19:40 |
| SamYaple | thanks for the clarity | 19:40 |
| jlk | only the jobs in your repo (if that's where the secret is defined) can access the secret | 19:40 |
| jlk | always good to check! And to bubble up things that aren't clear | 19:40 |
| pabelanger | and if they can, quietly let us know | 19:40 |
| SamYaple | heh will do | 19:40 |
| fungi | SamYaple: now, this _does_ point out a safety concern... don't want to write jobs which run arbitrary code which could be used to expose secrets that job uses | 19:40 |
| SamYaple | fungi: only use it on one line, and i set no_log | 19:41 |
| *** ralonsoh has quit IRC | 19:41 | |
| *** slaweq has quit IRC | 19:41 | |
| tosky | switching topic a bit, just to be sure: if a job defined in repo A is used in repo B, will repo A be cloned as well or should it explicitly be put into required-projects? | 19:41 |
| openstackgerrit | gordon chung proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 19:41 |
| SamYaple | tosky: i second that question | 19:41 |
| fungi | SamYaple: right, just pointing out you want to avoid patterns similar to `setup.py upload` or `npm upload` which run arbitrary code from the project being uploaded | 19:42 |
| *** mordred has joined #openstack-infra | 19:42 | |
| jlk | I thought that Zuul will automatically clone the repo the job exists in, in order to access the playbook | 19:42 |
| tosky | that makes sense; but does it copy it to the workspace too? | 19:42 |
| jlk | at least cloned to the executor, not necessarily exposed in teh workspace. that I'm less clear on | 19:42 |
| SamYaple | fungi: good info to have, thanks | 19:42 |
| fungi | tosky: repo B will be cloned automatically, but they _will_ need to add project A to the required-projects list for their variant | 19:42 |
| * mordred waves to everyone from the air again after reconnecting his irc bouncer to freenode | 19:43 | |
| pabelanger | these are exciting questions | 19:43 |
| tosky | fungi: oh, good to know | 19:43 |
| SamYaple | such depth. much wow | 19:43 |
| tosky | my use case is: a job derived from the (work-in-progress) devstack-tempest, which will be put in sahara-tests | 19:43 |
| jlk | fungi: "their variant" means what? | 19:43 |
| tosky | called sahara-tests-tempest | 19:43 |
| tosky | and that will be used in sahara and python-saharaclient (in addition to sahara-tests) | 19:43 |
| pabelanger | tosky: yah, you'll want to use required-projects | 19:44 |
| tosky | good to know :) | 19:44 |
| tosky | I would have probably discovered it, but one review less is better | 19:44 |
| fungi | jlk: when they put the job in their projects list, they can set parameters on it (including required-projects) and, unless i'm mistaken on the terminology, that technically makes it a variant of the original | 19:44 |
| pabelanger | tosky: yah, zuul does a good jobs telling you if something is missing. It should results in a syntax error | 19:44 |
| jlk | fungi: wouldn't that mean ANY application of a job is a variant? | 19:45 |
| tosky | now the question is: what is the reason for not making that repository available automatically? | 19:46 |
| fungi | jlk: oh, maybe i'm still mistaken on exactly what constitutes a job variant in that case | 19:46 |
| tosky | is there a use case where I would need only the playbooks/roles/zuul stuff but not the rest of its content? | 19:46 |
| * jlk thinks a bit | 19:47 | |
| pabelanger | first definition of a job is reference, anything else name the same as the reference job, is a variant | 19:47 |
| jlk | "exposure" of content isn't a built-in to Zuul. It's done as a pre-task | 19:47 |
| jlk | cloning happens on the executor, which uses it for running playbooks. What's on the remote node depends on the base job | 19:47 |
| fungi | pabelanger: okay, that's what i thought as well | 19:48 |
| pabelanger | had to double check in the docs :) | 19:48 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish cinder loci images to DockerHub https://review.openstack.org/512398 | 19:49 |
| tosky | ok, another question | 19:50 |
| mordred | tosky, jlk: yes, there is a difference in repos cloned so that zuul can use theirplaybooks/roles to execute jobs - and repos needed on the remote node for job content itself | 19:50 |
| jlk | tosky: jobs are free form, there's no real standard that the job (playbooks) make any use of the repo contents. | 19:50 |
| SamYaple | ok so fungi this also adds openstack/loci as a required-project, did i do that right https://review.openstack.org/#/c/512398/ ? | 19:50 |
| jlk | tosky: like the job could just be curling against remote URLs, and won't even need a node from nodepool | 19:50 |
| *** armax has quit IRC | 19:50 | |
| tosky | I see | 19:50 |
| jlk | so really, NOTHING is exposed on the remote node, until a pre-job delivers it to said node | 19:50 |
| *** markvoelker has joined #openstack-infra | 19:51 | |
| tosky | the documentation for a job says " If parent is set to null (which is only valid in a config-project), this is a base job." | 19:51 |
| mordred | for instance, the base job comes from project-config, so when running the base job pre-playbooks zuul clones project-config at is master state on the executor so that it has the roles/playbooks ... but we don't rsync project-config to the remote nodes because that wouldn't be very helpful | 19:51 |
| *** markvoelker has quit IRC | 19:51 | |
| jlk | and those pre-jobs are up to Zuul instance admins, and in this case, openstack-infra | 19:51 |
| pabelanger | SamYaple: you shouldn't need require-projects, since the job exists in openstack-loci | 19:51 |
| mordred | and, in fact, a job could be testing a change to project-config and would need the speculative state available for rsyncing to the remote build node so that the test of the patch can take place | 19:51 |
| *** markvoelker has joined #openstack-infra | 19:51 | |
| tosky | about parents: does it mean that if I try to define a job without a parent in a in-tree .zuul.yaml I should get an error, or that there is an implicit parent? | 19:52 |
| SamYaple | pabelanger: "19:42:51 < fungi> tosky: repo B will be cloned automatically, but they _will_ need to add project A to the required-projects list for their variant" | 19:52 |
| jlk | the take away is to use 'required-projects' if you require the content of the repo available during job execution. | 19:52 |
| mordred | tosky: there is an implicit parent called base | 19:52 |
| SamYaple | pabelanger: doesnt that mean project B (cinder) needs to add project A (loci) to it? | 19:52 |
| mordred | jlk: ++ | 19:52 |
| pabelanger | SamYaple: that isn't a job variant however | 19:52 |
| pabelanger | SamYaple: you are just adding a job to a pipeline | 19:52 |
| tosky | mordred: ah, ok, that explains why https://review.openstack.org/#/c/511080/ works | 19:52 |
| SamYaple | ah i see. i need ot research "variant" then | 19:53 |
| mordred | tosky: yes | 19:53 |
| fungi | pabelanger: but the project the job is defined in won't be automatically cloned, will it? just the project triggering the job i thought. are you suggesting it's the other way around? | 19:53 |
| *** edmondsw has joined #openstack-infra | 19:54 | |
| jlk | fungi: the repo where the playbooks exist will be cloned to the executor | 19:54 |
| *** panda has joined #openstack-infra | 19:54 | |
| pabelanger | fungi: we'd need to clone the jobs on the executor, because that is wehere the playbooks live | 19:54 |
| jlk | so that ansible has access to the playbooks | 19:54 |
| pabelanger | Yah, which is not needed on the remove node | 19:54 |
| *** ldnunes has quit IRC | 19:54 | |
| fungi | jlk: pabelanger: but to have it available on the job node you'd still need to add it to required-projects right? | 19:54 |
| jlk | it's just that the content won't then be pushed to the nodepool node. | 19:54 |
| pabelanger | fungi: yah, that is right | 19:55 |
| mordred | yah. there are two different cloning contexts | 19:55 |
| jlk | fungi: correct. | 19:55 |
| mordred | if you look at: | 19:55 |
| *** panda is now known as panda|rover | 19:55 | |
| mordred | http://logs.openstack.org/80/511080/18/check/sahara-image-elements-buildimages-spark/d26ceaf/zuul-info/inventory.yaml | 19:55 |
| *** ldnunes has joined #openstack-infra | 19:55 | |
| fungi | i assumed SamYaple wanted the job to also have access to loci | 19:55 |
| mordred | zuul.projects is the list of things that we're going to rsync to the remote nodes | 19:55 |
| fungi | but maybe that's an incorrect assumption | 19:55 |
| mordred | fungi: yes. if SamYaple is defining a loci job that might want to be added to the cinder pipelines | 19:55 |
| mordred | then that job should required-projects on openstack/loci | 19:56 |
| fungi | aha | 19:56 |
| mordred | it'll not add much to the loci invocations of that job | 19:56 |
| mordred | but it'll be needed for the cinder invocations | 19:56 |
| fungi | whereas if the job only ran against loci changes it wouldn't be needed in required-projects | 19:56 |
| jlk | yeah that can be defined with the job itself | 19:56 |
| jlk | fungi: right, it's implicit there | 19:56 |
| jlk | or well, self serving? | 19:56 |
| dhellmann | the tag-releases job is failing because add-sshkey doesn't have an "ssh_private_key" value. That doesn't feel like something I can fix on my own. Does someone have a few minutes to help out? http://logs.openstack.org/ef/ef00e18f9937e1544d2bedebf26328a7c3d741b8/release-post/tag-releases/e9fc7f2/job-output.txt.gz#_2017-10-16_19_53_58_645691 | 19:56 |
| fungi | or required-projects could be omitted from the job definition in loci, but then cinder would need to add it as a variant | 19:57 |
| pabelanger | I'll look at that | 19:57 |
| jlk | you could do it that way, but that's probably more error prone | 19:57 |
| jlk | easier to define it at the job level, so that consumers can just toss the job on and it'll work as expected | 19:57 |
| tosky | jlk: do you mean the job in A adding required-projects on A? | 19:57 |
| mordred | yes | 19:57 |
| *** markvoelker_ has joined #openstack-infra | 19:57 | |
| clarkb | dhellmann: I'm guessing that is beacuse the job wants to add an ssh key to push tags into gerrit? | 19:57 |
| *** ifat_afek has quit IRC | 19:57 | |
| mordred | since it's a job designed to be run by other repos and the job requires loci | 19:57 |
| dhellmann | clarkb : yes | 19:57 |
| jlk | tosky: correct, the canonical job definition in A can list required-projects of A | 19:57 |
| tosky | that makes sense | 19:58 |
| tosky | I will do it | 19:58 |
| clarkb | dhellmann: so you need to update the job to use the ssh_private_key of that user. If it is already a secret in projcet-config I think that is easy ish but if not ya an infra root will have to add it | 19:58 |
| mordred | jlk, fungi: https://review.openstack.org/#/c/500365/29/.zuul.yaml is an example - look at shade-functional-devstack-base | 19:58 |
| *** rhallisey has quit IRC | 19:58 | |
| dhellmann | clarkb : I understand most of those words, but I'm not sure where to start with that. Is that just a setting on the job definition? How do I figure out what value to use? | 19:58 |
| mordred | jlk, fungi: that puts name: openstack-infra/shade into required-projects: so that if I convinced keystoneauth to use that job it would have the complte set of repos | 19:58 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish cinder loci images to DockerHub https://review.openstack.org/512398 | 19:58 |
| clarkb | dhellmann: ya let me dig up some links (though I'm in the middle of updating local network hard ware so may disappear for short periods of time | 19:59 |
| SamYaple | ok fungi mordred , the job is already functional (it runs successfully from loci) so all the required-projects should be set correctly | 19:59 |
| jlk | well | 19:59 |
| dhellmann | clarkb : in the tag-releases job definition I see 2 secrets. "lp_creds" and "release_ssh_key" | 19:59 |
| jlk | in the case of running on loci jobs, it's assumed that the loci repo exists | 19:59 |
| jlk | on the remote node | 20:00 |
| mordred | yah. the project triggering the job gets added to the projects list automatically | 20:00 |
| jlk | SamYaple: the question is, does the job require content from the loci repo itself? | 20:00 |
| jlk | if it does, then you should add a required-projects: of loci in that definition | 20:00 |
| fungi | SamYaple: a job running against changes/commits for the repo it's defined in is a special case, because it will get its project required automatically by being the one which triggered the job | 20:00 |
| dhellmann | clarkb : and in secrets.yaml I see a release_ssh_key with a private_key that's a list (?) of encoded data | 20:00 |
| jlk | since the job is designed to (potentially) run in pipelines for other projects | 20:00 |
| *** markvoelker has quit IRC | 20:01 | |
| mordred | SamYaple: I'd add openstack/loci to required-projects in your base job: http://git.openstack.org/cgit/openstack/loci/tree/.zuul.yaml#n75 | 20:01 |
| SamYaple | oh jeez. THATS where you all are talking about it | 20:01 |
| tosky | mordred: about that shade example: what would have happened if you omitted the override-branch: master for shade? A run-time error because the, say, stable/ocata branch does not exist? | 20:01 |
| SamYaple | yes. i can add loci there | 20:01 |
| SamYaple | can i do a depends on in project-config to an untrusted repo? | 20:01 |
| mordred | tosky: no - it's just that the stable/ocata branch would be checked out - thing is, I don't actually care about that branch in that case - it would totally work if I hadn't put in that override though | 20:02 |
| mordred | SamYaple: you can - it won't do speculative job config though | 20:02 |
| tosky | mordred: and if the branch does not exist? | 20:02 |
| SamYaple | got it | 20:02 |
| mordred | tosky: it'll fall back to master using normal branch fallback logic | 20:03 |
| mordred | tosky: so if keystoneauth added a shade-functional test to their pipeline and then submitted a patch to stable/ocata of keystoneauth | 20:03 |
| mordred | tosky: without that branch line it would result in stable/ocata of shade being cloned if it existed (which is does) or master if it didn't | 20:03 |
| tosky | mordred: but I guess it's better to add it explicitly? (use case: sahara-tests, described above, is branchless, so technically I don't need that override-branch, but...) | 20:04 |
| mordred | tosky: but the thing I want to test with shade is that master of shade works with stable/ocata of keystoneauth | 20:04 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix the configuration for the release tagging key https://review.openstack.org/512401 | 20:04 |
| dhellmann | clarkb : is it possible this is the fix? ^^ | 20:04 |
| mordred | tosky: yah - sahara-tests being branchless means you do not need that override thing... | 20:04 |
| mordred | tosky: for branch matching shade's world-view is weird - odon't copy that part :) | 20:04 |
| tosky | mordred: ok :) | 20:04 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish cinder loci images to DockerHub https://review.openstack.org/512398 | 20:05 |
| *** hongbin has quit IRC | 20:05 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix release_ssh_key secret https://review.openstack.org/512402 | 20:05 |
| pabelanger | dhellmann: yah, you beat me to it | 20:05 |
| dhellmann | pabelanger : cool, thanks | 20:05 |
| dhellmann | we need to put some jsonschema validation on that file :-) | 20:06 |
| clarkb | pabelanger: dhellmann we also need to make sure that it is using that secret.ssh_private_key on the other end right? | 20:06 |
| SamYaple | phew. thanks all for sticking with me! i think i have a handle on it now | 20:06 |
| pabelanger | clarkb: it will once we update the secret | 20:06 |
| SamYaple | mordred: i did successfully push to dockerhub from a post job this morning :) | 20:06 |
| mordred | SamYaple: that patch looks fine. NOW - just for sake of being complete with the discussion - you could also just submit that patch to the cinder repo | 20:06 |
| dhellmann | clarkb : that key is listed in the set of secrets | 20:06 |
| dhellmann | for the job | 20:06 |
| dhellmann | I'm not sure if you mean go deeper than that in the validation? | 20:06 |
| mordred | SamYaple: project-config vs. cinder in this case is a social question, not a technical one | 20:06 |
| clarkb | dhellmann: in the place where we attempt to set the key | 20:07 |
| pabelanger | dhellmann: which server is that key for? | 20:07 |
| clarkb | pabelanger: its a user key for a gerrit user | 20:07 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 20:07 |
| dhellmann | pabelanger : it's used by the tag-releases job. I don't know what server. | 20:07 |
| pabelanger | ah, okay, so we don't SSH into the server | 20:07 |
| dhellmann | no | 20:07 |
| SamYaple | mordred: cinder repo has no .zuul.yaml or zuul.d yet | 20:07 |
| pabelanger | dhellmann: and we don't validate known_hosts? | 20:08 |
| mordred | SamYaple: cool. mostly wanted to just make sure it was understaood it was physically possible | 20:08 |
| mordred | pabelanger, dhellmann we should ALWAYS validate known_hosts | 20:08 |
| fungi | SamYaple: it could also be done in project-config in that case, but reviewers would want to see +1 from cinder infra liaison or ptl | 20:08 |
| SamYaple | mordred: and we talking about ti a bit, it seems like it makes sense to keep them seperate. because this in a post job in no way affects anything in cinder | 20:08 |
| mordred | no? | 20:08 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 20:08 |
| pabelanger | mordred: right, we likely should be adding ssh_known_hosts to that secret too | 20:08 |
| dhellmann | pabelanger , mordred : how do I make it do that? | 20:08 |
| mordred | pabelanger, dhellmann: that ssh key is the one used by git review to communicate with gerrit, yeah? | 20:08 |
| dhellmann | it's used to let us push tags | 20:09 |
| * mordred looks real quick | 20:09 | |
| SamYaple | fungi: patch is already up for project-config. have a +1 from the cinder team, if you need it from the PTL/cinder infra liasion ill look into that | 20:09 |
| mordred | yah. it's the ssh key for the account on gerrit IIRC - one sec | 20:09 |
| dhellmann | I see an "ssh_known_hosts" setting on some other keys | 20:09 |
| pabelanger | mordred: dhellmann: you might be able to use proposal_ssh_key for info | 20:09 |
| mordred | pabelanger, dhellmann: I'm reading the full job stack real quick ... | 20:10 |
| pabelanger | kk | 20:10 |
| dhellmann | this stuff isn't new, it just didn't migrate over cleanly I think | 20:10 |
| dhellmann | I'm assuming this is the key we were using before? | 20:10 |
| dhellmann | some guy named "Monty" put it in the file | 20:11 |
| *** rbrndt has joined #openstack-infra | 20:11 | |
| clarkb | pabelanger: dhellmann ya the add-sshkey role needs a ssh_private_key arg which we are not passing in | 20:11 |
| fungi | i'm still trying to piece together how the tag-releases job works... i see the pre playbook calls an add-sshkey role but where is that role defined? http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/release/pre.yaml#n13 | 20:11 |
| fungi | i don't see it in the roles directory in project-config | 20:11 |
| clarkb | it also takes ssh known hosts and fqdn to set that up | 20:11 |
| fungi | and i thought any role using a secret needed to be defined in the same repo as the secret | 20:11 |
| clarkb | so I think the problem is we aren't passing the secret's details into that role | 20:11 |
| pabelanger | clarkb: yah, 512401 fixes that problem. Just confirming known_hosts know | 20:11 |
| clarkb | fungi: any playbook | 20:11 |
| pabelanger | now* | 20:11 |
| jlk | mordred: from a social aspect, is it expected that we'll slowly move project definitions from project-config into the repos themselves? | 20:11 |
| mordred | pabelanger: ssh_key.ssh_known_hosts and ssh_key.fqdn | 20:12 |
| fungi | clarkb: oh, wow so as long as the playbook is in the same repo as the secret, the role using that secret can go anywhere? | 20:12 |
| * fungi looks in osj | 20:12 | |
| jlk | giving the project more explicit overview of changes to pipeline configurations that impact them? | 20:12 |
| mordred | fungi: yes. secrets are tied to playbooks | 20:12 |
| fungi | mordred: but roles can subvert that? | 20:12 |
| pabelanger | mordred: yah, so proposal_ssh_key is likely the format, with different private_key contents | 20:12 |
| pabelanger | dhellmann: ^ | 20:13 |
| dhellmann | pabelanger : the user value in the known_hosts entry for the proposal key is not the user I would expect the tag job to use. does that matter? | 20:13 |
| fungi | mordred: i guess the idea is you wouldn't intentionally pass a secret to a role you don't trust/control | 20:13 |
| pabelanger | dhellmann: I don't believe that is an issue | 20:13 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 20:13 |
| dhellmann | it has "gerrit-code-review@gerrit.openstack.org" and I don't know if that value is part of the fingerprint or just metadata | 20:13 |
| ianw | o/ ... nice to see no chatter in infra-incident :) | 20:13 |
| pabelanger | I think that is meta, fungi ^? | 20:13 |
| mordred | fungi: yah. and roles don't get passed in speculative versions to playbooks with secrets | 20:13 |
| fungi | pabelanger: dhellmann: it's irrelevant (a "comment" field in the key format) | 20:14 |
| pabelanger | ty | 20:14 |
| mordred | jlk: I think loci here is an interesting case ... | 20:14 |
| dhellmann | fungi : thanks | 20:14 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix the configuration for the release tagging key https://review.openstack.org/512401 | 20:15 |
| dhellmann | let's see if ^^ is closer | 20:15 |
| clarkb | jeblair: ok I think 01 is mostly stopped except it is still running zuul-executor proceses. I don't see any ansible though | 20:15 |
| mordred | jlk: in this case, rather than wanting cinder to run a job in check/gate - the loci project wants to 'subscribe' to commits from cinder so it can rebuild loci artifacts | 20:15 |
| clarkb | jeblair: is that a situation you want to look at? | 20:15 |
| mordred | jlk: I could see that being a thing to just do in project-config because it's not *actually* a thing that impacts the cinder project directly | 20:15 |
| pabelanger | +2 | 20:16 |
| jlk | yeah it's different. and post jobs don't really "block" changes from going in, right? | 20:16 |
| *** salv-orlando has joined #openstack-infra | 20:16 | |
| SamYaple | correct | 20:16 |
| mordred | jlk: or I could see us deciding socially that even doing that without cinder consent is rude ... it's a new fun question! :) | 20:16 |
| SamYaple | lots of testing there in recent days | 20:16 |
| jlk | mordred: the other question, can part of a project definition live in cinder, and the rest live in project-config? | 20:16 |
| pabelanger | jlk: mostly just consume resources IMO | 20:16 |
| mordred | jlk: yup! | 20:16 |
| SamYaple | in this case, i already have cinder PTL +1 o nthe patch | 20:16 |
| SamYaple | but i would prefer *not* to have to get everyones permission. i ll do what i need to though | 20:17 |
| *** yamamoto has joined #openstack-infra | 20:17 | |
| jlk | mordred: so they are merged, but ... hrm. | 20:17 |
| mordred | jlk: in fact, pretty much every project that has in-repo config has some project definition in project-config and some in their repo | 20:17 |
| jlk | mordred: how much merging is done? | 20:17 |
| jlk | can both list jobs for a "check" pipeline, and you get the sum total of both listings? | 20:17 |
| mordred | jlk: since we are requiring that projects keep the 'system-required' project-template in their project definition in project-config | 20:17 |
| mordred | jlk: yes | 20:17 |
| jlk | okay. | 20:17 |
| mordred | jlk: so putting things in project-config is like the "the openstack project requires python projects run pep8" .. and then putting things in in-repo is like "also, cinder wants to run cinder-foo-bar" | 20:18 |
| * mordred waves hands | 20:18 | |
| SamYaple | did merge-check evlove into system-required? | 20:18 |
| jlk | mordred: that makes sense | 20:19 |
| jlk | since it does merging in that style that totally makes sense. | 20:19 |
| mordred | SamYaple: yes | 20:19 |
| SamYaple | cool. cause that was new to me | 20:19 |
| jlk | how that merging is done, should be documented, if it isn't already | 20:19 |
| mordred | jlk: it's like a shared responsibility model | 20:19 |
| clarkb | but also we aren't running merge-check right now | 20:19 |
| mordred | jlk: totally | 20:19 |
| clarkb | I'm half expecting it to just go away because gerrit | 20:19 |
| mordred | clarkb: yah - but the project-template we have documented everyone should have in their project is now 'system-required' | 20:19 |
| *** ijw has joined #openstack-infra | 20:19 | |
| clarkb | ya | 20:20 |
| mordred | which would let us put in other things globally if we needed/wanted | 20:20 |
| clarkb | pabelanger: is the situation ze01 is in familiar to you? | 20:20 |
| pabelanger | so, I would thinking about 'system-required' if we require all projects to have it, couldn't be have a global jobs regex some how, and it there, over having 1500 repos do it? | 20:20 |
| mordred | like a 'cleanup stale logs' triggered by abandon events that would delete logs for abandoned changesets from the log server (not saying that's a good idea - but as an example of a thing we might choose to put there) | 20:21 |
| clarkb | basically just a zuul daemon hanging out but no apparent jobs running | 20:21 |
| pabelanger | clarkb: let me look | 20:21 |
| *** wolverineav has joined #openstack-infra | 20:21 | |
| *** yamamoto has quit IRC | 20:22 | |
| mordred | pabelanger: yah - I floated the idea a little while ago of adding the ability to define a list of default project-templates for a tenant - but that got quickly put into the 'let's talk about that post v3 rollout' pile - I honestly have not thought through all of the ramifications ... it might be a TERRIBLE idea | 20:22 |
| pabelanger | clarkb: I see a few ssh-agent processes running, perhaps that is blocking the executor | 20:22 |
| dmsimard | infra-root: my mid-term solution for ara storage and inodes: https://review.openstack.org/#/c/511992/ let me know what you think | 20:22 |
| pabelanger | clarkb: I think have jeblair would be good too | 20:22 |
| jeblair | back | 20:22 |
| mordred | dmsimard: looking | 20:22 |
| mordred | \o/ it's a jeblair | 20:22 |
| jeblair | clarkb: gimme a sec on ze01 | 20:22 |
| clarkb | pabelanger: the last thing logged is about killing an ssh agent (and that agent doesn't appear to exist) | 20:22 |
| clarkb | jeblair: ok | 20:22 |
| *** jtomasek has quit IRC | 20:23 | |
| *** dprince has quit IRC | 20:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs in Mixmatch https://review.openstack.org/511764 | 20:23 |
| *** armax has joined #openstack-infra | 20:24 | |
| pabelanger | mordred: yah, post zuulv3 for sure :) | 20:24 |
| mordred | dmsimard: on first glance that looks pretty cool | 20:24 |
| jeblair | clarkb: i'm going to sigusr2 | 20:24 |
| clarkb | jeblair: ok | 20:25 |
| dmsimard | mordred: I've tested it and it works very well. I also suspect this could work well against swift or s3 by abstracting them with something like s3ql but I need to test it. | 20:25 |
| fungi | dmsimard: when it comes to test-driving that, we can add it to the logs-dev vhost on static.o.o without impacting logs.o.o | 20:25 |
| *** baoli has joined #openstack-infra | 20:25 | |
| mordred | fungi: ++ | 20:25 |
| jeblair | clarkb: tracebacks are crazy wrong :) | 20:25 |
| *** smatzek has quit IRC | 20:26 | |
| *** smatzek has joined #openstack-infra | 20:26 | |
| *** ldnunes has quit IRC | 20:26 | |
| *** baoli has quit IRC | 20:26 | |
| dmsimard | fungi: oh, that's a good idea. | 20:27 |
| *** baoli has joined #openstack-infra | 20:27 | |
| clarkb | jeblair: looks like one of the threads may be waiting to stop and then the others are all lined up behind it? | 20:27 |
| openstackgerrit | Merged openstack-infra/project-config master: fix the configuration for the release tagging key https://review.openstack.org/512401 | 20:27 |
| dmsimard | fungi: the only doubt I have is about "hardcoding" /ara/ which might clash with other users whom might not be saving their databases (such as openstack-ansible) | 20:27 |
| fungi | dhellmann: ^ | 20:27 |
| fungi | need anything retriggered? | 20:28 |
| fungi | or reenqueued i guess | 20:28 |
| jeblair | clarkb: first three look like they're waiting for the merger lock so they can run a job... | 20:28 |
| pabelanger | dmsimard: after the first load, will the database be used again? | 20:28 |
| dhellmann | fungi : I have a revert in process now, so I'll run it that way. thanks! | 20:28 |
| jeblair | 0064 is the main thread | 20:29 |
| dmsimard | pabelanger: the database is used by the web app so long as it gets queries for it, the middleware doesn't delete the database or anything | 20:29 |
| *** Swami has joined #openstack-infra | 20:29 | |
| pabelanger | dmsimard: I guess I am curious, if I constently hit f5, would I be using CPU each time? | 20:29 |
| pabelanger | or does ARA cache it some how | 20:30 |
| dmsimard | pabelanger: there is no static generation involved | 20:30 |
| pabelanger | okay | 20:30 |
| dmsimard | pabelanger: it's just reading from the database and serving routes | 20:30 |
| pabelanger | so, something to keep in mind then | 20:30 |
| *** smatzek has quit IRC | 20:30 | |
| dmsimard | pabelanger: it doesn't do 'ara generate' which consumes the CPU and generates all those files and directories | 20:30 |
| dmsimard | pabelanger: it's a native wsgi web app that reads from the database | 20:30 |
| pabelanger | kk | 20:30 |
| *** epende has quit IRC | 20:30 | |
| pabelanger | http://logs.openstack.org/76/512376/1/gate/legacy-openstackci-beaker-centos-7/bc2dd8b/ is an interesting POST_FAILURE. I think that is an aborted job from ze02 but we didn't re-run it | 20:32 |
| pabelanger | going to add to etherpad to triage later | 20:32 |
| jeblair | pabelanger: that would be a zuul issue, go ahead and add it there | 20:33 |
| *** hasharAway has quit IRC | 20:33 | |
| jeblair | let's leave the triage section for "i have no idea what this issue is or who can help with it" | 20:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add flag to allow disabling sudo revocation https://review.openstack.org/512408 | 20:34 |
| jeblair | since we have folks who specialize in job issues and different folks who specialize in zuul issues | 20:34 |
| clarkb | 02 is near being down according to the process listing at least | 20:35 |
| pabelanger | sure | 20:35 |
| *** kgiusti has quit IRC | 20:35 | |
| dhellmann | there's a new error with tag-releases: http://logs.openstack.org/6b/6b2b2f10af366ea3c2a7f4c056ce16a7ccde16f1/release-post/tag-releases/7dd1a54/job-output.txt.gz#_2017-10-16_20_35_00_427589 | 20:36 |
| mordred | pabelanger, clarkb, jlvillal : did the patches for releasenotes building fix ironic-inspector? | 20:36 |
| jeblair | mordred: can we talk about the sudo thing? | 20:36 |
| dhellmann | that one is odd because it looks like the task failed but didn't cause the job to fail | 20:36 |
| mordred | jeblair: totally - although Im landing right now so it might need to be in a little bit ... | 20:37 |
| mordred | jeblair: feel free to -2 it to block it until we talk | 20:37 |
| jeblair | mordred: okay i was thinking we should add a new tox-functional job. cause i don't think we want unittest jobs to not revoke sudo | 20:37 |
| dhellmann | and I'm not sure where to look for the add-launchpad-credentials role | 20:38 |
| jlk | jeblair that sounds better to me as well, don't let consumers opt-out of system protections | 20:38 |
| fungi | mordred: the first patch fixed the tox-siblings issue in pre, but the second patch did not solve the rsync issue in post with missing tox logs for some envs | 20:38 |
| * fungi updates the pad with that status | 20:39 | |
| mordred | fungi: oh - yah - I remember this from scrollback now - bleh | 20:39 |
| jeblair | mordred: left comment on change so it's got a pin in it for later | 20:39 |
| mordred | jeblair: kk. that works for me too. cool | 20:39 |
| mordred | dhellmann: it's in zuul-jobs | 20:39 |
| dhellmann | mordred : thanks. I think this is just another key name error in the task definition | 20:40 |
| pabelanger | 133847 | 20:41 |
| pabelanger | oops | 20:41 |
| mordred | dhellmann: it's also possible that the template module wants src and dest not src and path | 20:41 |
| clarkb | jeblair: do you think we should continue to wait for 01? | 20:41 |
| mordred | dhellmann: yes - that's it | 20:41 |
| dhellmann | mordred : yeah | 20:41 |
| dhellmann | mordred : I wish I had time to help, but maybe we can recruit a small group to work on adding schema validation to all of these files? | 20:42 |
| dhellmann | seems like something that would be useful for ansible anyway, maybe there is already a linter | 20:42 |
| jeblair | clarkb: i think i've got a handle on the deadlock. why don't you kill and proceed on ze01, i'll write a patch. | 20:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use dest for template: not path https://review.openstack.org/512409 | 20:42 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: fix the template instruction for writing launchpadlib/creds https://review.openstack.org/512410 | 20:42 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config master: drop legacy aodh jobs https://review.openstack.org/512411 | 20:42 |
| mordred | dhellmann: ^^ there's that one - and yes, adding more linting is DEFINITELY athing that would be good | 20:42 |
| * mordred abandons his patch | 20:42 | |
| clarkb | jeblair: will do, thanks | 20:43 |
| *** tpsilva has quit IRC | 20:43 | |
| clarkb | jeblair: will I need to clean up those ssh-agents too? | 20:43 |
| openstackgerrit | gordon chung proposed openstack-infra/openstack-zuul-jobs master: remove legacy aodh jobs https://review.openstack.org/512412 | 20:43 |
| mordred | dhellmann: +2 | 20:43 |
| jeblair | clarkb: not critical, but good to be tidy | 20:43 |
| clarkb | rgr | 20:43 |
| jeblair | mordred, dhellmann, pabelanger: iirc, we had linting there and then discovered a substantial bug in the linter | 20:44 |
| dhellmann | mordred : otoh, this whack-a-mole thing gives that fast feedback loop of "debug, fix, rinse, repeat" :-) | 20:44 |
| mordred | dhellmann: :) | 20:44 |
| mordred | jeblair: yes - this is very true | 20:44 |
| jlvillal | mordred: Not sure if anyone else responded. But no it did not fix ironic-inspector. Have a POST_FAILURE now. | 20:44 |
| jeblair | hopefully we can turn it back on soon | 20:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 20:44 |
| SamYaple | what is infracloud-chocolate and infracloud-vanilla? and why are all my jobs timing out on them from being so slow :( | 20:44 |
| dhellmann | jeblair : ah. it seems like it would be pretty complicated to get it all 100%, so maybe it can be done incrementally | 20:44 |
| mordred | having a playbook that uses the role and running a syntax check on that playbook gets a fairly decently far distance down the path | 20:44 |
| jlvillal | mordred: That was about an hour ago though. | 20:44 |
| clarkb | fs backup is running now | 20:44 |
| dhellmann | as I said, i wish I could help instead of just complaining :-/ | 20:45 |
| pabelanger | Yah, I am hoping to start work on linters tomorrow | 20:45 |
| SamYaple | jobs that typically take 3-5m to run are timing out at 20m :/ | 20:45 |
| pabelanger | if pressure on zuulv3-issues is down | 20:45 |
| pabelanger | SamYaple: which job? | 20:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 20:45 |
| SamYaple | pabelanger: https://review.openstack.org/#/c/512400/ | 20:46 |
| dhellmann | mordred : should I be concerned that the task failed but the job didn't? | 20:46 |
| dhellmann | pabelanger : ++ | 20:46 |
| SamYaple | pabelanger: zuul is about to comment, but over half of the jobs hit the time cap of 20m and failed. all of those are on infra-canilla/chocolate | 20:46 |
| clarkb | jeblair: also the fix for slow shutdown seemed to run into job failures | 20:47 |
| clarkb | jeblair: thinking we may want to get both sets of fixes in with the next round of restarts | 20:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 20:48 |
| openstackgerrit | Brian Rosmaita proposed openstack-infra/project-config master: Remove workflow +1 on glance_store from swift-core https://review.openstack.org/511517 | 20:49 |
| pabelanger | SamYaple: some ran in infracloud, are you using docker reverse proxy cache | 20:50 |
| pabelanger | sounds like networking | 20:50 |
| *** jcoufal_ has joined #openstack-infra | 20:50 | |
| SamYaple | pabelanger: indeed. im using all the infra mirrors possible for everything | 20:50 |
| *** ihrachys_ has joined #openstack-infra | 20:51 | |
| pabelanger | SamYaple: can only suggest collecting docker log file, with debug enabled to see what is going on | 20:52 |
| SamYaple | fair enough | 20:52 |
| SamYaple | thanks! | 20:53 |
| *** ihrachys has quit IRC | 20:53 | |
| pabelanger | SamYaple: also possible the reverse proxy cache is cold, and caching docker things. | 20:53 |
| dmsimard | I feel noob for asking this, but does anyone know why I keep screwing up the etherpad formatting ? Especially around font types and copy/pastes. | 20:53 |
| *** jcoufal has quit IRC | 20:53 | |
| clarkb | jeblair: what is /var/lib/zuul/git? | 20:53 |
| dmsimard | clarkb: zuul merger root ? | 20:54 |
| clarkb | jeblair: it appears to be empty on some executors and not in others. | 20:54 |
| pabelanger | SamYaple: this doesn't look right either: http://logs.openstack.org/00/512400/2/check/loci-keystone/cda54e6/job-output.txt.gz#_2017-10-16_20_31_01_994285 | 20:54 |
| clarkb | oh ya I guess 01 would've been its own merger at one point | 20:54 |
| clarkb | jeblair: can I delete that to free up space and inodes on 01? | 20:54 |
| fungi | i was under the impression all the executors were also running merger daemons | 20:54 |
| SamYaple | pabelanger: ill start collecting more logs and get back to you | 20:54 |
| dmsimard | fungi: that would make sense, actually, if an executor could use a merger instance hosted locally | 20:55 |
| SamYaple | this might be a problem with async. i hadnt used it before these playbooks | 20:55 |
| clarkb | fungi: not that I haev seen unless its a thread in the main zuul-executor | 20:55 |
| *** jcoufal has joined #openstack-infra | 20:55 | |
| dmsimard | fungi: less traffic and opportunities for failure | 20:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 20:55 |
| *** edmondsw has quit IRC | 20:56 | |
| pabelanger | SamYaple: yah, your likey the first one to be using async task on zuulv3 too :) | 20:57 |
| pabelanger | 512376 has been in gate pipeline for over 2h25mins | 20:57 |
| *** jcoufal_ has quit IRC | 20:57 | |
| pabelanger | going to see why that is | 20:58 |
| clarkb | jeblair: you have a shell on 01 from the 14th that is in /var/lib/zuul | 20:58 |
| clarkb | jlvillal: so I can't unmount the fs, can you shut that shell down? | 20:58 |
| clarkb | er jeblair ^ | 20:58 |
| * jlvillal was confused... | 20:58 | |
| pabelanger | I think 512376 had a job running on ze02 | 20:59 |
| clarkb | pabelanger: ya I'm worreid 02 is in the same boat as 01 and will need more forceful stopping | 20:59 |
| pabelanger | 2017-10-16 18:48:10,741 DEBUG zuul.AnsibleJob: [build: 2a418f7682104223b5d41155a00ad0a4] Abort: no process is running | 20:59 |
| pabelanger | that is the last thing in log | 20:59 |
| pabelanger | for the build | 20:59 |
| pabelanger | clarkb: kk | 21:00 |
| pabelanger | I'll hold off debuging for now | 21:00 |
| *** smatzek has joined #openstack-infra | 21:00 | |
| dmsimard | pabelanger: hey now that we're back on v3, I'll do a recheck on https://review.openstack.org/#/c/504238/ .. could you fetch executor logs when it inevitably fails ? | 21:01 |
| jeblair | clarkb: that was pre-power outage, i've lost it | 21:01 |
| jeblair | clarkb: feel free to kill as needed | 21:01 |
| clarkb | jeblair: will do | 21:01 |
| pabelanger | dmsimard: will have to be tomorrow, just about to afk for now | 21:02 |
| pabelanger | I have an local meetup to attend this evening | 21:02 |
| dmsimard | pabelanger: ok, np, maybe another infra-root can help.. I've been trying to nail down that bug ever since the ptg :) | 21:02 |
| *** sree has joined #openstack-infra | 21:02 | |
| *** rockyg has joined #openstack-infra | 21:02 | |
| clarkb | jeblair: did you see question about deleting /var/lib/zuul/git ? | 21:02 |
| clarkb | jeblair: it has ~8GB on 01 but seems empty elsewhere | 21:02 |
| *** smatzek has quit IRC | 21:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Executor: stop jobs in parallel on shutdown https://review.openstack.org/512383 | 21:04 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| jeblair | clarkb: missed that. it can be deleted; that was probably a puppet install error (installed it as a merger rather than executor perhaps) | 21:05 |
| jeblair | clarkb: updated that change ^ with the deadlock fix (since they touch the same code) | 21:05 |
| clarkb | jeblair: cool, I'll be sure to delete the contents before starting the service | 21:05 |
| clarkb | I'll leave the dir there as I think puppet may be creating it as it is everywhere (just only full of stuff on 01) | 21:06 |
| *** mat128 has quit IRC | 21:06 | |
| jeblair | dmsimard: i can grab logs | 21:06 |
| dmsimard | jeblair: ok, the recheck is fired, I'll let you know when I got the finger url. | 21:07 |
| *** sree has quit IRC | 21:07 | |
| jeblair | dmsimard: kk | 21:07 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: fix the template instruction for writing launchpadlib/creds https://review.openstack.org/512410 | 21:09 |
| dmsimard | jeblair: finger://ze08.openstack.org/e974e5b38df9480ab1385fb45e404497 | 21:09 |
| openstackgerrit | boden proposed openstack-infra/project-config master: zuul changes for vmware-nsx https://review.openstack.org/512426 | 21:10 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 21:11 |
| dmsimard | I guess I'll add the include_role post failure (truncated json) issue to the zuulv3-issues pad. | 21:13 |
| clarkb | 01 is starting now with fs reinoded | 21:15 |
| clarkb | jeblair: why did you remove the if not self._running from mergerJobDispatch in ps2? | 21:17 |
| jeblair | clarkb: oh forgot to mention that. it was because merge jobs are fast, and we can afford to let a merge job we just got go ahead and finish while we're running down the real jobs. that will keep someone from getting a MERGER_ERROR. | 21:18 |
| *** yamamoto has joined #openstack-infra | 21:18 | |
| clarkb | jeblair: and to clarify the git/ dir thing earlier. Executors do run their own mergers in process (as a thread) and they use executor-git/ not git/ ? | 21:19 |
| jeblair | dmsimard: why is the intergration job defined in openstack-zuul-jobs instead of zuul-jobs? | 21:19 |
| jeblair | clarkb: correct. there may be raisins there. | 21:20 |
| jeblair | clarkb: (at the very least, it lets you run a merger process on an executor without collision (or special configuration)) | 21:21 |
| *** bobh has quit IRC | 21:21 | |
| prometheanfire | is openstack running checks for bad ssh/gpg keys for the infineon rsa badness? | 21:23 |
| clarkb | jeblair: 01 looks possibly unhappy | 21:23 |
| clarkb | jeblair: exit codes are all 250 | 21:23 |
| jeblair | dmsimard: i asked because in my comment on 509436 i really wanted to say 'squash all 3 changes together' | 21:23 |
| jeblair | clarkb: go ahead and stop it | 21:23 |
| *** florianf has quit IRC | 21:23 | |
| clarkb | ok stopping now | 21:23 |
| *** yamamoto has quit IRC | 21:24 | |
| clarkb | 2017-10-16 21:23:57,140 DEBUG zuul.AnsibleJob: [build: dab61331cba040298f98311cf695adb1] Ansible output: b"ERROR! Unexpected Exception: 'NoneType' object has no attribute 'isatty'" | 21:24 |
| clarkb | that appears to be the cause | 21:24 |
| odyssey4me | evening all - I'm working on a starting set of in-repo jobs for OSA and need a little guidance... for now I'm doing a like-for-like replacement with https://review.openstack.org/512414 but it's failing because it doesn't have sudo access... where exactly is sudo revoked? | 21:25 |
| *** florianf has joined #openstack-infra | 21:25 | |
| clarkb | odyssey4me: jeblair figured out it is revoked by the unittest job which is parent to tox which is parent to openstack-tox | 21:25 |
| jeblair | odyssey4me: mordred started working on this between takeoffs and landings in https://review.openstack.org/512408 | 21:26 |
| odyssey4me | clarkb jeblair ah ok - what's the options in this case? I'm happy to propose something or review. | 21:26 |
| odyssey4me | ah, that looks like a start - thanks | 21:27 |
| jeblair | odyssey4me: i think the outstanding question is will the result be that osa just writes a new job completely from scratch, or is there a sort of tox-functional-test paradigm that's repeated enough we should make a new general purpose parent job for that. | 21:27 |
| jeblair | odyssey4me: i suspect the latter | 21:28 |
| clarkb | to see full traceback use -vvv | 21:28 |
| odyssey4me | jeblair fair question - honestly our situation is that for now we need to use tox as it's the quickest way to get like-for-like tests to what we had with jenkins... but in time we will rewrite how the tests are executed to be rid of tox | 21:28 |
| jeblair | clarkb: okay i suspect something about the control persist has gone haywire | 21:28 |
| jeblair | odyssey4me: *nod* i think other projects will continue to use tox even after you stop | 21:29 |
| jeblair | clarkb: i guess we should have switched to verbose before starting :/ | 21:29 |
| odyssey4me | oh absolutely, although I'm not sure they'll need sudo though | 21:29 |
| jeblair | odyssey4me: i think a lot of them do (for *functional* tests) | 21:30 |
| odyssey4me | the flag patch is a good one, with the comments already suggested | 21:30 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config master: drop legacy panko jobs https://review.openstack.org/512429 | 21:30 |
| jeblair | odyssey4me: i have very strong feelings about the use of sudo in unit tests, so i just don't want us to accidentally make it easy for someone to disable that for a tox-py35 job. | 21:31 |
| odyssey4me | yeah, agreed | 21:31 |
| clarkb | jeblair: thoughts on what we should do next on 01? | 21:31 |
| jeblair | clarkb: i kind of want to start it again, turn on verbose, get an error, then stop | 21:31 |
| jeblair | clarkb: and hope all that happens fast enough all the jobs it grabs get abort/retried | 21:32 |
| clarkb | jeblair: ok I think it is still running since the last stop but clost to being done | 21:32 |
| *** ijw has quit IRC | 21:32 | |
| clarkb | jeblair: how do I turn on verbose? | 21:32 |
| dmsimard | jeblair: pretty sure I added them in zuul-jobs first and someone asked to move them in ozj | 21:32 |
| jeblair | clarkb: 'zuul-executor verbose' | 21:32 |
| jeblair | dmsimard: was that someone me? :) | 21:33 |
| clarkb | oh I'd run it in the foreground then? | 21:33 |
| jeblair | clarkb: no, that sends a command to the socket | 21:33 |
| clarkb | ah | 21:33 |
| dmsimard | jeblair: maybe | 21:33 |
| smcginnis | If someone has a moment for pointers, looks like we still have failures in the release-post jobs: http://logs.openstack.org/81/81f73d804ccf51f985b902d84bbf88ef13f2c32b/release-post/tag-releases/ce6f50f/job-output.txt.gz#_2017-10-16_21_29_35_217979 | 21:33 |
| clarkb | jeblair: should I go ahead and kill the existing processes, start it, turn on verbose then stop it then? | 21:33 |
| clarkb | also I think 02 is ready to be manually stopped maybe | 21:34 |
| jeblair | clarkb: yeah, though wait for a 250 before stopping | 21:34 |
| openstackgerrit | gordon chung proposed openstack-infra/openstack-zuul-jobs master: remove legacy panko jobs https://review.openstack.org/512430 | 21:34 |
| jeblair | dmsimard: is that ze08 job still running? | 21:34 |
| clarkb | jeblair: also comment on https://review.openstack.org/#/c/512383/2 | 21:34 |
| * clarkb proceeds on 01 | 21:34 | |
| dmsimard | jeblair: nop | 21:35 |
| jeblair | dmsimard: ok i'll grab error | 21:35 |
| *** jcoufal_ has joined #openstack-infra | 21:35 | |
| dmsimard | jeblair: ok, making something to eat before meeting, I'll catch up. | 21:35 |
| jeblair | dmsimard: http://paste.openstack.org/show/623764/ do we need to set verbose for that too? | 21:36 |
| jeblair | clarkb: i see errors | 21:36 |
| clarkb | ya about to stop it | 21:36 |
| clarkb | ok init stop run. Will watch it to see if I have to manually stop it soon | 21:37 |
| dmsimard | jeblair: hmm, I guess so ? Are we able to do that without reloading executors ? | 21:37 |
| jeblair | dmsimard: yes, though we'll have to do it on all the executors since we don't know where it will end up | 21:37 |
| *** jcoufal has quit IRC | 21:38 | |
| jeblair | well... actually... it should take effect for any playbooks started after we set it... | 21:38 |
| jeblair | dmsimard: so if i'm really fast, i may just be able to do it on one | 21:38 |
| jeblair | dmsimard: anyway, i'll have to shelve that for a minute while i switch to the thing with clarkb | 21:38 |
| dmsimard | jeblair: sure, I can't look at it immediately anyway | 21:39 |
| jeblair | which, btw, is: http://paste.openstack.org/show/623765/ | 21:39 |
| clarkb | oh I just made a paste too :) | 21:39 |
| clarkb | yours has slightly more context so we'll go with it | 21:39 |
| *** claudiub has quit IRC | 21:40 | |
| smcginnis | Any idea if this is a known issue? http://logs.openstack.org/81/81f73d804ccf51f985b902d84bbf88ef13f2c32b/release-post/tag-releases/ce6f50f/job-output.txt.gz#_2017-10-16_21_29_35_217979 | 21:40 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Harden exception handling around instance deletes https://review.openstack.org/512413 | 21:41 |
| clarkb | smcginnis: I tknow pabelanger and dhellmann were looking at related failures but that one looks new to me (but I've been working other things) | 21:43 |
| *** ijw has joined #openstack-infra | 21:43 | |
| smcginnis | clarkb: Yeah, dhellmann fixed a few things, now this appears to be the next roadblock. | 21:44 |
| clarkb | jeblair: so sys.stdin is None which makes sense for a daemon I think? | 21:44 |
| SamYaple | pabelanger: looks like it was async and me :/ . I was not retrying enough times when polling the async job, so the job wasn't timing out, the "task" was failing because it didn't complete fast enough. | 21:44 |
| SamYaple | a slower host would aggrevate that issue | 21:45 |
| smcginnis | The playbooks/release/tag.yaml file it's point to looks fine to me, so not really sure where to take it from here. | 21:45 |
| *** chlong has quit IRC | 21:45 | |
| jeblair | clarkb: yeah; i switched to using the nullwrap driver for the popen implementation rather than bubblewrap; there must be something about that | 21:45 |
| odyssey4me | if I want to make use of a project template, for example https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n253 - how do I do that in my pipeline? | 21:46 |
| dmsimard | odyssey4me: you want to create a new template or use an existing one ? | 21:47 |
| odyssey4me | dmsimard use an existing one for now | 21:47 |
| dmsimard | odyssey4me: nothing special, it is shared by default | 21:48 |
| odyssey4me | yeah, but how do I configure my project to use it :) | 21:48 |
| dmsimard | Oh, one sec | 21:48 |
| *** MasterOfBugs has quit IRC | 21:49 | |
| *** pramodrj07 has quit IRC | 21:49 | |
| clarkb | jeblair: this is the wrapped=False bits of the persistent ssh change? | 21:49 |
| jeblair | clarkb: ya | 21:49 |
| jeblair | clarkb: i'm having trouble replicating this locally (though i haven't gone full-on daemon yet) | 21:50 |
| clarkb | jeblair: ya my guess is it works without being a daemon because when you aren't a daemon sys.stdin is a thing | 21:50 |
| clarkb | jeblair: but part of daemonization is to close that fd | 21:50 |
| *** aeng has joined #openstack-infra | 21:50 | |
| dmsimard | odyssey4me: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-project.templates | 21:51 |
| clarkb | ok ze01 never fully stopped I am going to kill the processes now | 21:51 |
| dmsimard | It's a list attribute, see beginning of https://raw.githubusercontent.com/openstack-infra/project-config/master/zuul.d/projects.yaml for example | 21:51 |
| odyssey4me | ah, thanks dmsimard - I misread that earlier | 21:51 |
| clarkb | but then I need to step away for a bit (will return to hopeflly happier zuul and continue to restart things on new filesystem | 21:52 |
| clarkb | alright 01 is in that steady state now | 21:54 |
| clarkb | I'll be back in about 30-45 minutes | 21:54 |
| *** jcook has quit IRC | 22:00 | |
| fungi | meeting time? | 22:01 |
| *** brett has joined #openstack-infra | 22:04 | |
| brett | Hi, hoping to get a couple of simple gerrit reviews for Jenkins Job Builder changes: | 22:05 |
| brett | https://review.openstack.org/#/c/505532/ | 22:05 |
| brett | https://review.openstack.org/#/c/505560/ | 22:05 |
| *** Shrews has quit IRC | 22:06 | |
| jeblair | brett: you might try in the #openstack-jjb channel | 22:06 |
| brett | @jeblair thanks | 22:07 |
| *** brett has left #openstack-infra | 22:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Set ansible stdin to /dev/null https://review.openstack.org/512434 | 22:08 |
| jeblair | clarkb, SpamapS, fungi: ^ i *think* that will fix it. this is very difficult to test though. so i will manually apply that on ze01 and restart. | 22:08 |
| *** jcoufal_ has quit IRC | 22:11 | |
| jeblair | seems better so far | 22:11 |
| fungi | is it legitimate to use jinja parameter substitution in a shell task? | 22:11 |
| jlk | it should be fine, I think | 22:12 |
| jlk | unless you're having it specifically not run the bits through the templating engine. | 22:12 |
| fungi | the tag-releases job seems to think this is a templating error in project-config:playbooks/release/tag.yaml: | 22:13 |
| fungi | RELEASES_DIR=/home/fungi/src/{{ zuul.project.canonical_name }} | 22:13 |
| jlk | hrm. | 22:14 |
| fungi | ERROR! failed at splitting arguments, either an unbalanced jinja2 block or quotes: RELEASES_DIR=$HOME/src/{{ zuul.project.canonical_name }} | 22:14 |
| *** salv-orlando has quit IRC | 22:14 | |
| fungi | no quote marks anywhere in there, and the jinja doesn't _seem_ imbalanced | 22:15 |
| jlk | oh, I wonder if it's because of the | thing that it's not properly quoting the bits to template. | 22:15 |
| jlk | I wonder. | 22:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Executor: stop jobs in parallel on shutdown https://review.openstack.org/512383 | 22:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Set ansible stdin to /dev/null https://review.openstack.org/512434 | 22:17 |
| jeblair | fungi: it's the "'" in it's | 22:18 |
| jeblair | ansible hates contractions | 22:18 |
| *** panda|rover is now known as panda|rover|off | 22:19 | |
| fungi | d/oh! | 22:19 |
| jeblair | *also* that shouldn't be a contraction! | 22:19 |
| jeblair | *This* is why grammar is important! :) | 22:19 |
| fungi | i keep forgetting quotes in comments throw off jinja there | 22:19 |
| fungi | smcginnis: ^ *sigh* | 22:19 |
| jlk | yeah that is kinda dumb. | 22:19 |
| jlk | probably something COULD be done | 22:19 |
| fungi | patch on the way | 22:20 |
| smcginnis | fungi: Thanks | 22:20 |
| *** dingyichen has quit IRC | 22:20 | |
| *** yamamoto has joined #openstack-infra | 22:21 | |
| smcginnis | fungi, jeblair: So it's the "it's" in the comment on line 6 throwing it off? | 22:21 |
| jeblair | smcginnis: yep | 22:21 |
| smcginnis | jeblair: And you're right, should be its. :D | 22:21 |
| smcginnis | fungi: I can take care of that if you have better things to do. | 22:22 |
| *** rloo has left #openstack-infra | 22:22 | |
| smcginnis | fungi: Just let me know. | 22:22 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Correct grammar in tag-releases comment line https://review.openstack.org/512437 | 22:22 |
| fungi | smcginnis: already done | 22:22 |
| fungi | took way longer to write the commit message | 22:23 |
| smcginnis | fungi: If only they could all be that easy. ;) | 22:23 |
| clarkb | I hate that ansible bug | 22:23 |
| fungi | smcginnis: i enjoyed that i could masquerade that as a really pedantic grammar patch ;) | 22:23 |
| clarkb | jeblair: I'm reviewing the fix change now, also I see you saw my can of worms commetn so reviewing that too | 22:23 |
| johnsom | zuulv3 noob question: I have a horizon plugin that requires horizon installed for the openstack-tox-py27/35 gates. Should I just create a new job with openstack-tox-py27 as the parent that includes the horizon package? This used tox_install.sh before. | 22:23 |
| jeblair | fungi: i enjoyed +2ing it. | 22:24 |
| jeblair | johnsom: i think there's a project-template for that now | 22:24 |
| *** edmondsw has joined #openstack-infra | 22:25 | |
| *** aviau has quit IRC | 22:25 | |
| ianw | mordred: (sorry, catching up) ... are we happy with increased permissions for the "tox as other user" issue. should i abandon https://review.openstack.org/#/c/512194/ (zuul-cloner-shim: don't link if users different) | 22:25 |
| johnsom | jeblair I looked on the docs pages and nothing jumped out at me, any hints to the name? | 22:25 |
| *** aviau has joined #openstack-infra | 22:25 | |
| jeblair | johnsom: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n335 | 22:25 |
| johnsom | jeblair Perfect! Thanks | 22:25 |
| *** yamamoto has quit IRC | 22:26 | |
| *** masber has joined #openstack-infra | 22:26 | |
| jeblair | clarkb: afaict the fix is working on ze01. | 22:26 |
| clarkb | jeblair: re stdin looks like regardless of wrapper method we'll just set stdin to dev/null which should be fine since all of ansible et al's input are via config and command line args | 22:26 |
| jeblair | clarkb: yep, that's my thinkin | 22:27 |
| johnsom | Hmm, this may not be getting updated: https://docs.openstack.org/infra/openstack-zuul-jobs/jobs.html as I don't see openstack-python-jobs-horizon there. | 22:27 |
| SamYaple | if someone on my team were to ask infra to decrypt the secrets for them, would infra be able to do that (planning hit-by-a-bus scenarios) | 22:27 |
| clarkb | SamYaple: as long as we don't lose the secret key yes. I'm not sure we have those in the backup rotation yet though | 22:28 |
| jeblair | johnsom: yes, i think we forgot to add support for project-templates to zuul-sphinx (which is what renders that). a terrible mistake in retrospect -- those are way more likely to be referenced than jobs. we'll correct it soon, but it'll probably take a few days to plumb everything through. | 22:28 |
| clarkb | jeblair: I don't think I can do anymore restarts until those two changes merge then. Also should I plan to restart 01 again after those merge so that it is running of of HEAD? | 22:29 |
| johnsom | NP | 22:29 |
| SamYaple | clarkb: cool. but if you lose the private key then someone will have to go in an reset the secrets anyway | 22:29 |
| johnsom | I will stick that somewhere in the FAQ etherpad | 22:29 |
| jeblair | clarkb: we should really add that to backups! :) | 22:29 |
| *** andreww has quit IRC | 22:29 | |
| clarkb | jeblair: we probably should | 22:29 |
| *** edmondsw has quit IRC | 22:29 | |
| jeblair | or, i guess, tell everyone if we aren't so they know what to expect. :) | 22:30 |
| * jeblair adds item to infra meeting agenda | 22:30 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Switch cistatus page to zuul v3 https://review.openstack.org/508546 | 22:32 |
| *** ijw has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Correct grammar in tag-releases comment line https://review.openstack.org/512437 | 22:33 |
| *** ijw has joined #openstack-infra | 22:34 | |
| jeblair | SpamapS: are you available to merge https://review.openstack.org/512383 ? | 22:35 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Fix octavia-dashboard py27/35 jobs https://review.openstack.org/512439 | 22:35 |
| *** esberglu has quit IRC | 22:36 | |
| *** florianf has quit IRC | 22:37 | |
| *** felipemonteiro_ has quit IRC | 22:37 | |
| *** gouthamr has quit IRC | 22:37 | |
| jeblair | clarkb: i'm inclined to just merge that with the 2 of us, and ask SpamapS to retro-review it later | 22:40 |
| clarkb | jeblair: wfm | 22:40 |
| clarkb | jeblair: if you can work on that (and the kick.sh I am going to start prepping 02 | 22:41 |
| dmsimard | Interesting how k8s didn't develop their own "stackalytics": https://devstats.k8s.io/dashboard/db/companies-stats?orgId=1&var-period=y&var-metric=activity&var-repogroup_name=All&var-repogroup=all&var-companies=All | 22:41 |
| clarkb | jeblair: the fs backup takes long enough that you should be done merging things by the time I am ready to start the service again | 22:41 |
| clarkb | dmsimard: they kind of did, its a bunch of bigtable goolgle stuff on top of the github integration aiui | 22:42 |
| clarkb | ok 02 is all stopped now. Moving ahead with the fs work | 22:44 |
| SpamapS | jeblair: I'm OTP for a bit, available in 14 for sure. | 22:46 |
| *** Shrews has joined #openstack-infra | 22:47 | |
| *** wolverineav has quit IRC | 22:47 | |
| *** rbrndt has quit IRC | 22:50 | |
| *** rockyg has quit IRC | 22:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Import legacy playbooks https://review.openstack.org/512164 | 22:50 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Fix Octavia jobs to be voting again https://review.openstack.org/512444 | 22:51 |
| clarkb | restoring fs backup on 02 now (then will be ready to start the service whenever code is installed) | 22:52 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Executor: stop jobs in parallel on shutdown https://review.openstack.org/512383 | 22:54 |
| *** sdague has quit IRC | 22:56 | |
| *** lbragstad has quit IRC | 22:57 | |
| *** gouthamr has joined #openstack-infra | 22:57 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Set ansible stdin to /dev/null https://review.openstack.org/512434 | 23:01 |
| *** gordc has quit IRC | 23:01 | |
| jeblair | clarkb: i'll start kicking things | 23:01 |
| clarkb | thanks | 23:01 |
| clarkb | I'm ready to start zuul on 02 as soon as that is done | 23:01 |
| *** aeng has quit IRC | 23:02 | |
| *** smarcet has joined #openstack-infra | 23:03 | |
| *** smarcet has quit IRC | 23:04 | |
| *** dhinesh has quit IRC | 23:05 | |
| *** esberglu has joined #openstack-infra | 23:11 | |
| clarkb | jeblair: the git repo on 02 looks updates but pip doesn't seem to know that zuul is installed | 23:13 |
| clarkb | jeblair: is it still running? | 23:13 |
| jeblair | clarkb: just finished | 23:14 |
| jeblair | clarkb: make sure you use pip3 | 23:14 |
| clarkb | oh right | 23:14 |
| clarkb | thanks | 23:14 |
| clarkb | ok I am starting zuul-executor on 02 now | 23:14 |
| jeblair | ++ | 23:14 |
| *** esberglu has quit IRC | 23:15 | |
| clarkb | jeblair: did you want to restart 01 too so that it is running the same code as the others? | 23:15 |
| clarkb | I guess I should restart 03 too | 23:15 |
| clarkb | and 09 | 23:15 |
| *** salv-orlando has joined #openstack-infra | 23:15 | |
| jeblair | clarkb: yeah, we should do those, but maybe after the others? | 23:16 |
| clarkb | ok | 23:16 |
| clarkb | I'll do 4-8 then 10 | 23:16 |
| clarkb | 2017-10-16 23:16:40,264 DEBUG zuul.AnsibleJob: [build: 21bb90abd9394f3bbfe1f7b3561952b5] Ansible exit code: 0 | 23:16 |
| clarkb | 02 lgtm | 23:16 |
| * clarkb stops on 04 | 23:16 | |
| clarkb | and 05 | 23:17 |
| *** ijw has quit IRC | 23:17 | |
| jeblair | clarkb: 02 lgtm too | 23:17 |
| *** tosky has quit IRC | 23:17 | |
| *** aeng has joined #openstack-infra | 23:19 | |
| *** mat128 has joined #openstack-infra | 23:19 | |
| clarkb | 16384 bytes per inode is apparently an ext4 default (or a value close to that). What do we thinkabout updating the mkfs in the system-config script for using the ephemeral drive to just hardcode -i 8192? | 23:20 |
| clarkb | seems like we tend to need more inodes than not and the actual cost isn't that large to have extra inodes | 23:20 |
| clarkb | ok 04 is stopped migrating its fs now | 23:20 |
| dmsimard | When we generate ara reports (on the executors), the reports are stored in the bubblewrap env right ? Those are discarded/deleted after every build ? | 23:21 |
| *** salv-orlando has quit IRC | 23:21 | |
| clarkb | the bubblewrap envs are deleted unless we set keep on the job | 23:22 |
| *** yamamoto has joined #openstack-infra | 23:22 | |
| clarkb | 05 has also stopped quickly so proceding to migrate it too | 23:24 |
| jeblair | keep is an executor run-time flag, but yes | 23:25 |
| jeblair | (just wanted to clarify you can't change that in a job definition) | 23:25 |
| clarkb | jeblair: is it global to all jobs? | 23:26 |
| *** gongysh has joined #openstack-infra | 23:27 | |
| jeblair | clarkb: yes | 23:27 |
| *** gongysh has quit IRC | 23:27 | |
| jeblair | it's kind of a blunt debugging tool | 23:27 |
| clarkb | it is a big hammer | 23:28 |
| *** yamamoto has quit IRC | 23:28 | |
| *** gmann_afk is now known as gmann | 23:30 | |
| *** ijw has joined #openstack-infra | 23:30 | |
| clarkb | 04 is starting back up now | 23:31 |
| clarkb | 2017-10-16 23:31:56,554 DEBUG zuul.AnsibleJob: [build: 1c66f0d4b3804a91801b87c4a3c8e232] Ansible complete, result RESULT_NORMAL code 0 | 23:32 |
| clarkb | stopping 06 now | 23:32 |
| *** ijw has quit IRC | 23:33 | |
| clarkb | 05 starting now | 23:35 |
| clarkb | 2017-10-16 23:35:49,245 DEBUG zuul.AnsibleJob: [build: 1ca9202922b245e2ab7e20cbedfe8209] Ansible complete, result RESULT_NORMAL code 0 | 23:36 |
| *** wolverineav has joined #openstack-infra | 23:36 | |
| *** jascott1 has joined #openstack-infra | 23:37 | |
| *** aeng has quit IRC | 23:40 | |
| *** aeng has joined #openstack-infra | 23:41 | |
| SamYaple | i just want to say, great job on zuulv3. even the seemingly obscure scenarios seem already well thoughtout | 23:41 |
| SamYaple | im more and more impressed the more i use it | 23:41 |
| clarkb | 07 has been stopped | 23:41 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Import legacy playbooks https://review.openstack.org/512164 | 23:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Handle tox environments via tox config https://review.openstack.org/512448 | 23:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Add support for project-templates https://review.openstack.org/512449 | 23:46 |
| jeblair | SamYaple: thanks! | 23:46 |
| *** bobh has joined #openstack-infra | 23:46 | |
| clarkb | 06 starting now | 23:47 |
| clarkb | 2017-10-16 23:47:13,795 DEBUG zuul.AnsibleJob: [build: 98eecfb7a381464088656bb2ed778356] Ansible complete, result RESULT_NORMAL code 0 | 23:47 |
| clarkb | on to 08 | 23:47 |
| SamYaple | jeblair: my dreams are almost fully realized :) new docker image pushed per commit to openstack project | 23:47 |
| mordred | pabelanger, clarkb, jlvillal : https://review.openstack.org/512448 should handle the tox directory thing much more comprehensively | 23:47 |
| *** jascott1 has quit IRC | 23:47 | |
| *** jascott1 has joined #openstack-infra | 23:48 | |
| * mordred shuts laptop lid to eat airplane dinner | 23:48 | |
| jeblair | SamYaple: yay! | 23:48 |
| SamYaple | fun fact: food tastes different on airplanes because our tastebuds get all smushed from decompression leading to bland tastes | 23:49 |
| clarkb | 07 is done 2017-10-16 23:51:32,890 DEBUG zuul.AnsibleJob: [build: 4733a27cc2274b00aeab22e5202371e4] Ansible complete, result RESULT_NORMAL code 0 | 23:51 |
| clarkb | stopping 10 now | 23:52 |
| *** ijw has joined #openstack-infra | 23:52 | |
| *** gildub has joined #openstack-infra | 23:55 | |
| *** ijw has quit IRC | 23:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] start at templating out the zuulv3 jobs https://review.openstack.org/512450 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!