clarkb | (something to wathc out for I guess but should be non issue for now) | 00:00 |
---|---|---|
*** pabelanger has quit IRC | 00:01 | |
*** mriedem1 has joined #openstack-infra | 00:05 | |
*** mriedem has quit IRC | 00:05 | |
*** mriedem1 is now known as mriedem | 00:08 | |
*** thorst has joined #openstack-infra | 00:08 | |
*** thorst has quit IRC | 00:09 | |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_in_repo_branch https://review.openstack.org/507708 | 00:14 |
jeblair | mordred, clarkb, jlk: that change was running in a loop locally without error for a long enough time that i think that has fixed it. | 00:15 |
jlk | k | 00:15 |
clarkb | jeblair: the z-c change has the +2s, I didn't approve it as not entirely sure how to check it has fixed things, I guess just recheck project-config changes? | 00:21 |
jeblair | clarkb: yeah. let's go ahead and land it. won't make things worse. :) | 00:21 |
jeblair | Shrews: can retro-review it :) | 00:21 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Always create dest dir in zuul-cloner shim https://review.openstack.org/507704 | 00:30 |
clarkb | that update should apply immediately right? | 00:31 |
* clarkb looks for a change to recheck | 00:31 | |
clarkb | 507222 has been rechecked | 00:32 |
*** hemna_ has joined #openstack-infra | 00:34 | |
*** nicolasbock has quit IRC | 00:35 | |
*** yamahata has quit IRC | 00:36 | |
clarkb | looks like post failures | 00:36 |
clarkb | which is a different error I think | 00:36 |
clarkb | progress? | 00:36 |
* fungi had forgotten that insta-applying configuration when changes merge to the repo was a v3 feature | 00:37 | |
clarkb | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-centos-7/297d448/job-output.txt.gz#_2017-09-27_00_35_03_642505 | 00:37 |
*** stakeda has joined #openstack-infra | 00:37 | |
clarkb | and fedora is the permission denied thing | 00:39 |
clarkb | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-debian-jessie/a25ef8b/job-output.txt.gz#_2017-09-27_00_37_37_648816 jessie is just broken maybe? | 00:39 |
*** Apoorva_ has joined #openstack-infra | 00:40 | |
clarkb | so that is an improvement over the old situation but still not fixed | 00:40 |
clarkb | fungi: security.debian.org does not resolve earlier in that lo | 00:40 |
clarkb | it does resolve locally though, so possible network problems? | 00:41 |
fungi | strange | 00:41 |
fungi | it certainly _should_ resolve | 00:41 |
fungi | or else, like, all debian users stop getting quick security updates | 00:41 |
clarkb | ya, I mean it totally resolves from here | 00:41 |
*** nicolasbock has joined #openstack-infra | 00:42 | |
clarkb | fungi: and is that libxml2-dev thing a broken package/repo? it deps on the old version but apt is going to install the new version | 00:42 |
ianw | clarkb: isn't the fedora fail the same network issue there? http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/fc748ef/job-output.txt.gz#_2017-09-27_00_34_48_852890 | 00:42 |
*** Apoorva has quit IRC | 00:43 | |
clarkb | ianw: oh it is and centos7 suffers the permission denied issue too | 00:43 |
clarkb | so I think those are the same problem | 00:43 |
ianw | oh i see, just above that | 00:43 |
clarkb | so if we fix /home/zuul permissions that may make both of thoes pass | 00:44 |
clarkb | then we are just left with debian packaging catching up with reality | 00:45 |
clarkb | jeblair: is there already a plan for fixing the homedir thing? | 00:46 |
ianw | (sorry, catching up ...) | 00:48 |
ianw | it's not really homedir permissions, but inability to log in? | 00:48 |
ianw | Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). | 00:48 |
ianw | or we've borked the authorized_keys or something? | 00:49 |
*** xarses has joined #openstack-infra | 00:49 | |
clarkb | ianw: there was discussion of homedir perms at some point earlier today and I may have misparsed, could be auth failed because networking failed | 00:49 |
clarkb | is it permission denied because of fs perms or ssh auth? | 00:50 |
ianw | to me, http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/fc748ef/job-output.txt.gz#_2017-09-27_00_34_46_189587 seems to say that rsync can not get in | 00:50 |
*** LindaWang has joined #openstack-infra | 00:51 | |
jeblair | clarkb: the homedir perms from earlier were that under fedora, /home/zuul was 700. that's fine for most things, but caused us to upload the logs to logs.o.o with 700 (because of the rsync command). | 00:51 |
*** mugsie has quit IRC | 00:51 | |
*** dingyichen has joined #openstack-infra | 00:51 | |
clarkb | gotcha so in this case likely actually what ianw says, auth | 00:52 |
jeblair | clarkb, ianw: yeah, did we obliterate ~/.ssh ? | 00:52 |
ianw | that seems unlikely? did something bindep install fiddle things somehow? | 00:53 |
jeblair | maybe the newly fixed zuul-cloner shim did it? | 00:54 |
clarkb | jeblair: possibly, are we copying into the root of the homedir? | 00:54 |
*** Apoorva_ has quit IRC | 00:54 | |
jeblair | clarkb: yes | 00:55 |
clarkb | but that should add and not delete old things right? unless we have a .ssh/ in project-config | 00:55 |
clarkb | also it totally worked on ubuntu | 00:55 |
jeblair | clarkb: agreed | 00:55 |
jeblair | (i have to run now) | 00:56 |
clarkb | probably a good next step is putting an auto hold on that job, then looking at the node afterwards since root ssh shouldn't be touched | 00:57 |
*** xarses has quit IRC | 00:57 | |
* clarkb goes to figure that out | 00:57 | |
ianw | clarkb: you read my mind :) | 00:57 |
clarkb | arg my ssh key got unloaded /me first gets an ssh key loaded | 00:58 |
ianw | the suspect is https://review.openstack.org/#/c/507704/1/roles/fetch-zuul-cloner/templates/zuul-cloner-shim.py.j2 ? | 00:58 |
clarkb | yes | 00:59 |
*** esberglu has joined #openstack-infra | 00:59 | |
clarkb | ok autohold in place /me rechecks again | 01:00 |
clarkb | did the centos 7 job | 01:01 |
ianw | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-fedora-26/fc748ef/job-output.txt.gz#_2017-09-27_00_33_42_734036 | 01:02 |
ianw | mkdir -p /home/zuul ... after zuul-cloner run, where's that come from? | 01:03 |
*** yamamoto has quit IRC | 01:03 | |
*** yamamoto_ has joined #openstack-infra | 01:03 | |
*** esberglu has quit IRC | 01:03 | |
*** Sukhdev has quit IRC | 01:03 | |
clarkb | looks like the script that invokes the z-c shim | 01:03 |
clarkb | based on the level of +'s in that log | 01:03 |
clarkb | the jessie job looks happy now so going to blame that on repo syncage in mirrors | 01:04 |
*** thorst has joined #openstack-infra | 01:06 | |
*** thorst has quit IRC | 01:06 | |
ianw | https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/bindep-fallback-fedora-26/run.yaml#n28 | 01:07 |
ianw | still not seeing where a mkdir comes into that | 01:07 |
clarkb | that job doesn't seem to use a debian mirror | 01:07 |
clarkb | so that was upstream | 01:07 |
dims | clarkb : fungi : the mogan stuff can be done anytime they already stopped all work until the repo is switched - http://lists.openstack.org/pipermail/openstack-dev/2017-September/thread.html#122509 | 01:07 |
clarkb | ianw: line 19 of that file maybe? | 01:07 |
clarkb | ianw: ya thats cleanup for the z-c invocation so it runs after z-c | 01:08 |
ianw | clarkb: hmm, yep on the cleanup path, ok | 01:08 |
clarkb | ianw: ssh root@104.130.198.116 | 01:09 |
clarkb | /home/zuul/.ssh/authorized_keys lgtm on first glance | 01:10 |
clarkb | maybe selinux? | 01:11 |
*** cuongnv has joined #openstack-infra | 01:11 | |
clarkb | type=USER_LOGIN msg=audit(1506474438.992:778): pid=3179 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="zuul" exe="/usr/sbin/sshd" hostname=? addr=104.239.143.247 terminal=ssh res=failed' | 01:12 |
clarkb | I'm going to add my public key to that authorized_keys file and see if I can reproduce | 01:13 |
clarkb | ya it asks for my password | 01:15 |
fungi | so it's trashed .ssh (possibly entire homedir)? | 01:15 |
clarkb | Sep 27 01:14:56 centos-7-rax-ord-0000035830 sshd[3257]: Authentication refused: bad ownership or modes for directory /home/zuul | 01:15 |
fungi | aha | 01:15 |
ianw | ls -ld /home/zuul | 01:15 |
ianw | drwxrwxr-x. 20 zuul zuul 4096 Sep 27 01:05 /home/zuul | 01:15 |
ianw | ya | 01:15 |
fungi | i bet it's world-readable .ssh | 01:15 |
clarkb | should it be 755? | 01:16 |
fungi | we inadvertently chmod it? | 01:16 |
clarkb | drwx------. 2 zuul zuul 4096 Sep 27 01:14 .ssh | 01:16 |
fungi | aha, group-writeable is probably the issue, yeah | 01:17 |
clarkb | fungi: the z-c shim copy may carry over its perms? | 01:17 |
clarkb | i'm going to try chmod real quick | 01:17 |
clarkb | yup that fixed ssh for me | 01:17 |
ianw | "cp -al /home/zuul/src/git.openstack.org/openstack-infra/project-config/. /home/zuul" | 01:17 |
ianw | does the -a fiddle it? | 01:17 |
clarkb | ianw: ya man page says that preserves mode | 01:18 |
clarkb | I'm guessing centos and fedora have a different umask than debuntu which is why this breaks? | 01:19 |
clarkb | we could possibly fix this on an image level with a umask update | 01:19 |
clarkb | or have a pre task ste it | 01:19 |
clarkb | I too now need to look at dinner tasks | 01:21 |
clarkb | I'll let others think about how best to address this problem | 01:21 |
clarkb | (another option is move everything a dir level deeper | 01:21 |
ianw | yep, "cp -al /some/dir /tmp/foo" changes /tmp/foo | 01:21 |
fungi | that's mildly surprising behavior | 01:21 |
ianw | or "cd /home/zuul ; cp -al $path ." ? | 01:21 |
fungi | but yeah, i guess it's down to umask | 01:21 |
clarkb | ianw: oh that might be simplest | 01:22 |
*** jdandrea_ has quit IRC | 01:28 | |
tonyb | Am I okay to +W stuff in openstack/releases? | 01:30 |
clarkb | tonyb: I think so, all of that stuff should still be processed on zuulv2 | 01:30 |
clarkb | we're in a weird halfway spot for the moment while we get project-config on zuulv3 properly | 01:31 |
prometheanfire | /tmp/in_target.d/install.d/60-zuul-worker: line 25: die: command not found | 01:32 |
prometheanfire | boooo? | 01:32 |
tonyb | clarkb: Thanks. | 01:32 |
clarkb | tonyb: but may be try a single thing and make sure I haven't mis interpreted the situation | 01:32 |
*** dimak has quit IRC | 01:32 | |
clarkb | prometheanfire: need more context | 01:32 |
fungi | prometheanfire: is zuul telling you to fork off and die? | 01:32 |
*** lihi has quit IRC | 01:33 | |
fungi | pretty rude, if so | 01:33 |
ianw | prometheanfire: that's pretty closely related to your issue of finding somehwere to have a library function in the chroot i think | 01:33 |
tonyb | clarkb: Yeah I'll go slow and careful :) | 01:33 |
*** dimak has joined #openstack-infra | 01:33 | |
prometheanfire | trying to build a nodepool image | 01:33 |
*** lihi has joined #openstack-infra | 01:33 | |
*** ijw has quit IRC | 01:33 | |
prometheanfire | that's not my code this time | 01:33 |
clarkb | ZUUL_USER_SSH_PUBLIC_KEY=${ZUUL_USER_SSH_PUBLIC_KEY:-$HOME/.ssh/id_rsa.pub} | 01:34 |
clarkb | weird that it didn't fail in the extra data script since there is a die there too | 01:35 |
prometheanfire | die "Can not find Zuul public key!" | 01:35 |
clarkb | ya in the extra data script for that element its copying the public key into the chroot | 01:35 |
prometheanfire | ah, I'll just put a $HOME/.ssh/id_rsa.pub in there | 01:36 |
clarkb | and the line I pasted above is where it looks for the public key file | 01:36 |
prometheanfire | make it mine for the test | 01:36 |
prometheanfire | take another 3 hours to test :P | 01:36 |
prometheanfire | turns out building a gentoo image takes a while | 01:36 |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner: enter directory before copying https://review.openstack.org/507713 | 01:37 |
*** caphrim007 has quit IRC | 01:37 | |
clarkb | ianw: comment on ^ otherwise I think its worth using that route | 01:38 |
*** caphrim007 has joined #openstack-infra | 01:38 | |
ianw | clarkb: thanks, durr | 01:42 |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner: enter directory before copying https://review.openstack.org/507713 | 01:42 |
*** gildub has joined #openstack-infra | 01:48 | |
clarkb | dims: assuming things are moving with zuul tomorrow I can take a look at the mogan repo stuff, won't tonight because its late enough I don't want to be force pushing anything | 01:49 |
dims | clarkb yep of course, sounds good. thanks! | 01:50 |
*** Douhet has quit IRC | 01:51 | |
*** Douhet has joined #openstack-infra | 01:52 | |
mordred | clarkb, ianw: looks good - +3 | 01:53 |
ianw | does that clear known issues with project-config? | 01:54 |
*** gildub has quit IRC | 01:54 | |
clarkb | ianw: ya I think once those two jobs work project-config is gateable on zuulv3 | 01:54 |
mordred | yup | 01:54 |
*** Apoorva has joined #openstack-infra | 01:54 | |
clarkb | mordred: re tonyb's question about releases above, was I correct | 01:55 |
clarkb | basically zuulv2 is still managing all that and should be safe for release team to continue to make releases? | 01:55 |
tonyb | clarkb: well the 1 thing I tried worked ;P | 01:55 |
mordred | yes - we haven't landed the switch to have v3 take over | 01:55 |
mordred | only project-config is in a weird state | 01:55 |
mordred | and once that ianw patch lands and fixes fedora/centos and https://review.openstack.org/#/c/507222/ turns green - we'll be in solid shape | 01:56 |
mordred | also - yay for this turning up on that project-config patch and not, you know, on 10k jobs all at the same time | 01:56 |
*** hongbin has joined #openstack-infra | 01:58 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: zuul-cloner: enter directory before copying https://review.openstack.org/507713 | 02:02 |
mordred | \o/ | 02:02 |
clarkb | want to recheck thtat change now? | 02:02 |
mordred | I've rechecked https://review.openstack.org/#/c/507222 | 02:02 |
mordred | fingers crossed | 02:02 |
clarkb | still failed | 02:05 |
clarkb | ran with new code too according to log (I see the cd) | 02:06 |
clarkb | I really need to afk now though | 02:06 |
*** thorst has joined #openstack-infra | 02:07 | |
ianw | hmm, post_failure is different though? having a look | 02:08 |
ianw | bummer | 02:13 |
ianw | http://logs.openstack.org/22/507222/3/check/legacy-bindep-fallback-centos-7/74453e9/job-output.txt.gz#_2017-09-27_02_04_04_608530 ... the cd bit worked | 02:13 |
*** aeng has quit IRC | 02:16 | |
*** SumitNaiksatam has joined #openstack-infra | 02:18 | |
*** srobert has joined #openstack-infra | 02:20 | |
*** gongysh has joined #openstack-infra | 02:38 | |
*** esberglu has joined #openstack-infra | 02:42 | |
*** esberglu has quit IRC | 02:47 | |
*** erlon has quit IRC | 02:48 | |
*** thorst has quit IRC | 02:48 | |
*** kjackal_ has joined #openstack-infra | 02:55 | |
*** liujiong has joined #openstack-infra | 03:00 | |
*** ijw has joined #openstack-infra | 03:12 | |
*** rakhmerov has quit IRC | 03:14 | |
jeblair | mordred, ianw, clarkb, Shrews: if we don't find a way to make this work, we can probably just decide to change $WORKSPACE from /home/zuul to /home/zuul/workspace, and add a step to legacy-base to ensure that dir is created. | 03:16 |
ianw | jeblair: i've been testing and i think drop the -a | 03:16 |
jeblair | (or even go so far as to make it /home/zuul/$JOB to be even more jenkins/zuulv2 like) | 03:16 |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Revert "zuul-cloner: enter directory before copying" https://review.openstack.org/507723 | 03:16 |
*** ijw has quit IRC | 03:17 | |
ianw | jeblair: ^ | 03:17 |
jeblair | ianw: cool. i'm not really here -- just wanted to throw that out there as an option -- there isn't anything terribly magical about the current value of WORKSPACE -- it just seemed convenient at the time. less so in retrospect. | 03:18 |
ianw | http://paste.openstack.org/show/621999/ | 03:18 |
jeblair | there may be other reasons to move workspace down too -- like not surprising any jobs by having a .ssh dir inside the git working dir | 03:19 |
jeblair | i may be talking myself into that :) | 03:20 |
jeblair | anyway... g'night. | 03:20 |
ianw | yeah, it's an option | 03:20 |
*** nicolasbock has quit IRC | 03:21 | |
openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 03:26 |
*** tojuvone has joined #openstack-infra | 03:27 | |
*** tojuvone has left #openstack-infra | 03:27 | |
*** rakhmerov has joined #openstack-infra | 03:28 | |
*** vivsoni has joined #openstack-infra | 03:32 | |
adriant | I seem to be having some trouble getting a package published on pypi by the CI system :( | 03:37 |
adriant | I pushed a signed tag, the tag does exist, a job did appear to run in zuul, openstackci is owner in pypi, but no pypi update :( | 03:38 |
adriant | project in question is python-adjutantclient | 03:40 |
adriant | I mean, I can always manually push the new package to pypi but I thought I should do it properly | 03:40 |
clarkb | adriant: http://logs.openstack.org/26/26f0b531fa5ce0bd0e413c0c9e3f9cb8ab95516b/release/python-adjutantclient-tarball/5eca9b6/console.html#_2017-09-27_03_20_57_640721 it failed to make the tarball sdist because no venv env in tox.ini | 03:43 |
adriant | clarkb: bah! ty | 03:44 |
adriant | clarkb: so after I add that how do I retrigger the release? | 03:44 |
adriant | actually | 03:45 |
adriant | nvm | 03:45 |
adriant | new commit, new tag | 03:45 |
*** thorst has joined #openstack-infra | 03:45 | |
*** Sukhdev has joined #openstack-infra | 03:46 | |
*** gongysh has quit IRC | 03:48 | |
zhongjun | Since the manila-image-elements patch has been merged (https://review.openstack.org/#/c/507640/), but the image at http://tarballs.openstack.org/manila-image-elements/images/ is still dated 2017-09-21. Does anyone know how to update image in tarballs. thanks | 03:49 |
*** aeng has joined #openstack-infra | 03:53 | |
ianw | zhongjun: does it comes from a periodic job or what? | 03:54 |
ianw | it's a post job | 03:58 |
ianw | http://logs.openstack.org/ff/ffabcc1a591c55e60e93da9a1f2325a8ca46ed0d/post/gate-manila-publishimage-generic-ubuntu-xenial/29adc11/console.html | 03:58 |
*** krtaylor has quit IRC | 04:00 | |
zhongjun | ianw: The image build from manila-image-elements code. I am not sure if it is a periodic job or anything else | 04:01 |
*** coolsvap has joined #openstack-infra | 04:01 | |
*** krtaylor has joined #openstack-infra | 04:01 | |
ianw | zhongjun: i can try a rerun, per the above link the post job failed. but that was during a time of fairly big instability so let's see if it works on rerun... | 04:06 |
*** krtaylor has quit IRC | 04:08 | |
zhongjun | ianw: the post job failed because of we use the worry image from tarball | 04:09 |
ianw | zhongjun: the job ^^ failed before it did anything cloning repos | 04:10 |
zhongjun | http://logs.openstack.org/40/507640/1/check/gate-manila-tempest-dsvm-generic-scenario-custom-image-ubuntu-xenial-nv/9018a54/logs/devstacklog.txt.gz#_2017-09-26_18_58_48_637 | 04:10 |
ianw | i guess you're saying the current gate is broken because the post job failed to upload the right image? | 04:11 |
zhongjun | Yes | 04:13 |
zhongjun | The current image is not correct | 04:14 |
ianw | zhongjun: ok, i reran the publish job for you, logs are at -> http://logs.openstack.org/ff/ffabcc1a591c55e60e93da9a1f2325a8ca46ed0d/post/gate-manila-publishimage-generic-ubuntu-xenial/rerun/console.html | 04:15 |
ianw | manila-service-image-None.qcow22017-09-27 04:12 323M | 04:16 |
ianw | zhongjun: that seems up to date now? | 04:16 |
*** thorst has quit IRC | 04:16 | |
*** ykarel|away has joined #openstack-infra | 04:17 | |
zhongjun | Yes, thank you | 04:17 |
*** jaosorior has joined #openstack-infra | 04:19 | |
*** ianychoi_ has joined #openstack-infra | 04:20 | |
*** Apoorva_ has joined #openstack-infra | 04:21 | |
*** hongbin has quit IRC | 04:21 | |
*** peterlisak_ has joined #openstack-infra | 04:22 | |
*** purp_too has joined #openstack-infra | 04:22 | |
*** gongysh has joined #openstack-infra | 04:22 | |
*** kashyap` has joined #openstack-infra | 04:22 | |
*** zz_ja has joined #openstack-infra | 04:22 | |
*** inc01 has joined #openstack-infra | 04:23 | |
*** ykarel|away is now known as ykarel|training | 04:23 | |
*** spotz_ has joined #openstack-infra | 04:24 | |
*** admcleod_ has joined #openstack-infra | 04:24 | |
*** thorre_se has joined #openstack-infra | 04:24 | |
*** bradm_ has joined #openstack-infra | 04:24 | |
*** rkukura_ has joined #openstack-infra | 04:24 | |
*** colettecello has joined #openstack-infra | 04:25 | |
*** matbu_ has joined #openstack-infra | 04:25 | |
*** spiffxp_ has joined #openstack-infra | 04:25 | |
*** migi_ has joined #openstack-infra | 04:26 | |
*** Sukhdev has quit IRC | 04:29 | |
*** Apoorva has quit IRC | 04:29 | |
*** hemna_ has quit IRC | 04:29 | |
*** mriedem has quit IRC | 04:29 | |
*** xinliang has quit IRC | 04:29 | |
*** rkukura has quit IRC | 04:29 | |
*** danpawlik has quit IRC | 04:29 | |
*** thorre has quit IRC | 04:29 | |
*** cburgess has quit IRC | 04:29 | |
*** spotz has quit IRC | 04:29 | |
*** yolanda has quit IRC | 04:29 | |
*** ianychoi has quit IRC | 04:29 | |
*** inc0 has quit IRC | 04:29 | |
*** jrist has quit IRC | 04:29 | |
*** leifmadsen has quit IRC | 04:29 | |
*** Diabelko has quit IRC | 04:29 | |
*** flaper87 has quit IRC | 04:29 | |
*** jab has quit IRC | 04:29 | |
*** jcook has quit IRC | 04:29 | |
*** chason has quit IRC | 04:29 | |
*** abelur has quit IRC | 04:29 | |
*** sambetts|afk has quit IRC | 04:29 | |
*** purp has quit IRC | 04:29 | |
*** peterlisak has quit IRC | 04:29 | |
*** ericyoung has quit IRC | 04:29 | |
*** migi has quit IRC | 04:29 | |
*** stephenfin has quit IRC | 04:29 | |
*** matbu has quit IRC | 04:29 | |
*** admcleod has quit IRC | 04:29 | |
*** honza has quit IRC | 04:29 | |
*** dmsimard has quit IRC | 04:29 | |
*** spiffxp has quit IRC | 04:29 | |
*** kashyap has quit IRC | 04:29 | |
*** bradm has quit IRC | 04:29 | |
*** gothicmindfood has quit IRC | 04:29 | |
*** rkukura_ is now known as rkukura | 04:29 | |
*** peterlisak_ is now known as peterlisak | 04:29 | |
*** honza has joined #openstack-infra | 04:29 | |
*** stephenfin has joined #openstack-infra | 04:30 | |
*** abelur has joined #openstack-infra | 04:30 | |
*** honza is now known as Guest85888 | 04:30 | |
*** thorre_se is now known as thorre | 04:30 | |
*** chason has joined #openstack-infra | 04:30 | |
*** sambetts_ has joined #openstack-infra | 04:30 | |
*** cburgess has joined #openstack-infra | 04:30 | |
*** ericyoung has joined #openstack-infra | 04:30 | |
*** claudiub has joined #openstack-infra | 04:30 | |
*** leifmadsen has joined #openstack-infra | 04:32 | |
*** jcook has joined #openstack-infra | 04:32 | |
*** jklare has quit IRC | 04:32 | |
*** edwarnicke has quit IRC | 04:32 | |
*** nhicher has quit IRC | 04:33 | |
*** edwarnicke has joined #openstack-infra | 04:33 | |
*** slagle has quit IRC | 04:34 | |
*** trown|outtypewww has quit IRC | 04:34 | |
*** mdbooth has quit IRC | 04:34 | |
*** jklare has joined #openstack-infra | 04:34 | |
*** trown has joined #openstack-infra | 04:35 | |
*** nhicher has joined #openstack-infra | 04:35 | |
*** kjackal_ has quit IRC | 04:35 | |
*** xinliang has joined #openstack-infra | 04:36 | |
*** udesale has joined #openstack-infra | 04:36 | |
*** jrist has joined #openstack-infra | 04:36 | |
*** flaper87 has joined #openstack-infra | 04:36 | |
*** yolanda has joined #openstack-infra | 04:37 | |
*** dmsimard has joined #openstack-infra | 04:37 | |
*** hemna_ has joined #openstack-infra | 04:37 | |
*** Diabelko has joined #openstack-infra | 04:37 | |
*** danpawlik has joined #openstack-infra | 04:37 | |
*** Apoorva_ has quit IRC | 04:40 | |
*** sree has joined #openstack-infra | 04:40 | |
adriant | clarkb: ty again! package now updated on pypi :) | 04:41 |
*** slagle has joined #openstack-infra | 04:41 | |
*** ankkumar has joined #openstack-infra | 04:42 | |
*** armax has joined #openstack-infra | 04:42 | |
*** liujiong has quit IRC | 04:44 | |
*** psachin has joined #openstack-infra | 04:47 | |
*** mdbooth has joined #openstack-infra | 04:48 | |
zhongjun | ianw: sorry, the image name is not correct, the correct image name is "manila-service-image-master.qcow2" | 04:58 |
ianw | zhongjun: hmm, i may have missed an argument there | 05:01 |
ianw | zhongjun: running it again with a refname of master | 05:01 |
zhongjun | ianw: thank you, how to rerun it? | 05:04 |
*** zhurong has joined #openstack-infra | 05:13 | |
ianw | zhongjun: it's a manual process, i guess you could commit some dummy change. anyway, check again, i think it's right now | 05:13 |
*** thorst has joined #openstack-infra | 05:13 | |
*** rlandy has quit IRC | 05:14 | |
zhongjun | ianw: yeah. It's right now, thank you again | 05:17 |
*** dhajare has joined #openstack-infra | 05:18 | |
*** esberglu has joined #openstack-infra | 05:25 | |
*** esberglu has quit IRC | 05:29 | |
tonyb | Can an infra core look at: https://review.openstack.org/#/c/505656/5 I want to make sure I'm processing the file correctly, and avoid +W'ing my own change | 05:32 |
ankkumar | Hi AJaeger_ | 05:35 |
ankkumar | Can you please review this patch https://review.openstack.org/#/c/462695/ | 05:35 |
*** thorst has quit IRC | 05:47 | |
*** ltomasbo has quit IRC | 05:49 | |
*** jpena|off has quit IRC | 05:52 | |
*** amoralej has quit IRC | 05:53 | |
*** jamesmcarthur has joined #openstack-infra | 05:53 | |
*** jamesmcarthur has quit IRC | 05:58 | |
openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add NFS test gate to OpenStack-Helm https://review.openstack.org/507682 | 05:58 |
ianw | zhongjun: np, i'll delete the bad one | 06:01 |
tonyb | ianw: Thanks! | 06:01 |
openstackgerrit | Flavio Percoco proposed openstack-infra/irc-meetings master: Reflect TC's office hours in the meetings calendar https://review.openstack.org/507490 | 06:02 |
*** pahuang has quit IRC | 06:04 | |
*** aeng has quit IRC | 06:06 | |
openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: JJB meetings are being held in dedicated channel https://review.openstack.org/496314 | 06:07 |
openstackgerrit | Merged openstack-infra/irc-meetings master: Update channel list from project-config https://review.openstack.org/505656 | 06:09 |
openstackgerrit | Merged openstack-infra/irc-meetings master: Proposal for change of FWaaS meeting https://review.openstack.org/507172 | 06:10 |
openstackgerrit | Merged openstack-infra/irc-meetings master: Change heat meeting schedule to 1300 UTC Wed. https://review.openstack.org/506901 | 06:10 |
*** pgadiya has joined #openstack-infra | 06:16 | |
*** jtomasek has quit IRC | 06:18 | |
openstackgerrit | Merged openstack-infra/irc-meetings master: JJB meetings are being held in dedicated channel https://review.openstack.org/496314 | 06:19 |
*** armax has quit IRC | 06:19 | |
*** fbouliane has quit IRC | 06:19 | |
*** ltomasbo has joined #openstack-infra | 06:21 | |
*** fbouliane has joined #openstack-infra | 06:22 | |
*** hashar has joined #openstack-infra | 06:33 | |
*** andreas_s has joined #openstack-infra | 06:36 | |
*** rcernin has joined #openstack-infra | 06:39 | |
*** rossella_s has joined #openstack-infra | 06:40 | |
*** e0ne has joined #openstack-infra | 06:40 | |
*** armax has joined #openstack-infra | 06:41 | |
*** yamamoto_ has quit IRC | 06:41 | |
*** thorst has joined #openstack-infra | 06:45 | |
*** yamamoto has joined #openstack-infra | 06:45 | |
*** armax has quit IRC | 06:50 | |
*** pgadiya has quit IRC | 06:55 | |
*** rossella_s has quit IRC | 06:56 | |
*** Guest61942 has quit IRC | 07:03 | |
*** yamamoto_ has joined #openstack-infra | 07:04 | |
*** yamamoto has quit IRC | 07:04 | |
*** florianf has joined #openstack-infra | 07:05 | |
*** iyamahat has joined #openstack-infra | 07:05 | |
*** iyamahat has quit IRC | 07:06 | |
*** iyamahat has joined #openstack-infra | 07:06 | |
*** sree has quit IRC | 07:06 | |
*** sree has joined #openstack-infra | 07:07 | |
*** e0ne has quit IRC | 07:08 | |
*** yamamoto_ has quit IRC | 07:10 | |
*** sree has quit IRC | 07:11 | |
*** esberglu has joined #openstack-infra | 07:13 | |
*** sree has joined #openstack-infra | 07:13 | |
*** feisky has joined #openstack-infra | 07:15 | |
*** lamt has quit IRC | 07:16 | |
*** Dinesh_Bhor has quit IRC | 07:17 | |
*** thorst has quit IRC | 07:17 | |
*** esberglu has quit IRC | 07:17 | |
*** tdasilva has quit IRC | 07:18 | |
*** yamamoto has joined #openstack-infra | 07:18 | |
openstackgerrit | Merged openstack/diskimage-builder master: Actually sort mount-point list https://review.openstack.org/504819 | 07:20 |
*** Dinesh_Bhor has joined #openstack-infra | 07:21 | |
*** tesseract has joined #openstack-infra | 07:21 | |
*** jpich has joined #openstack-infra | 07:29 | |
*** jaosorior has quit IRC | 07:31 | |
*** shardy has joined #openstack-infra | 07:35 | |
*** threestrands has quit IRC | 07:38 | |
*** mat128 has joined #openstack-infra | 07:38 | |
*** tdasilva has joined #openstack-infra | 07:42 | |
*** mat128 has quit IRC | 07:44 | |
*** Hal has joined #openstack-infra | 07:45 | |
*** Hal is now known as Guest19173 | 07:45 | |
*** iyamahat has quit IRC | 07:46 | |
*** jaosorior has joined #openstack-infra | 07:46 | |
*** amoralej has joined #openstack-infra | 07:47 | |
*** jpena has joined #openstack-infra | 07:47 | |
*** ijw has joined #openstack-infra | 07:47 | |
*** ykarel|training is now known as ykarel | 07:53 | |
*** alexchadin has joined #openstack-infra | 07:53 | |
*** kashyap` is now known as kashyap | 07:57 | |
*** ccamacho has joined #openstack-infra | 07:57 | |
*** hashar has quit IRC | 07:58 | |
*** ijw has quit IRC | 08:02 | |
*** yamahata has joined #openstack-infra | 08:03 | |
*** ralonsoh has joined #openstack-infra | 08:04 | |
*** hashar has joined #openstack-infra | 08:04 | |
*** efried has quit IRC | 08:08 | |
*** zhurong has quit IRC | 08:11 | |
*** egonzalez has joined #openstack-infra | 08:12 | |
*** thorst has joined #openstack-infra | 08:14 | |
*** Guest19173 has quit IRC | 08:15 | |
*** makowals has quit IRC | 08:17 | |
*** efried has joined #openstack-infra | 08:19 | |
*** baoli has joined #openstack-infra | 08:28 | |
*** yamahata has quit IRC | 08:29 | |
*** zhurong has joined #openstack-infra | 08:29 | |
*** dizquierdo has joined #openstack-infra | 08:30 | |
*** makowals has joined #openstack-infra | 08:32 | |
*** baoli has quit IRC | 08:32 | |
*** dingyichen has quit IRC | 08:35 | |
*** alexchadin has quit IRC | 08:41 | |
*** iyamahat has joined #openstack-infra | 08:42 | |
*** alexchadin has joined #openstack-infra | 08:42 | |
*** iyamahat has quit IRC | 08:43 | |
*** iyamahat has joined #openstack-infra | 08:43 | |
*** derekh has joined #openstack-infra | 08:47 | |
*** alexchadin has quit IRC | 08:48 | |
*** chlong has quit IRC | 08:49 | |
*** thorst has quit IRC | 08:50 | |
*** iyamahat has quit IRC | 08:53 | |
*** jamesmcarthur has joined #openstack-infra | 08:53 | |
*** jascott1 has quit IRC | 08:54 | |
*** jascott1 has joined #openstack-infra | 08:55 | |
*** udesale has quit IRC | 08:57 | |
*** udesale has joined #openstack-infra | 08:57 | |
*** jamesmcarthur has quit IRC | 08:58 | |
*** jascott1 has quit IRC | 08:59 | |
*** jtomasek has joined #openstack-infra | 09:01 | |
*** esberglu has joined #openstack-infra | 09:01 | |
*** claudiub|2 has joined #openstack-infra | 09:05 | |
*** esberglu has quit IRC | 09:06 | |
*** claudiub|3 has joined #openstack-infra | 09:06 | |
*** makowals has quit IRC | 09:07 | |
*** alexchadin has joined #openstack-infra | 09:07 | |
*** alexchadin has quit IRC | 09:08 | |
*** claudiub has quit IRC | 09:08 | |
*** alexchadin has joined #openstack-infra | 09:08 | |
*** claudiub|2 has quit IRC | 09:10 | |
*** alexchadin has quit IRC | 09:11 | |
*** markmcd has quit IRC | 09:14 | |
*** electrofelix has joined #openstack-infra | 09:15 | |
*** makowals has joined #openstack-infra | 09:16 | |
*** e0ne has joined #openstack-infra | 09:16 | |
*** markmcd has joined #openstack-infra | 09:18 | |
*** yamamoto has quit IRC | 09:19 | |
*** Guest85888 is now known as honza | 09:22 | |
*** pbourke has quit IRC | 09:23 | |
*** pbourke has joined #openstack-infra | 09:24 | |
*** tosky has joined #openstack-infra | 09:25 | |
*** feisky has quit IRC | 09:25 | |
*** numans has quit IRC | 09:25 | |
*** iyamahat has joined #openstack-infra | 09:26 | |
*** askb has quit IRC | 09:26 | |
*** askb has joined #openstack-infra | 09:27 | |
*** sambetts_ is now known as sambetts | 09:27 | |
*** numans has joined #openstack-infra | 09:28 | |
openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 09:30 |
*** iyamahat has quit IRC | 09:31 | |
*** askb has quit IRC | 09:32 | |
*** askb has joined #openstack-infra | 09:33 | |
*** jtomasek has quit IRC | 09:38 | |
*** electrofelix has quit IRC | 09:46 | |
*** thorst has joined #openstack-infra | 09:47 | |
*** afazekas is now known as afazekas|seek4fo | 09:48 | |
*** hashar is now known as hasharAway | 09:50 | |
*** sdague has joined #openstack-infra | 09:50 | |
*** esberglu has joined #openstack-infra | 10:00 | |
*** LindaWang has quit IRC | 10:01 | |
*** stakeda has quit IRC | 10:02 | |
*** d0ugal has joined #openstack-infra | 10:03 | |
*** esberglu has quit IRC | 10:04 | |
*** cuongnv has quit IRC | 10:05 | |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 10:08 |
*** dizquierdo has quit IRC | 10:11 | |
*** LindaWang has joined #openstack-infra | 10:16 | |
*** thorst has quit IRC | 10:18 | |
*** gongysh has quit IRC | 10:19 | |
*** zhurong has quit IRC | 10:20 | |
*** yamamoto has joined #openstack-infra | 10:20 | |
*** yamahata has joined #openstack-infra | 10:27 | |
*** egonzalez has quit IRC | 10:31 | |
*** jkilpatr has quit IRC | 10:34 | |
*** andreas_s_ has joined #openstack-infra | 10:37 | |
*** andreas_s has quit IRC | 10:40 | |
*** jascott1 has joined #openstack-infra | 10:42 | |
*** apuimedo_ has joined #openstack-infra | 10:43 | |
*** andreas_s_ has quit IRC | 10:45 | |
*** apuimedo has quit IRC | 10:45 | |
*** apuimedo_ is now known as apuimedo | 10:45 | |
*** jascott1 has quit IRC | 10:46 | |
*** ankkumar has quit IRC | 10:47 | |
*** andreas_s has joined #openstack-infra | 10:47 | |
*** d0ugal has quit IRC | 10:48 | |
*** udesale has quit IRC | 10:52 | |
*** vponomaryov has quit IRC | 10:54 | |
*** hemna_ has quit IRC | 10:57 | |
*** nicolasbock has joined #openstack-infra | 11:00 | |
*** d0ugal has joined #openstack-infra | 11:02 | |
*** jkilpatr has joined #openstack-infra | 11:02 | |
*** nicolasbock has quit IRC | 11:04 | |
*** jkilpatr has quit IRC | 11:07 | |
*** kjackal_ has joined #openstack-infra | 11:10 | |
*** pbourke has quit IRC | 11:11 | |
*** timothyb89 has quit IRC | 11:11 | |
*** jaosorior is now known as jaosorior_sick | 11:12 | |
*** thorst has joined #openstack-infra | 11:15 | |
*** vponomaryov has joined #openstack-infra | 11:16 | |
*** jkilpatr has joined #openstack-infra | 11:21 | |
*** kjackal_ has quit IRC | 11:21 | |
*** strigazi has quit IRC | 11:23 | |
*** strigazi has joined #openstack-infra | 11:24 | |
*** egonzalez has joined #openstack-infra | 11:25 | |
*** pbourke has joined #openstack-infra | 11:25 | |
*** thorst has quit IRC | 11:27 | |
*** pcaruana has joined #openstack-infra | 11:27 | |
*** udesale has joined #openstack-infra | 11:28 | |
*** kiennt26 has joined #openstack-infra | 11:29 | |
*** jtomasek has joined #openstack-infra | 11:41 | |
*** rossella_s has joined #openstack-infra | 11:41 | |
*** dizquierdo has joined #openstack-infra | 11:42 | |
*** rossella_s has quit IRC | 11:47 | |
*** dave-mccowan has joined #openstack-infra | 11:49 | |
*** bobh has joined #openstack-infra | 11:50 | |
Shrews | jeblair: mordred: i'm inclined to go with changing $WORKSPACE | 11:54 |
*** jpena is now known as jpena|lunch | 11:54 | |
*** thorst has joined #openstack-infra | 11:55 | |
*** dprince has joined #openstack-infra | 11:56 | |
mordred | Shrews, jeblair: yah - me too | 11:58 |
*** inc01 has quit IRC | 11:59 | |
*** dave-mccowan has quit IRC | 12:01 | |
*** kiennt26 has quit IRC | 12:02 | |
*** mat128 has joined #openstack-infra | 12:03 | |
*** wolverineav has joined #openstack-infra | 12:04 | |
*** dave-mccowan has joined #openstack-infra | 12:04 | |
*** trown is now known as trown|outtypewww | 12:05 | |
*** trown|outtypewww is now known as trown|afk | 12:05 | |
*** mat128 has quit IRC | 12:06 | |
*** mat128 has joined #openstack-infra | 12:08 | |
*** mat128 has quit IRC | 12:08 | |
*** mat128 has joined #openstack-infra | 12:08 | |
*** afazekas|seek4fo is now known as afazekas | 12:09 | |
*** bobh has quit IRC | 12:10 | |
Shrews | i wonder why there is a "mkdir -p $WORKSPACE" *after* we are already in said workspace doing zuul-cloner-shim things? http://logs.openstack.org/22/507222/4/check/legacy-bindep-fallback-centos-7/7b2fdba/job-output.txt.gz#_2017-09-27_03_31_45_052288 | 12:10 |
Shrews | i can't find where that's coming from | 12:10 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put legacy jobs down a directory level into workspace https://review.openstack.org/507834 | 12:14 |
mordred | Shrews: how's that look? ^^ | 12:15 |
*** dtantsur|afk is now known as dtantsur | 12:16 | |
*** alexchadin has joined #openstack-infra | 12:16 | |
*** pblaho has quit IRC | 12:18 | |
*** Hal has joined #openstack-infra | 12:18 | |
*** threestrands has joined #openstack-infra | 12:18 | |
*** threestrands has quit IRC | 12:18 | |
*** threestrands has joined #openstack-infra | 12:18 | |
*** Hal is now known as Guest64180 | 12:18 | |
*** pblaho has joined #openstack-infra | 12:19 | |
Shrews | mordred: lgtm, but don't we need to change the workspace in more than just those locations? i would have thought that would be defined in a base job or such | 12:21 |
mordred | not really - the converted JJB jobs are all self-contained for the most part | 12:22 |
mordred | the whole idea of 'WORKSPACE' isn't really a real thing in v3 | 12:22 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add workspace directory for converted legacy jobs https://review.openstack.org/507835 | 12:23 |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: We should await ws.send_str() https://review.openstack.org/507671 | 12:25 |
Shrews | oh, that mkdir is coming from https://review.openstack.org/#/c/507835/1/playbooks/legacy/zuul-nodepool/run.yaml | 12:29 |
mordred | Shrews: ah - so it is | 12:30 |
*** gouthamr has joined #openstack-infra | 12:30 | |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 12:30 |
Shrews | mordred: i think that trap is still safe... yeah? | 12:31 |
Shrews | yeah, nothing is removed now, so it shouldn't do anything | 12:32 |
Shrews | so many pieces | 12:33 |
*** kjackal_ has joined #openstack-infra | 12:34 | |
*** gouthamr_ has joined #openstack-infra | 12:37 | |
*** gouthamr has quit IRC | 12:38 | |
mordred | Shrews: yah | 12:40 |
mordred | srrsly | 12:40 |
*** rlandy has joined #openstack-infra | 12:44 | |
*** gcb has quit IRC | 12:45 | |
*** amoralej is now known as amoralej|lunch | 12:48 | |
*** jcoufal has joined #openstack-infra | 12:51 | |
*** mat128 has quit IRC | 12:53 | |
*** Guest64180 has quit IRC | 12:53 | |
Shrews | i think we still want the revert of ianw's change: https://review.openstack.org/507723 | 12:54 |
mordred | yah - we can at least land it and even test to see if it works while we're waiting on others to awaken | 12:55 |
*** alexchadin has quit IRC | 12:55 | |
mordred | I agree with jeblair from above (and pabelanger_ from yesterday) that adding a workspace dir to the mix may still be nicer for other reasons | 12:55 |
*** pabelanger_ is now known as pabelanger | 12:56 | |
*** kiennt26 has joined #openstack-infra | 12:56 | |
*** alexchadin has joined #openstack-infra | 12:56 | |
*** jpena|lunch is now known as jpena | 12:57 | |
Shrews | yeah | 12:57 |
*** esberglu has joined #openstack-infra | 12:58 | |
pabelanger | mordred: just legacy jobs? | 13:00 |
*** jdandrea_ has joined #openstack-infra | 13:01 | |
*** eharney has joined #openstack-infra | 13:01 | |
mordred | pabelanger: yah | 13:02 |
pabelanger | ++ | 13:02 |
mordred | we don't need it for non-legacy jobs | 13:02 |
pabelanger | Yup, agree | 13:02 |
mordred | pabelanger: check out https://review.openstack.org/507835 and https://review.openstack.org/507834 | 13:02 |
pabelanger | looking | 13:03 |
mordred | pabelanger: (need to land 834 and get the filter plugin change rolled out before 835 will work) | 13:03 |
*** tpsilva has joined #openstack-infra | 13:03 | |
mordred | just rechecked https://review.openstack.org/#/c/507222 to see if ianw's change improved things too | 13:06 |
pabelanger | mordred: comment on 507834 | 13:06 |
*** kgiusti has joined #openstack-infra | 13:07 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 13:08 |
*** mriedem has joined #openstack-infra | 13:09 | |
mordred | pabelanger: I believe we don't want the slash - but one sec ... | 13:09 |
pabelanger | I know the trailing / for copy task means everything inside the directory | 13:10 |
pabelanger | cannot remember for sync | 13:10 |
mordred | pabelanger: nope. you're right, I'm wrong | 13:10 |
mordred | http://logs.openstack.org/03/506803/1/check/gate-bindep-bindep-fallback-ubuntu-trusty/69c2538/_zuul_ansible/post_playbook | 13:10 |
mordred | that's the one that's been working- it has the slash | 13:10 |
pabelanger | kk | 13:10 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put legacy jobs down a directory level into workspace https://review.openstack.org/507834 | 13:11 |
*** jistr is now known as jistr|call | 13:12 | |
mordred | pabelanger, Shrews: there ya go ^^ | 13:12 |
*** krtaylor has joined #openstack-infra | 13:13 | |
mordred | WOOT | 13:13 |
mordred | ianw's change fixed project-config jobs | 13:13 |
Shrews | sweeeet | 13:13 |
pabelanger | Nice | 13:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add workspace directory for converted legacy jobs https://review.openstack.org/507835 | 13:14 |
pabelanger | +3 on 834 | 13:14 |
Shrews | the "Conflicts With" section on gerrit UI is neat here: https://review.openstack.org/#/c/507835/ | 13:15 |
pabelanger | oh, guess we don't have mirrors working on debian-jessie | 13:15 |
Shrews | "400 too many terms in query" | 13:15 |
pabelanger | we pulling from ftp.us.debian.org | 13:15 |
mordred | hrm | 13:15 |
pabelanger | I think that is okay for now, but going to make note of it | 13:16 |
mordred | ++ | 13:17 |
*** kjackal_ has quit IRC | 13:17 | |
pabelanger | 507222 is now green! | 13:18 |
pabelanger | \o/ | 13:18 |
*** trown|afk is now known as trown | 13:18 | |
mordred | woot. so - we're actually set to flip the switch!!! | 13:18 |
* mordred hands pies to pabelanger and Shrews | 13:18 | |
pabelanger | actually, now that I think about it | 13:19 |
pabelanger | why did legacy bindep-fallback run on that change? | 13:19 |
pabelanger | in zuulv2 the jobs only run when we change bindep-fallback.txt | 13:19 |
pabelanger | mordred: Hmm, we don't seem to have files section on bindep-fallback template now | 13:21 |
pabelanger | https://review.openstack.org/#/c/507219/10/zuul/layout.yaml are original | 13:21 |
pabelanger | line 2787 | 13:21 |
Shrews | the legacy jobs don't have any matchers | 13:22 |
pabelanger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n968 is new jobs | 13:22 |
Shrews | why, i do not know | 13:22 |
pabelanger | kk, might be a regression | 13:23 |
*** amoralej|lunch is now known as amoralej | 13:23 | |
mordred | yah - maybe there is a bug in expansion of filters and project-templates in the migration script? | 13:24 |
*** kjackal_ has joined #openstack-infra | 13:25 | |
mnaser | i thought all the matchers went into the projects.yaml file? | 13:25 |
mordred | Shrews: I'm on a call - any chance you have a sec to look at that? (you'll probably have to pull an older version of project-config) | 13:25 |
*** baoli has joined #openstack-infra | 13:26 | |
*** lbragstad has joined #openstack-infra | 13:27 | |
Shrews | yep, just lemme finish eating bfast real quick | 13:27 |
*** sbezverk has joined #openstack-infra | 13:27 | |
mordred | \o/ | 13:28 |
*** chlong has joined #openstack-infra | 13:29 | |
*** chlong has quit IRC | 13:31 | |
pabelanger | mordred: Shrews: I also think we need to add legacy-project-config-layout for project-config project. Possible we missed it by mistake? | 13:32 |
pabelanger | in v2.5 that was the job we always ran, regardless of files settings | 13:32 |
mordred | pabelanger: nope- omitted on purpose | 13:32 |
pabelanger | Hmm | 13:33 |
pabelanger | okay, that could be an issue | 13:33 |
mordred | it's a job to check the zuul v2 layout.yaml | 13:33 |
mordred | we don't have that in v3 anymore - and zuul does syntax checking on its config files directly | 13:33 |
pabelanger | mordred: right, but all the other jobs have potential file matchers. So, there will be a case now if we touch a file that doesn't match, no job will run | 13:34 |
pabelanger | so, we should pick a new job and remove the file matches, to always run | 13:34 |
mordred | ah - great point | 13:34 |
pabelanger | maybe docs job | 13:35 |
mordred | yah | 13:35 |
mordred | pabelanger: or we could add noop-jobs | 13:38 |
mordred | pabelanger: if we don't have an actual test to run on a given piece of content, I'm not sure running the specs site index generation will tell us anything more than noop would | 13:39 |
pabelanger | mordred: k, lets see what others say. We also still need to add a linters job for project-config, so possible that might be the new job | 13:41 |
mordred | pabelanger: ++ | 13:42 |
mordred | pabelanger: let's DEFINITELY at least at noop jobs while we figure it all out so that we don't wind up with no gate jobs :) | 13:42 |
pabelanger | yah | 13:42 |
mordred | remote: https://review.openstack.org/507860 Add noop-jobs as a fallback for project-config | 13:45 |
* Shrews trying to create a minimal test case for debugging matcher bug | 13:46 | |
*** lifeless has quit IRC | 13:46 | |
mordred | Shrews: oh jeez. good luck with that :) | 13:46 |
Shrews | i don't know how else to do it | 13:47 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 13:47 |
mordred | oh! I ahve discovered another issue ... | 13:50 |
mordred | in the 'add projects to gerrit main.yaml' patch, I didn't add projects that are listed as retired | 13:50 |
mordred | but we have some projects that are retired that have jobs in zuul.yaml | 13:51 |
*** alexchadin has quit IRC | 13:51 | |
pabelanger | k | 13:51 |
mordred | let's just add them for now | 13:52 |
mordred | or - I take that back - leme remove them from projects.yaml | 13:53 |
*** jamesmcarthur has joined #openstack-infra | 13:54 | |
*** lifeless has joined #openstack-infra | 13:54 | |
*** dave-mccowan has quit IRC | 13:54 | |
*** camunoz has joined #openstack-infra | 13:54 | |
*** udesale__ has joined #openstack-infra | 13:55 | |
fungi | so it's safe to mix no-op jobs with real jobs in v3 now? | 13:56 |
fungi | or were you suggesting a custom no-op job? | 13:56 |
*** udesale has quit IRC | 13:56 | |
*** hemna_ has joined #openstack-infra | 13:58 | |
*** dangers_away has left #openstack-infra | 13:58 | |
mordred | fungi: I do not know the answer to that question | 13:58 |
Shrews | compare (^gate-bindep-bindep-fallback.*$, gate-{name}-bindep-fallback-centos-7) | 13:58 |
*** rtjure has quit IRC | 13:58 | |
*** baoli has quit IRC | 13:58 | |
Shrews | that's what the migrate script is comparing for job names to apply matchers. we aren't doing the format() there | 13:59 |
*** baoli has joined #openstack-infra | 13:59 | |
mordred | Shrews: ah. that's unfortunate | 14:00 |
Shrews | not sure where to get {name} value from, atm | 14:01 |
openstackgerrit | Mohammed Naser proposed openstack-infra/system-config master: Add Ceph Luminous Debian mirrors https://review.openstack.org/507864 | 14:01 |
mordred | Shrews: well- we do that substitution in Job | 14:02 |
mordred | I believe | 14:02 |
mordred | Shrews: oh - aren't you in a project context there? | 14:02 |
Shrews | mordred: nope. processing templates | 14:02 |
mordred | hrm | 14:03 |
Shrews | mordred: right below the "# Handle matchers" comment in getNewJobs | 14:03 |
*** rtjure has joined #openstack-infra | 14:03 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 14:04 |
*** nhicher has quit IRC | 14:05 | |
Shrews | so this only affects templates where {name} is used. other templates should be fine | 14:05 |
*** pblaho has quit IRC | 14:05 | |
*** nhicher has joined #openstack-infra | 14:05 | |
mordred | where name is used and where it doesn't have an explicit project: matcher defined - since those get handled in the project-template expansion step | 14:06 |
Shrews | oh, my layout file might be wrong too | 14:06 |
mordred | Shrews: oh yah - you need to update project-config to be before we removed the project-config jobs from layout perhaps? | 14:07 |
Shrews | compare (^gate-project-config-bindep-fallback.*$, gate-{name}-bindep-fallback-centos-7) | 14:08 |
Shrews | that's better. still need to get name="project-config" from somewhere | 14:08 |
Shrews | mordred: if we aren't in project context, we cant do this here, can we? | 14:09 |
Shrews | *sigh* | 14:09 |
*** sree has quit IRC | 14:09 | |
*** yamamoto has quit IRC | 14:10 | |
*** felipemonteiro has joined #openstack-infra | 14:10 | |
*** sree has joined #openstack-infra | 14:11 | |
*** felipemonteiro_ has joined #openstack-infra | 14:12 | |
*** mat128 has joined #openstack-infra | 14:12 | |
mordred | Shrews: you might have to ... throw in a for loop and do a check for every project name? | 14:12 |
*** camunoz has quit IRC | 14:13 | |
*** camunoz has joined #openstack-infra | 14:13 | |
openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Remove promotion logic from the toci scripts https://review.openstack.org/507076 | 14:13 |
Shrews | i think we might need to duplicate that logic for cases where the job name wouldn't match in template context. trying that now | 14:15 |
*** yamamoto has joined #openstack-infra | 14:15 | |
*** Goneri has joined #openstack-infra | 14:15 | |
*** felipemonteiro has quit IRC | 14:16 | |
*** sree has quit IRC | 14:16 | |
jeblair | mordred: is there a change to the migration script to add workspace/ ? | 14:18 |
*** alex_xu has quit IRC | 14:18 | |
mordred | jeblair: yes. | 14:19 |
*** srobert has quit IRC | 14:19 | |
mordred | jeblair: https://review.openstack.org/507834 | 14:19 |
mordred | jeblair: and also to the filter | 14:19 |
jeblair | mordred: ah right, included in the filter change | 14:19 |
mordred | jeblair: so - other than the thing Shrews is looking at, we're good to go (And even that could be fixed as a follow up) | 14:20 |
*** yamamoto has quit IRC | 14:20 | |
mordred | jeblair: also - see fungi's question about noop jobs - and the noop discussion with pabelanger about project-config jobs | 14:20 |
mordred | jeblair: also - good morning! | 14:20 |
jeblair | mordred: i'm a little worried that Shrews thing will affect a lot of projects... | 14:20 |
*** alex_xu has joined #openstack-infra | 14:21 | |
mordred | yah | 14:21 |
openstackgerrit | Luigi Toscano proposed openstack/os-testr master: Fix .testr.conf detection: test path follows discover https://review.openstack.org/503877 | 14:21 |
mordred | jeblair: I think we should give him a sec before we flip the switch | 14:22 |
*** mat128 has quit IRC | 14:23 | |
jeblair | mordred, fungi: it's safe inasmuch as it's not going to pull us into any shared queues, but i think it's bad form, so i'd like to avoid it if possible. i think temporarily adding it is fine. but i'd like to remove it shortly. | 14:23 |
jeblair | mordred: good morning! | 14:23 |
Shrews | well, i don't think is simple. the bindep template doesn't get expanded into project-config (no 'skip-if' to reference it). So we can't apply the matchers will processing the project. It has to be while processing the template, but that doesn't have a referencing project | 14:23 |
Shrews | s/will/while/ | 14:23 |
Shrews | so i'm not clear on how to handle this | 14:24 |
openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Remove promotion logic from the toci scripts https://review.openstack.org/507076 | 14:24 |
fungi | do we know if it affects any other jobs/templates besides that one? | 14:25 |
*** jamesmcarthur has quit IRC | 14:25 | |
*** hemna_ has quit IRC | 14:25 | |
Shrews | fungi: good question. i don't know offhand. if it's only a few, we could hand fix it | 14:26 |
* Shrews needs to make another cup o' joe. bias | 14:26 | |
jeblair | the problem is basically: matchers that match jobs applied by a template, right? | 14:26 |
jeblair | rather | 14:26 |
jeblair | the problem is basically: matchers that match jobs with an embedded project name applied by a template, right? | 14:27 |
*** dave-mccowan has joined #openstack-infra | 14:27 | |
Shrews | jeblair: yes | 14:27 |
*** hongbin has joined #openstack-infra | 14:27 | |
*** jistr|call is now known as jistr | 14:27 | |
Shrews | and it will really only be 'files' matchers, i think | 14:29 |
SpamapS | oh my, it's an early morning jeblair | 14:30 |
jeblair | andreaf: i was thinking that perhaps we could use a job variable to list files that should be saved by a devstack/tempest job; then it would be really easy for jobs to specify additional files to save (since that's such a common thing) | 14:30 |
jeblair | mordred: ^ | 14:30 |
pabelanger | fungi: check-tripleo-jobs template seems to be what I expect for files | 14:31 |
Shrews | i guess 'branch' and 'voting' might be affected there, too | 14:32 |
Shrews | mordred: jeblair: the simplest solution seems to be *always* expand templates | 14:34 |
*** hemna_ has joined #openstack-infra | 14:34 | |
Shrews | then apply matchers after expansion | 14:34 |
Shrews | or expand templates where {name} is referenced | 14:35 |
mordred | Shrews: I could see the second perhaps ... losing all the templates would kinda suck | 14:37 |
Shrews | yeah | 14:38 |
*** udesale__ has quit IRC | 14:38 | |
*** dave-mccowan has quit IRC | 14:38 | |
mordred | Shrews: oh - what if we expanded all the templates in the project template matcher thing - then applied the matchers- and if there were no changes (nothing matched) we go with the template rather than the expaned set of jobs | 14:38 |
SpamapS | +1 | 14:38 |
SpamapS | that's what I was going to say | 14:38 |
SpamapS | speculative future calculation is a zuul tradition | 14:39 |
mordred | :) | 14:39 |
*** threestrands has quit IRC | 14:40 | |
*** makowals has quit IRC | 14:40 | |
Shrews | mordred: i think use of "expand" in that sentence is confusing me for some reason. can you use different words? | 14:40 |
*** jascott1 has joined #openstack-infra | 14:41 | |
mordred | Shrews: while processing a project in writeProject - do a pass of of expandTemplateIntoProject for ALL of the templates - but on a copy of the project | 14:41 |
mordred | Shrews: then, check the results and see whether exapnding a given template and applying matchers resulted in jobs with matchers/variants | 14:42 |
mordred | if so - then it's a template that needs to be expanded in the real pass - if not, it's not | 14:42 |
Shrews | mordred: ah. yeah, i think that would work | 14:43 |
mordred | (so it would have to be a loop of for template in project['template'] - expandTemplateIntoProject(template, project.copy) - if doIHaveMatchers(): templates_to_expand.append(template) | 14:43 |
mordred | *waves hands* | 14:43 |
*** alex_xu has quit IRC | 14:44 | |
*** dave-mccowan has joined #openstack-infra | 14:44 | |
*** jascott1 has quit IRC | 14:45 | |
*** jaypipes has joined #openstack-infra | 14:49 | |
*** dave-mcc_ has joined #openstack-infra | 14:49 | |
*** alex_xu has joined #openstack-infra | 14:49 | |
*** rbrndt has joined #openstack-infra | 14:50 | |
*** dave-mccowan has quit IRC | 14:51 | |
*** xarses has joined #openstack-infra | 14:52 | |
*** andreww has joined #openstack-infra | 14:54 | |
mordred | Shrews: oh - although I guess it's not actualy applying applyProjectMatchers after expandTemplateIntoProject - it would be a copy of the Handle matchers logic against the jobs in the template expansion | 14:55 |
*** gongysh has joined #openstack-infra | 14:55 | |
clarkb | re workspace/ change I'm not going to check every single file changed, but looking at a sample few lgtm. Did not approve as unsure if above discussion involves waiting for some reason (still catching up) | 14:56 |
*** xarses has quit IRC | 14:56 | |
*** mat128 has joined #openstack-infra | 14:56 | |
*** baoli has quit IRC | 14:56 | |
mordred | clarkb: the workspace change to ozj is safe to land - the project-config change to add projects to pipelines in v3 and go live is not | 14:56 |
jeblair | i think we could go ahead and merge workspace; the current discussion is likely to produce another patch, but it'll probably be easier to build on it? | 14:57 |
clarkb | ok I'll approve then | 14:57 |
mordred | jeblair, clarkb: if we land the ozj change, we can re-check the project-config change to verify it's still working | 14:57 |
mordred | jeblair: yah- the workspace change is also to playbooks, the upcoming template one will be a patch to the pipeline definitions in project-config | 14:57 |
jeblair | mordred: ack | 14:58 |
*** baoli has joined #openstack-infra | 14:59 | |
mordred | jeblair, Shrews: I *think* from looking at it that if we wanted to it would not be super hard (if we get speculative expansion going - since that will ensure all things get touched at least once) ... to build a patch that keeps track of what instances of a given new job name and matchers applied to them - so that if every instance of a given job had the same matchers applied, we could apply it to the | 15:00 |
mordred | job definition and not expand the template | 15:00 |
mordred | which means I *think* what I'm saying is slightly different than what I suggested above. If instead of doing a speculative stab at applyProjectMatchers in writeProject - we actually just did two entire loops of _calling_ writeProject .... | 15:01 |
mordred | one that calls a version of writeProject that expands every template regardless of need and records informatoin about which expansions were useful ... | 15:02 |
clarkb | does zuul need restarting to apply 834 properly before we can use 835? | 15:02 |
mordred | and then a second one that uses that information to knmow what to expand | 15:02 |
mordred | and update toJobDict to also read the recorded information | 15:02 |
*** jdandrea_ has quit IRC | 15:03 | |
*** ihrachys has joined #openstack-infra | 15:03 | |
mordred | Shrews, jeblair: a) does that make sense and b) is it worth trying to do? | 15:04 |
*** andreas_s_ has joined #openstack-infra | 15:04 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add workspace directory for converted legacy jobs https://review.openstack.org/507835 | 15:04 |
mordred | k. rechecking project-config patch | 15:05 |
*** ykarel has quit IRC | 15:05 | |
*** jdandrea_ has joined #openstack-infra | 15:06 | |
tosky | the proposal bot started rebasing the same patch over and over (https://review.openstack.org/#/c/505885/), is it related to the zuul v3 changes | 15:06 |
tosky | ? | 15:06 |
pabelanger | tosky: no, a result of upgrading gerrit | 15:06 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 15:07 |
*** andreas_s has quit IRC | 15:07 | |
tosky | oh, the other even | 15:07 |
tosky | event* | 15:07 |
tosky | ok | 15:07 |
Shrews | mordred: maybe? i'm still trying to code the first thing | 15:07 |
fungi | tosky: known issue. it's on the list of things to fix soon after zuul v3 cutover is behind us | 15:07 |
pabelanger | tosky: I believe we have a task to make the proposal bot smarter | 15:07 |
tosky | but will it stop at some point? I mean, is there a condition that triggers that over and over, out of the blue? | 15:08 |
mordred | Shrews: ok. cool. how about we just ignore me for right now and roll with it if you get the first thing going | 15:08 |
clarkb | tosky: changes to openstack/requirements merging | 15:08 |
fungi | tosky: the short explanation is that gerrit used to reject new patchsets if the only appreciable change was the commit id but the diff itself was the same. now it only emits a warning in those cases and accepts the new patchset | 15:08 |
fungi | and yes, it should only happen when the requirements-core reviewers approve a change to the global-requirements.txt file | 15:09 |
tosky | oh, so now every change to openstack/requirements will retrigger this? I see | 15:09 |
tosky | thanks | 15:09 |
fungi | which is why we didn't treat it as a higher priority | 15:09 |
fungi | not every change to openstack/requirements, just changes which alter the global-requirements.txt file | 15:09 |
fungi | so, for example, daily updates to upper-constraints.txt shouldn't trigger that | 15:09 |
*** yamamoto has joined #openstack-infra | 15:10 | |
*** makowals has joined #openstack-infra | 15:10 | |
*** srobert has joined #openstack-infra | 15:12 | |
*** makowals has quit IRC | 15:13 | |
Shrews | mordred: i would not be opposed to us trying to code the two things in parallel. whoever gets something that seems to work first wins | 15:15 |
*** chlong has joined #openstack-infra | 15:17 | |
mordred | Shrews: ok - I can take a stab at the thing I said | 15:17 |
openstackgerrit | Luigi Toscano proposed openstack/os-testr master: Fix .testr.conf detection: test path follows discover https://review.openstack.org/503877 | 15:18 |
clarkb | fungi: tosky I think I see the problem in the script, the diff it does is against HEAD (eg tip of master) and not against the pervious patchset if the previous patchset exists | 15:20 |
dirk | clarkb: fungi: that sounds fixable | 15:21 |
clarkb | dirk: ya we'll just need to fetch any existing change and diff against that instead before pushing | 15:21 |
fungi | clarkb: though while we're in there for an overhaul, i think that's probably also a good time to add the reqeusted logic to create a new commit instead of amending if the existing change is already approved | 15:22 |
*** gongysh has quit IRC | 15:23 | |
dirk | fungi: you mean w+1 with approved? | 15:23 |
dirk | I think it just searches for an open review with a special topic | 15:23 |
dirk | Hmm | 15:24 |
* dirk is already out of office for today | 15:24 | |
fungi | yeah, we would likely check the workflow label for a +1 | 15:24 |
dirk | I can take a look tomorrow | 15:24 |
dirk | In return, a few reviews on the stuff that I have pending wouldn't hurt ;-) | 15:25 |
*** gouthamr_ has quit IRC | 15:26 | |
fungi | dirk: any particular changes of yours which are more urgent than others? is there open work on the suse images? i think a lot of stuff is just blocked on the zuul v3 cutover work so we don't create new churn in job configs we'd have to track in two places | 15:27 |
clarkb | I think the 42.2 mirror cleanup and wheels for 42.3 are likely high on that list? | 15:28 |
jeblair | i'm approving the master version of the case-sensitive labels change | 15:28 |
jeblair | only a day late | 15:28 |
fungi | jeblair: oh, thanks! i guess we said it was merging this week | 15:29 |
pabelanger | clarkb: fungi: just looking into debian-jessie image for nodepool, right now we only have bindep-fallback and ansible-hardening using it. Maybe we should see what openstack-ansible is using it for | 15:29 |
*** Sukhdev has joined #openstack-infra | 15:30 | |
pabelanger | evrardjp: odyssey4me: mhayden:^ seems you're the only job using debian-jessie now. Is that something you plan on growing it the future? Would you be sadface if we removed the image from nodepool? | 15:32 |
*** inc0 has joined #openstack-infra | 15:32 | |
odyssey4me | pabelanger mhayden is OOO right now, but I'm pretty sure it'd be sadface | 15:32 |
odyssey4me | if it's a burden to maintain then I'd suggest working out an exit plan with mhayden - maybe he can figure out some external CI for additional platforms | 15:33 |
openstackgerrit | Merged openstack-infra/zuul master: Case sensitive label matching https://review.openstack.org/482856 | 15:33 |
clarkb | ok 507222 is happy now, I guess its just a matter of getting the file matcher stuff in place then we move forward | 15:33 |
Shrews | mordred: so... ugh, template expansion expands a v3 template into a v3 project. our speculative expansion needs to compare v2 template job names, which presents a problem since we've lost those at that point | 15:33 |
dirk | clarkb: the 42.2 clean should be mergable any time? | 15:34 |
*** dprince has quit IRC | 15:34 | |
pabelanger | odyssey4me: okay, not a burden currently. Something we talked about a while back since we removed deb-packaging repos. But if you are using it, we should be able to support it still. | 15:34 |
Shrews | mordred: i guess i could loop through the stored jobs looking for the old name? | 15:34 |
clarkb | dirk: it will need an infra-root to clean up the afs side of things, I think the change itself is likely safe though (just don't want to forget to clean up afs as necessary) | 15:34 |
dirk | Most open review i have are related to enabling tumbleweed, but that's not particularly urgent | 15:34 |
odyssey4me | thanks pabelanger | 15:35 |
*** dtantsur is now known as dtantsur|afk | 15:35 | |
dirk | clarkb: the rm -rf in there should do the cleaning? | 15:35 |
clarkb | dirk: it won't remove the volume though | 15:35 |
*** dprince has joined #openstack-infra | 15:35 | |
clarkb | I'd have to doule check the volume setup there to know if that is necessary though | 15:35 |
dirk | clarkb: I thought it's just one volume overall ? | 15:36 |
clarkb | dirk: also the sync after the rm -rf might need to be run manually as it will be slow (due to the large delta) | 15:36 |
clarkb | assuming it is one volume | 15:36 |
clarkb | so either volume needs to be removed entirely, or volume sync needs to be watched closely | 15:36 |
fungi | odyssey4me: pabelanger: i don't know that the debian-jessie images are especially burdensome on us are they? granted, it seems silly that we lack debian-stretch images a couple months after release... so i'd be inclined to suggest that we should either have a plan to get stretch working and move jobs to it, or to retire debian images instead | 15:36 |
openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Improved Missing Scope error https://review.openstack.org/507900 | 15:37 |
pabelanger | clarkb: dirk: ya, I would say we can just manually delete it and manually release volume, just to be safe. Rather then having puppet do it | 15:37 |
pabelanger | shouldn't need to delete any volumes at this time | 15:37 |
fungi | odyssey4me: pabelanger: where said plan might entail keeping jessie around for a cycle or two after stretch is working, but at least we should have a plan | 15:37 |
jeblair | fungi, clarkb, mordred, jlk, tobiash: i'm going to tag zuul 2.6.0 from master now. sound ok? | 15:37 |
fungi | jeblair: sounds good, thanks | 15:38 |
odyssey4me | fungi pabelanger sounds sensible, although I'm not privy to what the requirement is from an image point of view | 15:38 |
clarkb | jeblair: yup | 15:38 |
*** andreas_s_ has quit IRC | 15:38 | |
pabelanger | fungi: odyssey4me: Ya, I don't actually know how long jessie gets updates for | 15:39 |
odyssey4me | fungi perhaps pop an email to the ML with the [ansible-hardening] tag to solicit feedback on where to go next... mhayden should pick it up when he can, or perhaps some other consumers of the role will respond with their suggestions | 15:39 |
*** electrofelix has joined #openstack-infra | 15:39 | |
fungi | odyssey4me: pabelanger: likely the bulk of the work is just testing http://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/debian-minimal for stretch, and then fixing anything there which isn't quite working | 15:39 |
*** psachin has quit IRC | 15:40 | |
*** vhosakot has joined #openstack-infra | 15:40 | |
pabelanger | fungi: odyssey4me: It shouldn't be too hard now to updated nodepool-dsvm job to start building stretch, then we just need to confirm our project-config elements work | 15:40 |
jeblair | pushed zuul 2.6.0 | 15:40 |
pabelanger | fungi: odyssey4me: once zuulv3 is migrated, we can loop back | 15:41 |
fungi | yeah, i might even have time to poke at it myself at that point | 15:41 |
*** jascott1 has joined #openstack-infra | 15:42 | |
*** rcernin has quit IRC | 15:43 | |
mordred | Shrews: getOldJob can get you from new name to old name | 15:45 |
*** yamahata has quit IRC | 15:45 | |
Shrews | mordred: yep, found that. though i'm trying a different approach | 15:46 |
*** jascott1 has quit IRC | 15:46 | |
jeblair | mordred: i think the nodesets you used in the migration collided with what we wanted to do in https://review.openstack.org/504259 | 15:48 |
openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/507905 | 15:48 |
jeblair | mordred: i think we need to have the migration use "legacy-devstack-single-node" | 15:48 |
jeblair | mordred: should we just fix that after the migration? | 15:48 |
jeblair | andreaf: once we do that, we can land the nodesets in this change and then you'll have a way to select a node to run tempest on: https://review.openstack.org/504259 | 15:49 |
mordred | jeblair: for all jobs? | 15:50 |
mordred | jeblair: we can just set the nodeset on legacy-base in OZJ | 15:50 |
jeblair | mordred: no just devstack. | 15:50 |
fungi | pabelanger: as to when security support ends for jessie, it depends on how you define it. jessie (now as "oldstable") is supported by the debian security team until probably june (~a year from the stretch release date), at which point the volunteer lts team takes over providing crucial updates planned through april 2020: https://wiki.debian.org/LTS | 15:51 |
jeblair | mordred: devstack-single-node is used by legacy-dsvm-base and legacy-puppet-openstack-integration. so i think i'd just change those 2 places. | 15:51 |
openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/507905 | 15:51 |
mordred | jeblair: we can set that on legacy-dsvm-base and ... yah | 15:51 |
mordred | jeblair: what you said | 15:51 |
jeblair | mordred: assuming that doing so now would complicate the migration script/process, i'll just put than on the post-cutover list | 15:52 |
jeblair | zuul 2.6.0 is on pypi; sending followup email | 15:53 |
*** trown is now known as trown|afk | 15:53 | |
*** erlon has joined #openstack-infra | 15:55 | |
*** xyang1 has joined #openstack-infra | 15:56 | |
clarkb | how does one remove an autohold on a job? I want to cleanup after the debugging last night | 15:56 |
clarkb | set count to 0 maybe? | 15:56 |
*** iyamahat has joined #openstack-infra | 15:56 | |
*** egonzalez has quit IRC | 15:57 | |
*** rossella_s has joined #openstack-infra | 15:57 | |
pabelanger | clarkb: I _think_ it gets auto removed once the jobs are held | 15:57 |
*** iyamahat has quit IRC | 15:58 | |
clarkb | I see, then just delete the nodes on the nodepool side? | 15:58 |
pabelanger | ya, that is what I have been doing. And seems to have work | 15:58 |
*** dizquierdo has quit IRC | 15:59 | |
openstackgerrit | Merged openstack-infra/openstackid master: Improved Missing Scope error https://review.openstack.org/507900 | 16:00 |
*** bswartz has joined #openstack-infra | 16:01 | |
openstackgerrit | Merged openstack-infra/release-tools master: Fix options to allow --warn-exit in eol-branch.sh https://review.openstack.org/505593 | 16:01 |
clarkb | hrm I wonder if the node that got held was already deleted. I don't see it on nl01/02 or nodepool.o.o | 16:02 |
*** yamahata has joined #openstack-infra | 16:02 | |
mordred | jeblair: you can go ahead and change legacy-dsvm-base and legacy-puppet-openstack-integration - we only reference them by parent name | 16:03 |
*** timothyb89 has joined #openstack-infra | 16:03 | |
openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Print status of playbook run in console https://review.openstack.org/507912 | 16:03 |
*** Rockyg has joined #openstack-infra | 16:03 | |
pabelanger | looks like issue with inap | 16:03 |
mgagne | pabelanger: yes | 16:03 |
mordred | jeblair: that said - it's possible we'll have places where a thing gets parent: legacy-dsvm-base but defines a different nodeset | 16:03 |
mgagne | pabelanger: we are investigating right now | 16:03 |
*** sbezverk has quit IRC | 16:03 | |
mordred | jeblair: (like the multinode jobs) | 16:03 |
pabelanger | mgagne: Yay | 16:04 |
*** LindaWang has quit IRC | 16:04 | |
pabelanger | mgagne: also, you must have irc highlight for inap :) | 16:04 |
mordred | jeblair: so we may need to add a nodeset name mapping - which would complicate this patch, so let's do that next | 16:04 |
mgagne | pabelanger: of course | 16:04 |
*** Apoorva has joined #openstack-infra | 16:04 | |
mgagne | pabelanger: but also, I was about to communicate the current issue | 16:05 |
mordred | jeblair: although, if you felt like making an old-node-name to new-legacy-devstack-nodeset-name mapping and sticking it in zuul/mapping.yaml in project-config - I thnk that would likely be the easiest way to add that and to work on it in parallel | 16:05 |
openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/507905 | 16:05 |
openstackgerrit | Merged openstack-infra/release-tools master: Set up gerrit remote manually in eol_branch.sh https://review.openstack.org/505594 | 16:05 |
openstackgerrit | Merged openstack-infra/release-tools master: Check the gerrit remote is as expected, eol_branch https://review.openstack.org/505595 | 16:05 |
mordred | jeblair, pabelanger: https://review.openstack.org/507905 makes me realize we need to update the new project guide to mention adding the project to zuul/main.yaml | 16:05 |
mordred | jeblair, Shrews: I think I have the second crazy version written - testing now | 16:06 |
pabelanger | mordred: good idea | 16:06 |
jeblair | mordred: i'll just wait until we're finished :) | 16:06 |
Shrews | mordred: good. mine still is not working, but i did find we are calling writeProject() twice in writeJobs() for no apparent reason | 16:07 |
mordred | Shrews, jeblair: I predict you're REALLY going to like the globla variables and nested for loops in this patch | 16:09 |
fungi | mordred: what you really need is to liberally sprinkle it with goto | 16:09 |
*** r-daneel has joined #openstack-infra | 16:12 | |
*** hemna_ has quit IRC | 16:15 | |
*** iyamahat has joined #openstack-infra | 16:16 | |
*** jpich has quit IRC | 16:16 | |
mordred | fungi: dont' think I haven't | 16:17 |
mordred | I did just add "This may be the ugliest code I've ever written." to the commit message :) | 16:18 |
*** trown|afk has quit IRC | 16:18 | |
Shrews | mordred: how can i get the short-name used for {name}? | 16:19 |
jeblair | mordred: always pushing yourself to superlatives! that's great! | 16:19 |
Shrews | within writeProject() | 16:19 |
mordred | Shrews: project.split('/')[1] | 16:19 |
mordred | Shrews: project['name'].split('/')[1] | 16:20 |
Shrews | k. thought maybe that may already be done | 16:20 |
mordred | it might be - but I've long since lost track of sanity in this codebase | 16:21 |
*** iyamahat has quit IRC | 16:23 | |
openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add new repo for manila tempest plugin https://review.openstack.org/507905 | 16:24 |
*** tesseract has quit IRC | 16:24 | |
*** rossella_s has quit IRC | 16:24 | |
*** yamahata has quit IRC | 16:30 | |
*** dizquierdo has joined #openstack-infra | 16:30 | |
mgagne | pabelanger: service is now restored | 16:31 |
*** camunoz has quit IRC | 16:31 | |
fungi | mgagne: are you still seeing issues with puppet jobs in ovh-gra1 today? | 16:32 |
*** Hal has joined #openstack-infra | 16:33 | |
*** jascott1 has joined #openstack-infra | 16:33 | |
mgagne | fungi: wrong ping? =) | 16:33 |
*** Hal is now known as Guest72849 | 16:33 | |
pabelanger | mgagne: thanks, checking nodepool now | 16:34 |
*** florianf has quit IRC | 16:34 | |
fungi | mgagne: oh, right, it was mnaser oops | 16:34 |
mnaser | fungi i dont think so! | 16:35 |
*** jascott1 has quit IRC | 16:35 | |
fungi | mnaser: thanks, it probably cleared up on its own in that case | 16:35 |
pabelanger | OpenStackCloudHTTPError: (500) Server Error for url: https://compute.api.mtl01.cloud.iweb.com/v2/addf341f4fa44528b1ac73e2f5c68a21/servers The server has either erred or is incapable of performing the requested operation. | 16:35 |
pabelanger | mgagne: I'm seeing ^now | 16:36 |
mordred | ooh - that's a nice error :) | 16:36 |
*** jascott1 has joined #openstack-infra | 16:36 | |
mordred | I like the use of "erred" | 16:36 |
pabelanger | okay, things happening | 16:36 |
*** jascott1 has quit IRC | 16:36 | |
pabelanger | delete requests were successful | 16:36 |
pabelanger | but still issues launching | 16:37 |
*** jascott1 has joined #openstack-infra | 16:37 | |
mgagne | pabelanger: ok, will check | 16:37 |
pabelanger | danke! | 16:37 |
*** kjackal_ has quit IRC | 16:39 | |
*** david-lyle has quit IRC | 16:40 | |
*** SumitNaiksatam has quit IRC | 16:40 | |
*** rossella_s has joined #openstack-infra | 16:40 | |
*** gouthamr has joined #openstack-infra | 16:43 | |
pabelanger | mgagne: there we go | 16:43 |
pabelanger | building now | 16:44 |
mgagne | pabelanger: :D | 16:44 |
mgagne | now begins the root cause analysis :-/ | 16:44 |
pabelanger | Yah | 16:44 |
*** hemna_ has joined #openstack-infra | 16:45 | |
*** gouthamr_ has joined #openstack-infra | 16:45 | |
mordred | jeblair: you konw what's great? putting in a debug error message about a missing value - but putting it BEFORE the if checking for the missing valueso that it prints every iteration | 16:46 |
jeblair | :) | 16:47 |
*** bobh has joined #openstack-infra | 16:47 | |
*** gouthamr has quit IRC | 16:49 | |
*** derekh has quit IRC | 16:49 | |
*** d0ugal has quit IRC | 16:50 | |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Remove superfluous writeProject() call https://review.openstack.org/507928 | 16:50 |
clarkb | mordred: at least you know it is working? | 16:50 |
mordred | clarkb: :) | 16:50 |
*** e0ne has quit IRC | 16:52 | |
*** xyang1 has quit IRC | 16:52 | |
*** yamamoto has quit IRC | 16:55 | |
*** yamamoto has joined #openstack-infra | 16:59 | |
clarkb | dims: your new mogan tree will require a force push over the top of the existing tree right? eg there is no common ancestor and it will not fast forward? | 17:00 |
dims | yes, it's a replace kind-of-thing | 17:00 |
*** sree has joined #openstack-infra | 17:00 | |
mordred | clarkb, dims: you could make a merge commit merging dims tree into the existing tree - and do the merge with --theirs | 17:00 |
mordred | which should create a fast-forwardable commit with the same results | 17:01 |
dims | mordred : my tree has stuff from mogan, nova and cinder | 17:01 |
clarkb | mordred: ya, but I think they are trying to make the history better reflect the nova heritage of the code, which won't be as obvious with that merge commit | 17:01 |
*** kiennt26 has quit IRC | 17:01 | |
mordred | nod. | 17:01 |
dims | yes, i did a lot of git filtering to make it work | 17:01 |
clarkb | the other question I had was, is there any reason to preserve the old tree? should be make a branch called old-master based off of current master before doing the push? | 17:01 |
dims | clarkb : sure for safety | 17:02 |
*** Apoorva_ has joined #openstack-infra | 17:03 | |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove superfluous writeProject() call https://review.openstack.org/507928 | 17:04 |
*** jpena is now known as jpena|off | 17:05 | |
*** Apoorva has quit IRC | 17:06 | |
*** sree has quit IRC | 17:06 | |
clarkb | old-master is now a branch on mogan (we can change the name if that one is ugly) | 17:06 |
*** dave-mcc_ has quit IRC | 17:06 | |
*** hrubi has quit IRC | 17:07 | |
*** tosky has quit IRC | 17:10 | |
*** dave-mccowan has joined #openstack-infra | 17:11 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Run ansible setup at start of job to verify network https://review.openstack.org/507933 | 17:11 |
clarkb | dims: actually looking at the two git trees I think it might be a fast forward | 17:13 |
jeblair | clarkb, jlk, pabelanger, odyssey4me, mordred: ^ when you have a sec, can you take a look at https://review.openstack.org/507933 ? it addresses the network error we saw yesterday. it's not urgent though. | 17:13 |
dims | clarkb : your git-fu is way stronger than me. so sure :) | 17:13 |
*** david-lyle has joined #openstack-infra | 17:13 | |
*** pcaruana has quit IRC | 17:14 | |
fungi | clarkb: if they started as a fork of nova/cinder/something which inherited the initial commit from nova, even if they later deleted/replaced a bunch of files, then that sounds plausible | 17:14 |
clarkb | dims: there are new commits in your tree on top of 7744129c83839ab36801856f283fb165d71af32e, can you just propose those as normal changes and have them go through code review? You'll need the permission to push merge commits but we should be able to add it | 17:14 |
clarkb | this way normal mogan review process applies which is a nice papertrail for what is happening | 17:14 |
*** Guest72849 has quit IRC | 17:16 | |
tobiash | jeblair: is there a reason to put the setup playbook into zuul instead of the base job? | 17:17 |
pabelanger | jeblair: left comment, but looks good in general | 17:18 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 17:18 |
clarkb | also did a file not get added to the commit? | 17:18 |
mordred | tobiash: it's a network validation check playbook with a different timeout so that it doesn't take ages to fail if there's an issue | 17:19 |
clarkb | or is setup_playbook already a thing? | 17:19 |
mordred | no - this is a whole new thing to address the issue we saw yesterday | 17:20 |
clarkb | right I get that | 17:20 |
clarkb | its juts we make a directory for the playbook but there is no content that I can see | 17:20 |
pabelanger | I think we are creating it for ansible.cfg file | 17:20 |
pabelanger | then just use CLI | 17:21 |
*** trown has joined #openstack-infra | 17:21 | |
clarkb | pabelanger: we also pass the playbook to the cli command | 17:21 |
clarkb | but maybe thats just to honor an internal function api contract and is otherwise unused | 17:21 |
*** rossella_s has quit IRC | 17:22 | |
*** rbrndt has quit IRC | 17:22 | |
pabelanger | clarkb: ya, playbook there is just the folder, which runAnsible uses to setup some things with secrets | 17:23 |
jeblair | yeah, it's just there for the ansible.cfg file; every playbook gets one | 17:24 |
*** kjackal_ has joined #openstack-infra | 17:24 | |
jeblair | but more to the point, in order to get one, we need a "playbook" directory, even though we don't have a playbook. it's a slight api abuse but keeps things very simple | 17:24 |
tobiash | mordred: crazy, I thought ansible would have a per task timeout, but there doesn't seem to exist one | 17:25 |
*** Hal has joined #openstack-infra | 17:25 | |
mordred | tobiash: does 'ansible hostname -m setup' work as expected for windows hosts? | 17:25 |
*** Hal is now known as Guest90591 | 17:26 | |
tobiash | mordred: I have to check that tomorrow | 17:26 |
pabelanger | tobiash: in fact there is gather_timeout (default: 10) but doesn't seem to apply in this case | 17:26 |
tobiash | Hopefully yes | 17:26 |
mordred | tobiash: I would certainly hope so -otherwise fact gathering on windows would be super broken | 17:28 |
mordred | pabelanger, tobiash: I think those timeouts aren't applying becaue this is the ssh transport layer pausing - so I think the place that implements the timeout isn't being reached | 17:29 |
tobiash | mordred: according to the docs, this should work | 17:29 |
*** shardy has quit IRC | 17:29 | |
mordred | pabelanger, tobiash: it might be worthwhile for us to file a bug upstream if we can produce a nice test case | 17:29 |
pabelanger | mordred: agree with transport layer | 17:30 |
pabelanger | adding upstream bug is good too | 17:30 |
*** yamamoto has quit IRC | 17:31 | |
*** yamamoto has joined #openstack-infra | 17:31 | |
clarkb | speaking of iwndows did the idea of using certificate authentication work out? | 17:37 |
clarkb | tobiash: ^ | 17:37 |
tobiash | clarkb: yes, it works and I had already a working hello world job with this | 17:38 |
mordred | \o/ | 17:38 |
clarkb | nice | 17:38 |
mordred | that's exciting | 17:38 |
tobiash | (using hard codes certs for the start) | 17:38 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Run ansible setup at start of job to verify network https://review.openstack.org/507933 | 17:38 |
tobiash | Console log streaming could get a challenge though | 17:39 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Run ansible setup at start of job to verify network https://review.openstack.org/507933 | 17:41 |
tobiash | As the zuul stream module won't work on windows | 17:41 |
dmsimard | clarkb: hey, just wanted to mention I haven't forgotten I'm still on the hook for two specs 1) puppetboard to ara and 2) proactive monitoring | 17:42 |
dmsimard | soon (tm) | 17:42 |
clarkb | dmsimard: no problem, I think everyone is swamped under zuulv3 anyways :) | 17:42 |
tobiash | How is the cutover going? | 17:42 |
SpamapS | Seems like it's getting close | 17:43 |
* SpamapS observing from afar | 17:43 | |
clarkb | tobiash: mordred and shrews are sorting out a files matcher rule problem for project templates aiui, but once that is done we are ready to unleash it to everything | 17:43 |
mordred | yah | 17:44 |
tobiash | \o/ | 17:44 |
Shrews | mordred and I are racing to see who can produce the worst spaghetti code EVA | 17:44 |
*** camunoz has joined #openstack-infra | 17:44 | |
mordred | Shrews: hang on - you think you're a match for my abilities in producing TERRIBLE spaghetti code? | 17:45 |
*** trown has quit IRC | 17:46 | |
mordred | Shrews: you write code that's way too nice and well thought out for that battle | 17:46 |
* tobiash just had spaghetti for dinner :) | 17:46 | |
Shrews | i'm trying. this migration has brought out the worst in my abilities | 17:46 |
*** electrofelix has quit IRC | 17:47 | |
mordred | tobiash: mmm. probably tastier than what I'm producing | 17:47 |
mordred | Shrews: uhoh. I mean - you've worked on MySQL - this could get tricky | 17:47 |
*** rbrndt has joined #openstack-infra | 17:47 | |
Shrews | now where can i place this new THD object.... | 17:48 |
*** Swami has joined #openstack-infra | 17:48 | |
*** e0ne has joined #openstack-infra | 17:49 | |
jeblair | Spaghetti Battle: Whose routine will reign supreme?! | 17:49 |
* jeblair bites into bell pepper | 17:49 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 17:50 |
clarkb | now I need to watch some alton brown | 17:50 |
mordred | jeblair, Shrews, pabelanger, clarkb: ^^ ok - that is now at least running and initial spot-check is promising | 17:50 |
*** dizquierdo has quit IRC | 17:50 | |
mordred | I'm going to push up patches with the results of running it so that we can look at diffs | 17:50 |
*** amoralej is now known as amoralej|off | 17:51 | |
*** sambetts is now known as sambetts|afk | 17:52 | |
mordred | ok. I see one bug -fixing real quick | 17:52 |
*** jtomasek has quit IRC | 17:53 | |
Shrews | mordred: do we need the 2 deepcopy's of expand_project? | 17:54 |
mordred | Shrews: no clue - just seemed safer | 17:54 |
Shrews | lol, spoken like a true spaghetti artist! | 17:55 |
*** jascott1 has quit IRC | 17:56 | |
clarkb | what is with the change to the shell script? nrelated maybe? | 17:56 |
*** jascott1 has joined #openstack-infra | 17:57 | |
*** jascott1 has quit IRC | 17:57 | |
*** jascott1 has joined #openstack-infra | 17:57 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 17:58 |
clarkb | I definitely think a diff of the results would be helpful in understanding what is going on at least for me | 17:58 |
mordred | yah- those are coming | 17:58 |
mordred | clarkb: I just noticed it when running - it was doing the final logic when I was not running it that way | 17:58 |
mordred | clarkb: so - yah - unrelated - could be split | 17:59 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 18:00 |
mordred | ok - there's that patch updated | 18:01 |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 18:01 |
*** raissa_ has quit IRC | 18:01 | |
*** bobh has quit IRC | 18:01 | |
*** raissa_ has joined #openstack-infra | 18:02 | |
clarkb | I just asked my browser to render a 26k line diff | 18:02 |
clarkb | it got mad | 18:02 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update job definitions with per-job filters https://review.openstack.org/507943 | 18:03 |
mordred | infra-root: ^^ that's the other half of the output to diff (it's much smaller) | 18:03 |
mordred | one of the pieces of new logic is one that determines if matchers/variant info apply to all instances of a job, and if so it puts it on the job and does not put it on the entry in teh project pipeline definition | 18:04 |
clarkb | mordred: in 943 I don't see the bindep jobs with new matchers | 18:04 |
clarkb | I guess that implies it is in 179 instead | 18:04 |
mordred | yah - and I don't see it there either - so we may not be all the way home yet | 18:05 |
*** ralonsoh has quit IRC | 18:06 | |
mordred | oh - I thnk I see it | 18:06 |
*** raissa_ has quit IRC | 18:07 | |
Shrews | mordred: you need the shortname for the project i think? | 18:07 |
Shrews | mordred: orig_name = orig_name.format(name=project['name']) | 18:07 |
mordred | Shrews: ++ | 18:07 |
Shrews | i had the same issue using similar | 18:08 |
pabelanger | I'm wondering if the usage of branches: master is actually needed in some cases | 18:08 |
mordred | yah- it doens't put it all the places | 18:08 |
mordred | so those _should_ be places where there is a matcher that's saying "only run this on the master branch" | 18:08 |
pabelanger | added comment to 507943 about branches | 18:08 |
mordred | pabelanger: but would be great to verify that | 18:08 |
pabelanger | kk | 18:09 |
pabelanger | let me look | 18:09 |
*** jkilpatr has quit IRC | 18:10 | |
pabelanger | Ah, okay. So bindep-fallback is special it seems | 18:10 |
pabelanger | which in zuulv3, isn't an issue | 18:11 |
pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml?id=d5e62922ebe92da7eae69792edc185d17e9549ae#n2783 | 18:12 |
pabelanger | our branch filter could be removed now | 18:12 |
pabelanger | branches: ^stable/(newton|ocata).*$ | 18:14 |
pabelanger | mordred: I think that is going to be an issue^ | 18:14 |
pabelanger | does branches support regex in zuulv3? | 18:14 |
Shrews | pabelanger: yes https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.branches | 18:15 |
jeblair | pabelanger: yes it supports regexes | 18:15 |
*** pcaruana has joined #openstack-infra | 18:15 | |
pabelanger | Shrews: thanks! Was just looking for docs | 18:15 |
* jlk looks around wonders how things go | 18:15 | |
jeblair | it's also a list, which is probably a friendlier way to write that in the future | 18:15 |
jeblair | i'm going to go grab lunch; biab. | 18:16 |
*** Sukhdev has quit IRC | 18:17 | |
mordred | jeblair, pabelanger, Shrews: OH - so - there is a good reason why we're not seeing the legacy-bindep-fallback jobs be updated ... | 18:18 |
mordred | and that's because my copy of project-config already has that removed from layout.yaml - lemme fix that real quick | 18:18 |
pabelanger | ah | 18:19 |
Shrews | mordred: i'm also fairly certain you need the project shortname fix, otherwise you'll end up with 'gate-openstack-infra/project-config-bindep...' job names | 18:19 |
mordred | Shrews: oh - yah - I already did that one locally | 18:20 |
Shrews | cool cool cool | 18:20 |
Shrews | just making sure | 18:20 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 18:21 |
mordred | Shrews: ^^ there's the latest - also added sorted() calls to emitting irrelevant-files - as their order is changing from run to run making it hard to see changes in behavior | 18:21 |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Run ansible setup at start of job to verify network https://review.openstack.org/507933 | 18:23 |
*** yamahata has joined #openstack-infra | 18:23 | |
*** raissa_ has joined #openstack-infra | 18:23 | |
*** jkilpatr has joined #openstack-infra | 18:23 | |
clarkb | mordred: don't you need to take [1] of the project name split not [0] ? | 18:24 |
pabelanger | cool, once we stop / start executors we could try bringing sto2 back online | 18:24 |
mordred | clarkb: yes.thank you. | 18:24 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update job definitions with per-job filters https://review.openstack.org/507943 | 18:24 |
mordred | that ^^ is mostly just to baseline with sorted irrelevant-files | 18:26 |
Shrews | sooo... just looking at openstack/governance project as an example, it has a 'files' job matcher but i do not see that applied, and i'm guessing because we automatically remove the 'ubuntu-xenial' portion from the job name | 18:30 |
Shrews | matcher name: ^gate-governance-tox-validate-ubuntu-xenial$ to be specific | 18:31 |
Shrews | this may actually be a different bug | 18:31 |
openstackgerrit | Jesse Keating proposed openstack-infra/openstack-zuul-jobs master: Remove gratuitous use of xargs https://review.openstack.org/507691 | 18:32 |
*** pcrews has quit IRC | 18:32 | |
Shrews | i actually do not see that project in the generated output O.o | 18:32 |
*** pcrews has joined #openstack-infra | 18:33 | |
Shrews | wait... where ARE our legacy projects? | 18:34 |
pabelanger | Shrews: you mean 507179 ? | 18:35 |
Shrews | pabelanger: ah, yes! thx. | 18:36 |
Shrews | ok, ignore me then | 18:36 |
* Shrews might have been staring at this too long | 18:36 | |
mordred | Shrews: that's ok - I just realized that my local fixup of layout.yaml so that I was migrating the right thing still wasn't complete :) | 18:37 |
*** rwsu has quit IRC | 18:38 | |
pabelanger | Hmm, I think I might be a little confused | 18:39 |
pabelanger | mordred: Shrews: looking at legacy-api-guide job in 507943, is it going to be an issue that we duplicate the branch and files in project-template? http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n102 | 18:40 |
pabelanger | or, can those be removed now? | 18:40 |
clarkb | pabelanger: I think if I'm reading it correctly the project-template takes precendence | 18:41 |
clarkb | pabelanger: so should be fine | 18:41 |
clarkb | (just extra verbose) | 18:41 |
EmilienM | hi, we're debugging a Depends-On issue, on https://review.openstack.org/#/c/487488/ - it seems like the patch in Depends-On wasn't seen when deploying the repo http://logs.openstack.org/88/487488/27/check/gate-python-tripleoclient-python27-ubuntu-xenial/35e09ce/console.html#_2017-09-27_10_33_11_179279 - any thoughts ? | 18:42 |
pabelanger | clarkb: right, I'm thinking if these jobs are around for a bit, it will be confusing to update | 18:42 |
clarkb | EmilienM: unittests typically don't have the machinery required to handle depends on like that | 18:43 |
mordred | yes - there is a level of extra verbosity in some of these - but they're not optimized for long-term maint. the intent is that everything with legacy- prefix be deleted | 18:43 |
clarkb | EmilienM: since they are not integration tests | 18:43 |
Shrews | pabelanger: should be fine. we can't expect the migration script to clean up things like that. will have to be a post-migration thing | 18:43 |
mordred | EmilienM, clarkb: that's fixed in v3 | 18:43 |
clarkb | mordred: sort of, tox will still do the wrong thing | 18:44 |
mordred | nope. tox will do the right thing in v3 :) | 18:44 |
clarkb | which is fine imo | 18:44 |
clarkb | wait what? | 18:44 |
mordred | as long as you add the otherproject to required-projects | 18:44 |
clarkb | how, that will break (among other things) contraints | 18:44 |
EmilienM | matbu_: fyi ^ | 18:44 |
EmilienM | mordred: thx for the infos | 18:44 |
matbu_ | clarkb: hmm thx, so we need to merged the depends-on before ? | 18:44 |
matbu_ | mordred: ? | 18:45 |
clarkb | mordred: and tox will rewrite both requirements.txt and constraints? | 18:45 |
mordred | it won't break constraints - things are installed via constraints - then thesystem is checked for sibling repos that have been prepared as part of required-projects, and if they exist, they will be added to the tox venv | 18:45 |
clarkb | mordred: gotcha, that could create unexpected behavior in the other direction (due to bypassing constraints) but at least that unexpected behavior is likely more expected | 18:45 |
mordred | it doesn't rewrite reuqirements - it installs the sibling repo into the tox venv after tox has done normal processing | 18:45 |
smcginnis | Is the zuul upgrade still going on? | 18:46 |
mordred | clarkb: yah - you still have to opt in by having a job that lists the other project in required-projects ... | 18:46 |
mordred | clarkb: but the only thing you need for tox-based cross-testing of multiple repos is to mention the other repo | 18:46 |
mordred | which hopefully will make such things easier for people who want them | 18:46 |
clarkb | matbu_: yes, current tox jobs install what is in your requiremnts files, optionally with constraints applied. This isn't currently compatible with depends-on | 18:47 |
pabelanger | smcginnis: yup, we still have zuulv2.5 gating | 18:47 |
matbu_ | clarkb: ack thx you for the infos :) | 18:47 |
pabelanger | Shrews: okay, wfm | 18:47 |
Shrews | mordred: is another update to 507943 forthcoming? | 18:48 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 18:48 |
clarkb | and 719 and 941 I think | 18:48 |
clarkb | there is 941 | 18:48 |
mordred | clarkb, Shrews: ^^ ok - this is still not fully working - but it's now properly expanding the template - it's now just not applying the matchers and variants to it | 18:48 |
* mordred will push up the other repos to - so as that other people can look | 18:49 | |
smcginnis | pabelanger: OK, thanks. Wasn't sure if I missed something. | 18:49 |
clarkb | smcginnis: we're sorting out the last problems in the migration output (that we are aware of) before making the big switch throw | 18:49 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 18:49 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: DNM Re-add v2 references for project-config https://review.openstack.org/507947 | 18:49 |
jeblair | mordred: for tox siblings -- i'm not entirely sure listing in required-projects is necessary. i think a depends-on (or just being ahead in the gate) will trigger it. | 18:50 |
smcginnis | clarkb: ++ | 18:50 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update job definitions with per-job filters https://review.openstack.org/507943 | 18:51 |
mordred | jeblair: cool for depends-on - just being ahead in the gate might be a little weird and we might want to look at / think about that (once we're done with this) | 18:51 |
*** Rockyg has quit IRC | 18:53 | |
*** sbezverk has joined #openstack-infra | 18:54 | |
*** tosky has joined #openstack-infra | 18:55 | |
pabelanger | jeblair: want to restart zuul-executors to pickup 507933? | 18:55 |
*** dprince has quit IRC | 18:56 | |
mordred | pabelanger, jeblair: if restarting is happening- also restarting zuul-web on zuulv3.o.o to pick up Shrews two patches to it is not a terrible idea either | 18:56 |
jeblair | pabelanger: sure, i can do that in about 15 mins, or feel free to if you want to do it now | 18:57 |
pabelanger | don't mind waiting, give me time to refresh coffee | 18:58 |
openstackgerrit | Merged openstack-infra/irc-meetings master: Reflect TC's office hours in the meetings calendar https://review.openstack.org/507490 | 18:59 |
*** slaweq_ has joined #openstack-infra | 19:00 | |
*** pcaruana has quit IRC | 19:00 | |
*** sree has joined #openstack-infra | 19:01 | |
*** hemna_ has quit IRC | 19:02 | |
*** r-daneel has quit IRC | 19:02 | |
*** r-daneel has joined #openstack-infra | 19:03 | |
*** mrunge_ has joined #openstack-infra | 19:04 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add projects.required to zuul vars https://review.openstack.org/507951 | 19:05 |
jeblair | mordred, clarkb: ^ if we want to narrow the scope of tox siblings | 19:06 |
*** sree has quit IRC | 19:06 | |
mordred | jeblair: ++ I think we do | 19:06 |
*** mrunge has quit IRC | 19:06 | |
jeblair | mordred: i'm going to write the other half of the change real quicklike | 19:06 |
mordred | jeblair: I think adding a job that does required-projects is great - and since it can be done speculatively it's still pretty flexible | 19:06 |
mordred | jeblair: but auto-putting it there just because it shares a queue and is stacked seems ... less idea :) | 19:07 |
mordred | ideal | 19:07 |
dmsimard | infra-root: because I'm sure you have a bunch of free time between the zuul v3 shenanigans, I just wanted to point out that unless mistaken, the multinode stacks are ready to land: https://review.openstack.org/#/q/topic:zuulv3-multinode+(status:open+OR+status:merged) | 19:07 |
*** e0ne has quit IRC | 19:08 | |
Shrews | mordred: ok, had to step away to rest my eyes a bit. do you need help debugging your current iteration of the migration script? | 19:08 |
*** hasharAway is now known as hashar | 19:09 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: sqlreporter: use newrev instead of refs/heads/ https://review.openstack.org/507954 | 19:11 |
*** coolsvap has quit IRC | 19:11 | |
mordred | Shrews: ok - I think I see the latest issue - one sec, I might need help fixing it | 19:11 |
Shrews | ok | 19:11 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Only run tox-siblings on required projects https://review.openstack.org/507955 | 19:13 |
*** vivsoni has quit IRC | 19:14 | |
*** vivsoni has joined #openstack-infra | 19:14 | |
jeblair | i'm restarting the executors now | 19:16 |
*** jcoufal has quit IRC | 19:17 | |
*** dprince has joined #openstack-infra | 19:20 | |
jeblair | and restarted zuul-web | 19:21 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Delay executor noop packets https://review.openstack.org/507961 | 19:26 |
*** bobh has joined #openstack-infra | 19:26 | |
jeblair | based on the number of times ze08 has appeared in our job logs since we started this, i think we're going to want that ^ in before too long. | 19:26 |
*** lukebrowning has joined #openstack-infra | 19:27 | |
mordred | jeblair: ++ | 19:31 |
*** rwsu has joined #openstack-infra | 19:32 | |
mordred | this migration script bug is not fun | 19:32 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add ref_url column to the buildset reporter table https://review.openstack.org/503714 | 19:38 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add oldrev/newrev column to the buildset reporter table https://review.openstack.org/507954 | 19:38 |
*** eharney has quit IRC | 19:44 | |
Shrews | mordred: so... i run your version and i get matchers on the bindep jobs on my shortened example. gonna try on the entire project-config files | 19:46 |
mordred | Shrews: oh yay! | 19:47 |
*** colettecello has quit IRC | 19:47 | |
mordred | Shrews: I do not get the matchers - I just get the bindep jobs expanded | 19:47 |
mordred | Shrews: oh - wait - you get them on the jobs? | 19:47 |
Shrews | mordred: yes | 19:47 |
*** gothicmindfood has joined #openstack-infra | 19:47 | |
mordred | Shrews: I think the specific case now is the following | 19:47 |
mordred | Shrews: job in a project-template applied to two different projects. global matcher that applies to only one of them | 19:48 |
mordred | Shrews: so that the matchers need to be output into the project-pipeline definition rather than the job | 19:48 |
*** markvoelker has quit IRC | 19:48 | |
Shrews | project-pipeline ? | 19:49 |
mordred | sorry - into the project - in writeProject | 19:49 |
*** markvoelker has joined #openstack-infra | 19:49 | |
jeblair | pabelanger, jlk, clarkb: can you look at https://review.openstack.org/507951 and https://review.openstack.org/507961 ? | 19:52 |
jlk | opened | 19:53 |
mordred | Shrews: BOOM | 19:55 |
mordred | Shrews: got it | 19:55 |
Shrews | \o/ | 19:55 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 19:56 |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update job definitions with per-job filters https://review.openstack.org/507943 | 19:56 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 19:57 |
mordred | infra-root: please look at those ^^ as carefully as possible | 19:57 |
*** jascott1 has quit IRC | 19:57 | |
*** jascott1 has joined #openstack-infra | 19:57 | |
mordred | we'll want to remove the DNM patch in the project-config chain before we land it | 19:58 |
jeblair | mordred: what dnm? | 19:59 |
Shrews | interesting. i still see the matchers getting applied to jobs | 20:00 |
mordred | jeblair: there is a patch I shoved in front of https://review.openstack.org/507179 to add entries back to layout.yaml so that the migration script would pick them up to generate the jobs | 20:00 |
mordred | Shrews: they'll get applied to jobs if all projects that use that job have the same matchers | 20:00 |
Shrews | oh, now i get it | 20:01 |
mordred | btw - legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-container-upgrades is fun to look at | 20:01 |
mordred | it has both files and irrelevant files :) | 20:01 |
*** jascott1 has quit IRC | 20:02 | |
mordred | also - it apparently publishes artifacts as we have based it on the publish-openstack-artifacts base job | 20:02 |
jeblair | i wonder if anyone knew what caused that job to run. :) | 20:02 |
mordred | jeblair: I think we might need a legacy-publish-openstack-artifacts to use for the things that publish artifacts - that does the nodepool dir writing | 20:03 |
jeblair | mordred: wow, it uses that too? | 20:04 |
*** mat128 has quit IRC | 20:04 | |
mordred | yah | 20:04 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route https://review.openstack.org/503268 | 20:04 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 20:04 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 20:04 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 20:04 |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add projects.required to zuul vars https://review.openstack.org/507951 | 20:05 |
*** jascott1 has joined #openstack-infra | 20:05 | |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add artifact publication job for legacy jobs https://review.openstack.org/507971 | 20:06 |
mordred | jeblair: ^^ | 20:06 |
*** yamamoto has quit IRC | 20:08 | |
jeblair | mordred: apparently puppet-dashboard needs to be added to main.yaml | 20:09 |
Shrews | mordred: i hope we never have to return to this code in the future | 20:09 |
mordred | jeblair: oh - actually - it can just be removed - it's a retired project it just still has entries in layout.yaml - I cleaned that up earlire | 20:09 |
mordred | jeblair: it's the only one | 20:09 |
*** jascott1 has quit IRC | 20:10 | |
fungi | 507943 really shows how much we were missing because of that, i guess | 20:11 |
*** jkilpatr has quit IRC | 20:14 | |
*** r-daneel has quit IRC | 20:14 | |
*** r-daneel has joined #openstack-infra | 20:15 | |
pabelanger | back now, catching up | 20:16 |
jeblair | mordred: are you working on a revision to 179 without dashboard? | 20:16 |
pabelanger | 507961 got -1 from zuul | 20:17 |
Shrews | mordred: overall concept seems sound to me, and the new code is doing the things i'd expect you'd have to do. | 20:17 |
*** dhajare has quit IRC | 20:17 | |
Shrews | i say we run with it | 20:17 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Delay executor noop packets https://review.openstack.org/507961 | 20:18 |
mordred | jeblair: yes | 20:18 |
Shrews | i did not +3 in case others want to examine the new migrate.py | 20:18 |
jeblair | pabelanger: thx | 20:18 |
openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Removed confusing comment https://review.openstack.org/507973 | 20:21 |
openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate master: Remove other references to DEVSTACK_GATE_TEMPEST_STRESS https://review.openstack.org/507974 | 20:21 |
fungi | i clearly didn't read the entirety of the converted projects set, but it looks sane and nothing in the script jumped out at me as particularly buggish | 20:22 |
jeblair | no "#BUG: ..." comments at least | 20:22 |
fungi | hah | 20:25 |
Shrews | well, working on a alternative solution in parallel, i saw a lot of similarities in things i needed to do (although my solution hit a rather rude dead end) | 20:26 |
mordred | fungi, pabelanger: could I bother you for a +3 on https://review.openstack.org/#/c/507971/ real quick? | 20:26 |
clarkb | I'm lunching and picking up csa box back to computer in half hour ornso | 20:26 |
mordred | clarkb: by then we'll likely be ready to do things | 20:26 |
*** yamamoto has joined #openstack-infra | 20:27 | |
openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Update parents of converted artifact jobs https://review.openstack.org/507975 | 20:27 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 20:27 |
mordred | ok. that discards the DNM patch and also deals with puppet-dashboard | 20:28 |
*** yamamoto has quit IRC | 20:31 | |
*** gouthamr_ has quit IRC | 20:32 | |
*** jdandrea_ has quit IRC | 20:32 | |
*** jpena|off has quit IRC | 20:33 | |
*** ltomasbo has quit IRC | 20:34 | |
*** gouthamr has joined #openstack-infra | 20:35 | |
*** amoralej|off has quit IRC | 20:35 | |
jlk | suppose I should do the food in the face thing. | 20:35 |
pabelanger | Ya, I'll be stepping away from computer in about 45mins to do the same | 20:36 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add support for result data in child jobs https://review.openstack.org/504808 | 20:36 |
*** ltomasbo has joined #openstack-infra | 20:37 | |
*** jpena|off has joined #openstack-infra | 20:38 | |
*** amoralej has joined #openstack-infra | 20:38 | |
pabelanger | +2 on 507179 | 20:38 |
pabelanger | bindep-fallback looks correct | 20:38 |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add artifact publication job for legacy jobs https://review.openstack.org/507971 | 20:39 |
*** gouthamr has quit IRC | 20:39 | |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Add post step to Tempest base job https://review.openstack.org/506835 | 20:40 |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results. https://review.openstack.org/507980 | 20:40 |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 20:41 |
mordred | pabelanger: \o/ | 20:42 |
*** gouthamr has joined #openstack-infra | 20:42 | |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Delay executor noop packets https://review.openstack.org/507961 | 20:43 |
Shrews | anyone else want to look at https://review.openstack.org/507943 or shall i merge it? | 20:43 |
jeblair | Shrews: go for it | 20:44 |
*** gouthamr has quit IRC | 20:45 | |
mordred | ++ | 20:45 |
jeblair | i think we're waiting on 943 and 975 to merge, then i believe we can throw the switch (179). is that correct? | 20:46 |
mordred | jeblair: yes | 20:46 |
mordred | jeblair: shall we go ahead and merge https://review.openstack.org/#/c/507941/ too? | 20:46 |
Shrews | +A'd 941 | 20:46 |
mordred | Shrews: ++ | 20:46 |
jeblair | ah missed that one, thx | 20:47 |
jeblair | maybe on purpose. ;) | 20:47 |
mordred | jeblair: don't EVER LOOK INSIDE THAT FILE | 20:47 |
Shrews | srsly | 20:47 |
*** esberglu has quit IRC | 20:47 | |
mordred | if there are any additional issues, it will in fact be easier to just deal with them as bug fixes to the new jobs :) | 20:48 |
*** esberglu has joined #openstack-infra | 20:48 | |
jeblair | i'm going to upgrade the executors and, once those changes land, restart them. i'd like to do that before the big switch throwing. | 20:48 |
jeblair | mostly for the load balancing change. | 20:48 |
mordred | jeblair: ++ | 20:48 |
mordred | jeblair: yah - I'd lke to use more executors than just 01 and 08 | 20:48 |
mordred | jeblair: should we squash https://review.openstack.org/#/c/507275 onto the shift nodepool quota patch? | 20:49 |
jeblair | mordred: well, what we should really do is add tripleo in one patch, then update the shift quota patch to include it | 20:50 |
jeblair | otherwise, reverts get confusing | 20:50 |
mordred | nod | 20:50 |
* mordred fixes | 20:50 | |
*** rwsu has quit IRC | 20:52 | |
*** esberglu has quit IRC | 20:52 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update job definitions with per-job filters https://review.openstack.org/507943 | 20:53 |
clarkb | ok back to computer | 20:54 |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update parents of converted artifact jobs https://review.openstack.org/507975 | 20:54 |
*** wolverineav has quit IRC | 20:54 | |
*** wolverineav has joined #openstack-infra | 20:55 | |
jeblair | oh we need a scheduler restart too to fix the 'dump all the file content to log' thing. | 20:55 |
*** kgiusti has left #openstack-infra | 20:55 | |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add an additional pass through project templates https://review.openstack.org/507941 | 20:58 |
jeblair | andreaf: regarding 507980, we should have a subunit->html role in zuul-jobs | 20:58 |
jeblair | restarting executors now, then scheduler. | 20:59 |
*** wolverineav has quit IRC | 20:59 | |
andreaf | jeblair: yeah we do but includes also the archive and copy part which I'd rather do with the save-file role | 20:59 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add tripleo-test-cloud-rh1 into nl01 https://review.openstack.org/507275 | 21:00 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 21:00 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add OVH to nodepool v3 https://review.openstack.org/507984 | 21:00 |
mordred | jeblair, clarkb: ok - reworked that into a stack - the first 2 can land right now | 21:00 |
*** e0ne has joined #openstack-infra | 21:00 | |
mordred | andreaf: we're going to have eversomuch fun constructing job bits aren't we? | 21:01 |
*** iyamahat has joined #openstack-infra | 21:01 | |
jeblair | andreaf: what does save-file do differently? | 21:02 |
clarkb | mordred: adding the two clouds (rh1 and ovh) can mrege now ? | 21:02 |
*** sree has joined #openstack-infra | 21:02 | |
jeblair | zuulv3 is restarting | 21:02 |
andreaf | jeblair: I'm tyring to make general purpose roles so that they can be re-used easily | 21:03 |
andreaf | jeblair: the zuul-jobs role assumes that the user has read/write access in the folder where the test results are stored | 21:04 |
*** akscram1 has quit IRC | 21:04 | |
*** e0ne has quit IRC | 21:04 | |
*** eharney has joined #openstack-infra | 21:04 | |
andreaf | the way I'm doing it is to use a stage folder where I generate archive files to be copied (it can be /tmp) | 21:05 |
*** e0ne has joined #openstack-infra | 21:05 | |
jeblair | andreaf: right, i'm supportive of that. :) the roles in zuul-jobs are intended to be reusable too, so i wondered if it's something we can improve there | 21:05 |
*** akscram1 has joined #openstack-infra | 21:05 | |
* mordred is excited for andreaf to start hacking on things in zuul-jobs too :) | 21:06 | |
andreaf | jeblair: sure that's the kind of feedback I was hoping to get :) | 21:06 |
jeblair | \o/ | 21:07 |
*** sree has quit IRC | 21:07 | |
jeblair | zuulv3 is up again | 21:08 |
*** esberglu has joined #openstack-infra | 21:08 | |
jeblair | mordred: you may want to recheck your changes | 21:08 |
mordred | jeblair: https://review.openstack.org/#/c/507984 if you have a sec | 21:08 |
*** bobh has quit IRC | 21:08 | |
mordred | jeblair: kk | 21:08 |
*** thorst has quit IRC | 21:09 | |
mordred | jeblair: https://review.openstack.org/#/c/507275/ is +3d but not showing up in status - am I missing something? | 21:09 |
jeblair | mordred: yeah, no +1 vote; needs a recheck to go through check pipeline | 21:09 |
mordred | ah | 21:09 |
mordred | cool | 21:10 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 21:10 |
*** thorst has joined #openstack-infra | 21:11 | |
*** rwsu has joined #openstack-infra | 21:12 | |
mordred | infra-root: I've updated the etherpad list of steps - and split the rollback section into hard and soft (soft meaning rollback to today's state - hard meaning remove all the projects from v3) | 21:12 |
*** LeoBud has joined #openstack-infra | 21:13 | |
*** bnemec has quit IRC | 21:14 | |
*** rossella_s has joined #openstack-infra | 21:14 | |
*** thorst has quit IRC | 21:15 | |
*** jpena|off has quit IRC | 21:16 | |
*** ltomasbo has quit IRC | 21:16 | |
*** amoralej has quit IRC | 21:16 | |
mordred | jeblair: so - the noop jobs patch: https://review.openstack.org/#/c/507860/ - I'm worried that once we flip the switch we'll stop being able to land patches to project-config because of the various job exclusions - how about I grab that patch and also add the legacy-bindep-fallback changes to it and we land that before we flip the switch so that switch-flipping doesn't impact project-config's | 21:16 |
mordred | operation | 21:17 |
jeblair | mordred: why bring bindep into it? why not just merge noop? | 21:17 |
*** ltomasbo has joined #openstack-infra | 21:18 | |
*** jpena|off has joined #openstack-infra | 21:18 | |
jeblair | trying to avoid churn on the big patches | 21:18 |
*** amoralej has joined #openstack-infra | 21:18 | |
*** LeoBud has quit IRC | 21:20 | |
mordred | jeblair: mostly was thinking about rollback - but kk | 21:22 |
clarkb | so I understand the noop jobs patch we need that so that we run something if nothing else were to run? | 21:22 |
jeblair | mordred: i think we can leave noop in even after rollback | 21:23 |
clarkb | does this still have the problem of putting everything with noop jobs into a common gate pipeline? | 21:23 |
openstackgerrit | Merged openstack-infra/project-config master: Add tripleo-test-cloud-rh1 into nl01 https://review.openstack.org/507275 | 21:23 |
jeblair | clarkb: no it doesn't have that problem, but we should still clean up our jobs and remove it because it's bad form and silly. | 21:23 |
mordred | jeblair: yah - I meant getting bindep-fallback jobs out so that we're not running 6 un-needed jobs if we do rollback - but yes- let's not churn the big patch | 21:23 |
jeblair | clarkb: we're mostly entertaining it here because so much stuff is in flux. | 21:24 |
mordred | ++ | 21:24 |
clarkb | gotcha | 21:24 |
*** e0ne has quit IRC | 21:24 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add noop-jobs as a fallback for project-config https://review.openstack.org/507860 | 21:24 |
*** aeng has joined #openstack-infra | 21:24 | |
jeblair | publish-openstack-specs-site failed: http://logs.openstack.org/7c/62c3f92e19e7282959de85e75bcb84166562597c/post/publish-openstack-specs-site/aa7b784/job-output.txt.gz | 21:28 |
*** yamamoto has joined #openstack-infra | 21:28 | |
clarkb | I think I saw a patch for that problem yesterday | 21:29 |
*** thorst has joined #openstack-infra | 21:29 | |
clarkb | we don't seem to log the tox command though? | 21:29 |
openstackgerrit | Merged openstack-infra/project-config master: Add OVH to nodepool v3 https://review.openstack.org/507984 | 21:30 |
jeblair | clarkb: :( it's in the .json file though -- tox -evenv -vv python setup.py build_sphinx | 21:30 |
*** srobert_ has joined #openstack-infra | 21:31 | |
mordred | jeblair: that should have been fixed by https://review.openstack.org/#/c/507678/ | 21:31 |
clarkb | http://git.openstack.org/cgit/openstack/openstack-specs/tree/setup.py the file is there though | 21:32 |
*** slaweq_ has quit IRC | 21:32 | |
jeblair | mordred: it's not a siblings file issue | 21:32 |
mordred | clarkb: this isn't for openstack-specs | 21:32 |
mordred | the issue now will be that that job uses openstack-tox-docs as a base job - one sec | 21:32 |
jeblair | ya that sounds more like it; i imagine this job needs to be specialer | 21:33 |
mordred | yu[ | 21:33 |
*** ihrachys has quit IRC | 21:33 | |
*** thorst has quit IRC | 21:34 | |
jeblair | actually, it's not parentage (its parent is base); it just re-uses playbooks from python-docs | 21:34 |
*** yamamoto has quit IRC | 21:34 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Don't use the docs cantrip for specs-site https://review.openstack.org/507989 | 21:35 |
mordred | jeblair: yes. ^^ | 21:35 |
*** srobert has quit IRC | 21:35 | |
mordred | jeblair: the main thign is that it certainly shouldn't be running python setup.py build_sphinx | 21:35 |
*** srobert_ has quit IRC | 21:36 | |
*** yamahata has quit IRC | 21:36 | |
*** yamahata has joined #openstack-infra | 21:37 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Emit a message about the final tox invocation https://review.openstack.org/507990 | 21:37 |
mordred | jeblair, clarkb: that should help debugging issues with tox jobs without having to go to the json | 21:37 |
jeblair | mordred: why don't we have zuul_stream emit the command? | 21:38 |
mordred | jeblair: because it's VERY chatty if we do it for every command | 21:39 |
clarkb | is it worse than bash set -x is/was | 21:39 |
*** andreww has quit IRC | 21:39 | |
mordred | jeblair: we could also remove the name: from that task though - which should cause it to emit the command since there is no name | 21:39 |
openstackgerrit | Merged openstack-infra/project-config master: Add noop-jobs as a fallback for project-config https://review.openstack.org/507860 | 21:40 |
clarkb | mordred: I both like that idea adn not like it, the nice thing about named task is their grepability | 21:40 |
clarkb | but that is a simple fix for getting logging behavior we do want | 21:40 |
mordred | clarkb: yah | 21:40 |
jeblair | mordred: i'd like to spend some more time on this once you refactor zuul_stream and add testing | 21:41 |
mordred | agre | 21:41 |
*** xarses has joined #openstack-infra | 21:42 | |
*** dprince has quit IRC | 21:42 | |
jeblair | because i feel like "this is a command i ran" is something i almost always want to know, even more than many things which are currently in the streaming log. so if it's too chatty, i'd like to be able to explore how to get what we need out of it. as it is, i almost always have to go to json to debug, and i don't think adding ansible debug lines in strategic places is scalable. | 21:42 |
*** jascott1 has joined #openstack-infra | 21:43 | |
*** dizquierdo has joined #openstack-infra | 21:43 | |
jeblair | mordred: i think we're no longer waiting on anything, is that right? | 21:45 |
clarkb | I was just going to ask the same thing :) | 21:45 |
*** dave-mccowan has quit IRC | 21:45 | |
mordred | jeblair: correct | 21:45 |
mordred | I believe we're ready to rock and roll | 21:45 |
jeblair | mordred: and if my clock is right -- you're ready to eod, so should we pick this up tomorrow? :) | 21:46 |
mordred | jeblair: we keep saying that :) | 21:46 |
mordred | jeblair: have I mentioned how much I did not enjoy today's migration script bug? | 21:46 |
* Shrews echos mordred's disdain for said bug | 21:47 | |
mordred | jeblair: but yes - I need to EOD in 45 minutes (we're taking scuba classes and they do not enjoy it when we're late) | 21:47 |
mordred | jeblair: fwiw, I have no evening things tomorrow or friday - so as soon as we don't need me to be able to squeeze an extra couple of hours in, I'll be free to do so | 21:47 |
*** wolverineav has joined #openstack-infra | 21:47 | |
*** kjackal_ has quit IRC | 21:48 | |
mordred | that said - we can probably judge immediate rollback need in 45 minutes | 21:48 |
clarkb | I'm good with holding off too, I expect we'll be fielding lots of questions and working to iron out bugs after the switch and well having a good chunk of the day to that would be nice | 21:48 |
clarkb | as well as extra eyeballs to untangle things | 21:49 |
mordred | yah | 21:49 |
jeblair | i agree with clarkb | 21:49 |
openstackgerrit | Merged openstack-infra/project-config master: Don't use the docs cantrip for specs-site https://review.openstack.org/507989 | 21:49 |
*** jascott1 has quit IRC | 21:51 | |
*** andreww has joined #openstack-infra | 21:51 | |
mordred | kk. I shall write email update | 21:51 |
*** xarses has quit IRC | 21:52 | |
*** jascott1 has joined #openstack-infra | 21:53 | |
openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 21:53 |
mordred | infra-root: https://etherpad.openstack.org/p/OmUqeXyu4R - that look ok for an update? | 21:54 |
*** baoli has quit IRC | 21:54 | |
jeblair | mordred: lgtm | 21:54 |
jeblair | the specs site job got farther -- http://logs.openstack.org/34/9480d876595193ee72ca3a1935213e261d42aa34/post/publish-openstack-specs-site/c1a5cbf/job-output.txt.gz#_2017-09-27_21_52_28_504954 | 21:55 |
clarkb | I'll even try to wake early so that west coast is ready early :) | 21:55 |
clarkb | mordred: so message lgtm | 21:55 |
jeblair | looks like we need to collect the index from a different directory | 21:56 |
*** jkilpatr has joined #openstack-infra | 21:56 | |
clarkb | outdir = os.path.join(template_path, 'output') | 21:57 |
jeblair | yah, and: 2017-09-27 21:52:21.307878 | ubuntu-xenial | Writing '/home/zuul/src/git.openstack.org/openstack-infra/project-config/specs/output/index.html' | 21:57 |
*** jascott1 has quit IRC | 21:57 | |
*** yamamoto has joined #openstack-infra | 21:57 | |
clarkb | specs/output? | 21:57 |
*** mriedem is now known as mriedem_away | 21:57 | |
*** yamamoto has quit IRC | 21:57 | |
*** dave-mccowan has joined #openstack-infra | 21:57 | |
*** thorst has joined #openstack-infra | 21:58 | |
*** kjackal_ has joined #openstack-infra | 21:58 | |
mordred | yah - need to tell it a diffrent src location ... patch coming | 21:58 |
*** dizquierdo has quit IRC | 21:59 | |
*** hashar has quit IRC | 22:00 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Grab spec site content from where it actually comes https://review.openstack.org/507996 | 22:01 |
mordred | clarkb, jeblair: ^^ | 22:01 |
jeblair | mordred: sure about no trailing / ? | 22:02 |
*** inc0 has quit IRC | 22:02 | |
*** slaweq_ has joined #openstack-infra | 22:03 | |
*** andreww has quit IRC | 22:03 | |
*** dave-mccowan has quit IRC | 22:03 | |
mordred | jeblair: nope! | 22:03 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Grab spec site content from where it actually comes https://review.openstack.org/507996 | 22:03 |
jeblair | oh weird, that's actually not the default for that role | 22:04 |
jeblair | the default has no trailing / | 22:04 |
*** xarses has joined #openstack-infra | 22:04 | |
*** kjackal_ has quit IRC | 22:05 | |
jeblair | maybe that's because by default, the role is being used for docs-draft, so ending up in a subdir is desired... | 22:05 |
jeblair | so actually thinking this one through -- i'm guessing that we *do* want a trailing / and ps2 is good. | 22:06 |
jeblair | that's where i'm putting my nickel. | 22:06 |
mordred | jeblair: \o/ | 22:07 |
*** rwsu has quit IRC | 22:07 | |
clarkb | Im gonna have to read the manpage | 22:08 |
*** Goneri has quit IRC | 22:08 | |
clarkb | ya trailing / is right | 22:08 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Expand docs for fetch-sphinx-output role https://review.openstack.org/507999 | 22:08 |
jeblair | clarkb, mordred: ^ missing docs (ironic!) | 22:09 |
*** andreww has joined #openstack-infra | 22:09 | |
*** xarses has quit IRC | 22:09 | |
*** rossella_s has quit IRC | 22:10 | |
openstackgerrit | Merged openstack-infra/project-config master: Grab spec site content from where it actually comes https://review.openstack.org/507996 | 22:17 |
pabelanger | Hmm, I think nl01 is not happy | 22:18 |
pabelanger | KeyError: 'tripleo-centos-7' | 22:19 |
pabelanger | on nodepool list | 22:19 |
openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add missing label for tripleo-centos-7 https://review.openstack.org/508004 | 22:20 |
pabelanger | and fix | 22:20 |
pabelanger | os_client_config.exceptions.OpenStackConfigException: Region BHA1 is not a valid region name for cloud ovh. Valid choices are BHS1,GRA1. Please note that region names are case sensitive. | 22:21 |
pabelanger | getting that on nl02 | 22:21 |
fungi | okay, finished with dinner and catching up | 22:22 |
openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix typo with ovh-bhs1 https://review.openstack.org/508005 | 22:23 |
pabelanger | and nl02 fix | 22:23 |
*** rossella_s has joined #openstack-infra | 22:23 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Only run tox-siblings on required projects https://review.openstack.org/507955 | 22:23 |
*** rlandy is now known as rlandy|afk | 22:26 | |
pabelanger | I've added 508004 and 508005 to blockers section on zuulv3-cutover etherpad | 22:26 |
jeblair | mordred, clarkb: specs site run looks good and specs.o.o has a recent update time now | 22:27 |
jeblair | so! next failure: http://logs.openstack.org/97/4102e7b6abcd6d5f5245e11597e0c5693111ab97/post/publish-openstack-python-docs-infra-index/b9db5a6/job-output.txt.gz | 22:27 |
jeblair | pabelanger: +2+2 | 22:28 |
fungi | reviewing the nodepool config fixes since those seem urgent | 22:28 |
jeblair | looks like basically the same error | 22:29 |
fungi | also, agreed on this being a good point to pause and then try to throw the freakishly big dr. frankenstein's lab like knife switch tomorrow and cackle insanely | 22:30 |
*** krtaylor has quit IRC | 22:30 | |
*** inc0 has joined #openstack-infra | 22:30 | |
pabelanger | tox -evenv python setup.py build_sphinx also doen't work for me locally on project-config | 22:31 |
pabelanger | looks like it should be: tox -einfra-docs | 22:32 |
jeblair | pabelanger: yeah, i'm working on a patch | 22:33 |
pabelanger | jeblair: kk | 22:34 |
*** slaweq_ has quit IRC | 22:36 | |
*** iyamahat has quit IRC | 22:39 | |
*** slaweq_ has joined #openstack-infra | 22:39 | |
*** ari[m] has joined #openstack-infra | 22:40 | |
openstackgerrit | Merged openstack-infra/project-config master: Add missing label for tripleo-centos-7 https://review.openstack.org/508004 | 22:40 |
*** krtaylor has joined #openstack-infra | 22:43 | |
*** jaypipes has quit IRC | 22:48 | |
*** abelur_ has quit IRC | 22:49 | |
*** abelur_ has joined #openstack-infra | 22:53 | |
*** rossella_s has quit IRC | 22:56 | |
*** bobh has joined #openstack-infra | 22:57 | |
*** yamamoto has joined #openstack-infra | 22:58 | |
openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add a role to write a root marker https://review.openstack.org/508010 | 23:00 |
openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Fix publish infra docs index job https://review.openstack.org/508011 | 23:00 |
jeblair | yowza that wasn't easy. | 23:00 |
jeblair | pabelanger, clarkb: ^ | 23:00 |
*** threestrands has joined #openstack-infra | 23:00 | |
*** threestrands has quit IRC | 23:00 | |
*** threestrands has joined #openstack-infra | 23:00 | |
*** rtjure has quit IRC | 23:01 | |
*** bobh has quit IRC | 23:02 | |
*** claudiub|3 has quit IRC | 23:03 | |
*** andreww has quit IRC | 23:04 | |
*** rbrndt has quit IRC | 23:05 | |
*** yamamoto has quit IRC | 23:05 | |
*** lukebrowning has quit IRC | 23:08 | |
*** slaweq_ has quit IRC | 23:13 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove gratuitous use of xargs https://review.openstack.org/507691 | 23:16 |
*** baoli has joined #openstack-infra | 23:18 | |
*** slaweq_ has joined #openstack-infra | 23:18 | |
*** baoli_ has joined #openstack-infra | 23:22 | |
*** baoli has quit IRC | 23:22 | |
*** lukebrowning has joined #openstack-infra | 23:23 | |
*** hongbin has quit IRC | 23:25 | |
*** lukebrowning has quit IRC | 23:28 | |
*** felipemonteiro_ has quit IRC | 23:29 | |
*** lukebrowning has joined #openstack-infra | 23:29 | |
*** thorst has quit IRC | 23:30 | |
*** mriedem_away is now known as mriedem | 23:31 | |
*** Sukhdev has joined #openstack-infra | 23:32 | |
*** lukebrowning has quit IRC | 23:34 | |
*** stakeda has joined #openstack-infra | 23:34 | |
*** krtaylor has quit IRC | 23:35 | |
*** lukebrowning has joined #openstack-infra | 23:36 | |
*** srobert has joined #openstack-infra | 23:39 | |
*** lukebrowning has quit IRC | 23:40 | |
*** dizquierdo has joined #openstack-infra | 23:41 | |
*** lukebrowning has joined #openstack-infra | 23:42 | |
*** ijw has joined #openstack-infra | 23:43 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add a role to write a root marker https://review.openstack.org/508010 | 23:44 |
clarkb | jeblair: ^ | 23:45 |
*** ijw has quit IRC | 23:45 | |
*** ijw has joined #openstack-infra | 23:45 | |
*** tpsilva has quit IRC | 23:46 | |
*** lukebrowning has quit IRC | 23:47 | |
*** slaweq_ has quit IRC | 23:48 | |
*** lukebrowning has joined #openstack-infra | 23:48 | |
*** krtaylor has joined #openstack-infra | 23:51 | |
*** krtaylor has quit IRC | 23:51 | |
*** slaweq_ has joined #openstack-infra | 23:52 | |
*** bnemec has joined #openstack-infra | 23:52 | |
*** lukebrowning has quit IRC | 23:53 | |
*** lukebrowning has joined #openstack-infra | 23:54 | |
*** iyamahat has joined #openstack-infra | 23:56 | |
*** xarses has joined #openstack-infra | 23:58 | |
*** r-daneel has quit IRC | 23:58 | |
*** yamahata has quit IRC | 23:58 | |
*** lukebrowning has quit IRC | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!