| jeblair | ianw: okay, i just commented out the vos release in a local copy of the script | 00:00 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: unbound: Add check for undefined default_ipv6 https://review.openstack.org/511653 | 00:00 |
| ianw | pabelanger / clarkb: ^ i feel like this is bug https://github.com/ansible/ansible/issues/23675 but i guess it's clear enough | 00:00 |
| ianw | jeblair: ++, fingers crossed | 00:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Bump reprepro timeout up to 90m https://review.openstack.org/511654 | 00:01 |
| pabelanger | jeblair: clarkb: ianw: ^reprepro timeout bump | 00:01 |
| ianw | pabelanger: is the cron update an hour though? | 00:01 |
| fungi | clarkb: find will spend its time iterating through directories too. i expect the shell glob to be an up-front cost vs find traversing the entire tree. in the end maybe they're similarly efficient | 00:01 |
| jeblair | ianw: 2h -- 0 */2 * * * | 00:01 |
| ianw | oh, that's reasonable then | 00:02 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Reparent ozj integration jobs to base-test for testing https://review.openstack.org/511646 | 00:02 |
| clarkb | ianw: I rebased ^ on your change which should end up testing it | 00:02 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Stop it with the moving then copying of cache files https://review.openstack.org/511260 | 00:02 |
| mnaser | ^^^ yay | 00:02 |
| ianw | clarkb: thanks | 00:02 |
| jeblair | we're past the update and on to 'checking state of mirror' | 00:02 |
| fungi | clarkb: basically i avoided find because we could provide an exact set of globs to filter vs find having to look inside every directory for a potential match | 00:03 |
| clarkb | fungi: ya | 00:03 |
| pabelanger | ianw: why the fact? and not just use both statements in the task below? | 00:03 |
| ianw | pabelanger: avoid duplicating it, thought it was a bit clearer | 00:03 |
| jeblair | 'checking state of mirror' takes about 15-20 mins. | 00:04 |
| pabelanger | ianw: you could make that a block too | 00:04 |
| fungi | clarkb: at least in my experience, with very wide and deep file trees, a specific shell glob will win over a subdirectory match with find | 00:04 |
| jeblair | clarkb: you want to do one mirror or all of em? | 00:04 |
| pabelanger | ianw: +3, we can refactor later | 00:04 |
| clarkb | jeblair: we probably want to do all of them? Since any of them could still be broken? | 00:04 |
| jeblair | clarkb: sorry, i meant switch one mirror host to read-write or all mirror hosts | 00:05 |
| jeblair | Missing file pool/main/a/apparmor/apparmor_2.10.95-0ubuntu2.6_amd64.deb | 00:05 |
| jeblair | Files are missing for 'apparmor'! | 00:05 |
| jeblair | There have been errors! | 00:05 |
| clarkb | jeblair: oh we probably want to flip all of them to get data back quicker | 00:05 |
| jeblair | ianw: how about i perform checksum.db removal on that file then re-run? | 00:06 |
| clarkb | but can also flip in the biggest region too and probably get data back pretty quick | 00:06 |
| *** sbezverk has quit IRC | 00:06 | |
| ianw | jeblair: ok ... odd that it didn't happen before? | 00:06 |
| fungi | i know i shouldn't jinx us, but here's hoping we don't have files spontaneously disappearing out of afs | 00:08 |
| jeblair | the script runs deleteifunereferenced | 00:08 |
| jeblair | i suspect we just deleted it ourselves | 00:08 |
| fungi | yeah, that seems far more likely | 00:08 |
| ianw | but only after the "update" phase? | 00:08 |
| clarkb | ianw: pabelanger http://logs.openstack.org/46/511646/2/infra-check/base-integration-centos-7/4d11e4f/job-output.txt.gz#_2017-10-13_00_05_07_944763 will we not test that premerge? | 00:08 |
| fungi | in other news, we're just about to hit the 2% free inodes mark | 00:09 |
| clarkb | ok I've got to take the kids for a walk. Back later to check on the ara on failure stuff | 00:09 |
| ianw | clarkb: do you mean unworkflow it? i'm ok with that | 00:10 |
| jeblair | ianw, fungi: that file is not in my dump of checksums.list from earlier | 00:11 |
| fungi | strange | 00:11 |
| jeblair | ianw, fungi: so that means that it hasn't been in the filesystem for a while | 00:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Set v3 nodepool citycloud timeout to 600 https://review.openstack.org/511656 | 00:11 |
| jeblair | (we didn't just delete it, and it didn't disappear) | 00:11 |
| fungi | yeah, extra strange | 00:11 |
| pabelanger | jeblair: fungi: clarkb: added launch-timeout to citycloud also. ^ See to be having issues building nodes, but when working usually 30 second boots | 00:11 |
| jeblair | i don't know why the check thinks we need it though | 00:12 |
| jeblair | it's not in the list of unreferenced files | 00:12 |
| *** smatzek has quit IRC | 00:13 | |
| fungi | does it exist in the source site reprepro copies from? | 00:13 |
| *** smatzek has joined #openstack-infra | 00:13 | |
| jeblair | fungi: i don't know | 00:13 |
| fungi | i'll check an upstream mirror | 00:14 |
| pabelanger | Hmm, debian-jessie failed again on mirrors | 00:15 |
| pabelanger | I'm going to hold one and see what is going on for zuulv3 | 00:15 |
| jeblair | pabelanger: can you be more specific? | 00:15 |
| pabelanger | http://logs.openstack.org/53/511653/1/infra-gate/base-integration-debian-jessie/25c9216/ara/result/983d76c0-aee7-4213-9036-30f45539b0c1/ | 00:15 |
| pabelanger | we should not be using upstream mirrors there | 00:16 |
| pabelanger | but our AFS | 00:16 |
| *** gmann_afk is now known as gmann | 00:16 | |
| jeblair | k | 00:16 |
| jeblair | fungi: that file is referenced by the xenial-security package list | 00:17 |
| ianw | hmm, yes one of the first things we tried was removing xenial-security & xenial-updates yesterday from the update | 00:17 |
| ianw | if things in them are borked, that might be related | 00:18 |
| *** smatzek has quit IRC | 00:18 | |
| *** xinliang has quit IRC | 00:18 | |
| fungi | jeblair: i see it at http://security.ubuntu.com/ubuntu/pool/main/a/apparmor/ | 00:18 |
| fungi | so the security mirror should in theory be wanting it | 00:19 |
| jeblair | ianw: we're telling it to mirror xenial-security now, yeah? | 00:19 |
| jeblair | so i'm surprised that reprepro update did not download it | 00:19 |
| jeblair | i'm considering running that again | 00:19 |
| ianw | jeblair: yes, that should definitely be in there | 00:19 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move tox_envlist into job variables for releasenote jobs https://review.openstack.org/511628 | 00:19 |
| ianw | i removed it for one run as we tried that, but put the original file back | 00:19 |
| *** sdague has quit IRC | 00:19 | |
| jeblair | running update made no changes | 00:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Use sources.list.d for Debian with APT https://review.openstack.org/511658 | 00:27 |
| pabelanger | jeblair: dmsimard: clarkb: ianw: ^ should be our fix for debian mirrors on zuulv3 | 00:27 |
| pabelanger | will confirm in logs first | 00:27 |
| ianw | huh, yeah that seems right | 00:28 |
| *** thorst has joined #openstack-infra | 00:29 | |
| jeblair | i'm trying to figure out what database files are involved in the 'check' command to see if we can figure out if we're missing something from one of them | 00:30 |
| tonyb | Huh, it turns out I don't actually understand the zuulv2 config well enough to write that tool. Rats. | 00:31 |
| jeblair | tonyb: here's an idea: take the console output of a layout check job and parse it | 00:31 |
| jeblair | tonyb: zuul outputs its "compiled" configuration into the log | 00:32 |
| *** thorst has quit IRC | 00:32 | |
| jeblair | tonyb: it'll tell you what jobs run for what projects under what pipelines. it will also have branch matching information in there | 00:32 |
| jeblair | tonyb: it's not intended to be parsable -- it's a debugging aid, but with a few heuristic regex matches, it may get pretty close to what you want | 00:33 |
| jeblair | tonyb: or, you know, forget i said anything and go do something fun. :) | 00:33 |
| ianw | clarkb: that didn't appear to use the unbound change? https://review.openstack.org/#/c/511646/ | 00:33 |
| clarkb | ianw: ya I wonder if it has to be post merge | 00:35 |
| clarkb | because it is prerun? | 00:35 |
| tonyb | jeblair: hehe :) Thanks for the idea I'll look at it | 00:35 |
| *** cody-somerville has joined #openstack-infra | 00:36 | |
| ianw | clarkb: yeah, might be too early | 00:36 |
| *** hamzy has joined #openstack-infra | 00:36 | |
| pabelanger | ianw: clarkb: which patch? | 00:37 |
| clarkb | pabelanger: 511646 and its parent | 00:37 |
| jeblair | ianw: i'm worried we now need to do step 4 -- packages.db. | 00:38 |
| *** aeng has quit IRC | 00:39 | |
| clarkb | I've approved the debian package fix which shold allow us to get in ianw's change and then test the ara on failure thing | 00:39 |
| ianw | jeblair: yeah, touching that dists/ directory i was very worried about :/ | 00:39 |
| *** Goneri has quit IRC | 00:39 | |
| pabelanger | looking | 00:39 |
| *** aeng has joined #openstack-infra | 00:39 | |
| jeblair | ianw: hrm, it's in packages.db | 00:40 |
| ianw | jeblair: security / updates aren't that big, should we drop them & reload? | 00:41 |
| ianw | i think that means remove from the config file | 00:41 |
| pabelanger | clarkb: ya, base-test won't be able to use unmerged role. You'd need to write a test in ozj, which then uses unbound role | 00:41 |
| ianw | it tells you they're not there, clear them out | 00:41 |
| ianw | run delete unreferenced | 00:41 |
| ianw | then start again? | 00:42 |
| jeblair | ianw: is there any chance when you ran rereference you did so with a config without xenial-security? | 00:42 |
| ianw | i did not intentionally do that, but isn't not impossible | 00:42 |
| ianw | that was a terrible sentence | 00:43 |
| ianw | i may have done that | 00:43 |
| clarkb | pabelanger: best test is what I want to test though, we'll just merge ianws fix and recheck | 00:43 |
| *** claudiub|2 has quit IRC | 00:43 | |
| jeblair | ianw: hrm. i was wondering if running rereference might be a way to fix this, but that's only likely if there was something missing from your earlier run of it. | 00:43 |
| pabelanger | clarkb: right, but that is in project-config (trusted playbook) it won't be able to use the unmerged role | 00:43 |
| *** kaisers1 has joined #openstack-infra | 00:44 | |
| pabelanger | ya, merge / recheck should work | 00:44 |
| jeblair | ianw: xenial-security|main|amd64 pool/main/a/apparmor/apparmor_2.10.95-0ubuntu2.6_amd64.deb | 00:44 |
| jeblair | that's from references.db | 00:45 |
| jeblair | ianw: so that looks correct | 00:45 |
| ianw | jeblair: to flesh out my idea, it's remove them from /etc/reprepro/ubuntu/distributions, then run "reprepro --delete clearvanished", then an update, then put them back | 00:46 |
| jeblair | ianw: your idea is sounding like the best approach. | 00:46 |
| *** kaisers has quit IRC | 00:46 | |
| *** LindaWang has joined #openstack-infra | 00:47 | |
| *** kiennt26 has joined #openstack-infra | 00:47 | |
| jeblair | reprepro --confdir /etc/reprepro/ubuntu -VVVV --delete clearvanished | 00:47 |
| jeblair | ianw: ^? | 00:47 |
| ianw | yep | 00:47 |
| ianw | it will tell you to run that if you modify distributions anyway | 00:48 |
| ianw | at least it did for me | 00:48 |
| *** kiennt26 has quit IRC | 00:48 | |
| jeblair | 403 files lost their last reference. | 00:49 |
| jeblair | (dumpunreferenced lists such files, use deleteunreferenced to delete them.) | 00:49 |
| *** kiennt26 has joined #openstack-infra | 00:49 | |
| jeblair | i will not run deleteunreferenced | 00:49 |
| ianw | no, yeah i never ran thta | 00:49 |
| jeblair | ianw: so now, still with xenial-security omitted, run update? | 00:49 |
| ianw | i think so, and see if it works? | 00:50 |
| *** eroux has quit IRC | 00:50 | |
| ianw | hopefully it just finishes all ok ... | 00:50 |
| jeblair | okay that's done | 00:50 |
| jeblair | no mention of xenial-security | 00:50 |
| pabelanger | ze05 doesn't appear to be running zuul-merger | 00:50 |
| jeblair | pabelanger: executors are not supposed to run mergers | 00:50 |
| pabelanger | so, zm05 | 00:51 |
| pabelanger | also zm08 too | 00:51 |
| ianw | jeblair: that seems good?! in theory we put it back in, it starts fresh ... | 00:51 |
| pabelanger | and zm06 and zm07 have hung git processes | 00:51 |
| *** cody-somerville has quit IRC | 00:51 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use sources.list.d for Debian with APT https://review.openstack.org/511658 | 00:51 |
| jeblair | ianw: okay, replaced distributions file, running update again | 00:52 |
| pabelanger | adding to zuulv3 etherpad so we can look at | 00:52 |
| jeblair | ianw: it's getting lots of debs | 00:53 |
| ianw | 403 deb's hopefully, from the prior output | 00:53 |
| jeblair | i lost count after 3 | 00:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Bump reprepro timeout up to 90m https://review.openstack.org/511654 | 00:54 |
| *** huanxie has joined #openstack-infra | 00:55 | |
| pabelanger | okay, added zuul-merger issues to zuulv3-issues etherpad, getting late for me to look into them. Can help in the morning on those | 00:56 |
| jeblair | ianw: done. i'll run checkpool and check now manually? | 00:56 |
| ianw | jeblair: ++ ... have we fixed it?! | 00:57 |
| ianw | (we being you :) | 00:57 |
| jeblair | the error happened at the end of the check command last time, so 15m away from finding out | 00:58 |
| jeblair | probably shoulda dropped the -VVVV for check | 01:00 |
| *** ramishra has quit IRC | 01:00 | |
| *** ramishra has joined #openstack-infra | 01:01 | |
| *** cody-somerville has joined #openstack-infra | 01:02 | |
| *** cody-somerville has joined #openstack-infra | 01:02 | |
| jeblair | ianw: exit 0! | 01:04 |
| ianw | that's the best exit of them all | 01:04 |
| jeblair | ianw: i'm going to remove the unreferenced file for the previous run, and run my modified (no vos release) script again? | 01:04 |
| ianw | as in the output of dumpunreferenced? | 01:06 |
| ianw | is there much in there? | 01:06 |
| jeblair | ianw: er, i think there was. i just deleted it. | 01:06 |
| ianw | maybe just generate it and make sure we're not going to somehow delete everything | 01:07 |
| *** cody-somerville has quit IRC | 01:07 | |
| jeblair | ianw: the script generates the list and uses it on the *next* run | 01:07 |
| jeblair | which is why i deleted it this time, i didn't want to delete things from our previous run with bad data | 01:08 |
| jeblair | i'm running the script now. it has complete the update (a noop), and generated a new unreferenced list. it has 795 files in it. | 01:08 |
| jeblair | it's running a check now. | 01:08 |
| ianw | right, just looking at those files, seeing if they have a common thread | 01:09 |
| jeblair | if this works, i'm thinking run the script again. it will delete the unferenced files and then run another check. if that works, maybe we're ok? (or should we run it one more time?) | 01:09 |
| *** thorst has joined #openstack-infra | 01:10 | |
| *** stakeda has joined #openstack-infra | 01:10 | |
| ianw | jeblair: ok, so looking at that list, yeah it seems reasonable that they're just old packages | 01:10 |
| ianw | (rather than packages we want but have somehow decided to delete) | 01:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Switch to RW afs mirror https://review.openstack.org/511660 | 01:11 |
| *** cody-somerville has joined #openstack-infra | 01:11 | |
| jeblair | ianw: it just finished successfully! | 01:11 |
| jeblair | ianw: i'd like to grab dinner now... may i make the following suggestion: | 01:11 |
| jeblair | ianw: you run "/root/corvus/reprepro-mirror-update /etc/reprepro/ubuntu mirror.ubuntu" as many times as you like to feel comfortable, then merge 511660. | 01:12 |
| ianw | ok, let it do the vos release? | 01:12 |
| jeblair | ianw: /root/corvus/reprepro-mirror-update does not have the vos release | 01:13 |
| jeblair | 511660 will let us test the mirror without a vos release | 01:13 |
| ianw | oh, right yeah i see | 01:13 |
| jeblair | once we have jobs that have run with 511660 in place, we can revert it. if they worked, we can vos release and return to normal. if they did not work, we will still have the old read-only volume to keep things going while we regroup. | 01:14 |
| ianw | jeblair: ok, is cron turned off? | 01:15 |
| jeblair | i think clarkb maybe wanted to let that run for a while to increase the chance of random jobs hitting some bad package. | 01:15 |
| jeblair | ianw: yes, the cron entry is commented out, and mirror-update is in the emergency file. | 01:15 |
| ianw | ok, cool | 01:16 |
| jeblair | ianw: great, thanks! i'm off to dinner now. | 01:16 |
| *** Apoorva_ has joined #openstack-infra | 01:17 | |
| clarkb | ya was hoping to try and shake any problems loose before doing a vos release | 01:19 |
| clarkb | since at that point we are committed aiuu | 01:19 |
| *** aeng has quit IRC | 01:19 | |
| *** Apoorva has quit IRC | 01:20 | |
| *** ijw has quit IRC | 01:20 | |
| *** thorst has quit IRC | 01:20 | |
| *** Apoorva_ has quit IRC | 01:21 | |
| ianw | clarkb: yep, i think if we commit that change, even then just running apt-get update should pull in everything new, we can check the problem packages, gnutls, curl etc | 01:21 |
| ianw | ok, i ran the update and it removed the unreferenced files, the ones that all appeared to be just from regular updates | 01:21 |
| ianw | over the past few days | 01:22 |
| ianw | i'll run again, and it should basically do nothing now | 01:22 |
| ianw | reprepro is as confusing as hell, but surprisingly robust it seems | 01:24 |
| ianw | yay, as expected the update has done nothing | 01:25 |
| ianw | ergo, i believe we have a completely up-to-date mirror | 01:25 |
| *** cuongnv has joined #openstack-infra | 01:27 | |
| clarkb | nice | 01:28 |
| clarkb | I +2'd the rw path change | 01:28 |
| ianw | i think i'm going to write some of this down while it's still fresh in case it happens again | 01:30 |
| *** hongbin has joined #openstack-infra | 01:32 | |
| *** felipemonteiro has quit IRC | 01:36 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch to RW afs mirror https://review.openstack.org/511660 | 01:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: unbound: Add check for undefined default_ipv6 https://review.openstack.org/511653 | 01:39 |
| *** esberglu has joined #openstack-infra | 01:39 | |
| *** esberglu has quit IRC | 01:39 | |
| *** cody-somerville has quit IRC | 01:39 | |
| *** esberglu has joined #openstack-infra | 01:39 | |
| clarkb | I just rechecked the ozj check for ara only on failures | 01:40 |
| *** esberglu has quit IRC | 01:44 | |
| clarkb | no ara at http://logs.openstack.org/46/511646/2/infra-check/base-integration-ubuntu-xenial/d768565/ | 01:44 |
| clarkb | I think it works | 01:44 |
| *** psachin has joined #openstack-infra | 01:46 | |
| pabelanger | yah, looks good | 01:46 |
| *** markvoelker has joined #openstack-infra | 01:47 | |
| pabelanger | should see if we can do a failure next | 01:47 |
| *** iyamahat has quit IRC | 01:47 | |
| *** yamahata has quit IRC | 01:47 | |
| clarkb | we have failure already from when ipv6 didnt work | 01:49 |
| clarkb | look at the retry limit runs | 01:49 |
| pabelanger | isn't that before role merged? | 01:52 |
| *** dhinesh has quit IRC | 01:56 | |
| *** ihrachys_ has quit IRC | 01:59 | |
| *** thorst has joined #openstack-infra | 01:59 | |
| *** thorst has quit IRC | 02:00 | |
| clarkb | no the role we are testing already merged | 02:03 |
| clarkb | ianws change was ipv6 fix but we are testing ara on failure | 02:03 |
| *** daidv has joined #openstack-infra | 02:07 | |
| *** kiennt26 has quit IRC | 02:18 | |
| *** mgagne has quit IRC | 02:19 | |
| *** kiennt26 has joined #openstack-infra | 02:19 | |
| *** markvoelker has quit IRC | 02:21 | |
| *** thingee has quit IRC | 02:22 | |
| *** comstud has quit IRC | 02:22 | |
| *** mgagne has joined #openstack-infra | 02:22 | |
| *** mgagne is now known as Guest53680 | 02:22 | |
| *** thingee has joined #openstack-infra | 02:22 | |
| *** comstud has joined #openstack-infra | 02:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: firehose.rst: trivial syntax https://review.openstack.org/511677 | 02:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add reprepo documentation https://review.openstack.org/511678 | 02:32 |
| ianw | jeblair / pabelanger: ^ i hope that is never needed! | 02:32 |
| *** iyamahat has joined #openstack-infra | 02:32 | |
| ianw | ok, one thing about switching these to the .openstack.org is that i guess we've lost caching instantly | 02:36 |
| ianw | fresh trusty update -> http://paste.openstack.org/show/623515/ | 02:37 |
| clarkb | oh hrm | 02:38 |
| clarkb | looks happy though? | 02:39 |
| ianw | fresh xenial update -> http://paste.openstack.org/show/623516/ | 02:39 |
| ianw | both look happy, and that was the packages in question i think? | 02:39 |
| clarkb | ya gnutls | 02:39 |
| ianw | i'd feel confident running an update with vos release at this point | 02:40 |
| ianw | and switching back, so we get those caches going | 02:40 |
| clarkb | I havent been directly involved so will defer to you and jeblair's judgement on that | 02:41 |
| clarkb | but sounds like probably a reasonable idea because caches | 02:41 |
| *** dave-mccowan has quit IRC | 02:41 | |
| ianw | let's see if jeblair checks back within the next 20 mins with any other ideas, otherwise i think we go. i'll prep the revert | 02:42 |
| fungi | yeah, watching from the late-night sidelines, but sounds promising at any rate | 02:42 |
| jeblair | o/ | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Revert "Switch to RW afs mirror" https://review.openstack.org/511681 | 02:43 |
| jeblair | ianw: ++ revert and release | 02:45 |
| ianw | ok, i'll manually run the full /usr/local/bin/ update script now | 02:45 |
| ianw | logging to /tmp/ubuntu-update.log | 02:46 |
| *** hongbin has quit IRC | 02:50 | |
| *** links has joined #openstack-infra | 02:51 | |
| ianw | ... and done | 02:52 |
| ianw | and thus the mirrors should be healthy again! | 02:52 |
| fungi | and we have 20m inodes free on the logs site now | 02:53 |
| fungi | tomorrow's shaping up to be a quiet day for a change (i hope) | 02:53 |
| ianw | jeblair: should we let cron loose on it again? we could wait, at this point | 02:54 |
| *** bobh has quit IRC | 03:03 | |
| *** rosmaita has quit IRC | 03:06 | |
| *** iyamahat has quit IRC | 03:06 | |
| *** liusheng has quit IRC | 03:09 | |
| *** liusheng has joined #openstack-infra | 03:09 | |
| *** thorst has joined #openstack-infra | 03:10 | |
| openstackgerrit | Merged openstack-infra/system-config master: firehose.rst: trivial syntax https://review.openstack.org/511677 | 03:11 |
| *** udesale has joined #openstack-infra | 03:12 | |
| *** hongbin has joined #openstack-infra | 03:14 | |
| *** thorst has quit IRC | 03:15 | |
| jeblair | ianw: if you wanted to leave cron disabled, i (or maybe fungi?) could get that first thing in our morning | 03:15 |
| *** nicolasbock has quit IRC | 03:16 | |
| jeblair | ianw: though if you are inclined to re-enable it, it sounds like it ought to be safe now. | 03:16 |
| jeblair | your choice :) | 03:16 |
| ianw | jeblair: let's do it now, tomorrow will have enough to deal with :) i'll be around to keep and eye on the next few runs | 03:17 |
| ianw | i'll remove it from emergency | 03:17 |
| *** markvoelker has joined #openstack-infra | 03:18 | |
| *** yamahata has joined #openstack-infra | 03:21 | |
| *** smatzek has joined #openstack-infra | 03:22 | |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Switch to RW afs mirror" https://review.openstack.org/511681 | 03:23 |
| *** hongbin has quit IRC | 03:25 | |
| *** aeng has joined #openstack-infra | 03:27 | |
| *** priteau has joined #openstack-infra | 03:31 | |
| *** smatzek has quit IRC | 03:32 | |
| *** priteau has quit IRC | 03:36 | |
| *** yamahata has quit IRC | 03:36 | |
| *** edmondsw has joined #openstack-infra | 03:38 | |
| *** coolsvap has joined #openstack-infra | 03:41 | |
| *** jaosorior has joined #openstack-infra | 03:45 | |
| *** edmondsw has quit IRC | 03:45 | |
| *** markvoelker has quit IRC | 03:51 | |
| *** yamahata has joined #openstack-infra | 03:53 | |
| *** dbecker has quit IRC | 03:54 | |
| SamYaple | would https://github.com/openstack-infra/zuul/blob/feature/zuulv3/zuul/ansible/callback/zuul_stream.py#L69 be useable with async tasks? | 03:57 |
| *** daidv has quit IRC | 04:00 | |
| *** daidv has joined #openstack-infra | 04:01 | |
| *** dhinesh has joined #openstack-infra | 04:01 | |
| *** cuongnv has quit IRC | 04:03 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 04:04 | |
| *** cuongnv has joined #openstack-infra | 04:04 | |
| *** annp has joined #openstack-infra | 04:05 | |
| *** ykarel|afk has joined #openstack-infra | 04:06 | |
| *** dbecker has joined #openstack-infra | 04:09 | |
| *** kiennt26 has quit IRC | 04:15 | |
| *** huanxie has quit IRC | 04:18 | |
| *** gildub has quit IRC | 04:20 | |
| *** esberglu has joined #openstack-infra | 04:20 | |
| *** esberglu has quit IRC | 04:25 | |
| *** pcaruana has joined #openstack-infra | 04:27 | |
| *** sree has joined #openstack-infra | 04:29 | |
| *** ramishra has quit IRC | 04:37 | |
| *** ramishra has joined #openstack-infra | 04:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Create reprepro lockdir https://review.openstack.org/511380 | 04:41 |
| *** markvoelker has joined #openstack-infra | 04:49 | |
| *** armax has quit IRC | 04:50 | |
| *** armax has joined #openstack-infra | 04:51 | |
| *** armax has quit IRC | 04:51 | |
| *** armax has joined #openstack-infra | 04:51 | |
| *** huanxie has joined #openstack-infra | 04:52 | |
| *** armax has quit IRC | 04:52 | |
| *** armax has joined #openstack-infra | 04:52 | |
| *** armax has quit IRC | 04:53 | |
| *** bobh has joined #openstack-infra | 04:54 | |
| *** pcaruana has quit IRC | 04:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use native propose-translation jobs https://review.openstack.org/511435 | 04:56 |
| *** AJaeger has joined #openstack-infra | 04:57 | |
| *** AJaeger has quit IRC | 04:57 | |
| *** bobh has quit IRC | 04:58 | |
| *** claudiub|2 has joined #openstack-infra | 04:58 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add branches to all periodic jobs https://review.openstack.org/511533 | 04:58 |
| *** cody-somerville has joined #openstack-infra | 05:00 | |
| *** cody-somerville has joined #openstack-infra | 05:00 | |
| *** udesale has quit IRC | 05:01 | |
| *** AJaeger has joined #openstack-infra | 05:01 | |
| *** vaidy has quit IRC | 05:02 | |
| *** isviridov_away has quit IRC | 05:02 | |
| AJaeger | ianw, if you have some time to review a few of my open changes for project-config and openstack-zuul-jobs, please, it would be great... | 05:02 |
| *** cody-somerville has quit IRC | 05:04 | |
| *** thorst has joined #openstack-infra | 05:11 | |
| *** thorst has quit IRC | 05:15 | |
| ianw | AJaeger: you always provide for an interesting tour around some lesser known parts of these jobs :) | 05:15 |
| *** jaosorior has quit IRC | 05:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy job from openstackdocstheme https://review.openstack.org/511287 | 05:17 |
| *** chlong has joined #openstack-infra | 05:20 | |
| *** ykarel|afk has quit IRC | 05:21 | |
| *** markvoelker has quit IRC | 05:22 | |
| *** gouthamr has quit IRC | 05:24 | |
| *** edmondsw has joined #openstack-infra | 05:27 | |
| *** pcaruana has joined #openstack-infra | 05:30 | |
| *** jaosorior has joined #openstack-infra | 05:30 | |
| *** ykarel has joined #openstack-infra | 05:32 | |
| *** edmondsw has quit IRC | 05:32 | |
| *** slaweq_ has joined #openstack-infra | 05:32 | |
| *** priteau has joined #openstack-infra | 05:32 | |
| *** slaweq_ has quit IRC | 05:33 | |
| *** smatzek has joined #openstack-infra | 05:33 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert legacy-api-ref and -guide to native v3 https://review.openstack.org/511290 | 05:35 |
| *** priteau has quit IRC | 05:37 | |
| openstackgerrit | Merged openstack-dev/pbr master: Add reno for release notes management https://review.openstack.org/510679 | 05:37 |
| *** smatzek has quit IRC | 05:38 | |
| *** vaidy has joined #openstack-infra | 05:38 | |
| *** isviridov_away has joined #openstack-infra | 05:39 | |
| *** armaan has joined #openstack-infra | 05:39 | |
| *** pcaruana has quit IRC | 05:39 | |
| *** rcernin has joined #openstack-infra | 05:41 | |
| *** armaan has quit IRC | 05:43 | |
| *** nikhil_k has quit IRC | 05:46 | |
| *** andreas_s has joined #openstack-infra | 05:53 | |
| *** spectr has quit IRC | 05:58 | |
| *** spectr has joined #openstack-infra | 05:58 | |
| *** Hal has joined #openstack-infra | 05:59 | |
| *** Hal is now known as Guest90486 | 05:59 | |
| *** claudiub|2 has quit IRC | 06:01 | |
| *** cshastri has joined #openstack-infra | 06:06 | |
| *** dfflanders has joined #openstack-infra | 06:08 | |
| *** slaweq_ has joined #openstack-infra | 06:09 | |
| *** esberglu has joined #openstack-infra | 06:11 | |
| *** esberglu has quit IRC | 06:11 | |
| *** esberglu has joined #openstack-infra | 06:11 | |
| *** esberglu has quit IRC | 06:11 | |
| *** jbadiapa has joined #openstack-infra | 06:11 | |
| *** esberglu has joined #openstack-infra | 06:12 | |
| *** ykarel_ has joined #openstack-infra | 06:16 | |
| *** esberglu has quit IRC | 06:16 | |
| *** ykarel has quit IRC | 06:17 | |
| *** aeng has quit IRC | 06:18 | |
| *** markvoelker has joined #openstack-infra | 06:19 | |
| *** kiennt26 has joined #openstack-infra | 06:21 | |
| *** e0ne has joined #openstack-infra | 06:32 | |
| *** e0ne has quit IRC | 06:35 | |
| *** e0ne has joined #openstack-infra | 06:35 | |
| AJaeger | ianw: they were broken, so I had to learn something new ;) | 06:38 |
| AJaeger | ianw: thanks | 06:38 |
| evrardjp | tonyb: updated the review https://review.openstack.org/#/c/511479/1 with a comment as you asked. | 06:39 |
| *** ccamacho has joined #openstack-infra | 06:39 | |
| *** slaweq_ has quit IRC | 06:40 | |
| *** vsaienk0 has joined #openstack-infra | 06:41 | |
| *** slaweq_ has joined #openstack-infra | 06:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use build-openstack-api-ref for openstackdocstheme https://review.openstack.org/511318 | 06:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix v2 shade setup https://review.openstack.org/511733 | 06:47 |
| *** eumel8 has joined #openstack-infra | 06:48 | |
| AJaeger | ianw, yolanda, zuul v2 files are currently broken, we need the change above ^ | 06:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Set v3 nodepool citycloud timeout to 600 https://review.openstack.org/511656 | 06:48 |
| eumel8 | morning | 06:48 |
| yolanda | AJaeger, looking | 06:50 |
| AJaeger | morning, eumel8 and yolanda | 06:50 |
| *** martinkopec has joined #openstack-infra | 06:51 | |
| yolanda | pypi jobs will be enough to fix shade? | 06:51 |
| AJaeger | yolanda: should be ;) want to +2 and then I'll wait for test results? | 06:51 |
| yolanda | ok +2 | 06:51 |
| AJaeger | yolanda: if you have time, could you review https://review.openstack.org/#/c/511435/ and https://review.openstack.org/511404 and https://review.openstack.org/511563 as well, please? | 06:52 |
| *** markvoelker has quit IRC | 06:52 | |
| *** dfflanders has quit IRC | 06:52 | |
| AJaeger | and https://review.openstack.org/511555 | 06:53 |
| *** shardy has joined #openstack-infra | 06:54 | |
| eumel8 | yolanda: maybe it's a good the to get a merge: https://review.openstack.org/#/c/506795/ :) | 06:54 |
| *** mpaolino has joined #openstack-infra | 06:54 | |
| *** ykarel has joined #openstack-infra | 06:55 | |
| *** ykarel_ has quit IRC | 06:57 | |
| *** huanxie has quit IRC | 06:57 | |
| *** threestrands has quit IRC | 06:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix v2 shade setup https://review.openstack.org/511733 | 06:59 |
| *** bauzas is now known as bauwser | 07:01 | |
| *** vhosakot has quit IRC | 07:06 | |
| AJaeger | mordred: now you can release shade ^ | 07:06 |
| *** pcaruana has joined #openstack-infra | 07:07 | |
| *** dhajare has joined #openstack-infra | 07:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix publish-releasenotes https://review.openstack.org/511739 | 07:10 |
| AJaeger | mnaser: followup to your change ^ | 07:11 |
| *** thorst has joined #openstack-infra | 07:12 | |
| *** florianf has joined #openstack-infra | 07:13 | |
| *** priteau has joined #openstack-infra | 07:15 | |
| *** edmondsw has joined #openstack-infra | 07:15 | |
| *** gcb has quit IRC | 07:16 | |
| *** thorst has quit IRC | 07:17 | |
| *** slaweq_ has quit IRC | 07:17 | |
| *** armaan has joined #openstack-infra | 07:17 | |
| *** vsaienk0 has quit IRC | 07:18 | |
| *** edmondsw has quit IRC | 07:20 | |
| *** priteau has quit IRC | 07:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use system-required instead of merge-check https://review.openstack.org/511745 | 07:20 |
| *** slaweq_ has joined #openstack-infra | 07:20 | |
| *** tesseract has joined #openstack-infra | 07:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add system-required template https://review.openstack.org/511746 | 07:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove merge-check https://review.openstack.org/511747 | 07:22 |
| AJaeger | monty, jeblair, I've done the change for merge-check -> system-required ^ | 07:23 |
| AJaeger | Use topic:system-required for reviewing | 07:23 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Use system-required instead of merge-check https://review.openstack.org/511748 | 07:25 |
| *** adarazs has quit IRC | 07:25 | |
| *** dtantsur|afk has quit IRC | 07:26 | |
| *** spectr has quit IRC | 07:26 | |
| *** tesseract has quit IRC | 07:26 | |
| *** rcernin has quit IRC | 07:26 | |
| *** pcaruana has quit IRC | 07:26 | |
| *** martinkopec has quit IRC | 07:27 | |
| *** dtantsur has joined #openstack-infra | 07:31 | |
| *** adarazs has joined #openstack-infra | 07:32 | |
| *** e0ne has quit IRC | 07:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Document v3 periodic jobs https://review.openstack.org/511750 | 07:36 |
| AJaeger | jeblair, mordred: Let's document periodic jobs ^ | 07:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Document v3 periodic jobs https://review.openstack.org/511750 | 07:38 |
| *** martinkopec has joined #openstack-infra | 07:39 | |
| *** spectr has joined #openstack-infra | 07:40 | |
| *** pcaruana has joined #openstack-infra | 07:40 | |
| *** rcernin has joined #openstack-infra | 07:40 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** armaan has quit IRC | 07:41 | |
| *** spectr has quit IRC | 07:43 | |
| *** pcaruana has quit IRC | 07:44 | |
| *** pcaruana has joined #openstack-infra | 07:44 | |
| *** spectr has joined #openstack-infra | 07:44 | |
| *** Guest90486 has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove pin pip from beaker legacy jobs https://review.openstack.org/511361 | 07:46 |
| *** markvoelker has joined #openstack-infra | 07:50 | |
| *** slaweq_ has quit IRC | 07:53 | |
| *** andreas_s has quit IRC | 07:54 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| yolanda | AJaeger, how is zuul v3 status now? are we ready to review things? | 07:55 |
| AJaeger | yolanda: we can review v3 things for sure - but v3 only runs on infra stuff... | 07:56 |
| AJaeger | So, let's not suggest to folks to merge adding .zuul.yaml to their repos or update them - no tests are run currently for "normal projects" | 07:56 |
| AJaeger | yolanda: so, go ahead and review project-config, openstack-zuul-jobs etc. please | 07:56 |
| yolanda | ok | 07:57 |
| *** andreas_s has joined #openstack-infra | 07:57 | |
| AJaeger | yolanda: https://etherpad.openstack.org/p/zuulv3-issues has on the top of list of reviews if you like to use that list - or go directly over the repos | 07:59 |
| *** rossella_s has joined #openstack-infra | 08:01 | |
| *** slaweq_ has joined #openstack-infra | 08:02 | |
| yolanda | ok let me look | 08:03 |
| *** egonzalez has joined #openstack-infra | 08:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/openstack-zuul-jobs master: convert deploy-guide to native zuul v3 https://review.openstack.org/511564 | 08:07 |
| AJaeger | yolanda: why did you push 511564 again? A recheck should be enough - but only once project-config change is in | 08:07 |
| yolanda | i clicked the rebase button | 08:08 |
| yolanda | for that -1 from zuul | 08:08 |
| *** rossella_s has quit IRC | 08:08 | |
| AJaeger | yolanda: that -1 from zuul is normal for any change that dpeends on project-config | 08:08 |
| AJaeger | So, feel free to +2 - and I'll recheck and +A once the dependency is in. | 08:09 |
| tonyb | evrardjp: +2W | 08:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Add in merge-check templates for LOCI https://review.openstack.org/510649 | 08:09 |
| AJaeger | yolanda: See the message in 511564 "This change depends on a change to a config project. | 08:09 |
| yolanda | AJaeger , so is normal to see -1 without logs, if the project-config change is still not merged? | 08:09 |
| AJaeger | The syntax of the configuration in this change has been verified to be | 08:09 |
| AJaeger | correct once the config project change upon which it depends is | 08:09 |
| AJaeger | merged, but it can not be used until that occurs. | 08:09 |
| AJaeger | " | 08:09 |
| AJaeger | yolanda: yes! | 08:09 |
| yolanda | ok | 08:10 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Moving the OpenStack-Ansible meeting time and channel https://review.openstack.org/511479 | 08:10 |
| SamYaple | would love to get a review +2+W for this bindep patch https://review.openstack.org/#/c/506502/ | 08:10 |
| *** rossella_s has joined #openstack-infra | 08:10 | |
| SamYaple | if anyone has some time | 08:10 |
| *** yamahata has quit IRC | 08:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: remove legacy openstackdocstheme jobs https://review.openstack.org/511289 | 08:11 |
| *** thorst has joined #openstack-infra | 08:12 | |
| AJaeger | yolanda: since project-config is a trusted repo, some of these depends-on will not work and we have to do the dance of merge one change, recheck the next etc. So, feel free to +2 and I'll recheck once dependines are in... | 08:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-deploy-guide job https://review.openstack.org/511563 | 08:14 |
| *** armaan has joined #openstack-infra | 08:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use system-required instead of merge-check https://review.openstack.org/511745 | 08:15 |
| *** rossella_s has quit IRC | 08:16 | |
| *** thorst has quit IRC | 08:17 | |
| *** rossella_s has joined #openstack-infra | 08:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Removing collectd-ceilometer-plugin definition from zuul.d/projects.yaml https://review.openstack.org/510086 | 08:19 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use native propose-translation jobs https://review.openstack.org/511435 | 08:20 |
| *** andreas_s has quit IRC | 08:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update some legacy jobs https://review.openstack.org/511555 | 08:21 |
| *** gcb has joined #openstack-infra | 08:21 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add branches to all periodic jobs https://review.openstack.org/511533 | 08:22 |
| *** markvoelker has quit IRC | 08:23 | |
| *** andreas_s has joined #openstack-infra | 08:23 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused kuryr-libnetwork jobs https://review.openstack.org/511404 | 08:23 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Mixmatch https://review.openstack.org/511763 | 08:25 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Mixmatch https://review.openstack.org/511764 | 08:25 |
| *** andreas_s has quit IRC | 08:26 | |
| *** derekh has joined #openstack-infra | 08:26 | |
| *** andreas_s has joined #openstack-infra | 08:28 | |
| *** lucas-afk is now known as lucasagomes | 08:28 | |
| *** rossella_s has quit IRC | 08:28 | |
| *** coolsvap has quit IRC | 08:30 | |
| *** rossella_s has joined #openstack-infra | 08:31 | |
| *** dhinesh has quit IRC | 08:31 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update chair for doc team meeting https://review.openstack.org/511484 | 08:31 |
| *** ralonsoh has joined #openstack-infra | 08:32 | |
| *** slaweq_ has quit IRC | 08:35 | |
| *** jascott1 has quit IRC | 08:35 | |
| *** jascott1 has joined #openstack-infra | 08:35 | |
| *** kjackal_ has joined #openstack-infra | 08:35 | |
| *** ykarel is now known as ykarel|lunch | 08:36 | |
| tonyb | I'm trying to use owners.py and it's failing to get https://review.openstack.org/gitweb?p=openstack/governance.git;a=blob_plain;f=reference/projects.yaml;hb=oct-2017-elections with a 404 shoudl that still work? | 08:36 |
| *** johnthetubaguy has quit IRC | 08:39 | |
| *** jascott1 has quit IRC | 08:39 | |
| *** johnthetubaguy has joined #openstack-infra | 08:40 | |
| *** thorst has joined #openstack-infra | 08:41 | |
| *** e0ne has joined #openstack-infra | 08:41 | |
| *** slaweq_ has joined #openstack-infra | 08:44 | |
| *** armaan has quit IRC | 08:44 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: convert deploy-guide to native zuul v3 https://review.openstack.org/511564 | 08:44 |
| *** thorst has quit IRC | 08:45 | |
| *** huanxie has joined #openstack-infra | 08:46 | |
| *** kiennt26 has quit IRC | 08:50 | |
| AJaeger | tonyb: the updated gerrit caused a few changes, this could be one part of it. Best discuss with rest of team later... | 08:54 |
| *** electrofelix has joined #openstack-infra | 08:56 | |
| *** kaisers1 has quit IRC | 08:57 | |
| *** dtantsur has quit IRC | 08:57 | |
| *** dtantsur has joined #openstack-infra | 08:58 | |
| *** adarazs has quit IRC | 08:58 | |
| *** rcernin has quit IRC | 08:59 | |
| *** spectr has quit IRC | 08:59 | |
| *** panda|rover|off is now known as panda|rover | 08:59 | |
| *** martinkopec has quit IRC | 08:59 | |
| *** pcaruana has quit IRC | 08:59 | |
| ianw | tonyb: we switched to cgit, so ... no | 09:00 |
| ianw | i would get that from git.openstack.org, rather than review.o.o | 09:00 |
| ianw | that way it's load balanced & more reliable | 09:00 |
| *** andreas_s has quit IRC | 09:02 | |
| *** kiennt26 has joined #openstack-infra | 09:03 | |
| *** edmondsw has joined #openstack-infra | 09:04 | |
| *** adarazs has joined #openstack-infra | 09:05 | |
| *** andreas_s has joined #openstack-infra | 09:07 | |
| *** edmondsw has quit IRC | 09:08 | |
| *** ccamacho has left #openstack-infra | 09:09 | |
| *** andreas_s has quit IRC | 09:12 | |
| *** martinkopec has joined #openstack-infra | 09:12 | |
| *** rcernin has joined #openstack-infra | 09:12 | |
| *** spectr has joined #openstack-infra | 09:12 | |
| *** andreas_s has joined #openstack-infra | 09:12 | |
| *** pcaruana has joined #openstack-infra | 09:13 | |
| *** ccamacho has joined #openstack-infra | 09:14 | |
| *** jgr has joined #openstack-infra | 09:14 | |
| *** priteau has joined #openstack-infra | 09:16 | |
| *** andreas_s has quit IRC | 09:17 | |
| *** slaweq_ has quit IRC | 09:17 | |
| *** kiennt26 has quit IRC | 09:17 | |
| *** ykarel|lunch is now known as ykarel | 09:18 | |
| *** markvoelker has joined #openstack-infra | 09:20 | |
| *** priteau has quit IRC | 09:20 | |
| *** slaweq_ has joined #openstack-infra | 09:21 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Mogan https://review.openstack.org/511780 | 09:23 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Mogan https://review.openstack.org/511781 | 09:23 |
| openstackgerrit | Tovin Seven proposed openstack-infra/project-config master: Remove legacy oslo.versionedobjects job https://review.openstack.org/511783 | 09:24 |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy oslo.versionedobjects job https://review.openstack.org/511784 | 09:24 |
| *** gcb has quit IRC | 09:27 | |
| *** tosky has joined #openstack-infra | 09:28 | |
| tosky | AJaeger: morning, thanks for the review | 09:31 |
| AJaeger | morning, tosky - you're welcome | 09:31 |
| tosky | (as usual :) | 09:31 |
| tosky | did you see this error, which seems unrelated to tripleo? http://logs.openstack.org/54/509554/2/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq/8daa9af/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-10-12_23_46_57_067 | 09:31 |
| AJaeger | tosky: best ask others on the team later for that - might have been a networking issue | 09:32 |
| tosky | AJaeger: uhm, so should I not feel guilty if I try to recheck? | 09:33 |
| AJaeger | tosky: with that timing, infra was still not green, so go for the recheck... | 09:35 |
| tosky | oh, thanks! | 09:35 |
| *** Hal has joined #openstack-infra | 09:36 | |
| *** Hal is now known as Guest19388 | 09:36 | |
| *** andreas_s has joined #openstack-infra | 09:36 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Cliff https://review.openstack.org/511788 | 09:37 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Cliff https://review.openstack.org/511789 | 09:37 |
| *** chlong has quit IRC | 09:40 | |
| *** egonzalez has quit IRC | 09:42 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara-dashboard: use the openstack-python-jobs template https://review.openstack.org/511791 | 09:43 |
| *** priteau has joined #openstack-infra | 09:44 | |
| *** andreas_s has quit IRC | 09:45 | |
| *** andreas_s has joined #openstack-infra | 09:46 | |
| *** egonzalez has joined #openstack-infra | 09:46 | |
| *** huanxie has quit IRC | 09:46 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:47 | |
| *** markvoelker has quit IRC | 09:53 | |
| *** slaweq_ has quit IRC | 09:55 | |
| *** andreas_s has quit IRC | 09:55 | |
| *** armaan has joined #openstack-infra | 09:58 | |
| tosky | yolanda: I know that this is a minor review, but can you please take some time to check small IRC notification change for sahara channels: https://review.openstack.org/#/c/503936/ ? | 09:58 |
| tosky | especially now that we are in the time before two storms :) | 09:59 |
| *** shardy has quit IRC | 09:59 | |
| *** slaweq_ has joined #openstack-infra | 10:00 | |
| *** andreas_s has joined #openstack-infra | 10:00 | |
| openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add initial jobs for manila-tempest-plugin https://review.openstack.org/510548 | 10:02 |
| *** cuongnv has quit IRC | 10:03 | |
| *** sambetts|afk is now known as sambetts | 10:05 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara-dashboard: use the openstack-python-jobs template https://review.openstack.org/511791 | 10:08 |
| openstackgerrit | Raissa Sarmento proposed openstack-infra/project-config master: Add initial jobs for manila-tempest-plugin https://review.openstack.org/510548 | 10:09 |
| *** andreas_s has quit IRC | 10:09 | |
| *** florianf has quit IRC | 10:09 | |
| *** florianf has joined #openstack-infra | 10:09 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove openstack-api-ref jobs https://review.openstack.org/511796 | 10:13 |
| *** sbezverk has joined #openstack-infra | 10:14 | |
| *** andreas_s has joined #openstack-infra | 10:14 | |
| *** daidv has quit IRC | 10:15 | |
| *** armaan has quit IRC | 10:17 | |
| *** pbourke has quit IRC | 10:17 | |
| *** pbourke has joined #openstack-infra | 10:18 | |
| *** andreas_s has quit IRC | 10:19 | |
| *** andreas_s has joined #openstack-infra | 10:19 | |
| *** annp has quit IRC | 10:22 | |
| *** sdague has joined #openstack-infra | 10:24 | |
| *** boden has joined #openstack-infra | 10:27 | |
| *** andreas_s has quit IRC | 10:28 | |
| *** andreas_s has joined #openstack-infra | 10:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: v3 cleanup: Use openstack-python-jobs https://review.openstack.org/511803 | 10:30 |
| *** psachin has quit IRC | 10:33 | |
| *** gmann is now known as gmann_afk | 10:33 | |
| *** andreas_s has quit IRC | 10:34 | |
| *** huanxie has joined #openstack-infra | 10:34 | |
| *** andreas_s has joined #openstack-infra | 10:36 | |
| *** jascott1 has joined #openstack-infra | 10:37 | |
| *** sree has quit IRC | 10:41 | |
| *** thorst has joined #openstack-infra | 10:42 | |
| *** sree has joined #openstack-infra | 10:42 | |
| *** esberglu has joined #openstack-infra | 10:43 | |
| *** esberglu has quit IRC | 10:43 | |
| *** thorst_ has joined #openstack-infra | 10:45 | |
| *** sree has quit IRC | 10:46 | |
| *** jaosorior has quit IRC | 10:47 | |
| *** thorst has quit IRC | 10:47 | |
| *** pcaruana has quit IRC | 10:48 | |
| *** yamamoto has quit IRC | 10:48 | |
| *** thorst_ has quit IRC | 10:49 | |
| *** ociuhandu has quit IRC | 10:49 | |
| *** markvoelker has joined #openstack-infra | 10:50 | |
| AJaeger | jeblair, mordred, pabelanger, we have an infra-post job running for 5 h - apparently waiting for node. Job is publish-openstack-sphinx-docs-infraqueue, repo openstack-zuul-jobs, hash is 8c1d4e1b40a3f1a47e511785d5a54a1d3e11451f . | 10:54 |
| *** claudiub has joined #openstack-infra | 10:59 | |
| mordred | AJaeger: cool- I'll look in just a few | 11:00 |
| *** slaweq_ has quit IRC | 11:03 | |
| *** aperri has joined #openstack-infra | 11:07 | |
| *** nicolasbock has joined #openstack-infra | 11:09 | |
| *** slaweq_ has joined #openstack-infra | 11:10 | |
| *** armaan has joined #openstack-infra | 11:10 | |
| mordred | AJaeger: hrm. I'm not sure I'm awake enough to debug this yet | 11:10 |
| *** adarazs has left #openstack-infra | 11:13 | |
| *** nicolasbock has quit IRC | 11:14 | |
| AJaeger | mordred: it's not urgent - just wanted to point it out. Get awake first, please ;) | 11:14 |
| *** lucasagomes is now known as lucas-hungry | 11:14 | |
| AJaeger | mordred: once you're awake, hope you like https://review.openstack.org/511746 ;) | 11:15 |
| *** hashar has joined #openstack-infra | 11:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Stop translation of trove https://review.openstack.org/511813 | 11:20 |
| AJaeger | eumel8, ianychoi, kumarmn ^ | 11:20 |
| *** edmondsw has joined #openstack-infra | 11:20 | |
| *** markvoelker has quit IRC | 11:23 | |
| *** erlon has joined #openstack-infra | 11:23 | |
| *** sree has joined #openstack-infra | 11:24 | |
| *** alexchadin has joined #openstack-infra | 11:24 | |
| *** rosmaita has joined #openstack-infra | 11:25 | |
| *** Qiming has quit IRC | 11:26 | |
| *** nicolasbock has joined #openstack-infra | 11:26 | |
| eumel8 | AJaeger: I heart it, trove is in an unmaintanable state | 11:26 |
| *** aviau has quit IRC | 11:28 | |
| *** Qiming has joined #openstack-infra | 11:28 | |
| *** aviau has joined #openstack-infra | 11:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Stop translation of trove https://review.openstack.org/511813 | 11:28 |
| AJaeger | eumel8: just updated commit message to explain this can be changed back ^ | 11:28 |
| AJaeger | eumel8: could you +1 if you agree - it's not worth spending resources on it *currently* | 11:29 |
| eumel8 | AJaeger: sure, I would also set the projects read-only in Zanata to prevent useless translations | 11:30 |
| mordred | AJaeger: love it! | 11:31 |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| AJaeger | eumel8: yes, exactly - make it read-only... | 11:34 |
| AJaeger | mordred: three changes for this ;( https://review.openstack.org/#/c/511745 would be the next one... | 11:35 |
| mordred | AJaeger: yah - the story around moving/renaming is not as fun as other things | 11:35 |
| AJaeger | mordred: yeah ;( | 11:36 |
| *** smatzek has joined #openstack-infra | 11:37 | |
| AJaeger | yolanda and ianw reviewed nicely our open list - now we need one more review basically. mordred, if you have time to go over the etherpad reviews, it would be nice . But nothing urgent... | 11:37 |
| *** esberglu has joined #openstack-infra | 11:37 | |
| mordred | AJaeger: YOU KNOW - we could just locate the system-required project-template in project-config - there's no real need for it to be in openstack-zuul-jobs | 11:37 |
| eumel8 | AJaeger: trove-dashboard ist still in use? https://review.openstack.org/#/c/504749/ https://review.openstack.org/#/c/493761/ | 11:37 |
| mordred | AJaeger: cool - will look | 11:37 |
| AJaeger | eumel8: yeah - but they merge at least some stuff... | 11:37 |
| eumel8 | not for stable/pike... | 11:38 |
| AJaeger | mordred: shall I moe it over? | 11:38 |
| AJaeger | mordred: I quickly gave a -2 o nthe change in case we want to have the template there | 11:39 |
| AJaeger | right now we have no templates in project-config for zuul v3 | 11:39 |
| mordred | AJaeger: hrm. actually - no, if we leave it in ozj then we can do depends-on with chanes to it | 11:39 |
| mordred | AJaeger: so I think I'm wrong and the way it is is correct | 11:39 |
| AJaeger | ok | 11:40 |
| *** yamamoto has joined #openstack-infra | 11:40 | |
| AJaeger | lunch time - bbl | 11:41 |
| *** huanxie has quit IRC | 11:41 | |
| *** esberglu has quit IRC | 11:41 | |
| *** slaweq_ has quit IRC | 11:43 | |
| *** armaan has quit IRC | 11:44 | |
| *** armaan has joined #openstack-infra | 11:44 | |
| *** yamamoto has quit IRC | 11:45 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add system-required template https://review.openstack.org/511746 | 11:47 |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** baoli has joined #openstack-infra | 11:49 | |
| *** pblaho has joined #openstack-infra | 11:53 | |
| *** thorst has joined #openstack-infra | 11:53 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** martinkopec has quit IRC | 11:54 | |
| *** dprince has joined #openstack-infra | 11:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Collapse job header lines into a single statement https://review.openstack.org/511821 | 11:57 |
| alexchadin | hi. how can I ask Zuul to recheck commit instead of Jenkins? | 11:57 |
| AJaeger | alexchadin: you cannot. | 11:58 |
| AJaeger | alexchadin: a recheck would recheck both systems | 11:58 |
| AJaeger | but currently zuul v3 is only running on a few repos due to the problems we had elsewhere the last days. | 11:59 |
| alexchadin | AJaeger: oh, thanks for quick reply | 11:59 |
| AJaeger | alexchadin: best wait until zuul v3 is running check on all jobs again and then recheck | 11:59 |
| AJaeger | alexchadin: why did you recheck 510848 ? | 12:00 |
| AJaeger | alexchadin: ah, sorry - that's fine... | 12:01 |
| alexchadin | AJaeger: Zuul shows error with one job: multinode-integration-debian-jessie | 12:01 |
| AJaeger | alexchadin: didn't remember that dependencies were in... | 12:01 |
| alexchadin | AJaeger: +1 | 12:01 |
| *** hashar has quit IRC | 12:01 | |
| *** armaan has quit IRC | 12:01 | |
| *** baoli_ has joined #openstack-infra | 12:02 | |
| *** armaan has joined #openstack-infra | 12:02 | |
| *** andreas_s has quit IRC | 12:02 | |
| *** tpsilva has joined #openstack-infra | 12:02 | |
| *** baoli has quit IRC | 12:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix publish-releasenotes https://review.openstack.org/511739 | 12:05 |
| *** LindaWang has quit IRC | 12:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: sahara-dashboard: use the openstack-python-jobs template https://review.openstack.org/511791 | 12:07 |
| odyssey4me | howdy all - IIRC someone on the ML mentioned an environment variable which was specific to zuul v2 and would not be carried forward... but I can't seem to find it - does anyone know which env var I'm referring to? | 12:07 |
| mordred | AJaeger, ianw, yolanda: great work while the rest of us slept! I've +A'd all the things I can +A | 12:08 |
| mordred | odyssey4me: probably ZUUL_REF | 12:08 |
| mordred | I think | 12:08 |
| mordred | hang on ... | 12:08 |
| mordred | odyssey4me: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3 | 12:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Add initial jobs for manila-tempest-plugin https://review.openstack.org/510548 | 12:09 |
| mordred | odyssey4me: that is the filter plugin that turns things in to legacy env vars | 12:09 |
| *** felipemonteiro has joined #openstack-infra | 12:09 | |
| odyssey4me | ok, so ZUUL_URL, ZUUL_REF and ZUUL_COMMIT are v2 only | 12:09 |
| mordred | odyssey4me: that said - the intent is for ALL of the ZUUL_* env vars to go away - the things in that are just for transition purposes | 12:10 |
| odyssey4me | we're going to build a bit of a shim to detect the version and use z-c for v2 only | 12:10 |
| mordred | odyssey4me: woot! | 12:10 |
| *** baoli_ has quit IRC | 12:10 | |
| odyssey4me | that'll help us get jobs working with both active | 12:10 |
| odyssey4me | cool - thanks a mill | 12:10 |
| *** lucas-hungry is now known as lucasagomes | 12:10 | |
| *** hashar has joined #openstack-infra | 12:11 | |
| mordred | I'm a fan of this plan | 12:11 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Use system-required instead of merge-check https://review.openstack.org/511748 | 12:11 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Document v3 periodic jobs https://review.openstack.org/511750 | 12:12 |
| odyssey4me | we're also trying to be better citizens - https://review.openstack.org/#/q/topic:infra-log-inode-reduction - apologies for hogging the inodes :/ | 12:12 |
| *** bobh has joined #openstack-infra | 12:13 | |
| *** martinkopec has joined #openstack-infra | 12:14 | |
| *** felipemonteiro has quit IRC | 12:14 | |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| *** baoli has joined #openstack-infra | 12:17 | |
| *** shardy has joined #openstack-infra | 12:19 | |
| *** sree has quit IRC | 12:20 | |
| *** sree has joined #openstack-infra | 12:21 | |
| *** armaan has quit IRC | 12:21 | |
| *** armaan has joined #openstack-infra | 12:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix doc typo that missed important words https://review.openstack.org/509905 | 12:21 |
| *** eharney has joined #openstack-infra | 12:22 | |
| *** slaweq_ has joined #openstack-infra | 12:23 | |
| mordred | odyssey4me: no worries! thanks for cleaning that up! | 12:23 |
| mordred | odyssey4me: (turns out it's a constant learning experience around here) | 12:24 |
| *** LindaWang has joined #openstack-infra | 12:24 | |
| *** martinkopec has quit IRC | 12:24 | |
| odyssey4me | mordred yep, that's what we love about it! | 12:24 |
| *** sree has quit IRC | 12:25 | |
| *** hashar has quit IRC | 12:25 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-.*python{34,35} jobs https://review.openstack.org/511557 | 12:25 |
| *** efried is now known as fried_rice | 12:26 | |
| AJaeger | thanks, mordred! | 12:27 |
| *** trown|outtypewww is now known as trown | 12:28 | |
| AJaeger | odyssey4me: see http://lists.openstack.org/pipermail/openstack-dev/2017-October/123049.html | 12:28 |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| odyssey4me | AJaeger ah, LOG_PATH | 12:28 |
| *** baoli has quit IRC | 12:29 | |
| odyssey4me | tyvm | 12:29 |
| *** yamamoto has quit IRC | 12:29 | |
| *** martinkopec has joined #openstack-infra | 12:29 | |
| *** baoli has joined #openstack-infra | 12:29 | |
| *** ociuhandu has joined #openstack-infra | 12:30 | |
| *** hashar has joined #openstack-infra | 12:32 | |
| *** martinkopec has quit IRC | 12:32 | |
| *** mat128 has joined #openstack-infra | 12:32 | |
| *** baoli has quit IRC | 12:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to ensure per-node output dirs exist https://review.openstack.org/511823 | 12:35 |
| *** gfidente has joined #openstack-infra | 12:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use system-required instead of merge-check https://review.openstack.org/511745 | 12:36 |
| AJaeger | mordred: rebased since a just merged change introduced merge-check again ^. Do you want to +2 again? | 12:36 |
| *** ociuhandu has quit IRC | 12:39 | |
| *** hashar has quit IRC | 12:40 | |
| *** Qiming has quit IRC | 12:41 | |
| mordred | AJaeger: I do! | 12:41 |
| AJaeger | thanks! | 12:41 |
| *** Qiming has joined #openstack-infra | 12:42 | |
| *** dave-mccowan has joined #openstack-infra | 12:42 | |
| *** stakeda has quit IRC | 12:43 | |
| AJaeger | mordred: do we need a rebase of https://review.openstack.org/495426 ? | 12:43 |
| *** jpena is now known as jpena|lunch | 12:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Set zuul_output_dir in site-variables https://review.openstack.org/511827 | 12:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use ensure-output-dirs in base-test https://review.openstack.org/511828 | 12:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use ensure-output-dirs in base https://review.openstack.org/511829 | 12:44 |
| *** hashar has joined #openstack-infra | 12:45 | |
| *** links has quit IRC | 12:45 | |
| *** esberglu has joined #openstack-infra | 12:45 | |
| *** jcoufal has joined #openstack-infra | 12:47 | |
| dmsimard | odyssey4me: thanks for inodes :D | 12:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Mark publish-openstack-artifacts and pipelines as post-review https://review.openstack.org/495426 | 12:47 |
| mordred | AJaeger: ^^ yes! | 12:47 |
| *** dave-mccowan has quit IRC | 12:47 | |
| dmsimard | odyssey4me: fwiw ara is part of zuul v3 by default, perhaps there is something to do there once you move to native v3 jobs -- the equivalent to https://review.openstack.org/#/c/511818/ was implemented in https://review.openstack.org/#/c/511622/ | 12:48 |
| dmsimard | odyssey4me: that said, we'll be exploring different options on how to best serve ara with the least strain on the infrastructure | 12:48 |
| *** pcaruana has joined #openstack-infra | 12:48 | |
| pabelanger | morning | 12:50 |
| *** alexchadin has quit IRC | 12:50 | |
| *** alexchadin has joined #openstack-infra | 12:51 | |
| mordred | odyssey4me, dmsimard: yah - I'm quite interested in what the integration story is for openstack-ansible and v3 ... once the dust settles I have this hunch the openstack-ansible repos are going to be great case studies to figure out how to make the story great for projects that are already natively ansible | 12:51 |
| AJaeger | mordred: there're a couple of more changes by you in project-config, zuul-jobs, and zuul - some of these look needed as well. Any further hidden gems in there that need rebasing and push? | 12:51 |
| mordred | morning pabelanger ! | 12:51 |
| AJaeger | morning, pabelanger ! | 12:51 |
| *** eumel8 has quit IRC | 12:52 | |
| *** tlaxkit has joined #openstack-infra | 12:52 | |
| mordred | AJaeger: that's an excellent quesiton :) | 12:52 |
| dmsimard | mordred: I'm also interested. Interestingly, we (RDO) have an Ansible framework for running gate jobs outside the gate -- tl;dr, it installs dependencies, sets up RDO trunk repos, clones a project such as puppet-openstack-integration and then uses run_tests.sh. With v3 and native ansible, the lines blur quite a bit. | 12:53 |
| mordred | dmsimard: yah. that's all stuff I'd expect v3 to do - especially the cloning repos part | 12:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Removes legacy collectd-ceilometer-plugin jobs https://review.openstack.org/510087 | 12:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove openstack-api-ref jobs https://review.openstack.org/511796 | 12:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy watcher jobs https://review.openstack.org/510848 | 12:54 |
| AJaeger | Same question for pabelanger - just saw https://review.openstack.org/#/c/486199/ and wondered whether we need that... | 12:55 |
| mordred | pabelanger, AJaeger: yah - that does look like a thign that would be nice to merge | 12:55 |
| AJaeger | pabelanger: question was: there're a couple of more changes by you in project-config, zuul-jobs, and zuul - some of these look needed as well. Any further hidden gems in there that need rebasing and push? | 12:56 |
| *** slaweq_ has quit IRC | 12:56 | |
| AJaeger | mordred, pabelanger: Let me rebase 486199... | 12:56 |
| *** spectr has quit IRC | 12:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Limit add-build-sshkey to localhost https://review.openstack.org/486199 | 12:58 |
| AJaeger | done ^ | 12:58 |
| *** ociuhandu has joined #openstack-infra | 12:58 | |
| odyssey4me | mordred yeah, I'm looking forward to thinning out some of these extra layers - right now we do job -> tox -> script -> ansible... we should at least be able to do job (ansible) -> script -> ansible to start with... then see where we can also ditch the script bits if that makes sense | 12:58 |
| odyssey4me | our script bits are usually to get ansible there - we may be able to only use those bits if you're testing outside of zuul | 12:59 |
| odyssey4me | anyway - we'll see... early days | 12:59 |
| mordred | odyssey4me: yah - I also want to explore making some elements of ansible -> ansible richer/better in some manner | 12:59 |
| openstackgerrit | boden proposed openstack-infra/project-config master: zuul v3 gate changes for vmware-nsx https://review.openstack.org/511836 | 13:00 |
| *** slaweq_ has joined #openstack-infra | 13:01 | |
| evrardjp | mordred: odyssey4me maybe in the future, we can rely on directly using our playbooks, and we just set a variable of our python interpreter, and we are good. | 13:01 |
| *** spectr has joined #openstack-infra | 13:01 | |
| evrardjp | future! | 13:01 |
| odyssey4me | evrardjp hmm, that could be interesting - use a pre task to get our ansible there, then flip to doing the run with our venv and our version of ansible | 13:04 |
| odyssey4me | neat idea | 13:04 |
| *** mriedem has joined #openstack-infra | 13:05 | |
| evrardjp | yeah, simple too. | 13:05 |
| evrardjp | basically reproducing work locally just says: "Installs ansible and those dependencies" | 13:05 |
| odyssey4me | evrardjp yeah, and we have run_tests.sh to do that for humans | 13:06 |
| evrardjp | who needs humans when we can have Zuul! | 13:06 |
| evrardjp | we can haz* | 13:06 |
| AJaeger | mordred: abandon https://review.openstack.org/#/c/501475/ ? rebase: https://review.openstack.org/#/c/500615/ and https://review.openstack.org/#/c/500614/ ? | 13:06 |
| *** jistr is now known as jistr|mtg | 13:08 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/infra-specs master: Flat Git Namespace https://review.openstack.org/461878 | 13:11 |
| mordred | AJaeger: abandoned the first - the second two are post-v3 things to discuss with folks whether they like it or not | 13:12 |
| AJaeger | ok | 13:12 |
| AJaeger | Just rechecked an ansible linters job - see http://logs.openstack.org/63/495463/3/infra-check/openstack-tox-linters/0e539b6/job-output.txt.gz | 13:12 |
| AJaeger | ERROR! the role 'fetch-javscript-tarball' was not found in /home/zuul/src/git.openstack.org/openstack-infra/project-config/playbooks/javascript/roles:/home/zuul/src/git.openstack.org/openstack-infra/project-config/.tox/linters/src/zuul-jobs/roles:/home/zuul/src/git.openstack.org/openstack-infra/project-config/playbooks/javascript | 13:12 |
| AJaeger | ah, see the error - will fix! | 13:12 |
| *** ociuhandu has quit IRC | 13:14 | |
| alexchadin | AJaeger: do we also need to add .zuul files to appropriate python clients? | 13:14 |
| AJaeger | alexchadin: you can - if you run legacy jobs, see https://docs.openstack.org/infra/manual/zuulv3.html | 13:14 |
| *** hashar has quit IRC | 13:15 | |
| ttx | fungi: posted a second rev on the flat-git-namespace spec. I listed a number of analysis tasks that I haven't completed yet. One of them is to explore if smart rewriting/redirecting on git.o.o would not spare us from having to fix all those hardcoded references at once. | 13:15 |
| alexchadin | AJaeger: yeah, we have tempest job for client | 13:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix typo in role https://review.openstack.org/511840 | 13:16 |
| *** baoli has joined #openstack-infra | 13:20 | |
| alexchadin | AJaeger: should projects take appropriate rally jobs into their repos or rally team does it into its own repo? | 13:21 |
| alexchadin | AJaeger: they have a lot of project specific jobs | 13:21 |
| AJaeger | alexchadin: something to discuss with rally team ;) Might make sense for them to have all in one place... | 13:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to fetch output from nodes https://review.openstack.org/511843 | 13:22 |
| AJaeger | for now: I suggest you leave them in project-config, move other jobs, convert them - and discuss... | 13:23 |
| AJaeger | No need to move the rally jobs *now* | 13:23 |
| *** kgiusti has joined #openstack-infra | 13:23 | |
| alexchadin | AJaeger: waiting for an answer from rally ptl side :) | 13:24 |
| mordred | alexchadin, AJaeger: yah - I think ultimately having a set of good v3 rally jobs in the rally repo would be the pattern we'd advocate for | 13:25 |
| mordred | similarly - we're aiming to have base devstack jobs in the devstack repo and then base tempest jobs in the tempest repo | 13:26 |
| AJaeger | mordred: something to document? ^ | 13:27 |
| mordred | AJaeger: yah - probably so ... there is a difference between "you can define jobs in your own repo - andyou can also use jobs from someone else's repo" and "we recommend that projects like tempest and rally define jobs in their repos that other projects can consume" | 13:28 |
| AJaeger | mordred: yes, there is - I know. We could still say that projects can define jobs for multiple projects etc. - without naming them. | 13:29 |
| *** baoli has quit IRC | 13:30 | |
| *** smatzek has quit IRC | 13:30 | |
| *** yamamoto has joined #openstack-infra | 13:30 | |
| *** nikhil_k has joined #openstack-infra | 13:30 | |
| mordred | AJaeger: oh - no, sorry - I was saying I think we shoujld spell out the second thing because people might not infer it correctly just from documenting the first thing | 13:31 |
| *** baoli has joined #openstack-infra | 13:31 | |
| mordred | the first is 'you can do this' - the second is 'you should do this' | 13:31 |
| *** Qiming has quit IRC | 13:31 | |
| *** baoli has quit IRC | 13:32 | |
| *** chlong has joined #openstack-infra | 13:33 | |
| *** slaweq has quit IRC | 13:33 | |
| AJaeger | ah, I see ;) /me should not multi-task... | 13:33 |
| AJaeger | bbl | 13:33 |
| *** Qiming has joined #openstack-infra | 13:33 | |
| *** slaweq_ has quit IRC | 13:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add fetch-output to base-test https://review.openstack.org/511850 | 13:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add fetch-output to base job https://review.openstack.org/511851 | 13:36 |
| *** baoli has joined #openstack-infra | 13:39 | |
| *** yamamoto has quit IRC | 13:40 | |
| *** slaweq has joined #openstack-infra | 13:41 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** armaan has quit IRC | 13:41 | |
| *** armaan has joined #openstack-infra | 13:42 | |
| odyssey4me | mordred what's the path where the git repositories are put by v3? | 13:42 |
| odyssey4me | z-c (v2) had /opt/git ... IIRC the path in zuulv3 is something along the line of /home/zuul/src/openstack.org/<repo> ? | 13:43 |
| odyssey4me | oh, that'd be <org>/<repo> | 13:44 |
| *** rcernin has quit IRC | 13:44 | |
| dmsimard | AJaeger: I think https://review.openstack.org/#/c/504788/ is good to go | 13:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update upload-logs to process docs as well https://review.openstack.org/511853 | 13:44 |
| AJaeger | dmsimard: wanted jeblair give a chance to review 504788 again - but if he doesn't manage it today, I'll do again... | 13:45 |
| dmsimard | works for me | 13:45 |
| dmsimard | mordred: commented on https://review.openstack.org/#/c/511843/ | 13:45 |
| *** andreas_s has joined #openstack-infra | 13:47 | |
| *** kaisers has joined #openstack-infra | 13:47 | |
| *** dansmith is now known as superdan | 13:47 | |
| *** d0ugal has quit IRC | 13:48 | |
| *** dave-mccowan has joined #openstack-infra | 13:49 | |
| *** gouthamr has joined #openstack-infra | 13:49 | |
| *** yamamoto has joined #openstack-infra | 13:50 | |
| *** gouthamr has quit IRC | 13:51 | |
| *** andreas_s has quit IRC | 13:51 | |
| *** gouthamr has joined #openstack-infra | 13:51 | |
| *** dave-mcc_ has joined #openstack-infra | 13:55 | |
| *** dave-mccowan has quit IRC | 13:57 | |
| *** smatzek has joined #openstack-infra | 13:59 | |
| *** smatzek has quit IRC | 13:59 | |
| *** smatzek has joined #openstack-infra | 13:59 | |
| *** smatzek has quit IRC | 13:59 | |
| *** eumel8 has joined #openstack-infra | 14:00 | |
| *** smatzek has joined #openstack-infra | 14:00 | |
| *** bnemec is now known as beekneemech | 14:00 | |
| pabelanger | mordred: AJaeger: re: 486199 I don't think we need that any more, it was more of an optimization. So, I can WIP for now, and maybe revisit in the future | 14:01 |
| AJaeger | pabelanger: sure | 14:01 |
| *** mriedem has quit IRC | 14:03 | |
| *** ociuhandu has joined #openstack-infra | 14:03 | |
| pabelanger | ianw: jeblair: great work on ubuntu mirror | 14:03 |
| *** d0ugal has joined #openstack-infra | 14:03 | |
| pabelanger | mordred: looks like new shade was released, did we want to restart nodepool-builders? | 14:04 |
| *** aperri has quit IRC | 14:05 | |
| AJaeger | pabelanger: let's first debug why we have an infra-post job for 8+ hours... | 14:05 |
| *** andreas_s has joined #openstack-infra | 14:05 | |
| AJaeger | (or at least discuss that;) | 14:05 |
| pabelanger | looking | 14:05 |
| *** claudiub has quit IRC | 14:06 | |
| * AJaeger needs to step out for a bit... | 14:06 | |
| pabelanger | wow | 14:06 |
| *** mriedem has joined #openstack-infra | 14:06 | |
| pabelanger | we have 31 held debian-jessie nodes | 14:06 |
| pabelanger | for zuulv3 | 14:06 |
| pabelanger | I suspect, we no longer have capicity for launch the needed node for the job | 14:07 |
| pabelanger | ya, I suspect nodepool v3 is wedged | 14:08 |
| pabelanger | we have plenty of ready nodes for xenial | 14:08 |
| *** trown is now known as trown|brb | 14:08 | |
| pabelanger | Yah, for some reason the autohold I did for debian-jessie last night, has held 31 instances of debian-jessie | 14:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Upload fetched artifacts to log server https://review.openstack.org/511858 | 14:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only publish artifacts when zuul.change is not defined https://review.openstack.org/511859 | 14:09 |
| pabelanger | Shrews: ^do you have a moment to help debug why? | 14:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Handle basic artifacts in base-test job https://review.openstack.org/511860 | 14:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Process artifacts in base and base-minimal https://review.openstack.org/511861 | 14:09 |
| Shrews | pabelanger: looking | 14:10 |
| *** armax has joined #openstack-infra | 14:11 | |
| *** alexchadin has quit IRC | 14:12 | |
| Shrews | pabelanger: they are held because you've put an autohold on somewhere | 14:12 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/infra-specs master: Flat Git Namespace https://review.openstack.org/461878 | 14:12 |
| Shrews | pabelanger: the only comment is "pabelanger" so that's all the info i have | 14:12 |
| pabelanger | Shrews: ya, I put an autohold o zuulv3 last night | 14:13 |
| pabelanger | but, I assumed it was for 1 node | 14:13 |
| *** slaweq has quit IRC | 14:13 | |
| pabelanger | Shrews: Did I have to delete my autohold? | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to fetch output from nodes https://review.openstack.org/511843 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update upload-logs to process docs as well https://review.openstack.org/511853 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Upload fetched artifacts to log server https://review.openstack.org/511858 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only publish artifacts when zuul.change is not defined https://review.openstack.org/511859 | 14:13 |
| Shrews | pabelanger: what command did you use? | 14:13 |
| pabelanger | 1 sec | 14:13 |
| pabelanger | sudo zuul autohold --tenant openstack --project openstack-infra/openstack-zuul-jobs --job base-integration-debian-jessie --reason pabelanger | 14:14 |
| pabelanger | that was on zuulv3.o.o | 14:14 |
| *** trown|brb is now known as trown | 14:15 | |
| *** hongbin has joined #openstack-infra | 14:16 | |
| pabelanger | mordred: https://review.openstack.org/511656/ when you have a moment | 14:16 |
| Shrews | pabelanger: that would hold any nodes requested by that job for 1 run of that job. i don't know how many nodes that job uses, but you shouldn't need to delete your autohold (unless you specified a count arg). but you can try if you want using --count=0 | 14:18 |
| Shrews | pabelanger: but you'll need to manually delete those nodes | 14:18 |
| pabelanger | Shrews: ya, that's how I understood it too. But, not sure why autohold held more then 1 node, in fact 31 :) | 14:19 |
| *** Qiming has quit IRC | 14:19 | |
| erlon | @guys, I'm trying to increase the amount of log my CI can store. After systemd, all devstack logfiles are trimmed and usually we loose a lot of useful information. Does anyone knows how to increase that? | 14:19 |
| pabelanger | Shrews: okay, I can delete them now, but is that something we want to add to zuulv3-issues and debug with you and jeblair? | 14:19 |
| *** Qiming has joined #openstack-infra | 14:19 | |
| pabelanger | Shrews: before I delete, I'd also wouldn't mind you seeing my 100-0000416513 has been pending for some time | 14:21 |
| *** huanxie has joined #openstack-infra | 14:21 | |
| *** yamamoto has quit IRC | 14:21 | |
| *** sree has joined #openstack-infra | 14:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use system-required instead of merge-check https://review.openstack.org/511745 | 14:22 |
| *** slaweq has joined #openstack-infra | 14:23 | |
| *** andreas_s has quit IRC | 14:23 | |
| pabelanger | Shrews: okay, looks like 100-0000416513 has been stuck, retrying in citycloud-kna1 | 14:23 |
| pabelanger | which, doesn't have quota any more | 14:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Don't use in-line yaml lists https://review.openstack.org/511876 | 14:24 |
| tinwood | sorry, very quick question to anyone you knows. Is zuul v3 running, or is it back to zuul v2? Sorry, if this a dumb question; not sure where to look to see what is the version. e.g. status.openstack.org/zuul/ shows v2.6.0 which means I'm guessing v2 is running. | 14:24 |
| pabelanger | tinwood: yah, we are still on zuulv2.5 ATM. | 14:25 |
| pabelanger | tinwood: we're looking to start rollout again, this coming Sunday 22:00 UTC | 14:25 |
| *** sree has quit IRC | 14:25 | |
| tinwood | pabelanger, thanks for the update. | 14:26 |
| Shrews | pabelanger: yeah, no quota. but nodes are deleting | 14:27 |
| pabelanger | Shrews: okay, and we'd never release the request to another provider right? | 14:28 |
| *** andreas_s has joined #openstack-infra | 14:28 | |
| Shrews | pabelanger: nope | 14:28 |
| AJaeger | jeblair: thanks for fixing my typos and grammar! | 14:28 |
| *** egonzalez has quit IRC | 14:29 | |
| AJaeger | pabelanger: but the debian jessie is not the note the post job needs... | 14:29 |
| *** ramishra has quit IRC | 14:29 | |
| pabelanger | AJaeger: mordred: ^ see discussion with Shrews. We're basically blocked on infra-post job, until we get more quota in citycloud-kna1 | 14:29 |
| *** yamahata has joined #openstack-infra | 14:30 | |
| *** rbrndt has joined #openstack-infra | 14:31 | |
| pabelanger | I admit, I'd don't fully understand why we try forever for a single provider, but seems like we are more fragile to getting into these blocking scenarios. Where, if we know we no longer have quota, why not release the request for another provider to try? | 14:31 |
| pabelanger | because, we do have other providers with ready xenial nodes, but not getting used | 14:31 |
| jeblair | pabelanger: why is there no quota? | 14:32 |
| Shrews | jeblair: out of 10 total, 8 are deleting, 1 in hold, 1 is ready but of the wrong label | 14:33 |
| pabelanger | we have 8 nodes in deleting state, 1 held and 1 ready currently in kna1, which gets us 10 max-servers | 14:33 |
| jeblair | are the deleting nodes stuck? | 14:33 |
| pabelanger | I think they are | 14:34 |
| pabelanger | looking at logs now | 14:34 |
| Shrews | the deletes are timing out | 14:34 |
| *** Dinesh_Bhor has quit IRC | 14:34 | |
| pabelanger | yah, see that now | 14:35 |
| Shrews | no idea why. currently searching scheduler logs for a hold issue, so if someone else could look at the timeout, that would be helpful | 14:35 |
| *** dave-mcc_ is now known as dave-mccowan | 14:35 | |
| *** andreas_s has quit IRC | 14:36 | |
| *** cshastri has quit IRC | 14:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove merge-check https://review.openstack.org/511747 | 14:38 |
| pabelanger | Hm | 14:38 |
| pabelanger | 3432242d-846c-4b5a-9fd8-309edb280cd5 in citycloud-kna1, is currently BUILD state | 14:39 |
| pabelanger | and see when we send a delete, it is not affecting it | 14:39 |
| pabelanger | however, citycloud-kna1 is working currently. I can see nodepool v2 launching servers | 14:40 |
| *** yamamoto has joined #openstack-infra | 14:41 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Support pike->master undercloud upgrades https://review.openstack.org/511885 | 14:42 |
| *** caphrim007_ has quit IRC | 14:43 | |
| *** rcernin has joined #openstack-infra | 14:45 | |
| *** LindaWang has quit IRC | 14:45 | |
| *** spectr has quit IRC | 14:46 | |
| *** LindaWang has joined #openstack-infra | 14:46 | |
| Shrews | jeblair: hrm, did we have a recent gear release maybe? Seeing this for autohold jobs: http://paste.openstack.org/show/623582/ | 14:47 |
| Shrews | oh, that may not be gear related | 14:47 |
| *** e0ne has quit IRC | 14:47 | |
| Shrews | grrr | 14:47 |
| jeblair | Shrews: we did have a recent gear release, but it only had one change, and that was a send mutex | 14:47 |
| *** bobh has quit IRC | 14:48 | |
| Shrews | jeblair: yeah, i saw JSON decode error and asked too quickly | 14:49 |
| *** claudiub has joined #openstack-infra | 14:50 | |
| *** andreas_s has joined #openstack-infra | 14:50 | |
| *** huanxie has quit IRC | 14:51 | |
| *** slaweq has quit IRC | 14:56 | |
| AJaeger | mordred: do you want to abandon https://review.openstack.org/#/c/508767/ - seeing jeblair's comment? | 14:56 |
| *** jistr|mtg is now known as jistr | 14:58 | |
| Shrews | pabelanger: I think I see the bug causing the autohold issue (though I don't understand the underlying cause yet). I believe you'll need to do the autohold clear (using --count=0) to stop this from recurring for now | 14:59 |
| fungi | tonyb: ianw: oh, wow, sorry about that... i knew we'd switched to linking cgit instead of gitweb in the gerrit webui but didn't realize we'd actually disabled access to the gitweb interface on review.o.o entirely. i agree that'll break a number of scripts which were depending on it (especially ones which want to retrieve things like configuration from the all-projects "repo" which we don't mirror | 15:00 |
| fungi | anywhere) | 15:00 |
| *** vhosakot has joined #openstack-infra | 15:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Re-enable v3 pipelines https://review.openstack.org/511899 | 15:01 |
| AJaeger | team, here's change to enable the pipelines again once we're ready to start ^ | 15:02 |
| pabelanger | Shrews: k, let me do that now | 15:02 |
| pabelanger | Shrews: I can then start the clean up process for debian-jessie | 15:02 |
| *** rcernin has quit IRC | 15:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update upload-logs to process docs as well https://review.openstack.org/511853 | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Upload fetched artifacts to log server https://review.openstack.org/511858 | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only publish artifacts when zuul.change is not defined https://review.openstack.org/511859 | 15:03 |
| *** slaweq has joined #openstack-infra | 15:03 | |
| mordred | dmsimard: thanks for the reviews! | 15:03 |
| mordred | AJaeger: looking | 15:03 |
| clarkb | fungi: we didnt so much disable gitweb as it completely stopped working :( | 15:04 |
| *** andreas_s has quit IRC | 15:04 | |
| pabelanger | Shrews: okay, held nodes have been deleted | 15:05 |
| mordred | pabelanger: re: BUILDING/delete -- yah, deleting a node in building state is unpossible | 15:07 |
| clarkb | erlon: the systemd switch in devstack shouldnt have changed what you are logging if you are using devstack-gate. It grabs the service logs out of journalctl | 15:08 |
| clarkb | erlon: generally though look to journalctl to find the logs | 15:08 |
| *** mpaolino has quit IRC | 15:08 | |
| *** andreas_s has joined #openstack-infra | 15:09 | |
| *** huanxie has joined #openstack-infra | 15:09 | |
| erlon | clarkb: yeap, but by default systemd uses Nopersistent logging, which I think is the reason why our logs are so short (2M vs 15M on infra) | 15:10 |
| erlon | clarkb: may be is there any configuration on your nodepool scripts to set that? | 15:10 |
| mordred | what is nopersistent? | 15:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Better exception handling during autohold https://review.openstack.org/511902 | 15:12 |
| Shrews | jeblair: pabelanger: ^^ | 15:12 |
| erlon | mordred: does not use disk to store logs, saves on /run | 15:12 |
| clarkb | erlon: our image builds force systemd to write to disk rather than its ring buffer | 15:12 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 15:13 |
| clarkb | erlon: you can create the dir it looks for or change the config from auto to persistent | 15:13 |
| erlon | clarkb: hmm, ok, other than that is there anything else? | 15:13 |
| chandankumar | AJaeger: tosky is this correct now https://review.openstack.org/#/c/511194/ ? | 15:13 |
| clarkb | no thats it | 15:13 |
| *** andreas_s has quit IRC | 15:13 | |
| chandankumar | mordred: hello | 15:13 |
| erlon | clarkb: good, ill try that | 15:13 |
| tosky | chandankumar: remove also publish-openstack-python-branch-tarball, it is part of openstack-python-jobs | 15:14 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 15:15 |
| *** smarcet has joined #openstack-infra | 15:15 | |
| smarcet | fungi: morning how are u doing ? :) just wanna let u know that this one https://review.openstack.org/#/c/497616/ seems that got stuck | 15:16 |
| chandankumar | mordred: on review.openstack.org people are not able to add me as a reviewer | 15:16 |
| chandankumar | mordred: and on this https://review.openstack.org/#/admin/groups/1842,members and also i am not able add new members here | 15:17 |
| chandankumar | mordred: ianw found that on my email-id <chkumar@redhat.com> there are two account, that's why people are not able to add me as a reviewer | 15:17 |
| chandankumar | mordred: my Account ID 12393 is this please have a look, thanks :-) | 15:18 |
| chandankumar | tosky: done | 15:18 |
| *** LindaWang1 has joined #openstack-infra | 15:20 | |
| *** LindaWang has quit IRC | 15:22 | |
| *** alex_xu has quit IRC | 15:22 | |
| *** LindaWang1 is now known as LindaWang | 15:22 | |
| *** florianf has quit IRC | 15:23 | |
| *** florianf has joined #openstack-infra | 15:23 | |
| *** annp has joined #openstack-infra | 15:24 | |
| *** LindaWang has quit IRC | 15:26 | |
| rajinir | Setting up new workspace by the builds is taking an hour now. Used to take 3-5 mins. What could be the cause? | 15:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Better exception handling during autohold https://review.openstack.org/511902 | 15:26 |
| rajinir | https://www.irccloud.com/pastebin/iIslATLA/Settung%20up%20workspace | 15:27 |
| clarkb | chandankumar: I'll take a look in a few minutes, need to get to real keyboard | 15:27 |
| *** LindaWang has joined #openstack-infra | 15:27 | |
| clarkb | rajinir: you should look in the workspace setup log to see where it spent its time | 15:27 |
| clarkb | rajinir: is this from an infra run job or third party ci? | 15:28 |
| rajinir | clarkb: Thirdparty CI | 15:28 |
| *** Qiming has quit IRC | 15:29 | |
| rajinir | clarkb: thanks let me check the logs | 15:30 |
| *** alex_xu has joined #openstack-infra | 15:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 15:32 |
| *** Qiming has joined #openstack-infra | 15:33 | |
| *** fanzhang_ has joined #openstack-infra | 15:34 | |
| AJaeger | anybody for a quick +3 ^ - I just added a newline | 15:34 |
| chandankumar | AJaeger: thanks you fix it :-) | 15:34 |
| *** smatzek has quit IRC | 15:35 | |
| dmsimard | AJaeger: hm, I wonder if we should be doing this at all or if we should instead make it go straight to v3 | 15:35 |
| dmsimard | AJaeger: once we switch to v3, this project won't have any legacy jobs. | 15:35 |
| AJaeger | dmsimard: wait, wait - that change is exactly the policy we want ;) | 15:36 |
| AJaeger | dmsimard: read https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 15:36 |
| AJaeger | dmsimard: there's no legacy job in that one | 15:36 |
| *** slaweq has quit IRC | 15:36 | |
| dmsimard | AJaeger: first time I see system-required o_O /me searches | 15:37 |
| AJaeger | dmsimard: we merged it today ;) | 15:37 |
| dmsimard | oh no wonder then | 15:37 |
| AJaeger | dmsimard: new name for merge-check | 15:37 |
| dmsimard | AJaeger: oh, okay | 15:37 |
| AJaeger | dmsimard: we want to have it easy to add more to it later | 15:37 |
| AJaeger | dmsimard: see backscroll from 17hours earlier | 15:37 |
| dmsimard | :( | 15:37 |
| *** smatzek has joined #openstack-infra | 15:38 | |
| *** dtantsur is now known as dtantsur|afk | 15:39 | |
| *** huanxie has quit IRC | 15:40 | |
| pabelanger | mordred: jeblair: clarkb: AJaeger: Now that we have an idea of the infra-post jobs being stuck, I'd like to see if we are ready to do a restart on nb04.o.o for new shade release and rax uploads | 15:41 |
| *** slaweq has joined #openstack-infra | 15:41 | |
| *** annp has quit IRC | 15:41 | |
| dmsimard | AJaeger, chandankumar: +1 in https://review.openstack.org/#/c/511194/ but added a comment | 15:43 |
| clarkb | dmsimard: did you see I think I got ara on failure tested last night and it is working. So next step is to add it to base if that hasnt happened yet | 15:44 |
| dmsimard | clarkb: yup, I saw. Let me see if it's in base yet, otherwise I can submit it. | 15:44 |
| tosky | dmsimard: no jobs defined? I thought that adding the template you get the jobs defined there automatically | 15:45 |
| Shrews | pabelanger: did you do anything for cleaning up the instances stuck 'deleting'? | 15:45 |
| dmsimard | tosky: the jobs that are being added are zuul v2 jobs, which will work and run today (or whenever it merges) | 15:46 |
| dmsimard | tosky: however, since the v3 "migration script" has already ran and we won't be running it again, these v2 jobs won't be migrated to v3 | 15:46 |
| pabelanger | Shrews: only clean up was when I deleted held debian-jessie node | 15:46 |
| pabelanger | that opened quota for xenial to build | 15:46 |
| pabelanger | however, we still have stuck nodes in delete state | 15:46 |
| dmsimard | tosky: therefore you need to setup a zuul.yaml file in python-tempestconf with your jobs and layout there | 15:47 |
| *** bobh has joined #openstack-infra | 15:47 | |
| tosky | dmsimard: where is openstack-python-jobs defined? | 15:47 |
| dmsimard | tosky: there is an equivalent template for that in v3, hang on | 15:47 |
| Shrews | pabelanger: and those are because we tried to delete them while they were building? | 15:47 |
| *** dhajare has quit IRC | 15:47 | |
| pabelanger | Shrews: yah, according to mordred, which will never delete | 15:48 |
| pabelanger | so, should like we might need to get citycloud to help | 15:48 |
| tosky | dmsimard: doesn't zuul.d/project.yaml contain zuul v3 jobs definitions? Otherwise the legacy- jobs defined there wouldn't make sense | 15:48 |
| dmsimard | tosky: there is openstack-python35-jobs and openstack-python-jobs (py27+pep8) | 15:49 |
| tosky | dmsimard: and that review adds openstack-python-jobs | 15:49 |
| dmsimard | tosky: the jobs exists, they just aren't applied to openstack/python-tempestconf | 15:49 |
| dmsimard | tosky: that adds it for zuul v3 | 15:50 |
| dmsimard | zuulv2*** | 15:50 |
| dmsimard | oh fuck | 15:50 |
| dmsimard | it adds it to zuul v3 | 15:50 |
| * dmsimard hides | 15:50 | |
| tosky | exactly :) | 15:50 |
| pabelanger | but, like I asked above, couldn't we release the request to another provider if we know quota is full? And have provider B fullfill the request? In the case this morning, we had plenting of xenial nodes available over all | 15:50 |
| tosky | I went through that already, so I know the feeling | 15:50 |
| dmsimard | tosky: yeah.. I'll blame that on friday and lack of caffeine. | 15:51 |
| dmsimard | tosky: sorry if I confused you, I was totally off track | 15:52 |
| *** chlong has quit IRC | 15:53 | |
| tosky | dmsimard: no, no problem, really, better clarify this now | 15:53 |
| tosky | next week it will be (another) roller coaster | 15:53 |
| * AJaeger sends coffee and weekend to dmsimard ;) | 15:53 | |
| fungi | clarkb: oh, so the gitweb in 2.13 was basically broken and that's what urged us to switch our linking to git.o.o? that makes more sense but i guess i had forgotten it already | 15:53 |
| tosky | speaking of which... what is the current plan for the re-deployment of zuul v3? Monday? | 15:53 |
| dmsimard | AJaeger: weekend, yes, please send me more weekends | 15:53 |
| dmsimard | tosky: not today :D | 15:53 |
| *** caphrim007 has joined #openstack-infra | 15:53 | |
| tosky | definitely :D | 15:54 |
| dmsimard | tosky: so far it's looking like sunday, there'll be an email sent out I suspect. | 15:54 |
| AJaeger | pabelanger: go ahead! | 15:54 |
| *** Qiming has quit IRC | 15:55 | |
| fungi | smarcet: thanks for the heads up, i've rechecked it so hopefully will merge shortly | 15:55 |
| clarkb | fungi: ya | 15:55 |
| fungi | i'll keep a closer eye on it this time, sorry! | 15:55 |
| *** LindaWang has quit IRC | 15:56 | |
| *** Qiming has joined #openstack-infra | 15:57 | |
| clarkb | chandankumar: when you login what account id shows up at https://review.openstack.org/#/settings/ is it 12393? | 15:58 |
| *** lucasagomes is now known as lucas-afk | 15:58 | |
| clarkb | chandankumar: or is it 8944? | 15:59 |
| *** Apoorva has joined #openstack-infra | 16:00 | |
| clarkb | chandankumar: it looks like you may have ended up using both accounts? If you can clarify which one is the one you are currently using and whether or not that is the one you would like to continue using that would be helpful | 16:01 |
| pabelanger | okay, stopping nb04 to pickup latest shade | 16:01 |
| *** Apoorva has quit IRC | 16:01 | |
| fungi | clarkb: one common pattern there too is users who end up submitting changes with the username/ssh keys on their earlier account but logging into the webui and reviewing with their later account | 16:02 |
| Shrews | pabelanger: i'm going to add a check to nodepool to not try to delete an instance that is building, but yeah, we might need provider help to cleanup the ones that are still building | 16:03 |
| clarkb | fungi: chandankumar interesting, both accounts have an ssh username set | 16:03 |
| Shrews | pabelanger: oh, wait plz | 16:03 |
| clarkb | chandankumar: so maybe you can clarify which ssh user you are using as well | 16:03 |
| fungi | clarkb: but also sometimes they end up switching usernames with the new account, so double-checking both the account id number showing in the webui _and_ the ssh username they're using to submit changes can be useful | 16:03 |
| Shrews | pabelanger: i want to watch builder logs but i'm stuck doing too much at the same time. gimme a minute? | 16:03 |
| *** reed has quit IRC | 16:04 | |
| pabelanger | Shrews: sorry, builder already stopped | 16:04 |
| Shrews | pabelanger: ok, go ahead and start it | 16:05 |
| pabelanger | Shrews: is it builder you want or nodepool-launcher? | 16:05 |
| Shrews | pabelanger: builder. that's where i discovered the shade bug | 16:05 |
| pabelanger | kk | 16:05 |
| pabelanger | Hmm | 16:05 |
| pabelanger | i see a lot of open mounts from DIB | 16:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Recommendations and examples for v3 job usage https://review.openstack.org/511915 | 16:05 |
| AJaeger | mordred: I tried it myself ^ | 16:05 |
| pabelanger | let me see if I can manually clean them up | 16:05 |
| pabelanger | ianw: when you are back, we might want to audit nb03 and see why we are leaving mounts from dib^ | 16:06 |
| clarkb | chandankumar: to summarize I think there are at least two pieces of info that we need to untangle which account to use going forward. First is what account id yo usee in the web ui settings page and second what ssh user you use to push code. And so that its written down somewhere the two accounts are 8944 and 12393 | 16:06 |
| *** reed has joined #openstack-infra | 16:07 | |
| Shrews | mordred: fyi, nodepool builder being restarted with newest shade. | 16:07 |
| pabelanger | mordred: Shrews: I think it might be faster to reboot nb04 then manually unmount all the open mount points from dib | 16:08 |
| pabelanger | any objections? | 16:08 |
| pabelanger | infra-root^ | 16:08 |
| Shrews | pabelanger: ok, i'm off | 16:08 |
| clarkb | dmsimard: after a quick skim of project-config changes I'm not seeing one for doing ara on failure on base. | 16:09 |
| *** makowals has quit IRC | 16:09 | |
| *** baoli_ has joined #openstack-infra | 16:09 | |
| pabelanger | okay, rebooting nb04 | 16:10 |
| clarkb | if we can get that in I think the next step is considering reenabling check/periodic on v3 | 16:10 |
| AJaeger | clarkb: https://review.openstack.org/511899 is the patch for that | 16:10 |
| clarkb | AJaeger: ty | 16:10 |
| pabelanger | Shrews: okay, back | 16:11 |
| *** Apoorva has joined #openstack-infra | 16:11 | |
| *** baoli__ has joined #openstack-infra | 16:12 | |
| *** kjackal_ has quit IRC | 16:12 | |
| *** ykarel is now known as ykarel|afk | 16:12 | |
| *** kjackal_ has joined #openstack-infra | 16:12 | |
| *** baoli has quit IRC | 16:13 | |
| *** claudiub has quit IRC | 16:13 | |
| *** slaweq has quit IRC | 16:14 | |
| *** baoli_ has quit IRC | 16:15 | |
| pabelanger | OpenStackCloudException: Multiple matches found for ubuntu-trusty-1507781835 | 16:15 |
| *** yamahata has quit IRC | 16:15 | |
| pabelanger | that was same as yesterday | 16:15 |
| pabelanger | and think mordred suggested we write a shade script to clean up leaked images? | 16:16 |
| AJaeger | clarkb: I squeezed the three changes we did for disabling together into one in 511899 | 16:16 |
| *** lnxnut_ has quit IRC | 16:16 | |
| *** slaweq has joined #openstack-infra | 16:16 | |
| mordred | pabelanger: yah - I can do that real quick | 16:16 |
| *** slaweq has quit IRC | 16:18 | |
| *** slaweq has joined #openstack-infra | 16:18 | |
| *** ykarel|afk has quit IRC | 16:20 | |
| *** kiennt26 has joined #openstack-infra | 16:20 | |
| dmsimard | clarkb: I thought I had sent a change for base failure for ara, guess not. I'll send one now. | 16:21 |
| *** pcaruana has quit IRC | 16:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Do not delete instances that are 'building' https://review.openstack.org/511917 | 16:25 |
| *** huanxie has joined #openstack-infra | 16:26 | |
| Shrews | pabelanger: i believe that ^^^ will prevent us from trying to delete a building instance until it changes status. I need a test for it still. | 16:26 |
| *** kjackal_ has quit IRC | 16:27 | |
| *** felipemonteiro has joined #openstack-infra | 16:27 | |
| *** jpich has quit IRC | 16:28 | |
| *** fried_rice is now known as fried_rice_injer | 16:28 | |
| *** fried_rice_injer is now known as friedrice_injera | 16:29 | |
| *** yamamoto has quit IRC | 16:29 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Only generate ARA reports on failure for base jobs https://review.openstack.org/511918 | 16:32 |
| *** felipemonteiro has quit IRC | 16:32 | |
| *** apetrich has quit IRC | 16:32 | |
| dmsimard | infra-root: ^ enabling limited report generation for v3 jobs as per discussed | 16:32 |
| *** apetrich has joined #openstack-infra | 16:32 | |
| pabelanger | Shrews: I'll defer to both jeblair and mordred on 511917. I'm not sure how best to handle it | 16:33 |
| *** derekh has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Configure OVB jobs to use local mirrors for images https://review.openstack.org/511434 | 16:33 |
| *** xarses has joined #openstack-infra | 16:33 | |
| Shrews | pabelanger: k, but if deletes for a building instance get lost in the ether, then without that code, we'll have deleter threads that stick around forever (or until a restart) | 16:34 |
| smarcet | fungi: thx u :) | 16:34 |
| *** snapiri has quit IRC | 16:34 | |
| *** rpittau has quit IRC | 16:34 | |
| fungi | smarcet: yw, looks like it should merge at any moment now | 16:35 |
| *** fanzhang_ has quit IRC | 16:35 | |
| openstackgerrit | Merged openstack-infra/system-config master: OpenStackID Release 1.0.23 https://review.openstack.org/497616 | 16:35 |
| mordred | pabelanger, Shrews: I have written a script in ~nodepool on nl01.openstack.org called cleanup.py | 16:36 |
| *** dhajare has joined #openstack-infra | 16:36 | |
| pabelanger | mordred: looks good, did you want to run it now? | 16:39 |
| *** camunoz has joined #openstack-infra | 16:41 | |
| Shrews | pabelanger: hrm, actually, i think the delete may timeout and we'd try to delete again later, which makes my change unnecessary (though slightly more efficient). I might abandon it | 16:42 |
| mordred | pabelanger: biggest question - it will delete all copies of an image that has duplicates - which might leave us without a particular image on a cloud - I think that should be ok thought, right? | 16:44 |
| Shrews | that is, in fact, what it's currently doing | 16:44 |
| fungi | Shrews: i manually delete instances in a building state semi-regularly, so at least some deployments work just fine that way | 16:44 |
| fungi | and waiting into it reaches active before issuing the delete would therefore be at least marginally less efficient in those | 16:45 |
| fungi | s/into/until | 16:45 |
| *** trown is now known as trown|lunch | 16:45 | |
| pabelanger | mordred: yah, I think so. It would be good to audit rackspace again after we run your script to see if any other leaked images are found | 16:46 |
| mordred | pabelanger: kk. I run now | 16:46 |
| Shrews | fungi: manual intervention is needed for these in citycloud-kna1 | 16:48 |
| fungi | that's pretty wacky. i wonder if they got stuck for "other" reasons | 16:49 |
| fungi | and not simply because we tried to delete them while they were building | 16:49 |
| Shrews | fungi: there are 8. | 16:49 |
| *** jtomasek has quit IRC | 16:49 | |
| Shrews | fungi: all fedora-26, interestingly | 16:49 |
| fungi | i've seen that happen when new instances get scheduled to a broken compute node | 16:50 |
| fungi | i wonder if whatever was broken was related to that particular image (maybe with nova's image cache?) | 16:50 |
| pabelanger | fungi: yah, I am thinking a hypervisor issue myself | 16:50 |
| fungi | though probably pointless to speculate | 16:50 |
| pabelanger | we should reach out to citycloud however | 16:50 |
| Shrews | i have to afk for a bit | 16:51 |
| pabelanger | Shrews: mordred just seen a new (for me) exception on nb04.o.o debug.log | 16:51 |
| pabelanger | when you have time | 16:51 |
| fungi | basically you have one compute node with something wrong with the particular cached image, where any time you ask it to boot that it never exits building... then it becomes a tarpit slowly accumulating broken building instances over some span of time | 16:51 |
| fungi | until eventually most of your quota is taken up by them | 16:52 |
| mordred | pabelanger: awesome! | 16:52 |
| clarkb | can we get reviews on https://review.openstack.org/#/c/511918/1 to only run ara on failure, then we should consider https://review.openstack.org/511899 to reenable v3 pipelines | 16:53 |
| pabelanger | looking | 16:54 |
| *** huanxie has quit IRC | 16:56 | |
| mordred | pabelanger: OpenStackCloudHTTPError: (409) Client Error for url: https://sto2.citycloud.com:9292/v2/images/4a8ccead-b90a-47a7-a968-178f2ec79994/file Conflict (Inner Exception: Expecting value: line 1 column 1 (char 0)) | 16:56 |
| pabelanger | clarkb: left comment | 16:56 |
| mordred | pabelanger: is that the new exception of which you speak? | 16:56 |
| pabelanger | mordred: yah, that's it | 16:56 |
| mordred | kk. thanks | 16:56 |
| pabelanger | dmsimard: ^comment on 511899 | 16:57 |
| clarkb | dmsimard: see pabelanger's comment | 16:57 |
| *** wolverineav has joined #openstack-infra | 16:58 | |
| clarkb | pabelanger: dmsimard I could imagine there may be cases where overriding that to true could be useful for some reason, I doubt projects would go overriding it, but wouln't be the first time I've been suprirsed by job configs :) | 16:58 |
| *** epende has quit IRC | 16:59 | |
| pabelanger | clarkb: dmsimard: ya, I think we should just set it at http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml#n5 just to be safe, then we don't have to find jobs and track down extra inode usage again | 16:59 |
| dmsimard | clarkb: ara runs by default and requires to be trusted, we can't run "manually" in base-minimal integration tests. | 16:59 |
| pabelanger | this is only short term any ways | 16:59 |
| clarkb | dmsimard: one of the things that make base-minimal minimal is it does not run ara | 17:00 |
| dmsimard | I'm not really worried about singular jobs overriding the config, if anything, it leaves us an easy way to test things | 17:00 |
| *** AJaeger has left #openstack-infra | 17:00 | |
| *** AJaeger has quit IRC | 17:00 | |
| clarkb | dmsimard: so setting the var there is mostly noop | 17:00 |
| *** AJaeger has joined #openstack-infra | 17:00 | |
| *** epende has joined #openstack-infra | 17:00 | |
| pabelanger | well, if jobs override it, we have no idea, until we run out of inodes | 17:01 |
| pabelanger | that would be my concern | 17:01 |
| dmsimard | clarkb: we do run ara in minimal (exactly because we can't test it anyway) http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base-minimal/post-logs.yaml#n5 | 17:01 |
| *** dhajare has quit IRC | 17:01 | |
| *** jascott1 has quit IRC | 17:01 | |
| dmsimard | pabelanger: let's not exaggerate the impact of ara on inodes | 17:01 |
| clarkb | dmsimard: huh the comment is wrong above base minimal then "doesn't validate the node and does not generate an ARA report" | 17:01 |
| *** jascott1 has joined #openstack-infra | 17:02 | |
| clarkb | dmsimard: we can clean that up in a followup | 17:02 |
| dmsimard | clarkb: oops ? | 17:02 |
| pabelanger | dmsimard: sure, but this is the main reason why we are doing this today for ARA, otherwise, why bother? | 17:02 |
| dmsimard | pabelanger: it would require projects to consciously and explicitely override that variable, I feel if someone does that, they know what they're doing, and that someone could be one of us | 17:03 |
| *** jpena is now known as jpena|off | 17:03 | |
| pabelanger | I think in general, we shouldn't allow jobs to override anything in base | 17:04 |
| *** e0ne has joined #openstack-infra | 17:04 | |
| pabelanger | but will defer to others | 17:04 |
| clarkb | ya I think generally not overriding base is a good idea. Since its the highly opinionated base for everyting | 17:05 |
| dmsimard | Any vars in ansible can be overridden with extra vara | 17:05 |
| *** shardy has quit IRC | 17:05 | |
| dmsimard | If we want people to not override anything in base, we need to think about a lot of different things, like preventing vars from being passed as extra vars for trusted content | 17:06 |
| *** slaweq has quit IRC | 17:06 | |
| dmsimard | Unless that's already the case, but I don't know. | 17:06 |
| *** shardy has joined #openstack-infra | 17:06 | |
| clarkb | extra vars are controlled by zuul directly iirc? | 17:07 |
| clarkb | like its locked up in the zuul config because thos eare passed on ansible invocation | 17:07 |
| *** Goneri has joined #openstack-infra | 17:07 | |
| dmsimard | Hmmm, vars are set in the inventory, not as extra vars, you're right | 17:08 |
| dmsimard | extra-vars are also 'almighty' in the sense that even if you try to override it with a set_fact, it won't take effect | 17:08 |
| pabelanger | only secrets and site vars are extra_vars today | 17:08 |
| *** yamahata has joined #openstack-infra | 17:09 | |
| *** yamamoto has joined #openstack-infra | 17:09 | |
| *** yamamoto has quit IRC | 17:09 | |
| *** jtomasek has joined #openstack-infra | 17:09 | |
| *** kiennt26 has quit IRC | 17:09 | |
| pabelanger | 2017-10-13 17:09:48,426 INFO nodepool.builder.UploadWorker.9: Image build opensuse-423-0000000525 in rax-dfw is ready | 17:10 |
| pabelanger | yay | 17:10 |
| pabelanger | Shrews: mordred ^ upload to rax | 17:10 |
| mordred | pabelanger: woot | 17:10 |
| * dirk wonders why opensuse was exciting | 17:12 | |
| pabelanger | what isn't exciting about opensuse :D | 17:12 |
| clarkb | mordred: did you want to weigh in on the ara on failure where to set the var thing? chnage 511918 | 17:12 |
| pabelanger | dirk: we've been having issues uploading to rax over the last few days, new shade release appears to have fixed it | 17:13 |
| *** vhosakot has quit IRC | 17:13 | |
| *** jascott1 has quit IRC | 17:13 | |
| dirk | pabelanger: ah, thanks for info | 17:14 |
| *** dhajare has joined #openstack-infra | 17:16 | |
| *** sambetts is now known as sambetts|adk | 17:20 | |
| *** sambetts|adk is now known as sambetts|afk | 17:20 | |
| *** ijw has joined #openstack-infra | 17:20 | |
| mordred | clarkb: looking | 17:20 |
| *** andreas_s has joined #openstack-infra | 17:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove api-guide remains https://review.openstack.org/511924 | 17:21 |
| odyssey4me | mordred another one - our pike tests are still frequent, so I ported the patch back: https://review.openstack.org/511923 | 17:21 |
| *** dhinesh has joined #openstack-infra | 17:22 | |
| dmsimard | clarkb, pabelanger: so, circling back to the base jobs and ara report generation. My method allows it to be overridden on a need basis but we can enforce if it we want. Which do we want ? IMO enforcing it just makes it harder to perhaps selectively tests things (such as an error not showing up in a false positive, for example, or whatever it could be) | 17:23 |
| dmsimard | I'll settle with the majority here | 17:23 |
| clarkb | dmsimard: I think I'm fine either way, mordred was looking at it as of a couple minutes ago so curious what he says | 17:23 |
| clarkb | I do think we should get that in and then enable v3 pipelines again though | 17:24 |
| *** dhinesh_ has joined #openstack-infra | 17:25 | |
| *** andreas_s has quit IRC | 17:25 | |
| pabelanger | clarkb: before we bring zuulv3 back online, we should look at zuulv3 mergers, they are currently down (all). I noted on the zuulv3-issues etherpad last night | 17:26 |
| pabelanger | zm05 to zm08 | 17:26 |
| clarkb | pabelanger: says that jeblair is looking at it? | 17:27 |
| mordred | pabelanger, clarkb, dmsimard: I'm fine with it on the job and allowing a job to override it - there might be a valid reasons it needs to be set on a case by case basis | 17:28 |
| pabelanger | clarkb: yah, see that now. Just wanted to make sure we maybed solved it before start up with jobs again | 17:28 |
| clarkb | pabelanger: we should probably make sure jeblair is looking at it (but maybe that is why jeblair has been so quiet, busy digging into that?) | 17:28 |
| *** dhinesh has quit IRC | 17:29 | |
| pabelanger | sure | 17:29 |
| pabelanger | we're also making good process on rackspace uploads now on nb04.o.o | 17:29 |
| pabelanger | mordred: did your shade script on nb04 also finish? | 17:29 |
| mordred | pabelanger: still runnning | 17:31 |
| AJaeger | trivial fix for a typo in a job: https://review.openstack.org/511840 - reviews welcome | 17:32 |
| AJaeger | mordred: I worked some more on zuulv3 manual, what do you think of https://review.openstack.org/511915 ? And let's approve https://review.openstack.org/#/c/511876/ , please | 17:33 |
| AJaeger | thanks, clarkb and pabelanger ! | 17:34 |
| clarkb | pabelanger: you good with mordreds comment? | 17:34 |
| *** camunoz has quit IRC | 17:36 | |
| *** slaweq has joined #openstack-infra | 17:37 | |
| *** camunoz has joined #openstack-infra | 17:37 | |
| *** dhajare has quit IRC | 17:37 | |
| *** e0ne has quit IRC | 17:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo in role https://review.openstack.org/511840 | 17:38 |
| pabelanger | clarkb: yah, just wanted to point it out and make sure all on board. | 17:38 |
| pabelanger | +3 | 17:39 |
| clarkb | pabelanger: thanks | 17:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Improve the comment for the base-minimal job https://review.openstack.org/511932 | 17:41 |
| dmsimard | clarkb: ^ fixed your comment | 17:41 |
| *** tosky has quit IRC | 17:41 | |
| *** trown|lunch is now known as trown | 17:41 | |
| clarkb | dmsimard: thanks | 17:42 |
| *** rbrndt has quit IRC | 17:42 | |
| *** armax has quit IRC | 17:42 | |
| *** nicolasbock has quit IRC | 17:42 | |
| inc0 | good morning, are you planning Friday afternoon deployment? | 17:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Only generate ARA reports on failure for base jobs https://review.openstack.org/511918 | 17:42 |
| dmsimard | inc0: for zuul v3? no. | 17:43 |
| inc0 | just checking, any chances you can enable some nodepool for zuulv3 so we can work on our jobs over weekend? | 17:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: WIP: Fix api-guide/api-ref publishing https://review.openstack.org/511933 | 17:43 |
| *** gfidente has quit IRC | 17:43 | |
| AJaeger | mordred: the publish jobs for api-guide/api-ref are wrong ^ | 17:43 |
| AJaeger | mordred: that change is incomplete - do you have a good idea on how to get the servicename? ^ | 17:44 |
| pabelanger | inc0: were talking about enabling zuulv3 check pipelines again now, but unsure just yet | 17:44 |
| dmsimard | clarkb, mordred: did we settle on sunday ? are we sending an email out ? There was some concerns, here for example: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123584.html | 17:44 |
| pabelanger | but mostly wanting to make sure were stable from recent CI outage | 17:44 |
| *** jtomasek has quit IRC | 17:45 | |
| clarkb | pabelanger: we seemto have stabilized around 62 million free inodes. its been floating up and down around that number today | 17:46 |
| clarkb | dmsimard: I thought so, mordred did email go out? I saw ianw had a followup email on the outages | 17:46 |
| pabelanger | dmsimard: yah, I think I'd rather rollout zuulv3 and help projects get jobs working. | 17:47 |
| dmsimard | pabelanger: rollout as in, enable check and keep v2 gating ? | 17:47 |
| AJaeger | clarkb: haven't seen an annoucement by mordred today | 17:47 |
| *** jascott1 has joined #openstack-infra | 17:47 | |
| AJaeger | dmsimard: switch off v3 | 17:47 |
| pabelanger | dmsimard: no, rollout zuulv3 and stop zuulv2 | 17:47 |
| AJaeger | dmsimard: switch off v2 I mean | 17:47 |
| pabelanger | I don't think logs.o.o will be able to keep up with 2 zuuls | 17:48 |
| inc0 | how about roll out partially, without legacy jobs, and first thing after weekend roll 3? | 17:48 |
| dmsimard | inc0: rolling without legacy jobs would be a bit complicated, I think ? | 17:49 |
| inc0 | that will allow few silent days (although over weekend) for no lifers like me to work on gates | 17:49 |
| *** dosaboy has quit IRC | 17:49 | |
| inc0 | we can just comment *a lot* of lines in one file;) | 17:49 |
| inc0 | https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml | 17:50 |
| clarkb | I've got to pop out for lunch shortly (driving into town) but if mordred (or someone) can send that email saying plan is for sunday that would be great | 17:50 |
| clarkb | inc0: rough plan is turn on pipelines again today since friday and weekend is low volume allowing people to iterate on jobs | 17:50 |
| clarkb | then about 2200UTC Sunday we will flip things back to v3 | 17:51 |
| jlvillal | Not sure if anybody would like to review: https://review.openstack.org/#/c/509670/ Support the log level "CRITICAL" | 17:51 |
| inc0 | cool, that'd be awesome, thanks! | 17:51 |
| *** armax has joined #openstack-infra | 17:52 | |
| *** camunoz has quit IRC | 17:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Improve the comment for the base-minimal job https://review.openstack.org/511932 | 17:53 |
| *** camunoz has joined #openstack-infra | 17:54 | |
| *** baoli__ has quit IRC | 17:54 | |
| fungi | clarkb: i expect we'll start gaining ground on the inode count again once activity trails off over the next few hours | 17:55 |
| *** zaneb has quit IRC | 17:55 | |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: build_job: return queue item identifier https://review.openstack.org/511937 | 17:56 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add get_queue_item() method https://review.openstack.org/511938 | 17:56 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: better get_build_info() example https://review.openstack.org/511939 | 17:56 |
| fungi | but yeah, we've been hovering at 93% used inodes since i woke up | 17:56 |
| clarkb | ok I'v egot to pop out now, I'll be back around probably 1:30 PDT or so. Would be great if someone can send an email with status update | 17:57 |
| *** ralonsoh has quit IRC | 18:01 | |
| *** panda|rover is now known as panda|rover|off | 18:01 | |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: build_job: return queue item identifier https://review.openstack.org/511937 | 18:04 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add get_queue_item() method https://review.openstack.org/511938 | 18:04 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: better get_build_info() example https://review.openstack.org/511939 | 18:04 |
| *** baoli has joined #openstack-infra | 18:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: WIP: Fix api-guide/api-ref publishing https://review.openstack.org/511933 | 18:07 |
| *** yamamoto has joined #openstack-infra | 18:10 | |
| *** pblaho has quit IRC | 18:12 | |
| *** tosky has joined #openstack-infra | 18:13 | |
| *** armaan has quit IRC | 18:13 | |
| *** Guest53680 is now known as mgagne | 18:14 | |
| *** mgagne has quit IRC | 18:14 | |
| *** mgagne has joined #openstack-infra | 18:14 | |
| mgagne | We would like to schedule a maintenance next Tuesday (October 17) on inap-mtl01 to perform an upgrade on Nova at around ~6:00pm EDT. Would that be ok with you? API will be unavailable but existing instances will continue to be reachable. | 18:15 |
| *** smarcet has quit IRC | 18:16 | |
| clarkb | mgagne: should be fine, should we expect to disable nodepool against that cloud beforehand? | 18:18 |
| mgagne | clarkb: what do you recommend? IIRC, last time, we didn't do anything to that regard. | 18:18 |
| clarkb | nodepool should gracefully handle it it will just hit your api (or try to) and fail | 18:19 |
| clarkb | if that is ok on your end we may just leave it to run on its own | 18:19 |
| fungi | it's not strictly necessary, just helps if you want our load to go away before you start working | 18:19 |
| *** yamamoto has quit IRC | 18:19 | |
| mgagne | we are updating Nova, we won't update anything related to hypervisor themselves so instances can stay | 18:20 |
| *** eharney has quit IRC | 18:20 | |
| thingee | am I understanding right that the post jobs for publishing docs is leaning on the instructions with tox? | 18:20 |
| fungi | thingee: yeah, they basically run `tox -e venv python setup.py build_sphnix` | 18:21 |
| fungi | er, sphinx | 18:21 |
| fungi | helps to spell it correctly | 18:22 |
| thingee | AJaeger: sounds perfect to me | 18:22 |
| fungi | i think technically it ends up being `tox -e venv -- python setup.py build_sphinx` but basically the same | 18:22 |
| thingee | AJaeger: whoops sorry wrong channel :) | 18:22 |
| AJaeger | thingee: we can talk here as well ;) | 18:23 |
| fungi | mordred has a proposal out to change the pti to expect projects to just run sphinx-build without needing tox | 18:23 |
| fungi | but the our jobs won't reflect that until the tc has had a chance to debate and approve something along those lines | 18:24 |
| *** rbrndt has joined #openstack-infra | 18:26 | |
| *** r0llerd3rby has joined #openstack-infra | 18:26 | |
| dmsimard | fungi: on my end I just run tox -e docs ? | 18:26 |
| fungi | dmsimard: yeah, a lot of projects include that in their tox.ini today as a convenience | 18:27 |
| fungi | but it's technically not what our pti demands official projects support | 18:27 |
| *** ldnunes has quit IRC | 18:27 | |
| *** ldnunes has joined #openstack-infra | 18:28 | |
| *** mriedem1 has joined #openstack-infra | 18:29 | |
| dmsimard | looking at what python-jobs provides, it circles back to https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/macros.yaml#L314 and then ultimately https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-docs.sh which sounds way more complicated than it should be | 18:29 |
| fungi | infra-root: since 511918 (Only generate ARA reports on failure for base jobs) has merged, are we missing anything else before we reenable check and tripleo-check for zuulv3.o.o? | 18:30 |
| fungi | i can send the announcement about sunday, but it would be nice if the announcement could mention we're running v3 check jobs again | 18:30 |
| AJaeger | fungi, AFAIK we're ready to +2A https://review.openstack.org/511899 ;) | 18:31 |
| fungi | thanks AJaeger! | 18:31 |
| *** mriedem has quit IRC | 18:31 | |
| clarkb | pabelanger had concerns about the zuul merger | 18:31 |
| clarkb | so that might need debugging too | 18:31 |
| *** eharney has joined #openstack-infra | 18:31 | |
| fungi | oh, right, jeblair may or may not be looking into the mergers | 18:31 |
| *** mriedem1 is now known as mriedem | 18:32 | |
| *** eharney has quit IRC | 18:32 | |
| pabelanger | yah, I think we need input from jeblair on zuulv3 mergers, zm05 to zm08 are not running anything ATM | 18:32 |
| *** eharney has joined #openstack-infra | 18:32 | |
| *** electrofelix has quit IRC | 18:32 | |
| pabelanger | nb04.o.o is pretty much recovered, rackspace has been getting new images uploaded since we restarted with lastest shade | 18:33 |
| mordred | pabelanger: awesome! | 18:33 |
| *** dprince has quit IRC | 18:33 | |
| pabelanger | and we're just building a new xenial DIB now with latest AFS mirror content | 18:34 |
| fungi | yeah, looks like jeblair's shell sessions on most of those mergers are only idle for a few hours, so presumably he was doing something on them at least that recently | 18:34 |
| pabelanger | that should go live in the next hour | 18:34 |
| *** zaneb has joined #openstack-infra | 18:34 | |
| dmsimard | clarkb: where is https://review.openstack.org/#/c/501887/ used ? it mentions puppet, afaik we no longer use puppet on nodepool nodes so that would be for infrastructure nodes ? | 18:35 |
| pabelanger | and looking at reprepro on mirror-update.o.o, it appears to be happy again | 18:35 |
| *** claudiub has joined #openstack-infra | 18:35 | |
| fungi | dmsimard: correct | 18:36 |
| fungi | dmsimard: related to the mechanism we use for adding and disabling our individual admin shell accounts on our control-plane and similarly persistent servers | 18:37 |
| *** pcrews has quit IRC | 18:38 | |
| fungi | dmsimard: and he's repeatedly rechecking it in hopes of triggering a nondeterministic failure we get in those validation jobs | 18:38 |
| fungi | so we can hopefully get additional clues as to what's causing it | 18:38 |
| dmsimard | okay | 18:39 |
| *** armax_ has joined #openstack-infra | 18:40 | |
| fungi | dmsimard: when it fails, it looks like this http://logs.openstack.org/16/497616/1/gate/gate-infra-puppet-apply-3-centos-7/154b032/puppetapplytest29.final.out.FAILED | 18:41 |
| dmsimard | fungi: that sounds faulty indeed, the key should lay on a single line and start with ssh-rsa, ssh-dsa etc | 18:42 |
| fungi | yep! | 18:42 |
| fungi | so what we want to know is why it does this, and why it's only occasional (usually the job succeeds) | 18:42 |
| *** armax has quit IRC | 18:42 | |
| *** armax_ is now known as armax | 18:42 | |
| *** lihi has quit IRC | 18:42 | |
| dmsimard | do we want to merge that ? or is it self tested ? | 18:43 |
| *** slaweq has quit IRC | 18:43 | |
| *** kjackal_ has joined #openstack-infra | 18:43 | |
| dmsimard | oh, it's self tested, okay http://logs.openstack.org/87/501887/2/check/gate-infra-puppet-apply-3-centos-7/52c6356/console.html#_2017-10-13_00_00_27_432766 | 18:44 |
| fungi | yeah, it's fine for us to merge it, i think, because then the next time any of our changes hits that nondeterministic error, we can get more detail to help us track it down | 18:44 |
| dmsimard | maybe we should merge it and add an e-r query | 18:44 |
| *** lihi has joined #openstack-infra | 18:45 | |
| *** gouthamr has quit IRC | 18:47 | |
| fungi | i don't think we index those logs | 18:47 |
| fungi | into logstash/elasticsearch | 18:47 |
| fungi | but i could be mistaken | 18:47 |
| *** slaweq has joined #openstack-infra | 18:48 | |
| *** dosaboy has joined #openstack-infra | 18:48 | |
| *** felipemonteiro has joined #openstack-infra | 18:48 | |
| *** dosaboy has quit IRC | 18:48 | |
| *** rhallisey has quit IRC | 18:49 | |
| *** dosaboy has joined #openstack-infra | 18:50 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:50 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding check/publish jobs for contributor guide https://review.openstack.org/511946 | 18:53 |
| *** felipemonteiro has quit IRC | 18:53 | |
| thingee | AJaeger: I think I did that right? ^ | 18:54 |
| thingee | I didn't update zuul/layout.yml ... it looks like that would involve creating a legacy job. | 18:54 |
| thingee | but I'm also very confused. :) | 18:55 |
| fungi | pabelanger: without running zuul-merger processes, how is zuulv3 running a job for 511946,1 now? | 18:57 |
| pabelanger | fungi: each executor also has a merger process, so we do some merging | 18:58 |
| fungi | ahh | 18:58 |
| pabelanger | but will be a scaling issue without main | 18:58 |
| fungi | got it | 18:58 |
| AJaeger | thingee: just use the template and you#re good to go. | 18:59 |
| *** jcoufal has quit IRC | 18:59 | |
| AJaeger | This is v3 setup- which will be fully used by Monday, no need to hazzle with v2 until then | 18:59 |
| jeblair | fungi, pabelanger: i put my name next to the merger issue on the etherpad. :) | 19:00 |
| jeblair | fungi: i understand the issue, it's not a blocker. | 19:01 |
| mordred | pabelanger, Shrews: fwiw, I'm still poking at the image cleanup thing | 19:05 |
| *** ijw has quit IRC | 19:05 | |
| *** baoli has quit IRC | 19:05 | |
| *** baoli has joined #openstack-infra | 19:06 | |
| * AJaeger signs off and wishes everybody a nice evening, weekend etc | 19:10 | |
| *** AJaeger has quit IRC | 19:10 | |
| *** baoli has quit IRC | 19:10 | |
| *** florianf has quit IRC | 19:10 | |
| *** florianf has joined #openstack-infra | 19:11 | |
| *** baoli has joined #openstack-infra | 19:12 | |
| *** friedrice_injera is now known as fried_rice | 19:12 | |
| *** baoli has quit IRC | 19:13 | |
| fungi | jeblair: so we can approve 511899 safely without waiting to have merger daemons running on zm05-08 again? | 19:14 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 19:14 |
| *** baoli has joined #openstack-infra | 19:15 | |
| jeblair | fungi: yep. i just did so. | 19:15 |
| fungi | thanks! | 19:15 |
| *** florianf has quit IRC | 19:15 | |
| dmsimard | mordred: see this patch ^ otherwise https://review.openstack.org/#/c/511821/ could have landed and broken every gate | 19:15 |
| fungi | i'll get the announcement drafted up in an etherpad and link it in here in a moment | 19:15 |
| *** florianf has joined #openstack-infra | 19:15 | |
| mordred | dmsimard: yay! | 19:15 |
| fungi | unless mordred already had one in progress? | 19:16 |
| * dmsimard proposes we volounteer mordred announcer in chief | 19:16 | |
| *** armaan has joined #openstack-infra | 19:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Re-enable v3 pipelines https://review.openstack.org/511899 | 19:17 |
| fungi | we need infra-branded bullhorns/megaphones | 19:17 |
| * fungi watches zuulv3 for signs of life in check pipelines | 19:18 | |
| *** baoli has quit IRC | 19:19 | |
| *** r0llerd3rby has quit IRC | 19:20 | |
| *** slaweq has quit IRC | 19:21 | |
| *** slaweq has joined #openstack-infra | 19:21 | |
| *** andreas_s has joined #openstack-infra | 19:21 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for emit-job-header and set-zuul-log-path-fact https://review.openstack.org/511948 | 19:21 |
| fungi | zuulv3 check lives again | 19:22 |
| pabelanger | fungi: https://review.openstack.org/511656/ would be a nice one to land also. caps launch-timeout on citycloud to 10mins | 19:22 |
| *** eggshell has joined #openstack-infra | 19:24 | |
| *** gouthamr_ has joined #openstack-infra | 19:24 | |
| dmsimard | pabelanger: 10 minutes ? wow | 19:29 |
| *** florianf has quit IRC | 19:30 | |
| *** florianf has joined #openstack-infra | 19:30 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding check/publish jobs for contributor guide https://review.openstack.org/511946 | 19:31 |
| fungi | infra-root: proposed announcement is https://etherpad.openstack.org/p/X6d2vXrmDm | 19:32 |
| *** ijw has joined #openstack-infra | 19:33 | |
| *** andreas_s has quit IRC | 19:34 | |
| mordred | pabelanger, Shrews: ok. dupicate images should be all taken care of | 19:36 |
| *** florianf has quit IRC | 19:37 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for fetch-zuul-cloner https://review.openstack.org/511953 | 19:37 |
| *** florianf has joined #openstack-infra | 19:37 | |
| dmsimard | fungi: so this means that sunday v2 is no longer gating, correct ? I feels it needs an explicit mention | 19:40 |
| dmsimard | brb | 19:40 |
| fungi | dmsimard: starting at 22:00 sunday, yes | 19:41 |
| SamYaple | two days until i can publish to dockerhub | 19:41 |
| fungi | i'll clarify, thanks | 19:41 |
| *** Swami has joined #openstack-infra | 19:41 | |
| fungi | SamYaple: and 2.5 days until dockerhub humbly requests that you cease and desist? ;) | 19:41 |
| SamYaple | fungi: haha no. my images are like 40mb, ill be the least of thier worries | 19:42 |
| SamYaple | but if they do, quay.io is totally cool wit hthis | 19:42 |
| jeblair | fungi: ++ | 19:43 |
| fungi | dmsimard: does my extra sentence in the tl;dr provide the necessary illumination? | 19:43 |
| dmsimard | fungi: yes | 19:44 |
| *** ccamacho has quit IRC | 19:46 | |
| fungi | awesome | 19:47 |
| boden | hi, where can I find zuul v3 logs if I have a -1V from Zuul, but no jobs (logs) listed on the patch? https://review.openstack.org/#/c/511542/ | 19:49 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: build_job: return queue item identifier https://review.openstack.org/511937 | 19:49 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add get_queue_item() method https://review.openstack.org/511938 | 19:49 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: better get_build_info() example https://review.openstack.org/511939 | 19:49 |
| fungi | boden: hit the "toggle ci" button at the bottom and expand the commend from the "Zuul" user | 19:49 |
| pabelanger | mordred: yah, nodepool looks happy again | 19:50 |
| boden | fungi thank you | 19:50 |
| mtreinish | fungi: is there any plans to fix that somehow. (probably add on to the js for the tbale) I've see a bunch of people confused by that | 19:51 |
| fungi | boden: looks like it should be a - instead of a _ | 19:51 |
| boden | fungi: ya, my bad :( | 19:51 |
| pabelanger | fungi: ++ | 19:52 |
| fungi | mtreinish: depends on what you mean by "plans" but i think most of us assume we'll amend the ci results table to show a row for "syntax error" or something as well as teach it about some of the new job failure types | 19:52 |
| *** Swami has quit IRC | 19:53 | |
| mtreinish | fungi: I guess that assumption was it. I just wanted to know if it's on the radar for a future work item (I'm assuming there is a long to do list for all the things that need to be updated post migration) | 19:54 |
| fungi | mtreinish: yep, i expect if anyone who's not totally turned off by hacking on javascript wanted to take that up, none of us would object | 19:54 |
| fungi | otherwise, we'll likely get around to it soon-ish | 19:55 |
| fungi | okay, licking the stamp for the announcement now | 19:55 |
| *** smarcet has joined #openstack-infra | 19:56 | |
| *** andreas_s has joined #openstack-infra | 19:57 | |
| *** Swami has joined #openstack-infra | 19:58 | |
| mtreinish | fungi: heh, well my js is terrible. So I'm probably not volunteering for that :) | 19:58 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: get_build_console_output: fix docs for "number" parameter https://review.openstack.org/511954 | 19:59 |
| *** smarcet has quit IRC | 20:00 | |
| fungi | mtreinish: you could add it to the zuul section of the zuulv3-issues etherpad if you want, so we don't forget it needs doing at least | 20:00 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for fetch-zuul-cloner https://review.openstack.org/511953 | 20:01 |
| mtreinish | fungi: done | 20:01 |
| dmsimard | pabelanger: btw I had a comment on https://review.openstack.org/#/c/510155/ | 20:02 |
| *** andreas_s has quit IRC | 20:02 | |
| pabelanger | dmsimard: possible, I haven't had a change to test that playbook yet. will update shortly | 20:03 |
| fungi | thanks mtreinish | 20:05 |
| jeblair | that's not really an issue with zuul | 20:06 |
| fungi | good point, we could create a gerrit section | 20:06 |
| *** felipemonteiro__ has joined #openstack-infra | 20:06 | |
| fungi | i'll do that now | 20:06 |
| *** smatzek has quit IRC | 20:09 | |
| *** felipemonteiro_ has quit IRC | 20:09 | |
| *** iyamahat has joined #openstack-infra | 20:10 | |
| fungi | looking through the issues pad, "Devstack-gate jobs do not find cached files in ~zuul when running under the zuul user in v3" is solved now, yes? | 20:11 |
| fungi | i'm double-checking a v3 job now to confirm | 20:12 |
| fungi | er, build i mean | 20:12 |
| clarkb | ya 511260 fixed that | 20:12 |
| *** sdague has quit IRC | 20:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove -e from requirements https://review.openstack.org/511957 | 20:15 |
| jeblair | pabelanger, mordred, fungi: ^ that should fix the mergers | 20:15 |
| jeblair | (they were never started with the new git timeout code because of that error) | 20:15 |
| *** andreas_s has joined #openstack-infra | 20:16 | |
| pabelanger | ah.. +2 | 20:16 |
| fungi | completely agree with that fix | 20:18 |
| *** AJaeger has joined #openstack-infra | 20:19 | |
| AJaeger | team, please check the zuul v3 failures in https://review.openstack.org/#/c/511501/ - this looks really broken, we should not get RETRY_LIMIT on a simple pep8 anymore.... | 20:19 |
| AJaeger | Sorry, no time to investigate - I'm gone, just saw this and wanted to pass on... | 20:20 |
| * AJaeger waves good night | 20:20 | |
| *** AJaeger has quit IRC | 20:20 | |
| *** kgiusti has left #openstack-infra | 20:20 | |
| jeblair | and really gone! | 20:20 |
| fungi | clarkb: yeah, confirmed we have a v3 devstack run here which didn't need to download get-pip.py, so i'll move that issue to the archive pad: http://logs.openstack.org/97/509697/4/check/congress-devstack-py35-api-mysql/dce7f2d/logs/devstack-early.txt.gz#_2017-10-13_19_41_45_676 | 20:20 |
| jeblair | the thing aj pointed out is it apparently needs horizon to run its pep8 jobs | 20:22 |
| pabelanger | looking | 20:22 |
| jeblair | http://logs.openstack.org/01/511501/2/check/openstack-tox-pep8/8d9a420/job-output.txt.gz#_2017-10-13_19_38_16_444933 | 20:22 |
| *** thorst has quit IRC | 20:23 | |
| pabelanger | odd | 20:23 |
| *** thorst has joined #openstack-infra | 20:23 | |
| pabelanger | ah, sahara-dashboard | 20:25 |
| pabelanger | maybe it is using horizon apis | 20:25 |
| fungi | curious why that retried instead of failing | 20:25 |
| pabelanger | http://git.openstack.org/cgit/openstack/sahara-dashboard/tree/tools/tox_install.sh | 20:26 |
| pabelanger | seems to have its own tox installer | 20:26 |
| pabelanger | unsure why that would be | 20:27 |
| jeblair | fungi: in order to support the tox-siblings functionality, we do a tox env creation with no commands first, in the pre-playbook. that's what failed. | 20:27 |
| tosky | uh | 20:27 |
| fungi | pabelanger: most projects have a pip wrapper like that one, called from tox, to shoehorn in constraints files so that local devs can optionally use them | 20:27 |
| *** thorst has quit IRC | 20:28 | |
| pabelanger | fungi: interesting, didn't know that | 20:28 |
| fungi | pabelanger: most of the constraints-using projects anyway | 20:28 |
| tosky | uhm, maybe we can kill it | 20:28 |
| tosky | or maybe not | 20:29 |
| tosky | it was added recently, and from outside the sahara team | 20:29 |
| *** andreas_s has quit IRC | 20:29 | |
| tosky | added by tonyb | 20:29 |
| *** trown is now known as trown|outtypewww | 20:30 | |
| jeblair | remote: https://review.openstack.org/511501 WIP Initial Zuul v3 jobs | 20:30 |
| jeblair | i pushed a revision | 20:30 |
| *** Apoorva_ has joined #openstack-infra | 20:30 | |
| tosky | jeblair: oh, interesting | 20:31 |
| tosky | I couldn't proceed with the experiments because zuul v3 stopped voting | 20:31 |
| jeblair | oh, i didn't add pep8 there, i'll push another | 20:31 |
| jeblair | i'm pretty sure this is going to prompt an interesting conversation with ajaeger and mordred next week | 20:32 |
| fungi | pabelanger: oh, i should clarify, specifically most constraints-using libraries which are themselves represented in the constraints list need a wrapper like that around pip to first edit themselves out of the constraints list before installing. see commit message on https://review.openstack.org/413354 for details | 20:32 |
| tosky | jeblair: one question: if the job is defined locally, is its definition merged with the definition from project-config, or does it replace that definition completely? | 20:32 |
| jeblair | tosky: but i think it's useful to at least push this forward till it's working so we have something more definitive to talk about :) | 20:32 |
| jeblair | tosky: merged | 20:32 |
| jeblair | tosky: so i'm about to add pep8 to that change, which will basically alter the pep8 definition that comes from project-config to add the required project | 20:33 |
| jeblair | tosky: pushed ps3 -- hope you don't mind | 20:34 |
| jeblair | er, ps4 rather | 20:34 |
| *** Apoorva has quit IRC | 20:34 | |
| tosky | I will complete it with the dashboard job | 20:34 |
| tosky | but probably on Monday | 20:34 |
| *** andreas_s has joined #openstack-infra | 20:34 | |
| tosky | so does it mean that zuul v3 is voting again on all projects? | 20:35 |
| tosky | I have another recheck to launch then | 20:35 |
| jeblair | tosky: voting in check, yes | 20:35 |
| pabelanger | fungi: ya, I wonder if tox-siblings might help with that now | 20:35 |
| fungi | tosky: yes, and come monday it'll be gating on them | 20:35 |
| pabelanger | I haven't look much at tox-siblings myself just yet | 20:35 |
| fungi | pabelanger: maybe within the ci, but not for the local developer use case | 20:35 |
| pabelanger | fungi: true | 20:35 |
| fungi | pabelanger: oh, except maybe it does once we have a good story around local devs firing their tests with ansible? | 20:36 |
| fungi | so i think the answer to that is, not so much just yet, but perhaps soon | 20:36 |
| pabelanger | yah, will be interesting to see if more devs embrace ansible locally, or continue along with bash scripts | 20:38 |
| pabelanger | wait and see approach | 20:38 |
| *** pcrews has joined #openstack-infra | 20:38 | |
| tosky | jeblair: would it make sense to define a set of openstack-tox-{pep8,py27,py35}-dashboards jobs with the added dependency on horizon that can be used by all horizon dashboards? | 20:39 |
| mordred | fungi, pabelanger: yes - tox-siblings should help with the tox/pip/constraints thing | 20:39 |
| *** andreas_s has quit IRC | 20:39 | |
| fungi | jeblair: thanks for the clarification on the RETRY_LIMIT for that one... that means we should occasionally expect RETRY_LIMIT failures on changes adding/modifying job configuration when it runs afoul of the tox-siblings pre task, but not for changes which aren't touching job configuration? | 20:39 |
| mordred | also, tonyb and I looked a little bit in to potentially using -c$UPPER_CONSTRAINTS_FILE in the tox.ini in the requirements line - which should make the need for the wrapper script go away for hte libraries that are themselves in constraints | 20:39 |
| mordred | so there are 2 different things that are different in the new world order that will help us simplify how this works today | 20:40 |
| mordred | I think we were waiting to push on the 'use -c in tox.ini' approach until v3 settled down to reduce confusion | 20:40 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add statsd to zuul.conf https://review.openstack.org/511018 | 20:40 |
| fungi | mordred: how does that solution you discussed with tonyb work around the need to filter the project itself out of teh constraints list? | 20:41 |
| mordred | yes | 20:41 |
| mordred | becaues it's not putting it into the install command entry liek we do today | 20:41 |
| mordred | but instead only in the requirements entry | 20:41 |
| jeblair | tosky: yeah, that would work. let's run it by andreas monday | 20:41 |
| fungi | ohhh | 20:41 |
| mordred | so that it gets applied when tox installs the requirements | 20:41 |
| mordred | but not when it installs the repo itself | 20:41 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config master: Revert "neutron: Make grenade-neutron-dvr-multinode job non-voting" https://review.openstack.org/505318 | 20:42 |
| fungi | mordred: got it, but not when tox installs other things (unless also explicitly copied to commands in the other tox envs which don't use the normal requirements? | 20:42 |
| *** ihrachys has joined #openstack-infra | 20:42 | |
| jeblair | wow. git just *completely* bombed on a merge conflict resolution. like, i think i'm going to need to rewrite the patch. | 20:43 |
| fungi | wha? that's unfortunate | 20:43 |
| jeblair | i guess maybe i could try redoing it with another merge resolve strategy | 20:44 |
| mordred | tosky, jeblair: might also want to consider an openstack-tox-dashboards project-template that adds requirements into the requirements in the pipeline definitions | 20:44 |
| jeblair | i've never done that before | 20:44 |
| *** thorst has joined #openstack-infra | 20:44 | |
| *** wolverineav has quit IRC | 20:44 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove -e from requirements https://review.openstack.org/511957 | 20:44 |
| jeblair | mordred, tosky: ^ yeah that's better i think | 20:44 |
| fungi | jeblair: i've done that very occasionally when i want to emulate the behavior of gerrit/zuul's git merge conflict resolution choices | 20:45 |
| tosky | mordred, jeblair: template which uses the normal jobs instead of single extended jobs? | 20:45 |
| tosky | why not | 20:45 |
| jeblair | tosky: yeah, it'd be the project-template version of what i did in that patch | 20:45 |
| *** edmondsw has quit IRC | 20:45 | |
| *** Apoorva_ has quit IRC | 20:46 | |
| mordred | tosky: yah - that way it's still the 'openstack-tox-pep8' job that's getting run, just with specific additions (mostly I don't think the difference matters, but in this particular case (the core tox jobs) it might be nicer for the jobs themselves to be 'the same') | 20:46 |
| mordred | physically each option will do th eright thing - this is more about managing how we think about things | 20:46 |
| *** thorst has quit IRC | 20:48 | |
| pabelanger | +1 openstack-tox-dashboards | 20:49 |
| *** andreas_s has joined #openstack-infra | 20:52 | |
| *** smatzek has joined #openstack-infra | 20:53 | |
| *** wolverineav has joined #openstack-infra | 20:59 | |
| clarkb | looks like v3 is alive again! | 21:00 |
| pabelanger | okay, nb04 is done with images for today! Which means we ubuntu DIBs now live, for a few hours, using latest reprepro ubuntu.mirrors | 21:00 |
| pabelanger | and I haven't seen any issues | 21:01 |
| pabelanger | \o/ | 21:01 |
| *** iyamahat has quit IRC | 21:02 | |
| *** iyamahat_ has joined #openstack-infra | 21:02 | |
| pabelanger | clarkb: yah, just seen https://review.openstack.org/510991/ pass all zuulv3 jobs (requirements) | 21:02 |
| clarkb | looks like we are still at 62 million ish inodes free | 21:02 |
| pabelanger | that is great | 21:02 |
| clarkb | wow | 21:02 |
| *** slaweq has quit IRC | 21:04 | |
| *** andreas_s has quit IRC | 21:05 | |
| jeblair | fungi: strategy=resolve strategy-options=patience :) | 21:08 |
| mordred | \o/ | 21:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 21:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 21:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 21:09 |
| jeblair | hopefully that did the right thing. i think tests will tell us if it chewed it up undetectably | 21:09 |
| dmsimard | jeblair: can we start rewriting zuul v3 in puppet now ? https://twitter.com/laserllama/status/918884663363620865 | 21:09 |
| *** andreas_s has joined #openstack-infra | 21:10 | |
| jeblair | i don't even know why i follow twitter links :) | 21:11 |
| *** huanxie has joined #openstack-infra | 21:12 | |
| *** ihrachys has quit IRC | 21:12 | |
| dmsimard | ok, here's the direct one: https://puppet.com/products/puppet-bolt | 21:12 |
| dmsimard | also, bit of friday trolling never hurt anyone :D | 21:12 |
| dmsimard | clarkb: do we have/should we have a cacti graph for inodes ? | 21:13 |
| jeblair | dmsimard: no/yes | 21:13 |
| *** ldnunes has quit IRC | 21:13 | |
| clarkb | ++ | 21:13 |
| tonyb | fungi: Correct it does result in a small ammount of duplication but if gets rid of 1 reason to use tox_install.sh | 21:13 |
| tonyb | mordred: Yeah while it isn't directly related to the v3 work I am waiting for the switch befoer I more forward. | 21:14 |
| dmsimard | ok let's see if I can figure it out | 21:14 |
| fungi | tonyb: i'm 200% in favor of any option which gets rid of even some of the random shell scripts people are cargo culting from repo to repo | 21:14 |
| *** smatzek has quit IRC | 21:14 | |
| * tonyb should send an email saying 'we're chaning the way we use constratints and would like testers' | 21:15 | |
| jeblair | dmsimard: it' probably requires admin privs to add; unfortunately our cacti system largely pedates our automation, so about the only thing that's been automated is adding new hosts | 21:15 |
| * tonyb kinda started with shade ;P | 21:15 | |
| dmsimard | jeblair: ah, I was trying to see if the configuration was stored in system-config cacti things | 21:15 |
| fungi | dmsimard: it _should_ be but likely isn't. if you want to hack on some sort of templating we're missing there, i'm happy to get you copies of configuration | 21:16 |
| tonyb | pabelanger, tosky there's a review for dragonflow? that make tox_install work with both v2 and v3 | 21:17 |
| dmsimard | fungi: last time I've done any meaningful amount of cacti was years ago, so something to pattern off of would be great, yes :) | 21:17 |
| tonyb | Havign said that maybe it's worth doign the constratints before v3? | 21:17 |
| fungi | dmsimard: lmk when you have bandwidth to work on it and i'll dig up the configs on the server | 21:17 |
| dmsimard | fungi: it was either that or trying to figure something out for our ara situation mid term | 21:18 |
| fungi | tonyb: "before v3" is... well, the next 48 hours and change | 21:18 |
| fungi | (if all goes well) | 21:18 |
| jeblair | 48 hours and 42 minutes | 21:18 |
| jeblair | not that anyone's counting | 21:18 |
| fungi | ;) | 21:19 |
| clarkb | "the final countdown!" | 21:19 |
| jeblair | big red clock | 21:19 |
| fungi | na-na-na-naaaa | 21:19 |
| dmsimard | fungi: looks like it's readily available over snmp so it shouldn't be very hard to add http://www.net-snmp.org/docs/mibs/UCD-SNMP-MIB.txt | 21:19 |
| *** andreas_s has quit IRC | 21:19 | |
| jeblair | yeah, i think the way to do it would be to add it to the "data query" that's used for partitions | 21:19 |
| fungi | dmsimard: yes, i'm painfully aware, we simply hadn't been querying those oids previously because we didn't have frequent issues with inode capacity anywhere | 21:19 |
| dmsimard | fungi: oh, sure, just saying I would add it if I could :D | 21:20 |
| jeblair | that way it pulls in a data template for each partition and you get set up with a data source automatically | 21:20 |
| tonyb | fungi: Yeah the 'plan' was to use proposal bit to keep the cargo culted scripts in sync but that seemed unpopular so I never implemented it | 21:21 |
| tonyb | fungi: Well it wont happen in 48 hours ;P | 21:21 |
| jeblair | it looks like the '-e' fix has gone in and things have been reinstalled correctly on the mergers, so i will start them now | 21:22 |
| *** wolverineav has quit IRC | 21:22 | |
| jeblair | done | 21:22 |
| tonyb | fungi: to be clear the proposal mordred and I are proposing will only remove the ommon case the sibling projects thing is still up in the air | 21:23 |
| tonyb | fungi: and I don't have a good solution that works with depends-on and local development | 21:24 |
| fungi | tonyb: yep, i think the compelling story there is to make it easy for people to invoke job playbooks locally | 21:24 |
| fungi | pulling in the same tox-sibling playbook we use in ci | 21:24 |
| tonyb | fungi: Interesting, that kinda moves away from the make OpenStack more std. but seems like a move in the right direction | 21:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Handle glance image pagination links better https://review.openstack.org/511963 | 21:25 |
| *** dhinesh_ has quit IRC | 21:25 | |
| tonyb | fungi: so WRT gitweb should I just switch owners.py to using cgit? | 21:25 |
| *** armax has quit IRC | 21:25 | |
| clarkb | tonyb: yes I think so, gitweb was unfunctioning upon upgrade and review-dev had been running cgit (it was something we had planned on doing a long time ago just didn't show up until we were more forced into it) | 21:26 |
| fungi | tonyb: yes | 21:26 |
| fungi | sorry for not realizing that before today | 21:26 |
| tonyb | okay patch incomming ;P | 21:26 |
| tonyb | fungi: np there's a lot going on ;P | 21:26 |
| fungi | we didn't intentionally take down gitweb, it was a casualty in the war on gerrit | 21:27 |
| tonyb | :) | 21:27 |
| clarkb | I feel like gerrit is winning | 21:27 |
| jeblair | tonyb: while we have a moment -- is the elections site okay? do we need to do any forced publication or anything like that? just wanted to check in on it specifically because things have been hectic recently. | 21:27 |
| mordred | tonyb: which thing becomes harder from a 'run locally' perspective? the 'run with the tip of a different project' part? | 21:27 |
| tonyb | that may be the least positive things I've heard clarkb say ;P | 21:28 |
| fungi | dmsimard: so poking around a bit, i think what you'll want is a copy of our /var/lib/cacti/linux_host.xml file | 21:28 |
| clarkb | tonyb: ha | 21:28 |
| tonyb | jeblair: election is okay for right now but in ~week we need the publish jobs working | 21:28 |
| dmsimard | fungi: there's one already in system-config but it dates back to 5 years ago, is that the same one ? | 21:28 |
| fungi | dmsimard: oh! yes, probably | 21:28 |
| fungi | it's 6867 lines long | 21:28 |
| tonyb | jeblair: I can push some trivial chnages next week to test that and thenreact accordingly | 21:28 |
| dmsimard | fungi: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/cacti/linux_host.xml | 21:29 |
| tonyb | jeblair: but mordred had patches that looked to fix it? | 21:29 |
| mordred | I did? | 21:29 |
| tonyb | mordred: Yup install this gitrepo at this SHA | 21:29 |
| jeblair | dmsimard: i at least tried not to have it all be server side magic :) | 21:30 |
| tonyb | mordred: the static-site publishing things you did ~2 weeks ago | 21:30 |
| dmsimard | jeblair: you write all that XML by hand? :p | 21:30 |
| mordred | tonyb: ah nod | 21:30 |
| dmsimard | fungi: I've already entered an ara rabbit hole worth exploring so I'll circle back to inodes in cacti unless someone beats me to it | 21:30 |
| jeblair | dmsimard, fungi: that's an export of all the custom goodness that we do to actually get correct data. it will at the least, let us recreate all the templates if needed, as well as replicate it on a local machine. | 21:30 |
| jeblair | (you import that template into cacti, and it creates a bunch of database records for those things) | 21:32 |
| *** mat128 has quit IRC | 21:32 | |
| tonyb | mordred: this is the hack we're currently using: https://git.openstack.org/cgit/openstack/castellan-ui/tree/tools/tox_install.sh?id=198f56de0dd67d1e404beebd971ec2d9eb291696#n72 | 21:33 |
| *** andreas_s has joined #openstack-infra | 21:33 | |
| mordred | tonyb: ah - wild :) | 21:33 |
| mordred | tonyb: so - fwiw, when I want to run tox tests with a git state of a different project ... | 21:34 |
| mordred | tonyb: I tend to do: .tox/py35/bin/pip install -e ../other-project ... and then just run tox -epy35 again as normal | 21:34 |
| mordred | tonyb: but I also don't use git branch and instead use git reset - so I'm a little weird | 21:34 |
| *** dave-mccowan has quit IRC | 21:34 | |
| tonyb | mordred: Hmm okay that's cute. I wonder how the devs in affected projects will feel about it? | 21:35 |
| *** felipemonteiro__ has quit IRC | 21:36 | |
| mordred | tonyb: well - it shouldn't matter too much - that script won't break tox-siblings - it just won't use the exact same mechanism to get the other repo | 21:36 |
| *** armax has joined #openstack-infra | 21:36 | |
| mordred | tonyb: so if we want to keep that tox_install script to enable that usage - I don't think anything will be amiss | 21:37 |
| *** slaweq has joined #openstack-infra | 21:37 | |
| mordred | Shrews, clarkb, SamYaple, pabelanger: https://review.openstack.org/511963 'fixes' the issue seen with extra /v2 url segments | 21:38 |
| tonyb | mordred: okay. the zuul-cloner shim will work once the require-projects is setup correctly, but fail before that correct? | 21:38 |
| *** thorst has joined #openstack-infra | 21:38 | |
| mordred | tonyb: that's right | 21:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 21:39 |
| mordred | tonyb: but also the zuul-cloner shim is not needed in the gate once required-projects are set up | 21:39 |
| mordred | in v3 | 21:39 |
| *** anupn has joined #openstack-infra | 21:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Specify duplicate project defn behavior https://review.openstack.org/511025 | 21:39 |
| tonyb | mordred: okay that's what I thought. | 21:40 |
| *** esberglu has quit IRC | 21:40 | |
| anupn | Hi all, is it possible to run pxe_ipmitool_tinyipa job without swift service? | 21:41 |
| * tonyb will try to clean all that up next week'ish | 21:41 | |
| mordred | OH | 21:41 |
| mordred | https://git.openstack.org/cgit/openstack/castellan-ui/tree/tox.ini#n20 | 21:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 21:41 |
| mordred | I fully see what's going on there | 21:41 |
| mordred | now | 21:41 |
| anupn | I understand for agent_* drivers in Ironic, i have to use swift service | 21:41 |
| *** Apoorva has joined #openstack-infra | 21:41 | |
| mordred | wow ... SO ... | 21:41 |
| tonyb | mordred: sorry yeah I shoudl have included that url too | 21:42 |
| fungi | tonyb: lmk if you need help with the gitweb->cgit patch for owners.py, though if memory serves i corralled all the gitweb interaction into one function for ease of updating | 21:42 |
| *** thorst has quit IRC | 21:42 | |
| *** huanxie has quit IRC | 21:42 | |
| *** camunoz has quit IRC | 21:42 | |
| fungi | tonyb: and happy to review any patch for that, just give me a heads up in case i miss it | 21:43 |
| tonyb | fungi: Yeah, I was spliting it out again but I'll see what that looks like. | 21:43 |
| mordred | tonyb: if I can make a recommendation - we should have that os: logic (or maybe the install_from_git logic) look to see if ~/src/git.openstack.org/$PROJ exists, and if so use it ... or, hrm. lemme think about this a little more | 21:43 |
| tonyb | mordred: Yup exactly, that's what drgaonflow does | 21:44 |
| mordred | tonyb: (I obviously totally see the issue this is solving for - mostly trying to help think about it in ways that don't continue to use zuul-cloner) | 21:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use weakref for change cache https://review.openstack.org/511355 | 21:44 |
| tonyb | mordred: I think the zuul-cloner stuff can go once we pass the point of never going back to v2 | 21:44 |
| mordred | tonyb: cool. I'd also love to, if we can, avoid double-installing required-projects | 21:45 |
| tonyb | mordred: +1 | 21:45 |
| mordred | tonyb: I think we may be back to "it's really unfortunate that we don't publish server projects to pypi" | 21:45 |
| tonyb | mordred: in *theory* we could pull them from the .zuul.yaml | 21:45 |
| mordred | tonyb: can't count on that - it might also be set in a central repo | 21:46 |
| tonyb | mordred: Yeah I'm looking at doign that and had good conversations at the PTG along those lines where I was given a heap of probelms to consider | 21:46 |
| mordred | tonyb: yah - I think I gave you some of those :) | 21:46 |
| *** andreas_s has quit IRC | 21:46 | |
| tonyb | mordred: Yeah (to problems) and Yeah to things bing in the central repos | 21:47 |
| mordred | tonyb: neutron and horizon at least both have pypi records | 21:47 |
| mordred | (that we own) | 21:47 |
| tonyb | mordred: True that may be enough for the near term. | 21:47 |
| mordred | keystone is the one that has someone else on it | 21:47 |
| tonyb | mordred: Yeah but it hasn't been used since ~2012 | 21:48 |
| mordred | yah - maybe we could convince Dan Crosta to release it to us | 21:48 |
| tonyb | mordred: possibly, or ask the cheeseshop to do it. I want to generate the list of all conflicts before we look to hard at that | 21:49 |
| mordred | ++ | 21:49 |
| jeblair | mordred: tox-siblings does normal tox installation, then over-installs siblings, right? | 21:49 |
| mordred | yes | 21:49 |
| mordred | tonyb: I'd like to see if we can just start doing that - because the MAIN issue with these is that there is no other way for tox to install horizon into castellan-ui currently other than via this kind of thing - so normal dev work that isn't trying to simular depends-on locally is still borked | 21:50 |
| jeblair | mordred: and when you say you want to avoid double installing, you mean you want to avoid the step where we install from git.openstack.org, and then install from ~zuul | 21:51 |
| *** boden has quit IRC | 21:51 | |
| mordred | jeblair: yah | 21:51 |
| jeblair | mordred: publication to pypi means we'd still be on a double-install path, but we're installing from mirorrs, so the ci cost is small. | 21:51 |
| tonyb | mordred: Yup. it *is* a requirement that we can't express :( so we need to look at fixing that anyway | 21:51 |
| mordred | tonyb: ++ | 21:51 |
| mordred | jeblair: exactly | 21:51 |
| mordred | jeblair: the normal case is 'install from pypi' which is how we install things in tox | 21:51 |
| *** andreas_s has joined #openstack-infra | 21:51 | |
| mordred | the exceptional case that's likely mostly used in the gate and not locally is "please use this newer version" | 21:52 |
| mordred | so if we published even just neutron and horizon we could avoid openstack-tox-dashboard-jobs and openstack-tox-jobs-networking since those projects could all actually express their actual requirements | 21:52 |
| jeblair | i suspect the 'newer version' thing may be much more used for projects like this than other types, but still yes. | 21:53 |
| mordred | jeblair: ++ | 21:53 |
| jeblair | mordred: we *could* avoid those templates, but i'm not certain it's desirable -- if these projects *are* depending on in-development software, then they'd want to keep the required-projects entries and make use of tox-siblings. | 21:54 |
| jeblair | it is, however, a *choice* at that point which is desirable in all cases :) | 21:54 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Better exception handling during autohold https://review.openstack.org/511902 | 21:54 |
| mordred | jeblair: yes - I'd like for them to be able to test both things if they want | 21:55 |
| *** Nil_ has quit IRC | 21:55 | |
| mordred | "I want to test against the most recent release" and "I want to test against master" | 21:55 |
| jeblair | ya | 21:55 |
| *** andreas_s has quit IRC | 21:56 | |
| mordred | ZOMG I ACTUALLY LOGGED IN TO STORYBOARD!!!! | 21:56 |
| *** iyamahat_ has quit IRC | 21:57 | |
| *** iyamahat has joined #openstack-infra | 21:57 | |
| jeblair | mordred: celebrate with a trivial +3? :) https://review.openstack.org/511876 | 21:58 |
| SamYaple | mordred: celebrate with a non-trival +3? :) https://review.openstack.org/#/c/506502/ | 22:01 |
| openstackgerrit | Tony Breeds proposed openstack-infra/system-config master: Switch to cgit from gitweb https://review.openstack.org/511970 | 22:04 |
| *** bobh has quit IRC | 22:07 | |
| tonyb | fungi: I'm going AFK for $some_time feel free to take over/replace ^^ whichever works for you | 22:07 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Don't use in-line yaml lists https://review.openstack.org/511876 | 22:07 |
| *** florianf has quit IRC | 22:09 | |
| *** ijw has quit IRC | 22:09 | |
| *** florianf has joined #openstack-infra | 22:10 | |
| *** andreas_s has joined #openstack-infra | 22:10 | |
| *** slaweq has quit IRC | 22:10 | |
| *** tpsilva has quit IRC | 22:10 | |
| *** florianf has quit IRC | 22:10 | |
| *** zaneb has quit IRC | 22:11 | |
| *** tlaxkit has quit IRC | 22:11 | |
| openstackgerrit | Merged openstack-infra/bindep master: Add new syntax to allow matching multiple profile https://review.openstack.org/506502 | 22:13 |
| *** andreas_s has quit IRC | 22:14 | |
| *** andreas_s has joined #openstack-infra | 22:14 | |
| fungi | tonyb: thanks, i'll take a gander at it | 22:16 |
| *** slaweq has joined #openstack-infra | 22:19 | |
| jeblair | the statsd patches for zuulv3 are ready for review. it's a series of 3 patches starting at https://review.openstack.org/511017 | 22:21 |
| jeblair | it might be handy to have those in soonish | 22:21 |
| *** gouthamr_ has quit IRC | 22:23 | |
| mordred | jeblair: looking | 22:23 |
| *** cody-somerville has joined #openstack-infra | 22:24 | |
| SamYaple | w00t! thanks mordred. been waiting for that patch for a while | 22:24 |
| *** Swimingly has quit IRC | 22:25 | |
| mordred | SamYaple: it's like I'm your personal santa claus this week | 22:25 |
| SamYaple | the one time im thankful for chrismas in october | 22:26 |
| *** esberglu has joined #openstack-infra | 22:26 | |
| *** Swimingly has joined #openstack-infra | 22:26 | |
| *** andreas_s has quit IRC | 22:27 | |
| mordred | :) | 22:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Handle glance image pagination links better https://review.openstack.org/511963 | 22:31 |
| *** esberglu has quit IRC | 22:31 | |
| *** dbecker has quit IRC | 22:32 | |
| mordred | jeblair: also - you might find https://review.openstack.org/#/c/511963/ amusing | 22:32 |
| mordred | jeblair: for some definitions of 'amusing' | 22:32 |
| jeblair | mordred: you had me at 'pagination' | 22:34 |
| *** baoli has joined #openstack-infra | 22:34 | |
| *** andreas_s has joined #openstack-infra | 22:37 | |
| fungi | jeblair: was 511073 the one you had conflict resolution issues with? the diff for it is... nigh unreadable | 22:40 |
| fungi | as best i can tell it's shuffling a lot of class definitions around | 22:41 |
| fungi | i'm going to have to trust that the tests prove all the bits are still there | 22:42 |
| *** Swami has quit IRC | 22:45 | |
| *** andreas_s has quit IRC | 22:45 | |
| *** baoli has quit IRC | 22:45 | |
| *** andreas_s has joined #openstack-infra | 22:46 | |
| *** rbrndt has quit IRC | 22:49 | |
| jeblair | fungi: yeah.. it's *basically* just moving the Job class before the Executor class so that the executor can reference it in a class var | 22:49 |
| fungi | ahh, the gerrit webui does a little better job at showing that | 22:49 |
| jeblair | fungi: tbh, i could probably split that into two changes if you want -- one that only reorders the classes, then one that makes the actual changes (there aren't many) | 22:49 |
| fungi | but having to keyword search for statsd to find what got added | 22:49 |
| fungi | that would make it easier to see what's going on (i hate to be _that_ reviewer, but...) | 22:50 |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 22:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 22:51 |
| *** slaweq has quit IRC | 22:52 | |
| *** Apoorva has quit IRC | 22:53 | |
| *** Apoorva has joined #openstack-infra | 22:53 | |
| *** andreas_s has quit IRC | 22:54 | |
| *** andreas_s has joined #openstack-infra | 22:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Re-order executor/job classes https://review.openstack.org/511977 | 22:57 |
| fungi | jeblair: thanks! | 22:57 |
| jeblair | fungi, mordred: ^ there we go | 22:57 |
| jeblair | we'll see if i extracted that correctly :) | 22:57 |
| mordred | jeblair: oh - that's much better - I stand by my +2 - but I'm glad I was right from eyeballing | 22:58 |
| *** slaweq has joined #openstack-infra | 23:00 | |
| *** baoli has joined #openstack-infra | 23:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 23:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 23:03 |
| mordred | ok - now that shade's gate jobs are passing again normally - let's see how those fancy new v3 devstack jobs do | 23:03 |
| jeblair | oh where's that move devstack to devstack change? | 23:03 |
| mordred | oh - good question | 23:04 |
| mordred | jeblair: https://review.openstack.org/#/c/508906/ | 23:04 |
| mordred | jeblair: I haven't updated it in a while - might need to re-sync with devstack-gate repo ... should we go ahead and delete from d-g and add to devstack real quick before the rollout? | 23:05 |
| *** priteau has quit IRC | 23:05 | |
| jeblair | mordred: ++ | 23:06 |
| openstackgerrit | Lin Yang proposed openstack-infra/project-config master: Add OpenStack client check to python-rsdclient https://review.openstack.org/511650 | 23:06 |
| jeblair | mordred: you want to prep the remove patch real quick? | 23:07 |
| fungi | jeblair: you've got 2x +2 on the statsd stack though i left comments/questions on a couple | 23:07 |
| *** xarses has quit IRC | 23:08 | |
| mordred | jeblair: I need to afk real quick - will be back in an hour and will do so then | 23:09 |
| jeblair | mordred: k | 23:09 |
| SamYaple | What is the release schedule for bindep? i realize it is a non-standard release cycle | 23:10 |
| SamYaple | is it a "whnever we feel like it" deal? | 23:10 |
| *** wolverineav has joined #openstack-infra | 23:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Make statsd import failure raise exception https://review.openstack.org/511978 | 23:12 |
| jeblair | fungi: thanks, responded! | 23:13 |
| fungi | SamYaple: it has a release frequency proportional to the frequency with which people bug us about it, though i'm unsure of the scalar constant involved | 23:13 |
| *** andreas_s has quit IRC | 23:13 | |
| jeblair | i have a feeling we're about to collect more experimental data on that contast | 23:13 |
| jeblair | constant | 23:13 |
| fungi | indeed! | 23:13 |
| SamYaple | youve always been good at predictions jeblair :) | 23:15 |
| SamYaple | as long as there is a tagged release from now to anytime before queens I will be happy | 23:16 |
| *** wolverineav has quit IRC | 23:18 | |
| *** priteau has joined #openstack-infra | 23:18 | |
| *** eumel8 has quit IRC | 23:20 | |
| *** esberglu has joined #openstack-infra | 23:20 | |
| *** priteau has quit IRC | 23:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 23:22 |
| *** andreas_s has joined #openstack-infra | 23:22 | |
| *** smatzek has joined #openstack-infra | 23:23 | |
| *** ilpianista has quit IRC | 23:23 | |
| *** fried_rice is now known as efried | 23:23 | |
| *** shu-mutou-AWAY has quit IRC | 23:23 | |
| *** esberglu has quit IRC | 23:25 | |
| *** smatzek has quit IRC | 23:26 | |
| *** ilpianista has joined #openstack-infra | 23:27 | |
| *** claudiub has quit IRC | 23:27 | |
| *** gouthamr has joined #openstack-infra | 23:29 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-order executor/job classes https://review.openstack.org/511977 | 23:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 23:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 23:31 |
| *** andreas_s has quit IRC | 23:31 | |
| *** slaweq has quit IRC | 23:33 | |
| *** hongbin has quit IRC | 23:35 | |
| *** andreas_s has joined #openstack-infra | 23:36 | |
| *** kzaitsev_pi has quit IRC | 23:36 | |
| *** zaneb has joined #openstack-infra | 23:36 | |
| *** bobh has joined #openstack-infra | 23:37 | |
| *** kzaitsev_pi has joined #openstack-infra | 23:38 | |
| *** gouthamr_ has joined #openstack-infra | 23:43 | |
| *** slaweq has joined #openstack-infra | 23:43 | |
| *** gouthamr has quit IRC | 23:46 | |
| *** Apoorva has quit IRC | 23:46 | |
| *** superdan is now known as dansmith | 23:47 | |
| pabelanger | nice^ | 23:48 |
| *** markvoelker has quit IRC | 23:49 | |
| *** Goneri has quit IRC | 23:54 | |
| *** andreas_s has quit IRC | 23:54 | |
| jeblair | power outages here | 23:55 |
| *** gouthamr_ has quit IRC | 23:57 | |
| *** andreas_s has joined #openstack-infra | 23:59 | |
| *** smatzek has joined #openstack-infra | 23:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuulv3-dev.o.o https://review.openstack.org/511986 | 23:59 |
| clarkb | jeblair: because fire? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!