| *** elyezer has quit IRC | 00:11 | |
| *** elyezer has joined #zuul | 00:23 | |
| *** Wei_Liu has joined #zuul | 02:16 | |
| *** sshnaidm has quit IRC | 02:40 | |
| *** harlowja has joined #zuul | 03:11 | |
| *** xinliang has quit IRC | 03:16 | |
| *** xinliang has joined #zuul | 03:16 | |
| *** harlowja has quit IRC | 03:18 | |
| *** elyezer has quit IRC | 03:35 | |
| *** elyezer has joined #zuul | 05:48 | |
| *** xinliang has quit IRC | 06:02 | |
| *** xinliang has joined #zuul | 06:14 | |
| *** xinliang has quit IRC | 06:14 | |
| *** xinliang has joined #zuul | 06:14 | |
| *** yolanda_ has quit IRC | 06:44 | |
| *** elyezer has quit IRC | 07:10 | |
| *** elyezer has joined #zuul | 07:15 | |
| Wei_Liu | tristanC: I found out that the zuul scheduler is required to be restarted if I change main.yaml, is there anyway let zuul scheduler load main.yaml automatically? | 07:33 |
|---|---|---|
| tristanC | Wei_Liu: it's not automatic, but you can signal SIGHUP to trigger a dynamic reconfiguration | 07:57 |
| *** AJaeger has quit IRC | 08:26 | |
| *** AJaeger has joined #zuul | 08:31 | |
| *** sshnaidm has joined #zuul | 08:37 | |
| Wei_Liu | tristanC: would this action affect the running job? | 08:44 |
| tristanC | tristanC: it should not affect running jobs | 08:45 |
| *** jpena|off is now known as jpena | 08:48 | |
| *** threestrands_ has joined #zuul | 10:33 | |
| *** threestrands has quit IRC | 10:36 | |
| *** jappleii__ has joined #zuul | 10:37 | |
| *** jappleii__ has quit IRC | 10:38 | |
| *** jappleii__ has joined #zuul | 10:39 | |
| *** threestrands_ has quit IRC | 10:41 | |
| rcarrillocruz | odyssey4me: hey jesse, how's going | 10:42 |
| rcarrillocruz | turns out i started writing a nodepool ansible inventory | 10:42 |
| rcarrillocruz | iirc you wanted to deploy a just nodepool CI (with maybe jenkins) | 10:42 |
| rcarrillocruz | curious if that plugin would be of interest of you, as i'm writing it and happy to put $things to return per hosts | 10:43 |
| rcarrillocruz | thinking on returning now groups by label | 10:43 |
| rcarrillocruz | and state | 10:43 |
| rcarrillocruz | [ready[ | 10:43 |
| rcarrillocruz | [used] | 10:43 |
| rcarrillocruz | etc | 10:43 |
| odyssey4me | @rcarrillocruz hmm, actually that'd be pretty nifty I think | 10:47 |
| odyssey4me | yeah, there needs to be some way of communicating between jenkins and nodepool essentially... | 10:48 |
| odyssey4me | I guess if there's a dynamic inventory, jenkins could just run a playbook once per minute - effectively being a poll | 10:49 |
| odyssey4me | but then how would jenkins inform nodepool when it's using the node, and when it's done with a node? | 10:49 |
| *** evrardjp has joined #zuul | 10:51 | |
| rcarrillocruz | jenkins would need to request the lock on a node, then release it when done | 10:53 |
| rcarrillocruz | i.e. do what zuul does in zk.py | 10:53 |
| rcarrillocruz | i've always thought it would be useful to have a 'lock' verb on nodepool cli | 10:53 |
| rcarrillocruz | like, when zuul takes a node | 10:53 |
| rcarrillocruz | it sets the state of the node to 'in-use' | 10:53 |
| rcarrillocruz | would be nice if you could do that from CLI | 10:54 |
| rcarrillocruz | on non-zuul CIs | 10:54 |
| odyssey4me | yeah, that'd be a simple way of implementing it actually | 10:54 |
| rcarrillocruz | in the uber rough CI we have in place, we simply have one node per platform | 10:55 |
| rcarrillocruz | and we run a cronjob | 10:55 |
| rcarrillocruz | to run int tests for each one of them | 10:55 |
| rcarrillocruz | that way, we don't need to schedule | 10:55 |
| rcarrillocruz | there's always one ready | 10:55 |
| rcarrillocruz | once we finish | 10:55 |
| rcarrillocruz | we delete the node | 10:55 |
| rcarrillocruz | and nodepool recreates | 10:55 |
| rcarrillocruz | but that's super janky | 10:55 |
| rcarrillocruz | should be simple enough to do a kazoo script that requests a node, then release post job | 10:56 |
| odyssey4me | at this point we're kinda ok to go with janky, but obviously we'd like to work towards something better and more maintainable | 10:56 |
| rcarrillocruz | odyssey4me: if janky is ok, we use that in prod , it works | 10:56 |
| odyssey4me | yeah, that's our current plan | 10:56 |
| rcarrillocruz | except we use the openstack inventory | 10:56 |
| rcarrillocruz | i figure the nodepool inventory will open more interesting things | 10:56 |
| rcarrillocruz | anyway, i'll ping you when i have it up | 10:56 |
| rcarrillocruz | curious to see what kind of info you think would be useful to pull | 10:57 |
| odyssey4me | For the moment we're on hold with this - I'm hoping we can get back to it next month. Unfortunately other work was prioritised above it. | 10:57 |
| odyssey4me | I did a rather janky implementation which got me to a point... https://gist.github.com/odyssey4me/d1a202d6e340d165513f9cec1d19d5f0 | 10:58 |
| odyssey4me | it implemented a little CLI thing to request/release nodes and a daemon to watch changes in state | 10:59 |
| odyssey4me | the idea being that when the state changes, do something (like register the node in jenkins) | 10:59 |
| odyssey4me | Shrews very rightly said that we should be using the same API as zuul, instead of directly changing the data in zookeeper... hence my notes about how hacky it is. | 11:01 |
| *** elyezer has quit IRC | 11:04 | |
| * odyssey4me wonders if he needs to remind mordred to republish https://review.openstack.org/520657 to the master branch :) | 11:04 | |
| *** electrofelix has joined #zuul | 11:06 | |
| *** elyezer has joined #zuul | 11:06 | |
| *** JasonCL has joined #zuul | 11:14 | |
| *** JasonCL has quit IRC | 11:15 | |
| *** JasonCL has joined #zuul | 11:15 | |
| *** _ari_|DevConf is now known as _ari_|conf | 11:32 | |
| *** _ari_|conf is now known as _ari_|brno | 11:32 | |
| *** elyezer has quit IRC | 12:23 | |
| *** elyezer has joined #zuul | 12:25 | |
| *** sshnaidm has quit IRC | 12:28 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** sshnaidm has joined #zuul | 12:43 | |
| *** sshnaidm_ has joined #zuul | 13:11 | |
| *** sshnaidm has quit IRC | 13:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 13:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 13:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 13:16 |
| *** rlandy has joined #zuul | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| rcarrillocruz | odyssey4me, Shrews : the fact there's interest for non-zuul interactions with nodepool, yet the api for interacting for it is within zuul tree (zk.py), it begs the question if there should be at some point a very light nodepool-lib | 13:55 |
| odyssey4me | yeah, that'd be nice - obviously it'd make it much easier to implement the right bits in various code bases, even if just in nodepool and zuul to start with | 13:57 |
| Shrews | rcarrillocruz: odyssey4me: that was actually my plan (the lib idea) when we started zuulv3. but it was decided that we'd just break it out later if we needed to. | 14:07 |
| odyssey4me | Shrews it'd be super-nice if nodepool standalone had the CLI options to take nodes through their life cycle... that'd make integration with third party tooling far simpler | 14:08 |
| Shrews | odyssey4me: can't do that because of the requirement to hold locks during the lifecycle | 14:09 |
| *** dmsimard is now known as dmsimard|afk | 14:17 | |
| *** myoung is now known as myoung|reboot | 14:20 | |
| *** myoung|reboot is now known as myoung | 14:29 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 14:58 |
| *** elyezer has quit IRC | 15:03 | |
| *** elyezer has joined #zuul | 15:05 | |
| *** myoung is now known as myoung|brb | 15:24 | |
| pabelanger | I'm not sure if I am going to explain it properly, but I'm seeing a lot of dynamic reload configurations related to a single project this morning in zuul. openstack/networking-odl | 15:30 |
| pabelanger | our results queue grows large, due to reloads being processed, eventually we finish that, results contain new info, and reloads happen again with openstack/networking-odl | 15:31 |
| pabelanger | is it possible that job results (pass / fail) could be triggering a reload for openstack/networking-odl ? I new when a new patchset was added into check, it would do so. But I am seeing a lot of Re-enqueing change ... in queue .. | 15:32 |
| pabelanger | 2018-01-29 15:33:18,172 DEBUG zuul.Pipeline.openstack.gate: Re-enqueing change <Change 0x7fbdb66afd30 528717,52> in queue <ChangeQueue gate: openstack/networking-odl> for example | 15:33 |
| pabelanger | http://paste.openstack.org/show/657071/ | 15:35 |
| *** myoung|brb is now known as myoung | 15:37 | |
| Shrews | clarkb: i left a comment on https://review.openstack.org/538529. I think there's something else happening there. | 15:47 |
| clarkb | Shrews: that is the behavior I saw with extra logging added to the test | 15:48 |
| clarkb | basially there would be an extra node in a deleting state and we'd double delete it never deleting the second node so quota would not free up | 15:49 |
| clarkb | happened about one in 10 test runs locally | 15:49 |
| Shrews | clarkb: but... there's no node | 15:49 |
| Shrews | so i'm confused | 15:49 |
| Shrews | clarkb: oh! we create an artificial one for leaked instances | 15:52 |
| Shrews | so yeah, we could be racing that one | 15:52 |
| Shrews | clarkb: changed to +3 :) | 15:54 |
| clarkb | thanks, good to know nodepool isnt being extra weird here :) | 15:54 |
| Shrews | we could have also just compared the node id's there, but state also works | 15:54 |
| clarkb | fwiw I added a print(nodes) just above the loop to figure it out | 15:54 |
| Shrews | nodepool is just complex. even though I wrote most of it, i have to continually re-inspect code to remind myself of how it works :/ | 15:55 |
| Shrews | s/wrote/re-wrote/ | 15:56 |
| corvus | i'm not feeling well today -- would anyone else like to chair the meeting? | 15:58 |
| corvus | or we could cancel it | 16:02 |
| Shrews | corvus: i think i can stumble my way through the commands. but i think we can probably skip this one maybe? | 16:02 |
| Shrews | corvus: we'll work it out. go rest | 16:03 |
| clarkb | I'm taking care of kids all this week due to some last minute travel plans by my wife. Happy to help run a meeting if the kids will let me sit still long enough | 16:03 |
| corvus | Shrews: ok thanks | 16:03 |
| *** jappleii__ has quit IRC | 16:05 | |
| Shrews | I don't see anything on https://wiki.openstack.org/wiki/Meetings/Zuul so maybe we can just skip it today | 16:05 |
| pabelanger | +1 | 16:10 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix race in test_failed_provider https://review.openstack.org/538529 | 16:21 |
| *** jpena is now known as jpena|brb | 16:44 | |
| *** JasonCL has quit IRC | 17:05 | |
| *** JasonCL has joined #zuul | 17:08 | |
| *** sshnaidm_ is now known as sshnaidm | 17:09 | |
| *** JasonCL has quit IRC | 17:10 | |
| *** elyezer has quit IRC | 17:23 | |
| *** elyezer has joined #zuul | 17:23 | |
| *** jpena|brb is now known as jpena | 17:31 | |
| corvus | tristanC: i have one small request (for another test) on 535553 -- then i think we can merge it (and strike an item off the 3.0 blocker list!) | 17:40 |
| *** myoung is now known as myoung|food | 17:42 | |
| *** JasonCL has joined #zuul | 17:48 | |
| *** weshay|ruck is now known as weshay|ruck|brb | 18:09 | |
| *** myoung|food is now known as myoung | 18:14 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 18:17 |
| *** jpena is now known as jpena|off | 18:18 | |
| *** weshay|ruck|brb is now known as weshay | 18:20 | |
| *** weshay is now known as weshay|ruck | 18:20 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Partial revert for disabled provider change https://review.openstack.org/538995 | 18:28 |
| Shrews | Can I get some eyes on 538995? I don't want a just merged change to accidentally hit production. It's also a good demonstration for "always include tests". :) | 18:30 |
| * Shrews reprimands himself | 18:30 | |
| clarkb | Shrews: looks like more than a partial revert as the max servers stuff is removed entirely? | 18:34 |
| Shrews | clarkb: no, it keeps the "accept max-servers less than 0" part | 18:34 |
| Shrews | clarkb: which was something you asked for a while back, iirc | 18:35 |
| clarkb | its only checking max_concurrency not max_servers | 18:35 |
| Shrews | correct | 18:36 |
| clarkb | oh I see it is supposed to fall through a bit further | 18:37 |
| Shrews | right | 18:37 |
| Shrews | i mean, i can revert the entire thing and add the new test and -1 support separately if that's a cleaner solution | 18:38 |
| Shrews | but that's 2, maybe 3, reviews compared to this single one | 18:39 |
| clarkb | no I see it now just had to pull up the old change and do mental diffing | 18:39 |
| pabelanger | tobiash: looking at 536930, the test for test_over_quota might also need to be updated. If I understand, request 2 would never be fulfilled now, so we need a 3rd? | 18:39 |
| tobiash | pabelanger: yeah it also needs updating | 18:41 |
| tobiash | but have to look at this closer | 18:41 |
| tobiash | probably tomorrow morning | 18:41 |
| pabelanger | ack | 18:41 |
| tobiash | I think request 2 might be paused now as it is intended and we need to check for this, delete request 1 and check if request 2 now gets processed | 18:42 |
| tobiash | at least that's what I remember from a few months ago | 18:42 |
| tobiash | but will take a closer look tomorrow | 18:42 |
| clarkb | I'm not sure how many of us here knew spearce but https://groups.google.com/forum/#!topic/repo-discuss/B4P7G1YirdM is worth a read if you knew him | 18:48 |
| clarkb | tl;dr cancer sucks | 18:48 |
| Shrews | clarkb: oh wow | 18:54 |
| *** sshnaidm is now known as sshnaidm|afk | 19:12 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 19:23 |
| *** harlowja has joined #zuul | 19:25 | |
| Shrews | mordred: pabelanger: clarkb: SpamapS: Any thoughts on skipping today's Zuul meeting in 2hrs? Nothing on the agenda and corvus will be afk. Unless someone has something to discuss, I vote push to next week. | 20:00 |
| SpamapS | +1, I can't attend | 20:01 |
| pabelanger | the only real thing I wanted to discuss was migrating zuul.o.o to zookeeper cluster, due to outage this weekend. But more a heads up of something we want to do sooner then later, and if there is any concerns | 20:01 |
| pabelanger | and how we might do that, but that is more an openstack-inra thing | 20:01 |
| clarkb | I don't have anything urgent to go over | 20:01 |
| SpamapS | pabelanger: oh, did you have to dump the zk db? | 20:01 |
| SpamapS | or super long snap recovery? | 20:02 |
| Shrews | pabelanger: yeah, we could cover that at tomorrow's infra meeting | 20:02 |
| pabelanger | SpamapS: we haven't done it yet. But we did loose zookeeper host for 4 hours yesterday | 20:02 |
| pabelanger | Shrews: yup, wfm | 20:02 |
| Shrews | SpamapS: iirc, we greatly reduced our snapshot count a while back | 20:03 |
| Shrews | due to diskspace issues | 20:03 |
| Shrews | though for the life of me, i cannot find where we set that :/ | 20:10 |
| pabelanger | system-config? | 20:10 |
| clarkb | its a puppet module setting iirc | 20:10 |
| Shrews | codesearch.o.o of snapCount doesn't pull up anything | 20:12 |
| clarkb | purge_interval => 6, | 20:14 |
| clarkb | in system-config/manifests/site.pp | 20:14 |
| clarkb | the puppet module defaulted to higher snapcount out of the box iirc | 20:17 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 20:19 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 20:21 |
| clarkb | the length of the outage was due to rax saying leave the node alone | 20:21 |
| clarkb | then they brought back the hardware for it in a broken way aiui | 20:21 |
| pabelanger | yes | 20:26 |
| SpamapS | bummer | 20:44 |
| *** elyezer has quit IRC | 21:08 | |
| *** elyezer has joined #zuul | 21:09 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Partial revert for disabled provider change https://review.openstack.org/538995 | 21:43 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 21:43 |
| *** myoung is now known as myoung|bbl | 21:46 | |
| fungi | no meeting today? | 22:02 |
| Shrews | fungi: nope | 22:02 |
| fungi | thanks! | 22:03 |
| *** threestrands has joined #zuul | 22:05 | |
| *** threestrands has quit IRC | 22:05 | |
| *** threestrands has joined #zuul | 22:05 | |
| *** threestrands_ has joined #zuul | 22:07 | |
| *** threestrands has quit IRC | 22:08 | |
| *** threestrands_ has quit IRC | 22:08 | |
| *** threestrands has joined #zuul | 22:08 | |
| *** dtruong has quit IRC | 22:13 | |
| *** dmellado has quit IRC | 22:17 | |
| *** dmellado has joined #zuul | 22:20 | |
| *** threestrands_ has joined #zuul | 22:21 | |
| *** threestrands has quit IRC | 22:23 | |
| *** jappleii__ has joined #zuul | 22:35 | |
| *** jappleii__ has quit IRC | 22:36 | |
| *** jappleii__ has joined #zuul | 22:37 | |
| *** threestrands_ has quit IRC | 22:37 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 22:43 |
| *** dtruong has joined #zuul | 23:04 | |
| *** weshay|ruck is now known as weshay|ruck|afk | 23:07 | |
| *** rlandy is now known as rlandy|bbl | 23:11 | |
| *** Wei_Liu has quit IRC | 23:13 | |
| *** logan- has quit IRC | 23:45 | |
| *** clarkb has quit IRC | 23:45 | |
| *** smyers has quit IRC | 23:45 | |
| *** logan- has joined #zuul | 23:51 | |
| *** clarkb has joined #zuul | 23:51 | |
| *** smyers has joined #zuul | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!