| corvus | logan-: ++ | 00:30 |
|---|---|---|
| *** jiapei has joined #zuul | 01:34 | |
| *** dtruong2_ has joined #zuul | 03:24 | |
| *** spsurya_ has joined #zuul | 03:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: rstrip baseurl trailing / https://review.openstack.org/581945 | 03:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: rstrip baseurl trailing / https://review.openstack.org/581945 | 04:25 |
| logan- | thanks tristanC | 04:26 |
| tristanC | logan-: you're welcome, thanks for the report :-) | 04:28 |
| *** bhavik1 has joined #zuul | 05:20 | |
| *** bhavik1 has quit IRC | 05:56 | |
| *** dtruong2_ has quit IRC | 05:57 | |
| *** nchakrab has joined #zuul | 06:07 | |
| *** gtema has joined #zuul | 07:15 | |
| *** adam_g has quit IRC | 07:53 | |
| *** adam_g has joined #zuul | 07:54 | |
| *** sshnaidm|off has quit IRC | 08:27 | |
| *** hashar has joined #zuul | 08:29 | |
| *** bhavik1 has joined #zuul | 08:37 | |
| *** bhavik1 has quit IRC | 08:41 | |
| *** electrofelix has joined #zuul | 08:46 | |
| *** sshnaidm has joined #zuul | 08:48 | |
| *** sshnaidm is now known as sshnaidm|rover | 08:48 | |
| *** sambetts|afk is now known as sambetts | 08:57 | |
| tobiash | Shrews: I think we might have some problem in the static driver | 09:17 |
| tobiash | Shrews: I'm debugging an issue where I have a node request for a static node, there is a zNode in state ready but it still doesn't get this node | 09:17 |
| tobiash | Shrews: the request is in state requested and I see the following regularly in the logs: http://paste.openstack.org/show/725669/ | 09:18 |
| tobiash | Shrews: then the handler pauses and when it unpauses it does the same again | 09:20 |
| *** jiapei has quit IRC | 09:34 | |
| odyssey4me | hi folks, can I get some reviews for https://review.openstack.org/581348 - while it may be less than perfect, at least it's an improvement to help new developers... the current docs are quite broken | 09:53 |
| odyssey4me | also, tobiash do you think I should apply https://review.openstack.org/580728 to https://review.openstack.org/581329 given that I've left part of that in the test addition | 09:54 |
| odyssey4me | I could also merge https://review.openstack.org/580728 into https://review.openstack.org/581329 if you like | 09:55 |
| tobiash | odyssey4me: do they conflict? | 10:18 |
| odyssey4me | they do not, however if I do the patch then it'll be easier to get two cores to approve ;) | 10:18 |
| tobiash | odyssey4me: are you sure you mean https://review.openstack.org/580728 ? | 10:20 |
| tobiash | that doesn't look related | 10:20 |
| odyssey4me | tobiash: this is the part I mean: https://review.openstack.org/#/c/580728/1/nodepool/tests/test_launcher.py | 10:20 |
| odyssey4me | ie, the clean-up | 10:20 |
| odyssey4me | in my patch I did leave the 'finished waiting' debug line, so I can clean that up if you like | 10:21 |
| tobiash | odyssey4me: I don't see that line in your new test so I think it's fine as is | 10:23 |
| odyssey4me | oh, heh, I cleaned it out yesterday | 10:23 |
| odyssey4me | nvm, nothing to see here :p | 10:23 |
| tobiash | 580728 is just a trivial stylistic cleanup that can be drive-by approved by any zuul-core | 10:25 |
| tobiash | mordred: wanna do a quick review? ^ | 10:25 |
| tobiash | Shrews: I think I understood my nodepool issue. I have a static provider with a pool containing multiple static nodes of different labels | 10:27 |
| tobiash | Shrews: so now I get a request for label A which is fulfilled, after that I get another request for label A which gets pending and pauses the handler because I only have one node of label A | 10:28 |
| tobiash | Shrews: after that I get a third request but now for label B which is free but not serviced because there is a common handler for the pool that is blocked until the second A request can be fulfilled | 10:29 |
| tobiash | Shrews: I can do a workaround by having only one label per pool but I think we should make it possible to handle this within the same pool | 10:30 |
| tobiash | Shrews: otherwise at some point I'll have 50 or more pools that each declines almost every node request which will just create synchronization contention and reduce reaction times of the whole system | 10:31 |
| tobiash | s/reduce/increase | 10:31 |
| openstackgerrit | Roman Gorshunov proposed openstack-infra/zuul master: Fix: correct path to the project's public key https://review.openstack.org/582143 | 10:54 |
| *** hashar is now known as hasharAway | 11:21 | |
| *** hasharAway has quit IRC | 11:27 | |
| openstackgerrit | Roman Gorshunov proposed openstack-infra/zuul master: Fix: correct path to the project's public key https://review.openstack.org/582143 | 11:29 |
| openstackgerrit | Roman Gorshunov proposed openstack-infra/zuul master: Fix: correct path to the project's public key https://review.openstack.org/582143 | 11:41 |
| Shrews | tobiash: so sounds to me that nodepoool is working as designed, but the request handling is suboptimal given the nature of the static driver | 12:03 |
| Shrews | tobiash: sounds like a different request handling mechanism is needed, but that's a larger change | 12:04 |
| Shrews | tobiash: i've seen other requests for modifying that algorithm, but i think any change to it for something more complex is going to require coordination between the launchers | 12:05 |
| Shrews | and i don't think we're ready to tackle that yet | 12:05 |
| tobiash | htm, that will create a problem for me in the long run :/ | 12:09 |
| tobiash | ok, I'll create a pool per label for now | 12:10 |
| *** nchakrab_ has joined #zuul | 12:30 | |
| *** nchakrab has quit IRC | 12:33 | |
| odyssey4me | Shrews: if you could put https://review.openstack.org/581329 into your review queue we'd certainly appreciate it, thanks again for all the help getting those tests started | 12:34 |
| *** rlandy has joined #zuul | 12:38 | |
| *** bhavik1 has joined #zuul | 12:50 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Cleanup test_over_quota https://review.openstack.org/580728 | 12:51 |
| *** bhavik1 has quit IRC | 13:10 | |
| *** dkranz has quit IRC | 13:11 | |
| *** nchakrab_ has quit IRC | 13:15 | |
| *** nchakrab has joined #zuul | 13:16 | |
| *** nchakrab has quit IRC | 13:37 | |
| *** nchakrab has joined #zuul | 13:38 | |
| *** pwhalen has quit IRC | 13:41 | |
| *** nchakrab has quit IRC | 13:43 | |
| *** pwhalen has joined #zuul | 13:45 | |
| *** pwhalen has joined #zuul | 13:45 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Remove redhat-rpm-config bindep https://review.openstack.org/581252 | 13:52 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Fix github setup in zuul from scratch https://review.openstack.org/581253 | 13:52 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: fix zuul from scratch user and group creation https://review.openstack.org/581254 | 13:52 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions for deploying zuul with openSUSE https://review.openstack.org/581255 | 13:52 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions on building static web components https://review.openstack.org/581256 | 13:52 |
| *** hwoarang has quit IRC | 14:12 | |
| *** dkranz has joined #zuul | 14:13 | |
| *** nchakrab has joined #zuul | 14:16 | |
| *** nchakrab_ has joined #zuul | 14:22 | |
| mordred | tobiash: do you have a list of the alpine packages needed to install zuul you could share? | 14:23 |
| *** nchakrab has quit IRC | 14:26 | |
| tobiash | mordred: http://paste.openstack.org/show/725719/ | 14:32 |
| mordred | tobiash: thanks! | 14:32 |
| tobiash | mordred: but that was a growing list which is not necessary minimal today | 14:33 |
| tobiash | mordred: executor additionally needs 'bubblewrap' | 14:33 |
| *** GonZo2000 has joined #zuul | 14:34 | |
| *** GonZo2000 has quit IRC | 14:34 | |
| *** GonZo2000 has joined #zuul | 14:34 | |
| tobiash | mordred: but I'd recommend ubuntu (probably bionic) so wheels from pypi can be used | 14:35 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: zuul-web: jobs list endpoint: return 404 when tenant not found https://review.openstack.org/582200 | 14:36 |
| *** mugsie has quit IRC | 14:36 | |
| *** mugsie has joined #zuul | 14:36 | |
| *** mugsie has quit IRC | 14:36 | |
| *** mugsie has joined #zuul | 14:36 | |
| corvus | tobiash, Shrews: the only quick fix i can think of would be to have the static handler never pause, however, that might generate *a lot* of zk traffic. a second option i thought about would be to internally make separate pool workers per-label (so automate what tobiash is doing), but support for multiple labels makes that complicated. | 14:37 |
| tobiash | corvus: if the static handler never pauses would it grab another request then? | 14:39 |
| tobiash | corvus: we could add a backoff to reduce the zk traffic | 14:40 |
| Shrews | corvus: pausing is not controlled by the driver. we'd have to come up with a mechanism to do that | 14:40 |
| rcarrillocruz | i remember there was at some point a story for 'trigger zuul on $package update URL' | 14:40 |
| corvus | Shrews: i know, but it would be a one-liner to change that | 14:40 |
| corvus | Shrews: basically change 'self.paused = True' to 'self.setPaused(True)' and then make setPaused a noop in static | 14:43 |
| corvus | then we can keep using the runHandler method | 14:43 |
| corvus | mind you, i'm not ready to say that's a *good* idea, just an idea. :) | 14:43 |
| *** nchakrab_ has quit IRC | 14:44 | |
| corvus | tobiash: yeah, if there are no 'paused' handlers in a pool worker, it continues handling new requests | 14:44 |
| mordred | tobiash: I have trickier ways so I don't need to reuse those wheels - will show you something soon | 14:44 |
| tobiash | corvus: I think if we would do that we'd need some backoff mechanism (per request?) | 14:45 |
| corvus | tobiash: yeah, that's sort of the missing piece -- how to avoid piling up a bunch of active requests, each of which does a full query of all the zk nodes every few seconds? the rest of the framework isn't really designed for that. that's what paused is for. | 14:47 |
| corvus | Shrews: currently, if a request is paused, we just keep trying that one request each time through the poolworker loop, right? | 14:50 |
| Shrews | corvus: correct | 14:50 |
| corvus | there's no poolworker sleep, right? | 14:52 |
| corvus | so in that situation, we basically just have one thread reading zk nodes continually? | 14:52 |
| Shrews | i don't think so. i'd have to look at the code again | 14:52 |
| corvus | considering that, we may actually not generate any more traffic this way | 14:53 |
| corvus | instead of one handler reading all the nodes over and over, we'd have N handlers reading all the nodes in sequence | 14:53 |
| Shrews | every pool worker reads all requests every 10s, unless there is a paused handler | 14:56 |
| corvus | oh i missed the 10s sleep | 14:57 |
| tobiash | just wanted to say that it retries every 10s according to the logs ;) | 14:58 |
| corvus | there's still a bunch of details that would need to be worked out. there's currently no way to resume a request in progress other than if it's paused, and i don't think we'd want to have more than one request for a label active at a time. so, in the end, even this 'quick' fix may be a substantial change that needs to be thought through. | 14:59 |
| tobiash | I think we should at least document the current behavior and the workaround | 15:00 |
| corvus | it may be better to rethink the algorithm for static nodes from scratch, then see if there's a way to support that. | 15:00 |
| corvus | tobiash: yes | 15:00 |
| *** hashar has joined #zuul | 15:00 | |
| tobiash | so I'm ok for now with the workaround given that I need to do further for us more important optimizations in zuul as we're now at a point where users are complaining about stalls, node failures, ... | 15:02 |
| tobiash | but it's just many users find many bugs... | 15:03 |
| tobiash | so I'm much of my time in fire fighting mode atm | 15:04 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: zuul-web: jobs list endpoint: Add test and fix tenant not found 500 error https://review.openstack.org/581810 | 15:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 15:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add alpine packages to bindep.txt https://review.openstack.org/582276 | 15:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move the database libraries to requirements https://review.openstack.org/582277 | 15:20 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: zuul-web: config_errors endpoint: return 404 when tenant not found https://review.openstack.org/582281 | 15:34 |
| *** hashar has quit IRC | 15:39 | |
| pabelanger | Question around 3pci jobs, in the case of a downstream zuul (rdoproject) loading a job from upstream (openstack.org), if downstream only wanted to run jobs on master branch (devstack) and not stable branches, would it make sense to express some how in tenant configuration to only include the job@master and ignore all other branches? We had an issue where zuul properly loaded the master, because we fixed it in | 15:55 |
| pabelanger | https://review.openstack.org/581441 but I didn't backport properly. I think we should fix it for devstack, but does raise the question of loading job per branch | 15:55 |
| corvus | pabelanger: i'm confused -- why don't you just specify on the project where you're running the job to only run it on master? | 15:58 |
| corvus | mordred: looking at 582277 -- that one feels a little weird. we have, at the request of some of our users, tried to keep the requirements file to the minimum requirements. could we perhaps use [extras] in the requirements files to tell pbrx to add that in when making an image, but it isn't required for everyone? | 16:03 |
| pabelanger | corvus: I believe I did, maybe did it wrong. But I think the traceback from zuul is happening much early then that, we see a traceback when zuul scheduler does a reconfiguration, when it tries to load configs from stable branches, it fails due to syntax issues | 16:03 |
| pabelanger | let me get traceback | 16:03 |
| mordred | corvus: yeah - we could do that - I agree, it feels weird | 16:04 |
| pabelanger | https://tree.taiga.io/project/morucci-software-factory/issue/1495 sorry, it isn't the best format, but comment 1 from tristanC shows the error | 16:04 |
| *** nchakrab has joined #zuul | 16:05 | |
| corvus | pabelanger: oh, i see. yeah, i think that mostly should be fixed in devstack. now, we also have talked about listing specific branches in the main.yaml file, but i don't think i would recommend using that as a solution in this case even if it had been implemented. | 16:06 |
| corvus | pabelanger: to be clear, i understand the issue from the text you wrote here. that link did not help. :( | 16:06 |
| clarkb | not in a great place to dig in right now but one thing I have noticed with third party check is that kata-containers/proxy "changes"/PRs take a long time to queue in the zuul status | 16:07 |
| clarkb | probably something we need to look into for better user communication | 16:07 |
| clarkb | and now meetings | 16:07 |
| pabelanger | corvus: okay, thanks. Yes, that is what I was trying to ask, some list of branches in main.yaml. And agree fixing in devstack is best case here but wanted to raise the question | 16:08 |
| corvus | pabelanger, mordred: have time to chat about pabelanger's issues so far with third party ci? | 16:13 |
| pabelanger | I do yes | 16:13 |
| *** hwoarang has joined #zuul | 16:17 | |
| *** hwoarang has quit IRC | 16:17 | |
| *** hwoarang has joined #zuul | 16:17 | |
| *** hwoarang has quit IRC | 16:18 | |
| *** nchakrab has quit IRC | 16:22 | |
| mordred | corvus, pabelanger: yes - was just finishing up a chat with notmyname | 16:22 |
| corvus | pabelanger, mordred: cool -- want to use the pbx server and chat on the phone? | 16:25 |
| pabelanger | sure, let me see if I have my headphones | 16:25 |
| *** spsurya_ has quit IRC | 16:26 | |
| mordred | corvus: yah - gimme just a sec and I can join up | 16:27 |
| pabelanger | I do, which room. will join in a minute | 16:27 |
| corvus | how about 6000 | 16:28 |
| corvus | https://wiki.openstack.org/wiki/Infrastructure/Conferencing | 16:28 |
| corvus | other folks are welcome to join, but also we can take notes in https://etherpad.openstack.org/p/7X6jiX8fTE and produce a summary of conclusions | 16:29 |
| *** nchakrab has joined #zuul | 16:31 | |
| corvus | postponed ~3 hours | 16:39 |
| pabelanger | +1 | 16:40 |
| *** acozine1 has joined #zuul | 16:41 | |
| *** gtema has quit IRC | 16:58 | |
| *** GonZo2000 has quit IRC | 17:02 | |
| *** sambetts is now known as sambetts|afk | 17:03 | |
| jlk | I know y'all don't use GitHub directly, but this could be useful https://blog.github.com/2018-07-12-security-vulnerability-alerts-for-python/ | 17:08 |
| jlk | https://github.com/openstack-infra/zuul/network/dependencies | 17:10 |
| jlk | hrm, looks like it hasn't picked up requirements.txt yet | 17:11 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 17:31 | |
| *** nchakrab has quit IRC | 17:34 | |
| *** sshnaidm|afk has quit IRC | 17:36 | |
| *** electrofelix has quit IRC | 17:42 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 17:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix broken templates for projects in dependent pipelines https://review.openstack.org/581888 | 18:01 |
| corvus | fbo: ^ that may be of interest to you | 18:01 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: client: show subcommand propose a more meaningful help message https://review.openstack.org/582321 | 18:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop publishing docs to docs.openstack.org https://review.openstack.org/582323 | 18:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Stop publishing docs to docs.openstack.org https://review.openstack.org/582325 | 18:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Stop publishing to docs.openstack.org https://review.openstack.org/582326 | 18:12 |
| *** harlowja has joined #zuul | 18:24 | |
| *** hwoarang has joined #zuul | 18:58 | |
| *** hwoarang has quit IRC | 18:58 | |
| *** hwoarang has joined #zuul | 18:58 | |
| *** acozine1 has quit IRC | 19:01 | |
| *** dkranz has quit IRC | 19:02 | |
| *** dkranz has joined #zuul | 19:18 | |
| corvus | pabelanger, mordred: i'm back from lunch whenever you're ready | 19:36 |
| pabelanger | corvus: yes, 5mins please | 19:40 |
| pabelanger | okay, joining again now | 19:44 |
| pabelanger | corvus: mordred: connected to 6000@pbx.openstack.org | 19:45 |
| *** acozine1 has joined #zuul | 20:09 | |
| *** dkranz has quit IRC | 20:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 20:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role for installing docker and configuring registry mirror https://review.openstack.org/580730 | 21:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 21:09 |
| mordred | corvus, pabelanger: that ^^ is worth reviewing/re-reviewing now | 21:09 |
| mordred | pabelanger: I updated the zuul-jobs role one more time with a line that had been in the job definition in the zuul repo but really belonged in the install-docker role | 21:09 |
| corvus | mordred: oh that's nice and fairly intuitive. it might be good to put a comment in zuul's setup.cfg stating that though, so people know what those keywords mean | 21:14 |
| corvus | mordred: can extras go in requirements.txt instead of setup.cfg? | 21:15 |
| corvus | apparently so? https://www.python.org/dev/peps/pep-0508/#extras | 21:17 |
| mordred | corvus: that's how you indicate consuming them | 21:18 |
| mordred | corvus: so no, for us to declare an extra, we have to put it in setup.cfg | 21:19 |
| corvus | that's a really confusing bit of documentation | 21:20 |
| corvus | apparently "when the extra is used in a dependency specification" doesn't mean what i thought it did. | 21:21 |
| corvus | yep. confirmed. i'm wrong for wanting wrong things: https://github.com/pypa/pip/issues/1161 | 21:22 |
| corvus | apparently we should not be using requirements.txt | 21:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Specify a prefix for building the images https://review.openstack.org/582396 | 21:22 |
| mordred | corvus: yah. we're wrong to want to express dependencies | 21:22 |
| mordred | corvus: if you express specific dependencies, then you make it harder for a developer who wants to install your software with a different set of dependencies to get that done | 21:23 |
| mordred | corvus: so explicitly listing your actual dependncies is wrong think | 21:23 |
| mordred | also, listing them declaratively in requirements.txt is wrong thing - if you list them at all, they should be listed inside of a python executable file as a list argument to a key of a dict, and they should not specify versions | 21:24 |
| corvus | i am learing so much about how to be a better programmer | 21:25 |
| mordred | corvus: apparently delivering web apps and vendoring your depends is the ultimate in good programming and you should do more of it | 21:25 |
| *** dkranz has joined #zuul | 21:38 | |
| Shrews | corvus: do you want an opportunity to review https://review.openstack.org/581329 ? | 21:47 |
| corvus | Shrews: yep, will do, thanks | 21:48 |
| *** rlandy is now known as rlandy|afk | 21:55 | |
| corvus | tristanC, tobiash: https://review.openstack.org/581553 approved with comment | 21:55 |
| *** sshnaidm|afk has joined #zuul | 21:57 | |
| corvus | tristanC: https://review.openstack.org/581793 approved with comment | 22:00 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add ability to ignore provider quota for a pool https://review.openstack.org/581329 | 22:03 |
| odyssey4me | w00t! thanks tobiash corvus Shrews :) | 22:05 |
| corvus | odyssey4me: thank you :) | 22:06 |
| * mordred hands odyssey4me what he thinks is probably still mostly a bunny | 22:08 | |
| * odyssey4me considers the existential crisis of what is mostly a bunny, while sipping a perfectly good beverage. | 22:09 | |
| odyssey4me | ok folks, I'm out for the night - cheerio! | 22:12 |
| * mordred panics, realizing he is NOT sipping a perfectly good beverage | 22:12 | |
| mordred | have a good one odyssey4me ! | 22:12 |
| openstackgerrit | Merged openstack-infra/zuul master: timer: skip projects not using the pipeline https://review.openstack.org/581553 | 22:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 22:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Specify a prefix for building the images https://review.openstack.org/582396 | 22:33 |
| openstackgerrit | Merged openstack-infra/zuul master: zuul-web: key endpoint: return 404 when tenant or project not found https://review.openstack.org/581793 | 22:36 |
| *** acozine1 has quit IRC | 22:38 | |
| *** rlandy|afk is now known as rlandy | 23:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Build container images using pbrx https://review.openstack.org/580160 | 23:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Specify a prefix for building the images https://review.openstack.org/582396 | 23:36 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!