| *** dims has quit IRC | 00:42 | |
| *** JasonCL has quit IRC | 01:07 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: docs: add build status documentation https://review.openstack.org/561489 | 01:08 |
|---|---|---|
| *** 18WAAQUZ1 has joined #zuul | 01:14 | |
| *** 18WAAQUZ1 has quit IRC | 02:06 | |
| *** JasonCL has joined #zuul | 02:15 | |
| *** JasonCL has quit IRC | 02:16 | |
| *** Wei_Liu has joined #zuul | 02:34 | |
| *** JasonCL has joined #zuul | 02:36 | |
| *** JasonCL has quit IRC | 02:58 | |
| *** AJaeger has quit IRC | 06:03 | |
| *** bhavik1 has joined #zuul | 06:41 | |
| *** snapiri has joined #zuul | 06:49 | |
| *** hashar has joined #zuul | 06:50 | |
| *** threestrands has quit IRC | 07:47 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** bhavik1 has quit IRC | 08:19 | |
| *** electrofelix has joined #zuul | 08:40 | |
| *** amito has joined #zuul | 08:48 | |
| *** JasonCL has joined #zuul | 09:49 | |
| electrofelix | hitting a problem with zuul v2, it seems that in a patch queue, any subsequent change that adds a Depends-On entry creating a cycle can prevent the original patch (that does not have a cycle) from being properly enqueued at: https://github.com/openstack-infra/zuul/blob/2.6.0/zuul/connection/gerrit.py#L116-L117 | 11:08 |
| electrofelix | Get the following traceback http://paste.openstack.org/show/719290/ | 11:09 |
| electrofelix | Is it safe to change the exception behaviour to be caught in _updateChanges to skip adding to needs_changes at https://github.com/openstack-infra/zuul/blob/2.6.0/zuul/source/gerrit.py#L320-L322 | 11:14 |
| electrofelix | This looks to be fixed in v3 to correctly skip enqueuing subsequent changes rather than the parent change, so just looking for a bit of guidence on how to bandaid this for v2 until we're in a position to migrate? | 11:17 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 11:17 |
| electrofelix | Sorry looks like the exception is actually thrown at https://github.com/openstack-infra/zuul/blob/2.6.0/zuul/source/gerrit.py#L308-L310 my mistake | 11:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 11:31 |
| *** elyezer_ has quit IRC | 11:33 | |
| *** AJaeger has joined #zuul | 11:40 | |
| *** elyezer_ has joined #zuul | 11:46 | |
| *** dmsimard is now known as dmsimard|off | 12:16 | |
| *** dims has joined #zuul | 12:30 | |
| *** rlandy has joined #zuul | 12:37 | |
| *** snapiri has quit IRC | 12:49 | |
| *** snapiri has joined #zuul | 12:57 | |
| openstackgerrit | Merged openstack-infra/zuul master: Tell geard to use keepalives https://review.openstack.org/425248 | 13:33 |
| *** jimi|ansible has quit IRC | 14:11 | |
| fungi | electrofelix: i didn't even realize we'd fixed it in v3 and was still assuming it behaved the same | 14:14 |
| fungi | but yeah, we've been well aware of that symptom all the way back to when depends-on got introduced | 14:15 |
| fungi | the idea was that just to be as safe as possible, zuul aborted all processing for any set of changes where a dependency loop appears | 14:15 |
| fungi | it tended to be annoying to have to track down, since it involved searching the scheduler's log for that particular exception or working it out by searching around in gerrit | 14:17 |
| fungi | i think we've mostly resolved that we could probably safely leave a comment on changes involved in a loop, though determining which change(s) to comment on if you're not going to comment on all of them could be a bit of a challenge | 14:18 |
| fungi | i also don't know if anyone's working on that yet | 14:18 |
| corvus | electrofelix: if i had a simple answer, i'd give it, but i think not only did we change some handling around dependencies in the gerrit driver, but also some aspects of change caching -- they may be interrelated. | 14:21 |
| Shrews | Just FYI, I am afk while having electrical work done on the house this morning. Hope to be fully online soonish | 14:52 |
| *** mugsie has quit IRC | 14:53 | |
| electrofelix | corvus: I think that's a sufficient answer to suggest it's not safe for us to make that change ;-) | 14:53 |
| *** mugsie has joined #zuul | 14:53 | |
| *** mugsie has quit IRC | 14:53 | |
| *** mugsie has joined #zuul | 14:53 | |
| electrofelix | fungi corvus: sounds like we're probably better off living with the issue until we migrate | 14:53 |
| electrofelix | thanks | 14:53 |
| fungi | electrofelix: yeah, we managed to deal with it for years at the change volume sustained by the openstack project, so it's not impossible | 14:58 |
| *** ssbarnea_ has joined #zuul | 14:59 | |
| fungi | electrofelix: basically i got to the point where if zuul was inexplicably not enqueuing a change that looked like it should be, first thing i'd do is grep the logs for that exception | 14:59 |
| fungi | because more often than not that was the problem | 14:59 |
| *** hashar is now known as hasharAway | 15:16 | |
| *** myoung is now known as myoung|brb | 15:44 | |
| *** jpena is now known as jpena|brb | 15:47 | |
| *** mgagne_ is now known as mgagne | 15:50 | |
| *** myoung|brb is now known as myoung | 15:50 | |
| corvus | fungi, clarkb, mordred, pabelanger: the foundation marketing folks suggested putting a video of the zuul simulation bit from the zuul overview talk on the website; i've made one, and it's come in at about 2MB. that's "big", but probably not too big to check into the website repo. but i wonder if we should anticipate other large media, and create a zuul-website-media git repo for large blobs. that way just | 15:54 |
| corvus | editing the content doesn't require you to pull down a bunch of "big" files? | 15:54 |
| *** dkranz has quit IRC | 15:56 | |
| *** dkranz has joined #zuul | 15:57 | |
| pabelanger | that sounds great to me | 15:57 |
| fungi | yeah, i like that idea | 15:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Don't use pip internals https://review.openstack.org/561659 | 16:04 |
| clarkb | corvus: https://bugs.chromium.org/p/gerrit/issues/detail?id=3829 may be of use? | 16:05 |
| corvus | remote: https://review.openstack.org/561660 Add zuul-website-media project | 16:06 |
| corvus | clarkb: oh, do you think we need to do that? i'm assuming we can handle several mb files in git without special support, just that it's inconvenient to clone repos that have a bunch. | 16:08 |
| clarkb | I don't think we need to do that, just pointing it out as an option | 16:09 |
| corvus | clarkb: i guess if we did that, we still need a storage space? | 16:09 |
| clarkb | I'm also not sure how git replication would work with that | 16:09 |
| clarkb | corvus: ya | 16:09 |
| *** jpena|brb is now known as jpena|off | 16:32 | |
| *** elyezer_ has quit IRC | 16:38 | |
| *** jpena|off is now known as jpena | 16:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Test base job secrets https://review.openstack.org/561030 | 16:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 16:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 16:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 16:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 16:39 |
| *** elyezer_ has joined #zuul | 16:40 | |
| pabelanger | does zuul still support github without a GitHub application? IIRC, there was the ability to configure each project manually to use webhooks right? | 16:52 |
| rcarrillocruz | think so, but GH app is the recommend way of doing so | 16:56 |
| rcarrillocruz | iirc SpamapS is using hooks ? | 16:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove extra words from gerrit setup docs https://review.openstack.org/561672 | 16:57 |
| pabelanger | rcarrillocruz: yah, right now I don't want to setup an org, can't see a way of using them from a personal repo | 16:58 |
| pabelanger | but testing the webhooks directly on a repo | 16:58 |
| rcarrillocruz | yup, i had to create rcarrillocruz-org for that purpose | 16:58 |
| rcarrillocruz | for tinkering with it | 16:58 |
| corvus | pabelanger: you should be able to create an app under your account | 16:59 |
| corvus | pabelanger: https://developer.github.com/apps/building-github-apps/creating-a-github-app/ | 17:00 |
| corvus | "You can create and register a GitHub App under your personal account or under any organization you have administrative access to." | 17:00 |
| mrhillsman | i really wanted to try to fix but figured you all would be quicker | 17:00 |
| mrhillsman | it is not possible right now to encrypt secrets unless i am just totally wrong and that is reasonable since i am not a developer by nature | 17:00 |
| pabelanger | corvus: Oh, cool! Thanks | 17:01 |
| mrhillsman | in rpclistener.py the last function handle_get_key is broken | 17:01 |
| mrhillsman | again, could just be me, but this never works - (trusted, project) = tenant.getProject(args.get("project")) - as tenant.getProject fails because tenant = self.sched.abide.tenants.get(args.get("tenant")) returns None | 17:02 |
| mrhillsman | when webapp.py was around this URL works - http://{address}:8001/tenant/keys/connection/project-org/project.pub | 17:04 |
| corvus | mrhillsman: i don't think there's anything fundamentally broken in handle_get_key -- if the tenant argument passed to it exists, it should work. if it's returning None there, it must be getting an invalid tenant name. | 17:05 |
| mrhillsman | i'm 100% sure the tenant exists | 17:05 |
| clarkb | mrhillsman: rightb ut didn't you say the tenant was failign to load the other day? | 17:05 |
| mrhillsman | and other URLs reflect that | 17:05 |
| corvus | mrhillsman: it's possible we neglected to update the encrypt_secrets.py script with the api route changes | 17:06 |
| tobiash | pabelanger: an org is basically the same as the personal space | 17:06 |
| mrhillsman | api/tenants - shows it, api/tenant/tenant/info, etc | 17:06 |
| tobiash | pabelanger: and installation of the app can be done either on an org, the personal space or invididual repos | 17:06 |
| mrhillsman | nah, i'm looking at the scheduler trying to handle the key_get job and it always complains about tenant.getProject | 17:07 |
| mrhillsman | because tenant is None | 17:07 |
| tobiash | pabelanger: oh corvus already said that | 17:07 |
| pabelanger | tobiash: yah, thanks. I'll try that in a moment | 17:07 |
| * tobiash should read complete backlog before replying | 17:07 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: getting details on pip and virtualenv https://review.openstack.org/561675 | 17:07 |
| corvus | mrhillsman: do you have multiple tenants, or are you single-tenant whitelabel (like openstack)? | 17:07 |
| mrhillsman | single tenant | 17:08 |
| mrhillsman | waiting on scheduler to restart and i will give you link | 17:08 |
| corvus | mrhillsman: i just encrypted a secret on openstack with: python tools/encrypt_secret.py https://zuul.openstack.org/api openstack-infra/zuul | 17:09 |
| corvus | (we should not need 'api/' in there; i'll fix that) | 17:09 |
| corvus | but it should at least work if you add that for now | 17:10 |
| mrhillsman | well, i could deal with the tool failing and being fixed | 17:10 |
| mrhillsman | but i should be able to see the public key in the browser | 17:10 |
| mrhillsman | unless that has been disabled | 17:10 |
| pabelanger | AttributeError: 'NoneType' object has no attribute 'public_key' | 17:10 |
| pabelanger | mrhillsman: is that what you got?^ | 17:10 |
| mrhillsman | ok | 17:11 |
| mrhillsman | http://37.187.92.93:9000/api/tenants | 17:11 |
| mrhillsman | you see the tenant | 17:11 |
| mrhillsman | http://37.187.92.93:9000/api/tenant/jitcoding/info | 17:11 |
| mrhillsman | tenant info | 17:11 |
| mrhillsman | so 100% sure it exists | 17:11 |
| mrhillsman | under webapp.py | 17:11 |
| mrhillsman | http://80.158.22.48:8001/openlab/keys/github/gophercloud/gophercloud.pub | 17:11 |
| mrhillsman | that works | 17:11 |
| mrhillsman | when i take that and adjust the URL based on the new structure | 17:12 |
| mrhillsman | zuul-web crashes | 17:12 |
| openstackgerrit | Merged openstack-infra/zuul master: Add Gerrit docs to Zuul From Scratch https://review.openstack.org/558600 | 17:12 |
| openstackgerrit | Merged openstack-infra/zuul master: Add static driver doc to Zuul From Scratch https://review.openstack.org/558802 | 17:12 |
| corvus | mrhillsman: okay, you're directly accessing zuul-web, without a proxy server, so it's effectively a multi-tenant environment as far as the urls go | 17:13 |
| mrhillsman | i map the URL so am i doing the mapping wrong? | 17:13 |
| tobiash | mrhillsman: how does your mapping look like? | 17:13 |
| mrhillsman | http://37.187.92.93:9000/api/tenant/jitcoding/keys/github/jitcoding/common-jobs.pub | 17:13 |
| mrhillsman | if you click, zuul-web crashes ;) | 17:13 |
| tobiash | ok, I got once 404 and now nothing ;) | 17:14 |
| mrhillsman | https://www.irccloud.com/pastebin/3GNtXKEd/ | 17:14 |
| pabelanger | mrhillsman: I can reproduce that atrributeerror | 17:15 |
| pabelanger | if I use the wrong path info | 17:15 |
| corvus | mrhillsman: '/api/tenant/{tenant}/key/{project:.*}.pub' is the api url | 17:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: getting details on pip and virtualenv https://review.openstack.org/561675 | 17:15 |
| corvus | mrhillsman: so it should be: http://37.187.92.93:9000/api/tenant/jitcoding/key/jitcoding/common-jobs.pub | 17:15 |
| pabelanger | but I get {"error_description": "Internal error"} not a crash | 17:15 |
| corvus | mrhillsman: note "key" instead of "keys" and there's no more connection name | 17:16 |
| mrhillsman | ok will try that, but i think none of the variations worked | 17:16 |
| clarkb | fwiw if it is properly hard crashing it should probably be updated to not do that | 17:16 |
| tobiash | mrhillsman: do you have a recent zuul-web and scheduler? | 17:16 |
| mrhillsman | restarting zuul-web as it does not respond to any further requests | 17:16 |
| mrhillsman | yep | 17:16 |
| mrhillsman | 3.0.1 | 17:16 |
| mrhillsman | ok it is working -_- | 17:17 |
| *** electrofelix has quit IRC | 17:18 | |
| mrhillsman | api/tenant/{tenant}/key/{project:.*}.pub is confusing for a novice like myself | 17:19 |
| mrhillsman | i'll create a doc patch | 17:19 |
| *** jpena is now known as jpena|off | 17:19 | |
| mrhillsman | in particular {project:.*} | 17:19 |
| mrhillsman | i thought it was project:{stuff_here} or just project name, and a number of other things probably | 17:20 |
| pabelanger | speaking of keys, I wonder in the case of 3pci or using git driver for git.zuul-ci.org zuul-base-jobs, if we really need zuul to create keys for those projects, since they are not under the control of the local zuul | 17:21 |
| mrhillsman | and knowing from previous issue with github payload that api was added figured only needed to add that for all the other things | 17:21 |
| pabelanger | I noticed they were generated when I first created my main.yaml | 17:21 |
| mrhillsman | it does right for key encryption? i thought that was the point/purpose, how else would it be done securely | 17:22 |
| mrhillsman | this has been painful for the last couple of days hehe | 17:22 |
| mrhillsman | at least i was able to show the zuul-web crashing even though i have not idea how to fix :) | 17:24 |
| mrhillsman | corvus are you doing the patch for the encryption tool? if not i can | 17:24 |
| corvus | mrhillsman: i am | 17:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: getting details on pip and virtualenv https://review.openstack.org/561675 | 17:25 |
| mrhillsman | ok cool | 17:25 |
| tobiash | mrhillsman: what's still strange is the zuul-web crash | 17:25 |
| tobiash | I tried that in my environment and couldn't get it to crash | 17:25 |
| pabelanger | same | 17:26 |
| mrhillsman | hrmm | 17:26 |
| fungi | a traceback from the zuul-web log would be most helpful | 17:26 |
| tobiash | mrhillsman: you said you have some mapping in front of zuul-web, can you show how that looks like? | 17:26 |
| mrhillsman | zuul-web does not error which is quite strange but no longer serves any requests | 17:26 |
| fungi | assuming it manages to write to its log before dying in flames | 17:26 |
| tobiash | maybe that has somthing to do with it | 17:26 |
| pabelanger | I just rebuilt with 3.0.1 to confirm also | 17:26 |
| mrhillsman | what is also weird is it ain't doing it now | 17:27 |
| mrhillsman | lol | 17:27 |
| mrhillsman | but y'all saw it | 17:27 |
| tobiash | so a heisenbug? | 17:27 |
| mrhillsman | at least i think someone did, they said 404, then stuck loading | 17:27 |
| mrhillsman | now when you click - http://37.187.92.93:9000/api/tenant/jitcoding/keys/github/jitcoding/common-jobs.pub | 17:28 |
| mrhillsman | 404s every time | 17:28 |
| tobiash | mrhillsman: your paste (https://www.irccloud.com/pastebin/3GNtXKEd/) was from the scheduler? | 17:28 |
| mrhillsman | yeah | 17:28 |
| tobiash | mrhillsman: do you have debug logging enabled in zuul-web | 17:29 |
| tobiash | ? | 17:29 |
| Shrews | zuul-web stopping could very well be related to the non-async async nature | 17:29 |
| mrhillsman | there we go | 17:29 |
| mrhillsman | http://37.187.92.93:9000/api/tenant/jitcoding/key/github/jitcoding/common-jobs.pub | 17:29 |
| mrhillsman | that caused it | 17:29 |
| mrhillsman | with the scheduler saying the same as you pasted | 17:29 |
| mrhillsman | zuul-web - 2018-04-16 17:29:15,001 DEBUG zuul.RPCClient: Waiting for job completion | 17:30 |
| Shrews | yep. that's what i'm trying to correct now | 17:30 |
| mrhillsman | ++ | 17:30 |
| corvus | why isn't the job completing? | 17:30 |
| tobiash | mrhillsman: so it looks like the scheduler doesn't answer and zuul-web blocks waiting for the response | 17:30 |
| mrhillsman | ++ | 17:31 |
| mrhillsman | but that is even weird because | 17:31 |
| mrhillsman | 2018-04-16 17:29:15,007 INFO gear.Connection.b'Zuul RPC Listener': Sending packet to <gear.Connection 0x7f3ae6bfcd30 host: 10.15.211.180 port: 4730>: <gear.Packet 0x7f3ae4020940 type: WORK_EXCEPTION handle: b'H:gearman00:299'> | 17:31 |
| mrhillsman | so gearman not responding to zuul-web? | 17:32 |
| corvus | we should fix zuul-web to not block, but the more worrying thing is why it isn't completing. if we don't fix *that*, even if we do fix the async stuff, well end up with infinitely growing pending jobs (and stress on the asyncio executor thread pool) | 17:32 |
| tobiash | the scheduler should send work exception http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/rpclistener.py#n112 | 17:33 |
| mrhillsman | it is isn't it? 2018-04-16 17:29:15,007 INFO gear.Connection.b'Zuul RPC Listener': Sending packet to <gear.Connection 0x7f3ae6bfcd30 host: 10.15.211.180 port: 4730>: <gear.Packet 0x7f3ae4020940 type: WORK_EXCEPTION handle: b'H:gearman00:299'> | 17:34 |
| tobiash | mrhillsman: are you running the integrated geard or a separate gearman server? | 17:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Don't use pip internals https://review.openstack.org/561659 | 17:34 |
| mrhillsman | separate gearman | 17:34 |
| tobiash | which gearman? | 17:34 |
| mrhillsman | gearman-job-server installed via packaged ubuntu16.04.4 | 17:34 |
| mrhillsman | s/packaged/packaging | 17:35 |
| tobiash | mrhillsman: that could be the problem | 17:35 |
| tobiash | back in zuulv2 I had problems getting zuul to work with that | 17:35 |
| mrhillsman | should i install from upstream and try again? | 17:35 |
| tobiash | that might not support the WORK_EXCEPTION | 17:35 |
| corvus | or maybe it doesn't pass work_exception to the client. | 17:35 |
| corvus | the gear protocol has traditionally been under-specified. | 17:36 |
| tobiash | probably | 17:36 |
| mrhillsman | https://github.com/gearman/gearmand/issues/70 | 17:36 |
| clarkb | I think Shrews' change to stick it on the asyncio threadpool with a timeout will at least prevent it from getting worse over time | 17:37 |
| clarkb | unless DoS'd in a short period of time | 17:37 |
| mrhillsman | https://bugs.launchpad.net/gearmand/+bug/405732/comments/1 | 17:38 |
| openstack | Launchpad bug 405732 in Gearman "WORK_EXCEPTION never forwarded to client" [Undecided,Invalid] - Assigned to Eric Day (eday) | 17:38 |
| tobiash | mrhillsman: back in v2 times I ran gearmand from fedora | 17:38 |
| corvus | clarkb: i disagree, it trades one problem for another | 17:38 |
| mrhillsman | consider using a combination of WORK_WARNING and WORK_FAIL messages | 17:38 |
| tobiash | but now in v3 I just run it within the scheduler | 17:38 |
| corvus | clarkb: see what i wrote above about zuul-web accumulating stuck jobs indefinitely | 17:38 |
| Shrews | eday! wow, almost forgot about him | 17:38 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: trying to upgrade pip before running tox https://review.openstack.org/561675 | 17:38 |
| tobiash | as this is afaik the only way to do proper encryption with client cert auth | 17:38 |
| clarkb | corvus: they will timeout though right? | 17:38 |
| corvus | clarkb: no | 17:39 |
| mrhillsman | so maybe i should install from upstream rather than packaging | 17:39 |
| clarkb | corvus: https://review.openstack.org/#/c/560044/5/zuul/driver/github/githubconnection.py they will if that change goes in? I had thought that had merged for some reason | 17:39 |
| tobiash | mrhillsman: so you probably want to enable the gearman server in the zuul-scheduler | 17:39 |
| tobiash | or run the python geard | 17:40 |
| corvus | mrhillsman: i recommend using zuul's internal gearman server for now. i think supporting something else is a moderate development effort -- not a quick fix. | 17:40 |
| corvus | clarkb: the code that calls has no provision for timing out. what happens in that case? does the asyncio executor kill the thread? i thought that was not recommended in python. i assume it just returns control to the greenthread without doing anything to the underlying pthread. and those accumulate forever? | 17:42 |
| mrhillsman | ok | 17:42 |
| corvus | clarkb, Shrews: everytime i look at that, i think we need a proper gearman client for zuul-web, just like zuul-executor. | 17:43 |
| corvus | clarkb, Shrews: i'm becoming more convinced of that | 17:43 |
| Shrews | corvus: that's what you suggested last week, right? IIRC | 17:44 |
| * Shrews digging back into that now that I have electricity again | 17:44 | |
| corvus | Shrews: yeah. but as a 'improvement'. i'm thinking now it may be more important. | 17:44 |
| clarkb | "Returns result of the Future or coroutine. When a timeout occurs, it cancels the task and raises asyncio.TimeoutError. To avoid the task cancellation, wrap it in shield()." | 17:44 |
| clarkb | I read that as the thread pool thread will be freed and the code that is blocking will be stopped | 17:45 |
| clarkb | but likely need to test it or read the source to be sure | 17:45 |
| corvus | clarkb: i don't read that as necessarily implying it does anything with the underlying thread. it could just stop waiting for it. but i also agree it doesn't discount that. :) | 17:46 |
| clarkb | https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel talks about the implementation | 17:46 |
| clarkb | it causes an exception to be raised | 17:46 |
| clarkb | if they are mangling python in such a way that a blocking io call can raise that exception I think we are ok there | 17:47 |
| clarkb | if not then ya it would likely tie up threads | 17:47 |
| corvus | clarkb: okay, that's reasonable and could mean we don't end up with a thread sitting there forever | 17:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update encrypt_secret to be API aware https://review.openstack.org/561681 | 17:49 |
| corvus | mordred: ^ can you take a look at that change | 17:49 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: trying to upgrade pip before running tox https://review.openstack.org/561675 | 17:50 |
| tobiash | corvus: tested 561681 against my multitenant environment | 17:55 |
| tobiash | and works | 17:55 |
| corvus | tobiash: thanks! i tested it against openstack | 17:56 |
| tobiash | on friday we thought we should make the quite the same change | 17:57 |
| tobiash | but you were faster ;) | 17:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: DNM: trying to upgrade pip before running tox https://review.openstack.org/561675 | 17:59 |
| corvus | tobiash: oh, sorry, i was deep into making a video on friday and may have missed some chat | 18:02 |
| pabelanger | so, looking to get some help on github connection driver, I believe I have it setup properly, as I can see the following: | 18:03 |
| pabelanger | 2018-04-16 18:01:35,288 INFO zuul.GithubConnection: Starting GitHub connection: github.com | 18:03 |
| pabelanger | and I see zuul-merger clone git repos from my github.com namespace | 18:03 |
| clarkb | corvus: tobiash: thinking out loud here, would it make sense to have white labels pass through the nroaml tenant'd urls too? then tooling like this could be made simpler and just always provide a tenant? | 18:03 |
| pabelanger | however, http://162.253.55.78:9000/api/connection/github.com/payload is 404 | 18:03 |
| clarkb | I guess humans won't necessarily know what the tenant is in that case | 18:03 |
| pabelanger | and my github app isn't green | 18:03 |
| pabelanger | I see the following in the github ui: 401: X-Hub-Signature header missing. | 18:05 |
| clarkb | corvus: flake8 failed on line length | 18:06 |
| corvus | pabelanger: did you create a webhook secret? | 18:07 |
| pabelanger | corvus: yes | 18:07 |
| tobiash | pabelanger: do you test with POST? | 18:07 |
| pabelanger | corvus: oh wait, it is now empty | 18:07 |
| openstackgerrit | Merged openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 18:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update encrypt_secret to be API aware https://review.openstack.org/561681 | 18:08 |
| pabelanger | corvus: okay, thank you. That was it | 18:08 |
| pabelanger | I some how cleared it | 18:09 |
| tobiash | with something like https://github.com/travist/jsencrypt we might be able to add secret encryption as a feature to zuul-web at some point in time | 18:10 |
| tobiash | I think that would improve usability especially for unexperienced users | 18:11 |
| corvus | ++ | 18:14 |
| tobiash | today I split out base job and moved log upload into its own protected base-logs job in front of base | 18:33 |
| tobiash | that made log upload more resilient to retry-limit in our deployment | 18:34 |
| *** harlowja has joined #zuul | 18:40 | |
| mrhillsman | so the version of gearman by ubuntu source is 1.0.6 | 18:45 |
| mrhillsman | latest is 1.1.18 | 18:45 |
| mrhillsman | with the latest version | 18:45 |
| mrhillsman | { | 18:45 |
| mrhillsman | error_description: "Internal error" | 18:45 |
| mrhillsman | } | 18:45 |
| mrhillsman | and no zuul-web crashing | 18:46 |
| mrhillsman | sorry, by ubuntu packaging 16.04 is 1.0.6 | 18:46 |
| mrhillsman | 18.04 will install 1.1.18 | 18:46 |
| mrhillsman | i was able to compile/install on 16.04 and got the internal error for work_exception and zuul-web still running | 18:47 |
| pabelanger | yah, that is what I see in local setup | 18:47 |
| pabelanger | using latest gear | 18:47 |
| mrhillsman | and zuul-web spits out a trace rather than hanging | 18:48 |
| mrhillsman | https://www.irccloud.com/pastebin/LiEhxzTW/ | 18:48 |
| mrhillsman | guess i said all that to say on 18.04 using the packaged version of gearman-job-server will be fine as it relates to WORK_EXCEPTION handling | 18:50 |
| *** jimi|ansible has joined #zuul | 18:50 | |
| *** jimi|ansible has joined #zuul | 18:50 | |
| clarkb | mrhillsman: out of curiousity was there a specific reason to run zuul with external C gearman server? | 19:04 |
| clarkb | (I don't think we tell people to run zuul that way) | 19:04 |
| *** openstackgerrit has quit IRC | 19:05 | |
| mrhillsman | yes, docs say internal is recommended, but i personally work to keep things separate from an operational experience perspective | 19:06 |
| clarkb | mrhillsman: in that case you can run geard separately | 19:06 |
| mrhillsman | not my own experience as being better/more than another | 19:06 |
| mrhillsman | i was not aware of geard | 19:07 |
| mrhillsman | docs just say i needed gearman so me not having any idea what gearman is/was i searched and gearman.org is where i landed with instructions to apt install gearman... | 19:08 |
| pabelanger | geard works well, what I've been tested with recently | 19:20 |
| mrhillsman | pabelanger where do you get it from? | 19:25 |
| clarkb | mrhillsman: its part of the gear pypi package which installing zuul from source will install for you | 19:25 |
| mrhillsman | ah ok | 19:25 |
| mrhillsman | i was searching and not finding | 19:26 |
| Shrews | ok, i'm not sure i can fix this zuul-web async problem | 19:26 |
| tobiash | pabelanger: does it also work well with encryption? | 19:27 |
| tobiash | pabelanger: having it separated from the scheduler process would be better for me from operational point of view | 19:28 |
| tobiash | Sometimes zuul-web hangs when the scheduler (and with it geard) is restarted | 19:29 |
| Shrews | corvus: i think we have a greater fundamental issue with the gearman handling in zuul-web. It's fundamentally designed to do request->response, so it can't handle more than 1 request at a time. The log streamer solves this by creating a new response object for each request. | 19:30 |
| Shrews | corvus: So I don't know how to add the async gearman calls without first solving that issue | 19:31 |
| Shrews | tobiash: did you add the gearman code to zuul-web by chance? | 19:31 |
| tobiash | Shrews: I don't know anymore, I think most of that work was done by tristanC | 19:32 |
| pabelanger | tobiash: I haven't tested, but should know more in the next day. I am running it with SSL | 19:33 |
| Shrews | I think I'm going to have to enlist the original author with a "PLZ HALP" | 19:33 |
| Shrews | but i'm also hoping my understanding of the code is incorrect too | 19:34 |
| mrhillsman | pabelanger: geard you cannot set the host though right? | 19:35 |
| mrhillsman | i see port option, host is 127.0.0.1 | 19:35 |
| pabelanger | mrhillsman: yah, i think it binds to 0.0.0.0 by default, i've been meaning to push up a change to support a config file and allow for more specific binding | 19:37 |
| corvus | pabelanger: maybe start with a command line option? :) | 19:37 |
| corvus | Shrews: i'm not sure i follow; can you point me at the log streaming code you're referencing? | 19:38 |
| pabelanger | corvus: ack | 19:38 |
| mrhillsman | ah i see | 19:38 |
| mrhillsman | yeah, command line option is needed | 19:38 |
| corvus | tobiash, pabelanger: maybe zuul-web hangs because of the async issue Shrews is working on. let's fix that. | 19:39 |
| Shrews | corvus: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/__init__.py#n127 | 19:39 |
| mrhillsman | it uses systemd to run gearmand :) | 19:39 |
| mrhillsman | i was looking in the code to find where it is hardcoding 127.0.0.1 but maybe it is my server | 19:39 |
| tobiash | corvus: yes that might be it | 19:39 |
| corvus | tobiash, pabelanger: (externalizing the server won't fix the problem, it will just alter when it happens to when the gearman server is restarted) | 19:40 |
| pabelanger | I actually haven't seen an issue myself locally | 19:41 |
| Shrews | corvus: that code prepares a new response, fulfilling it as it can, allowing other responses to be handled similarly. Looking at the GearmanHandler code, I see no similar mechanism. Eg., http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/__init__.py#n293 | 19:41 |
| tobiash | Shrews, corvus: the scheduler makes that more asynchronous and event based right? | 19:41 |
| pabelanger | but also haven't been testing zuul-web much yet | 19:41 |
| tobiash | Maybe that model also fits for zuul-web | 19:41 |
| Shrews | corvus: GearmanHandler.processRequest() just returns the web response. So I think that means we are locked into waiting on the response for each request, right? | 19:43 |
| corvus | Shrews: yes, but if we 'await' then we service other requents in the interim | 19:44 |
| Shrews | corvus: but how are the responses to the requests kept synchronized? | 19:44 |
| clarkb | Shrews: aiohttp keeps each session straight | 19:44 |
| corvus | yeah, there's still a unique call stack / context for each request. | 19:45 |
| Shrews | clarkb: oh, is that where the magic is hidden | 19:45 |
| Shrews | i suppose that makes sense | 19:45 |
| Shrews | i'm no web dev | 19:46 |
| corvus | it's just like threads, but you have to do all the work yourself instead of having the benefit of 30 years of computer science...nevermind. | 19:46 |
| corvus | Shrews: so key_get should be able to, basically, run submitJob in an executor, then await job completion, then return the result | 19:47 |
| Shrews | corvus: "await job completion" == handleWorkComplete() called, yeah? | 19:48 |
| corvus | Shrews: yep -- the missing piece we don't have anywhere yet is having handleWorkComplete throw something on the asyncio loop to indicate that whatever we're "await"ing in key_get is done. | 19:49 |
| corvus | Shrews: if that is too much to bite off for now, we can merge the rpcclient.submitJob change -- maybe add a TODO? | 19:52 |
| Shrews | lemme keep poking a bit | 19:53 |
| *** hasharAway has quit IRC | 20:01 | |
| *** openstackgerrit has joined #zuul | 20:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update encrypt_secret to be API aware https://review.openstack.org/561681 | 20:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 20:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 20:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 20:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 20:12 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 20:32 |
| *** ssbarnea_ has quit IRC | 20:44 | |
| pabelanger | SpamapS: are you also running zuul from a virtualenv? I notice a few of your pastebin showing nodepool | 20:53 |
| pabelanger | SpamapS: if so, did you also put ansible inside a virtualenv? I am guessing you might be using symlinks so bwrap mount is able to find it? | 20:54 |
| *** harlowja has quit IRC | 20:55 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 20:58 |
| corvus | pabelanger: if you install ansible into the zuul-executor's venv, it should get it automatically | 20:59 |
| pabelanger | corvus: yes, gets pulled in via requirements, however isn't in the path right now. Currently looking to see why that is | 20:59 |
| pabelanger | corvus: I suspect it is because I don't source venv/bin/activate for the virtualenv, but directly call venv/bin/zuul-executor with service scripts | 21:01 |
| *** JasonCL has quit IRC | 21:02 | |
| corvus | pabelanger: that sounds likely | 21:04 |
| openstackgerrit | Arun S A G proposed openstack-infra/nodepool master: Do not hardcode zookeeper host to localhost https://review.openstack.org/561729 | 21:12 |
| *** JasonCL has joined #zuul | 21:13 | |
| openstackgerrit | Arun S A G proposed openstack-infra/nodepool master: Do not hardcode zookeeper host to localhost https://review.openstack.org/561729 | 21:14 |
| openstackgerrit | Merged openstack-infra/zuul master: Support regex matching of github status https://review.openstack.org/561177 | 21:20 |
| openstackgerrit | Merged openstack-infra/zuul master: docs: add build status documentation https://review.openstack.org/561489 | 21:28 |
| *** AJaeger has quit IRC | 22:10 | |
| *** AJaeger has joined #zuul | 22:22 | |
| *** jimi|ansible has quit IRC | 22:25 | |
| *** smyers has quit IRC | 22:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update encrypt_secret to be API aware https://review.openstack.org/561681 | 22:27 |
| *** smyers has joined #zuul | 22:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add license https://review.openstack.org/561747 | 22:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Add license and zuul.yaml https://review.openstack.org/561747 | 22:46 |
| *** JasonCL has quit IRC | 22:50 | |
| *** JasonCL has joined #zuul | 22:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 22:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Merge zuul-website-media when publishing site https://review.openstack.org/561749 | 22:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website-media master: Run zuul-website jobs https://review.openstack.org/561750 | 22:56 |
| clarkb | I'm going to mention this in this channel too https://review.openstack.org/#/c/561659/2 is an attempt at fixing tox-siblings in zuul-jobs to work with pip10 | 23:16 |
| clarkb | we don't currently see this as a problem on openstack infra because our images still have global pip 9 which is what the jobs are using even if tox installs pip10 | 23:16 |
| clarkb | so this both fixes needing an external pip to do tox-siblings and addresses the lack of public api in the pip10 code for this | 23:16 |
| SpamapS | pabelanger: yes, and yes | 23:29 |
| *** dkranz has quit IRC | 23:38 | |
| *** rlandy has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!