| mrhillsman | thx | 00:04 |
|---|---|---|
| mrhillsman | maybe i am overcomplicating things | 00:04 |
| *** JasonCL has quit IRC | 00:04 | |
| mrhillsman | because i can see pipelines listed on the status page getting out of hand | 00:04 |
| *** JasonCL has joined #zuul | 00:05 | |
| EmilienM | why zuul is always restarted when code is updated? | 00:05 |
| clarkb | EmilienM: the same reason you restart nova when code is updated | 00:05 |
| clarkb | EmilienM: you need to run the new executables | 00:05 |
| EmilienM | clarkb: mutable config? | 00:05 |
| clarkb | EmilienM: its not config, its the actual code | 00:05 |
| EmilienM | ok | 00:06 |
| EmilienM | right | 00:06 |
| clarkb | the config is dynamically updated all the time | 00:06 |
| EmilienM | that's good | 00:06 |
| clarkb | every time .zuul.yaml change merges for example | 00:06 |
| EmilienM | I'm wondering if containerizing zuul would help | 00:07 |
| clarkb | I think the only config you may need to restart for currently is the ini config which has stuff like driver connection details in it | 00:07 |
| clarkb | EmilienM: you'd still have to restart it to pick up new code | 00:07 |
| clarkb | (same as nova) | 00:07 |
| pabelanger | i think we could CD nodepool-launcher today, right? Since all that data is stored in zookeeper | 00:07 |
| clarkb | pabelanger: we do CD nodepool launcher today | 00:07 |
| pabelanger | we do? | 00:08 |
| clarkb | yes we continuously deploy it | 00:08 |
| clarkb | straight from master | 00:08 |
| pabelanger | right :) What is the restart part called | 00:08 |
| *** harlowja has quit IRC | 00:08 | |
| clarkb | thats part of it | 00:08 |
| clarkb | it just happens to be the manual step in the process | 00:08 |
| pabelanger | deploy / restart on each commit. I think we could write a pipeline in zuul to do that, without a downtime on nodepool-launcher | 00:09 |
| pabelanger | IIUC | 00:09 |
| clarkb | needing to restart processes to pick up new code is somewhat orthogonal to CD and containers | 00:09 |
| clarkb | unless you have some way to transition from one process into a new fork (which is doable in some cases see weechat for example) you have to do that | 00:10 |
| clarkb | pabelanger: its like a one line chagne in existing puppet if you want to do that | 00:10 |
| clarkb | subscribe => Exec[nodepool-install] in the service | 00:10 |
| pabelanger | clarkb: yah, agree. But would be cool to drive it directly from zuul.o.o :) | 00:11 |
| clarkb | EmilienM: the major issue is the zuul scheduler is still a spof. So you can't restart it without a downtime | 00:11 |
| EmilienM | I see | 00:11 |
| clarkb | executors can be restarted without downtime, nodepool launchers without downtime and nodepool builders. Zuul web too I think if you ran them behind a load balacner | 00:12 |
| pabelanger | EmilienM: FWIW: tobiash is deploying zuul / nodepool in containers on openshift | 00:12 |
| clarkb | but if you have to retart the scheduler its a downtime | 00:12 |
| pabelanger | yah | 00:12 |
| pabelanger | I think nodepool-builder needs a little more work to clean up diskimage-builder on shutdown, but agree no downtime for it | 00:13 |
| clarkb | the fingergateway holds open long term connections but I think it too could be load balacned | 00:13 |
| clarkb | its just users would noticed when it flapped but immediately be able to reconnect | 00:14 |
| EmilienM | pabelanger: sweet | 00:14 |
| *** JasonCL has quit IRC | 00:29 | |
| *** JasonCL has joined #zuul | 00:31 | |
| tristanC | is there a role/task to accept inbound port 19885 (zuul-console) for slave running firewalld? | 00:35 |
| clarkb | tristanC: looks like we bake that into our images | 00:36 |
| clarkb | quick grepping doesn't show it in the jobs | 00:36 |
| tristanC | or maybe we should just wait for the zuul_stream refactor where the console stream goes through ssh tunnel | 00:38 |
| tristanC | otherwise, this deserve a bit more of documentation | 00:38 |
| *** dtruong_ has joined #zuul | 00:42 | |
| *** nguyenhai_ has joined #zuul | 00:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 00:44 |
| pabelanger | yah, we back it into a DIB element today | 00:44 |
| *** dtruong has quit IRC | 00:46 | |
| *** nguyenhai has quit IRC | 00:46 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 00:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-review job https://review.openstack.org/535223 | 00:49 |
| *** odyssey4me has quit IRC | 00:51 | |
| *** odyssey4me has joined #zuul | 00:51 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-upload-to-galaxy job https://review.openstack.org/532084 | 00:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-spec job https://review.openstack.org/532085 | 00:55 |
| tristanC | thanks for the information, that's good to know | 00:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 01:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: model: add empty branch attribute to the Tag object https://review.openstack.org/560700 | 01:10 |
| *** JasonCL has quit IRC | 01:14 | |
| *** gouthamr has joined #zuul | 01:46 | |
| *** gouthamr has quit IRC | 01:49 | |
| *** gouthamr has joined #zuul | 01:50 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: launcher: handle ZK session loss during handler poll https://review.openstack.org/556335 | 01:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 01:57 |
| *** kmalloc has quit IRC | 02:06 | |
| pabelanger | clarkb: clarkb: tobiash: replied to 3.0.1 ML post about maybe also removing zuul-cloner, since it is broken and no longer needed. | 02:17 |
| *** sshnaidm has quit IRC | 02:26 | |
| *** gouthamr has quit IRC | 02:33 | |
| *** gouthamr has joined #zuul | 02:35 | |
| *** gouthamr_ has joined #zuul | 02:40 | |
| *** gouthamr has quit IRC | 02:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 02:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 02:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: openstack: convert rate to float https://review.openstack.org/537250 | 02:59 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add emit-job-report role https://review.openstack.org/548425 | 03:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: add variant-description https://review.openstack.org/549748 | 03:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-changes: update for the new api url https://review.openstack.org/557656 | 03:20 |
| *** jimi|ansible has quit IRC | 03:24 | |
| *** jimi|ansible has joined #zuul | 03:27 | |
| *** jimi|ansible has joined #zuul | 03:27 | |
| *** gouthamr_ is now known as gouthamr | 03:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object https://review.openstack.org/560700 | 03:39 |
| *** rlandy has quit IRC | 04:04 | |
| *** gouthamr has quit IRC | 04:10 | |
| openstackgerrit | Merged openstack-infra/zuul master: Report to all reporters even if one fails https://review.openstack.org/557859 | 04:29 |
| openstackgerrit | Merged openstack-infra/zuul master: Reorganize "Zuul From Scratch" document https://review.openstack.org/556988 | 04:29 |
| *** openstackgerrit has quit IRC | 05:48 | |
| *** openstackgerrit has joined #zuul | 06:02 | |
| openstackgerrit | Merged openstack-infra/nodepool master: openstack: convert rate to float https://review.openstack.org/537250 | 06:02 |
| *** threestrands has joined #zuul | 06:05 | |
| *** hashar has joined #zuul | 06:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix implicit role for repos named ansible https://review.openstack.org/560554 | 06:52 |
| *** electrofelix has joined #zuul | 07:24 | |
| *** xinliang has quit IRC | 07:30 | |
| *** jpena|off is now known as jpena | 07:33 | |
| *** xinliang has joined #zuul | 07:35 | |
| *** sshnaidm has joined #zuul | 07:52 | |
| *** JasonCL has joined #zuul | 09:49 | |
| *** kmalloc has joined #zuul | 10:01 | |
| *** threestrands has quit IRC | 10:24 | |
| *** odyssey4me has quit IRC | 10:55 | |
| *** odyssey4me has joined #zuul | 10:55 | |
| *** sshnaidm is now known as sshnaidm|lnch | 10:58 | |
| *** odyssey4me has quit IRC | 11:00 | |
| *** odyssey4me has joined #zuul | 11:00 | |
| *** Wei_Liu1 has joined #zuul | 11:10 | |
| *** Wei_Liu has quit IRC | 11:10 | |
| *** Wei_Liu1 is now known as Wei_Liu | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:53 | |
| *** JasonCL has quit IRC | 12:04 | |
| *** dkranz has quit IRC | 12:06 | |
| *** rlandy has joined #zuul | 12:28 | |
| *** sshnaidm|lnch is now known as sshnaidm | 12:32 | |
| *** pwhalen has quit IRC | 13:10 | |
| *** pwhalen has joined #zuul | 13:18 | |
| *** pwhalen has joined #zuul | 13:18 | |
| *** gouthamr has joined #zuul | 13:32 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| *** JasonCL has joined #zuul | 13:38 | |
| *** JasonCL_ has joined #zuul | 13:42 | |
| *** JasonCL_ has quit IRC | 13:42 | |
| *** JasonCL_ has joined #zuul | 13:43 | |
| mordred | clarkb, Shrews, tobiash: sorry, I was out yesterda - I see shade chat in the scrollback but it seems all is good and nothing needed from me on that? | 13:43 |
| *** JasonCL has quit IRC | 13:43 | |
| *** JasonCL_ has quit IRC | 13:45 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:47 | |
| Shrews | mordred: i don't think so? | 13:49 |
| *** JasonCL has joined #zuul | 13:50 | |
| mordred | Shrews: cool | 13:54 |
| mordred | Shrews: glad I could be of assistence then | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 14:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 14:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 14:05 |
| mordred | tristanC: sweet! with the multi-tenant dashboard job and the CORS patch I can see the errors you listed ... I think we're close to that patch working :) | 14:07 |
| *** mugsie has quit IRC | 14:08 | |
| fungi | clarkb: pabelanger: i don't think it's so much zuul-scheduler as a spof which leads to disruption, but rather the fact that it maintains its state in process memory which doesn't persist between restarts (granted, solving that isn't as simple as externalizing the state storage because you may want the new execution to create some different state than its predecessor so you're stuck having to define an api | 14:16 |
| fungi | contract around your state serialization) | 14:16 |
| fungi | the spof is even less of an issue if we get to the point where we have some intermediate message service buffering events while the scheduler is down so that it can request them once it's running again | 14:17 |
| clarkb | thats true, though people tend to notice via the side effects and not necessarily the state graphs themselves | 14:31 |
| clarkb | (jobs restarting and dashboard going away) | 14:31 |
| clarkb | oh and missing gerrit events | 14:33 |
| clarkb | I think once we address those whether or not the scheduler remains a spof will be less user visiible | 14:33 |
| *** JasonCL has quit IRC | 14:34 | |
| pabelanger | right, I agree with the scheduler points. My comments last night were more I think we are at a point in time where we could start doing restart of nodepool on a per commit bases, if we were comfortable with that. However, I know in the past corvus (I think) expressed an issue with doing that since non infra-root contributors could approve a commit which might break something in openstack-infra, then have no | 14:36 |
| pabelanger | ability to restore the service | 14:36 |
| clarkb | I think if we made the nodepool integration jobs voting (they aren't voting right now right?) I'd be a lot more comfortable with that | 14:37 |
| dmsimard | could the scheduler dump it's state before starting ? | 14:38 |
| pabelanger | they still are non-voting on zuul, IIRC | 14:38 |
| dmsimard | before restarting* | 14:38 |
| pabelanger | dib and glean, yes | 14:38 |
| dmsimard | like doing a restart would do the equivalent of our current manual dump/restore | 14:38 |
| pabelanger | dmsimard: in fact, I think the scheduler support that today, we just don't use it? | 14:38 |
| pabelanger | or maybe did at one point in time | 14:38 |
| corvus | it can be done. does someone want to do it? | 14:39 |
| clarkb | my concern with putting a lot of effort into that is it doesn't really address the problems users do notice with restarts | 14:40 |
| clarkb | all the jobs have to start over and we miss gerrit events and the dashboard returns an error in the interim | 14:40 |
| *** JasonCL has joined #zuul | 14:40 | |
| clarkb | it would reduce operator overhead for doing a graceful restart though | 14:40 |
| corvus | yes, i wouldn't put a lot of effort into it. but it would be a good minor improvement | 14:41 |
| pabelanger | agree | 14:41 |
| dmsimard | would just be a quality of operator life improvement | 14:42 |
| dmsimard | (there needs to be more of those) | 14:42 |
| corvus | i'm happy to discuss implementation with a volunteer | 14:42 |
| tristanC | would be nice if the scheduler would start the gerrit connection early and store events until configuration is completed | 14:46 |
| corvus | tristanC: it does | 14:46 |
| corvus | perceived downtime due to missing events should be as low as about 30 seconds (time to restart scheduler process) | 14:46 |
| corvus | (sometimes it takes a few seconds to stop cleanly) | 14:47 |
| tristanC | oh that may be correct, it's just that irc notices mention a 15 minutes downtime | 14:47 |
| *** dkranz has joined #zuul | 14:48 | |
| corvus | tristanC: if you're talking about yesterday's openstack-infra restart, that's not representative. | 14:50 |
| *** mugsie has joined #zuul | 14:54 | |
| *** mugsie has quit IRC | 14:54 | |
| *** mugsie has joined #zuul | 14:54 | |
| tristanC | mordred: yes, can't wait to rebase the rest of my dashboard changes on top of angular 5 :) | 14:54 |
| Shrews | fwiw, I would be very against auto restarting nodepool after every commit. Even making the non-voting jobs voting | 14:55 |
| *** gouthamr has quit IRC | 14:56 | |
| Shrews | I'm at a chiro appt now, but happy to discuss that more when at a computer | 14:56 |
| *** gouthamr has joined #zuul | 15:02 | |
| *** JasonCL has quit IRC | 15:03 | |
| *** mugsie has quit IRC | 15:04 | |
| *** mugsie has joined #zuul | 15:06 | |
| *** mugsie has quit IRC | 15:06 | |
| *** mugsie has joined #zuul | 15:06 | |
| *** JasonCL has joined #zuul | 15:06 | |
| *** mugsie has quit IRC | 15:08 | |
| *** mugsie has joined #zuul | 15:08 | |
| *** mugsie has quit IRC | 15:08 | |
| *** mugsie has joined #zuul | 15:08 | |
| pabelanger | Shrews: any specific reason why? | 15:14 |
| mordred | tristanC: ++ | 15:27 |
| pabelanger | toabctl: I don't think our new quota handing logic in nodepool applies to volumes right now, does that sound right? | 15:27 |
| pabelanger | tobiash: ^ | 15:28 |
| pabelanger | toabctl: sorry | 15:28 |
| toabctl | np :) | 15:28 |
| mordred | pabelanger: I believe you are correct- we're only doing nova quota calcs atm | 15:29 |
| pabelanger | mordred: k, I'll see if I can update it, noticed we were still trying to launch nodes in vexxhost when we hit quota on volumes | 15:29 |
| pabelanger | mordred: and on that note, might you have an idea how we could track leaked volumes in nodepool / shade? Seeing that scenario in vexxhost currently. mnaser is working on the openstack side to see why that might be | 15:30 |
| mordred | pabelanger: we'll need to add a get_block_storage_limits to shade - but that should be easy enough | 15:31 |
| mordred | pabelanger: as for the otherthing, nothing springs to mind if we mark them as terminate-on-delete when we do boot-from-volume - but I'll poke for a sec | 15:31 |
| clarkb | pabelanger: yes that was mentioned yesterday by tobiash as something that would need improving to support volumes better | 15:32 |
| mordred | pabelanger: oh - if the clouds are new enough, I think we could add a tag to the volume via the block_device_mapping_v2 structure | 15:33 |
| pabelanger | mordred: k, lets see what mnaser says :) | 15:33 |
| clarkb | does nova add any metdata by default in that case (possibly something we could use to support older clouds too) | 15:34 |
| mordred | clarkb: I'm not sure- that would be ideal | 15:36 |
| clarkb | ideally "this is the disk for instance with uuid foo" | 15:37 |
| clarkb | then if nodepool doesn't see a uuid foo we can delete it | 15:37 |
| clarkb | or similar | 15:37 |
| *** JasonCL has quit IRC | 15:38 | |
| *** JasonCL has joined #zuul | 15:39 | |
| clarkb | pabelanger: is there a change to remove zuul-cloner yet? | 15:43 |
| clarkb | pabelanger: if not is that somethign I should just go ahead and push up? | 15:43 |
| AJaeger | clarkb: there's one up - but you break everyobdoy if you do | 15:44 |
| pabelanger | clarkb: not yet, wanted to make sure everybody was okay first | 15:44 |
| pabelanger | but will do now | 15:44 |
| AJaeger | clarkb: https://review.openstack.org/513506 | 15:44 |
| clarkb | AJaeger: I don't think it should break anyone | 15:44 |
| * clarkb looks | 15:44 | |
| clarkb | AJaeger: oh sorry, I meant remove zuul-cloner from zuul itself | 15:45 |
| AJaeger | clarkb: sorry, in a call and didn't follow discussion. But if we remove zuul-cloner from our images, we have breakage | 15:45 |
| clarkb | AJaeger: the change you link likely would be disruptive | 15:45 |
| clarkb | AJaeger: ya different items, but good to keep that in mind | 15:45 |
| AJaeger | clarkb: ok. | 15:45 |
| *** JasonCL has quit IRC | 15:45 | |
| *** JasonCL has joined #zuul | 15:49 | |
| pabelanger | yah, we still want to do that, but protected with zuul-cloner role in zuul-jobs. We've actually already removed zuul-cloner from DIB images | 15:53 |
| tobiash | pabelanger: right, currently there is no quota handling of volumes in nodepool | 15:54 |
| tobiash | Which should be added also now due to boot from volume support | 15:55 |
| tobiash | But it should be easy to add that | 15:57 |
| tobiash | The quota handling code is pretty generic | 15:58 |
| corvus | pabelanger, clarkb, tobiash: i'd like to tag 301, zuul-cloner notwithstanding (i think we should remove it, but i don't think we need to block on it) | 15:59 |
| clarkb | corvus: ya I don't think we need to block on it either | 15:59 |
| clarkb | new users won't really know that it is a thing to worry about so shouldn't affect them (it doesn't work anyways) | 15:59 |
| pabelanger | clarkb: corvus: wfm, I can push it up shortly. debuging nodepool.o.o issue right now | 16:00 |
| tobiash | corvus: fine for me | 16:04 |
| corvus | weird, when i run 'zuul --version' locally on master i get 3.0.1.dev27 | 16:05 |
| corvus | but zuul.o.o says dev51 | 16:05 |
| corvus | oh, maybe that's just from when the tox env was created | 16:05 |
| corvus | maybe that doesn't get updated | 16:05 |
| corvus | yep | 16:06 |
| tobiash | corvus: I guess 560554 is for the next release then | 16:07 |
| corvus | tobiash: yep | 16:07 |
| tobiash | I think it should go latest in the next release as without it you cannot run jobs defined in a repo named ansible | 16:09 |
| pabelanger | which section of reno would removal of zuul-cloner go under? | 16:09 |
| tobiash | I have it in my staging branch so this is not an issue for me | 16:10 |
| tobiash | pabelanger: upgrade notice maybe? | 16:10 |
| corvus | pabelanger: i'm not sure it needs one? | 16:10 |
| pabelanger | tobiash: yah, that or other is what I am thinking | 16:10 |
| pabelanger | corvus: okay, that works too | 16:10 |
| corvus | (i can't think of how we'd expect someone to alter their behavior between 3.0.0 and 3.0.2 based on this) | 16:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove zuul-cloner command https://review.openstack.org/560958 | 16:11 |
| clarkb | corvus: ++ it never worked on 3.x if you want a working version you are already explicitly using v2 | 16:12 |
| corvus | 11a80ccebd5108bea1724e688c36ae281bc220b4 is what openstack-infra is running; i'll tag that as 3.0.1 | 16:12 |
| pabelanger | +1 | 16:13 |
| tobiash | ++ | 16:13 |
| corvus | pushed | 16:13 |
| *** JasonCL has quit IRC | 16:19 | |
| corvus | oh, i bet we don't rebuild the docs on release tags | 16:22 |
| corvus | that means we're going to need to land a change before we see the release notes updated | 16:22 |
| *** JasonCL has joined #zuul | 16:27 | |
| fungi | we could add a release pipeline docs job? | 16:27 |
| fungi | (for future sanity) | 16:27 |
| corvus | ++ | 16:27 |
| corvus | i'll get on that | 16:27 |
| fungi | presumably reenqueuing the master branch tip into the post pipeline after the tag is pushed will also get us that in the interim? | 16:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Publish docs in release pipeline https://review.openstack.org/560964 | 16:29 |
| corvus | fungi: you're right. clever. :) | 16:29 |
| corvus | but i went ahead and approved 560958, so there'll be a change landing shortly | 16:30 |
| tobiash | corvus: added a question on 560964 | 16:34 |
| *** gouthamr has quit IRC | 16:38 | |
| corvus | tobiash: mordred and i answered. | 16:47 |
| corvus | i could go either way on whether that's desirable :) | 16:48 |
| corvus | mordred: if i'm following correctly, we have no actionable patches from you at the moment all the javascript stuff is either in error or based on outdated patchsets. | 16:49 |
| tobiash | corvus: that makes me think about which version of the docs we want to publish | 16:53 |
| corvus | tobiash: i'm pretty happy to publish CD master for the most part, but we can certainly also publish specific versions. | 16:54 |
| tobiash | Latest master or latest tag or both? | 16:54 |
| clarkb | openstack does a project/latest/ which tracks master or latest release and then also project/$release | 16:54 |
| clarkb | s/or/and/ | 16:54 |
| *** dtruong_ has quit IRC | 16:56 | |
| fungi | it's probably also fine to just consider that for now documentation is still under heavier churn while it catches up in places we realize we've missed, but that at some later point it'll make more sense to switch to just publishing when tagged (especially if we're tagging new point releases extremely frequently) | 16:56 |
| *** dtruong has joined #zuul | 16:57 | |
| corvus | Shrews, clarkb: see comment on https://review.openstack.org/560044 | 17:03 |
| SpamapS | I've always liked the paradigm of having a /latest and /{tag} | 17:03 |
| clarkb | corvus: ya I think I used 30 seconds for the database query | 17:04 |
| clarkb | corvus: maybe we use 30 seconds to be inline with the library default? | 17:04 |
| clarkb | corvus: we can also make it set no timeout and have the underlying lib timeout for us instead | 17:05 |
| corvus | clarkb: i think that'd be my preference unless there's an overriding reason to go less | 17:05 |
| corvus | clarkb: that function won't ever timeout though | 17:05 |
| clarkb | 30 seconds is probably a good value then | 17:05 |
| corvus | clarkb: that's RPCClient.submitJob, not gear.Client.submitJob | 17:05 |
| corvus | clarkb: RPCClient.submitJob not only calls gear.Client.submitJob, it *also* waits for the job to finish | 17:06 |
| clarkb | gotcha | 17:06 |
| corvus | i wrote a treatise on this in my review of the gear async zuul-web change | 17:06 |
| *** electrofelix has quit IRC | 17:06 | |
| corvus | i'm pretty sure we're confusing ourselves here, due to the appropriation of the rpcclient by zuul-web | 17:06 |
| corvus | this change: https://review.openstack.org/560026 | 17:07 |
| corvus | clarkb: can you +3 https://review.openstack.org/560090 ? | 17:10 |
| *** hashar is now known as hasharAway | 17:11 | |
| clarkb | corvus: done | 17:12 |
| *** leifmadsen has quit IRC | 17:13 | |
| *** gouthamr has joined #zuul | 17:14 | |
| *** mugsie has quit IRC | 17:14 | |
| *** mugsie has joined #zuul | 17:15 | |
| *** mugsie has quit IRC | 17:15 | |
| *** mugsie has joined #zuul | 17:15 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add build-sphinx check/gate jobs https://review.openstack.org/560090 | 17:17 |
| *** jpena is now known as jpena|off | 17:17 | |
| *** leifmadsen has joined #zuul | 17:21 | |
| Shrews | corvus: clarkb: Ok. I need to familiarize myself with the async stuff in gear, but I'm going to complete the docs changes before doing that. | 17:22 |
| Shrews | If 30s is better, fine w/ me | 17:22 |
| corvus | Shrews: zuul/executor/client.py is a good example of the pattern | 17:23 |
| openstackgerrit | Merged openstack-infra/zuul master: zuul-changes: update for the new api url https://review.openstack.org/557656 | 17:23 |
| mordred | corvus: yes. that is correct. | 17:23 |
| mordred | corvus: although my goal is that by end of day that stack should be good to go | 17:23 |
| corvus | Shrews: and thinking more about it, i say just drop the rpcclient from zuul-web and make a new client just for that. | 17:24 |
| corvus | mordred: ++ | 17:24 |
| mordred | corvus: fwiw - the angular5 patch now has a preview version working against the softwarefactory multi-tenant deploy too - which has shown a place where things were broken for multi-tenant, so yay | 17:24 |
| corvus | i am caught up on reviews -- but in order to do that, i have had to ignore the 126 changes which are failing tests, outdated, or marked WIP. :) | 17:25 |
| mordred | corvus: it has also made me very much want to figure out a $something where we have js layer tests that we run against a zuul either in zuul unit or functional tests somehow | 17:26 |
| mordred | corvus: since there are currently some things only verified by looking at those draft publications - which isn't really sustainable long-term | 17:26 |
| mordred | corvus: \o/ | 17:26 |
| corvus | mordred: yeah, what can we do for that? selenium, or is there some other js thing? | 17:26 |
| mordred | corvus: yah - there's a bunch of good js things | 17:27 |
| mordred | corvus: the main question is more how to wire it all up | 17:27 |
| mordred | corvus: I'm kind of thinking about perhaps a unittest similar to the current test_web tests - but that the thing it does is trigger the js test suite | 17:28 |
| mordred | like "does the js test suite pass against the zuul this unit test just spun up" | 17:28 |
| corvus | mordred: yeah, that's a promising approach | 17:29 |
| mordred | corvus: but that's a bit handwavey - I wanna fix the multi-tenant issue in the angular5 patch first - and the nodeenv patch - but then I'll figure that out | 17:29 |
| corvus | as long as we're not testing that we wrote the same javascript twice. ;) | 17:30 |
| mordred | corvus: aw. but that's such a fun test | 17:30 |
| Shrews | could a non-corvus persona review this rather simple nodepool change? https://review.openstack.org/552538 | 17:30 |
| Shrews | TIA | 17:30 |
| clarkb | Shrews: I'll take a look | 17:30 |
| mordred | Shrews: how non-corvus does the person need to be? | 17:31 |
| *** corvus is now known as jeblair | 17:31 | |
| Shrews | mordred: very | 17:31 |
| *** jeblair is now known as corvus | 17:31 | |
| Shrews | and a jeblair doesn't count | 17:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 17:33 |
| mordred | corvus: if those turn green ^^ they should be ready to go | 17:34 |
| mordred | (they were red before beause I removed a job that was still in use elsewhere :) ) | 17:34 |
| clarkb | oh I should review the nodeenv change too | 17:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove zuul-tox-py35 https://review.openstack.org/560983 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 17:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 17:39 |
| mordred | clarkb: yes please | 17:39 |
| clarkb | its unfortunate that setuptools environment markers aren't capable of checking things outside of some basic python stuff | 17:40 |
| mordred | yah | 17:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add Gerrit docs to Zuul From Scratch https://review.openstack.org/558600 | 17:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add static driver doc to Zuul From Scratch https://review.openstack.org/558802 | 17:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add example nodepool-launcher systemd service file https://review.openstack.org/560993 | 17:53 |
| mrhillsman | "If a job has an empty or no nodeset definition, it will still run and may be able to perform actions on the Zuul executor." | 17:56 |
| mrhillsman | may be able to perform actions on the Zuul executor...can i get a bit more on this part of that sentence | 17:57 |
| clarkb | mordred: I've +2'd it but left some thoughts. There definitely feels like maybe there is a trade off there | 17:57 |
| clarkb | mordred: particularly in spin up time costs. But for local usage that is a one time cost for the most part | 17:57 |
| clarkb | mrhillsman: your playbooks can run tasks on the localhost host (the executor's bwrap container for that job). You won't be able to read much data off of disk or even write arbitrary shell I think. So you're constarined in what you can do to the "safe" modules | 17:58 |
| mrhillsman | thx | 17:58 |
| clarkb | mrhillsman: one example of how we use that setup for us is hitting the read the docs api to trigger a docs rebuild in read the docs iirc | 17:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 17:58 |
| mrhillsman | got it | 17:59 |
| Shrews | corvus: Re: your comment in 558830 ^ about the chmod, I have no idea if that is necessary, but is just grabbed from the instructions that were already merged | 18:00 |
| pabelanger | systemd files should be 0644, likely a warning message on boot if that was missing | 18:01 |
| pabelanger | IIRC | 18:01 |
| openstackgerrit | Merged openstack-infra/nodepool master: Handle ZK session loss during node launch https://review.openstack.org/552538 | 18:03 |
| clarkb | Shrews: in https://review.openstack.org/#/c/558830/4/doc/source/admin/zuul-from-scratch.rst shouldn't the systemctl commands be in the order of enable, start, status? instead of start status enable | 18:05 |
| Shrews | clarkb: i dunno. i am systemd ignorant and just copying already merged text | 18:06 |
| clarkb | Shrews: apparently enable/disable is just for autostarting on boot type stuff | 18:07 |
| clarkb | you can explicitly start a disabled unit so it should be fine as is | 18:07 |
| Shrews | It's funny how this reorg has brought about many questions on already reviewed/merged instructions. :) | 18:07 |
| Shrews | ok. i can change it if need be | 18:08 |
| *** Wei_Liu1 has joined #zuul | 18:08 | |
| pabelanger | I'm just looking at systemd files again and had questions around daemon / non-daemon things for systemd. Our examples are not passing -d, which I believe the preference of corvus. But I believe clarkb or fungi was saying there is something around forking which might not work for type=simple? | 18:08 |
| clarkb | I think its fine. I'm guessing the intent there was to start it, check the status as good then enable it to come back on next boot | 18:08 |
| pabelanger | does our daemon logic in nodepool / zuul call fork() ? | 18:08 |
| clarkb | pabelanger: it does | 18:08 |
| clarkb | pabelanger: it does the proper unix daemonization process which involves a double fork | 18:09 |
| *** Wei_Liu has quit IRC | 18:09 | |
| *** Wei_Liu1 is now known as Wei_Liu | 18:09 | |
| pabelanger | okay, so then we need type=forking in our systemd service files I believe | 18:09 |
| pabelanger | https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type= | 18:09 |
| Shrews | pabelanger: happy to make that change if you point me to the proper location | 18:09 |
| clarkb | and probably set the pidfile too (based on that doc) | 18:09 |
| clarkb | the alternative is to pass -d and let systemd do it for you | 18:10 |
| pabelanger | Shrews: yah,I can point but I think fungi was the expert here :) | 18:10 |
| clarkb | hrm I should make lunch before this phone call | 18:10 |
| openstackgerrit | Merged openstack-infra/zuul master: gerrit: recognize project-created event https://review.openstack.org/560274 | 18:11 |
| pabelanger | Shrews: okay, lets comments on 560993 | 18:11 |
| Shrews | pabelanger: thx! | 18:12 |
| Shrews | corvus: looks like a RFC on 560993 to you as well | 18:13 |
| *** Wei_Liu has quit IRC | 18:14 | |
| mordred | clarkb: agree re: the tradeoffs ... that said, these days nodeenv is actually downloading/installing pre-built node, so you don't have to actually do a build. but yes, also, it should be a one-time cost (or at least no worse than thecost of rebuilding a virtualenv anyway) | 18:14 |
| clarkb | mordred: oh I didn't realize it was just a download now rather than a download and compile. That is nice | 18:15 |
| *** openstackgerrit has quit IRC | 18:19 | |
| Shrews | pabelanger: do the zuul service files need the same changes you mentioned for the np file? | 18:20 |
| Shrews | pabelanger: for https://review.openstack.org/558830 | 18:20 |
| pabelanger | Shrews: yah, we should sync the chance across all, once we figure out how it looks in 558830 | 18:21 |
| pabelanger | err, 560993 | 18:21 |
| corvus | Shrews: i'm not systemd expert, but i can agree that we don't want "-d", so if systemd experts say that means type=forking then i guess that's what we should do :) | 18:28 |
| Shrews | "Computer! Make it so." | 18:28 |
| clarkb | I think it will work without type set to forking but systemd may have a harder time checking service status | 18:29 |
| corvus | Shrews: the important thing is that when folks come up to us at conferences and ask us detailed questions about systemd files, we point at clarkb and pabelanger and run | 18:29 |
| clarkb | uh oh | 18:29 |
| Shrews | corvus: so important | 18:29 |
| tobiash | corvus: what's the reason to not use -d? | 18:30 |
| tobiash | I always used -d (with systemd earlier and now docker) | 18:31 |
| corvus | Shrews, pabelanger, clarkb: i have rtfm now and agree that forking + pidfile is what we want :) | 18:31 |
| corvus | tobiash: it currently comingles debug log settings as well | 18:31 |
| corvus | tobiash: which may be what you want :) | 18:32 |
| tobiash | So maybe I want to switch to forking without knowing that | 18:32 |
| pabelanger | yah, I used simple and -d for local testing, but plan on trying forking now :) | 18:32 |
| *** openstackgerrit has joined #zuul | 18:33 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add example nodepool-launcher systemd service file https://review.openstack.org/560993 | 18:33 |
| tobiash | I'm not sure if forking is going well inside a container | 18:34 |
| corvus | tobiash: it's probably not what you want for that | 18:34 |
| corvus | tobiash: i think we should separate "don't fork" and "debug log" command line options to make this easier | 18:34 |
| Shrews | .. attr:: pidfile | 18:36 |
| Shrews | :default: /var/run/zuul-schedurecr/zuul-scheduler.pid | 18:36 |
| Shrews | lol | 18:36 |
| Shrews | wonder how that passed review?? ;) | 18:36 |
| tobiash | So with -d debug output to stdout is enabled regardless of the logging config? | 18:37 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 18:38 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented scheduler PID file default https://review.openstack.org/561010 | 18:39 |
| clarkb | Shrews: I want to say the default may actually be /var/run/zuul/zuul-$service.pid | 18:41 |
| * clarkb looks | 18:41 | |
| clarkb | '/var/run/zuul/%s.pid' % self.app_name <- yup | 18:42 |
| Shrews | clarkb: oh, so the docs are all wrong | 18:42 |
| Shrews | clarkb: i'll fix 'em up good | 18:42 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 18:43 |
| tobiash | Shrews: added a question on ^ | 18:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 18:44 |
| Shrews | tobiash: i don't think we should depend on folks keep the source repo around, IMO | 18:45 |
| Shrews | I mean, they should be able to figure out they can do that, if they really want to | 18:45 |
| tobiash | Good point | 18:48 |
| corvus | Shrews: ha! that typo has my fingerprints all over it. dvorak has 'crl' all right next to each other. usually i just mash them all when writing the word schedlclcr and it works out | 18:50 |
| Shrews | lol | 18:51 |
| corvus | Shrews: but one more comment on that | 18:52 |
| openstackgerrit | Merged openstack-infra/zuul master: configloader: add variant-description https://review.openstack.org/549748 | 18:53 |
| Shrews | corvus: gah | 18:53 |
| Shrews | corvus: yup, you are correct | 18:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 18:55 |
| Shrews | seemed like such a simple fix on the surface | 18:55 |
| * Shrews hits documentation iceberg | 18:56 | |
| * mordred hands Shrews an icepick | 18:56 | |
| Shrews | mordred: most kind hearted folks would throw a lifevest, but yeah, this icepick will help | 18:57 |
| Shrews | :-P | 18:58 |
| tobiash | Is today something special? Zuul seems to be much busier than normally. Had today a change that waited an hour for nodes around 10:00 utc | 18:59 |
| clarkb | Shrews: https://review.openstack.org/558830 needs the same pidfile update as latest ps for 561010? (I'm not actually sure myself) | 18:59 |
| Shrews | clarkb: yeah, fixing that now | 19:00 |
| Shrews | thx | 19:00 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add sample systemd service files. https://review.openstack.org/558830 | 19:01 |
| corvus | tobiash: yeah, problem with images in rax; pabelanger debugging in #openstack-infra | 19:02 |
| corvus | tobiash: it's annoying, but it's also kinda fun -- this is what we mean when we say we're redundant at the cloud level. we've lost 30% of our capacity, and the only user visible impact is the backlog. | 19:04 |
| tobiash | That's pretty cool | 19:05 |
| tobiash | I'm looking forward to the point where we can say that also about the control plane | 19:06 |
| corvus | indeed :) | 19:07 |
| openstackgerrit | Merged openstack-infra/nodepool master: launcher: handle ZK session loss during handler poll https://review.openstack.org/556335 | 19:07 |
| corvus | unfortunately, the backlog means our doc update isn't going to show up for a while. i'd like to send out the release announcement after that lands (so i can link to the right anchor). so that will probably be much later today, or even tomorrow. | 19:10 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-rackspace for the fun, can see the time where we rolled back to working images :D | 19:12 |
| *** sshnaidm|mtg is now known as sshnaidm|afk | 19:13 | |
| *** gouthamr has quit IRC | 19:20 | |
| *** JasonCL has quit IRC | 19:28 | |
| *** gouthamr has joined #zuul | 19:30 | |
| *** JasonCL_ has joined #zuul | 19:30 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove zuul-cloner command https://review.openstack.org/560958 | 19:31 |
| openstackgerrit | Merged openstack-infra/zuul master: Publish docs in release pipeline https://review.openstack.org/560964 | 19:31 |
| *** JasonCL_ has quit IRC | 19:35 | |
| mordred | corvus: woot. https://review.openstack.org/#/c/560104/ landed | 19:40 |
| mordred | corvus: sigh. PASSED TESTS | 19:40 |
| corvus | mordred: you're getting into the tyops spirit | 19:40 |
| mordred | corvus: dman stragiht | 19:40 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Make the yaml parser aware of '!encrypted/' tags https://review.openstack.org/559963 | 19:44 |
| corvus | oh, that [nodenv]install_command thing is neat | 19:45 |
| corvus | mordred: we probably don't need those for nodepool / remote venvs, but i guess it doesn't hurt too much? | 19:46 |
| mordred | corvus: yah - I wasn't 100% sure if we needed them or not | 19:48 |
| mordred | (although I agree, it's likely not a big hurt) | 19:48 |
| *** gouthamr has quit IRC | 19:50 | |
| *** gouthamr has joined #zuul | 19:51 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: role: Inject public keys in case of failure https://review.openstack.org/535803 | 19:57 |
| clarkb | https://review.openstack.org/#/c/559852/ appears to have hit the lose zk connection possibly due to cpu contention problem | 20:02 |
| clarkb | Shrews: ^ does that look like a correct analysis to you? don't want to recheck it and get it in if we think it may be the cause of the failure (but unrelated tests failed too) | 20:02 |
| clarkb | we might want to consider reducing the concurrency in the gate? | 20:03 |
| Shrews | clarkb: looking | 20:03 |
| *** snapiri has quit IRC | 20:04 | |
| Shrews | 2018-04-12 17:08:33,637 kazoo.client WARNING Connection dropped: outstanding heartbeat ping not received | 20:05 |
| Shrews | clarkb: seems most likely | 20:05 |
| clarkb | ok I am going to recheck the change in that case and then see if there is a not crazy way to get testr and tox to work together on choosing a concurrency level | 20:06 |
| Shrews | that is odd though. i haven't seen failures quite like that yet | 20:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 20:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Test base job secrets https://review.openstack.org/561030 | 20:11 |
| *** snapiri has joined #zuul | 20:11 | |
| corvus | (that's just a rebase on a new test which should show the series failing) | 20:12 |
| *** JasonCL has joined #zuul | 20:16 | |
| *** JasonCL has quit IRC | 20:18 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 20:20 |
| openstackgerrit | Merged openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 20:20 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix documented PID file defaults https://review.openstack.org/561010 | 20:20 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Reduce test concurrency by 1 cpu https://review.openstack.org/561037 | 20:21 |
| clarkb | Shrews: corvus ^ I think I figured out a non invasive way to support this | 20:21 |
| corvus | clarkb: what's the current failure rate due to that error? | 20:23 |
| clarkb | corvus: let me check logstash | 20:23 |
| openstackgerrit | Merged openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 20:27 |
| clarkb | using the specific kazoo message above and searching against zuul-tox-py35 just this one time in last 7 days according to logstash | 20:27 |
| clarkb | I know shrews has run into it before though (whcih is why I pinged shrews earlier) | 20:27 |
| corvus | clarkb: which provider? | 20:27 |
| clarkb | this job was inap | 20:28 |
| corvus | maybe we still want to do it? it might be worth several rechecks and comparing runtimes (especially if we hit multiple providers) to see what the tradeoff is | 20:29 |
| clarkb | ++ | 20:29 |
| Shrews | clarkb: i have? O.o | 20:29 |
| clarkb | Shrews: I seem to recall you debugging an issue related to zk dying in the test runs | 20:30 |
| clarkb | with pabelanger maybe? | 20:30 |
| clarkb | (where dying is connections are lost otherwise unhappy) | 20:30 |
| Shrews | i've seen that before (rarely), but not the cpu contention stuff | 20:30 |
| corvus | i left a wip note on that change | 20:31 |
| clarkb | oh I thoughti t had been attributed to cpu contention | 20:31 |
| clarkb | fwiw I think we hae noticed real slowdowns post meltdown | 20:32 |
| corvus | (conference talk idea ^) | 20:32 |
| clarkb | qemu + nova in particular. I wouldn't be surprised if that is also playing a part in this with the higher cost of context switches | 20:32 |
| clarkb | one of the known issues was nova had no way of exposing the pcid instruction to guests which was recently fixed. Hopefully we see that fix roll out to the clouds soonish | 20:33 |
| clarkb | (so we were worst case meltdown performance hit everywhere rather than best case) | 20:33 |
| clarkb | s/instruction/feature/ | 20:33 |
| *** dkranz has quit IRC | 20:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix typo in build-sphinx-docs docstring https://review.openstack.org/561040 | 20:41 |
| *** JasonCL has joined #zuul | 20:42 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: document that the build-reno-releasenotes job accepts sphinx_python https://review.openstack.org/561041 | 20:43 |
| *** gouthamr_ has joined #zuul | 20:43 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: document that the build-reno-releasenotes job accepts sphinx_python https://review.openstack.org/561041 | 20:49 |
| *** JasonCL has quit IRC | 20:51 | |
| *** JasonCL has joined #zuul | 20:51 | |
| *** JasonCL has quit IRC | 20:55 | |
| mrhillsman | question: looking at http://status.openlabtesting.org/stream.html?uuid=305a88df898f474694f4c3bd12870631&logfile=console.log | 20:56 |
| mrhillsman | if you search for zun | 20:56 |
| mrhillsman | can see that the depends-on syntax is working | 20:56 |
| mrhillsman | HEAD is now at ebe01a5 [WIP] Support enabling Zun on devstack | 20:57 |
| mrhillsman | but the actual code from that does not appear to be being used | 20:57 |
| clarkb | mrhillsman: can you link to that change? | 20:58 |
| mrhillsman | sure | 20:58 |
| mrhillsman | https://github.com/gophercloud/gophercloud/pull/927#issuecomment-380917196 | 20:58 |
| *** pwhalen has quit IRC | 20:58 | |
| mrhillsman | i think the console is gone now :( | 20:59 |
| SpamapS | mrhillsman: do you not have a post job that sets the result and stores the log somewhere? | 20:59 |
| mrhillsman | yeah, it is there | 20:59 |
| clarkb | mrhillsman: that seems to be a dummy change, not the update to enable zun | 20:59 |
| mrhillsman | so the update is in the first comment depends-on there | 21:00 |
| mrhillsman | https://github.com/theopenlab/openlab-zuul-jobs/pull/142 | 21:00 |
| *** JasonCL has joined #zuul | 21:01 | |
| *** JasonCL has joined #zuul | 21:02 | |
| *** pwhalen has joined #zuul | 21:02 | |
| *** pwhalen has joined #zuul | 21:02 | |
| clarkb | mrhillsman: is openlab-zuul-jobs a trusted repo? | 21:02 |
| clarkb | or config repo? whatever the term is | 21:03 |
| mrhillsman | yes, but let me confirm | 21:03 |
| *** JasonCL has quit IRC | 21:03 | |
| mrhillsman | it is | 21:03 |
| *** JasonCL has joined #zuul | 21:03 | |
| clarkb | mrhillsman: the job config of trusted repos won't be speculatively applied | 21:04 |
| clarkb | mrhillsman: you have to merge them first | 21:04 |
| mrhillsman | ah ok | 21:04 |
| mrhillsman | i thought that "Already on master" shed some light on the why | 21:04 |
| clarkb | mrhillsman: I think the git checkouts will still checkout the right ref | 21:04 |
| mrhillsman | was wondering why it pulled in the change it appeared but then said that | 21:04 |
| mrhillsman | got it, thx | 21:05 |
| clarkb | but your job config in zuul itself won't be updated | 21:05 |
| *** rlandy is now known as rlandy|afk | 21:05 | |
| mrhillsman | cool, ty sir | 21:05 |
| clarkb | mrhillsman: and the reason for that is to prevent someone from pushing up a commit that exposes your secrets or otherwise updates how sensitive jobs run without being reviewed by a human first | 21:05 |
| mrhillsman | totally understand | 21:06 |
| *** gouthamr has quit IRC | 21:06 | |
| clarkb | I should clarify I don't think they necessarily have to merge first they just have to get into a trusted pipeline | 21:06 |
| mrhillsman | also after reading through docs the last couple days, which i have more to do, now that i look at our main.yaml i think it needs some changing in general | 21:06 |
| mrhillsman | what's a "trusted pipeline"? docs search returns empty | 21:07 |
| mrhillsman | or what qualifies a pipeline as trusted is probably better question | 21:08 |
| clarkb | mrhillsman: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n69 | 21:08 |
| corvus | clarkb: i think config changes to trusted-projects always have to land before being used | 21:08 |
| clarkb | corvus: ah ok | 21:08 |
| mrhillsman | ah ok, need to read post-review, pabelanger mentioned something of this to me not long ago | 21:09 |
| mrhillsman | thx for clarity corvus | 21:09 |
| clarkb | so there are two levels of control here. The first being for trusted repos themselves and the second (post-review) for consuming secrets in general | 21:09 |
| mrhillsman | i think the docs say that | 21:09 |
| mrhillsman | trusted-projects having to land first | 21:09 |
| mrhillsman | but again, i think we have it wrong | 21:10 |
| mrhillsman | openlab-zuul-jobs should be ok not being trusted | 21:10 |
| mrhillsman | or we need to move some things out of it | 21:11 |
| mrhillsman | i'm leaning towards the latter | 21:11 |
| mrhillsman | thx again for the clarity | 21:11 |
| corvus | mordred: i got an npm error which i don't understand: http://paste.openstack.org/show/719100/ | 21:11 |
| *** hongbin has joined #zuul | 21:12 | |
| pabelanger | mrhillsman: yah, that was related to how openlab as using secrets and untrusted jobs | 21:13 |
| pabelanger | which, would leak on proposed PRs | 21:13 |
| mrhillsman | yep, i remember you saying that, still have more reading to do but experimenting and reading is helping :) | 21:14 |
| mrhillsman | i have a test environment setup so when i look at the live main.yaml it looks awkward based on my reading and definitely the change i mentioned should not be restricted to trusted-project constraints | 21:15 |
| *** gouthamr has joined #zuul | 21:15 | |
| *** JasonCL has quit IRC | 21:16 | |
| corvus | mordred: okay, apparently "rm -rf node_modules" then "npm run build" fixed that... | 21:24 |
| corvus | mordred: is that... erm, the only way to fix such errors? it seems... nuclear. | 21:24 |
| *** openstackstatus has quit IRC | 21:27 | |
| *** openstack has joined #zuul | 21:31 | |
| *** ChanServ sets mode: +o openstack | 21:31 | |
| *** hasharAway has quit IRC | 21:33 | |
| mordred | corvus: sorry - was debugging jaascript in a different window ... | 21:44 |
| clarkb | initial data back on reducing concurrency of testr is it took 12 minutes 8 seconds on ovh-bhs1. http://logs.openstack.org/52/559852/4/gate/tox-py35/696a8da/job-output.txt http://logs.openstack.org/10/561010/3/gate/tox-py35/bef3fc5/job-output.txt http://logs.openstack.org/83/560983/1/check/tox-py35/c0c3e8f/job-output.txt http://logs.openstack.org/65/560265/1/check/tox-py35/cc666ea/job-output.txt are all zuul | 21:46 |
| clarkb | tox-py35 jobs that have run recetnly on the same cloud region and range from ~11-13 minutes in runtime | 21:46 |
| mordred | corvus: in this particular case, the issue stems from the fact that we landed the patch to upgrade webpack from 2 to 4 , so the locally installed version of webpack did not know how to deal with the config files | 21:46 |
| clarkb | so I don't think this makes the jobs significantly slower ( at least not in that cloud ). I guess it doesn't tell us anything about reliability. I have rechecked it to continue to get more info | 21:46 |
| mordred | corvus: you could also have run "yarn install" (which would have updated your installed depends to match the declared depends) | 21:46 |
| mordred | corvus: (and also would not re-download all the things from node_modules) - but, it is always safe to just delete node_modules and start over | 21:47 |
| clarkb | mordred: does it not realize that it needed to upgrade webpack 4 before packing things? | 21:48 |
| clarkb | mordred: seems like that is something yarn should know how to do? | 21:48 |
| clarkb | or maybe its npm? | 21:48 |
| *** JasonCL has joined #zuul | 21:52 | |
| *** gouthamr has quit IRC | 21:54 | |
| *** JasonCL has quit IRC | 21:56 | |
| mordred | clarkb: it's two different actions - 'npm run build' is like 'python setup.py bdist_wheel' - and 'yarn install' is like 'pip install -r requirements.txt' - if you did the pip install, then did a git pull that pulled in a new entry in requirements, you'd need to run pip install again - python setup.py bdist_wheel would not install changed requirements for you | 21:57 |
| mordred | also - webpack is actually the top-level command line tool - 'npm run build' actually just runs 'webpack --env=prod' | 21:58 |
| mordred | (but is able to resolve the webpack command from node_modules/.bin/webpack so you don't have to type that) | 21:58 |
| corvus | mordred: ok, 'yarn install' is the thing i was missing here, thanks | 21:59 |
| corvus | mordred: also, i know the feeling (debugging js in another window :) | 21:59 |
| mordred | corvus: :) | 21:59 |
| mordred | corvus: which js are you debugging? | 21:59 |
| corvus | mordred: i'm making the status page do terrible things :) | 22:00 |
| corvus | (i'm not having any current issues; i'm making progress) | 22:00 |
| mordred | corvus: yay! | 22:02 |
| *** hongbin has left #zuul | 22:02 | |
| *** harlowja has joined #zuul | 22:24 | |
| *** JasonCL has joined #zuul | 22:31 | |
| *** JasonCL has quit IRC | 22:35 | |
| *** JasonCL has joined #zuul | 22:58 | |
| *** JasonCL has quit IRC | 23:00 | |
| *** JasonCL has joined #zuul | 23:01 | |
| *** JasonCL has quit IRC | 23:03 | |
| *** JasonCL has joined #zuul | 23:06 | |
| *** JasonCL has quit IRC | 23:07 | |
| *** JasonCL has joined #zuul | 23:13 | |
| *** JasonCL has quit IRC | 23:15 | |
| *** gouthamr has joined #zuul | 23:21 | |
| *** JasonCL has joined #zuul | 23:33 | |
| *** JasonCL has quit IRC | 23:34 | |
| *** JasonCL has joined #zuul | 23:36 | |
| -openstackstatus- NOTICE: The Etherpad service at https://etherpad.openstack.org/ is being restarted to pick up the latest release version; browsers should see only a brief ~1min blip before reconnecting automatically to active pads | 23:41 | |
| *** JasonCL has quit IRC | 23:41 | |
| *** JasonCL has joined #zuul | 23:44 | |
| *** JasonCL has quit IRC | 23:49 | |
| *** JasonCL has joined #zuul | 23:50 | |
| *** JasonCL has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!