| *** tosky has quit IRC | 00:02 | |
| *** iurygregory has quit IRC | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:17 | |
| *** jamesmcarthur has joined #zuul | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #zuul | 00:21 | |
| *** holser has quit IRC | 00:23 | |
| *** CrayZee has quit IRC | 00:35 | |
| *** snapiri has joined #zuul | 00:35 | |
| *** iurygregory has joined #zuul | 00:43 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| *** jamesmcarthur has joined #zuul | 00:45 | |
| *** rlandy has quit IRC | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** jamesmcarthur has joined #zuul | 00:49 | |
| *** ikhan has joined #zuul | 01:00 | |
| *** zenkuro has quit IRC | 01:03 | |
| *** zenkuro has joined #zuul | 01:03 | |
| *** hamalq has quit IRC | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #zuul | 01:35 | |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** harrymichal has quit IRC | 01:59 | |
| *** jamesmcarthur has joined #zuul | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| *** jamesmcarthur has joined #zuul | 02:15 | |
| *** y2kenny has joined #zuul | 02:28 | |
| y2kenny | Has anyone run into issue with the latest nodepool while creating kubernetes namespace? | 02:31 |
|---|---|---|
| *** ajitha has joined #zuul | 02:38 | |
| *** jamesmcarthur has quit IRC | 02:47 | |
| *** jamesmcarthur has joined #zuul | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** jamesmcarthur has joined #zuul | 02:50 | |
| *** zenkuro has quit IRC | 03:09 | |
| *** bhavikdbavishi has joined #zuul | 03:26 | |
| *** y2kenny has quit IRC | 03:27 | |
| *** bhavikdbavishi has quit IRC | 04:25 | |
| *** bhavikdbavishi has joined #zuul | 04:34 | |
| *** bhavikdbavishi1 has joined #zuul | 04:37 | |
| *** bhavikdbavishi has quit IRC | 04:39 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:39 | |
| *** bhavikdbavishi has quit IRC | 05:01 | |
| *** bhavikdbavishi has joined #zuul | 05:02 | |
| *** ykarel|away has joined #zuul | 05:03 | |
| *** bhavikdbavishi has quit IRC | 05:18 | |
| *** bhavikdbavishi has joined #zuul | 05:19 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** iurygregory has quit IRC | 05:33 | |
| *** jfoufas1 has joined #zuul | 05:42 | |
| *** jamesmcarthur has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #zuul | 05:52 | |
| *** vishalmanchanda has joined #zuul | 05:52 | |
| *** jamesmcarthur has quit IRC | 05:57 | |
| *** jfoufas1 has quit IRC | 05:59 | |
| *** jamesmcarthur has joined #zuul | 06:00 | |
| *** zbr5 has joined #zuul | 06:04 | |
| *** jfoufas1 has joined #zuul | 06:05 | |
| *** zbr has quit IRC | 06:06 | |
| *** zbr5 is now known as zbr | 06:06 | |
| *** ykarel_ has joined #zuul | 06:17 | |
| *** ykarel|away has quit IRC | 06:19 | |
| *** ykarel_ is now known as ykarel | 06:26 | |
| *** bhavikdbavishi1 has joined #zuul | 06:29 | |
| *** bhavikdbavishi has quit IRC | 06:30 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:30 | |
| *** sanjayu_ has joined #zuul | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:37 | |
| *** bhavikdbavishi has quit IRC | 07:26 | |
| *** openstackgerrit has joined #zuul | 07:36 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Cache unparsed config files in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771455 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store tenants in unparsed abide as dict https://review.opendev.org/c/zuul/zuul/+/771456 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Tenant read/write lock in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771458 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Lock pipelines during processing https://review.opendev.org/c/zuul/zuul/+/771459 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Lock global event queues during processing https://review.opendev.org/c/zuul/zuul/+/771460 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store tenant layout state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771461 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Configure unique command socket path per scheduler https://review.opendev.org/c/zuul/zuul/+/771462 | 07:36 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 07:36 |
| openstackgerrit | Daniel Blixt proposed zuul/zuul-jobs master: Use urlencoded filenames in test fixtures https://review.opendev.org/c/zuul/zuul-jobs/+/771566 | 07:38 |
| Phoenikzz | tobiash: In early december we discussed https://review.opendev.org/c/zuul/zuul-jobs/+/764062, which I have now abandoned in favour of https://review.opendev.org/c/zuul/zuul-jobs/+/771566 where I went with your suggestion instead. Happy to get feedback | 07:42 |
| *** jcapitao has joined #zuul | 07:51 | |
| *** sanjayu__ has joined #zuul | 07:52 | |
| *** sanjayu_ has quit IRC | 07:55 | |
| *** hashar has joined #zuul | 08:03 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| *** jamesmcarthur has joined #zuul | 08:37 | |
| *** smyers has quit IRC | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:42 | |
| *** smyers has joined #zuul | 08:43 | |
| *** bhavikdbavishi has joined #zuul | 08:45 | |
| *** tosky has joined #zuul | 08:47 | |
| *** bhavikdbavishi1 has joined #zuul | 08:47 | |
| *** bhavikdbavishi has quit IRC | 08:49 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 08:49 | |
| *** nils has joined #zuul | 08:50 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** harrymichal has joined #zuul | 09:07 | |
| *** lyr4 has joined #zuul | 09:17 | |
| *** lyr has quit IRC | 09:18 | |
| *** tristanC has quit IRC | 09:18 | |
| *** lyr4 is now known as lyr | 09:18 | |
| *** tristanC has joined #zuul | 09:20 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make buildset mandatory on build https://review.opendev.org/c/zuul/zuul/+/770900 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Implement ZooKeeper builds API https://review.opendev.org/c/zuul/zuul/+/770901 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Switch to ZooKeeper backed job execution and result events https://review.opendev.org/c/zuul/zuul/+/770902 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Merge ZooKeeper connection and client classes https://review.opendev.org/c/zuul/zuul/+/771442 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Refactor pipeline processing in run handler https://review.opendev.org/c/zuul/zuul/+/771452 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Dequeue superceded items via management event https://review.opendev.org/c/zuul/zuul/+/771453 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Text stream API for sharded Zookeeper data https://review.opendev.org/c/zuul/zuul/+/771454 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Cache unparsed config files in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771455 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Store tenants in unparsed abide as dict https://review.opendev.org/c/zuul/zuul/+/771456 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Tenant read/write lock in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771458 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Lock pipelines during processing https://review.opendev.org/c/zuul/zuul/+/771459 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Lock global event queues during processing https://review.opendev.org/c/zuul/zuul/+/771460 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Store tenant layout state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771461 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Configure unique command socket path per scheduler https://review.opendev.org/c/zuul/zuul/+/771462 | 09:30 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 09:30 |
| *** rpittau is now known as rpittau|bbl | 10:20 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: bindep: remove set_fact usage when converting string to list https://review.opendev.org/c/zuul/zuul-jobs/+/771585 | 10:24 |
| avass | we should probably have a policy to not override input variables inside roles and instead use 'private' variables with set_fact if they're needed | 10:25 |
| avass | since it's not possible to override variables set with include_role | 10:25 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:43 | |
| *** hashar has quit IRC | 10:50 | |
| *** bhavikdbavishi has quit IRC | 11:04 | |
| *** ikhan has quit IRC | 11:15 | |
| *** jcapitao is now known as jcapitao_lunch | 11:24 | |
| *** rpittau|bbl is now known as rpittau | 11:24 | |
| *** iurygregory has joined #zuul | 11:27 | |
| *** rlandy has joined #zuul | 12:20 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** evrardjp_ has joined #zuul | 12:52 | |
| *** evrardjp has quit IRC | 12:54 | |
| *** holser has joined #zuul | 12:57 | |
| *** jcapitao_lunch is now known as jcapitao | 13:02 | |
| *** jpena|lunch is now known as jpena | 13:28 | |
| *** zenkuro has joined #zuul | 13:38 | |
| tobiash | corvus: update to our scheduler performance problem. Our profiling revealed that our scheduler took 1/3 of its cpu time (10m during 30min yappi run) for status api formatting | 13:45 |
| tobiash | corvus: I think we can mitigate this easily to make the hard coded cache expiry in zuul-web configurable | 13:45 |
| tobiash | we're just rolling out a hot patched zuul-web to see if this improves our situation also in real life | 13:45 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: REST API: add api/tenant/{tenant}/jobs/running endpoint https://review.opendev.org/c/zuul/zuul/+/771648 | 13:48 |
| *** sanjayu__ has quit IRC | 13:53 | |
| tobiash | corvus: good news, this instantaneously fixed our event backlog :) | 14:05 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Cache tenant status longer https://review.opendev.org/c/zuul/zuul/+/771649 | 14:05 |
| mordred | tobiash: it was spending the time just formatting the data structure? | 14:14 |
| tobiash | yes | 14:15 |
| *** vishalmanchanda has quit IRC | 14:15 | |
| mordred | wow | 14:15 |
| tobiash | I think this could be useful for opendev as well | 14:15 |
| mordred | yeah - I imagine so | 14:16 |
| mordred | also - this should be improved by distributed scheduler, no? | 14:16 |
| mordred | (longer term - obviously) | 14:16 |
| corvus | yeah, the web process will be handling the formatting | 14:17 |
| *** irclogbot_3 has quit IRC | 14:27 | |
| *** irclogbot_3 has joined #zuul | 14:32 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: REST API: add api/tenant/{tenant}/jobs/running endpoint https://review.opendev.org/c/zuul/zuul/+/771648 | 14:52 |
| *** vishalmanchanda has joined #zuul | 14:53 | |
| tobiash | corvus: do you agree that we should make the cache time configurable in zuul-web? | 15:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add policy about overriding role input variables https://review.opendev.org/c/zuul/zuul-jobs/+/771655 | 15:00 |
| tobiash | I think we need at least 15s which might be too much for small deployments | 15:00 |
| *** hashar has joined #zuul | 15:04 | |
| tristanC | tobiash: would it be better to handle that with a caching proxy? | 15:11 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store tenants in unparsed abide as dict https://review.opendev.org/c/zuul/zuul/+/771456 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Refactor config/tenant (re-)loading https://review.opendev.org/c/zuul/zuul/+/771457 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Tenant read/write lock in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771458 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Lock pipelines during processing https://review.opendev.org/c/zuul/zuul/+/771459 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Lock global event queues during processing https://review.opendev.org/c/zuul/zuul/+/771460 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store tenant layout state in Zookeeper https://review.opendev.org/c/zuul/zuul/+/771461 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Configure unique command socket path per scheduler https://review.opendev.org/c/zuul/zuul/+/771462 | 15:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Support cross scheduler config loading https://review.opendev.org/c/zuul/zuul/+/771463 | 15:12 |
| tobiash | tristanC: I'd like to avoid a caching proxy in front of the api and further the caching is already implemented internally in zuul-web, the question is just about the duration | 15:12 |
| tristanC | tobiash: well we could drop the zuul-web internal cache... on the other hand it sounds easier to make the existing duration configurable | 15:14 |
| tobiash | dropping the zuul-web internal cache means breaking opendev immediately | 15:15 |
| tobiash | this would then mean that even smaller deployments need to think about adding a caching proxy | 15:16 |
| tobiash | extending this further to the authentication support this makes caching even harder (or broken) | 15:17 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: REST API: add api/tenant/{tenant}/jobs/running endpoint https://review.opendev.org/c/zuul/zuul/+/771648 | 15:50 |
| corvus | tobiash: oh i just had an idea -- what if we made it automatic? we could have the scheduler time how long it takes to format a status report, and cache it for some factor of that amount of time | 16:11 |
| tobiash | corvus: interesting idea, however I think that's quite complicated and probably hard to get right | 16:13 |
| corvus | tobiash: how long do you think your status formats are taking? | 16:14 |
| tobiash | I have no clue, we don't even have logs of that path yet | 16:15 |
| tobiash | all I know is the cumulative cpu time it took during profiling | 16:16 |
| corvus | tobiash: i think it would be pretty simple -- just log the start and end times of the status build, add it as an extra field in the data sent to the web server, then use that as the basis for the cache time | 16:17 |
| clarkb | it is probably variable and related to queue sizes too | 16:17 |
| corvus | clarkb: yeah, and this method would keep up with that | 16:17 |
| clarkb | so the value would likely change over time? | 16:17 |
| clarkb | ya that | 16:17 |
| corvus | tobiash: we could maybe do this in 2 changes -- first collect and send the data and log it. then we can guess a good multiplier and implement that as a second change. | 16:18 |
| tobiash | k, collecting the data is something we need to do anyways | 16:20 |
| *** ykarel has quit IRC | 16:21 | |
| tobiash | however I'm not yet completely convinced that this mechanism is not a little bit over engineered given that we were the first ones hitting this limit | 16:21 |
| corvus | tobiash: i'm sure it'll help opendev too | 16:22 |
| tobiash | but I see the benefit that small tenants then have less delay with this mechanism | 16:22 |
| corvus | tobiash: i'd really like to avoid adding tuning parameters if they're not necessary. it's daunting for new users, and computers should be able to figure this out for themselves | 16:22 |
| corvus | i honestly don't think it's going to be any more lines of code than a configurable paremeter :) | 16:23 |
| tobiash | yeah sure, I'm just thinking if it's worth to spend time in such a mechanism compared to a simple switch if there are just opendev, us and volvo who would need to tune that | 16:23 |
| corvus | i think it's important | 16:24 |
| tobiash | k, then let's try that | 16:24 |
| tobiash | corvus: just looking at that. If we want to keep the format I think we cannot add the time to dump the json to that part | 16:29 |
| corvus | tobiash: i don't think we need to keep the format | 16:29 |
| corvus | tobiash: we can do a simple check on the receiving side to determine if it's old or new format, then remove that after the next release, in order to make upgrades easier | 16:30 |
| tobiash | fyi, this was the status handling part of the profiling: https://paste.pics/d62dc92e4ed6e9a2dd4c431b28dbf9b1 | 16:33 |
| tobiash | times are cpu time and profiling time was ~30min | 16:33 |
| tobiash | so the json serialization is quite a bit of that | 16:33 |
| tobiash | corvus: what do you think about <duration in ms>\n<statusjson> as format? | 16:35 |
| tobiash | then zuul-web can judge by isint(<first line>) if it's the new format | 16:35 |
| corvus | tobiash: why not a dict? {duration: x, status: {}} ? | 16:35 |
| tobiash | woudldn't gear double serialize this then? | 16:36 |
| tobiash | current format is the already serialized dict | 16:36 |
| tobiash | it looks like gear wants a str or binary | 16:37 |
| tobiash | we can also put it into the dict and neglect the json serialization (and treat that as roughly proportional of the effort to create the dict) | 16:38 |
| corvus | tobiash: we should change formatStatusJSON to return a dict and handle the json serialization in the rpclistener | 16:39 |
| corvus | or... | 16:39 |
| tobiash | then we still cannot serialize the duration of the serialization into the serialization result | 16:40 |
| corvus | tobiash: yeah, put the timing checks inside formatStatusJSON and return the new dict there | 16:40 |
| corvus | no either works | 16:40 |
| corvus | we either: 1) have formatStatusJson return a dict, then json.dumps({duration:x, status:status}) inside the rpc listener | 16:41 |
| corvus | or 2) have formatStatusJSON calculate the time, enclose the status dict in the new result dict, and then return a serialized string of that | 16:41 |
| corvus | it's the same thing, just on either side of the return from formatStatusJSON | 16:42 |
| tobiash | yeah, so this neglects the json.dumps as part of the duration | 16:42 |
| corvus | tobiash: the 159s grey square? | 16:44 |
| tobiash | yes | 16:44 |
| corvus | tobiash: we could 1) ignore that and just account for it in the multiplier; 2) adopt a simple newline delimited format like you suggest; 3) send a separate gearman message (job.data[0] and job.data[1]). | 16:48 |
| tobiash | a different idea would be to judge by payload size so cache for x seconds per y kb | 16:48 |
| corvus | tobiash: yeah, that would work too | 16:48 |
| corvus | tobiash: that would be very simple to implement. i like it. | 16:49 |
| tobiash | the payload size would have the advantage that the protocol and scheduler don't need to be changed | 16:49 |
| corvus | yep, that approach has my vote :) | 16:49 |
| tobiash | :) | 16:49 |
| tobiash | so first patch, log the updates together with payload size | 16:50 |
| corvus | (especially since this will all be very different with distributed schedulers) | 16:50 |
| corvus | tobiash: and go ahead and include the times in that log msg too, i'd say, just for our own interest | 16:50 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Enhance logging of status_get requests https://review.opendev.org/c/zuul/zuul/+/771669 | 16:55 |
| corvus | tobiash: suggestion inline | 16:58 |
| tobiash | ++ | 16:59 |
| *** evrardjp has joined #zuul | 17:03 | |
| *** jcapitao has quit IRC | 17:03 | |
| *** evrardjp_ has quit IRC | 17:06 | |
| *** sanjayu__ has joined #zuul | 17:08 | |
| *** jfoufas1 has quit IRC | 17:09 | |
| *** jamesmcarthur has joined #zuul | 17:10 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Enhance logging of status_get requests https://review.opendev.org/c/zuul/zuul/+/771669 | 17:11 |
| *** sanjayu__ has quit IRC | 17:13 | |
| *** nils has quit IRC | 17:19 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Enhance logging of status_get requests https://review.opendev.org/c/zuul/zuul/+/771669 | 17:21 |
| *** evrardjp_ has joined #zuul | 17:25 | |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| *** evrardjp has quit IRC | 17:28 | |
| *** evrardjp_ has quit IRC | 17:48 | |
| *** evrardjp has joined #zuul | 17:51 | |
| *** evrardjp_ has joined #zuul | 18:19 | |
| *** cloudnull has quit IRC | 18:19 | |
| *** cloudnull has joined #zuul | 18:20 | |
| *** evrardjp has quit IRC | 18:22 | |
| *** cloudnull5 has joined #zuul | 18:26 | |
| *** cloudnull has quit IRC | 18:27 | |
| *** cloudnull5 is now known as cloudnull | 18:27 | |
| *** y2kenny has joined #zuul | 18:31 | |
| *** jpena is now known as jpena|off | 18:32 | |
| *** evrardjp has joined #zuul | 18:32 | |
| y2kenny | Has anyone run into issue with the latest nodepool while nodepool creates kubernetes namespace? | 18:32 |
| *** evrardjp_ has quit IRC | 18:35 | |
| corvus | y2kenny: what kind of issue? | 18:35 |
| tobiash | y2kenny: can you paste logs | 18:36 |
| tobiash | ? | 18:36 |
| y2kenny | unfortunately I didn't keep the log but something related to create_namespace not found or something. | 18:36 |
| corvus | y2kenny: are you running from master or a release? | 18:36 |
| y2kenny | I can try to reproduce it again (I sort of recovered quickly but switching to the tagged build from dockerhub.) | 18:36 |
| y2kenny | corvus: I was running nodepool:latest from dockerhub | 18:37 |
| corvus | gotcha | 18:37 |
| tobiash | I'd suspect https://review.opendev.org/c/zuul/nodepool/+/765873 | 18:37 |
| corvus | tobiash: agree | 18:37 |
| corvus | y2kenny: since https://review.opendev.org/765873 has landed you may want to try :latest on dockehub again | 18:37 |
| tobiash | a log would help us to see if this can be fixed easily or if we have to revert that | 18:37 |
| avass | yep that's probably the only thing that has touched it | 18:37 |
| corvus | oh i was thinking 873 might have fixed it | 18:37 |
| y2kenny | I was trying to read into the code but I wasn't familiar with kubernetes python library | 18:37 |
| corvus | i honestly think it could go either way | 18:37 |
| avass | is this starting a pod or giving zuul an entire namespace? | 18:38 |
| corvus | avass: namespace | 18:38 |
| tobiash | I guess 873 might have had missed one use case in the tests | 18:38 |
| avass | I could see if i can reproduce it in my system | 18:38 |
| avass | corvus: yeah but nodepool always creates a namespace :) | 18:38 |
| y2kenny | avass: I am using namespace only | 18:39 |
| *** vishalmanchanda has quit IRC | 18:40 | |
| avass | y2kenny: got it, I'll see how easy I can get that set up | 18:40 |
| tobiash | just double checked, both namespace and pod use cases are covered in the functional k8s test | 18:45 |
| y2kenny | sorry I forgot which paste bin equivalent you guys use, can someone be so kind to post the domain? | 18:45 |
| tobiash | http://paste.openstack.org/ | 18:45 |
| avass | tobiash: oh are they using the same logic? | 18:45 |
| y2kenny | tobiash: thanks | 18:45 |
| y2kenny | error log is here: http://paste.openstack.org/show/801796/ | 18:46 |
| y2kenny | doesn't happen for 3.14.0 tag on dockerhub | 18:46 |
| tobiash | the client is not initialized according to the log | 18:48 |
| tobiash | sounds similar to https://storyboard.openstack.org/#!/story/2008533 | 18:48 |
| avass | yep I'm getting the same error | 18:48 |
| tobiash | but I'm not sure I grok that correctly | 18:48 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 18:50 |
| tobiash | y2kenny: do you have a separate config or is your use case that it should use the implicit in cluster config? | 18:50 |
| y2kenny | tobiash: this is implicit in cluster | 18:51 |
| y2kenny | like nodepool was launched within the k8s cluster | 18:51 |
| y2kenny | in fact, the entire zuul deployment is deployed on a k8s cluster | 18:52 |
| tobiash | ah I think I've understood the issue | 18:52 |
| tobiash | just a sec | 18:52 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Fix k8s driver when using in cluster config https://review.opendev.org/c/zuul/nodepool/+/771704 | 18:54 |
| tobiash | if I've understood the issue correctly this should be the fix ^ | 18:54 |
| avass | I can probably test that quick | 18:55 |
| tristanC | tobiash: i'll add a depends-on with https://review.opendev.org/c/zuul/zuul-operator/+/771703 if it reproduce the issue | 18:55 |
| tobiash | ah and the functional test couldn't catch this since the test doesn't run within the k8s cluster | 18:56 |
| tristanC | the zuul-operator integration test runs a nodepool configured with the implicit in cluster config | 18:56 |
| tobiash | cool | 18:56 |
| avass | oh nice | 18:58 |
| *** hashar is now known as hasharAway | 19:00 | |
| avass | that seems to work btw | 19:04 |
| avass | https://zuul.vassast.org/status | 19:04 |
| *** harrymichal has quit IRC | 19:22 | |
| *** harrymichal has joined #zuul | 19:23 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 19:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: Test new openshift version fix from nodepool https://review.opendev.org/c/zuul/zuul-operator/+/770158 | 19:29 |
| *** jamesmcarthur has quit IRC | 19:39 | |
| *** jamesmcarthur has joined #zuul | 19:53 | |
| *** y2kenny has quit IRC | 19:57 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 20:00 |
| *** maxamillion has joined #zuul | 20:04 | |
| *** jamesmcarthur has quit IRC | 20:05 | |
| *** jamesmcarthur has joined #zuul | 20:35 | |
| *** jamesmcarthur_ has joined #zuul | 20:41 | |
| *** harrymichal_ has joined #zuul | 20:41 | |
| *** tosky has quit IRC | 20:41 | |
| *** harrymichal has quit IRC | 20:42 | |
| *** tosky has joined #zuul | 20:42 | |
| *** fbo has joined #zuul | 20:42 | |
| *** harrymichal_ is now known as harrymichal | 20:42 | |
| tristanC | tobiash: oops, i was incorrect, the operator tests are using a provided kubeconfig, not the default one. | 20:42 |
| *** jamesmcarthur has quit IRC | 20:42 | |
| *** smyers has quit IRC | 20:42 | |
| corvus | tristanC: i think avass said he verified it, so maybe we should go ahead and +3 it? | 20:43 |
| *** smyers has joined #zuul | 20:43 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 20:48 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: Test new openshift version fix from nodepool https://review.opendev.org/c/zuul/zuul-operator/+/770158 | 20:48 |
| *** jamesmcarthur_ has quit IRC | 21:00 | |
| avass | corvus: yep it worked in my deployment for pods at least, but seeing as it was a problem with reading the config file it should work for namespaces as well | 21:03 |
| avass | and my deployment uses rbac | 21:03 |
| tristanC | avass: thanks for the test! | 21:07 |
| avass | np :) | 21:08 |
| *** jamesmcarthur has joined #zuul | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:16 | |
| *** jamesmcarthur has joined #zuul | 21:17 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 21:18 |
| *** jamesmcarthur has quit IRC | 21:20 | |
| *** jamesmcarthur has joined #zuul | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** ajitha has quit IRC | 21:47 | |
| *** hasharAway has quit IRC | 22:14 | |
| openstackgerrit | Merged zuul/nodepool master: Fix k8s driver when using in cluster config https://review.opendev.org/c/zuul/nodepool/+/771704 | 22:19 |
| *** hamalq has joined #zuul | 22:25 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add kubernetes namespace job test https://review.opendev.org/c/zuul/zuul-operator/+/771703 | 22:33 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Gerrit: Add SSH review tags https://review.opendev.org/c/zuul/zuul/+/771733 | 22:35 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Require diskimage-builder 3.6.0 https://review.opendev.org/c/zuul/nodepool/+/771734 | 22:37 |
| ianw | fungi: ^ | 22:37 |
| fungi | thanks! | 22:40 |
| *** cloudnull8 has joined #zuul | 22:44 | |
| *** cloudnull has quit IRC | 22:46 | |
| *** cloudnull8 is now known as cloudnull | 22:46 | |
| *** Tahvok has quit IRC | 23:14 | |
| *** Tahvok has joined #zuul | 23:14 | |
| *** harrymichal_ has joined #zuul | 23:16 | |
| *** harrymichal has quit IRC | 23:19 | |
| *** harrymichal_ is now known as harrymichal | 23:19 | |
| *** maxamillion has quit IRC | 23:25 | |
| *** jamesmcarthur has joined #zuul | 23:29 | |
| *** rlandy has quit IRC | 23:30 | |
| *** jamesmcarthur has quit IRC | 23:34 | |
| openstackgerrit | Merged zuul/nodepool master: Require diskimage-builder 3.6.0 https://review.opendev.org/c/zuul/nodepool/+/771734 | 23:39 |
| *** tosky has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!