| *** jamesmcarthur has quit IRC | 00:02 | |
| *** jamesmcarthur has joined #zuul | 00:14 | |
| *** jamesmcarthur has quit IRC | 00:56 | |
| *** jamesmcarthur has joined #zuul | 01:24 | |
| *** jamesmcarthur has quit IRC | 01:29 | |
| *** armstrongs has quit IRC | 02:35 | |
| *** jamesmcarthur has joined #zuul | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** swest has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #zuul | 03:01 | |
| *** swest has joined #zuul | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:21 | |
| *** jamesmcarthur has joined #zuul | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:29 | |
| *** bhavikdbavishi has joined #zuul | 03:44 | |
| *** jamesmcarthur has joined #zuul | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:14 | |
| *** jamesmcarthur has joined #zuul | 04:15 | |
| *** evrardjp has quit IRC | 05:36 | |
| *** evrardjp has joined #zuul | 05:36 | |
| *** bhavikdbavishi has quit IRC | 05:41 | |
| *** jamesmcarthur has quit IRC | 05:43 | |
| *** bhavikdbavishi has joined #zuul | 06:22 | |
| *** saneax has joined #zuul | 06:39 | |
| *** bhavikdbavishi has quit IRC | 07:25 | |
| *** dpawlik has joined #zuul | 07:44 | |
| *** jcapitao has joined #zuul | 08:08 | |
| *** avass has joined #zuul | 08:22 | |
| *** tosky has joined #zuul | 08:32 | |
| *** arxcruz|off is now known as arxcruz|rover | 08:37 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** saneax has quit IRC | 09:15 | |
| *** saneax has joined #zuul | 09:32 | |
| *** bhavikdbavishi has joined #zuul | 09:36 | |
| *** dpawlik has quit IRC | 10:11 | |
| *** dpawlik has joined #zuul | 10:13 | |
| *** avass has quit IRC | 10:42 | |
| *** sshnaidm is now known as sshnaidm|pto | 10:53 | |
| *** harrymichal has joined #zuul | 11:05 | |
| *** harrymichal has quit IRC | 11:09 | |
| *** harrymichal has joined #zuul | 11:09 | |
| *** harrymichal has quit IRC | 11:14 | |
| *** rlandy has joined #zuul | 11:44 | |
| *** evrardjp has quit IRC | 11:44 | |
| *** evrardjp has joined #zuul | 11:50 | |
| *** rfolco has joined #zuul | 11:58 | |
| *** jcapitao is now known as jcapitao_lunch | 12:03 | |
| *** evrardjp has quit IRC | 12:27 | |
| *** avass has joined #zuul | 12:37 | |
| *** evrardjp has joined #zuul | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:56 | |
| *** jcapitao_lunch is now known as jcapitao | 13:23 | |
| *** bhavikdbavishi has quit IRC | 13:39 | |
| * AJaeger updated zuul-website for OpenDev changes, please review https://review.opendev.org/714261 | 13:41 | |
| *** zxiiro has joined #zuul | 13:42 | |
| *** sgw has joined #zuul | 13:42 | |
| openstackgerrit | Merged x/pbrx master: Fix tox-py35 job https://review.opendev.org/700901 | 13:54 |
|---|---|---|
| *** jpena|lunch is now known as jpena | 13:59 | |
| *** avass has quit IRC | 14:41 | |
| *** evrardjp has quit IRC | 15:07 | |
| *** evrardjp has joined #zuul | 15:11 | |
| *** bhavikdbavishi has joined #zuul | 15:22 | |
| *** jamesmcarthur has joined #zuul | 15:43 | |
| tristanC | corvus: the stack at https://review.opendev.org/714165 adds a convenient integration test. That change belows already have >= 2+2 , would you mind if I +Workflow those changes? | 15:45 |
| tristanC | The changes below* | 15:45 |
| *** panda is now known as panda|off | 15:46 | |
| tristanC | The zuul-operator doesn't setup nodepool yet, i was planning on adding the launcher service by extending the integration test to use a container nodeset. Just waiting for feedback on the open changes. | 15:49 |
| corvus | tristanC: yes i think it's fine to +w those, i'll look at the rest asap | 15:49 |
| Shrews | corvus: is the nodepool zk+tls change ready for review now? | 15:51 |
| corvus | Shrews: i believe so | 15:52 |
| corvus | Shrews: also zuul | 15:52 |
| Shrews | k. will look at them both | 15:52 |
| corvus | Shrews: (i think at the end of my stack on zuul there's complete end-to-end usage with quick-start) | 15:52 |
| Shrews | *nod* | 15:52 |
| *** y2kenny has joined #zuul | 15:57 | |
| y2kenny | Hi, I have got more question for you experts. For the executor and scheduler images on DockerHub, how are they generated and what version are they? It seems to be updated very frequently but there's no tag associated with them. | 16:00 |
| clarkb | y2kenny: they are generated by zuul jobs that run whenever changes merge. This means every time we land a change those images get updated | 16:01 |
| y2kenny | ah ok. So it's the tip of the tip. | 16:01 |
| clarkb | ya I think we may push tagged versions for each zuul reelase too | 16:02 |
| clarkb | and that tag would correspond to the zuul version? | 16:02 |
| clarkb | mordred: corvus ^ is that the case | 16:02 |
| y2kenny | I think that would be a great idea | 16:02 |
| y2kenny | at least I can fix my production deployment at a specific tag | 16:03 |
| *** klindgren_ has joined #zuul | 16:03 | |
| *** klindgren has quit IRC | 16:03 | |
| mordred | clarkb: we do not push docker hub tags for zuul releases yet | 16:04 |
| clarkb | y2kenny: fwiw we run zuul from near tip and every change is reasonably well tested. This doesn't eliminate risk, but we do everything we can to make running zuul from trunk possible | 16:04 |
| mordred | there is an issue that needs to be solved before we can which is tied with the upcoming "require db" step | 16:05 |
| clarkb | mordred: we could rebuild the images right? | 16:05 |
| mordred | so - it's a thing we'd like to do but are not doing atm | 16:05 |
| clarkb | as an intermediate step | 16:05 |
| mordred | clarkb: we could - but then those images would not have gone through any testing | 16:05 |
| clarkb | ya, but risk there seems low particularly if we keep testing image builds in general | 16:05 |
| *** mattw4 has joined #zuul | 16:06 | |
| y2kenny | I am ok with running at tip. But as an admin supporting a system, it would be nice to have a fixed tag to fall back to if there are any accident at the tip | 16:06 |
| mordred | yeah. it's potentially worth discussing | 16:06 |
| mordred | y2kenny: yah. for sure | 16:06 |
| y2kenny | I am sure you guys have excellent CI practice ;) | 16:06 |
| corvus | we try to walk the walk :) | 16:08 |
| *** panda|off has quit IRC | 16:09 | |
| y2kenny | Another question... for the scheduler, is it fair to say that the conf and tenant files are the only "critical states"? As in, if I want to have failover mirror, it can bring things back up just the same as long as the conf and tenant files are shared? | 16:10 |
| tobiash | y2kenny: also the private keys of the repos | 16:10 |
| y2kenny | oh right, yes. | 16:10 |
| y2kenny | but other than those, I can replicate the schedler pretty much as if it's stateless? | 16:11 |
| * mnaser believes so | 16:11 | |
| tobiash | yes, except the current queue | 16:11 |
| y2kenny | right... I think can accept that small lost | 16:12 |
| y2kenny | I have been trying to disect the quick-start tutorial into a k8s cluster and I am trying to figure out how to map some of the pieces over | 16:13 |
| tobiash | the cost depends on the size :), it can be quite significane on opendev or our environment hence the work on scale out scheduler | 16:13 |
| mnaser | y2kenny: fyi there is a zuul-operator project and also helm charts | 16:13 |
| mnaser | https://opendev.org/zuul/zuul-helm | 16:13 |
| mnaser | https://opendev.org/zuul/zuul-operator | 16:13 |
| y2kenny | mnaser: I have seen those but I wanted to understand the zuul system as I go, that's why I am doing it the hard way. | 16:14 |
| mnaser | y2kenny: gotcha, the helm charts might be a good reference point to look at in terms of volume mounts, why things are staefulset vs deployments, etc :) | 16:14 |
| *** sshnaidm|pto has quit IRC | 16:15 | |
| tobiash | mnaser: ah you chose statefulset for the scheduler, interesting | 16:16 |
| mnaser | tobiash: statefulset with replica=1 to maintain a stable hostname i think was a goal | 16:16 |
| y2kenny | which brings me to another question. How do you debug the connectivity between the executor and the scheduler/gearman? | 16:16 |
| mnaser | good question.. i haven't had to :-P | 16:17 |
| y2kenny | is ssl required for the connection? | 16:17 |
| mnaser | it is not, but it is recommended | 16:17 |
| clarkb | secrets are passed over that connection | 16:17 |
| *** sshnaidm|pto has joined #zuul | 16:17 | |
| clarkb | strongly recommended for that reason | 16:17 |
| tobiash | mnaser: actually for the scheduler it doesn't matter the stable hostname is currently only required for executors. So for the scheduler it's just a matter of tast between statefulset and deployment with re-create strategy | 16:18 |
| y2kenny | so far I have been able to put zookeeper and nodepool in to my k8s cluster but keep the scheduler /gearman on a separate machine | 16:18 |
| y2kenny | when I try to start a separate executor in the cluster to connect to gearman, the log seems to show something stuck | 16:19 |
| tobiash | y2kenny: openssl s_client is quite useful to debug connectivity issues to gearman | 16:19 |
| tobiash | y2kenny: I guess you're running gearman from the scheduler? | 16:20 |
| y2kenny | tobiash: yes | 16:20 |
| y2kenny | I was able to netcat from the executor to gearman | 16:20 |
| tobiash | try openssl s_client with the ca and client cert | 16:20 |
| y2kenny | at least with the status | 16:20 |
| y2kenny | ok... I haven't turned on ssl | 16:21 |
| y2kenny | and I noticed for the quick-start tutorial the executor connects to gearman directly on the same host | 16:22 |
| y2kenny | um... actually.. now that I looked into it... seems like it's connected? | 16:23 |
| y2kenny | that's weird | 16:23 |
| *** klindgren has joined #zuul | 16:24 | |
| tobiash | y2kenny: the openssl command for debugging ssl connections to gearman can be found here: https://zuul-ci.org/docs/zuul/howtos/troubleshooting.html#gearman-jobs | 16:25 |
| *** klindgren_ has quit IRC | 16:25 | |
| y2kenny | tobiash: thanks | 16:26 |
| y2kenny | ok... so may be the problem lied else where (or may be there is a delay) | 16:26 |
| tobiash | y2kenny: generating the gearman certs is described here: https://opendev.org/zuul/zuul/src/branch/master/tests/fixtures/gearman | 16:26 |
| tobiash | (as an example) | 16:27 |
| tobiash | actually I don't find that part in the docs | 16:27 |
| y2kenny | what I am trying to do is to have a executor inside the cluster to talk to nodes that are only visible within the cluster. | 16:27 |
| y2kenny | Initially it didn't work because I forgot to set the zone in the executor config | 16:28 |
| y2kenny | but even after I set it, the nodes are still not reachable | 16:28 |
| y2kenny | in order to use zone, do I have to set it for all the executor? | 16:29 |
| y2kenny | i.e., for executor that's not configured with zone, it is actually any? | 16:29 |
| y2kenny | (so in order to exclude unreachable nodes from certain executor, all executor will have to have a zone setting?) | 16:31 |
| *** hashar has joined #zuul | 16:33 | |
| *** panda has joined #zuul | 16:47 | |
| *** panda is now known as panda|off | 16:48 | |
| *** rfolco has quit IRC | 16:51 | |
| *** rfolco has joined #zuul | 16:51 | |
| *** jcapitao has quit IRC | 17:07 | |
| Shrews | corvus: hrm, i wasn't aware that our zuul vs. nodepool zookeeper configurations had become so radically different. Did I imagine that these had started out as identical?? | 17:09 |
| corvus | Shrews: no -- one's in a conffile, the other is in a yaml, so structurally they have to be different | 17:09 |
| Shrews | i thought they at least had the same section names. i must've been drunk when i imagined that | 17:10 |
| corvus | i'm wondering if we should move some nodepool stuff into a conffile, to mirror what zuul is like -- conffile is service config, yaml is content. but nodepool has some more grey areas there than zuul | 17:11 |
| corvus | y2kenny: i think tobiash and pabelanger are the zone experts... but let me see if i can answer that | 17:11 |
| fungi | Shrews: my conffiles are best read in the same state of insobriety in which i write them | 17:12 |
| Shrews | fungi: you win Monday | 17:12 |
| tobiash | y2kenny: if you configure an executor for a zone it will process exclusivly jobs with nodes from that zone | 17:14 |
| corvus | and it looks like it will *not* run unzoned jobs | 17:15 |
| tobiash | correct | 17:15 |
| corvus | so i think it works the way i think y2kenny thought originally, and the answer to the question "in order to exclude unreachable nodes from certain executor, all executor will have to have a zone setting?" is no. | 17:15 |
| corvus | y2kenny: if it's not working as expected, you may want to double check that you set the zone attribute for the nodes in nodepool | 17:16 |
| *** jamesmcarthur has quit IRC | 17:18 | |
| *** jamesmcarthur has joined #zuul | 17:18 | |
| clarkb | why did https://review.opendev.org/#/c/714188/3 not build with a preview site artifact? | 17:19 |
| tobiash | related to that, I have a change up for review so a zoned executor can be optionally allowed to process unzoned jobs: https://review.opendev.org/673840 | 17:19 |
| clarkb | (also it would be great if a non OSF staff member could review that and hopefully get it landed so we can avoid having 3 OSFers get that in) | 17:19 |
| fungi | and same for https://review.opendev.org/713110 | 17:21 |
| corvus | clarkb: do we normally get preview site artifacts? | 17:22 |
| clarkb | I thought we did | 17:22 |
| fungi | i don't think we have so far for that repo | 17:22 |
| clarkb | but maybe I just always navigated the html dir instead | 17:23 |
| fungi | we'd need to use the zuul preview service to get them to look right anyway what with there being a number of absolute urls in there | 17:23 |
| corvus | maybe we used to have a success-url but then never updated it to an artifact? | 17:23 |
| clarkb | oh that could be | 17:23 |
| clarkb | https://80d06003c63fddf1b683-19b813b48e429f09de2f1c9eaa005f3d.ssl.cf2.rackcdn.com/714188/3/check/zuul-website-build/802c0bf/html/users.html is the page that was updated fwiw | 17:23 |
| corvus | fungi: oh, it used to all be relative, did we slack off? | 17:23 |
| fungi | corvus: possible it's only 713110 which needs fixing for that then | 17:24 |
| fungi | since it does <img src="/images/... | 17:24 |
| mordred | I pulled off my +A in case we want to do that | 17:25 |
| fungi | jamesmcarthur: ^ maybe we do want to drop the leading / on those | 17:25 |
| jamesmcarthur | fungi: corvus: np - can fix | 17:25 |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add supporters to Community page https://review.opendev.org/713110 | 17:27 |
| *** hashar has quit IRC | 17:30 | |
| openstackgerrit | Merged zuul/zuul-website master: Promoting Zuul User Survey https://review.opendev.org/714188 | 17:34 |
| clarkb | mordred: ^ jamesmcarthur has a new ps up if you want to ack it again | 17:34 |
| mordred | clarkb, jamesmcarthur: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_357/713110/3/check/zuul-website-build/3571afe/html/community.html | 17:35 |
| mordred | looks a little off ... | 17:35 |
| jamesmcarthur | hmm | 17:36 |
| *** evrardjp has quit IRC | 17:36 | |
| jamesmcarthur | something is off for sure | 17:36 |
| *** evrardjp has joined #zuul | 17:36 | |
| *** dpawlik has quit IRC | 17:37 | |
| jamesmcarthur | give me a few - working | 17:38 |
| AJaeger | here's another change for zuul-website - update for opendev https://review.opendev.org/714261 | 17:38 |
| *** bhavikdbavishi has quit IRC | 17:40 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Set default secret mode to 0400 https://review.opendev.org/714501 | 17:43 |
| Shrews | corvus: the tls changes lgtm except for release notes. not sure what's up with the quickstart job change failing on the image builds | 17:48 |
| *** y2kenny has quit IRC | 18:00 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add integration test playbook https://review.opendev.org/714165 | 18:00 |
| *** dpawlik has joined #zuul | 18:02 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Trim whitespace from uri password for docker promote https://review.opendev.org/714506 | 18:03 |
| *** y2kenny has joined #zuul | 18:06 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Strip by default in tools/encrypt_secret https://review.opendev.org/714508 | 18:07 |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add supporters to Community page https://review.opendev.org/713110 | 18:08 |
| y2kenny | corvus, tobiash: ok I will give it a try (sorry I was afk and went for lunch.) I understand that the executor run exclusively zoned job but would non-zoned executor run zoned job? (it looked that way when I tried it but I could be missing something.) | 18:10 |
| tobiash | y2kenny: if there is an executor existing that is configured for the zone no other executor will get the job | 18:11 |
| y2kenny | ok. | 18:11 |
| tobiash | if there is no such executor, all non-zoned executors will get the job | 18:12 |
| y2kenny | Is there a way to check the number and status of connected executors from the scheduler/gearman side? | 18:12 |
| y2kenny | (or... is it the scheduler or gearman that is "executor-aware"? ... or both?) | 18:13 |
| clarkb | y2kenny: I think both but the easiest way to check is via gearman | 18:15 |
| clarkb | y2kenny: if you connect to the port and enter 'status' no quotes and a return after that is the gearman protocol command to get back some info like that | 18:15 |
| y2kenny | ok cool. Thanks. | 18:15 |
| clarkb | it will show you 4 columns, registered job name, builds queued for this job, builds running for this job, and registered workers for this job | 18:16 |
| clarkb | y2kenny: each executor will register as a worker for the execute function iirc | 18:16 |
| clarkb | so you'd look on that line to see how many executors are attached | 18:16 |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add supporters to Community page https://review.opendev.org/713110 | 18:17 |
| y2kenny | clarkb: yes I see some zone info in there as well. I will go play with this a bit | 18:22 |
| *** jpena is now known as jpena|off | 18:27 | |
| *** panda|off has quit IRC | 18:50 | |
| *** panda has joined #zuul | 18:51 | |
| openstackgerrit | Jimmy McArthur proposed zuul/zuul-website master: Add supporters to Community page https://review.opendev.org/713110 | 19:00 |
| jamesmcarthur | at long last, I believe I have fixed this if mordred: or corvus: want to give it another look: https://30472a6d122e9d2c7f5a-eb75a7eda94fb791869997c22d701b46.ssl.cf5.rackcdn.com/713110/6/check/zuul-website-build/cf3eaec/html/community.html | 19:06 |
| *** jamesmcarthur has quit IRC | 19:07 | |
| *** saneax has quit IRC | 19:07 | |
| mnaser | cccccclcthbbnlicilnrivgbjugbhlhtiugnfkdrgvfv | 19:11 |
| mnaser | oops, yubikey | 19:11 |
| openstackgerrit | Merged zuul/zuul-operator master: Update to dhall lang v14 https://review.opendev.org/710649 | 19:11 |
| openstackgerrit | Merged zuul/zuul-operator master: Add missing volumes for the web and merger service https://review.opendev.org/712811 | 19:11 |
| mordred | mnaser: I now own all of yoru things | 19:11 |
| mnaser | mordred: let me know if you find something interesting | 19:12 |
| *** jamesmcarthur has joined #zuul | 19:13 | |
| *** mattw4 has quit IRC | 19:13 | |
| openstackgerrit | Merged zuul/zuul-website master: Add supporters to Community page https://review.opendev.org/713110 | 19:17 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 19:34 |
| *** hashar has joined #zuul | 19:37 | |
| *** jamesmcarthur has quit IRC | 19:46 | |
| *** jamesmcarthur has joined #zuul | 19:47 | |
| *** jamesmcarthur has quit IRC | 19:49 | |
| *** jamesmcarthur has joined #zuul | 19:50 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 19:56 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 20:05 |
| *** jamesmcarthur has quit IRC | 20:06 | |
| *** mattw4 has joined #zuul | 20:06 | |
| *** jamesmcarthur has joined #zuul | 20:07 | |
| *** jamesmcarthur has quit IRC | 20:12 | |
| openstackgerrit | Merged zuul/zuul-operator master: Add missing input defaults https://review.opendev.org/713138 | 20:17 |
| *** jamesmcarthur has joined #zuul | 20:27 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 20:28 |
| clarkb | mordred: on that trim change, would that remove all leading and trailing whitespace? | 20:31 |
| clarkb | that could potentially be problematic for some passwds? | 20:31 |
| clarkb | hrm I guess docker wants it base64 encoded though? so maybe in this specific acse it is ok? | 20:31 |
| mordred | clarkb: yeah - it's mostly to align it with how we're running the docker login command currently | 20:33 |
| mordred | clarkb: but - honestly, I think maybe expecting passwords with leading or trailing spaces to work is likely to be a painful life choice no matter what | 20:35 |
| clarkb | mordred: I agree, but it is apparently a thing (see also that thread re horizon's behavior around this) | 20:35 |
| clarkb | pwgen doesn't even have an option for incorporating whitespace | 20:36 |
| fungi | i think it's fine if the encrypt tool keeps an option to not strip leading and trailing whitespace | 20:36 |
| mordred | well - that would mean not doing the | trim patch | 20:37 |
| clarkb | which I just approved | 20:37 |
| clarkb | I'll unapprove I guess? | 20:37 |
| fungi | no, i'm talking about the earlier discussion in #opendev about improvements to the zuul_encrypt.py script | 20:37 |
| mordred | yeah. those exist too - I made two changes | 20:38 |
| mordred | one to update zuul_encrypt - and one to trim in the job | 20:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Strip by default in tools/encrypt_secret https://review.opendev.org/714508 | 20:42 |
| mordred | fungi, clarkb: ^^ | 20:42 |
| *** dpawlik has quit IRC | 20:43 | |
| *** dpawlik has joined #zuul | 20:53 | |
| mnaser | mordred: thanks to the hold the cool artifact is now we have a role to collect all k8s-y things | 21:02 |
| mnaser | which should come in handy (cc tristanC ^) | 21:02 |
| mordred | mnaser: woot | 21:03 |
| *** panda is now known as panda|off | 21:05 | |
| tristanC | mnaser: that's good, i hope we can improve namespace log collection as it's currently not very practical (e.g. in the `docker` dir we can have many empty files) | 21:08 |
| mnaser | tristanC: yeah the POD is pretty much empty but at least we get the container logs cleanly there | 21:08 |
| tristanC | btw, there are quite a few changes waiting for review in https://review.opendev.org/#/q/project:zuul/zuul-operator+status:open | 21:10 |
| *** rfolco has quit IRC | 21:17 | |
| *** jamesmcarthur has quit IRC | 21:19 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add job_volumes CR spec attribute https://review.opendev.org/706642 | 21:20 |
| *** dpawlik has quit IRC | 21:20 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update attributes to camelCase https://review.opendev.org/707193 | 21:21 |
| *** jamesmcarthur has joined #zuul | 21:22 | |
| *** jamesmcarthur_ has joined #zuul | 21:25 | |
| *** jamesmcarthur has quit IRC | 21:25 | |
| y2kenny | A question about nodeset: for each node in a nodeset, I am required to provide a name and a label. From the documentation, that name seems to need to match the nodes name provided by nodepool but in practice that doesn't seem to be the case. What am I missing around this nodes and labels? | 21:33 |
| y2kenny | Is my setup working because all my playbooks has hosts: all? | 21:34 |
| clarkb | y2kenny: the label represents a class of node that nodepool can provide. Each node provided for that label will have a unique name in the cloud provider. From the job perspective the inventory will be filled out with the name in the nodeset | 21:35 |
| fungi | hosts: all applies to any and all nodes besides "localhost" (the executor) | 21:35 |
| clarkb | y2kenny: that allows you to not worry about specifics and focus on the groupings you've expressed in the nodeset when writing the job | 21:35 |
| openstackgerrit | Merged zuul/zuul-jobs master: add role for collecting the kubernetes pod&kubelet logs https://review.opendev.org/714521 | 21:35 |
| y2kenny | clarkb: that make sense but how come nodeset.nodes.name is a required field? Should I be able to just define a nodeset with labels only? | 21:36 |
| *** hashar has quit IRC | 21:37 | |
| clarkb | y2kenny: no because they represent differing things. The label maps onto nodepools availalbe classes and the name is what you get in the ansible inventory | 21:37 |
| clarkb | y2kenny: this would allow you to say I want a host called controller of nodepool class ubuntu-xenial-extra-large | 21:38 |
| y2kenny | Ohhhh... | 21:38 |
| clarkb | then you can refer to the controller as the logical entity without worrying if it is based on ubuntu or centos or whatever | 21:38 |
| clarkb | if you do a lot of cross platform tstuff this becomes really useful | 21:38 |
| y2kenny | I am wondering because the nodes name seems to be ignored. What I've done is to launch 3 nodes container instead of having one node (from the quick start) and those extra nodes are still fully utilize. | 21:39 |
| y2kenny | so instead of having a node name ubuntu-bionic with label of the same, I have name node0, node1, node2 and label unchanged. | 21:40 |
| y2kenny | (this is the nodepool provider setting) | 21:40 |
| y2kenny | but in the base job nodeset, I am still using name=ubuntu-bionic, label=ubuntu-bionic | 21:41 |
| clarkb | y2kenny: if you look at the inventory file for the job those names should be used there | 21:41 |
| clarkb | that then allows you to refer to those nodes by name in the job content | 21:42 |
| y2kenny | um... I am not sure which inventory file you are referring to. Is it suppose to go under zuul.d/ or part of the playbook/ (I am familiar with inventory for ansible but not sure how that fit into zuul.) | 21:46 |
| *** armstrongs has joined #zuul | 21:46 | |
| clarkb | y2kenny: it is part of the running job, and zuul-jobs has a role to collect it iirc. Let me show you an example from one of our jobs | 21:46 |
| clarkb | y2kenny: https://zuul.opendev.org/t/zuul/build/c9194ef26dfa42199ac1c974b2e41efa/log/zuul-info/inventory.yaml | 21:47 |
| clarkb | y2kenny: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/log-inventory is the role to log it. | 21:48 |
| clarkb | its a file used by ansible to know what it can talk to | 21:48 |
| *** jamesmcarthur_ has quit IRC | 21:49 | |
| y2kenny | um... this is different from the inventory file that I was thinking of. (I was thinking of the one that's in .ini format with various groupings of hosts/nodes.) | 21:50 |
| *** jamesmcarthur has joined #zuul | 21:50 | |
| clarkb | y2kenny: ansible accepts them in different serialization formats | 21:51 |
| clarkb | y2kenny: they are functionally the same iirc | 21:51 |
| y2kenny | ok. | 21:51 |
| *** jamesmcarthur has quit IRC | 21:52 | |
| *** jamesmcarthur has joined #zuul | 21:53 | |
| y2kenny | but I am still confused (sorry about this.) Let me see if I can ask the question in a different way... | 21:53 |
| fungi | here's one with custom node names: https://zuul.opendev.org/t/openstack/build/2cf56378a5c542a78d3a048fb50348ec/log/zuul-info/inventory.yaml | 21:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Allow pull-from-registry to be used in base jobs https://review.opendev.org/714541 | 21:54 |
| fungi | mimicking a production deployment, and renaming the various nodes with the names they might have in reality | 21:55 |
| fungi | this way we're able to run production deployment playbooks in the context of a multi-node zuul job | 21:55 |
| *** jamesmcarthur has quit IRC | 21:55 | |
| fungi | and even exercise the hostname patterns we tell ansible to filter for | 21:55 |
| y2kenny | ok... yea... I haven't even think about the idea of a multi-node zuul job yet. | 21:56 |
| fungi | all happening speculatively on a proposed change | 21:56 |
| *** armstrongs has quit IRC | 21:56 | |
| y2kenny | I am still trying to understand how zuul map a job that requires node type X to a collection of node with type X | 21:57 |
| *** jamesmcarthur has joined #zuul | 21:57 | |
| mordred | y2kenny: when you put a label in a nodeset, you're telling zuul "please get me a node of type X" | 21:57 |
| y2kenny | so in my test setup, I have defined a nodeset with 1 node (named and labeled ubuntu-bionic) | 21:57 |
| clarkb | let me try "diagram" it | 21:57 |
| * mordred lets clarkb go | 21:58 | |
| *** jamesmcarthur has quit IRC | 21:59 | |
| fungi | y2kenny: that's a fairly typical way to do it with a generic job where the job doesn't care what the name of the node is, or where you don't have multiple nodes within the same build you need to differentiate betwen | 21:59 |
| *** jamesmcarthur has joined #zuul | 21:59 | |
| y2kenny | Ok | 21:59 |
| fungi | the ability to specify node names in a nodeset is additional flexibility for when those things really do matter | 22:00 |
| y2kenny | so it is correct to say that the node.name in a nodeset is not the same as node.name in a nodepool's pool? | 22:00 |
| y2kenny | like no relationship? | 22:00 |
| fungi | correct | 22:01 |
| mordred | that's right | 22:01 |
| y2kenny | OOOOOOOOOOOOOOOO KAY | 22:01 |
| clarkb | https://etherpad.openstack.org/p/WRGEflEDZb | 22:01 |
| y2kenny | lol | 22:01 |
| fungi | the nodeset is how you would define that mapping | 22:01 |
| clarkb | fwiw | 22:01 |
| *** jamesmcarthur has quit IRC | 22:02 | |
| y2kenny | I think I was confused about that for the longest time. Now I think I get you mean with node name and inventory | 22:02 |
| mordred | \o/ | 22:02 |
| mordred | yay! | 22:02 |
| y2kenny | well... may be not... let me check that etherpad thing | 22:02 |
| y2kenny | ok... I think I got it | 22:04 |
| y2kenny | I haven't used the inventory stuff yet and that's probably why I got confused... as in, all my playbook are just hosts: all | 22:05 |
| clarkb | ya it makes a lot more sense when you start doing multinode stuff where you run actions per subset of nodes | 22:05 |
| y2kenny | does this mean you can actually use zuul to drive a test cluster? | 22:06 |
| y2kenny | I guess that's how you guys do openstack integration test and what not. | 22:07 |
| y2kenny | ok... yea... so the node.name in the nodeset is for the playbooks totally unrelated to the node.name in the nodepool | 22:08 |
| *** jamesmcarthur has joined #zuul | 22:08 | |
| clarkb | yup | 22:08 |
| y2kenny | the nodeset.node.label is related to the nodepool.node.label | 22:08 |
| y2kenny | Ok | 22:08 |
| clarkb | on the nodepool side its the actual name in the cloud provider | 22:08 |
| clarkb | on the zuul side its a logical name so that jobs can refer to things less concretely | 22:08 |
| y2kenny | yes... I think I got it now | 22:09 |
| y2kenny | this multi-node testing thing may open up some possibilities | 22:09 |
| y2kenny | very interesting. | 22:09 |
| y2kenny | This is pretty unique right? I don't recall having the mindset of multi-node testing when I was doing Jenkins. As far as I can remember, it's always pick a node from an approriate pool and do a specified task on it (either test or build) | 22:11 |
| y2kenny | Just to make sure I really understand this, what I just describe is covered by the labels right? | 22:12 |
| fungi | we basically hacked it together in jenkins ourselves at one point | 22:12 |
| fungi | giving a jenkins slave a set of "subnodes" that we granted it root ssh access into | 22:13 |
| clarkb | (though jenkins had no idea about them itself) | 22:13 |
| fungi | right, jenkins only knew about the "primary" node | 22:13 |
| clarkb | and ya I think this is fairly unique | 22:13 |
| fungi | when we replaced jenkins with ansible, we no longer had a need for any node in a nodeset to be "special" in that way | 22:14 |
| y2kenny | yea... that sounds nasty. I don't think we ever needed multi-node testing but our reboot/driver install requirements means that we have some custom test master that does things that jenkins doesn't know about | 22:14 |
| y2kenny | I'd say the test master is pretty much like the executor... but hacky and custom :) | 22:15 |
| fungi | but yeah, zuul tells nodepool it needs three nodes (maybe two ubuntu-bionic and one centos-8) and then nodepool assigns them and they become the nodeset for the associated build, then the job can reference each of them independently from the executor | 22:15 |
| fungi | y2kenny: you're not far off. our zuul executors are basically our jenkins master replacement. zuul v2 was a multi-jenkins-master aggregator | 22:16 |
| fungi | we had a jenkins plugin which gave jenkins masters the ability to communicate with the zuul scheduler over gearman protocol | 22:18 |
| mordred | (and also we had some users who had similar use caess of having a magical machien that knew about other nodes and did things there without our jenkinses knowing about it -w hich always felt fragile) | 22:18 |
| y2kenny | mordred: yes, totally agree with that. | 22:18 |
| y2kenny | ok. I think I will table this multi-node thing and revisit it down the road. This is great, thanks guys! I learned something today. | 22:21 |
| mordred | \o/ | 22:24 |
| corvus | y2kenny: if it helps you visualize it (i'm a visual person) -- here's a job that uses 3 hosts, each of which represents some part of our production control plane. there's a bastion ("bridge.opendev.org"), a load balancer ('gitea-lb01.opendev.org') and a web service ('gitea99.opendev.org'): https://zuul.opendev.org/t/openstack/build/e0395ebddaec4edb9b87643fbaaf5ecb/console | 22:39 |
| y2kenny | um... how come you guys have such a nice pretty console thing... I only have the Summary tab for my build. | 22:43 |
| y2kenny | thanks for the example though. That helps. | 22:43 |
| clarkb | y2kenny: if you collect the job-output.json file: https://zuul.opendev.org/t/zuul/build/c9194ef26dfa42199ac1c974b2e41efa/log/job-output.json that is what the UI ses to render the console tab | 22:44 |
| *** persia has quit IRC | 22:46 | |
| y2kenny | ok... I don't think I push the job-output.json to the web interface yet. Only dumping it into the log folder as per the quick start tutorial (haven't gone around to customize that bit just yet.) | 22:47 |
| clarkb | you might need to have it in the manifest too? | 22:47 |
| *** persia_ has joined #zuul | 22:47 | |
| corvus | oh, we might need a quickstart update for that | 22:48 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/generate-zuul-manifest | 22:48 |
| corvus | yep, i'll update qs | 22:48 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Be explicit about base container image https://review.opendev.org/714549 | 22:49 |
| y2kenny | This multi-node thing is pretty good... now I can probably use zuul to create a test for my CI infrastructure. Until now, all I've been doing is build a CI infra to test the product. We have minimal check on infrastructure config change but this will probably enable a more elaborate infrastructure test. | 22:51 |
| *** jamesmcarthur has quit IRC | 22:52 | |
| corvus | y2kenny: yes, that combined with zuul's speculative execution of its own config (where you can make a change to a zuul job and zuul will run with that change when it tests it before it lands) is very powerful | 22:52 |
| mordred | y2kenny: that was always a hole for us too - and it was always a bit terrifying to roll out some changes to the CI infrastructure - because there's nothing like preaching everything has to be tested - and then not having the testing system itself be tested and you break something :) | 22:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add generate-zuul-manifest to quick-start https://review.opendev.org/714551 | 22:55 |
| corvus | y2kenny: ^ there we go, i think that's the missing piece | 22:56 |
| y2kenny | mordred: yup, that's exactly it. | 22:57 |
| y2kenny | corvus: cool, I will add it and give it a try. I like pretty things. | 22:57 |
| corvus | it's pretty much the only way i look at logs now :) | 22:58 |
| fungi | it's certainly the first way i look at logs at the very least | 22:58 |
| fungi | (or sort of second if you count glancing at the abbreviated copy of the console log on the summary page, possibly third if i also looked at the console stream while the build was in progress!) | 22:59 |
| *** jamesmcarthur has joined #zuul | 23:04 | |
| openstackgerrit | Merged zuul/zuul master: Add generate-zuul-manifest to quick-start https://review.opendev.org/714551 | 23:47 |
| *** jamesmcarthur has quit IRC | 23:48 | |
| *** jamesmcarthur has joined #zuul | 23:52 | |
| *** jamesmcarthur has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!