| @kuprince:matrix.org | hi guys. I want to edit start-message for zuul pipeline. I added status_url in zuul.conf. but it is not working. I also added start-message: {{status_url}} in pipeline, but it is showing error - unhashable type: 'list'. Can someone tell me how can I do it? | 07:34 |
|---|---|---|
| @ecsantos:matrix.org | Prince Kumar: Hello, could you please upload your pipeline configuration file to https://paste.opendev.org/ so that I can take a look? | 07:42 |
| @ecsantos:matrix.org | And also, `[pipeline.start-message](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.start-message)` is not the same as `[web.status_url](https://zuul-ci.org/docs/zuul/latest/configuration.html#attr-web.status_url)` | 07:44 |
| @ecsantos:matrix.org | As a start, I'd try to use {status_url} instead of {{status_url}}, following the example from the docs | 07:46 |
| @kuprince:matrix.org | And can you guys please tell me what I need to do to make multi-node setup work? I have one switch node, and two peers nodes. I want to use switch node and peer node together to run my job. It should use one instance of switch node and one instance of peer node. | 07:55 |
| @kuprince:matrix.org | Right now, it is using one instance of both peer nodes, which I don't want it to use | 07:55 |
| @kuprince:matrix.org | I have a static nodepool defined with both peer nodes under one label. | 07:55 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] 826878: Cache serialized tenant status https://review.opendev.org/c/zuul/zuul/+/826878 | 08:09 | |
| @kuprince:matrix.org | > <@ecsantos:matrix.org> Prince Kumar: Hello, could you please upload your pipeline configuration file to https://paste.opendev.org/ so that I can take a look? | 08:28 |
| this is not working. | ||
| @kuprince:matrix.org | > <@ecsantos:matrix.org> And also, `[pipeline.start-message](https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.start-message)` is not the same as `[web.status_url](https://zuul-ci.org/docs/zuul/latest/configuration.html#attr-web.status_url)` | 08:29 |
| this is not working. I pasted the pipeline config on the given url | ||
| @ecsantos:matrix.org | Prince Kumar: You can also try Pastebin if you want | 08:31 |
| @kuprince:matrix.org | I pasted the config there on https://paste.opendev.org | 08:33 |
| @ecsantos:matrix.org | Prince Kumar: Oh, okay, I think I found it, it's this one [1], right? If so, I'd suggest trying {status_url} instead of {{status_url}}, with only one set of curly braces, as per the documentation [2] | 08:38 |
| [1] https://paste.opendev.org/show/812441/ | ||
| [2] https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.start-message | ||
| @kuprince:matrix.org | > <@ecsantos:matrix.org> Prince Kumar: Oh, okay, I think I found it, it's this one [1], right? If so, I'd suggest trying {status_url} instead of {{status_url}}, with only one set of curly braces, as per the documentation [2] | 08:50 |
| > | ||
| > [1] https://paste.opendev.org/show/812441/ | ||
| > [2] https://zuul-ci.org/docs/zuul/latest/config/pipeline.html#attr-pipeline.start-message | ||
| but where do we need to define status_url? | ||
| @kuprince:matrix.org | do we need to provide any value to it? | 08:50 |
| @ecsantos:matrix.org | Prince Kumar: The status_url variable is taken from the web.status_url option defined in your zuul.conf file [1] | 08:53 |
| [1] https://zuul-ci.org/docs/zuul/latest/configuration.html#attr-web.status_url | ||
| @kuprince:matrix.org | okay | 09:25 |
| @kuprince:matrix.org | Thanks | 09:25 |
| @kuprince:matrix.org | ecsantos: Can you please tell me about multi-node setup. I already configured a static nodepool with one switch node and two peer nodes. I want to run a job using one instance of switch node and one instance of peer node. | 09:29 |
| @kuprince:matrix.org | > <@ecsantos:matrix.org> Prince Kumar: The status_url variable is taken from the web.status_url option defined in your zuul.conf file [1] | 09:44 |
| > | ||
| > [1] https://zuul-ci.org/docs/zuul/latest/configuration.html#attr-web.status_url | ||
| I am getting one error when I made the changes. I can see the updated start-message with status_url, but getting error while loading tenant. | ||
| @kuprince:matrix.org | https://paste.opendev.org/show/812447/ | 09:44 |
| @kuprince:matrix.org | * ecsantos: https://paste.opendev.org/show/812447/ | 09:45 |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 827032: Install OVS from RDO Train Testing repository for CS8 https://review.opendev.org/c/zuul/zuul-jobs/+/827032 | 10:12 | |
| @ecsantos:matrix.org | Prince Kumar: Sorry, I don't understand what you mean by peer node and switch node. Could you elaborate further? | 12:31 |
| -@gerrit:opendev.org- daniel.pawlik https://matrix.to/#/@dpawlik:matrix.org proposed: [zuul/zuul-jobs] 827067: Change RDO train repository for Centos 8 stream https://review.opendev.org/c/zuul/zuul-jobs/+/827067 | 12:50 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Alfredo Moralejo: [zuul/zuul-jobs] 827032: Install OVS from RDO Train Testing repository for CS8 https://review.opendev.org/c/zuul/zuul-jobs/+/827032 | 13:19 | |
| @kuprince:matrix.org | > <@ecsantos:matrix.org> Prince Kumar: Sorry, I don't understand what you mean by peer node and switch node. Could you elaborate further? | 14:11 |
| switch node means server node, on which zuul is running, and peer node means worker nodes, where we want to run our jobs. | ||
| @ecsantos:matrix.org | Prince Kumar: Oh, okay, I think I got it. What kind of jobs do you want to run on the Zuul host? What do these jobs would execute? | 14:14 |
| @kuprince:matrix.org | these will execute few tests on cloud clusters. | 14:20 |
| @kuprince:matrix.org | > <@ecsantos:matrix.org> Prince Kumar: Oh, okay, I think I got it. What kind of jobs do you want to run on the Zuul host? What do these jobs would execute? | 14:20 |
| and will need to upload logs at the end | ||
| @ecsantos:matrix.org | Prince Kumar: I believe the usual approach is to run any sorts of tests on worker nodes, either virtual machines provided by an OpenStack cloud, AWS or GCE, or statically defined nodes (which I understand is your case). Jobs running on the same host that Zuul is running are very limited and lack isolation | 14:29 |
| @ecsantos:matrix.org | For example, the openstack-tox-pep8 job [1] from OpenDev tests a Git repository for compliance with PEP8, and is run on a worker node (an Ubuntu Focal virtual machine) | 14:31 |
| [1] https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L25-L72 | ||
| @ecsantos:matrix.org | Oops, I meant [1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L322-L332 | 14:32 |
| @ecsantos:matrix.org | While OpenDev's base job [2] employs a post playbook [3] that actually runs on the Zuul host (the Zuul Executor) | 14:36 |
| [2] https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L25-L72 | ||
| [3] https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L11-L28 | ||
| @ecsantos:matrix.org | The post playbooks of OpenDev's base job is a good starting point if your jobs need to upload logs. There are other Ansible roles to upload logs to other services [4][5][6][7] other than to Swift | 14:39 |
| [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs | ||
| [5] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-azure | ||
| [6] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-gcs | ||
| [7] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-s3 | ||
| -@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul-jobs] 827113: Add cargo-test job https://review.opendev.org/c/zuul/zuul-jobs/+/827113 | 14:44 | |
| @kuprince:matrix.org | > <@ecsantos:matrix.org> While OpenDev's base job [2] employs a post playbook [3] that actually runs on the Zuul host (the Zuul Executor) | 14:58 |
| > | ||
| > [2] https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L25-L72 | ||
| > [3] https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L11-L28 | ||
| I am able to run all jobs without any issue, and also able to upload-logs correctly. But the only issue I am having is that if I am assigning multiple nodes to a job, it is consuming one instance of each node and running the playbooks on all nodes. I only want it to run the playbooks on single node instance. | ||
| @kuprince:matrix.org | suppose, I have 3 nodes - 2 worker nodes and 1 server node. In playbooks, some tasks will run on worker node, and some tasks wil run on server node. Now, how will it choose which worker node to use to execute those tasks? I don't want to run all the tasks on both worker nodes. | 15:00 |
| @ecsantos:matrix.org | Prince Kumar: Ooh, okay, now I got you. If I'm not mistaken, the nodeset configuration item [1] is what you're looking for. For example, in the openstack/devstack project, they have defined the openstack-two-node-focal nodeset [2]. Jobs that consume this nodeset (I didn't find any good examples) can set in their playbooks that an arbitrary play should run on the compute1 node rather than in both controller and compute1 | 15:12 |
| [1] https://zuul-ci.org/docs/zuul/latest/config/nodeset.html | ||
| [2] https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L149-L177 | ||
| @ecsantos:matrix.org | Additionally, if you want to pass different job variables to different nodes in a multinode job, you can make use of the nodeset.groups [3] and job.group-vars [4] options | 15:15 |
| [3] https://zuul-ci.org/docs/zuul/latest/config/nodeset.html#attr-nodeset.groups | ||
| [4] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.group-vars | ||
| -@gerrit:opendev.org- Tristan Cacqueray proposed: | 15:38 | |
| - [zuul/zuul-jobs] 827113: Add cargo-test job https://review.opendev.org/c/zuul/zuul-jobs/+/827113 | ||
| - [zuul/zuul-jobs] 827130: Add packages support to ensure-cargo https://review.opendev.org/c/zuul/zuul-jobs/+/827130 | ||
| - [zuul/zuul-jobs] 827131: Update cargo-test job to support system packages https://review.opendev.org/c/zuul/zuul-jobs/+/827131 | ||
| @clarkb:matrix.org | Prince Kumar: yup ecsantos has got it. YOu address the specific nodes you want by name or group in your playbooks. The nodeset defines the node names and group memberships then you operate on specific subsets in ansible using those identifiers | 16:02 |
| @kuprince:matrix.org | Okay. Thanks ecsantos Clark | 16:09 |
| -@gerrit:opendev.org- Tristan Cacqueray proposed: | 17:53 | |
| - [zuul/zuul-jobs] 827113: Add cargo-test job https://review.opendev.org/c/zuul/zuul-jobs/+/827113 | ||
| - [zuul/zuul-jobs] 827130: Add packages support to ensure-cargo https://review.opendev.org/c/zuul/zuul-jobs/+/827130 | ||
| - [zuul/zuul-jobs] 827131: Update cargo-test job to support system packages https://review.opendev.org/c/zuul/zuul-jobs/+/827131 | ||
| @kuprince:matrix.org | Clark: ecsantos right now, I converted my playbooks so that it will use only one node to run all jobs. Now I defined all nodes under single label. What should I provide to job definition so that it will choose one node out of all nodes that we have in our nodeset? | 18:35 |
| @clarkb:matrix.org | Prince Kumar: you can put the nodes in different groups and then refer to that group in your playbook hosts: entries | 18:36 |
| @clarkb:matrix.org | Prince Kumar: as an alternative you can use the exact hostname in the nodeset | 18:36 |
| @kuprince:matrix.org | Clark: I want launcher to choose the nodes. I should not choose one. | 18:37 |
| @kuprince:matrix.org | can I show you my job definition and nodepool.yaml somehow? | 18:37 |
| @clarkb:matrix.org | yes, there are two layers here the nodepoo layer and the zuul layer. You let the nodepool layer select the node randomly out of what is available. Then you are spceific when you run the job | 18:38 |
| @clarkb:matrix.org | * yes, there are two layers here the nodepool layer and the zuul layer. You let the nodepool layer select the node randomly out of what is available. Then you are specific when you run the job | 18:38 |
| @kuprince:matrix.org | Can you explain a little more about the selection process in zuul layer? | 18:39 |
| @clarkb:matrix.org | Prince Kumar: in your jobs you say I want a nodeset from nodepool. Nodepool will semi randomly provide nodes that meet those criteria to the job. Then in your job you should explicitly say via the ansible hosts: directive which tasks run on which nodes | 18:42 |
| @clarkb:matrix.org | each job should only request the exact resources it needs via its nodeset | 18:42 |
| @clarkb:matrix.org | then rely on nodepool to do the resource balancing and allocation for all the jobs | 18:42 |
| @fungicide:matrix.org | you apply "labels" to resources in your nodepool configuration in order to classify them, then in zuul you create a nodeset which maps a set of classifications to names you want those nodes to have in your jobs (what "host" is specified in a play). zuul knows, based on the nodeset a job specifies, what labels to ask nodepool to provide representatives from | 18:43 |
| @kuprince:matrix.org | I understand how it is working between nodepool and zuul, but I am still not clear about how should I implement this. If somehow I can share my config with you, and you can tell what I need to change, that will be great. | 18:45 |
| @kuprince:matrix.org | Clark: It will be of great help to me. | 18:45 |
| @clarkb:matrix.org | Prince Kumar: maybe it will help if we give you an example. Let me work on that | 18:47 |
| @fungicide:matrix.org | it's probably good to step back and consider the overal design of your test jobs. you said you want three nodes in a job? how are those nodes differentiated, and where do they come from? do they have different configuration, different operating systems, different contents on their filesystems? | 18:47 |
| @fungicide:matrix.org | the answers to those questions will inform the most effective ways to label the resources you have, and to be able to map them to resources used within a specific job/build | 18:48 |
| @kuprince:matrix.org | https://paste.opendev.org/show/812453/ | 18:49 |
| @kuprince:matrix.org | Clark: fungi can you check the nodepool config once? | 18:49 |
| @kuprince:matrix.org | I have few tasks that I need to run, but need to run them using only one node. That one node, we need to choose from 3 nodesm that we have. | 18:50 |
| @fungicide:matrix.org | in a zuul deployment i help maintain, most of our test resources are identical except that they differ by what operating system is installed on them, so we label them by operating system and then in our jobs we specify how many nodes with which operating systems we need. but in other cases you might label them by specific hardware they have available (gpu vs non-gpu) or how much memory is on them if your jobs need different nodes with different amounts of ram | 18:50 |
| @kuprince:matrix.org | so right now, I have 2 VMs on which max-parallel-jobs it can run is 3. That means we have 3 instances of both VMs. Now I want to select one instance out of 6 instances, and run all tasks on that instance only. | 18:52 |
| @fungicide:matrix.org | Prince Kumar: i guess i was confused earlier when you said you wanted jobs with three nodes. if you want jobs to use only one node when they run, then your nodeset for them should contain only one node. give all three of your available servers the same label, and then when zuul makes a node request for a build, nodepool will assign an available one with the requested label from the nodeset | 18:52 |
| @jim:acmegating.com | don't use max-parallel-jobs unless you are very familiar with how the base jobs set up repos | 18:54 |
| @kuprince:matrix.org | fungi: so you are telling me that I should define only one node under nodeset for that particular job. Once all instances of that node get consumed, it will ask for other node instances with same label? Am I understanding it correctly? | 18:54 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/tNeRZ1rIH8eW4kFLsP6m | 18:55 |
| @clarkb:matrix.org | there is an example | 18:55 |
| @jim:acmegating.com | they also make this conversation very confusing if you're just learning how label selection works | 18:55 |
| @kuprince:matrix.org | > <@jim:acmegating.com> don't use max-parallel-jobs unless you are very familiar with how the base jobs set up repos | 18:55 |
| What does it mean? If we don't use this, how many jobs it will run on that particular node? | ||
| @jim:acmegating.com | just put howevery many vms you have in nodepool with the same label, then request one node of that label in zuul. | 18:55 |
| @fungicide:matrix.org | Prince Kumar: yes, once the build completes, zuul will let nodepool know it is done with the node which was assigned, and then nodepool will be able to assign it to a new request which may already be waiting for it | 18:55 |
| @jim:acmegating.com | > <@kuprince:matrix.org> What does it mean? If we don't use this, how many jobs it will run on that particular node? | 18:55 |
| 1 | ||
| @clarkb:matrix.org | basically what that shows is nodepool can provide two types of nodes that are designated by two different labels. Then there are three nodesets. One for a single node of the small label. One for a single node of the large label and one for two nodes one of each. The job then only needs a single node so it uses the single node nodeset and nodepool will randomly assign an available single node to that job | 18:56 |
| -@gerrit:opendev.org- Tristan Cacqueray proposed: | 18:56 | |
| - [zuul/zuul-jobs] 827130: Add packages support to ensure-cargo https://review.opendev.org/c/zuul/zuul-jobs/+/827130 | ||
| - [zuul/zuul-jobs] 827131: Update cargo-test job to support system packages https://review.opendev.org/c/zuul/zuul-jobs/+/827131 | ||
| @kuprince:matrix.org | Clark: So I need to assign ansible host: label to the node? | 18:57 |
| @clarkb:matrix.org | Prince Kumar: in this case ansible host will refer to the name or group in the nodeset | 18:57 |
| @fungicide:matrix.org | a nodeset is what assigns an ansible host: label to whatever node is provided by nodepool to fulfil that build's node request | 18:58 |
| @clarkb:matrix.org | so small/large/ubuntu in my example | 18:58 |
| @clarkb:matrix.org | let me add an example playbook to the etherpad | 18:58 |
| @fungicide:matrix.org | it seems like the fundamental challenge you have though, is that you want to parallelize the use of your servers so that you can run more jobs simultaneously than you have servers available to run them on. usually you'd do that with some separate isolation and resource management like openstack nova to subdivide them into virtual machines on demand, or kubernetes to provide separate containers on them, and then have nodepool treat the subdivided resources as individual nodes | 19:01 |
| @kuprince:matrix.org | I think I understand what you are trying to tell me Clark . That I am already using. But the problem I am having is different. | 19:01 |
| @kuprince:matrix.org | > <@fungicide:matrix.org> it seems like the fundamental challenge you have though, is that you want to parallelize the use of your servers so that you can run more jobs simultaneously than you have servers available to run them on. usually you'd do that with some separate isolation and resource management like openstack nova to subdivide them into virtual machines on demand, or kubernetes to provide separate containers on them, and then have nodepool treat the subdivided resources as individual nodes | 19:01 |
| yes. That is exactly what I am looking for. | ||
| @kuprince:matrix.org | Clark: Thanks for the ansible playbooks example, but I am already using them. | 19:03 |
| @fungicide:matrix.org | zuul/nodepool doesn't really directly solve that problem. but i think some folks have worked around it by setting up different ssh credentials going into different home directories on the server and listing each of those as separate nodes in nodepool. however you'd want to make sure your jobs aren't doing things with root access or anything similar which might conflict between the concurrent builds on them | 19:06 |
| @kuprince:matrix.org | That is already done by me. I had everything setup accordingly with ssh in place at all nodes. | 19:12 |
| @kuprince:matrix.org | fungi: These nodes are actually VMs, on which we want to run multiple jobs from zuul. | 19:13 |
| @fungicide:matrix.org | so each server has 3 different accounts on it, and you list the ssh connection info for each account as a different static node in nodepool? | 19:13 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 827167: Add release note for 5.0.0 https://review.opendev.org/c/zuul/nodepool/+/827167 | 19:14 | |
| @jim:acmegating.com | zuul-maint: ^ would you please review 827167 with high priority? | 19:15 |
| @kuprince:matrix.org | no. I have 3 VMs, on which I set max-parallel-jobs: 3. I have ssh connection info setup between these 3 VMs | 19:15 |
| @kuprince:matrix.org | so in zuul-web, I can see that I have a total of 9 nodes. | 19:16 |
| @fungicide:matrix.org | i'm probably still confused. you're saying you want to run 3 jobs at the same time on each vm (9 at the same time total)? | 19:17 |
| @kuprince:matrix.org | Now I want to select one node out of 9 nodes. all nodes are having same label | 19:17 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 827167: Add release note for 5.0.0 https://review.opendev.org/c/zuul/nodepool/+/827167 | 19:18 | |
| @kuprince:matrix.org | No. I want to run 1 job only. But it should be running on 1 node out of 9 nodes, that we have | 19:18 |
| @fungicide:matrix.org | and you've created 3 different users on each vm, and the ssh credentials for each is a separate node in your nodepool configuraiton? in that case just drop the max-parallel-jobs setting | 19:18 |
| @kuprince:matrix.org | then how will it work | 19:18 |
| @jim:acmegating.com | fungi: tristanC ianw tobiash Clark please take a look at https://review.opendev.org/827167 ASAP :) | 19:19 |
| @clarkb:matrix.org | corvus: should it say nearly identical to 4.12.0? | 19:19 |
| @clarkb:matrix.org | oh thats nodepool | 19:19 |
| @clarkb:matrix.org | I get it now :) | 19:19 |
| @jim:acmegating.com | Clark: heh, yeah i did get the # wrong the first time, but i think 4.4.0 is right this time :) | 19:20 |
| @fungicide:matrix.org | Prince Kumar: if you define 9 nodes in nodepool, three of which are different accounts on each of three virtual machines, then nodepool will assign them individually | 19:20 |
| @fungicide:matrix.org | from nodepool's perspective they're 9 different nodes, it's just that each trio shares a common ip address but with a different account name to log into | 19:21 |
| @kuprince:matrix.org | I defined 3 nodes in nodepool. but I also have max-parallel-jos: 3. Which gives us (3 * 3) nodes | 19:21 |
| @jim:acmegating.com | zuul-maint: thanks, i approved that so it can merge and i can go ahead and prep the tags | 19:22 |
| @kuprince:matrix.org | > <@fungicide:matrix.org> from nodepool's perspective they're 9 different nodes, it's just that each trio shares a common ip address but with a different account name to log into | 19:22 |
| yes. you can say that | ||
| @fungicide:matrix.org | give the same label to all 9 of those "nodes" and then in your job definition, use a nodeset with just one node in it. zuul will be able to get up to 9 nodes assigned from nodepool for 9 different builds at the same time | 19:23 |
| @fungicide:matrix.org | up to three of which might run on the same virtual machine | 19:24 |
| @kuprince:matrix.org | okay. That I am doing right now. | 19:24 |
| @kuprince:matrix.org | It seems that it is able to run it on other node, but it failed immediately. | 19:25 |
| @fungicide:matrix.org | in that case your problem is almost certainly the max-parallel-jobs setting. it probably tried to reuse the same node instead of assigning a different one | 19:26 |
| @kuprince:matrix.org | So when build is starting, It is cloning the projects on executor container, but not on worker node, even though we are running the playbooks on worker node. so I am synchronizing these two (worker and executor). Because of this it is unable to find the new build directory on other node. So it is failing. | 19:27 |
| @fungicide:matrix.org | you've already set up your parallelism faking three nodes with different accounts on each vm | 19:27 |
| @fungicide:matrix.org | the build directory on the node will be relative to the homedir of the distinct account you've told it to log into | 19:28 |
| @kuprince:matrix.org | If I don't synchronize between worker node and executor container, it will throw - FileNotFoundError | 19:32 |
| @fungicide:matrix.org | Prince Kumar: when you say you're "synchronizing" them, do you mean you're using one of the standard roles like https://www.zuul-ci.org/docs/zuul-jobs/general-roles.html#role-mirror-workspace-git-repos or https://www.zuul-ci.org/docs/zuul-jobs/general-roles.html#role-prepare-workspace ? | 19:33 |
| @kuprince:matrix.org | no. I was using ansible synchronize. But I think [2] is actually good. | 19:36 |
| @kuprince:matrix.org | fungi: How should we define this role? Any particular definition for this? | 19:38 |
| @fungicide:matrix.org | Prince Kumar: typically you'd add it to your base job definition | 19:55 |
| @kuprince:matrix.org | It is not working as we are expecting this. It is not synchronizing the complete build directory. Only src directory under zuul.executor.work_root | 19:56 |
| @fungicide:matrix.org | https://opendev.org/zuul/zuul-base-jobs/src/commit/a0739122f716eddf3edb66e3ffa48a18a4f141a8/playbooks/base/pre.yaml#L4 | 19:57 |
| @jim:acmegating.com | this is why i advised against using max-parallel-jobs | 19:57 |
| @fungicide:matrix.org | that's an example of how we do it | 19:57 |
| @fungicide:matrix.org | Prince Kumar: oh, you have other things you want copied to the nodes besides source code? typically you'd instrument that as part of your job definition as well, making sure you're copying exclusively to the remote workspace | 19:58 |
| @fungicide:matrix.org | each of the three accounts on the vm would have separate homedirs, so the workspaces between them shouldn't conflict in theory. you'll of course have to be extra careful none of the things jobs do write to shared locations on the nodes outside thosedistinct homedirs. that's why i say zuul/nodepool doesn't really solve this problem, most people use some separate resource provider layer like a cloud | 20:00 |
| @kuprince:matrix.org | I am having issue with my nodepool right now. | 20:02 |
| @kuprince:matrix.org | hosts: | 20:02 |
| 10.40.55.231: | ||
| ansible_connection: ssh | ||
| ansible_host: 10.40.55.215 | ||
| ansible_port: 22 | ||
| ansible_python_interpreter: auto | ||
| ansible_ssh_common_args: -o StrictHostKeyChecking=false | ||
| ansible_user: root | ||
| nodepool: | ||
| az: null | ||
| cloud: null | ||
| external_id: 10.40.55.215 | ||
| host_id: null | ||
| interface_ip: 10.40.55.215 | ||
| label: worker | ||
| private_ipv4: null | ||
| private_ipv6: null | ||
| provider: static-rack | ||
| public_ipv4: 10.40.55.215 | ||
| public_ipv6: null | ||
| region: null | ||
| @fungicide:matrix.org | using teh static node provider, you really miss out on most of the benefits of nodepool, because it's up to you to handle the lifecycle management (provisioning, contention, cleanup) | 20:02 |
| @kuprince:matrix.org | This is my inventory file from latest build. and it seems it is mapping the host with other node ip address | 20:03 |
| @kuprince:matrix.org | > <@fungicide:matrix.org> using teh static node provider, you really miss out on most of the benefits of nodepool, because it's up to you to handle the lifecycle management (provisioning, contention, cleanup) | 20:03 |
| okay | ||
| @fungicide:matrix.org | yeah, ``ansible_user: root`` is going to be a problem. you'd need some other account (three different accounts, one for each node which is sharing that vm) | 20:03 |
| @kuprince:matrix.org | This host is a single VM right now | 20:05 |
| @fungicide:matrix.org | did you put an ip address in your nodeset definition or something? that definitely looks strange | 20:08 |
| @kuprince:matrix.org | Yes. I defined the nodes with ip as their name | 20:09 |
| @fungicide:matrix.org | in the nodeset definition for zuul even? if so, that's not how it's intended to work. you're supposed to be giving it a name that the job's fabricated inventory will end up referring to it by (could even be the same as the node label you're using there, "worker") | 20:12 |
| @fungicide:matrix.org | your nodeset would just map the node name "worker" to the label "worker" and then your jobs can refer to ``host: worker`` that way | 20:12 |
| @kuprince:matrix.org | okay. But it seems to be working fine right now | 20:16 |
| @fungicide:matrix.org | you just said "i am having issue with my nodepool right now" so i'm not sure how to reconcile that with "it seems to be working fine right now" | 20:17 |
| @fungicide:matrix.org | anyway, an example of some simple nodesets where the label name is reused as the node name can be found here: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/nodesets.yaml | 20:18 |
| @kuprince:matrix.org | As we are only assigning one node in the job, and lettling launcher choose another node, in case first node instances are all consumed, so it is showing host name same as the node we defined for job, but actually choosing other node attributes and running job on other nodes. | 20:19 |
| @kuprince:matrix.org | so in inventory file, we are seeing the different mapping then what was I expecting . | 20:20 |
| @goneri:matrix.org | Can someone recheck this? https://review.opendev.org/c/zuul/zuul/+/822133 | 20:21 |
| @fungicide:matrix.org | my point was you seem to misunderstand how nodesets map labels to node names, as evidenced by the fact that there's an ip address (and not the same one) in your inventory as the host | 20:21 |
| @fungicide:matrix.org | * Prince Kumar: my point was you seem to misunderstand how nodesets map labels to node names, as evidenced by the fact that there's an ip address (and not the same one) in your inventory as the host | 20:21 |
| @kuprince:matrix.org | If I will remove ip address from its name, and then give some string as its name, in that case, we also need to change ssh config accordingly. | 20:24 |
| @fungicide:matrix.org | i'm not sure why that would entail a change in the ssh config. the basic requirement is that the ``label`` in your zuul nodeset matches a ``label`` in your nodepool provider configuration. that's the relationship zuul and nodepool use to coordinate node requests for builds | 20:34 |
| @kuprince:matrix.org | if server node wants to connect to worker node, it will do ssh, and it needs to resolve the name that is given to the node to do so. | 20:35 |
| @fungicide:matrix.org | what is "server node" in this context? i feel like i'm missing something important about your design goals. normally the zuul executor would connect to the worker nodes | 20:37 |
| @kuprince:matrix.org | my whole setup (zuul) is running on server node. We are not running any job on this node. now we have worker nodes, on which we want to run our jobs. so server node needs to connect with worker nodes via ssh | 20:38 |
| @jim:acmegating.com | fungi: should we maybe avoid merging anything right now? | 20:38 |
| @jim:acmegating.com | i mean, i do have the v5.0 sha.... but just wondering if we want to avoid merging anything not strictly necessary in case we change plans | 20:39 |
| @jim:acmegating.com | then again, that's just in tools/ so probably fine regardless? | 20:39 |
| @fungicide:matrix.org | corvus: oh! thanks, great point. if we've settled on the 5.0 ref, i'll block that change i just rechecked | 20:40 |
| @fungicide:matrix.org | done. i didn't consider that rechecking a previously approved change was going to interfere wit the release | 20:41 |
| @jim:acmegating.com | it *probably* won't... i just hadn't thought it through and wanted to raise it :) | 20:43 |
| @fungicide:matrix.org | Prince Kumar: okay, thanks, so you mean the executor needs to know how to connect to the nodes. i thought nodepool took care of setting up access keys for the executor to use, but to be honest i'm not all that familiar with caveats which may exist for the static node driver since i don't use it in the deployment i help manage | 20:43 |
| @kuprince:matrix.org | I think there is some config which do that, but I am doing all these things manually. | 20:44 |
| @fungicide:matrix.org | er, i guess the ssh_authorized_keys would be configured on each account you're using in each vm, so the executor should be using that regardless | 20:45 |
| @kuprince:matrix.org | yes. | 20:45 |
| @fungicide:matrix.org | in which case it shouldn't matter which ip address nodepool tells it belongs to a node which has been assigned | 20:45 |
| @fungicide:matrix.org | so changing the names in your nodeset definition shouldn't alter the ssh configuration anyway | 20:45 |
| @fungicide:matrix.org | assuming it's the same key which is used to connect to all of them | 20:46 |
| @kuprince:matrix.org | I think we have 2 components which need ssh files. one is executor and one is our nodepool. I remember that I needed to setup ssh files for both components differently. | 20:47 |
| @kuprince:matrix.org | But maybe I am doing something wrong. But I am sure that nodepool and executor needs ssh files to do their job. | 20:48 |
| @kuprince:matrix.org | I think executor needs to do ssh, when it runs your job on other nodes, and nodepool needs to do ssh to connect between different nodes | 20:49 |
| @jpew:matrix.org | I'm having some trouble getting keycloak to work to allow a user to do operations in the UI..... In the web logs I see `Could not fetch Identity Provider keys https://.../auth/realms/zuul/protocol/openid-connect/certs: Unable to find a algorithm for key: {'kid': 'ABC', 'kty': 'RSA', 'alg': 'RSA-OAEP', 'use': 'enc'...}` | 22:15 |
| @jpew:matrix.org | which ends up causing a 500 error on https://.../api/tenant/MyTenant/authorizations | 22:17 |
| @jpew:matrix.org | I don't really know where to go from here :( | 22:17 |
| @clarkb:matrix.org | jpew: that seems to originate in pyjwt here: https://github.com/jpadilla/pyjwt/blob/master/jwt/api_jwk.py#L47-L50 and the algorithms list is defined here: https://github.com/jpadilla/pyjwt/blob/master/jwt/algorithms.py#L84-L101 | 22:29 |
| @clarkb:matrix.org | notably the RSA algorithm is mapped to rs256 and that requires crypto. | 22:29 |
| @clarkb:matrix.org | and has_crypto checks if python cryptography is installed | 22:30 |
| @clarkb:matrix.org | it seems that is missing? | 22:30 |
| @clarkb:matrix.org | I suspect that if you ensure python cryptography is installed that it would work | 22:30 |
| @clarkb:matrix.org | What is odd is zuul lists cryptography in its requirements so it should already be there | 22:30 |
| @jpew:matrix.org | I'm using the zuul docker container from docker hub, let me check.... | 22:30 |
| @jpew:matrix.org | `import cryptography` works | 22:32 |
| @clarkb:matrix.org | I wonder if it is one of these sub imports that is failing then: https://github.com/jpadilla/pyjwt/blob/2.3.0/jwt/algorithms.py#L17-L51 (I switched to tag 2.3.0 instead of master as that is what should be in the container image) | 22:33 |
| @jpew:matrix.org | I wonder if the algorithm of RSA-OAEP is the problem | 22:33 |
| @jpew:matrix.org | There is a second key that has an RS256 key.... the one it's trying to use is a "enc" key, which doesn't seem correct | 22:34 |
| @clarkb:matrix.org | oh yup https://github.com/jpadilla/pyjwt/blob/2.3.0/jwt/api_jwk.py#L16-L17 | 22:34 |
| @clarkb:matrix.org | it tries to get alg first then if that isn't set gets kty | 22:34 |
| @clarkb:matrix.org | it knows how to handle that kty but not the alg of RSA-OAEP likely | 22:35 |
| @clarkb:matrix.org | I'm guessing that is something you can choose at token generation time | 22:38 |
| @jpew:matrix.org | Any idea where token generation is done? | 22:55 |
| @clarkb:matrix.org | jpew: I think keycloak does it after it gets the redirect from zuul | 22:55 |
| @clarkb:matrix.org | maybe there is a keycloak setting that needs to be made to selcet a more conservative algorithm? | 22:55 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 827167: Add release note for 5.0.0 https://review.opendev.org/c/zuul/nodepool/+/827167 | 23:45 | |
| @jim:acmegating.com | yay | 23:46 |
| @jim:acmegating.com | zuul-maint: how do these commits look for the 5.0.0 release of zuul and nodepool? | 23:47 |
| nodepool commit c34759b073050cc381fd709590b373ac430630ae (HEAD -> master, tag: 5.0.0, origin/master, refs/changes/67/827167/2) | ||
| zuul commit 930ee8faa3076233614565fcfbf55a4ee74551a7 (HEAD -> master, tag: 5.0.0, origin/master, refs/changes/98/826898/2) | ||
| @fungicide:matrix.org | corvus: looks great! i agree that matches with the current master branch states i have for both nodepool and zuul | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!