| *** s-shiono has joined #openstack-third-party-ci | 00:03 | |
| *** savihou has joined #openstack-third-party-ci | 06:47 | |
| *** savihou has quit IRC | 06:56 | |
| *** savihou has joined #openstack-third-party-ci | 06:56 | |
| *** savihou has quit IRC | 07:21 | |
| *** savihou has joined #openstack-third-party-ci | 07:22 | |
| *** wznoinsk has quit IRC | 08:16 | |
| *** mmedvede has quit IRC | 08:16 | |
| *** wznoinsk has joined #openstack-third-party-ci | 08:23 | |
| *** mmedvede has joined #openstack-third-party-ci | 08:23 | |
| *** wznoinsk has quit IRC | 08:50 | |
| *** mmedvede has quit IRC | 08:50 | |
| *** wznoinsk has joined #openstack-third-party-ci | 09:16 | |
| *** mmedvede has joined #openstack-third-party-ci | 09:16 | |
| *** s-shiono has quit IRC | 09:56 | |
| *** rfolco has joined #openstack-third-party-ci | 15:44 | |
| *** openstackgerrit_ has joined #openstack-third-party-ci | 15:47 | |
| *** openstackgerrit_ has quit IRC | 15:49 | |
| *** savihou has quit IRC | 16:32 | |
| *** lseki has joined #openstack-third-party-ci | 16:51 | |
| *** openstack has quit IRC | 17:02 | |
| *** openstack has joined #openstack-third-party-ci | 17:06 | |
| *** krtaylor has joined #openstack-third-party-ci | 17:09 | |
| lseki | Hey guys, I want to replace Jenkins with Ansible on my third party CI | 17:10 |
|---|---|---|
| asselin | lseki, i'm not sure anyone has done that yet. You should ask in #openstack-infra. | 17:11 |
| asselin | anyone --> 3rd party ci folks | 17:12 |
| lseki | oh I see | 17:12 |
| lseki | thanks asselin | 17:12 |
| krtaylor | lseki, I don't now how youd do that, one is a test job runner the other a deployment/configuration manager | 17:13 |
| krtaylor | seems like youd have to write a bunch of management tools that youd get for free with jenkins, any reason not to just use jenkins? | 17:14 |
| asselin | krtaylor, zuulv3 uses ansible instead of jenkins | 17:14 |
| krtaylor | oh, lseki did you mean using zuul? | 17:15 |
| lseki | krtaylor: I'm currently using Jenkins to run CI tests | 17:17 |
| lseki | Jenkins listens to Zuul's gearman and runs the jobs | 17:18 |
| krtaylor | yep, I kow that well :) | 17:19 |
| lseki | but for some reason, Jenkins starts queuing the jobs on its own Build Queue | 17:19 |
| krtaylor | re: zuul/ansible mmedvede have you looked at that yet? | 17:21 |
| lseki | and it seems that Jenkins assigns a specific nodepool node (e.g. d-p-c-local_01-25391) for the jobs in the queue | 17:21 |
| * krtaylor guesses that mmedvede may be off this week | 17:22 | |
| mmedvede | krtaylor: I looked at it awhile back and decided to wait until zuulv3 dust settles | 17:22 |
| lseki | this node may be deleted for some reason, before the job is run, and the queue gets stuck | 17:23 |
| lseki | so I thought replacing jenkins w/ ansible would eliminate this issue | 17:24 |
| krtaylor | lseki, why would it be deleted? | 17:24 |
| krtaylor | lseki, as mmedvede said, you might be asking for more/different problems until it gets all the kinks worked out, but infra would have more info on its readiness | 17:25 |
| mmedvede | lseki: are you using puppet-openstackci module? https://github.com/openstack-infra/puppet-openstackci | 17:25 |
| mmedvede | lseki: if you do not deviate from what it does, it should mostly work | 17:26 |
| lseki | krtaylor: power outage, network instability, and other bad things that happens in Brazil | 17:26 |
| krtaylor | hehheh, not just Brazil :) | 17:26 |
| lseki | mmedvede: yes, I'm using single_node_ci.pp which deploys me the whole stuff | 17:29 |
| lseki | including the problematic Jenkins :( | 17:30 |
| mmedvede | it is hard to tell what is going on from what you're saying. nodepool should be registering VMs with jenkins. Jenkins (through geraman plugin) then gets a job that might contain specific label to run the job on | 17:31 |
| mmedvede | or jenkins job itself can specify which label to run on | 17:32 |
| mmedvede | one jenkins starts running a job, you should see the node marked as 'used' in nodepool list | 17:32 |
| lseki | mmedvede: yes, that usually works as expected | 17:33 |
| mmedvede | and once a job is done, the node should be deleted by nodepool | 17:33 |
| lseki | the jenkins build is configured to run on nodes labeled "d-p-c" | 17:34 |
| lseki | and when it's working correctly, it happens as you described | 17:34 |
| mmedvede | lseki: you need to figure out who deletes your node | 17:36 |
| mmedvede | there is nothing jenkins can do if VM is just gone. In theory, if it is outside jenkins, you'd see the same problem with ansible runners | 17:36 |
| lseki | when a node is deleted, jenkins' build queue stops, and the next job says "No slave labeled d-p-c-local_01-xxx was found" | 17:37 |
| lseki | even if there are other nodes labeled d-p-c-local_01-yyy | 17:37 |
| lseki | I think jenkins is assigning a node to the job too early | 17:38 |
| mmedvede | lseki: sanity check, did you deal with jenkins security-170? | 17:38 |
| mmedvede | as described somewhere in puppet-openstackci, if you use newer jenkins and do not explicitly set certain option, jenkins would not accept zuul env variables | 17:39 |
| lseki | and unfortunately, some of my nodepool nodes will accidentally be deleted, for sure :( | 17:40 |
| lseki | yes, I configured Jenkins to accept env vars | 17:42 |
| asselin | lseki, it is true there are issues with jenkins, hense why zuulv3 switched to ansible (and for other reasons as well), but for 3rd party ci scale, it should work well enough | 17:42 |
| mmedvede | lseki: I do not have more ideas, did not experience what you're describing myself | 17:43 |
| mmedvede | I assume you checked nodepool logs for any errors | 17:44 |
| lseki | hmm have you guys ever seen a Build Queue appearing in Jenkins? | 17:47 |
| lseki | I'm not sure if it's something normal, since zuul has its own queue | 17:48 |
| mmedvede | lseki: I only see available workers and workers that are being used | 17:51 |
| mmedvede | not sure how build queue even looks in jenkins | 17:51 |
| lseki | mmedvede: it's something like this https://issues.jenkins-ci.org/secure/attachment/32833/build-queue.png | 17:52 |
| mmedvede | ok, I have "Build Queue" panel in mine. It always has "No builds in the queue." | 17:54 |
| lseki | the problem only occurs when 1) builds are inserted into Jenkins Build Queue AND 2) the node where the build was assigned is deleted before run | 17:55 |
| lseki | I thought solving 1) by replacing jenkins w/ ansible, which won't try to manage a queue | 17:59 |
| lseki | but maybe this queue shouldn't be populated at all | 18:00 |
| mmedvede | I am fairly certain you should not see anything in the queue | 18:01 |
| mmedvede | maybe it is worth going through https://github.com/openstack-infra/system-config/blob/5690f9971bdee17e724c660db62e33d6d21d2ac8/doc/source/jenkins.rst just in case | 18:04 |
| lseki | mmedvede: I'll read it, thanks! | 18:16 |
| lseki | thanks guys, I'll investigate why the build queue is being populated... it seems very wrong | 18:17 |
| *** rfolco has quit IRC | 20:41 | |
| *** lseki has quit IRC | 23:00 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
