| *** zbr has quit IRC | 00:09 | |
| *** yoctozepto has quit IRC | 00:53 | |
| *** yoctozepto has joined #zuul | 00:54 | |
| *** yoctozepto has quit IRC | 01:05 | |
| *** yoctozepto has joined #zuul | 01:05 | |
| *** tjgresha has quit IRC | 01:07 | |
| *** tjgresha has joined #zuul | 01:19 | |
| *** tjgresha has quit IRC | 01:22 | |
| *** fsvsbs has quit IRC | 01:53 | |
| *** rfolco has quit IRC | 01:54 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:11 | |
| *** swest has quit IRC | 02:21 | |
| *** rlandy|ruck has quit IRC | 02:34 | |
| *** swest has joined #zuul | 02:35 | |
| *** bhavikdbavishi has joined #zuul | 02:51 | |
| *** bhavikdbavishi1 has joined #zuul | 02:53 | |
| *** bhavikdbavishi has quit IRC | 02:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:55 | |
| *** bolg has joined #zuul | 04:13 | |
| *** mauved has quit IRC | 04:28 | |
| *** mauved has joined #zuul | 04:28 | |
| *** swest has quit IRC | 05:23 | |
| *** saneax has joined #zuul | 05:57 | |
| *** swest has joined #zuul | 06:07 | |
| *** saneax has quit IRC | 06:44 | |
| *** saneax has joined #zuul | 06:44 | |
| *** bhavikdbavishi has quit IRC | 06:52 | |
| *** bhavikdbavishi has joined #zuul | 06:56 | |
| *** bolg has quit IRC | 07:36 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** tosky has joined #zuul | 07:52 | |
| *** bolg has joined #zuul | 08:12 | |
| *** zbr has joined #zuul | 08:30 | |
| *** tosky has quit IRC | 08:32 | |
| *** zbr has quit IRC | 08:35 | |
| *** zbr has joined #zuul | 08:36 | |
| *** tosky has joined #zuul | 08:37 | |
| *** themroc has joined #zuul | 08:46 | |
| *** bolg has quit IRC | 08:58 | |
| *** bolg has joined #zuul | 09:07 | |
| zbr | pabelanger: hi! are you around? can you look again at https://review.opendev.org/#/c/693637/ ? | 09:21 |
|---|---|---|
| *** sshnaidm|off is now known as sshnaidm|rover | 09:30 | |
| *** irclogbot_2 has quit IRC | 09:39 | |
| *** irclogbot_1 has joined #zuul | 09:41 | |
| *** tosky has quit IRC | 09:41 | |
| *** tosky has joined #zuul | 09:42 | |
| *** bolg has quit IRC | 09:46 | |
| *** bolg has joined #zuul | 09:46 | |
| *** bolg has quit IRC | 09:51 | |
| *** tosky_ has joined #zuul | 09:52 | |
| *** tosky has quit IRC | 09:54 | |
| *** tosky has joined #zuul | 09:58 | |
| *** tosky_ has quit IRC | 10:01 | |
| *** tosky_ has joined #zuul | 10:05 | |
| *** tosky has quit IRC | 10:08 | |
| *** rfolco has joined #zuul | 10:08 | |
| *** bhavikdbavishi has quit IRC | 10:11 | |
| *** bhavikdbavishi has joined #zuul | 10:12 | |
| *** bolg has joined #zuul | 10:14 | |
| *** tosky has joined #zuul | 10:14 | |
| *** tosky_ has quit IRC | 10:17 | |
| *** tosky_ has joined #zuul | 10:21 | |
| *** tosky has quit IRC | 10:24 | |
| *** panda has quit IRC | 10:24 | |
| *** panda has joined #zuul | 10:29 | |
| *** bhavikdbavishi has quit IRC | 10:30 | |
| *** tosky has joined #zuul | 10:34 | |
| *** tosky_ has quit IRC | 10:36 | |
| *** tosky_ has joined #zuul | 10:43 | |
| *** tosky has quit IRC | 10:45 | |
| *** tosky_ has quit IRC | 10:46 | |
| *** tosky has joined #zuul | 10:46 | |
| *** hashar has joined #zuul | 10:51 | |
| *** tosky has quit IRC | 11:01 | |
| *** tosky has joined #zuul | 11:01 | |
| *** tosky_ has joined #zuul | 11:05 | |
| *** tosky has quit IRC | 11:06 | |
| *** tosky_ has quit IRC | 11:09 | |
| *** tosky has joined #zuul | 11:09 | |
| *** tosky has quit IRC | 11:17 | |
| *** tosky has joined #zuul | 11:23 | |
| *** hashar has quit IRC | 11:38 | |
| *** themroc has quit IRC | 11:43 | |
| *** tosky_ has joined #zuul | 11:47 | |
| *** tosky has quit IRC | 11:49 | |
| *** tosky has joined #zuul | 11:50 | |
| *** tosky_ has quit IRC | 11:53 | |
| *** themroc has joined #zuul | 11:55 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** openstackgerrit has quit IRC | 12:41 | |
| *** rlandy has joined #zuul | 12:50 | |
| *** rlandy is now known as rlandy|ruck | 12:54 | |
| *** sshnaidm|rover is now known as sshnaidm|ruck | 12:55 | |
| *** rlandy|ruck is now known as rlandy|over | 12:55 | |
| *** rlandy|over is now known as rlandy|rover | 12:56 | |
| *** jpena|lunch is now known as jpena | 13:27 | |
| mordred | SpamapS: most of it is in service of the nodepool-builder process, which I believe you don't run anyway. BUT - there is a change to the main dockerfile that *should* be no-impact on you at all, but would be good to make sure | 13:31 |
| *** bolg has quit IRC | 13:32 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 13:45 | |
| *** jangutter has quit IRC | 13:48 | |
| *** Goneri has joined #zuul | 13:48 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 13:59 | |
| pabelanger | zbr: replied | 14:37 |
| pabelanger | so, we are now running centos-8 jobs in zuul.a.c | 14:39 |
| pabelanger | for ansible lint | 14:39 |
| pabelanger | err | 14:39 |
| pabelanger | tox-linters | 14:39 |
| pabelanger | everything seems to work, if you add pip3 and virtualenv into images, like others | 14:39 |
| zbr | pabelanger: is your concern only related to virtuaenv? i was more concerned about epel-release issue | 14:39 |
| zbr | the old story: works *if* you do .... on your system. | 14:40 |
| pabelanger | zbr: yah, virtuelenv is main issue I see. But for epel-release, I think in the case of rdo and openstack-infra we need to install pip via DIB for now | 14:40 |
| pabelanger | until we can do a larger refactor | 14:40 |
| zbr | you did not comment on last version | 14:40 |
| zbr | look at https://review.opendev.org/#/c/693637/12/roles/bindep/tasks/install.yaml | 14:40 |
| zbr | which is less dangereus | 14:41 |
| pabelanger | but, I do think we need some sort of bootstrap-ansible role, early in job, for single place to add any dependency needed by ansible | 14:41 |
| pabelanger | oh, missed the latest patchset | 14:41 |
| pabelanger | yah, safer, but still think it is in wrong location. | 14:42 |
| zbr | i agree, you are not the first to mention this. issue is time, in fact i do already have ~that role somewhere else | 14:42 |
| pabelanger | yah | 14:42 |
| pabelanger | so, today we have unittests job | 14:42 |
| pabelanger | which is where bindep is called, I think | 14:42 |
| pabelanger | maybe we add bootstrap-ansible role | 14:43 |
| pabelanger | before bindep | 14:43 |
| zbr | in fact, virtualenv is not part of ansible. | 14:43 |
| zbr | and you know it well by now that there what goes into bootstrapping ansible is very cloudy, a huge gray area of module dependencies. | 14:43 |
| zbr | pip *may* need virtualenv, ipmi would need..., docker .... | 14:44 |
| zbr | in order to have some progress I just want to make bindep-role to work with epel-release. | 14:45 |
| zbr | i will propose generic ensure-ansible role in few weeks, as i do not have the hours needed this month. | 14:46 |
| pabelanger | right, so maybe by default we have the role not do anything, like noop but if user needs more ansible dependencies, they just add it to say ansible_bootstrap_packages / ansible_boostrap_method: (pip / package) | 14:46 |
| zbr | pabelanger: exactly default minimal reqs, but also receiving a list of optional requirements / labels. | 14:46 |
| pabelanger | yah | 14:47 |
| *** openstackgerrit has joined #zuul | 14:47 | |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Delete images by id https://review.opendev.org/693672 | 14:47 |
| zbr | i am bit surprised that this role we are talking about was not created by *ansible-team* itself, as it would have made sense. | 14:48 |
| pabelanger | ++ | 14:48 |
| zbr | it would likely be the most popular galaxy role ever. | 14:48 |
| *** shanemcd has quit IRC | 14:49 | |
| *** shanemcd has joined #zuul | 14:50 | |
| *** michael-beaver has joined #zuul | 14:51 | |
| *** tosky_ has joined #zuul | 15:00 | |
| *** tosky has quit IRC | 15:00 | |
| *** tosky_ is now known as tosky | 15:01 | |
| *** rfolco is now known as rfolco|brbr | 15:22 | |
| *** rfolco|brbr is now known as rfolco|brb | 15:22 | |
| *** pcaruana has joined #zuul | 15:28 | |
| *** bolg has joined #zuul | 15:31 | |
| *** tosky has quit IRC | 15:31 | |
| *** bhavikdbavishi has joined #zuul | 15:31 | |
| *** tosky has joined #zuul | 15:32 | |
| *** jhesketh has quit IRC | 16:05 | |
| *** jhesketh has joined #zuul | 16:21 | |
| *** themroc has quit IRC | 16:35 | |
| *** jpena is now known as jpena|off | 16:46 | |
| *** pcaruana has quit IRC | 17:03 | |
| *** bhavikdbavishi has quit IRC | 17:06 | |
| *** tjgresha has joined #zuul | 17:17 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: allow configure-mirrors to enable extra repos https://review.opendev.org/693887 | 17:19 |
| *** chandankumar is now known as rrrrrrrrrrrr | 17:29 | |
| *** rfolco|brb is now known as rfolco | 17:30 | |
| *** rrrrrrrrrrrr is now known as chandankumar | 17:34 | |
| pabelanger | Hmm, we have an in-use / locked node for 12hrs on zuul.a.c, but hard to see what is holding the lock | 17:34 |
| AJaeger | zuul-cores, please review https://review.opendev.org/692413 for zuul/project-config so that the GUI displays all changes | 17:36 |
| clarkb | AJaeger: see my quick comment on that change | 17:37 |
| *** chandankumar is now known as raukadah | 17:38 | |
| AJaeger | clarkb: good spot, will fix... | 17:39 |
| openstackgerrit | Andreas Jaeger proposed zuul/project-config master: Add success/failure message to pipelines https://review.opendev.org/692413 | 17:39 |
| pabelanger | so, I can see nodepool-launcher holding the lock of a in-use / locked node, but zuul-executor is done with it | 17:42 |
| pabelanger | any idea why that would be? | 17:42 |
| pabelanger | Shrews: ^ | 17:43 |
| Shrews | pabelanger: how do you know the launcher has the lock? | 17:43 |
| pabelanger | looking at zk-shell | 17:44 |
| pabelanger | let me get pb | 17:44 |
| pabelanger | http://paste.openstack.org/show/785990/ | 17:44 |
| pabelanger | 0000138761 in node in question | 17:44 |
| pabelanger | https://dashboard.zuul.ansible.com/t/ansible/nodes | 17:45 |
| pabelanger | can see it has been in-use for 13 hours | 17:45 |
| pabelanger | but, max job run times is capped at 3 hours | 17:45 |
| Shrews | oh, USED, not IN_USE | 17:47 |
| Shrews | that's different | 17:47 |
| pabelanger | Oh, yes used. Sorry about that | 17:48 |
| Shrews | pabelanger: launcher will lock a USED node when it attempts to delete it and the associated instance. You'll wanna check the launcher logs for that node ID | 17:48 |
| pabelanger | Shrews: http://paste.openstack.org/show/785991/ is the history of the node request, which was 2 nodes. But I only see 1 that was maked for deleted | 17:55 |
| pabelanger | the other seems to have not happened | 17:55 |
| pabelanger | Shrews: is there any thing else I can collect before restarting nodepool-launcher? Maybe dumping of threads? | 17:57 |
| *** igordc has joined #zuul | 17:59 | |
| Shrews | pabelanger: what timezone is that log? | 18:00 |
| pabelanger | UTC | 18:01 |
| Shrews | pabelanger: did you possibly do any changes with providers recently? like move them around, or disable, or what not? Otherwise, I don't see how that can possibly happen. | 18:03 |
| pabelanger | not recently no | 18:03 |
| Shrews | i have no explanation for why there isn't more in your logs. there should be *something*. afraid i don't have an answer | 18:05 |
| Shrews | unless the cleanup thread died | 18:06 |
| Shrews | you'd see a traceback for that, i think | 18:07 |
| openstackgerrit | Merged zuul/project-config master: Add success/failure message to pipelines https://review.opendev.org/692413 | 18:39 |
| SpamapS | Anybody else using the aws driver? I see these pretty frequently: http://paste.openstack.org/show/785993/ | 18:45 |
| SpamapS | I think it's a race.. if you try to read the instance too fast after getting the ID back, the API might not find it. | 18:45 |
| SpamapS | (nodepool tends to just retry and all is well, but this costs about 60s of waiting for a job to start since we have min-ready: 0 | 18:46 |
| *** tjgresha has quit IRC | 18:47 | |
| tobiash | SpamapS: not in production but I can check our logs for that | 18:47 |
| pabelanger | Shrews: no traceback, let me see if I can dump threads, then I'll restart | 18:47 |
| *** tjgresha has joined #zuul | 18:48 | |
| *** tjgresha has quit IRC | 18:56 | |
| *** tjgresha_ has joined #zuul | 18:56 | |
| SpamapS | tobiash: Just noticed, us-east-1 is having increased API error rates. | 18:57 |
| SpamapS | tobiash: so it may only happen when that's going on. :-P | 18:57 |
| SpamapS | But I've seen them in the past sporadically. It's just worse today. | 18:57 |
| *** tjgresha_ has quit IRC | 19:03 | |
| *** tjgresha has joined #zuul | 19:04 | |
| *** tjgresha has quit IRC | 19:07 | |
| *** michael-beaver has quit IRC | 19:10 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 19:42 | |
| *** Goneri has quit IRC | 19:51 | |
| *** panda has quit IRC | 20:18 | |
| *** panda has joined #zuul | 20:21 | |
| pabelanger | SpamapS: tobiash: we'll likely go all in on AWS in the near future, getting pushed hard by humans to us it for testing | 20:36 |
| pabelanger | mostly because of existing appliances that exist in that environment | 20:36 |
| *** rfolco has quit IRC | 21:01 | |
| *** rfolco has joined #zuul | 21:01 | |
| *** Goneri has joined #zuul | 21:02 | |
| *** rfolco has quit IRC | 21:06 | |
| *** aluria has quit IRC | 21:07 | |
| *** Goneri has quit IRC | 21:43 | |
| *** Goneri has joined #zuul | 21:46 | |
| *** Goneri has quit IRC | 21:57 | |
| *** Goneri has joined #zuul | 21:57 | |
| *** Goneri has quit IRC | 22:01 | |
| *** Goneri has joined #zuul | 22:01 | |
| *** aluria has joined #zuul | 22:14 | |
| *** homeski has joined #zuul | 22:23 | |
| SpamapS | pabelanger: I'm pretty happy with it. We spread our testing between 2 regions, 4 AZ's in each region. | 22:43 |
| SpamapS | pabelanger: Have definitely wanted to spend some time to add spot-instance support, but right now our monthly Zuul AWS bill is like $75 so it's not worth my time yet. | 22:44 |
| *** igordc has quit IRC | 23:06 | |
| *** igordc has joined #zuul | 23:31 | |
| pabelanger | SpamapS: great, hope to get some credentials soon (maybe this week) and try it out | 23:37 |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 23:41 |
| *** klindgren_ has quit IRC | 23:47 | |
| *** klindgren has joined #zuul | 23:47 | |
| *** klindgren has quit IRC | 23:50 | |
| *** klindgren has joined #zuul | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!