| kazsh | AJaeger_: thanks for your response, now we are getting review from Horizon team to split out head-dashboard from horizon repo & got comment we need to enable translation job. | 00:00 |
|---|---|---|
| AJaeger_ | kazsh: I'm still not getting how the translation jobs block you to release queens-2. | 00:01 |
| AJaeger_ | the translation jobs push to a server, then translations happen, and then it pushes back nearly complete translations whenever those are done. If we merge that change now, you might wait two year, or 2 weeks - for translations... | 00:01 |
| kazsh | I guess the translation job is not mandatory as you pointed out but just want to follow their review comment as much as possible before queens-2 | 00:02 |
| AJaeger_ | kazsh: exactly. It's good to have it - but it will only change content if the volunteer translators translate. And your change enables that... | 00:03 |
| kazsh | ah yea.. | 00:03 |
| AJaeger_ | and then you have to explictely import the changes | 00:03 |
| kazsh | your are correct, need to work that too...maybe I have to.. | 00:03 |
| dmsimard | mordred: gotcha on your comment, will look later | 00:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config master: Add log-classify project template https://review.openstack.org/525639 | 00:04 |
| kazsh | AJaeger_: ok, I understand that I don't need to be hurry up now, will wait the change will be merged later :) | 00:05 |
| AJaeger_ | kazsh: I'm sure rest of the config-core team will review 525028 later this week... | 00:06 |
| kazsh | AJaeger_: Noted with thanks mate :) | 00:06 |
| *** rbrndt has quit IRC | 00:07 | |
| *** caphrim007 has joined #openstack-infra | 00:10 | |
| *** signed8bit_Zzz is now known as signed8b_ | 00:10 | |
| *** caphrim007 has quit IRC | 00:14 | |
| *** jamesmcarthur has joined #openstack-infra | 00:16 | |
| smcginnis | fungi: Cool, thanks. Can we requeue that release job at some point (tomorrow)? | 00:18 |
| AJaeger_ | tristanC: did you see my comment on 525639 regarding noop-jobs/merging .zuul.yaml? | 00:19 |
| clarkb | johnsom: rm_work ok running your reproduce.sh has reproduced it | 00:19 |
| rm_work | \o/ | 00:19 |
| clarkb | johnsom: rm_work I think what I may do is keep this VM up | 00:19 |
| mordred | clarkb: looking | 00:19 |
| clarkb | give ovh the instance uuid so they can track it to a hypervisor then see if there is any info on their end | 00:20 |
| johnsom | Sweet. Thanks for the effort clarkb | 00:20 |
| rm_work | yes, thanks a ton!!! | 00:20 |
| clarkb | while at the same time try to reproduce it with vanilla devstack + tempest | 00:20 |
| mordred | clarkb: oh - goodie - we're not seeing the ipv6 addresses in teh config drive metadata on ovh? | 00:20 |
| fungi | smcginnis: if we get a second config-core reviewer to approve 525769 first and it merges, then sure, happy to, just remind me when you're ready | 00:20 |
| clarkb | johnsom: rm_work oh and it reproduced on cirros without config drive | 00:20 |
| clarkb | mordred: correct | 00:20 |
| johnsom | clarkb You can also give them the kernel.org bug so they can answer the questions they asked about the underlying host. | 00:20 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Create a test matrix role https://review.openstack.org/524402 | 00:20 |
| *** yamahata has quit IRC | 00:20 | |
| *** sdague has quit IRC | 00:20 | |
| clarkb | johnsom: have that handy? I'll make a note of it now | 00:21 |
| mordred | clarkb: sweet. do we get them via dhcp6 or RA or anything like that? | 00:21 |
| *** jamesmcarthur has quit IRC | 00:21 | |
| johnsom | Just a sec | 00:21 |
| clarkb | mordred: I don't think they are RAs because the interfaces haven't auto configured them | 00:21 |
| mordred | like- we should only ever need to pull them from metadata if we're not getting them via 'normal' means | 00:21 |
| clarkb | mordred: but I haven't really dug into that yet | 00:21 |
| mordred | spectacular | 00:21 |
| mordred | clarkb: but - so - in theory if we go and manually add the ipv6 info that's in the nova metadata we get working ipv6 on the node? | 00:21 |
| johnsom | clarkb https://bugzilla.kernel.org/show_bug.cgi?id=192521 | 00:21 |
| openstack | bugzilla.kernel.org bug 192521 in kvm "KVM: entry failed, hardware error 0x0" [High,New] - Assigned to virtualization_kvm | 00:21 |
| smcginnis | fungi: Thanks! | 00:21 |
| clarkb | mordred: I want to say on my personal VM I had to statically configure it | 00:21 |
| mordred | clarkb: I think I remember us poking at this a few months ago .. | 00:22 |
| clarkb | ya I think that is the theory | 00:22 |
| mordred | clarkb: oh- wait, no, I think I'm thinking about when we tried to add an ipv6 subnet to our citycloud network | 00:22 |
| mordred | clarkb: so I'm not sure I've poked at this at ovh yet | 00:22 |
| clarkb | mordred: ok, I will send ovh a second email tomorrow and cc you for that item | 00:23 |
| clarkb | I'm not feeling the energy to accurate write down either thing write now | 00:23 |
| mordred | clarkb: I think ovh has a custom neutron plugin or something? so it's possible that maybe that isn't plumbed through to the metadata pieces? or something | 00:23 |
| fungi | i vaguely recall we asked them about it in person and... damien? said they didn't make v6 automatic (yet) so it could be opt-in by customers manually configuring it | 00:23 |
| mordred | clarkb: yah- tomorrow sounds great | 00:23 |
| fungi | because some of their customers didn't want v6 or something | 00:23 |
| fungi | also possible i dreamt that entire conversation | 00:23 |
| clarkb | fungi: our instances have IPs according to the nova api at least though | 00:23 |
| clarkb | its just that the instances aren't being told about it | 00:23 |
| mordred | fungi: yah - I mean, I could see that and be fine with it - if the addresses wind up in config-drive metadata | 00:23 |
| fungi | right, but no ra or dhcp6 so that instances won't autoconfigure v6 in case customers didn't want working v6 | 00:24 |
| mordred | I *think* glean would do the right thing - but if it didn't, it wouldn't be super hard to fix | 00:24 |
| mordred | fungi: yah. | 00:24 |
| fungi | anyway, it seemed like a specious argument, and possibly an excuse for not having finished their v6 implementation | 00:24 |
| fungi | but i don't really know | 00:25 |
| *** dimak has quit IRC | 00:25 | |
| *** joadavis has joined #openstack-infra | 00:25 | |
| *** threestrands has quit IRC | 00:25 | |
| clarkb | johnsom: rm_work one more piece of info, kvm-spice was being used to boot things when it was working for me. Which then forks an qemu-system-x86_64. In the reproduction case it appears to just be qemu-system-x86)64 | 00:25 |
| *** sdague has joined #openstack-infra | 00:26 | |
| johnsom | clarkb Our gate job has "/usr/bin/qemu-system-x86_64" | 00:27 |
| clarkb | johnsom: rm_work as soon as I figure out which package includes kvm-spice can you add that to your change that runs on ovh and see if that changes anything? | 00:27 |
| AJaeger_ | is Zuul confused? The post queue has three entries - two of them in there for 9 hours, just waiting for docs job | 00:27 |
| johnsom | clarkb Yeah, we can probably do that. I wonder if that is a nova setup or just a bindep.... | 00:28 |
| clarkb | johnsom: aha! this could be it maybe? qemu-kvm installs it | 00:28 |
| clarkb | johnsom: and qemu-kvm is not present on the failed reproduction case | 00:28 |
| clarkb | whcih would also explain why enable-kvm is missing maybe | 00:28 |
| clarkb | I wonder if we are trying to use kvm in a qemu that isn't fully kvm capable? | 00:28 |
| clarkb | johnsom: you'll have to add it to the devstack package lists | 00:29 |
| johnsom | Hmm, I don't see qemu-kvm in the dpkg list on our failed case either | 00:29 |
| rm_work | ahh so maybe we're missing a bindep | 00:29 |
| rm_work | yeah that's what he's saying | 00:29 |
| clarkb | I'm not sure that bindep is used by devstack but ya | 00:29 |
| rm_work | in the failed case, it's not installed | 00:29 |
| rm_work | O_o | 00:29 |
| rm_work | that'd be ... too obvious, right? | 00:29 |
| clarkb | its on the list of things to change in devstack I think but it has long had its own way of listing package deps | 00:30 |
| AJaeger_ | infra-root, we only have 8 out of 10 executors according to grafana | 00:30 |
| rm_work | "didn't install qemu-kvm, kvm didn't work in qemu" | 00:30 |
| AJaeger_ | clarkb: devstack does not use bindep AFAIK | 00:30 |
| rm_work | ok how do i add this? | 00:30 |
| *** joadavis__ has joined #openstack-infra | 00:30 | |
| *** dimak has joined #openstack-infra | 00:31 | |
| AJaeger_ | infra-root, sorry, 9 executors currently | 00:31 |
| clarkb | rm_work: https://git.openstack.org/cgit/openstack-dev/devstack/tree/files/debs/n-cpu you add it to that file I think | 00:31 |
| *** joadavis__ has quit IRC | 00:31 | |
| clarkb | rm_work: you may be able to shim it into your devstack plugin temporarily to test it more easily but guessing longer term it shoudl live in nova's dep list | 00:31 |
| rm_work | clarkb: ah so we can't do it in-tree? | 00:31 |
| rm_work | ah right | 00:32 |
| rm_work | i want to test it in-tree | 00:32 |
| rm_work | so can i add like ... an apt-get install? | 00:32 |
| clarkb | ya you could do it that way | 00:32 |
| rm_work | k | 00:32 |
| clarkb | or if you add a files/debs/foo to your devstack plugin I think that works too | 00:32 |
| johnsom | Could be here too: http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova_plugins/functions-libvirt#n64 | 00:32 |
| AJaeger_ | infra-root, so the 12 min old post job just got a node for docs job while the 9hour old ones still don't have one ;( | 00:32 |
| rm_work | wait what | 00:32 |
| rm_work | ah | 00:33 |
| rm_work | i see it | 00:33 |
| rm_work | got it | 00:33 |
| *** tosky has quit IRC | 00:33 | |
| clarkb | AJaeger_: I think that can happen because zuul oeprates on requests now | 00:33 |
| rm_work | it's up for review, we'll see if we land on an ovh host | 00:33 |
| clarkb | AJaeger_: but I'm not sure when those go in (eg if it was 9 hours ago or maybe not too much before the request that did get fulfilled) | 00:34 |
| AJaeger_ | clarkb: if we have only 9 executors instead of 10, could that be the problem? | 00:34 |
| clarkb | AJaeger_: possibly if the 10th executor was responsible for that request somehow and it died? | 00:35 |
| clarkb | but I think zuul scheduelr makes all the requests then farms them out to executors once met | 00:35 |
| AJaeger_ | clarkb: all possible - question is whether we can / should do anything here to heal it? | 00:35 |
| clarkb | jeblair: ^ | 00:35 |
| clarkb | rm_work: johnsom if that does fix it, would be curious to see if it works in other clouds without that package or are we somehow pulling that package in on other clouds | 00:36 |
| *** signed8b_ is now known as signed8bit_Zzz | 00:36 | |
| pabelanger | AJaeger_: no, 1 executor is being limited by loadavg, it will come backonline shortly | 00:36 |
| clarkb | rm_work: johnsom: devstack jobs capture a dpkg -l listing you should be able to check | 00:36 |
| jeblair | clarkb: node requests come from scheduler | 00:36 |
| *** gmann_afk is now known as gmann | 00:36 | |
| clarkb | oh good I had remembered that design point correctly | 00:37 |
| AJaeger_ | pabelanger: indeed, back to 10 now | 00:37 |
| pabelanger | AJaeger_: refresh grafana now | 00:37 |
| pabelanger | AJaeger_: yah, ze06, you can see it was up to loadavg of 60 | 00:37 |
| AJaeger_ | so, is this related to the post queue? | 00:38 |
| pabelanger | I am not sure, did we lose an executor today? | 00:39 |
| rm_work | clarkb: it is NOT present on the working ones either | 00:39 |
| *** dhinesh_ has joined #openstack-infra | 00:39 | |
| pabelanger | AJaeger_: seems something happened about 10 hrs ago | 00:39 |
| clarkb | rm_work: interesting | 00:40 |
| clarkb | rm_work: the descript of the qemu-kvm package is Description: QEMU Full virtualization | 00:40 |
| clarkb | implying that it would be necessary for that feature | 00:40 |
| rm_work | actually hold up | 00:41 |
| rm_work | johnsom pointed out i may have looked at a bad example | 00:41 |
| rm_work | nope confirmed again | 00:42 |
| rm_work | i see zero "kvm" packages, in fact O_o | 00:42 |
| johnsom | clarkb Yeah, infracloud passes but does not have the qemu-kvm package. http://logs.openstack.org/37/453237/10/check/octavia-v1-dsvm-scenario/1dddc48/logs | 00:42 |
| *** dhinesh has quit IRC | 00:42 | |
| joadavis | Sorry to interrupt. Got a deep Zuul question - Can a project template be disabled per-branch? Example: monasca-api fails on running publish-openstack-sphinx-docs for stable/ocata. | 00:43 |
| clarkb | joadavis: generally yes, but iirc docs jobs are a policy mandated job for openstack projects? so those need to run (and pass) | 00:44 |
| clarkb | that said, there has been recent upheaval around how we run docs | 00:44 |
| clarkb | and the failures are likely related? | 00:44 |
| clarkb | fungi: mordred ^ I think you know more about that than I do | 00:44 |
| joadavis | The call to setup.py build_sphinx replies "invalid command" | 00:44 |
| clarkb | joadavis: that implies sphinx isn't being installed | 00:45 |
| joadavis | makes it hard to do code maintenance when old branches can't pass gate. :/ | 00:45 |
| clarkb | do you have a link to the failure? | 00:45 |
| AJaeger_ | joadavis: so, it works on master? | 00:45 |
| joadavis | http://logs.openstack.org/22/525522/1/check/build-openstack-sphinx-docs/e015468/job-output.txt.gz | 00:45 |
| * AJaeger_ signs off | 00:45 | |
| *** SumitNaiksatam has left #openstack-infra | 00:46 | |
| joadavis | witek hit that when working on https://review.openstack.org/#/c/525522/ | 00:46 |
| clarkb | ya there is no sphinx in the venv installation there | 00:46 |
| joadavis | so sphinx related gate works on master or pike branches (where I assume the venv has sphinx) | 00:47 |
| clarkb | master has it based on a quick read of test-requirements.txt and tox.ini | 00:48 |
| clarkb | reading ocata bow | 00:48 |
| clarkb | *now | 00:48 |
| clarkb | is missing from ocata after a similarly quick read | 00:48 |
| clarkb | one guess for why/how this broke is a dep was pulling it in and that dep has ceased to do so | 00:49 |
| clarkb | or maybe we just never ran docs on ocata and this is fallout from the zuulv3 transition? | 00:49 |
| *** hongbin has quit IRC | 00:49 | |
| joadavis | so is the better process to hack around on the old branch to try to get docs created, or to disable the docs gate (since the docs for Ocata have presumably already been created and published)? | 00:50 |
| clarkb | https://git.openstack.org/cgit/openstack/monasca-api/tree/docs?h=stable/ocata that probably explains why this doesn't work | 00:51 |
| clarkb | there aren't sphinx docs on that branch, thats odd | 00:51 |
| joadavis | hmm. how about older versions (looking...) | 00:51 |
| joadavis | oh, no stable/newton listed (probably cuz it was eol'd) | 00:52 |
| joadavis | huh, don't see anything sphinx-like for master either. | 00:54 |
| joadavis | oh, path issue - https://git.openstack.org/cgit/openstack/monasca-api/tree/doc?h=stable%2Fpike | 00:55 |
| clarkb | in this case you probably do want to disable the job | 00:57 |
| joadavis | yes, and just for stable/ocata, since stable/pike and on seem to be working | 00:57 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n8490 is where it is added I think | 00:58 |
| clarkb | you might need to remove that template and individually list all the jobs theer so that you can exclude just the docs job from the ocata branch | 00:58 |
| joadavis | line 8493, yeah | 00:58 |
| *** xarses has joined #openstack-infra | 01:00 | |
| joadavis | Example? Something like what is done for mon-trans on line 8684?? | 01:01 |
| *** gk__ has joined #openstack-infra | 01:01 | |
| gk__ | Hi Fungi, are you around? | 01:02 |
| *** esberglu has joined #openstack-infra | 01:02 | |
| *** cuongnv has joined #openstack-infra | 01:03 | |
| clarkb | joadavis: ya thats basically what you'd need to do, but that template is an aggregation of many jobs so you need to split them out and list them all then only exclude the docs job in that manner | 01:04 |
| clarkb | let me find a link to the template definition | 01:04 |
| *** vhosakot has quit IRC | 01:05 | |
| *** dhinesh_ has quit IRC | 01:05 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 01:05 | |
| *** dhinesh has joined #openstack-infra | 01:06 | |
| joadavis | ick. Well, at least we can drop the exception when ocata gets eol'd. | 01:06 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n22 | 01:06 |
| clarkb | you might be able to apply the branch exclusion to the template? I'm not actually sure | 01:06 |
| joadavis | clarkb: thanks for the help BTW | 01:06 |
| *** esberglu has quit IRC | 01:07 | |
| clarkb | but there are only 3 jobs listed there so not too bad to individually list them | 01:07 |
| *** bandini has quit IRC | 01:07 | |
| *** puiterwijk has quit IRC | 01:07 | |
| mordred | clarkb: I believe you shold be able to add branch matchers to the job in the pipeline definition | 01:07 |
| joadavis | applying the exclusion to the template would be nice. Can't tell if that is legal from reading https://docs.openstack.org/infra/zuul/user/config.html | 01:07 |
| joadavis | s/template/templates list/ | 01:08 |
| *** puiterwijk has joined #openstack-infra | 01:08 | |
| *** tristanC has quit IRC | 01:08 | |
| *** tristanC has joined #openstack-infra | 01:09 | |
| *** bandini has joined #openstack-infra | 01:09 | |
| *** frickler has quit IRC | 01:09 | |
| *** frickler has joined #openstack-infra | 01:10 | |
| *** gyee has quit IRC | 01:10 | |
| joadavis | I'll give exploding the template to jobs then limiting a try. Thanks again. | 01:13 |
| *** dhinesh has quit IRC | 01:15 | |
| *** liujiong has joined #openstack-infra | 01:20 | |
| *** dhinesh has joined #openstack-infra | 01:21 | |
| dmsimard | clarkb: I edited ethercalc for sprint, I'll have limited availability on dec 14/15, I realized that I'm helping to drive the RDO test days for the queens milestone | 01:23 |
| *** yamahata has joined #openstack-infra | 01:25 | |
| *** auggy has joined #openstack-infra | 01:25 | |
| *** dhinesh_ has joined #openstack-infra | 01:26 | |
| *** dhinesh has quit IRC | 01:26 | |
| clarkb | dmsimard: that should be fine I expect we'll get most things done early then will slow down later in the week as we hit the harder problems | 01:28 |
| dmsimard | mordred: with_first_found works for the doc/requirements.txt thing. | 01:28 |
| dmsimard | mordred: the caveat is that if no files are found at all, there's an error. Do we want to raise an error ? Otherwise we can let it continue without an error with a "skip: true" argument. | 01:29 |
| *** cshastri has joined #openstack-infra | 01:29 | |
| *** psachin has joined #openstack-infra | 01:29 | |
| dmsimard | mordred: btw the intent was totally not to steal your patch, feel free to carry that back into yours | 01:31 |
| *** liusheng has quit IRC | 01:32 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** kiennt26 has joined #openstack-infra | 01:36 | |
| *** mat128 has joined #openstack-infra | 01:36 | |
| mordred | dmsimard: I've got plenty of patches, I like yours ... I think maybe skip: true is a good thing to try? | 01:36 |
| dmsimard | mordred: worst that can happen is that it fails later | 01:37 |
| mordred | yah | 01:37 |
| *** aeng has quit IRC | 01:38 | |
| *** markvoelker has quit IRC | 01:38 | |
| *** caphrim007 has joined #openstack-infra | 01:39 | |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** caphrim007 has quit IRC | 01:43 | |
| *** caphrim007_ has joined #openstack-infra | 01:43 | |
| *** signed8bit_Zzz is now known as signed8b_ | 01:46 | |
| *** liusheng has joined #openstack-infra | 01:46 | |
| dmsimard | mordred: ok I'll send another patchset on top of yours with that in | 01:49 |
| *** zhurong has joined #openstack-infra | 01:49 | |
| *** mriedem_away has quit IRC | 01:50 | |
| *** rlandy has quit IRC | 01:50 | |
| *** namnh has joined #openstack-infra | 01:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: zk: automatically retry command when connection is lost https://review.openstack.org/523640 | 02:01 |
| *** mat128_ has joined #openstack-infra | 02:02 | |
| dmsimard | mordred: /me wonders at what point it's worth writing a sphinx ansible module | 02:03 |
| *** mat128 has quit IRC | 02:05 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: zk: automatically retry command when connection is lost https://review.openstack.org/525851 | 02:06 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add sphinx_python variable to sphinx role and job https://review.openstack.org/525688 | 02:07 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add sphinx_python variable to sphinx role and job https://review.openstack.org/525688 | 02:10 |
| *** mriedem has joined #openstack-infra | 02:10 | |
| *** oanson has quit IRC | 02:12 | |
| *** signed8b_ is now known as signed8bit_Zzz | 02:12 | |
| *** hongbin has joined #openstack-infra | 02:12 | |
| *** oanson has joined #openstack-infra | 02:12 | |
| *** rfolco has quit IRC | 02:16 | |
| *** bobh has joined #openstack-infra | 02:18 | |
| *** bobh has quit IRC | 02:20 | |
| *** mriedem has quit IRC | 02:23 | |
| *** pbourke_ has quit IRC | 02:25 | |
| *** dave-mccowan has joined #openstack-infra | 02:26 | |
| *** gk__ has quit IRC | 02:26 | |
| *** pbourke_ has joined #openstack-infra | 02:27 | |
| *** dave-mccowan has quit IRC | 02:30 | |
| *** hongbin_ has joined #openstack-infra | 02:33 | |
| *** Apoorva_ has joined #openstack-infra | 02:33 | |
| *** bobh has joined #openstack-infra | 02:34 | |
| *** Apoorva has quit IRC | 02:36 | |
| *** bobh has quit IRC | 02:36 | |
| *** hongbin has quit IRC | 02:37 | |
| *** Apoorva_ has quit IRC | 02:37 | |
| *** markvoelker has joined #openstack-infra | 02:39 | |
| *** dave-mccowan has joined #openstack-infra | 02:45 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 02:49 | |
| *** esberglu has joined #openstack-infra | 02:51 | |
| *** esberglu has quit IRC | 02:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/log-classify master: Add python project structure https://review.openstack.org/525858 | 02:52 |
| *** bobh has joined #openstack-infra | 02:52 | |
| *** esberglu has joined #openstack-infra | 02:52 | |
| *** esberglu has quit IRC | 02:57 | |
| *** signed8bit_Zzz has quit IRC | 03:00 | |
| *** signed8bit has joined #openstack-infra | 03:01 | |
| *** zhurong has quit IRC | 03:02 | |
| *** gongysh has joined #openstack-infra | 03:04 | |
| rosmaita | is there something going on with the gate? there are 45 gate jobs, but 8 of them are just sitting, some for almost 5 hours | 03:07 |
| *** zhurong has joined #openstack-infra | 03:09 | |
| rm_work | clarkb: for the record: http://logs.openstack.org/29/525729/2/check/octavia-v1-dsvm-py3x-scenario/db72e79/logs/ | 03:11 |
| rm_work | clarkb: qemu-kvm installed, same issue | 03:11 |
| rm_work | looks like it tries to start with spice tho? | 03:11 |
| rm_work | not sure if different, need to compare | 03:12 |
| *** bobh has quit IRC | 03:12 | |
| *** sree_ has joined #openstack-infra | 03:16 | |
| *** sree_ is now known as Guest50935 | 03:16 | |
| *** sree has joined #openstack-infra | 03:20 | |
| dmsimard | infra-root: can someone kick/ban xdl from #zuul, spamming things. | 03:21 |
| dmsimard | clarkb: those IRC permissions would've come in handy after all :/ | 03:22 |
| *** JasonCL has joined #openstack-infra | 03:22 | |
| *** Guest50935 has quit IRC | 03:22 | |
| * dmsimard goes to sleep cause he apparently never sleeps | 03:22 | |
| *** liujiong has quit IRC | 03:25 | |
| *** liujiong has joined #openstack-infra | 03:26 | |
| *** liujiong has quit IRC | 03:32 | |
| *** annp has joined #openstack-infra | 03:37 | |
| *** dhinesh_ has quit IRC | 03:40 | |
| openstackgerrit | Joseph Davis proposed openstack-infra/project-config master: monasca-api: Exclude stable/ocata from sphinx doc creation https://review.openstack.org/525864 | 03:40 |
| *** dhinesh has joined #openstack-infra | 03:40 | |
| openstackgerrit | Joseph Davis proposed openstack-infra/project-config master: monasca-api: Exclude stable/ocata from sphinx doc creation https://review.openstack.org/525864 | 03:43 |
| *** cody-somerville has quit IRC | 03:44 | |
| *** gongysh has quit IRC | 03:45 | |
| openstackgerrit | Joseph Davis proposed openstack-infra/project-config master: monasca-api: Exclude stable/ocata from sphinx doc creation https://review.openstack.org/525864 | 03:45 |
| *** aeng has joined #openstack-infra | 03:45 | |
| *** bobh has joined #openstack-infra | 03:53 | |
| *** ramishra has joined #openstack-infra | 03:55 | |
| *** _ari_ has joined #openstack-infra | 03:58 | |
| *** links has joined #openstack-infra | 03:59 | |
| *** _ari__ has quit IRC | 03:59 | |
| *** ntpttr_ has quit IRC | 03:59 | |
| *** tbarron has quit IRC | 03:59 | |
| *** ntpttr_ has joined #openstack-infra | 03:59 | |
| *** tbarron has joined #openstack-infra | 04:00 | |
| *** links has quit IRC | 04:01 | |
| *** links has joined #openstack-infra | 04:01 | |
| *** caphrim007_ has quit IRC | 04:04 | |
| *** dhinesh has quit IRC | 04:04 | |
| *** sdague has quit IRC | 04:05 | |
| *** dhinesh has joined #openstack-infra | 04:06 | |
| *** mgkwill has quit IRC | 04:07 | |
| *** cody-somerville has joined #openstack-infra | 04:07 | |
| *** mgkwill_ has joined #openstack-infra | 04:07 | |
| *** bobh has quit IRC | 04:08 | |
| *** yamamoto has joined #openstack-infra | 04:12 | |
| *** Apoorva has joined #openstack-infra | 04:12 | |
| *** bobh has joined #openstack-infra | 04:16 | |
| *** bobh has quit IRC | 04:17 | |
| *** ykarel__ has joined #openstack-infra | 04:19 | |
| ianw | i dunno who can kick | 04:20 |
| *** zhurong has quit IRC | 04:21 | |
| *** dave-mccowan has quit IRC | 04:23 | |
| *** gongysh has joined #openstack-infra | 04:24 | |
| *** sree_ has joined #openstack-infra | 04:26 | |
| *** sree_ is now known as Guest14002 | 04:27 | |
| *** sree has quit IRC | 04:28 | |
| *** sree has joined #openstack-infra | 04:31 | |
| *** Guest14002 has quit IRC | 04:32 | |
| *** david-lyle has joined #openstack-infra | 04:32 | |
| *** janki has joined #openstack-infra | 04:33 | |
| *** gouthamr has quit IRC | 04:36 | |
| *** hamzy has joined #openstack-infra | 04:40 | |
| *** namnh has quit IRC | 04:41 | |
| *** cuongnv has quit IRC | 04:41 | |
| *** daidv has quit IRC | 04:41 | |
| *** annp has quit IRC | 04:41 | |
| *** annp has joined #openstack-infra | 04:42 | |
| *** daidv has joined #openstack-infra | 04:42 | |
| *** namnh has joined #openstack-infra | 04:42 | |
| *** cuongnv has joined #openstack-infra | 04:42 | |
| *** david-lyle has quit IRC | 04:42 | |
| *** sree has quit IRC | 04:45 | |
| *** udesale has joined #openstack-infra | 04:46 | |
| *** sree has joined #openstack-infra | 04:49 | |
| *** sree_ has joined #openstack-infra | 04:50 | |
| *** sree_ is now known as Guest76573 | 04:50 | |
| *** sree has quit IRC | 04:53 | |
| *** janki has quit IRC | 04:54 | |
| *** Apoorva_ has joined #openstack-infra | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: [WIP] Move prepare-zanata-client to o-z-j https://review.openstack.org/525760 | 04:56 |
| *** Apoorva has quit IRC | 04:57 | |
| *** janki has joined #openstack-infra | 05:01 | |
| *** ykarel__ is now known as ykarel | 05:04 | |
| *** coolsvap has joined #openstack-infra | 05:07 | |
| *** zhurong has joined #openstack-infra | 05:08 | |
| *** threestrands has joined #openstack-infra | 05:09 | |
| *** threestrands has quit IRC | 05:09 | |
| *** cuongnv has quit IRC | 05:14 | |
| *** vivsoni_ has joined #openstack-infra | 05:19 | |
| vivsoni_ | Hi Team, Zuul check is failing in driverfixes/ocata branch | 05:19 |
| vivsoni_ | http://logs.openstack.org/19/525719/1/check/openstack-tox-py27/f0c6813/job-output.txt.gz#_2017-12-05_18_35_28_793241 | 05:19 |
| vivsoni_ | can someone please help ? | 05:19 |
| *** cuongnv has joined #openstack-infra | 05:19 | |
| *** csomerville has quit IRC | 05:22 | |
| *** csomerville has joined #openstack-infra | 05:23 | |
| Jeffrey4l | infra-core, if anyone could check this? http://logs.openstack.org/periodic/git.openstack.org/openstack/kolla/master/kolla-build-centos-source/cf1edb2/ara/result/025313a9-653d-42d1-9dd8-af49bb1058db/ | 05:24 |
| Jeffrey4l | kolla_dockerhub_creds is defined in the zuul.yaml file. | 05:24 |
| ianw | vivsoni_: that's not rally a zuul issue ... you'll have more luck asking cinder people i'd say | 05:24 |
| *** rosmaita has quit IRC | 05:26 | |
| vivsoni_ | ianw: ok. I will check.. Thanks | 05:27 |
| ianw | Jeffrey4l: has it ever worked, or is this a new problem? | 05:27 |
| Jeffrey4l | guess secrets can not configure through projects.jobs.secrets? https://github.com/openstack/kolla/blob/master/.zuul.yaml#L51 | 05:28 |
| *** csomerville has quit IRC | 05:28 | |
| Jeffrey4l | it worked. but broken by this patch https://github.com/openstack/kolla/commit/9846a27a6b453d4591979653d7ac0d52f7930fc7 | 05:28 |
| *** csomerville has joined #openstack-infra | 05:28 | |
| Jeffrey4l | ianw, ^^ | 05:28 |
| *** gongysh has quit IRC | 05:30 | |
| *** SumitNaiksatam_ has quit IRC | 05:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/log-classify master: Add python project structure https://review.openstack.org/525858 | 05:31 |
| *** esberglu has joined #openstack-infra | 05:33 | |
| *** pgadiya has joined #openstack-infra | 05:34 | |
| *** hongbin_ has quit IRC | 05:34 | |
| *** esberglu has quit IRC | 05:38 | |
| ianw | Jeffrey4l: yeah, you want the secret in the job definition, not the "jobs:" list | 05:42 |
| openstackgerrit | Merged openstack/diskimage-builder master: zypper: fix package removal https://review.openstack.org/517658 | 05:42 |
| Jeffrey4l | ianw, got. thaks. | 05:42 |
| *** Hal has quit IRC | 05:44 | |
| *** janki has quit IRC | 05:44 | |
| *** yamamoto_ has joined #openstack-infra | 05:50 | |
| *** yamamoto has quit IRC | 05:53 | |
| *** sree has joined #openstack-infra | 05:54 | |
| *** janki has joined #openstack-infra | 05:56 | |
| *** Guest76573 has quit IRC | 05:57 | |
| *** Apoorva_ has quit IRC | 05:59 | |
| *** rocky-shiyan has joined #openstack-infra | 05:59 | |
| *** yamamoto has joined #openstack-infra | 06:04 | |
| *** yamamoto_ has quit IRC | 06:07 | |
| *** dhajare has joined #openstack-infra | 06:07 | |
| *** udesale__ has joined #openstack-infra | 06:10 | |
| *** udesale has quit IRC | 06:12 | |
| *** mat128_ has quit IRC | 06:13 | |
| frickler | infra-root: https://review.openstack.org/520660 seems to be blocking integrated gate for 15h now with two jobs not getting executed | 06:14 |
| *** dsariel has joined #openstack-infra | 06:14 | |
| *** ykarel_ has joined #openstack-infra | 06:16 | |
| *** rwsu has quit IRC | 06:17 | |
| *** ykarel has quit IRC | 06:19 | |
| *** nmathew has joined #openstack-infra | 06:21 | |
| *** xinliang has quit IRC | 06:27 | |
| *** aeng has quit IRC | 06:29 | |
| *** rwsu has joined #openstack-infra | 06:30 | |
| frickler | seems related to the stuck post jobs, the two that AJaeger_ mentioned earlier are still there and have a similar start time | 06:30 |
| *** snapiri1 has joined #openstack-infra | 06:31 | |
| *** CrayZee has joined #openstack-infra | 06:31 | |
| *** sree_ has joined #openstack-infra | 06:31 | |
| *** sree_ is now known as Guest78058 | 06:32 | |
| *** eumel8 has joined #openstack-infra | 06:34 | |
| *** sree has quit IRC | 06:35 | |
| *** janki has quit IRC | 06:36 | |
| *** caphrim007 has joined #openstack-infra | 06:37 | |
| *** janki has joined #openstack-infra | 06:38 | |
| *** xarses has quit IRC | 06:38 | |
| *** xarses has joined #openstack-infra | 06:39 | |
| *** xinliang has joined #openstack-infra | 06:39 | |
| *** caphrim007 has quit IRC | 06:43 | |
| *** ykarel_ has quit IRC | 06:47 | |
| *** ykarel_ has joined #openstack-infra | 06:47 | |
| *** pgadiya has quit IRC | 06:51 | |
| CrayZee | Hi Infra, I have two zuul jobs that seem to be in some limbo state (running for over 15 hours). Could anyone take a look and maybe kill/restart them? | 06:56 |
| CrayZee | both are from the project dragonflow | 06:57 |
| CrayZee | sorry, two more at the gate running for over 11 hours as well | 06:58 |
| *** dhinesh has quit IRC | 06:58 | |
| *** udesale has joined #openstack-infra | 06:58 | |
| *** eumel8 has quit IRC | 07:00 | |
| *** udesale__ has quit IRC | 07:00 | |
| *** rcernin has quit IRC | 07:02 | |
| frickler | CrayZee: yeah, I noticed that earlier, need to wait for someone with root access to investigate | 07:03 |
| *** xarses_ has joined #openstack-infra | 07:04 | |
| *** xarses_ has quit IRC | 07:04 | |
| *** xarses_ has joined #openstack-infra | 07:05 | |
| *** xarses has quit IRC | 07:05 | |
| *** e0ne has joined #openstack-infra | 07:06 | |
| *** ykarel_ is now known as ykarel | 07:10 | |
| *** csomerville has quit IRC | 07:10 | |
| *** jtomasek has joined #openstack-infra | 07:10 | |
| *** jtomasek has quit IRC | 07:10 | |
| *** jtomasek has joined #openstack-infra | 07:11 | |
| *** dhajare has quit IRC | 07:12 | |
| *** andreas__ has joined #openstack-infra | 07:12 | |
| RuiChen | Anyone know why devstack repo don't have newton-eol tag? newton EOL in 2017-10-11 | 07:16 |
| *** eumel8 has joined #openstack-infra | 07:16 | |
| *** eumel8 has quit IRC | 07:23 | |
| *** esberglu has joined #openstack-infra | 07:23 | |
| *** esberglu has quit IRC | 07:28 | |
| frickler | RuiChen: better ask in #openstack-qa, some projects were late in doing EOL for newton, not sure whether that situation still persists | 07:28 |
| *** HeOS has joined #openstack-infra | 07:29 | |
| *** gongysh has joined #openstack-infra | 07:31 | |
| RuiChen | frickler: thanks, hope someone know that in #openstack-qa | 07:32 |
| *** makowals has joined #openstack-infra | 07:32 | |
| *** jascott1 has joined #openstack-infra | 07:33 | |
| *** dsariel has quit IRC | 07:36 | |
| *** jascott1 has quit IRC | 07:37 | |
| *** rcernin has joined #openstack-infra | 07:39 | |
| *** dhajare has joined #openstack-infra | 07:40 | |
| *** yamamoto_ has joined #openstack-infra | 07:42 | |
| *** yamamoto has quit IRC | 07:46 | |
| *** slaweq_ has joined #openstack-infra | 07:47 | |
| *** shu-mutou has quit IRC | 07:48 | |
| *** kjackal has quit IRC | 07:49 | |
| *** hashar has joined #openstack-infra | 07:49 | |
| *** eumel8 has joined #openstack-infra | 07:52 | |
| ianw | i'm just seeing if anything stands out with these stuck jobs | 07:55 |
| ianw | Execute job ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa (uuid: e2c9e68591474903941f8b3874ebdbcd) on nodes <NodeSet devstack-single-node OrderedDict([('primary', <Node 0001228408 primary:ubuntu-xenial>)])OrderedDict()> for change <Change 0x7f04f98f4908 525634,1> | 07:56 |
| *** dhajare has quit IRC | 07:57 | |
| ianw | 2017-12-05 15:02:46,381 DEBUG nodepool.NodeLauncher-0001228408: Node 0001228408 is running [region: IAD, az: None, ip: 104.239.229.185 ipv4: 104.239.229.185, ipv6: 2001:4802:7803:104:be76:4eff:fe20:40b7] | 07:58 |
| ianw | 2017-12-05 15:44:33,471 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0001228408 (state: used, allocated_to: 200-0001391746) | 07:59 |
| ianw | so, the node came and went, zuul clearly hasn't noticed | 07:59 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: monasca-api: Exclude stable/ocata from sphinx doc creation https://review.openstack.org/525864 | 07:59 |
| ianw | i'm going to restart it, and requeue jobs | 08:01 |
| CrayZee | ianw: Thanks for investigating, but I see that ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa for change 525634 succeeded? | 08:02 |
| *** dingyichen has quit IRC | 08:02 | |
| *** ildikov has quit IRC | 08:03 | |
| *** neith has quit IRC | 08:03 | |
| *** samueldmq has quit IRC | 08:04 | |
| *** ildikov has joined #openstack-infra | 08:04 | |
| *** samueldmq has joined #openstack-infra | 08:05 | |
| ianw | oh oh, it's not restarting | 08:05 |
| *** neith has joined #openstack-infra | 08:05 | |
| *** knikolla has quit IRC | 08:06 | |
| *** jpmaxman has quit IRC | 08:06 | |
| *** dbecker has joined #openstack-infra | 08:07 | |
| *** knikolla has joined #openstack-infra | 08:07 | |
| *** andrewbogott has quit IRC | 08:07 | |
| *** yamahata has quit IRC | 08:07 | |
| *** mnaser has quit IRC | 08:08 | |
| *** zhongjun has quit IRC | 08:08 | |
| *** shardy has joined #openstack-infra | 08:08 | |
| *** csatari has quit IRC | 08:09 | |
| *** gmann has quit IRC | 08:09 | |
| *** jpmaxman has joined #openstack-infra | 08:09 | |
| *** andrewbogott has joined #openstack-infra | 08:09 | |
| *** lin_yang has quit IRC | 08:10 | |
| *** gongysh has quit IRC | 08:10 | |
| *** mnaser has joined #openstack-infra | 08:11 | |
| *** florianf|afk is now known as florianf | 08:12 | |
| ianw | :/ ... i think it was just the pid file that didn't get cleaned up properly blocking it starting | 08:12 |
| *** csatari has joined #openstack-infra | 08:12 | |
| ianw | rather non-obvious though, as "service zuul-scheduler start" exits "OK" in that case. maybe that is something we can look at to make sure it gives an error failure | 08:13 |
| *** electrical has quit IRC | 08:14 | |
| *** alexchadin has joined #openstack-infra | 08:14 | |
| ianw | ok, i'm requeueing | 08:15 |
| *** electrical has joined #openstack-infra | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:19 | |
| *** ykarel is now known as ykarel|lunch | 08:20 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 08:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove workaround for periodic jobs https://review.openstack.org/525541 | 08:21 |
| ianw | #status log zuul-scheduler restarted due to very high number of stuck jobs. check/gate/triple-o requeued | 08:21 |
| openstackstatus | ianw: finished logging | 08:21 |
| ianw | it is now Wed Dec 6 08:20:40 UTC 2017 ... there were quite a number of stuck changes. the 525634,1 example above was a good one, it should have ended at 12/05 15:44 but zuul never noticed | 08:23 |
| ianw | jeblair: ^ if investigation needed. other jobs in the queue in scripts at ~ianw/tmp/restart-2017-12-06 if that helps | 08:24 |
| *** ralonsoh has joined #openstack-infra | 08:24 | |
| *** hashar has quit IRC | 08:26 | |
| *** e0ne has quit IRC | 08:32 | |
| *** hashar has joined #openstack-infra | 08:33 | |
| *** zhurong has quit IRC | 08:33 | |
| *** shu-mutou has joined #openstack-infra | 08:34 | |
| ianw | mtreinish: fetch-stestr-output File "/usr/os-testr-env/bin/subunit2html", line 11, in <module> ... UnicodeDecodeError: 'ascii' codec can't decode byte; see http://paste.openstack.org/show/628250/ if of interest ... clearly something up | 08:34 |
| *** Hal has joined #openstack-infra | 08:40 | |
| chandankumar | AJaeger_: hello | 08:41 |
| chandankumar | AJaeger_: https://review.openstack.org/524869 please have a look | 08:41 |
| *** udesale has quit IRC | 08:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 08:44 |
| *** ramishra has quit IRC | 08:45 | |
| *** jtomasek has quit IRC | 08:45 | |
| *** martinkopec has joined #openstack-infra | 08:46 | |
| *** dsariel has joined #openstack-infra | 08:47 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Creating a function that calls /usr/bin/timeout https://review.openstack.org/525594 | 08:49 |
| *** zhurong has joined #openstack-infra | 08:53 | |
| *** slaweq has joined #openstack-infra | 08:55 | |
| *** yamamoto_ has quit IRC | 08:56 | |
| *** kjackal has joined #openstack-infra | 08:56 | |
| *** ramishra has joined #openstack-infra | 08:57 | |
| *** slaweq has quit IRC | 09:00 | |
| *** rossella_s has joined #openstack-infra | 09:01 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| frickler | chandankumar: afaict both of AJaeger's remarks in that review are not responded to yet | 09:04 |
| *** yamamoto has joined #openstack-infra | 09:04 | |
| chandankumar | frickler: commented | 09:05 |
| *** ykarel_ has joined #openstack-infra | 09:10 | |
| *** ykarel|lunch has quit IRC | 09:11 | |
| *** esberglu has joined #openstack-infra | 09:11 | |
| *** esberglu has quit IRC | 09:12 | |
| *** esberglu has joined #openstack-infra | 09:12 | |
| *** esberglu has quit IRC | 09:12 | |
| *** esberglu has joined #openstack-infra | 09:13 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: [WIP] Add support function for fast forward upgrade testing. https://review.openstack.org/526000 | 09:15 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: [WIP] Add support function for fast forward upgrade testing. https://review.openstack.org/526000 | 09:16 |
| *** esberglu has quit IRC | 09:17 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: [WIP] Add support function for fast forward upgrade testing. https://review.openstack.org/526000 | 09:18 |
| *** raissa has joined #openstack-infra | 09:19 | |
| *** armaan has joined #openstack-infra | 09:21 | |
| *** HeOS has quit IRC | 09:26 | |
| *** HeOS has joined #openstack-infra | 09:31 | |
| Wei_Liu | ianw: hello, I had setup a local CI/CD env with zuulv3, when openstack-tox-pep8/py27 had been triggered, I found out that my new change in gerrit was not fetch to worker repo. Do you know how zuul fetch new change? | 09:31 |
| *** armaan has quit IRC | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** zhongjun has joined #openstack-infra | 09:35 | |
| *** d0ugal has quit IRC | 09:37 | |
| *** derekh has joined #openstack-infra | 09:39 | |
| *** d0ugal has joined #openstack-infra | 09:41 | |
| *** bhavik1 has joined #openstack-infra | 09:45 | |
| *** e0ne has joined #openstack-infra | 09:48 | |
| *** andreas_s has joined #openstack-infra | 09:53 | |
| *** electrofelix has joined #openstack-infra | 09:53 | |
| *** armaan has joined #openstack-infra | 09:53 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz data https://review.openstack.org/525603 | 09:54 |
| *** andreas__ has quit IRC | 09:56 | |
| *** linkmark has joined #openstack-infra | 09:57 | |
| *** sdague has joined #openstack-infra | 09:59 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** gmann has joined #openstack-infra | 10:05 | |
| *** rossella_s has quit IRC | 10:07 | |
| *** rossella_s has joined #openstack-infra | 10:08 | |
| *** yamamoto has joined #openstack-infra | 10:08 | |
| *** lin_yang has joined #openstack-infra | 10:08 | |
| *** ccamacho has joined #openstack-infra | 10:11 | |
| *** cuongnv has quit IRC | 10:12 | |
| *** yamamoto has quit IRC | 10:13 | |
| *** rossella_s has quit IRC | 10:13 | |
| *** annp has quit IRC | 10:14 | |
| *** andreas_s has quit IRC | 10:14 | |
| *** andreas_s has joined #openstack-infra | 10:15 | |
| *** andreas_s has quit IRC | 10:15 | |
| *** andreas_s has joined #openstack-infra | 10:15 | |
| *** kiennt26 has quit IRC | 10:15 | |
| *** rossella_s has joined #openstack-infra | 10:16 | |
| *** armaan has quit IRC | 10:16 | |
| *** dizquierdo has joined #openstack-infra | 10:22 | |
| *** Qiming has quit IRC | 10:24 | |
| *** namnh has quit IRC | 10:25 | |
| *** ldnunes has joined #openstack-infra | 10:28 | |
| *** bhavik1 has quit IRC | 10:29 | |
| *** Qiming has joined #openstack-infra | 10:30 | |
| *** katkapilatova has joined #openstack-infra | 10:30 | |
| *** raissa_ has joined #openstack-infra | 10:32 | |
| *** sambetts|afk is now known as sambetts | 10:32 | |
| *** raissa_ has quit IRC | 10:33 | |
| *** ganso has joined #openstack-infra | 10:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 10:34 |
| *** tosky has joined #openstack-infra | 10:37 | |
| *** zhurong has quit IRC | 10:37 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:39 | |
| *** d0ugal has quit IRC | 10:39 | |
| *** jtomasek has joined #openstack-infra | 10:40 | |
| zigo | What's that in tempest? | 10:40 |
| zigo | # Path to the script heat-config-notify (string value) | 10:40 |
| zigo | #heat_config_notify_script = heat-config-notify | 10:40 |
| zigo | Where may I find such a script? | 10:40 |
| zigo | Also, it'd be super nice if someone could provide to me a tempest.conf that works for heat. | 10:41 |
| zigo | I'm having a hard time configuring it. | 10:41 |
| *** dizquierdo has quit IRC | 10:46 | |
| *** hashar is now known as hasharAwayu | 10:46 | |
| *** hasharAwayu is now known as hasharAway | 10:46 | |
| *** alexchadin has quit IRC | 10:48 | |
| *** cshastri has quit IRC | 10:50 | |
| *** andreas_s has quit IRC | 10:52 | |
| *** adarazs is now known as adarazs_afk | 10:52 | |
| *** andreas_s has joined #openstack-infra | 10:53 | |
| *** sboyron has joined #openstack-infra | 10:54 | |
| *** hasharAway has quit IRC | 10:55 | |
| *** udesale has joined #openstack-infra | 10:55 | |
| tosky | AJaeger_: hi, would this kind of fix be acceptable for now? https://review.openstack.org/#/c/525251/ | 10:57 |
| *** hashar has joined #openstack-infra | 11:00 | |
| *** esberglu has joined #openstack-infra | 11:00 | |
| *** d0ugal has joined #openstack-infra | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Put test-requirements into an extra named 'test' https://review.openstack.org/492619 | 11:01 |
| *** andreas_s has quit IRC | 11:01 | |
| *** tosky_ has joined #openstack-infra | 11:03 | |
| *** andreas_s has joined #openstack-infra | 11:03 | |
| *** tosky has quit IRC | 11:03 | |
| *** tosky_ is now known as tosky | 11:03 | |
| *** esberglu has quit IRC | 11:04 | |
| *** caphrim007 has joined #openstack-infra | 11:06 | |
| *** ykarel_ has quit IRC | 11:06 | |
| *** ykarel_ has joined #openstack-infra | 11:06 | |
| *** sboyron has quit IRC | 11:08 | |
| *** caphrim007 has quit IRC | 11:10 | |
| *** andreas_s has quit IRC | 11:11 | |
| *** rfolco has joined #openstack-infra | 11:16 | |
| *** tosky has quit IRC | 11:16 | |
| *** andreas_s has joined #openstack-infra | 11:18 | |
| *** yamamoto has quit IRC | 11:19 | |
| *** udesale has quit IRC | 11:20 | |
| *** ykarel_ is now known as ykarel | 11:25 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Create a test matrix role https://review.openstack.org/524402 | 11:27 |
| *** ykarel has quit IRC | 11:29 | |
| *** ykarel has joined #openstack-infra | 11:29 | |
| *** andreas_s has quit IRC | 11:32 | |
| *** andreas_s has joined #openstack-infra | 11:32 | |
| *** ykarel_ has joined #openstack-infra | 11:34 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: tests: Increase coverage of requirements parsing https://review.openstack.org/518656 | 11:34 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Move packaging tests to test_packaging https://review.openstack.org/526036 | 11:34 |
| *** links has quit IRC | 11:34 | |
| *** andreas_s has quit IRC | 11:35 | |
| *** andreas_s has joined #openstack-infra | 11:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add new repo for mistral tempest plugin https://review.openstack.org/524869 | 11:36 |
| *** ykarel has quit IRC | 11:37 | |
| frickler | zigo: you might get more answers in #openstack-qa (for tempest) or #heat | 11:37 |
| *** rcernin has quit IRC | 11:39 | |
| zigo | frickler: Ok, thanks. | 11:39 |
| *** rosmaita has joined #openstack-infra | 11:40 | |
| frickler | ianw: infra-root: seems after your restart there is another job now in similar state at the top of integrated gate: https://review.openstack.org/519643 | 11:43 |
| *** wolverineav has joined #openstack-infra | 11:43 | |
| AJaeger_ | frickler: that still makes progress, doesn't it? http://zuulv3.openstack.org/static/stream.html?uuid=7ac3d6e24ce94b7ca43e21edb4b310ad&logfile=console.log | 11:47 |
| *** andreas_s has quit IRC | 11:48 | |
| *** links has joined #openstack-infra | 11:48 | |
| *** andreas_s has joined #openstack-infra | 11:48 | |
| frickler | zigo: for your second question, maybe this sample config from heat gate tests can help you? http://logs.openstack.org/43/521643/1/check/heat-functional-convg-mysql-lbaasv2/a864067/logs/tempest_conf.txt.gz | 11:48 |
| AJaeger_ | frickler, dmsimard thanks for volunteering as infra-root! | 11:48 |
| *** alexchadin has joined #openstack-infra | 11:49 | |
| *** janki has quit IRC | 11:52 | |
| *** smatzek has joined #openstack-infra | 11:53 | |
| *** dtantsur|afk is now known as dtantsur | 11:53 | |
| *** kiennt26 has joined #openstack-infra | 11:54 | |
| *** nmathew has quit IRC | 11:54 | |
| *** rossella_s has quit IRC | 11:55 | |
| *** rossella_s has joined #openstack-infra | 11:56 | |
| frickler | AJaeger_: oh, now that job is scheduled, it still was queued when I looked. so what we are seeing is simply backlog | 11:57 |
| AJaeger_ | ah... | 11:58 |
| *** jkilpatr has joined #openstack-infra | 12:00 | |
| Wei_Liu | hello, I had setup a local CI/CD env with zuulv3, when openstack-tox-pep8/py27 had been triggered, I found out that my new change in gerrit was not fetch to worker repo. And I did not find a job who did this in project project-config, zuul-jobs and openstack-zuul-jobs. So, how could zuul get the gerrit change during executing jobs? | 12:01 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Run monascaclient docs jobs on master only https://review.openstack.org/526043 | 12:02 |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** yamamoto has joined #openstack-infra | 12:05 | |
| *** openstackgerrit has joined #openstack-infra | 12:06 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Run monascaclient docs jobs on master only https://review.openstack.org/526043 | 12:06 |
| *** raissa has quit IRC | 12:07 | |
| *** links has quit IRC | 12:08 | |
| *** kiennt26 has quit IRC | 12:11 | |
| *** links has joined #openstack-infra | 12:13 | |
| *** dave-mccowan has joined #openstack-infra | 12:13 | |
| *** alexchadin has quit IRC | 12:16 | |
| *** alexchadin has joined #openstack-infra | 12:16 | |
| *** rossella_s has quit IRC | 12:17 | |
| *** rossella_s has joined #openstack-infra | 12:17 | |
| *** bobh has joined #openstack-infra | 12:18 | |
| *** adarazs_afk is now known as adarazs | 12:27 | |
| *** tosky has joined #openstack-infra | 12:28 | |
| *** Wei_Liu has quit IRC | 12:28 | |
| *** panda is now known as panda|lnch | 12:34 | |
| *** Goneri has quit IRC | 12:37 | |
| *** bhavik1 has joined #openstack-infra | 12:43 | |
| *** lucasagomes is now known as lucas-hungry | 12:44 | |
| *** slaweq_ has quit IRC | 12:45 | |
| *** efried_cya_wed is now known as efried | 12:46 | |
| *** janki has joined #openstack-infra | 12:46 | |
| *** bhavik1 has quit IRC | 12:46 | |
| dmsimard | AJaeger_: wait, volunteering ? No one told me it was a volunteer thing! (just kidding) | 12:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 12:51 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/openstack-zuul-jobs master: Add heat-tempest-plugin to grenade/sahara job https://review.openstack.org/525251 | 12:53 |
| *** tesseract has quit IRC | 12:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 12:55 |
| *** nmathew has joined #openstack-infra | 12:55 | |
| cmurphy | I've noticed a couple of times our unit tests time out after about 40 minutes - http://logs.openstack.org/05/523005/2/gate/openstack-tox-py27/b23aa06/job-output.txt.gz | 12:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 12:55 |
| cmurphy | they take a long time but they don't usually take THAT long, the python3 unit tests for the same patch are fine | 12:56 |
| cmurphy | is there something weird i should look for? | 12:56 |
| *** tesseract has joined #openstack-infra | 12:57 | |
| *** rossella_s has quit IRC | 12:59 | |
| *** bhavik1 has joined #openstack-infra | 13:01 | |
| *** rossella_s has joined #openstack-infra | 13:01 | |
| *** hughhalf has quit IRC | 13:03 | |
| cmurphy | here's another one, they both happened to run on vexxhost http://logs.openstack.org/35/484235/3/gate/openstack-tox-py27/0d971c4/job-output.txt.gz | 13:05 |
| *** gmann is now known as gmann_afk | 13:05 | |
| *** hughhalf has joined #openstack-infra | 13:05 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Creating a function that calls /usr/bin/timeout https://review.openstack.org/525594 | 13:07 |
| *** bhavik1 has quit IRC | 13:08 | |
| *** armaan has joined #openstack-infra | 13:09 | |
| *** verdurin has quit IRC | 13:12 | |
| frickler | AJaeger_: you broke designate ;) https://review.openstack.org/522640 http://logs.openstack.org/53/516753/3/check/requirements-check/75960e5/job-output.txt.gz#_2017-12-05_21_14_43_186747 | 13:13 |
| *** dprince has joined #openstack-infra | 13:14 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 13:14 |
| *** jkilpatr has quit IRC | 13:16 | |
| *** rossella_s has quit IRC | 13:17 | |
| *** dtantsur is now known as dtantsur|lunch | 13:18 | |
| *** armaan has quit IRC | 13:20 | |
| *** rhallisey has joined #openstack-infra | 13:20 | |
| *** rossella_s has joined #openstack-infra | 13:20 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 13:20 | |
| *** trown|outtypewww is now known as trown | 13:22 | |
| *** links has quit IRC | 13:23 | |
| *** markvoelker has quit IRC | 13:24 | |
| *** markvoelker has joined #openstack-infra | 13:25 | |
| *** verdurin has joined #openstack-infra | 13:25 | |
| AJaeger_ | frickler: Indeed, I did - in a way... | 13:27 |
| frickler | cmurphy: those logs look pretty normal to me, but three times the usual runtime is very slow indeed. not sure what to do with it, though, probably need to wait for more evidence | 13:28 |
| *** rlandy has joined #openstack-infra | 13:28 | |
| *** ykarel_ has quit IRC | 13:28 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 13:28 |
| *** rfolco is now known as rfolco|afk | 13:29 | |
| frickler | AJaeger_: I'm not sure though whether the test if broken or whether the requirements should be specified differently | 13:29 |
| frickler | s/if/is | 13:30 |
| *** ykarel_ has joined #openstack-infra | 13:30 | |
| *** Goneri has joined #openstack-infra | 13:30 | |
| AJaeger_ | frickler: just remove doc/requirements.txt | 13:31 |
| AJaeger_ | look at the content, it's not really needed. | 13:31 |
| *** bobh has quit IRC | 13:31 | |
| mordred | frickler: https://review.openstack.org/#/c/516753 is a very large patch :) | 13:31 |
| AJaeger_ | if you need help, tell me and I'll fix later... | 13:31 |
| AJaeger_ | mordred: check http://git.openstack.org/cgit/openstack/designate/tree/doc/requirements.txt - another item for our "obscure sphinx cases cabinett" | 13:32 |
| cmurphy | frickler: is it normal for zuul to kill it after 40 minutes? i had thought the timeout was 90 minutes but maybe that's historical or i'm misremembering | 13:32 |
| mordred | AJaeger_: yah - that one's fun | 13:33 |
| frickler | cmurphy: I think 40 minutes is the usual timeout now for the new tox-py jobs, let me check | 13:33 |
| *** rfolco|afk is now known as real_jesus | 13:34 | |
| frickler | mordred: the file causing the failure now hasn't been touched since 2014 | 13:34 |
| *** real_jesus is now known as rfolco | 13:34 | |
| mordred | frickler: yah - adding such a doc/requirements.txt would be a good way to get a repo to work on both infra and readthedocs up until now ... | 13:35 |
| mordred | the new way to support doing that is to shift doc requirements out of test-reuqirements and in to doc/requirements.txt - so that unittests don't need to install sphinx, and doc builds don't need all the unittest stuff... | 13:36 |
| mordred | BUT ... we're in a slightly awkward in-between place | 13:36 |
| mordred | and, of course, by slightly awkward, I mean AJaeger_ are losing our minds :) | 13:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 13:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pub route https://review.openstack.org/502530 | 13:37 |
| frickler | cmurphy: 30 minutes is the default timeout from the base job. I think it should be possible to increase that for your project when needed, but a couple of runs I just checked are in the 10 minute range | 13:39 |
| *** dtantsur|lunch is now known as dtantsur | 13:40 | |
| cmurphy | frickler: okay good to know, and yeah i would not want to have to increase the timeout for py27 unit tests | 13:40 |
| *** bobh has joined #openstack-infra | 13:40 | |
| *** mriedem has joined #openstack-infra | 13:41 | |
| cmurphy | will come back with more evidence if it keeps happening | 13:41 |
| *** nmathew- has joined #openstack-infra | 13:42 | |
| *** jkilpatr has joined #openstack-infra | 13:43 | |
| *** esberglu has joined #openstack-infra | 13:44 | |
| *** nmathew has quit IRC | 13:45 | |
| *** alexchadin has quit IRC | 13:45 | |
| *** psachin has quit IRC | 13:45 | |
| *** bobh has quit IRC | 13:45 | |
| *** esberglu has quit IRC | 13:48 | |
| *** rossella_s has quit IRC | 13:49 | |
| *** lucas-hungry is now known as lucasagomes | 13:49 | |
| *** rossella_s has joined #openstack-infra | 13:49 | |
| *** ykarel_ is now known as ykarel|afk | 13:53 | |
| *** wolverineav has quit IRC | 13:56 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Undercloud deploy: pass undercloud.yaml after docker.yaml https://review.openstack.org/526076 | 13:59 |
| *** nmathew- has quit IRC | 14:01 | |
| *** hamzy has quit IRC | 14:02 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 14:02 | |
| *** bobh has joined #openstack-infra | 14:03 | |
| *** rossella_s has quit IRC | 14:03 | |
| *** coolsvap has quit IRC | 14:04 | |
| *** rossella_s has joined #openstack-infra | 14:05 | |
| *** camunoz has joined #openstack-infra | 14:06 | |
| *** bobh has quit IRC | 14:07 | |
| *** dsariel has quit IRC | 14:08 | |
| *** sree has joined #openstack-infra | 14:08 | |
| *** ramishra has quit IRC | 14:11 | |
| *** adriant has quit IRC | 14:11 | |
| *** adriant has joined #openstack-infra | 14:11 | |
| *** udesale has joined #openstack-infra | 14:12 | |
| *** Guest78058 has quit IRC | 14:12 | |
| *** pbourke_ is now known as pbourke | 14:12 | |
| *** david-lyle has joined #openstack-infra | 14:13 | |
| *** slaweq_ has joined #openstack-infra | 14:13 | |
| *** bobh has joined #openstack-infra | 14:15 | |
| *** wolverineav has joined #openstack-infra | 14:16 | |
| *** armax has quit IRC | 14:17 | |
| *** dave-mccowan has quit IRC | 14:17 | |
| *** kgiusti has joined #openstack-infra | 14:18 | |
| *** bobh has quit IRC | 14:20 | |
| *** bobh has joined #openstack-infra | 14:22 | |
| *** signed8bit has quit IRC | 14:24 | |
| *** sree_ has joined #openstack-infra | 14:24 | |
| *** signed8bit has joined #openstack-infra | 14:25 | |
| *** sree_ is now known as Guest31633 | 14:25 | |
| *** sree has quit IRC | 14:26 | |
| *** smatzek has quit IRC | 14:26 | |
| *** esberglu has joined #openstack-infra | 14:27 | |
| *** bobh has quit IRC | 14:27 | |
| *** ykarel_ has joined #openstack-infra | 14:28 | |
| *** Guest31633 has quit IRC | 14:29 | |
| AJaeger_ | mordred: I'm recovering again slowly ... ;) | 14:30 |
| *** andreas_s has quit IRC | 14:31 | |
| *** ykarel|afk has quit IRC | 14:31 | |
| *** david-lyle has quit IRC | 14:31 | |
| *** andreas_s has joined #openstack-infra | 14:31 | |
| *** salv-orlando has joined #openstack-infra | 14:35 | |
| *** andreas_s has quit IRC | 14:36 | |
| *** signed8bit has quit IRC | 14:37 | |
| *** tesseract has quit IRC | 14:37 | |
| *** hamzy has joined #openstack-infra | 14:42 | |
| *** andreas_s has joined #openstack-infra | 14:42 | |
| *** bobh has joined #openstack-infra | 14:43 | |
| pabelanger | cmurphy: frickler: it is possible they are slow compute nodes in vexxhost. We've seen noisy neighbours before there | 14:43 |
| *** tesseract has joined #openstack-infra | 14:44 | |
| *** slaweq_ has quit IRC | 14:44 | |
| *** esberglu has quit IRC | 14:44 | |
| *** Wei_Liu has joined #openstack-infra | 14:45 | |
| dmsimard | mordred: so I have a shade question... why do we detach the floating IP before deleting a server ? Deleting the server without detaching it first yields the same result and is more "atomic" ? | 14:45 |
| dmsimard | mordred: I ask because of what is more than likely a race that we've discussed a while back, i.e: http://logs.rdoproject.org/nodepool_floating.txt | 14:46 |
| dmsimard | (that's v2 btw) | 14:46 |
| *** gouthamr has joined #openstack-infra | 14:46 | |
| *** rossella_s has quit IRC | 14:47 | |
| *** andreas_s has quit IRC | 14:48 | |
| *** bobh has quit IRC | 14:48 | |
| *** rossella_s has joined #openstack-infra | 14:49 | |
| *** armax has joined #openstack-infra | 14:49 | |
| *** raissa_ has joined #openstack-infra | 14:50 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add stackdumphandler to zuul-web https://review.openstack.org/526086 | 14:50 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Undercloud deploy: use docker-uc.yaml https://review.openstack.org/526076 | 14:52 |
| *** andreas_s has joined #openstack-infra | 14:52 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Creating a function that calls /usr/bin/timeout https://review.openstack.org/525594 | 14:52 |
| *** hamzy has quit IRC | 14:53 | |
| *** andreas_s has quit IRC | 14:53 | |
| *** andreas_s has joined #openstack-infra | 14:54 | |
| *** bobh has joined #openstack-infra | 14:54 | |
| *** raissa_ has quit IRC | 14:54 | |
| *** hamzy has joined #openstack-infra | 14:55 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Undercloud deploy: use docker-uc.yaml https://review.openstack.org/526076 | 14:56 |
| *** hamzy has quit IRC | 14:58 | |
| *** esberglu has joined #openstack-infra | 14:59 | |
| *** bobh has quit IRC | 14:59 | |
| *** mat128 has joined #openstack-infra | 14:59 | |
| *** hamzy has joined #openstack-infra | 15:00 | |
| *** ykarel_ is now known as ykarel | 15:01 | |
| *** raissa has joined #openstack-infra | 15:02 | |
| *** armaan has joined #openstack-infra | 15:03 | |
| *** salv-orlando has quit IRC | 15:03 | |
| *** owalsh_ has joined #openstack-infra | 15:03 | |
| *** owalsh has quit IRC | 15:05 | |
| *** caphrim007 has joined #openstack-infra | 15:06 | |
| *** hamzy has quit IRC | 15:06 | |
| *** ramishra has joined #openstack-infra | 15:07 | |
| *** salv-orlando has joined #openstack-infra | 15:10 | |
| *** caphrim007 has quit IRC | 15:10 | |
| *** dsariel has joined #openstack-infra | 15:11 | |
| weshay | anyone know if the rss feeds from openstack health are working properly? | 15:12 |
| weshay | about a week ago.. they started behaving a little odd. In that I was getting alerts from 5-7 prior to the actual job | 15:12 |
| weshay | EmilienM, FYI ^ | 15:12 |
| *** salv-orlando has quit IRC | 15:13 | |
| *** panda|lnch is now known as panda | 15:13 | |
| *** hamzy has joined #openstack-infra | 15:13 | |
| EmilienM | weshay: if you don't get RSS updates, it means no failures | 15:13 |
| *** bobh has joined #openstack-infra | 15:13 | |
| EmilienM | weshay: no failures == we probably have a bug in our subunit file generation | 15:13 |
| EmilienM | weshay: I remember sshnaidm|rover had a patch, I haven't checked if it worked | 15:14 |
| weshay | EmilienM, no I was getting rss updates/alerts on jobs from several days ago.. 5-7 days | 15:14 |
| EmilienM | when did we change the subunit file thing? | 15:14 |
| weshay | EmilienM, we should be seeing alerts on scen002 afaict | 15:14 |
| *** hamzy has quit IRC | 15:14 | |
| * weshay looks | 15:14 | |
| sshnaidm|rover | EmilienM, weshay maybe it doesn't work, worth to check | 15:14 |
| sshnaidm|rover | I mean this predefined fake subunit file | 15:14 |
| EmilienM | weshay: 9 days ago https://review.openstack.org/#/c/522955/ | 15:15 |
| EmilienM | so yeah, it might be that one | 15:15 |
| *** armaan has quit IRC | 15:16 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove workaround for periodic jobs https://review.openstack.org/525541 | 15:17 |
| *** janki has quit IRC | 15:17 | |
| *** hamzy has joined #openstack-infra | 15:17 | |
| *** bobh has quit IRC | 15:17 | |
| *** bobh has joined #openstack-infra | 15:18 | |
| *** hamzy has quit IRC | 15:18 | |
| *** dsariel has quit IRC | 15:18 | |
| *** ricolin has joined #openstack-infra | 15:20 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added initial jobs for mistral tempest plugin https://review.openstack.org/524952 | 15:21 |
| chandankumar | frickler: ^^ | 15:23 |
| ricolin | Hi I have a IRC issue in heat channel, since our meeting last week been cut off by ChanServ when it regain voice to `openstack`, our channel topic stays in the Meeting topic and won't get refreshed even we have another meeting. Anyone got any clue? | 15:24 |
| pabelanger | mordred: I'm helping debug an issue with nodepool in rdoproject with dmsimard, and wanted to ask about api-timeout. It looks like shade is sending a request to the cloud, but doesn't get anything back. Reading docs, I think api-timeout will help with that? looking at our nodepool.yaml file we seem to have it set to 60 | 15:24 |
| dmsimard | pabelanger: this is where you found the default ? https://github.com/openstack/os-client-config/blob/5209b3089a80118d79266c95fa3de922cf3ff473/os_client_config/defaults.py#L41 | 15:25 |
| fungi | if any config-core reviewers are on hand to approve AJaeger_'s 525769 change to fix the release jobs for projects running legacy-npm-upload i'll reenqueue the failing tag from yesterday for monasca-grafana-datasource | 15:25 |
| ricolin | checkout #heat if you wondering what I'm talking about | 15:25 |
| *** rossella_s has quit IRC | 15:25 | |
| smcginnis | fungi: Thanks for raising that, I literally just switched over here to look in to its status. | 15:26 |
| fungi | smcginnis: clearly we're in sync today | 15:26 |
| pabelanger | dmsimard: yah, I think that is it | 15:26 |
| *** david-lyle has joined #openstack-infra | 15:26 | |
| dmsimard | pabelanger: so if it's osc-config it means we can put it in clouds.yaml afaik, just not sure of the format (what key? in which dict?) | 15:27 |
| *** rossella_s has joined #openstack-infra | 15:28 | |
| *** rbrndt has joined #openstack-infra | 15:28 | |
| *** kiennt26 has joined #openstack-infra | 15:28 | |
| pabelanger | dmsimard: right, and it will be easy to test, you could create a new clouds.yaml file for rdocloud, then add api-timeout: 60, as a key under name of cloud | 15:28 |
| dmsimard | pabelanger: looks like just a key for the cloud, yeah | 15:29 |
| dmsimard | well, actually it's easy to test | 15:29 |
| pabelanger | dmsimard: http://paste.openstack.org/show/628284/ I think is syntax | 15:30 |
| *** smatzek has joined #openstack-infra | 15:30 | |
| dmsimard | I'll test locally on my laptop and just drop packets | 15:30 |
| *** smatzek has quit IRC | 15:30 | |
| pabelanger | cool | 15:30 |
| *** smatzek has joined #openstack-infra | 15:30 | |
| *** owalsh_ is now known as owalsh | 15:32 | |
| Wei_Liu | hello, I had setup a local CI/CD env with zuulv3, when openstack-tox-pep8/py27 had been triggered, I found out that my new change in gerrit was not fetch to worker repo. And I did not find a job who did this in project project-config, zuul-jobs and openstack-zuul-jobs. So, how could zuul get the gerrit change during executing jobs? | 15:32 |
| *** liusheng has quit IRC | 15:33 | |
| fungi | Wei_Liu: in zuul v3, the zuul-executor pushes a copy of the repository onto the job node's filesystem, checked out to the merge precalculated for testing | 15:34 |
| fungi | jobs generally don't (need to) clone/pull/fetch/checkout anything in most cases with zuul v3 | 15:34 |
| dmsimard | pabelanger: ok, it works with shade but openstackclient doesn't seem to care about it | 15:34 |
| dmsimard | ¯\_(ツ)_/¯ | 15:34 |
| *** david-lyle has quit IRC | 15:34 | |
| mtreinish | ianw: normally when I see that there is something wrong with the subunit output | 15:35 |
| mtreinish | ianw: I'll pull the subunit file from the log and take a look | 15:36 |
| dmsimard | pabelanger: wow cloud.list_servers() is a huge call | 15:36 |
| *** andrewbogott has quit IRC | 15:36 | |
| *** andrewbogott has joined #openstack-infra | 15:36 | |
| *** udesale has quit IRC | 15:36 | |
| pabelanger | dmsimard: okay, good. So, it is the right setting. | 15:37 |
| *** Apoorva has joined #openstack-infra | 15:37 | |
| fungi | Wei_Liu: you may want to look over https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#git-repositories | 15:37 |
| *** salv-orlando has joined #openstack-infra | 15:37 | |
| *** hemna_ has joined #openstack-infra | 15:37 | |
| pabelanger | Wei_Liu: what does your base job look like? In it, it should be using a role called prepare-workspace, that actually syncs the repo to the remote node | 15:38 |
| mtreinish | ianw: err, or I would if the subunit were there :) | 15:38 |
| mtreinish | I guess failing to generate the html made things bail out before it got to copying the subunit | 15:38 |
| fungi | pabelanger: i suppose that's what we mean in the user guide where it says "The base job in the standard library contains a pre-playbook which copies the repositories to all of the job’s nodes. It is recommended to always inherit from this base job to ensure that behavior." | 15:39 |
| mwhahaha | question for infra folks, what can we do to prevent 6 hour TIMED_OUT jobs? http://logs.openstack.org/68/525268/2/check/tripleo-ci-centos-7-containers-multinode/aabeaf0/ | 15:40 |
| Wei_Liu | fungi: the copy of the repository contains the merged code, right? how did the zuul executor checkout out to the merge precalculated for testing? | 15:40 |
| mwhahaha | i assume this is related to the 3 hour limit on the post tasks but something like this in the gate is a problem | 15:40 |
| fungi | Wei_Liu: the zuul-merger process builds the branch state(s) needed and then hands that off to the zuul-executor | 15:41 |
| pabelanger | fungi: yah, exactly. I'm glad we added that to docs :) | 15:41 |
| chandankumar | frickler: https://review.openstack.org/#/c/524952/ what i am doing wrong in this review? | 15:41 |
| fungi | pabelanger: is that section of the user guide correct? it looks like our "standard library" has a comment here which says "Assumes a 'base' job defined elsewhere" http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/zuul.yaml#n2 | 15:42 |
| *** trown is now known as trown|brb | 15:42 | |
| fungi | pabelanger: do we actually include a "base" job in our standard library, or expect you to carry a copy of the one from our project-config repo? | 15:42 |
| Wei_Liu | pabelanger: I use the base job defined in project-config, there is no prepare-workspace role | 15:43 |
| pabelanger | mwhahaha: there is a known bug in zuulv3 where we use the job timeout in each phase of runs / post-run, so if you have 3 hours for job timeout, it will stack up to 6. You can see http://logs.openstack.org/68/525268/2/check/tripleo-ci-centos-7-containers-multinode/aabeaf0/job-output.txt.gz#_2017-12-06_12_26_01_179916 | 15:43 |
| pabelanger | mwhahaha: obviously, we need to fix that | 15:43 |
| mwhahaha | pabelanger: i know but can we detect the node going away faster so we don't hang for 6 hours? | 15:43 |
| fungi | pabelanger: not just each phase, but each playbook too | 15:43 |
| pabelanger | fungi: yah, base is in zuul-jobs, but if a user shadows zuul-jobs, they could create a new base, which might not have it. In this case, a job-output.txt would be helpful | 15:44 |
| pabelanger | Wei_Liu: are you playbooks online some place? | 15:44 |
| pabelanger | Wei_Liu: you could use base from zuul-jobs, and not project-config | 15:44 |
| *** ykarel is now known as ykarel|afk | 15:45 | |
| fungi | pabelanger: where in zuul-jobs is a job named "base" defined? i'm really not finding it | 15:45 |
| *** salv-orlando has quit IRC | 15:45 | |
| pabelanger | mwhahaha: right now, I don't think so. You could check zuul.success in your post-run and still tasks you know that will fail. This looks to be a networking issue in our cloud provider | 15:45 |
| pabelanger | mwhahaha: inap? | 15:46 |
| Wei_Liu | pabelanger: no, my playbooks are behind the firewall | 15:46 |
| pabelanger | yah, so we can ask mgagne to maybe look and see what happen | 15:46 |
| dmsimard | mordred: there's a ginormous amount of calls when doing a cloud.list_servers() in shade around floating IPs. Looks like this, but it keeps going for like a minute.. there's around ~75 floating IPs in use in that tenant. Is that legit ? Looks like this: http://paste.openstack.org/raw/628286/ | 15:46 |
| mwhahaha | pabelanger: k just bringing it up because it causes problems | 15:47 |
| pabelanger | fungi: Oh, we removed it | 15:47 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul-jobs/commit/zuul.yaml?id=1a4f268af3d4be512fd793fc91affb1caf759fc5 | 15:47 |
| fungi | pabelanger: mwhahaha: i've heard it suggested that ansible should be able to figure out it's not connecting/getting any response and terminate the task in a reasonable amount of time rather than waiting for the general timeout to be reached, but so far nobody's stepped up with a solution | 15:47 |
| *** dave-mccowan has joined #openstack-infra | 15:47 | |
| fungi | pabelanger: we should probably update the user guide then to no longer suggest we include any "base" job in the standard library | 15:48 |
| dmsimard | yeah, ansible should be able to do that | 15:48 |
| pabelanger | fungi: yah, so I think we need to include an example of what base is. I am not sure we do right now | 15:48 |
| Wei_Liu | pabelanger: but there is no base playbook in zuul-jobs | 15:48 |
| fungi | pabelanger: i can embed a literal copy of our base job into that section of the user guide if you think that'll be sufficient? | 15:48 |
| jeblair | we need to add base back to zuul-jobs or add it to zuul-base-jobs, so there is a base job folks can use | 15:49 |
| pabelanger | Wei_Liu: yah, I am incorrect. It looks like I am giving wrong info. So what does your base/pre.yaml file look like for pre-run | 15:49 |
| fungi | jeblair: would putting a job called base into zuul-jobs in any way shadow/break the one we have in project-config? shouldn't right? | 15:50 |
| *** trown|brb is now known as trown | 15:50 | |
| jeblair | fungi: zuul-jobs is allowed to shadow project-config, so it will just be ignored by us (we'll be "overriding" it) | 15:50 |
| jeblair | fungi: oh, but the commit message at http://git.openstack.org/cgit/openstack-infra/zuul-jobs/commit/zuul.yaml?id=1a4f268af3d4be512fd793fc91affb1caf759fc5 still applies | 15:51 |
| jeblair | so we may need to go down the zuul-base-jobs path | 15:51 |
| pabelanger | yah | 15:52 |
| fungi | do we expect to provide multiple base jobs in zuul-base-jobs (i'm supposing so from the plurality of its name)? | 15:52 |
| pabelanger | maybe, but I think base jobs need to be called 'base', so not sure how to express that | 15:53 |
| Wei_Liu | pabelanger: http://paste.openstack.org/show/628287/, I commends the mirror roles since it always failed. | 15:54 |
| fungi | and is the idea that people could fork zuul-base-jobs to use as their config repository, or that they would choose whether to keep their base job in a config repository vs include the zuul-base-jobs repo? | 15:54 |
| pabelanger | Wei_Liu: yah, okay, I wouldn't expect that to work for you, unless you have cached git repos in your DIBs. | 15:54 |
| pabelanger | Wei_Liu: if you only have add-build-sshkeys, prepare-workspace, will it work? | 15:55 |
| pabelanger | Wei_Liu: http://paste.openstack.org/show/628288/ | 15:56 |
| pabelanger | that should be the minimal base to use | 15:56 |
| *** ianychoi has quit IRC | 15:56 | |
| *** ianychoi_ has joined #openstack-infra | 15:56 | |
| fungi | i'll move my base jobs design questions to #zuul for a (slightly) more appropriate audience | 15:56 |
| *** ianychoi_ is now known as ianychoi | 15:57 | |
| Wei_Liu | pabelanger: what is DIB? | 16:00 |
| Wei_Liu | diskimage-builder? | 16:00 |
| fungi | diskimage-builder | 16:00 |
| fungi | yes | 16:00 |
| pabelanger | Wei_Liu: yah, not sure if you are using it or cloud-images | 16:00 |
| *** rossella_s has quit IRC | 16:02 | |
| *** ykarel|afk has quit IRC | 16:02 | |
| Wei_Liu | pabelanger: yes, I am using it, but I did not modify the element "openstack-repos", it cached all openstack repos, not out local repos. | 16:04 |
| pabelanger | Wei_Liu: okay, I would try with prepare-workspace first, it is a simpler process of getting the git repos onto the node | 16:05 |
| *** rossella_s has joined #openstack-infra | 16:05 | |
| *** CrayZee1 has joined #openstack-infra | 16:06 | |
| mtreinish | ianw: my working guess is that the test suite emitted a unicode character in the test id or error message and it's blowing up there | 16:07 |
| dmsimard | mordred: the ginormous amount of floating ip calls aren't done if we're passing bare=False to list_servers.. I wonder if nodepool needs the stuff that is otherwise contained in the call ? | 16:07 |
| smcginnis | fungi: ;) | 16:07 |
| fungi | smcginnis: i'll reenqueue the tag once that change merges | 16:08 |
| fungi | should just be a few minutes | 16:08 |
| *** andreas_s has quit IRC | 16:08 | |
| smcginnis | Perfect, thanks fungi | 16:09 |
| *** CrayZee has quit IRC | 16:09 | |
| *** andreas_s has joined #openstack-infra | 16:09 | |
| *** CrayZee1 has quit IRC | 16:10 | |
| *** andreaf_ has joined #openstack-infra | 16:10 | |
| Wei_Liu | pabelanger: use-cached-repos also get the git repos into the remote node, but it did get the change from gerrit, could prepare-workspace get the change from gerrit? | 16:10 |
| *** CrayZee has joined #openstack-infra | 16:10 | |
| *** rossella_s has quit IRC | 16:11 | |
| pabelanger | Wei_Liu: yes, prepare-workspace will copy the contents of the src folder from zuul-executor to remote node | 16:12 |
| Wei_Liu | pabelanger: ok, I will try it first. Thank a lot | 16:12 |
| *** andreas_s has quit IRC | 16:13 | |
| *** erlon has joined #openstack-infra | 16:13 | |
| *** andreas_s has joined #openstack-infra | 16:13 | |
| *** rossella_s has joined #openstack-infra | 16:13 | |
| pabelanger | Wei_Liu: np! Looking forward to seeing you running zuulv3 :D | 16:15 |
| Wei_Liu | pabelanger: it looks like a big challenge, we will make it. | 16:17 |
| *** andreaf has quit IRC | 16:17 | |
| *** andreaf_ is now known as andreaf | 16:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing dependency for release job https://review.openstack.org/525769 | 16:18 |
| frickler | jeblair: clarkb: do we need to restart zuul to pick up new projects? see https://review.openstack.org/524952 still failing after https://review.openstack.org/524869 merged | 16:18 |
| *** vhosakot has joined #openstack-infra | 16:18 | |
| frickler | fungi: #heat seems to have the same issue like #openstack-meeting earlier this week (or was it last week?) see msg from ricolin earlier | 16:21 |
| *** gouthamr has quit IRC | 16:23 | |
| *** andreas_s has quit IRC | 16:24 | |
| fungi | frickler: ricolin: looks like #heat could stand to have some additional people with operator privs besides our general irc admins and sdake (see `/msg chanserv access #heat list`) so you can do things like change channel topics and ban disruptive users? | 16:24 |
| mordred | dmsimard: it shoulnd't | 16:24 |
| mordred | dmsimard: but I just got back online so may not have all the context yet | 16:24 |
| *** andreas_s has joined #openstack-infra | 16:24 | |
| dmsimard | mordred: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/provider.py?h=feature/zuulv3#n326 (same thing in nodepool v2 which is where we're seeing the high volume of calls) | 16:26 |
| mordred | andreaf: https://review.openstack.org/#/q/status:open+topic:devstack-functional | 16:26 |
| *** signed8bit has joined #openstack-infra | 16:26 | |
| fungi | ricolin: i'm happy to reset the channel topic to whatever you like, but even happier to just give you access to do that yourself ;) | 16:26 |
| *** andreas__ has joined #openstack-infra | 16:27 | |
| fungi | and anyone else you want to have chanserv access for that matter | 16:27 |
| mordred | dmsimard: yes - so, what you want is https://review.openstack.org/#/c/503754/ | 16:28 |
| AJaeger_ | frickler, we shouldn't anymore - zuul should find those . But it takes an hour for project creation. | 16:28 |
| mordred | dmsimard: which I need to rebase and get landed | 16:28 |
| mordred | dmsimard: and then released | 16:28 |
| AJaeger_ | frickler: so, first change merged 4+ hours ago - something is wrong ;( | 16:28 |
| *** andreas_s has quit IRC | 16:28 | |
| jeblair | Dec 6 11:40:58 zuulv3 puppet-user[15776]: (/Stage[main]/Zuul::Scheduler/Exec[zuul-reload]) Triggered 'refresh' from 1 events | 16:29 |
| jeblair | AJaeger_, frickler: ^ looks like the reload made it to zuul at least | 16:29 |
| dmsimard | mordred: so turning that on will prevent shade from poking every floating IP or something ? | 16:29 |
| mordred | dmsimard: you can also, I believe, as a workaround, set floating_ip_source: None which should also avoid the extra calls - but that's actually not the right thing to assert on a cloud that does have them but you just don't need them | 16:29 |
| dmsimard | mordred: we're actually using floating IPs | 16:29 |
| mordred | dmsimard: yes - the issue is that shade doesn't know whether you need a FIP or not, so it has to do some introspection to figure it out | 16:30 |
| jeblair | AJaeger_, frickler: checking zuul debug logs now | 16:30 |
| dmsimard | mordred: that info isn't in the detail call ? | 16:30 |
| mordred | oh - you are ... ah, that's different then | 16:30 |
| mordred | dmsimard: no, not in a way that can be relied upon | 16:30 |
| zigo | frickler: Sorry, I was away this afternoon, but indeed, thanks for the tempest.conf, that's very helpful for me. | 16:30 |
| *** rossella_s has quit IRC | 16:31 | |
| mordred | dmsimard: luckily for all of us, sometimes nova lies because it has stale cached info | 16:31 |
| *** andreas__ has quit IRC | 16:32 | |
| dmsimard | mordred: so the floating ip stuff is not directly related to the issue we're troubleshooting -- it seems like shade gets stuck on a call that probably never returns | 16:32 |
| dmsimard | mordred: we have shade 1.22.2 and there's no api-timeout setting, which seems to default to None in osc-config | 16:32 |
| mordred | dmsimard: sweet. SO - real quick -do you have cache/expiration settings set for your nodepool shade? | 16:32 |
| *** ricolin_ has joined #openstack-infra | 16:32 | |
| dmsimard | mordred: now, sure, we'll put an api-timeout setting (and cache settings) | 16:32 |
| mordred | dmsimard: there should be an api-timeout setting - that's a super old thing - can you point me to what's showing you it's not there? | 16:33 |
| *** Wei_Liu has quit IRC | 16:33 | |
| dmsimard | mordred: well it's not in our nodepool's api-timeout so yeah | 16:33 |
| dmsimard | er, nodepool's cloud.yaml | 16:34 |
| *** Hal has quit IRC | 16:34 | |
| *** rossella_s has joined #openstack-infra | 16:34 | |
| dmsimard | mordred: so that's likely getting None https://github.com/openstack-infra/shade/blob/baaf4155671d98002816597e51b2c3a33fe99390/shade/openstackcloud.py#L158 | 16:34 |
| *** ricolin has quit IRC | 16:34 | |
| jeblair | Exception: Gerrit error executing git-upload-pack openstack/mistral-tempest-plugin | 16:35 |
| mordred | yah - so - 2 things | 16:35 |
| jeblair | AJaeger_, frickler: ^ is that the project being added? | 16:35 |
| smcginnis | fungi: That patch merged. Ping me when you get a chance to requeue that and I'll watch the jobs. | 16:35 |
| *** HeOS has quit IRC | 16:35 | |
| *** dbecker has quit IRC | 16:35 | |
| mordred | dmsimard: a) that self.api_timeout setting in shade isn't used anymore since shade went REST - it's set on the adapter by os-client-config ... | 16:36 |
| mordred | sorry, it's set on the session, not the adapter, but it should be set on the Session in os-client-config | 16:36 |
| ricolin_ | fungi, yeah, I'm happy to learn about how to do this | 16:36 |
| frickler | jeblair: yes | 16:36 |
| mordred | dmsimard: but setting it in clouds.yaml *should* plumb it through properly and if it doesn't that's certainly a bug | 16:36 |
| dmsimard | mordred: yeah, I'm expecting that the value is plumbed properly | 16:37 |
| *** HeOS has joined #openstack-infra | 16:37 | |
| dmsimard | mordred: but shade probably shouldn't default api_timeout to None is what I'm saying | 16:37 |
| mordred | cool. just making sure :) | 16:37 |
| jeblair | frickler, AJaeger_: then it seems we have a sequencing issue. we now need to have zuul reload its configuration after gerrit is updated. we either need to solve this by updating infra's ansible to make sure that happens, or, in the mean time, we could split repo creation into two changes, and make sure the gerrit change is in place before the zuul change. | 16:38 |
| fungi | ricolin_: you should be able now to `/msg chanserv topic #heat whatever topic you want` | 16:38 |
| *** kiennt26 has quit IRC | 16:38 | |
| AJaeger_ | jeblair: we already split it in two changes, see the reference from frickler earlier. | 16:38 |
| dmsimard | mordred: it seems like there's some arbitrary timeout by requests or something, ex: http://paste.openstack.org/raw/628292/ | 16:38 |
| AJaeger_ | We could move the main.yaml change in the second change - and update infra-mnaula for that. | 16:38 |
| fungi | ricolin_: also you probably want to skim https://docs.openstack.org/infra/system-config/irc.html#basic-channel-operator-commands | 16:38 |
| mordred | dmsimard: what should a default for that be do you think? ... the default is set here, http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/defaults.py#n41 - fwiw | 16:39 |
| jeblair | AJaeger_: then we'll need 3 changes :) | 16:39 |
| *** nmathew- has joined #openstack-infra | 16:39 | |
| jeblair | AJaeger_: add to gerrit; add to zuul; add jobs | 16:39 |
| dmsimard | mordred: I dunno, are we expecting an API call that hasn't returned in more than 60 seconds to ever return ? | 16:40 |
| ricolin_ | fungi, thx will do:) | 16:41 |
| jeblair | AJaeger_, frickler: hrm, that's weird. git+gerrit should already be running before zuul | 16:41 |
| *** nmathew has joined #openstack-infra | 16:41 | |
| mordred | dmsimard: golly, that's an existential question ... can any software ever actually be expected to work? :) | 16:41 |
| jeblair | did the repo get created correctly in gerrit? | 16:41 |
| fungi | smcginnis: openstack/monasca-grafana-datasource 1.2.0 is what should have run jobs, right? | 16:41 |
| smcginnis | fungi: Sounds right, let me verify. | 16:42 |
| smcginnis | fungi: Correct | 16:42 |
| fungi | smcginnis: perfect, enqueued now | 16:42 |
| smcginnis | ++ | 16:42 |
| frickler | jeblair: git clone https://review.openstack.org/openstack/mistral-tempest-plugin seems to work fine | 16:43 |
| dmsimard | mordred: ok so was the floating IP stuff required in nodepool after all ? (i.e, would bare=False work?) | 16:43 |
| dmsimard | just curious as that seems like an expensive call | 16:44 |
| *** nmathew- has quit IRC | 16:44 | |
| fungi | inc0: can i go ahead and start importing kolla-kubernetes bugs from lp into sb any time now? if you're not quite ready to close down bug reporting for it in lp i can just rerun the import a second time right beforehand to pick up any subsequent new bugs/updates | 16:44 |
| mordred | dmsimard: it *should* wind up being 3 calls and being those 3 calls only once per nodepool poll period | 16:44 |
| *** nmathew- has joined #openstack-infra | 16:44 | |
| dmsimard | mordred: nope: https://paste.fedoraproject.org/paste/jsTBxWEAGInjwOBPYCHP5A/raw | 16:45 |
| *** nmathew- has quit IRC | 16:45 | |
| mordred | dmsimard: right... it should be that if the cache settings are enabled | 16:45 |
| mordred | should be 3 calls if the cache settings are enabled | 16:45 |
| dmsimard | mordred: it's the same thing with and without cache | 16:45 |
| inc0 | fungi: yes, for kolla-k8s it's totally fine | 16:45 |
| mordred | ok. that's a bug | 16:45 |
| dmsimard | mordred: bare=True returns immediately without poking every floating ip | 16:46 |
| inc0 | I'll be a bit more caucious about kolla and kolla-ansible tho because we'll need some time to teach operators new tool | 16:46 |
| fungi | inc0: thanks, i'll get it running (since i don't know how many bugs you have in total and thus how long the import will need to complete) | 16:46 |
| dmsimard | mordred: those are all differerent floating IPs, I'm not expecting cache to solve that problem | 16:46 |
| mordred | dmsimard: yes - but bare=True will not work in all cases - we need the full list to do what should be three calls so that we don't kill things at high load (otherwise each individual launch thread would need to do fip querying itself) | 16:47 |
| dmsimard | mordred: btw cache: 5 means 5 seconds ? or minutes ? | 16:47 |
| mordred | dmsimard: 5 seconds | 16:47 |
| mordred | and it shold - becaue the bug is that this is doing pushdown when it shouldn't be | 16:47 |
| mordred | dmsimard: there should be one list floating ips call to the cloud, not a bunchof specific gets | 16:47 |
| fungi | inc0: it's underway now and stories are beginning to appear at https://storyboard.openstack.org/#!/project/916 | 16:47 |
| mordred | dmsimard: and the lookup for each fip should get served out of the list information | 16:47 |
| fungi | inc0: i'll let you know once it finishes | 16:48 |
| dmsimard | mordred: behavior is the same in 1.22.2 and 1.25.0 fwiw | 16:48 |
| mordred | dmsimard: kk. I will investigave - the behavior you are showing me is *definitely* a bug, and one that is important to fix | 16:48 |
| *** nmathew has quit IRC | 16:48 | |
| *** gyee has joined #openstack-infra | 16:48 | |
| inc0 | thanks | 16:48 |
| dmsimard | mordred: want a bug in storyboard ? | 16:48 |
| mordred | dmsimard: nah. I mean, unless you just feel like making one | 16:49 |
| dmsimard | mordred: in case you get sidetracked :D | 16:49 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Add frickler as accessbot operator https://review.openstack.org/526125 | 16:49 |
| * dmsimard points at rabbit holes | 16:49 | |
| jeblair | frickler, AJaeger_: it looks like the ansible sequence happened in the correct order, however, the project-config change landed while it was in progress. so zuul got the project-config update in one ansible run, and gerrit got it in the next one. (iow, the project-config revision used in a cycle is not atomic) | 16:49 |
| mordred | dmsimard: :) | 16:50 |
| jeblair | if we pinned the project-config revision across ansible invocations within a single run of the run-all.sh script, this would work better. | 16:50 |
| fungi | i suppose our stdlib base job can't include post-logs because of the need for a site_logs secret? (this would be a common cause for people to put base in their config repo if so) | 16:51 |
| jeblair | fungi: yeah, we may want to make a stub role that copies logs to another location on the executor, like what we did in the early v3 days. | 16:51 |
| fungi | i'll omit it for now and then we can add in something suitable in a subsequent change | 16:52 |
| jeblair | k | 16:53 |
| pabelanger | mordred: dmsimard: if there is a description of the issue, I might have some time to poke into shade and see what is happening | 16:53 |
| *** hamzy has joined #openstack-infra | 16:53 | |
| pabelanger | could reproduce it easy, as I have an account on the cloud :) | 16:54 |
| *** hongbin has joined #openstack-infra | 16:54 | |
| dmsimard | mordred: https://storyboard.openstack.org/#!/story/2001394 | 16:54 |
| dmsimard | pabelanger: ^ | 16:54 |
| mordred | pabelanger, dmsimard: so - first thing - in shade.meta._get_supplemental_addresses (which is the code that does the thing dmsimard pasted) | 16:54 |
| dmsimard | brb | 16:55 |
| mordred | pabelanger, dmsimard: you'll see: if address.get('OS-EXT-IPS:type') == 'floating': \n# We have a floating IP that nova knows about, do nothing\n return server['addresses'] | 16:55 |
| *** ricolin_ has quit IRC | 16:55 | |
| mordred | that's the part that detects whether or not nova has a fip reference or not. so we have an escape hatch for "this cloud is working well and we don't need to query neutron" ... so we should verify real quick that our check is still valid | 16:56 |
| mordred | just to make sure that the cloud isn't returning a server with info filled in already | 16:56 |
| mordred | pabelanger: (if there's an easy way for you to reproduce what dmsimard did on that cloud but with a print of server['addresses'] at the top of that function, that would be helpful | 16:57 |
| dmsimard | I'll do it in a few | 16:58 |
| *** eumel8 has quit IRC | 16:58 | |
| mordred | the second thing is that in the try: block, the search_ports and search_floating_ips calls should be using cached/batched list results, rather than doing individual push-down calls - that's the thing i'm going to look at next | 16:58 |
| *** salv-orlando has joined #openstack-infra | 16:58 | |
| pabelanger | okay, let me start working on that | 16:59 |
| mordred | pabelanger, dmsimard: ok. I see the caching/batching bug - fix coming | 16:59 |
| *** katkapilatova has left #openstack-infra | 16:59 | |
| pabelanger | mordred: woot | 16:59 |
| smcginnis | fungi: I wonder if we really should have both publish-openstack-javascript-tarball and publish-openstack-python-tarball for that release. | 16:59 |
| fungi | smcginnis: it's entirely possible that the dependencies for the legacy-npm-upload job are chosen poorly | 17:00 |
| fungi | not knowing what's in that repo, i would ask the monasca team | 17:01 |
| *** dprince has quit IRC | 17:01 | |
| clarkb | johnsom: rm_work as a followup using my virt-install test method on the hypervisor your reproduce.sh setup also fails. So its not a difference in the xml definitions for the VMs as far as I can tell | 17:01 |
| clarkb | I think it must be in the host setup somewhere | 17:02 |
| johnsom | Hmmm | 17:02 |
| smcginnis | fungi: Failure: http://logs.openstack.org/b1/b18a06caa5955d6b72535cdb88e60b09f3e692d0/release/release-openstack-javascript/8365aff/job-output.txt.gz#_2017-12-06_17_01_32_437317 | 17:02 |
| clarkb | I'm also rerunning the vanilla devstack case on a different host to see if that consistently works or maybe I got a "good" hypervisor last time | 17:03 |
| mordred | pabelanger, dmsimard: I apologize for the batching bug. it's a bad bug and I think I remember landing the patch that introduced it. thank you very much for pointing it out | 17:03 |
| mordred | pabelanger, dmsimard: I'm amazed that the FIP clouds in infra's nodepool haven't completely fallen over | 17:03 |
| *** trown is now known as trown|lunch | 17:04 | |
| clarkb | its just citycloud right now iirc | 17:04 |
| mordred | I'm amazed that we have not broken citycloud | 17:06 |
| mordred | they deserve a high amount of praise :) | 17:06 |
| fungi | smcginnis: i don't know much about the javascript release workflow and tools... do we know if that ever worked? the install nvm task looks like it just clones a repo into ~/.nvm which seems insufficient to me for being able to call npm as an executable in the shell path | 17:06 |
| *** e0ne has quit IRC | 17:07 | |
| mordred | fungi: I worked on the translation of that from legacy to new-style | 17:07 |
| fungi | should the upload tarball to npm tash be calling ~/.nvm/nvm or something maybe? | 17:07 |
| fungi | or prepending ~/.nvm to $PATH? | 17:08 |
| smcginnis | Odd that it's complaining that the tar.gz file is not found even though it was just created. | 17:08 |
| fungi | mordred: these are the legacy jobs. are there outstanding changes to switch to v3 native? | 17:08 |
| *** hamzy has quit IRC | 17:08 | |
| fungi | we have a bunch of projects configured to run legacy-npm-upload in release | 17:09 |
| mordred | oh - awesome | 17:09 |
| *** hamzy has joined #openstack-infra | 17:09 | |
| mordred | well - we have a roles/upload-npm in zuul-jobs | 17:09 |
| *** salv-orlando has quit IRC | 17:09 | |
| mordred | fungi: in any case, after git clone to ~/.nvm, we should do "source ~/.nvm/nvm.sh" - which is missing from upload-nvm as wel | 17:09 |
| *** salv-orlando has joined #openstack-infra | 17:10 | |
| mordred | fungi: the nvm docs actually say to do: | 17:10 |
| mordred | export NVM_DIR="$HOME/.nvm" | 17:10 |
| mordred | [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm | 17:10 |
| fungi | mordred: given the limited number of js projects and infrequency of releases, i'm going to guess this simply hasn't been exercised since rollout | 17:10 |
| *** david-lyle has joined #openstack-infra | 17:10 | |
| mordred | fungi: yah - and rolling out the new stuff clearly wasn't finished (sorry) | 17:11 |
| dmsimard | mordred: ok so do you need the output of list servers or not ? | 17:11 |
| dmsimard | for the address thing | 17:11 |
| fungi | looks like we include the upload-npm role in the javascript playbook | 17:11 |
| smcginnis | I could have sworn we did a release for this that worked since the switch, but maybe it was one that failed and fell through the cracks with all the other fires at the time | 17:11 |
| mordred | dmsimard: yah - that's a different thing - basically, the code in question is there to work around an issue that happens in clouds, and it SEEMS like the cloud you're on is exhibiting that issue and that we are detecting it | 17:11 |
| mordred | dmsimard: however, it's possile thatour detection logic is flawed, so it's worth doublechecking | 17:12 |
| *** hamzy has quit IRC | 17:12 | |
| dmsimard | mordred: ok, getting it, one min | 17:12 |
| *** hamzy has joined #openstack-infra | 17:12 | |
| pabelanger | clarkb: and triple-test-cloud-rh1 | 17:12 |
| openstackgerrit | Mike Perez proposed openstack-infra/statusbot master: Thanks & Success bot provide confirmation site url https://review.openstack.org/510699 | 17:12 |
| smcginnis | mordred, fungi: So there's some jobs that need to be implemented yet for this? | 17:13 |
| mordred | smcginnis: there are some jobs half-implemented for this | 17:13 |
| openstackgerrit | Mike Perez proposed openstack-infra/statusbot master: Thanks & Success bot provide confirmation site url https://review.openstack.org/510699 | 17:13 |
| *** sdague has quit IRC | 17:13 | |
| AJaeger_ | fungi, mordred there's also https://review.openstack.org/518879 - and http://lists.openstack.org/pipermail/openstack-infra/2017-November/005670.html for the npm run job. | 17:13 |
| mordred | smcginnis: in zuul-jobs there are some jobs "nodejs-npm-run-test" , "nodejs-npm-run-lint" and "nodejs-npm-run-docs" (although that last one can get removed with the new sphinx jobs) | 17:14 |
| pabelanger | okay, I have to run into town for an errand, will be back shortly | 17:14 |
| *** salv-orlando has quit IRC | 17:14 | |
| smcginnis | mordred: Is that something you are or can finish implementing? | 17:15 |
| mordred | smcginnis: and there is release-openstack-javascript in project-config | 17:15 |
| *** xarses has joined #openstack-infra | 17:15 | |
| *** lucasagomes is now known as lucas-afk | 17:15 | |
| dmsimard | mordred: http://paste.openstack.org/raw/628297/ | 17:16 |
| *** JasonCL has quit IRC | 17:16 | |
| mordred | smcginnis: I can certainly re-engage with it - but I've got a couple of other bugs to finish up real quick | 17:16 |
| mordred | dmsimard: thanks - https://review.openstack.org/526127 Fix batching for floating ips and ports | 17:16 |
| smcginnis | mordred: Cool, thank you. | 17:17 |
| mordred | dmsimard: that should fix the "turning on caching settings for floating ips does not reduce the numer of calls" part | 17:17 |
| dmsimard | I'll try it real quick | 17:17 |
| mordred | dmsimard: hrm. that's troubling. u'OS-EXT-IPS:type': u'floating' is there in the addresses dict | 17:17 |
| dmsimard | mordred: that's get_server fwiw, I assumed that it would return the same thing as list_servers | 17:18 |
| mordred | dmsimard: yes - it should do | 17:18 |
| dmsimard | list_servers is very... big | 17:18 |
| mordred | dmsimard: can you try again adding in some more debugging to figure out why "if address.get('OS-EXT-IPS:type') == 'floating':" is not evaluating to true? | 17:19 |
| mordred | dmsimard: is it MAYBE a utf-8 vs. string thing perhaps? | 17:19 |
| dmsimard | maybe, trying your patch first | 17:19 |
| dmsimard | maybe worth mentioning this is python2 | 17:20 |
| dmsimard | I can try python3 after | 17:20 |
| mordred | I expect shade to work in python2 :) | 17:20 |
| dmsimard | yeah but unicode vs strings between py2 and py3 sucks | 17:20 |
| mordred | yup | 17:20 |
| *** rfolco is now known as rfolco_brb | 17:21 | |
| *** jtomasek has quit IRC | 17:22 | |
| dmsimard | mordred: ok your patch fixes it with cache enabled, +1'ing that | 17:22 |
| mordred | dmsimard: I cannot come up with a synthetic thing that causes the thing your code printed to not return true on that == test | 17:22 |
| mordred | dmsimard: woot | 17:22 |
| mordred | dmsimard: I want to add some tests for that, because clearly that's a thing that should not be allowed to break :) | 17:23 |
| *** yamamoto has quit IRC | 17:23 | |
| mordred | dmsimard: but I also want to get to the bottom of your other issue - which is that on your cloud you should never have hit the code which does this in the first place (unless those calls are being triggered from somewhere else and I'm wrong about where they're coming from for your case) | 17:24 |
| dmsimard | mordred: so you're saying with cache disabled it should detect properly the floating stuff and shouldn't be doing all those queries right ? | 17:25 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Creating a function that calls /usr/bin/timeout https://review.openstack.org/525594 | 17:26 |
| mordred | dmsimard: yes - based on that print, I expect that a get_server or a list_servers on that account should not trigger a flood of port/fip calls | 17:26 |
| dmsimard | mordred: k, checking | 17:26 |
| *** salv-orlando has joined #openstack-infra | 17:26 | |
| *** salv-orlando has quit IRC | 17:26 | |
| mordred | dmsimard: there may still be some port/fip calls during creatoin becaues of auto_ip ... you know, gotta create them - but list should not based on that addresses | 17:26 |
| *** salv-orlando has joined #openstack-infra | 17:27 | |
| *** Apoorva has quit IRC | 17:27 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add finger gateway https://review.openstack.org/525276 | 17:27 |
| dmsimard | mordred: it picks up floating IPs properly but still does all the calls I guess ? From adding a quick if/else http://paste.openstack.org/raw/628298/ | 17:28 |
| *** florianf has quit IRC | 17:29 | |
| *** dhill_ has quit IRC | 17:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Source nvm before using it https://review.openstack.org/526129 | 17:30 |
| *** sambetts is now known as sambetts|afk | 17:31 | |
| *** ralonsoh has quit IRC | 17:31 | |
| mordred | dmsimard: mind putting in a raise in def list_ports so we can see a traceback of where it's eing called from? | 17:32 |
| *** salv-orlando has quit IRC | 17:32 | |
| mordred | dmsimard: also - isn't pair-debugging fun? :) | 17:32 |
| *** david-lyle has quit IRC | 17:32 | |
| mordred | smcginnis, fungi: ^^ https://review.openstack.org/526129 should fix the nvm thing for the non-legacy thing | 17:32 |
| *** hamzy has quit IRC | 17:32 | |
| *** david-lyle has joined #openstack-infra | 17:32 | |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| frickler | mordred: AJaeger_: the patch to remove doc/reqs.txt is also failing the check, because it seems to test master branch, too: https://review.openstack.org/526097 | 17:33 |
| dmsimard | mordred: I'm cool with that, makes me dig into shade and nodepool which I'm not intimate with | 17:34 |
| dmsimard | mordred: your raise: http://paste.openstack.org/raw/628299/ | 17:34 |
| fungi | inc0: import of kolla-kubernetes is complete | 17:35 |
| dmsimard | mordred: you know what, let's take this to openstack-sdk so we're not spamming folks here | 17:36 |
| *** vhosakot has quit IRC | 17:38 | |
| *** vhosakot has joined #openstack-infra | 17:39 | |
| *** ramishra has quit IRC | 17:40 | |
| *** jpich has quit IRC | 17:43 | |
| *** gouthamr has joined #openstack-infra | 17:44 | |
| *** david-lyle has quit IRC | 17:45 | |
| *** hashar is now known as hasharDinner | 17:46 | |
| *** Kevin_Zheng has quit IRC | 17:46 | |
| *** xarses_ has quit IRC | 17:48 | |
| *** caphrim007 has joined #openstack-infra | 17:50 | |
| *** rfolco_brb is now known as rfolco | 17:52 | |
| *** dsariel has joined #openstack-infra | 17:54 | |
| *** shardy has quit IRC | 17:54 | |
| andreaf | mordred: I was wondering - are the patches to pull logs from consumer done now? we still have a role to sync files to the executor in devstack, and I would like to get rid of that and start staging things on {{ ansible_user_dir }} instead of /opt/stack/logs | 17:54 |
| *** RuiChen has quit IRC | 17:55 | |
| *** xarses_ has joined #openstack-infra | 17:55 | |
| *** signed8bit is now known as signed8bit_Zzz | 17:56 | |
| *** jascott1 has joined #openstack-infra | 17:56 | |
| AJaeger_ | frickler: best ask reuqirements team on #openstack-requirements - I don't know why they parse the old versoin and the new one. olnew should be enough | 17:57 |
| AJaeger_ | I mean: new should be enough - but I'm sure I miss something | 17:58 |
| *** jascott1 has quit IRC | 17:59 | |
| *** jascott1 has joined #openstack-infra | 17:59 | |
| AJaeger_ | mordred: just run into dhellmann at kubecon and heard you're here as well, so we can discuss sphinx problems in person if needed ;) | 18:00 |
| mordred | andreaf: not yet ... I was hoping to make good progress on that this week | 18:00 |
| *** rhallisey has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:01 | |
| *** rhallisey has joined #openstack-infra | 18:02 | |
| ihrachys | frickler, can we get https://review.openstack.org/#/c/525754/ in? this will unblock tempest plugin work in neutron | 18:03 |
| ihrachys | also related is https://review.openstack.org/#/c/525759/ but that's just grafana so not as critical | 18:04 |
| *** tesseract has quit IRC | 18:07 | |
| *** tosky has quit IRC | 18:09 | |
| AJaeger_ | infra-root, could you review https://review.openstack.org/#/c/525246/ and merge once there are no IRC meetings, please? It adds logging for the #kata channels | 18:10 |
| *** salv-orlando has joined #openstack-infra | 18:10 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 18:11 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: WIP: Git driver https://review.openstack.org/525614 | 18:12 |
| clarkb | johnsom: base devstack + tempest failed. So I am beginning to think lucky hypervisor maybe? | 18:12 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: WIP: Git driver https://review.openstack.org/525614 | 18:12 |
| clarkb | johnsom: I'm about to test on their published image instead of an infra image. Then with that should haev enough info to send them an email | 18:12 |
| *** david-lyle has joined #openstack-infra | 18:13 | |
| *** martinkopec has quit IRC | 18:13 | |
| *** bobh has quit IRC | 18:13 | |
| *** Apoorva has joined #openstack-infra | 18:15 | |
| *** Apoorva has quit IRC | 18:15 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Improve test case node_assignment_at_quota https://review.openstack.org/506134 | 18:15 |
| *** Apoorva has joined #openstack-infra | 18:15 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Reorg non detailed instance listing columns https://review.openstack.org/522103 | 18:16 |
| *** dhill_ has joined #openstack-infra | 18:16 | |
| *** david-lyle has quit IRC | 18:18 | |
| *** david-lyle has joined #openstack-infra | 18:18 | |
| *** dave-mccowan has quit IRC | 18:20 | |
| *** salv-orlando has quit IRC | 18:21 | |
| *** bobh has joined #openstack-infra | 18:21 | |
| *** dhinesh has joined #openstack-infra | 18:21 | |
| *** salv-orlando has joined #openstack-infra | 18:21 | |
| *** dhill_ has quit IRC | 18:22 | |
| *** dhill_ has joined #openstack-infra | 18:23 | |
| *** yamamoto has joined #openstack-infra | 18:24 | |
| *** rbrndt has quit IRC | 18:25 | |
| *** bobh has quit IRC | 18:25 | |
| *** salv-orlando has quit IRC | 18:26 | |
| *** dhill_ is now known as dhill|food | 18:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-base-jobs master: Initial boilerplate, packaging and testing https://review.openstack.org/526139 | 18:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-base-jobs master: Add generic base and base-test jobs/playbooks https://review.openstack.org/526140 | 18:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Move github webhook from webapp to zuul-web https://review.openstack.org/504267 | 18:29 |
| *** yamamoto has quit IRC | 18:30 | |
| *** vhosakot has quit IRC | 18:31 | |
| *** bobh has joined #openstack-infra | 18:31 | |
| *** trown|lunch is now known as trown | 18:32 | |
| *** david-lyle has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix mixed canonical/non-canonical project merge https://review.openstack.org/513331 | 18:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Lint and publish docs for zuul-base-jobs https://review.openstack.org/526144 | 18:35 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Document security group https://review.openstack.org/518620 | 18:35 |
| *** bobh has quit IRC | 18:36 | |
| *** dprince has joined #openstack-infra | 18:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-base-jobs master: Initial boilerplate, packaging and testing https://review.openstack.org/526139 | 18:37 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-base-jobs master: Add generic base and base-test jobs/playbooks https://review.openstack.org/526140 | 18:37 |
| *** hughhalf has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Changes for Ansible 2.4 https://review.openstack.org/505354 | 18:39 |
| *** e0ne has joined #openstack-infra | 18:39 | |
| *** hughhalf has joined #openstack-infra | 18:39 | |
| frickler | ihrachys: +3 to both | 18:40 |
| * ihrachys dances | 18:40 | |
| fungi | so much merriment in this channel | 18:40 |
| *** eumel8 has joined #openstack-infra | 18:41 | |
| johnsom | clarkb +1 Thanks! Ping me if you want a second set of eyes on anything | 18:42 |
| clarkb | johnsom: also I found chnage I5eb6bd137896eb9abfc4f8dbb41b41105e4820cd as the one that removed qemu-kvm | 18:44 |
| *** hamzy has joined #openstack-infra | 18:45 | |
| *** bobh has joined #openstack-infra | 18:45 | |
| *** hamzy has quit IRC | 18:46 | |
| *** hamzy has joined #openstack-infra | 18:46 | |
| johnsom | neat | 18:47 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron: also drop -api job from master gate https://review.openstack.org/525754 | 18:47 |
| clarkb | johnsom: and on further investigation that package basically only installs `kvm` executable as a wrapper around qemu-system-x86_64 | 18:47 |
| clarkb | (I'm learning so much about the mess that is using qemu and kvm) | 18:47 |
| *** eumel8 has quit IRC | 18:47 | |
| *** electrofelix has quit IRC | 18:48 | |
| johnsom | Thanks for sharing the info. At least two of us will be around to forget it by the time we need it again. grin | 18:49 |
| *** bobh has quit IRC | 18:50 | |
| *** signed8bit_Zzz is now known as signed8bit | 18:50 | |
| *** bobh has joined #openstack-infra | 18:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fixed grafana neutron board for new scenario/api job names https://review.openstack.org/525759 | 18:53 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Convert zuul.projects to a dict https://review.openstack.org/514119 | 18:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Source nvm before using it https://review.openstack.org/526129 | 18:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for shared ansible_host in inventory https://review.openstack.org/521324 | 18:56 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: github: add integration documentation https://review.openstack.org/522420 | 18:56 |
| *** esberglu has quit IRC | 18:56 | |
| clarkb | johnsom: and confirmed it happens if I devstack on their image as well | 18:57 |
| clarkb | johnsom: rm_work if you'd like to be cc'd on this email can you pm me email addrs that work for you? | 18:58 |
| *** bobh has quit IRC | 18:58 | |
| clarkb | or put it in here and see if the spam rate increases :) | 18:58 |
| *** sboyron has joined #openstack-infra | 19:00 | |
| rm_work | clarkb: same as my openstack address but i'll PM you | 19:00 |
| *** caphrim007 has quit IRC | 19:01 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 19:02 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Lint and publish docs for zuul-base-jobs https://review.openstack.org/526144 | 19:02 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Use zuul-base-jobs as a config repo https://review.openstack.org/526148 | 19:02 |
| *** caphrim007 has joined #openstack-infra | 19:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Fix zuul.projects type in docs https://review.openstack.org/526149 | 19:05 |
| clarkb | any other infra-roots want to be on the two emails to ovh? | 19:06 |
| clarkb | I'm going to cc mordred on the ipv6 one | 19:06 |
| *** jamesmcarthur has joined #openstack-infra | 19:06 | |
| pabelanger | clarkb: Shrews: so, now that we have zk0[1-3] online, we should maybe discuss how we want to migrate data into it | 19:08 |
| pabelanger | i think we'd need to do a snapshot save / restore | 19:08 |
| clarkb | pabelanger: is there not a way to join nodepool.o.o to the cluster, point all the clients to the new cluster members then delete nodepool.o.o? | 19:09 |
| clarkb | that would be the no downtime method if possible | 19:09 |
| *** sree has joined #openstack-infra | 19:09 | |
| pabelanger | clarkb: yah, I think we could. But, it is untested at this point. I could propose the patches to try it | 19:10 |
| pabelanger | clarkb: my concern is we might need to stop / start zk on nodepool to do that. | 19:10 |
| clarkb | pabelanger: my concern with it is that since the other nodes are greater in number they will treat their data as authoritative and overwrite nodepool.o.o's data | 19:11 |
| pabelanger | clarkb: yah, that too | 19:11 |
| *** rbrndt has joined #openstack-infra | 19:12 | |
| pabelanger | so, if we stop nodepool / zuul, take snapshot of zk, import into zk cluster, update nodepool / zuul to use zk cluster, that is likely safer | 19:12 |
| pabelanger | either route we go, i think there is an outage that needs to take place | 19:13 |
| *** sree has quit IRC | 19:13 | |
| inc0 | hey guys, we have discussion about how to test our images before publishing, if any of you got few minutes, we would appreciate feedback | 19:14 |
| inc0 | join us on #openstack-kolla if you can please | 19:15 |
| *** jascott1 has quit IRC | 19:16 | |
| *** sboyron has quit IRC | 19:16 | |
| *** jascott1 has joined #openstack-infra | 19:16 | |
| *** jascott1 has quit IRC | 19:17 | |
| *** jascott1 has joined #openstack-infra | 19:17 | |
| *** erlon has quit IRC | 19:17 | |
| *** signed8bit has quit IRC | 19:21 | |
| *** rhallisey has quit IRC | 19:24 | |
| *** esberglu has joined #openstack-infra | 19:24 | |
| *** rhallisey has joined #openstack-infra | 19:25 | |
| *** dave-mccowan has joined #openstack-infra | 19:25 | |
| *** yamamoto has joined #openstack-infra | 19:26 | |
| *** yamamoto has quit IRC | 19:31 | |
| *** ldnunes has quit IRC | 19:32 | |
| *** harlowja has quit IRC | 19:32 | |
| *** bobh has joined #openstack-infra | 19:32 | |
| *** wolverineav has quit IRC | 19:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add cloud quota handling https://review.openstack.org/503838 | 19:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 19:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 19:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 19:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 19:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Move api_timeout into clouds.yaml for nodepool https://review.openstack.org/526154 | 19:36 |
| *** harlowja has joined #openstack-infra | 19:36 | |
| Shrews | pabelanger: how is zk going to handle a cluster of 4 nodes? seems like a number that might be problematic if an odd number is required for majority voting or whatever | 19:37 |
| *** bobh has quit IRC | 19:38 | |
| Shrews | might want to just start with one of the zk0* nodes added to the current cluster of 1 (if that's the route you decide to go) | 19:38 |
| Shrews | or 2, or whatever | 19:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove api-timeout from nodepool-launcher yaml file https://review.openstack.org/526157 | 19:39 |
| pabelanger | Shrews: yah, we have 3 right now, so we can stop zk03 and add in nodepool.o.o, if we think that will replicate data. Then once good, we can take out nodepool.o.o and add back zk03. I think clarkb comment about will nodepool data get replicated into zk cluster happen or will the zk clustor say nodepool data is stale and wipe it is something to figure out. | 19:40 |
| pabelanger | Shrews: or, maybe we just stop everything, point to zk cluster and start with empty data set | 19:41 |
| clarkb | ya I'm really concerned that since nodepool.o.o is on the wrong side of quorum its data will be ignored/deleted/overwritten | 19:42 |
| clarkb | I am sure there are docs on this if we go digging though | 19:42 |
| *** wolverineav has joined #openstack-infra | 19:43 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul.projects type in docs https://review.openstack.org/526149 | 19:46 |
| clarkb | https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html looks fairly involved | 19:46 |
| clarkb | quick scan of that seems to say you can dynamically join one server at a time to an existing ensemble | 19:47 |
| *** [HeOS] has joined #openstack-infra | 19:47 | |
| clarkb | joiners know to look for an existing leader | 19:47 |
| clarkb | so that is how you tell it what direction data flows in | 19:48 |
| *** HeOS has quit IRC | 19:48 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Rename ssh_port to connection_port https://review.openstack.org/500800 | 19:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support username also for unmanaged cloud images https://review.openstack.org/500808 | 19:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection-type to provider diskimage https://review.openstack.org/503148 | 19:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't gather host keys for non ssh connections https://review.openstack.org/503166 | 19:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection-port to provider diskimage https://review.openstack.org/504112 | 19:49 |
| clarkb | reading that I'd be inclined to just go with the outage route as that seems simpler | 19:50 |
| *** jascott1 has quit IRC | 19:50 | |
| *** jascott1 has joined #openstack-infra | 19:51 | |
| pabelanger | clarkb: yah, think so too | 19:51 |
| *** rwsu has quit IRC | 19:51 | |
| clarkb | also our zk is too old | 19:51 |
| *** hasharDinner is now known as hashar | 19:52 | |
| clarkb | so we can't dynamic reconfig anyways | 19:52 |
| clarkb | I think that settles it :) | 19:52 |
| pabelanger | ah, was just trying to check that | 19:52 |
| pabelanger | yah | 19:52 |
| fungi | mordred: does it look to you like we have redundant (legacy and v3-native equivalent) jobs set here? http://lists.openstack.org/pipermail/release-job-failures/2017-December/000679.html | 19:54 |
| clarkb | ok I've sent email to ovh, its late tehre so don't expect immediate answers but will be curious to see what they say about the things | 19:55 |
| *** jascott1 has quit IRC | 19:55 | |
| fungi | mordred: specifically, release-openstack-javascript has an npm uploading post task, so i have a feeling the legacy-npm-upload job isn't actually needed there? | 19:56 |
| *** bobh has joined #openstack-infra | 20:01 | |
| *** CrayZee has quit IRC | 20:01 | |
| *** dhill|food has quit IRC | 20:01 | |
| smcginnis | fungi: publish-openstack-python-tarball fails too since it's not python. | 20:01 |
| *** dhill_ has joined #openstack-infra | 20:02 | |
| fungi | mordred: also i _think_ some of the post tasks are running on the wrong node (ara says ubuntu-xenial for all of them but shouldn't some stuff like the npm-upload task be happening on localhost?) | 20:02 |
| clarkb | https://auth.cloud.ovh.net/ is responding with 500s | 20:03 |
| pabelanger | oh, issues with OVH? | 20:03 |
| pabelanger | missed that | 20:03 |
| clarkb | (just general fyi that there may be ovh outage) | 20:03 |
| clarkb | pabelanger: I think it may have just started was going to clean up my servers for testing virt and ipv6 and ran into that | 20:03 |
| clarkb | was working not too long ago | 20:03 |
| fungi | mordred: at least the paths it's showing on http://logs.openstack.org/b1/b18a06caa5955d6b72535cdb88e60b09f3e692d0/release/release-openstack-javascript/76fe81c/ for npm-upload look like where the synchronize task for collect-artifacts would deposit them | 20:03 |
| fungi | but the upload command seems to claim the file doesn't exist | 20:04 |
| pabelanger | yah: http://grafana.openstack.org/dashboard/db/nodepool-ovh | 20:04 |
| clarkb | also thats without any of our auth details, just curling it directly | 20:04 |
| clarkb | so dot' | 20:04 |
| clarkb | er | 20:05 |
| clarkb | so don't expect it is our accounts | 20:05 |
| pabelanger | I'm checking http://status.ovh.net/ now | 20:05 |
| *** bobh has quit IRC | 20:06 | |
| ianw | i've live chatted with ovh support before and they were helpful, fwiw | 20:06 |
| *** caphrim007 has quit IRC | 20:07 | |
| pabelanger | clarkb: I see some maintenance on their DB right now | 20:07 |
| *** caphrim007 has joined #openstack-infra | 20:07 | |
| pabelanger | http://status.ovh.net/?do=details&id=14141 | 20:07 |
| pabelanger | says API down down | 20:08 |
| *** smatzek has quit IRC | 20:08 | |
| clarkb | I don't think its an emergency on our end | 20:09 |
| clarkb | mostly a head up when I noticed it :) | 20:09 |
| *** smatzek has joined #openstack-infra | 20:10 | |
| *** caphrim007_ has joined #openstack-infra | 20:11 | |
| *** caphrim007 has quit IRC | 20:11 | |
| *** smatzek_ has joined #openstack-infra | 20:12 | |
| *** smatzek has quit IRC | 20:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Rename ssh_port to connection_port https://review.openstack.org/500800 | 20:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support username also for unmanaged cloud images https://review.openstack.org/500808 | 20:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection-type to provider diskimage https://review.openstack.org/503148 | 20:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't gather host keys for non ssh connections https://review.openstack.org/503166 | 20:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add connection-port to provider diskimage https://review.openstack.org/504112 | 20:15 |
| dmsimard | infra-root: btw frickler and I still need to get https://review.openstack.org/#/c/525738/ and https://review.openstack.org/#/c/525748/ in | 20:16 |
| clarkb | thanks for the reminder I'll take a look today | 20:18 |
| *** jascott1 has joined #openstack-infra | 20:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Rename ssh_port to connection_port https://review.openstack.org/500799 | 20:19 |
| jeblair | +2; we usually let those get a lot of +2s before merging | 20:19 |
| clarkb | ++ | 20:20 |
| jeblair | frickler: looks like yours needs a rebase | 20:20 |
| *** dprince has quit IRC | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:22 | |
| *** lewo` has joined #openstack-infra | 20:23 | |
| *** smatzek_ has quit IRC | 20:27 | |
| *** yamamoto has joined #openstack-infra | 20:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added feature_cloud attribute to presentation serializer https://review.openstack.org/526177 | 20:29 |
| *** smatzek has joined #openstack-infra | 20:29 | |
| *** smatzek has quit IRC | 20:29 | |
| *** smatzek has joined #openstack-infra | 20:30 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added feature_cloud attribute to presentation serializer https://review.openstack.org/526177 | 20:30 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Fix cgit gerrit changeid link paths https://review.openstack.org/526178 | 20:31 |
| *** dtantsur|afk has quit IRC | 20:31 | |
| *** lewo has quit IRC | 20:31 | |
| *** fultonj has quit IRC | 20:31 | |
| *** cinerama` has quit IRC | 20:31 | |
| *** Anticime1 has quit IRC | 20:31 | |
| *** fmccrthy has quit IRC | 20:31 | |
| *** xgerman_ has quit IRC | 20:31 | |
| *** kklimonda has quit IRC | 20:31 | |
| *** kong has quit IRC | 20:31 | |
| *** berendt has quit IRC | 20:31 | |
| *** zhenguo has quit IRC | 20:31 | |
| *** dgonzalez has quit IRC | 20:31 | |
| *** khappone has quit IRC | 20:31 | |
| *** dave-mccowan has quit IRC | 20:31 | |
| *** hongbin has quit IRC | 20:31 | |
| *** gyee has quit IRC | 20:31 | |
| *** ianychoi has quit IRC | 20:31 | |
| *** rosmaita has quit IRC | 20:31 | |
| *** xinliang has quit IRC | 20:31 | |
| *** rocky-shiyan has quit IRC | 20:31 | |
| *** ekhugen_alt has quit IRC | 20:31 | |
| *** ilpianista_ has quit IRC | 20:31 | |
| *** masayukig[m] has quit IRC | 20:31 | |
| *** MasterofJOKers has quit IRC | 20:31 | |
| *** arxcruz has quit IRC | 20:31 | |
| *** peterlisak has quit IRC | 20:31 | |
| *** ajo has quit IRC | 20:31 | |
| *** larainema has quit IRC | 20:31 | |
| *** jd_ has quit IRC | 20:31 | |
| *** _Cyclone has quit IRC | 20:31 | |
| *** dtroyer has quit IRC | 20:31 | |
| clarkb | infra-root ^ is a really late followup to the Gerrit 2.13 upgrade. Noticed that wasn't working today | 20:31 |
| dmsimard | clarkb: I had a change already for that | 20:32 |
| clarkb | dmsimard: oh I'll abandon mine then, have a link to yours? | 20:32 |
| dmsimard | clarkb: https://review.openstack.org/#/c/518092/ | 20:32 |
| *** yamamoto has quit IRC | 20:32 | |
| *** bobh has joined #openstack-infra | 20:32 | |
| dmsimard | clarkb: hm it's not exactly the same | 20:32 |
| dmsimard | yours has an extra backslash | 20:33 |
| *** hashar_ has joined #openstack-infra | 20:33 | |
| dmsimard | I guess yours is the right one, I don't mind abandoning mine :p | 20:33 |
| clarkb | oh ya you'll need that backslash because its a shell script | 20:33 |
| *** uberjay has quit IRC | 20:33 | |
| clarkb | nah you can update yours | 20:33 |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| fungi | clarkb: i appoved 518092 | 20:34 |
| fungi | oh, missed that it was missing a backslash | 20:34 |
| dmsimard | fungi: I -Wd | 20:34 |
| *** aspiers[m] has quit IRC | 20:35 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Fix URL to Change-Id in commit headers https://review.openstack.org/518092 | 20:35 |
| dmsimard | ^ now with more backslashes | 20:35 |
| *** uberjay has joined #openstack-infra | 20:36 | |
| *** hashar has quit IRC | 20:37 | |
| fungi | clarkb: technically i think the backslash wasn't needed because # isn't the start of a comment unless preceded by whitespace (spaces, tabs, newline) | 20:37 |
| fungi | but it doesn't hurt | 20:37 |
| *** bobh has quit IRC | 20:38 | |
| *** adriant has quit IRC | 20:38 | |
| *** mikal has quit IRC | 20:38 | |
| *** jascott1 has quit IRC | 20:39 | |
| *** chason has quit IRC | 20:39 | |
| *** jascott1 has joined #openstack-infra | 20:39 | |
| *** chason has joined #openstack-infra | 20:40 | |
| *** adriant has joined #openstack-infra | 20:40 | |
| *** mikal has joined #openstack-infra | 20:43 | |
| *** dtantsur|afk has joined #openstack-infra | 20:47 | |
| *** fultonj has joined #openstack-infra | 20:47 | |
| *** cinerama` has joined #openstack-infra | 20:47 | |
| *** Anticime1 has joined #openstack-infra | 20:47 | |
| *** fmccrthy has joined #openstack-infra | 20:47 | |
| *** xgerman_ has joined #openstack-infra | 20:47 | |
| *** kklimonda has joined #openstack-infra | 20:47 | |
| *** kong has joined #openstack-infra | 20:47 | |
| *** berendt has joined #openstack-infra | 20:47 | |
| *** zhenguo has joined #openstack-infra | 20:47 | |
| *** dgonzalez has joined #openstack-infra | 20:47 | |
| *** khappone has joined #openstack-infra | 20:47 | |
| *** dave-mccowan has joined #openstack-infra | 20:47 | |
| *** hongbin has joined #openstack-infra | 20:47 | |
| *** gyee has joined #openstack-infra | 20:47 | |
| *** ianychoi has joined #openstack-infra | 20:47 | |
| *** rosmaita has joined #openstack-infra | 20:47 | |
| *** xinliang has joined #openstack-infra | 20:47 | |
| *** rocky-shiyan has joined #openstack-infra | 20:47 | |
| *** ekhugen_alt has joined #openstack-infra | 20:47 | |
| *** MasterofJOKers has joined #openstack-infra | 20:47 | |
| *** arxcruz has joined #openstack-infra | 20:47 | |
| *** peterlisak has joined #openstack-infra | 20:47 | |
| *** larainema has joined #openstack-infra | 20:47 | |
| *** ajo has joined #openstack-infra | 20:47 | |
| *** jd_ has joined #openstack-infra | 20:47 | |
| *** _Cyclone has joined #openstack-infra | 20:47 | |
| *** dtroyer has joined #openstack-infra | 20:47 | |
| *** calbers has quit IRC | 20:51 | |
| *** bobh has joined #openstack-infra | 20:52 | |
| *** jkilpatr has quit IRC | 20:54 | |
| *** calbers has joined #openstack-infra | 20:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Install SSH keys of admnins on RH clouds https://review.openstack.org/526187 | 20:55 |
| *** bobh has quit IRC | 20:56 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove file extension when building SimpleLayout https://review.openstack.org/525356 | 20:58 |
| clarkb | fungi: TIL didn't realize sh commenting was more than # | 20:59 |
| *** jamesmcarthur has quit IRC | 20:59 | |
| *** harlowja has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #openstack-infra | 21:03 | |
| *** harlowja has joined #openstack-infra | 21:03 | |
| *** bobh has joined #openstack-infra | 21:03 | |
| *** smatzek has quit IRC | 21:04 | |
| *** smatzek has joined #openstack-infra | 21:04 | |
| *** smatzek has quit IRC | 21:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuulv2 long lives servers https://review.openstack.org/526194 | 21:05 |
| pabelanger | clarkb: fungi: ^ removes puppet bits for static slaves | 21:05 |
| pabelanger | once we land that, I'll delete the nodes | 21:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add command socket support to zuul-merger https://review.openstack.org/523197 | 21:06 |
| clarkb | pabelanger: reading that I notice we may also have hiera data for them that can be cleaned up (because of the groups info being removed) | 21:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update playbook paths with extension https://review.openstack.org/525359 | 21:07 |
| clarkb | pabelanger: also does group signing need to be removed from the ansible group? | 21:07 |
| clarkb | er ansible group file ? /me looks | 21:07 |
| pabelanger | let me check | 21:07 |
| *** bobh has quit IRC | 21:07 | |
| *** salv-orlando has joined #openstack-infra | 21:08 | |
| pabelanger | yah, I see some hiera data | 21:08 |
| clarkb | modules/openstack_project/files/puppetmaster/groups.txt does haev a signing group too | 21:08 |
| clarkb | we might be able to remove the wheel mirror group from ^ too? | 21:09 |
| fungi | agreed, all that should be able to go buhbye | 21:09 |
| pabelanger | clarkb: same commit is fine? | 21:09 |
| *** kambiz has quit IRC | 21:09 | |
| fungi | i would | 21:10 |
| clarkb | pabelanger: ya I think we can likely bundle that together, but then that makes me wonder if the nodes should be deleted before we try puppeting them more ;) | 21:10 |
| *** dizquierdo has joined #openstack-infra | 21:10 | |
| *** sree has joined #openstack-infra | 21:10 | |
| *** yamahata has joined #openstack-infra | 21:10 | |
| clarkb | basically our inventory is created from what is running in the clouds so we may want to remove them from the cloud befor we remove them from the inventory metadata? | 21:10 |
| fungi | it won't matter if the references to the hiera vars are also gone | 21:10 |
| AJaeger_ | infra-root, could you review https://review.openstack.org/#/c/525246/ and merge once there are no IRC meetings, please? It adds logging for the #kata channels | 21:10 |
| pabelanger | clarkb: yah, we can delete them now | 21:10 |
| pabelanger | let me do that | 21:11 |
| *** bobh has joined #openstack-infra | 21:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuulv2 long lives servers https://review.openstack.org/526194 | 21:11 |
| *** kambiz has joined #openstack-infra | 21:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove zuulv2 long lived servers https://review.openstack.org/526194 | 21:13 |
| *** mat128 has quit IRC | 21:14 | |
| Zara | hm, question that came up elsewhere-- who was working on github+zuul? (just curiosity) | 21:14 |
| *** andreww has joined #openstack-infra | 21:14 | |
| clarkb | Zara: jlk and spamaps | 21:14 |
| clarkb | I think tobias may have too? | 21:14 |
| *** andreww has quit IRC | 21:14 | |
| *** sree has quit IRC | 21:14 | |
| pabelanger | yah, a few people for sure | 21:15 |
| clarkb | pabelanger: fungi and just to double check me we don't have those wheel mirror nodes anymore right becuase we build the wheels on single use instances now too | 21:15 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Save oooq generated files https://review.openstack.org/517136 | 21:15 |
| *** rhallisey has quit IRC | 21:15 | |
| Zara | thanks. | 21:15 |
| pabelanger | clarkb: yah, we deleted wheel mirrors a while back | 21:15 |
| pabelanger | maybe 2 weeks ago | 21:15 |
| *** xarses has quit IRC | 21:16 | |
| *** bobh has quit IRC | 21:16 | |
| *** salv-orlando has quit IRC | 21:16 | |
| *** andreww has joined #openstack-infra | 21:17 | |
| *** salv-orlando has joined #openstack-infra | 21:17 | |
| SpamapS | Zara: I'm using it every day. :) | 21:20 |
| SpamapS | (with a private GHE instance) | 21:20 |
| mwhahaha | any chance on getting https://review.openstack.org/#/c/525758/ merged so we can get CI working for the tripleo-upgrade repo? | 21:21 |
| *** salv-orlando has quit IRC | 21:21 | |
| *** ganso has quit IRC | 21:22 | |
| pabelanger | #status log proposal.slave.o.o / release.slave.o.o / signin01.ci.o.o have all been deleted and DNS records removed | 21:25 |
| openstackstatus | pabelanger: finished logging | 21:25 |
| pabelanger | clarkb: fungi: ^ | 21:25 |
| dmsimard | mwhahaha: +3 | 21:25 |
| mwhahaha | gracias | 21:25 |
| *** camunoz has quit IRC | 21:27 | |
| *** yamahata has quit IRC | 21:27 | |
| *** yamamoto has joined #openstack-infra | 21:28 | |
| *** liusheng has joined #openstack-infra | 21:30 | |
| *** harlowja has quit IRC | 21:32 | |
| *** bobh has joined #openstack-infra | 21:32 | |
| *** yamamoto has quit IRC | 21:33 | |
| *** gouthamr has quit IRC | 21:36 | |
| *** bobh has quit IRC | 21:37 | |
| rosmaita | anyone have time to help me troubleshoot a test error? i'm trying to convert a legacy job to zuul3, it's breaking during devstack setup in lib/databases/mysql:configure_database_mysql -- access denied when it tries to set the password. I'm pretty sure it has the correct password, it shows up in the db url connection strings for the services. This is the patch setting up the test: https://review.openstack.org/#/c/526064/ | 21:37 |
| rosmaita | and here is the error: http://logs.openstack.org/64/526064/2/check/glanceclient-dsvm-functional/da91985/job-output.txt.gz#_2017-12-06_20_52_43_496074 | 21:37 |
| rosmaita | i think maybe i missed a step somewhere | 21:38 |
| *** sdague has joined #openstack-infra | 21:38 | |
| *** openstackgerrit has quit IRC | 21:38 | |
| *** joadavis has quit IRC | 21:38 | |
| *** sai has quit IRC | 21:38 | |
| *** pahuang has quit IRC | 21:38 | |
| *** sshnaidm|afk has quit IRC | 21:38 | |
| *** akscram1 has quit IRC | 21:38 | |
| *** tlbr has quit IRC | 21:38 | |
| *** tushar has quit IRC | 21:38 | |
| *** SamYaple has quit IRC | 21:38 | |
| *** timothyb89 has quit IRC | 21:38 | |
| *** mandre has quit IRC | 21:38 | |
| *** waynr has quit IRC | 21:38 | |
| *** jzerebec1i has quit IRC | 21:38 | |
| *** ericyoung has quit IRC | 21:38 | |
| *** lifeless has quit IRC | 21:38 | |
| *** afazekas has quit IRC | 21:38 | |
| *** antonym has quit IRC | 21:38 | |
| *** andreww has quit IRC | 21:39 | |
| *** david-lyle has joined #openstack-infra | 21:39 | |
| *** xarses has joined #openstack-infra | 21:40 | |
| dmsimard | rosmaita: looking | 21:40 |
| dmsimard | jeblair: are job vars recursively merged ? | 21:41 |
| *** hughhalf has quit IRC | 21:41 | |
| dmsimard | jeblair: i.e, http://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n45 if I specify a var "devstack_localrc" in another job, does it overwrite it ? or would it merge values from https://review.openstack.org/#/c/526064/2/.zuul.yaml ? | 21:43 |
| dmsimard | it appears to be recursively merged fwiw, just double checking | 21:43 |
| *** dizquierdo has quit IRC | 21:43 | |
| *** hughhalf has joined #openstack-infra | 21:43 | |
| jeblair | dmsimard: yes, merged | 21:45 |
| dmsimard | zuul v3 is awesome | 21:45 |
| rosmaita | dmsimard: ty | 21:45 |
| *** david-lyle has quit IRC | 21:47 | |
| *** SamYaple has joined #openstack-infra | 21:49 | |
| rosmaita | dmsimard: i dont know if this matters, but i've got role run-devstack in my run.yaml, monty has it in pre.yaml in a similar shade job -- could that affect this? | 21:50 |
| dmsimard | rosmaita: that's fine, it's provided by the devstack repository | 21:52 |
| *** bobh has joined #openstack-infra | 21:52 | |
| dmsimard | rosmaita: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/run-devstack | 21:52 |
| *** bobh has quit IRC | 21:53 | |
| *** bobh has joined #openstack-infra | 21:53 | |
| rosmaita | ok, does it matter if it runs in pre.yaml or run.yaml ? | 21:54 |
| dmsimard | rosmaita: I'm not sure what's the problem tbh, I tried to compare with a job that worked http://logs.openstack.org/22/499022/4/check/devstack/d437355/job-output.txt.gz#_2017-12-04_09_15_11_792326 and I don't see anything particularly different | 21:54 |
| jeblair | pre/run shouldn't matter for this afaict | 21:54 |
| *** smatzek has joined #openstack-infra | 21:55 | |
| *** hashar_ has quit IRC | 21:56 | |
| dmsimard | rosmaita: comparing with shade the order isn't exactly the same | 21:56 |
| dmsimard | but that's because shade's job purpose is to test against a cloud (i.e, a devstack installation) | 21:57 |
| rosmaita | oh | 21:57 |
| dmsimard | so I would be wary of drawing too much from it as an example | 21:57 |
| rosmaita | ok, makes sense | 21:57 |
| dmsimard | rosmaita: looking at openstack-dev/devstack's roles, playbooks and zuul.yaml might give some insight | 21:58 |
| rosmaita | ok, i'll keep looking ... thanks (i was hoping this was a common thing you may have seen before, oh well!) | 21:58 |
| jeblair | i wonder if (a) something is happening in the wrong order (possibly due to the way we write local.conf). or (b) if we're somehow running something that removes local root access to mysql | 21:59 |
| jeblair | just brainstorming | 21:59 |
| *** smatzek has quit IRC | 21:59 | |
| *** aeng has joined #openstack-infra | 22:01 | |
| *** openstackgerrit has joined #openstack-infra | 22:01 | |
| *** joadavis has joined #openstack-infra | 22:01 | |
| *** waynr has joined #openstack-infra | 22:01 | |
| *** sai has joined #openstack-infra | 22:01 | |
| *** pahuang has joined #openstack-infra | 22:01 | |
| *** sshnaidm|afk has joined #openstack-infra | 22:01 | |
| *** akscram1 has joined #openstack-infra | 22:01 | |
| *** tlbr has joined #openstack-infra | 22:01 | |
| *** tushar has joined #openstack-infra | 22:01 | |
| *** timothyb89 has joined #openstack-infra | 22:01 | |
| *** mandre has joined #openstack-infra | 22:01 | |
| *** jzerebec1i has joined #openstack-infra | 22:01 | |
| *** ericyoung has joined #openstack-infra | 22:01 | |
| *** lifeless has joined #openstack-infra | 22:01 | |
| *** afazekas has joined #openstack-infra | 22:01 | |
| *** antonym has joined #openstack-infra | 22:01 | |
| *** rcernin has joined #openstack-infra | 22:01 | |
| openstackgerrit | Michael Turek proposed openstack/diskimage-builder master: Remove architecture rules on lshw dependency in ironic-agent https://review.openstack.org/526212 | 22:03 |
| *** aeng has quit IRC | 22:03 | |
| *** aeng has joined #openstack-infra | 22:03 | |
| *** rcernin has quit IRC | 22:03 | |
| *** rcernin has joined #openstack-infra | 22:03 | |
| *** e0ne has quit IRC | 22:06 | |
| *** e0ne has joined #openstack-infra | 22:07 | |
| *** e0ne has quit IRC | 22:07 | |
| *** e0ne has joined #openstack-infra | 22:08 | |
| *** smatzek has joined #openstack-infra | 22:08 | |
| *** e0ne has quit IRC | 22:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Move send_command() into ZuulApp https://review.openstack.org/523211 | 22:08 |
| *** e0ne has joined #openstack-infra | 22:08 | |
| *** e0ne has quit IRC | 22:09 | |
| *** e0ne has joined #openstack-infra | 22:10 | |
| *** e0ne has quit IRC | 22:10 | |
| *** e0ne has joined #openstack-infra | 22:11 | |
| *** e0ne has quit IRC | 22:11 | |
| *** e0ne has joined #openstack-infra | 22:12 | |
| *** e0ne has quit IRC | 22:12 | |
| *** dizquierdo has joined #openstack-infra | 22:12 | |
| *** e0ne has joined #openstack-infra | 22:12 | |
| *** smatzek has quit IRC | 22:13 | |
| *** e0ne has quit IRC | 22:13 | |
| *** kgiusti has left #openstack-infra | 22:13 | |
| rosmaita | jeblair thanks for the idea, i'll keep looking | 22:15 |
| *** tosky has joined #openstack-infra | 22:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add command socket support to zuul-scheduler https://review.openstack.org/523466 | 22:20 |
| *** jkilpatr has joined #openstack-infra | 22:21 | |
| *** salv-orlando has joined #openstack-infra | 22:27 | |
| *** xarses_ has quit IRC | 22:28 | |
| *** gouthamr has joined #openstack-infra | 22:28 | |
| *** xarses_ has joined #openstack-infra | 22:29 | |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| joadavis | clarkb: Any reason not to merge https://review.openstack.org/#/c/525864 ? :) | 22:30 |
| *** salv-orlando has quit IRC | 22:32 | |
| *** jascott1 has quit IRC | 22:33 | |
| *** jascott1 has joined #openstack-infra | 22:34 | |
| *** yamamoto has quit IRC | 22:35 | |
| *** harlowja has joined #openstack-infra | 22:36 | |
| *** jascott1 has quit IRC | 22:40 | |
| *** jascott1 has joined #openstack-infra | 22:41 | |
| *** jascott1 has quit IRC | 22:42 | |
| *** Goneri has quit IRC | 22:43 | |
| clarkb | joadavis: probably not, let me review | 22:45 |
| clarkb | also making tacos for lunch is a good use of lunch | 22:45 |
| *** jascott1 has joined #openstack-infra | 22:46 | |
| *** jascott1 has quit IRC | 22:46 | |
| *** stapler117 has joined #openstack-infra | 22:53 | |
| *** gouthamr has quit IRC | 22:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Allow running toci_gate_test on pre-provisioned ovb env https://review.openstack.org/518748 | 22:53 |
| openstackgerrit | Merged openstack-infra/project-config master: monasca-api: Exclude stable/ocata from sphinx doc creation https://review.openstack.org/525864 | 22:54 |
| clarkb | infra-root since I'm on a cloud feedback session I figure I'll send email to citycloud about ipv6, using la1 again, and our possibly out of sync nova quota. Anything else I should be communicating to them? | 22:54 |
| joadavis | Thanks, I didn't know your username in gerrit ot I would have added you. (today was pizza day here) | 22:54 |
| clarkb | oh those old stuck building servers, I'll mention those too | 22:55 |
| clarkb | joadavis: pizza sounds like a good use of lunch too | 22:56 |
| *** gouthamr has joined #openstack-infra | 22:56 | |
| dmsimard | clarkb: hum you can tell them nodepool will be nicer to them in the near future | 22:56 |
| *** stapler117 is now known as stapler118 | 22:56 | |
| jeblair | clarkb: i got nothin, thanks | 22:56 |
| dmsimard | We hunted part of this down today: https://storyboard.openstack.org/#!/story/2001394 | 22:57 |
| *** salv-orlando has joined #openstack-infra | 22:57 | |
| pabelanger | clarkb: nothing here | 22:57 |
| dmsimard | https://review.openstack.org/#/c/526127/ needs to merge and release though | 22:58 |
| *** dizquierdo has quit IRC | 22:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Enable logging on Kata IRC Channels https://review.openstack.org/525246 | 23:01 |
| *** katyafervent_awa has quit IRC | 23:05 | |
| *** salv-orlando has quit IRC | 23:05 | |
| *** salv-orlando has joined #openstack-infra | 23:05 | |
| *** igormarnat has quit IRC | 23:06 | |
| *** igormarnat has joined #openstack-infra | 23:06 | |
| *** katyafervent_awa has joined #openstack-infra | 23:09 | |
| *** mriedem has left #openstack-infra | 23:10 | |
| *** mriedem has quit IRC | 23:10 | |
| *** salv-orlando has quit IRC | 23:10 | |
| *** wolverineav has quit IRC | 23:10 | |
| *** wolverineav has joined #openstack-infra | 23:10 | |
| *** wolverineav has quit IRC | 23:11 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: WIP: Add a "New Story" view https://review.openstack.org/526219 | 23:19 |
| *** david-lyle has joined #openstack-infra | 23:22 | |
| clarkb | pabelanger: you've reached out to ctiycloud in the past, mind if I cc you? (thinking you may have mroe info on those building nodes that won't delete if they need it) | 23:26 |
| *** armaan has joined #openstack-infra | 23:26 | |
| *** openstack has joined #openstack-infra | 23:28 | |
| *** ChanServ sets mode: +o openstack | 23:28 | |
| *** mat128 has joined #openstack-infra | 23:29 | |
| *** yamamoto has joined #openstack-infra | 23:32 | |
| pabelanger | clarkb: go for it | 23:32 |
| *** gmann_afk is now known as gmann | 23:33 | |
| *** jamesmcarthur has quit IRC | 23:36 | |
| *** jamesmcarthur has joined #openstack-infra | 23:36 | |
| *** yamamoto has quit IRC | 23:37 | |
| *** stakeda has joined #openstack-infra | 23:39 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed bug on event update * location TBA * validation errors https://review.openstack.org/526223 | 23:40 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed bug on event update * location TBA * validation errors https://review.openstack.org/526223 | 23:41 |
| *** edmondsw has joined #openstack-infra | 23:42 | |
| *** sdague has quit IRC | 23:42 | |
| *** david-lyle has quit IRC | 23:44 | |
| *** armax has quit IRC | 23:45 | |
| *** rbrndt has quit IRC | 23:45 | |
| gmann | AJaeger_: thanks, actually 'irrelevant-files' is something which would get overridden from base job so my idea having few values of 'irrelevant-files' in base job and then further project specific values of 'irrelevant-files' in inherited job would not work here. | 23:45 |
| *** salv-orlando has joined #openstack-infra | 23:46 | |
| *** salv-orlando has quit IRC | 23:50 | |
| *** salv-orlando has joined #openstack-infra | 23:51 | |
| *** wolverineav has joined #openstack-infra | 23:52 | |
| *** salv-orlando has quit IRC | 23:55 | |
| *** david-lyle has joined #openstack-infra | 23:57 | |
| *** esberglu has quit IRC | 23:57 | |
| *** salv-orlando has joined #openstack-infra | 23:57 | |
| *** xarses has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!