| mnaser | i chalked it as release-post was low prio | 00:00 |
|---|---|---|
| smcginnis | corvus: Do you know if that failed? I just got a failure email with a RETRY_LIMIT failure, but no indication anywhere which job it was. | 00:00 |
| smcginnis | I think that was the only release one running. Or should have been. | 00:00 |
| mnaser | smcginnis: based on http://zuul.openstack.org/builds.html?job_name=tag-releases -- i think it did | 00:00 |
| *** eernst has joined #openstack-infra | 00:00 | |
| mnaser | the history shows the puppet fail then the RETRY_LIMIT | 00:00 |
| *** wolverineav has quit IRC | 00:01 | |
| *** wolverineav has joined #openstack-infra | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:02 | |
| clarkb | http://paste.openstack.org/show/679477/ is the error | 00:02 |
| clarkb | I think | 00:03 |
| pabelanger | which executor is that? | 00:03 |
| *** wolverin_ has joined #openstack-infra | 00:03 | |
| *** wolverineav has quit IRC | 00:03 | |
| clarkb | ze09 | 00:04 |
| clarkb | we have plenty of disk in /var/lib/zuul so I don't think its a disk error making git unhappy there | 00:04 |
| *** sshnaidm has quit IRC | 00:04 | |
| corvus | clarkb: did all 3 attempts happen on ze09? | 00:04 |
| clarkb | hrm the sha it has is indeed only 39 chars | 00:04 |
| clarkb | corvus: I'm not sure I found the last build in the scheduler because it is the one that reported retry limit | 00:05 |
| mnaser | by the way, super unrelated maybe, but i noticed that when i saw it in the queue, it wasn't the 8 character short commit id | 00:05 |
| clarkb | aha the f is/was missing from teh sha1 | 00:05 |
| corvus | clarkb: ah, but yeah, if the sha is wrong, that's probably the thing | 00:05 |
| clarkb | I'm guessing just an enqueing error? | 00:05 |
| mnaser | but it was the entire commit id which seems to be wrapping outside the little zuul box | 00:05 |
| clarkb | the 40th character in the sha1 should be an f | 00:05 |
| clarkb | but its missing in thel og and even in mnaser's link above :) | 00:05 |
| corvus | yeah, why is it missing everywhere? :) | 00:06 |
| smcginnis | Maybe didn't even get far enough for logs? | 00:06 |
| pabelanger | clarkb: corvus: I can see an attempt on ze01 too, same exception it seems | 00:06 |
| *** aeng has quit IRC | 00:06 | |
| pabelanger | build: ecbd96ad77144ad4a9aa9ea206b1848b | 00:06 |
| mnaser | maybe a typo in the re-enqueue? | 00:06 |
| clarkb | mnaser: yse I think that is what it was | 00:06 |
| mnaser | corvus> the fungiscript for that says sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release-post --project=openstack/releases --ref=refs/heads/master --newrev=475579f3f77cd102c6d199240d50b0248a539b8 | 00:06 |
| mnaser | the f was missing there too :p | 00:07 |
| *** yamamoto has quit IRC | 00:07 | |
| corvus | okay, it looks like i mis-copied the fungiscript output; when i run it now, it's correct | 00:08 |
| clarkb | cool | 00:08 |
| clarkb | maybe we also want to consider bubbling up these executor exceptions? | 00:08 |
| mnaser | thanks corvus and clarkb :) | 00:08 |
| corvus | re-enqueing now | 00:09 |
| *** sshnaidm has joined #openstack-infra | 00:09 | |
| mnaser | i even see it now with the short commit id rather than the big one in queue, cool | 00:10 |
| corvus | if aynone wants to add validation to that, it's in zuul/rpclistener.py :) | 00:10 |
| *** andreww has quit IRC | 00:10 | |
| * clarkb looks | 00:11 | |
| ianw | clarkb: theories on why nb03.openstack.org still not resolving in http://logs.openstack.org/02/546302/2/check/legacy-infra-puppet-apply-3-ubuntu-trusty/3132605/openstack-infra/system-config/applytest/puppetapplytest39.final.out.FAILED ? | 00:12 |
| *** eernst has quit IRC | 00:12 | |
| clarkb | corvus: do you want that in the client side argparse handling? | 00:12 |
| *** openstackstatus has quit IRC | 00:12 | |
| corvus | clarkb: i was thinking server-side for the most protection | 00:13 |
| clarkb | ianw: ipv6? | 00:13 |
| clarkb | ianw: guessing we don't have a AAAA record? | 00:13 |
| ianw | no | 00:13 |
| *** openstackstatus has joined #openstack-infra | 00:14 | |
| *** ChanServ sets mode: +v openstackstatus | 00:14 | |
| ianw | ohhhhh it's the ip6tables bit | 00:14 |
| *** wolverin_ has quit IRC | 00:15 | |
| *** wolverineav has joined #openstack-infra | 00:16 | |
| *** hongbin has quit IRC | 00:16 | |
| *** mriedem has quit IRC | 00:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Sort formats results from zookeeper https://review.openstack.org/546458 | 00:22 |
| mnaser | i'm working on finishing cutting the release for puppet-openstack and in https://review.openstack.org/#/admin/groups/1022,members -- other members seem to be afk right now, could an infra help me out? | 00:24 |
| tonyb | persia: only *really* old kernels shoudl say amd64, these dasy the kernel shoudl say x86_64 but some distros will report amd64 vs x86_64, and yeah you have the same issue with arm64 vs aarch64 and ppc64le vs ppc64el. Drives me nuts! | 00:25 |
| *** signed8bit is now known as signed8bit_Zzz | 00:25 | |
| clarkb | mnaser: I've added you (that group manages projects like puppet gnocchi and puppet neutron and puppet panko and governance says those are puppet openstack projects) | 00:27 |
| *** signed8bit_Zzz is now known as signed8bit | 00:27 | |
| mnaser | clarkb: yup, thank you so much | 00:27 |
| *** lbragstad has quit IRC | 00:32 | |
| *** eernst has joined #openstack-infra | 00:34 | |
| *** ihrachys has quit IRC | 00:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/546428 | 00:38 |
| *** tosky has quit IRC | 00:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Error when enqueue-ref doesn't get long enough rev https://review.openstack.org/546461 | 00:39 |
| *** michaelxin has quit IRC | 00:41 | |
| persia | tonyb: Eventually everyone moves to a standard. Sadly, there are far too many systems that require 5-year-old kernels still, so nothing invented in the last decade ever works right. | 00:43 |
| smcginnis | Hmm, different error, but I think we can ignore this one: http://logs.openstack.org/39/39673459f8a34f7a14ec4d0435699413ff9fbe30/tag/publish-openstack-releasenotes/23b684e/ara/result/4130f62f-cc9b-4871-b8cd-74c001951f36/ | 00:44 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Error when enqueue-ref doesn't get long enough rev https://review.openstack.org/546461 | 00:44 |
| smcginnis | mnaser: ^^ | 00:44 |
| mnaser | smcginnis: i followed it through, looks like it finished but that might have to do with the project not having release notes properly configured | 00:44 |
| pabelanger | mnaser: smcginnis: looks like a race. Was there more then 1 puppet module publishing? | 00:47 |
| mnaser | shouldn't be.. i think | 00:48 |
| *** d0ugal_ has joined #openstack-infra | 00:48 | |
| smcginnis | Well, actually, all of those puppet modules get queued up after they are tagged, so since there about 320 of them, probably. :) | 00:48 |
| tonyb | persia: I wonder exactly when it was deprecated .... Feels like it was > 10 years ago but hohum | 00:51 |
| *** daidv has joined #openstack-infra | 00:51 | |
| *** signed8bit is now known as signed8bit_Zzz | 00:52 | |
| persia | tonyb: It may be that nothing current reports "amd64", and just folk who have been dealing with architecture porting too long remember the string. | 00:54 |
| * smcginnis remembers over an hour trying to explain to a sales guy that amd64 didn't mean it wasn't an AMD processor. | 00:56 | |
| *** sshnaidm has quit IRC | 00:56 | |
| tonyb | persia: That could be true, and I'm probably in that latter group. | 00:56 |
| * smcginnis also remembers porting code to itanium *shudders* | 00:56 | |
| tonyb | ZOMG it's nearly 20 years since my first kernel patch ... how is that possible!? | 00:56 |
| smcginnis | You're old? | 00:56 |
| tonyb | smcginnis: itanium \o/ You win | 00:57 |
| persia | tonyb: You started 7 years late? | 00:57 |
| * smcginnis also remembers porting code to Netware, but won't go any further down that memory lane | 00:57 | |
| tonyb | persia: It's true ... I'm a late adopter this linux thing needed to prove itself ;P | 00:57 |
| tonyb | smcginnis: I am .... I'm offically a grumpy old man ;P | 00:58 |
| *** cuongnv has joined #openstack-infra | 00:58 | |
| smcginnis | tonyb: Welcome to the party. Now close the door before anyone else gets in. :) | 00:58 |
| tonyb | smcginnis: hehe :) | 00:58 |
| ianw | corvus: you wrote this bit -- scope.call_function('dns_a', [host['hostname']]) | 01:01 |
| ianw | can you do <% if scope.call_function(...) %> , or does a failure there raise an exception and error the whole thing out? | 01:01 |
| ianw | i can't find a lot of examples of this | 01:02 |
| corvus | ianw: i don't know | 01:03 |
| ianw | ok, i'll just try and see :) | 01:03 |
| *** yamamoto has joined #openstack-infra | 01:03 | |
| rm_work | is gerrit falling over, or is it just me? | 01:05 |
| *** aspiers has quit IRC | 01:05 | |
| corvus | rm_work: seems okay to me | 01:06 |
| rm_work | hmm, k | 01:06 |
| mnaser | corvus, rm_work: its not responding to me (or seems to just be stuck) | 01:07 |
| *** dave-mccowan has joined #openstack-infra | 01:07 | |
| clarkb | its that time of day again | 01:07 |
| clarkb | I wish I was in a better spot to dig in more | 01:07 |
| mnaser | oh true | 01:07 |
| mnaser | i remember | 01:07 |
| mnaser | and its back again now | 01:07 |
| rm_work | yeah | 01:08 |
| rm_work | seems to have just unstuck | 01:08 |
| *** yamamoto has quit IRC | 01:08 | |
| rm_work | must have been something temporary? | 01:08 |
| *** michaelxin has joined #openstack-infra | 01:08 | |
| *** cuongnv has quit IRC | 01:08 | |
| clarkb | rm_work: we've noticed that around 0100UTC performance degrades and apepars to be due to threads piling up. The number of http requests doesn't really chagne though | 01:08 |
| clarkb | so either someone is doing a daily massive query over http or its coming over ssh | 01:09 |
| rm_work | hmm, weird | 01:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-iptables master: Allow allowed_hosts to not have ipv6 interfaces https://review.openstack.org/546465 | 01:11 |
| clarkb | show-queue looks happy | 01:11 |
| clarkb | I'ev also got a show-connections but not sure I'll find anything there since its pretty bare of info | 01:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add nb03 matching, firewall and monitoring https://review.openstack.org/546302 | 01:11 |
| corvus | ianw: does nb03 not have ipv6? | 01:11 |
| ianw | corvus: no, afaict the linaro cloud, at least cn1, doesn't provide ipv6 | 01:12 |
| corvus | ianw: i guess it's there because we're building on the architecture? | 01:13 |
| ianw | yep | 01:14 |
| clarkb | user 19852 is using all 64 of its ssh connections | 01:14 |
| clarkb | 16422 using all of its connections too | 01:16 |
| *** slaweq has joined #openstack-infra | 01:17 | |
| corvus | itri disco ci / tintri ci respectively | 01:17 |
| clarkb | other than that I don't see anything too weird in the connection listing | 01:17 |
| *** dangers_away has quit IRC | 01:19 | |
| openstackgerrit | Emmet Hikory proposed openstack-infra/system-config master: Add ARM64 Ubuntu mirror https://review.openstack.org/546466 | 01:19 |
| *** slaweq has quit IRC | 01:21 | |
| openstackgerrit | Emmet Hikory proposed openstack-infra/openstack-zuul-jobs master: Vary Ubuntu mirror path based on architecture https://review.openstack.org/546434 | 01:23 |
| *** dangers_away has joined #openstack-infra | 01:25 | |
| *** stakeda has joined #openstack-infra | 01:31 | |
| *** rcernin has quit IRC | 01:36 | |
| *** rcernin has joined #openstack-infra | 01:36 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Add stable/queens integration checks https://review.openstack.org/545144 | 01:37 |
| *** rcernin has quit IRC | 01:42 | |
| *** rcernin has joined #openstack-infra | 01:43 | |
| *** rcernin has quit IRC | 01:53 | |
| *** rcernin has joined #openstack-infra | 01:53 | |
| *** slaweq has joined #openstack-infra | 01:54 | |
| fungi | corvus: clarkb: yeah, that pair of ci systems consistently seems to leak connections | 01:56 |
| *** cuongnv has joined #openstack-infra | 01:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add buildset.branch column https://review.openstack.org/546468 | 01:57 |
| *** s-shiono has joined #openstack-infra | 01:58 | |
| *** eernst has quit IRC | 01:58 | |
| *** slaweq has quit IRC | 01:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: add buildset.branch column https://review.openstack.org/546468 | 02:03 |
| *** salv-orl_ has joined #openstack-infra | 02:03 | |
| *** yamamoto has joined #openstack-infra | 02:04 | |
| *** salv-orlando has quit IRC | 02:07 | |
| jlvillal | clarkb, FYI. gerritbot seems to be working with regex. I proposed a patch to stable/ocata and it was announced on the ironic channel. | 02:07 |
| clarkb | nice | 02:07 |
| jlvillal | clarkb, I'm assuming the config changes got pushed out and gerritbot restarted :) | 02:07 |
| *** hongbin has joined #openstack-infra | 02:08 | |
| *** yamamoto has quit IRC | 02:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-iptables master: Allow allowed_hosts to not have ipv6 interfaces https://review.openstack.org/546465 | 02:13 |
| ianw | corvus: as an extra bonus, master of that dnsquery library seems to have (unintentionally) started to return an empty list for not-found, but the version we use raises an exception. i think ^ works with both http://paste.openstack.org/show/679722/ | 02:13 |
| *** yamahata has quit IRC | 02:15 | |
| *** niedbalski has quit IRC | 02:25 | |
| *** annp has joined #openstack-infra | 02:28 | |
| *** harlowja_ has quit IRC | 02:28 | |
| ianw | for reference, i guess that is an expected behaviour from https://github.com/dalen/puppet-dnsquery/commit/a733eadc2f1c7d7d82074c6597b18c9cec3e3ca0 | 02:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-iptables master: Allow allowed_hosts to not have ipv6 interfaces https://review.openstack.org/546465 | 02:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add support for Ansible extra-vars flag https://review.openstack.org/546474 | 02:48 |
| *** links has joined #openstack-infra | 02:49 | |
| *** slaweq has joined #openstack-infra | 02:58 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack-infra/system-config master: Remove pbr warnerrors in favor of sphinx check https://review.openstack.org/539812 | 02:58 |
| openstackgerrit | Nguyen Hung Phuong proposed openstack-infra/puppet-openstackci master: Remove pbr warnerrors in favor of sphinx check https://review.openstack.org/539804 | 02:59 |
| *** slaweq has quit IRC | 03:03 | |
| *** wolverineav has quit IRC | 03:05 | |
| *** yamamoto has joined #openstack-infra | 03:06 | |
| openstackgerrit | Emmet Hikory proposed openstack-infra/system-config master: Add ARM64 Ubuntu mirror https://review.openstack.org/546466 | 03:07 |
| *** kiennt26 has joined #openstack-infra | 03:09 | |
| *** yamamoto has quit IRC | 03:11 | |
| *** sree_ has joined #openstack-infra | 03:29 | |
| *** sree_ is now known as Guest31922 | 03:30 | |
| *** wolverineav has joined #openstack-infra | 03:35 | |
| *** udesale has joined #openstack-infra | 03:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add support for Ansible extra-vars flag https://review.openstack.org/546474 | 03:37 |
| *** udesale has quit IRC | 03:38 | |
| *** udesale has joined #openstack-infra | 03:39 | |
| *** wolverineav has quit IRC | 03:40 | |
| *** rlandy|rover has quit IRC | 03:42 | |
| *** lbragstad has joined #openstack-infra | 03:43 | |
| *** ykarel|away has joined #openstack-infra | 03:50 | |
| *** ykarel|away is now known as ykarel | 03:50 | |
| *** slaweq has joined #openstack-infra | 03:54 | |
| *** udesale has quit IRC | 03:55 | |
| *** yamahata has joined #openstack-infra | 03:59 | |
| *** slaweq has quit IRC | 03:59 | |
| *** larsks has joined #openstack-infra | 04:02 | |
| *** dougwig has quit IRC | 04:05 | |
| *** gongysh has joined #openstack-infra | 04:05 | |
| *** caphrim007 has joined #openstack-infra | 04:07 | |
| *** yamamoto has joined #openstack-infra | 04:07 | |
| *** cshastri has joined #openstack-infra | 04:07 | |
| *** ramishra has joined #openstack-infra | 04:10 | |
| *** d0ugal__ has joined #openstack-infra | 04:12 | |
| *** slaweq has joined #openstack-infra | 04:13 | |
| *** yamamoto has quit IRC | 04:13 | |
| *** d0ugal_ has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:17 | |
| *** wolverineav has joined #openstack-infra | 04:18 | |
| *** wolverineav has quit IRC | 04:21 | |
| *** wolverineav has joined #openstack-infra | 04:21 | |
| *** annp has quit IRC | 04:23 | |
| *** cuongnv has quit IRC | 04:23 | |
| *** cuongnv has joined #openstack-infra | 04:23 | |
| *** annp has joined #openstack-infra | 04:23 | |
| *** dave-mccowan has quit IRC | 04:34 | |
| *** psachin`` has joined #openstack-infra | 04:34 | |
| *** esberglu has quit IRC | 04:37 | |
| *** aeng has joined #openstack-infra | 04:39 | |
| *** sshnaidm has joined #openstack-infra | 04:41 | |
| *** eernst has joined #openstack-infra | 04:55 | |
| *** hongbin has quit IRC | 04:56 | |
| *** yamamoto has joined #openstack-infra | 05:04 | |
| *** psachin`` has quit IRC | 05:05 | |
| *** dhajare_ has joined #openstack-infra | 05:10 | |
| corvus | remote: https://review.openstack.org/546491 Prepare for dublin PTG update | 05:12 |
| corvus | clarkb, andreaf: ^ how about we use the infra-project-update branch of the publications repo for the ptg lunch update ( https://git.openstack.org/cgit/openstack-infra/publications/commit/?h=refs/heads/infra-project-update ) | 05:13 |
| corvus | clarkb, andreaf: i stubbed some stuff out in https://review.openstack.org/546491 -- why don't we stack changes on top of that to flesh it out. | 05:13 |
| *** udesale has joined #openstack-infra | 05:14 | |
| *** gyee has quit IRC | 05:27 | |
| *** lbragstad has quit IRC | 05:28 | |
| *** slaweq has joined #openstack-infra | 05:30 | |
| *** claudiub has joined #openstack-infra | 05:32 | |
| *** slaweq has quit IRC | 05:34 | |
| *** janki has joined #openstack-infra | 05:42 | |
| *** lpetrut has joined #openstack-infra | 05:48 | |
| *** yamamoto has quit IRC | 05:51 | |
| *** gongysh has quit IRC | 05:54 | |
| *** aspiers has joined #openstack-infra | 06:01 | |
| *** vivsoni_ has joined #openstack-infra | 06:02 | |
| *** yamamoto has joined #openstack-infra | 06:02 | |
| *** vivsoni has quit IRC | 06:02 | |
| *** kjackal has joined #openstack-infra | 06:05 | |
| *** vivsoni_ has quit IRC | 06:10 | |
| *** vivsoni has joined #openstack-infra | 06:11 | |
| *** inc0 has quit IRC | 06:17 | |
| *** aeng has quit IRC | 06:19 | |
| *** psachin`` has joined #openstack-infra | 06:22 | |
| openstackgerrit | Merged openstack-infra/zuul master: Error when enqueue-ref doesn't get long enough rev https://review.openstack.org/546461 | 06:24 |
| openstackgerrit | Merged openstack-infra/nodepool master: Sort formats results from zookeeper https://review.openstack.org/546458 | 06:25 |
| *** d0ugal__ has quit IRC | 06:27 | |
| *** dbecker has quit IRC | 06:30 | |
| *** d0ugal__ has joined #openstack-infra | 06:32 | |
| *** dbecker has joined #openstack-infra | 06:44 | |
| *** lpetrut has quit IRC | 06:48 | |
| *** d0ugal__ has quit IRC | 06:52 | |
| *** udesale_ has joined #openstack-infra | 06:54 | |
| *** udesale_ has quit IRC | 06:56 | |
| *** udesale_ has joined #openstack-infra | 06:56 | |
| *** udesale has quit IRC | 06:58 | |
| *** udesale_ has quit IRC | 07:02 | |
| *** udesale has joined #openstack-infra | 07:02 | |
| *** ykarel_ has joined #openstack-infra | 07:04 | |
| *** ykarel has quit IRC | 07:06 | |
| *** dsariel has joined #openstack-infra | 07:14 | |
| *** olaph has joined #openstack-infra | 07:17 | |
| *** olaph1 has quit IRC | 07:18 | |
| *** wolverineav has quit IRC | 07:18 | |
| *** andreas_s has joined #openstack-infra | 07:18 | |
| *** rcernin has quit IRC | 07:21 | |
| AJaeger | jlvillal: great that it works, thanks for getting this in! | 07:24 |
| *** HeOS has joined #openstack-infra | 07:25 | |
| *** eernst has quit IRC | 07:26 | |
| *** kjackal has quit IRC | 07:30 | |
| *** hashar has joined #openstack-infra | 07:32 | |
| *** yamahata has quit IRC | 07:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix post training-labs playbook https://review.openstack.org/546518 | 07:37 |
| *** slaweq has joined #openstack-infra | 07:38 | |
| AJaeger | ianw, frickler, that training-labs playbook is causing me headaches. Hope I get it right this time, could you review, please? ^ | 07:38 |
| *** udesale_ has joined #openstack-infra | 07:43 | |
| *** ykarel__ has joined #openstack-infra | 07:43 | |
| *** udesale has quit IRC | 07:44 | |
| *** ykarel_ has quit IRC | 07:46 | |
| *** pcaruana has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:52 | |
| AJaeger | and https://review.openstack.org/#/c/543139/ would be fine as well so that it's in the next images. | 07:52 |
| *** psachin`` has quit IRC | 07:54 | |
| *** alexchadin has joined #openstack-infra | 07:54 | |
| *** d0ugal has joined #openstack-infra | 07:55 | |
| *** d0ugal has quit IRC | 07:55 | |
| *** d0ugal has joined #openstack-infra | 07:55 | |
| *** pcichy_ has joined #openstack-infra | 07:56 | |
| *** agopi has quit IRC | 07:56 | |
| *** psachin has joined #openstack-infra | 07:56 | |
| *** lpetrut has joined #openstack-infra | 07:56 | |
| *** mancdaz has quit IRC | 07:57 | |
| *** pcichy has quit IRC | 07:57 | |
| *** wendar has quit IRC | 07:57 | |
| *** pcichy_ is now known as pcichy | 07:57 | |
| *** wendar has joined #openstack-infra | 07:58 | |
| *** mancdaz_ has joined #openstack-infra | 07:58 | |
| *** mancdaz_ is now known as mancdaz | 07:58 | |
| *** d0ugal has quit IRC | 08:00 | |
| *** d0ugal has joined #openstack-infra | 08:01 | |
| *** olaph1 has joined #openstack-infra | 08:01 | |
| *** olaph has quit IRC | 08:04 | |
| *** e0ne has quit IRC | 08:04 | |
| *** d0ugal has quit IRC | 08:06 | |
| *** amoralej|off is now known as amoralej | 08:08 | |
| *** ykarel__ is now known as ykarel | 08:09 | |
| *** udesale_ is now known as udesale | 08:09 | |
| *** kjackal has joined #openstack-infra | 08:11 | |
| *** vabada has quit IRC | 08:11 | |
| *** vabada has joined #openstack-infra | 08:12 | |
| *** d0ugal has joined #openstack-infra | 08:15 | |
| *** e0ne has joined #openstack-infra | 08:16 | |
| *** wolverineav has joined #openstack-infra | 08:18 | |
| *** florianf has joined #openstack-infra | 08:19 | |
| *** wolverineav has quit IRC | 08:23 | |
| *** slaweq_ has joined #openstack-infra | 08:23 | |
| *** s-shiono has quit IRC | 08:26 | |
| *** slaweq_ has quit IRC | 08:28 | |
| *** d0ugal has quit IRC | 08:29 | |
| *** zoli is now known as zoli|wfh | 08:31 | |
| *** zoli|wfh is now known as zoli | 08:32 | |
| *** wolverineav has joined #openstack-infra | 08:32 | |
| *** tesseract has joined #openstack-infra | 08:34 | |
| *** d0ugal has joined #openstack-infra | 08:34 | |
| *** wolverineav has quit IRC | 08:37 | |
| *** yamahata has joined #openstack-infra | 08:37 | |
| *** d0ugal has quit IRC | 08:39 | |
| *** electrofelix has joined #openstack-infra | 08:39 | |
| *** psachin has quit IRC | 08:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Tenant config dynamic loading of project from external script https://review.openstack.org/535878 | 08:44 |
| *** pcichy has quit IRC | 08:45 | |
| *** pcichy has joined #openstack-infra | 08:46 | |
| *** e0ne has quit IRC | 08:47 | |
| *** d0ugal has joined #openstack-infra | 08:48 | |
| *** d0ugal has quit IRC | 08:48 | |
| *** d0ugal has joined #openstack-infra | 08:48 | |
| *** alexchadin has quit IRC | 08:50 | |
| *** alexchadin has joined #openstack-infra | 08:51 | |
| *** psachin has joined #openstack-infra | 08:51 | |
| *** ralonsoh has joined #openstack-infra | 08:53 | |
| *** jpena|off is now known as jpena | 08:53 | |
| *** oidgar has joined #openstack-infra | 08:55 | |
| *** gfidente has joined #openstack-infra | 08:56 | |
| *** gfidente has joined #openstack-infra | 08:56 | |
| *** jpich has joined #openstack-infra | 08:57 | |
| *** d0ugal has quit IRC | 08:58 | |
| *** jtomasek has joined #openstack-infra | 09:01 | |
| *** wolverineav has joined #openstack-infra | 09:06 | |
| *** d0ugal has joined #openstack-infra | 09:07 | |
| *** wolverineav has quit IRC | 09:11 | |
| *** admcleod has quit IRC | 09:11 | |
| *** admcleod has joined #openstack-infra | 09:12 | |
| *** shardy has joined #openstack-infra | 09:12 | |
| *** hamzy_ has joined #openstack-infra | 09:17 | |
| *** hamzy has quit IRC | 09:19 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 09:23 |
| *** vipuls has quit IRC | 09:31 | |
| *** vipul has joined #openstack-infra | 09:33 | |
| andreaf | corvus clarkb ok that's fine for me | 09:36 |
| *** derekh has joined #openstack-infra | 09:39 | |
| *** seanhandley has joined #openstack-infra | 09:42 | |
| *** shardy has quit IRC | 09:43 | |
| *** armaan has joined #openstack-infra | 09:47 | |
| *** notmyname has quit IRC | 09:48 | |
| *** alexchadin has quit IRC | 09:50 | |
| *** alexchadin has joined #openstack-infra | 09:50 | |
| *** notmyname has joined #openstack-infra | 09:51 | |
| *** kiennt26 has quit IRC | 09:57 | |
| *** pcichy has quit IRC | 09:57 | |
| *** daidv has quit IRC | 09:58 | |
| *** wolverineav has joined #openstack-infra | 10:00 | |
| *** dtantsur|afk is now known as dtantsur | 10:02 | |
| *** wolverineav has quit IRC | 10:04 | |
| *** pcaruana has quit IRC | 10:05 | |
| *** d0ugal_ has joined #openstack-infra | 10:06 | |
| *** d0ugal has quit IRC | 10:06 | |
| *** cuongnv has quit IRC | 10:07 | |
| *** witek has quit IRC | 10:10 | |
| *** d0ugal_ has quit IRC | 10:11 | |
| *** d0ugal_ has joined #openstack-infra | 10:13 | |
| *** witek has joined #openstack-infra | 10:15 | |
| *** d0ugal__ has joined #openstack-infra | 10:17 | |
| *** abelur_ has quit IRC | 10:18 | |
| *** d0ugal_ has quit IRC | 10:18 | |
| odyssey4me | morning all, given that I'm seeing this fairly consistently in our builds, I suspect that the ubuntu mirrors may be out of date? http://logs.openstack.org/39/546239/1/check/openstack-ansible-functional-ubuntu-xenial/10034d0/logs/host/lxc-cache-prep-commands.log.txt.gz | 10:19 |
| *** pcaruana has joined #openstack-infra | 10:20 | |
| *** e0ne has joined #openstack-infra | 10:21 | |
| *** yamahata has quit IRC | 10:22 | |
| *** annp has quit IRC | 10:25 | |
| *** Guest31922 has quit IRC | 10:26 | |
| *** stakeda has quit IRC | 10:26 | |
| *** sree_ has joined #openstack-infra | 10:27 | |
| *** sree_ is now known as Guest95362 | 10:27 | |
| *** shardy has joined #openstack-infra | 10:27 | |
| *** caromeo has joined #openstack-infra | 10:30 | |
| *** fverboso has joined #openstack-infra | 10:30 | |
| *** Guest95362 has quit IRC | 10:31 | |
| *** priteau has joined #openstack-infra | 10:34 | |
| AJaeger | odyssey4me: you can check yourself - e.g. http://mirror.dfw.rax.openstack.org/ubuntu/dists/ | 10:34 |
| caromeo | could someone of you verify this review (https://review.openstack.org/#/c/546174/) in order to enable the zuul checks? thanks in advance... | 10:35 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add OSA nspawn host/container repo base jobs https://review.openstack.org/546294 | 10:36 |
| *** wolverineav has joined #openstack-infra | 10:41 | |
| *** wolverineav has quit IRC | 10:45 | |
| *** alexchadin has quit IRC | 10:48 | |
| *** kjackal has quit IRC | 10:53 | |
| *** d0ugal__ has quit IRC | 10:54 | |
| *** d0ugal has joined #openstack-infra | 10:54 | |
| *** d0ugal has quit IRC | 10:54 | |
| *** d0ugal has joined #openstack-infra | 10:54 | |
| *** dbecker has quit IRC | 10:55 | |
| *** katkapilatova has joined #openstack-infra | 11:02 | |
| *** panda|off is now known as panda | 11:05 | |
| *** manjeets has quit IRC | 11:05 | |
| *** abelur_ has joined #openstack-infra | 11:05 | |
| *** manjeets has joined #openstack-infra | 11:06 | |
| *** tosky has joined #openstack-infra | 11:06 | |
| *** seanhandley has left #openstack-infra | 11:09 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import openstack-ansible-os_panko role https://review.openstack.org/546571 | 11:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix post training-labs playbook https://review.openstack.org/546518 | 11:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import openstack-ansible-os_panko role https://review.openstack.org/546571 | 11:16 |
| *** pcaruana has quit IRC | 11:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add OSA os_panko repo base jobs https://review.openstack.org/546573 | 11:20 |
| *** d0ugal has quit IRC | 11:23 | |
| *** wolverineav has joined #openstack-infra | 11:26 | |
| *** kjackal has joined #openstack-infra | 11:26 | |
| *** kjackal_ has joined #openstack-infra | 11:27 | |
| *** d0ugal has joined #openstack-infra | 11:30 | |
| *** wolverineav has quit IRC | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** abelur_ has quit IRC | 11:31 | |
| *** pcaruana has joined #openstack-infra | 11:33 | |
| *** tosky has quit IRC | 11:34 | |
| *** udesale has quit IRC | 11:39 | |
| *** pbourke has quit IRC | 11:40 | |
| *** wolverineav has joined #openstack-infra | 11:41 | |
| *** yamamoto has quit IRC | 11:41 | |
| *** pbourke has joined #openstack-infra | 11:42 | |
| *** wolverineav has quit IRC | 11:45 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add individual core group for openstack-ansible-os_neutron https://review.openstack.org/546577 | 11:51 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Add new project 'osel' https://review.openstack.org/546260 | 11:55 |
| *** pcichy has joined #openstack-infra | 11:56 | |
| *** alexchadin has joined #openstack-infra | 11:56 | |
| *** armaan has quit IRC | 12:00 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Import openstack-ansible-os_panko role https://review.openstack.org/546571 | 12:04 |
| *** d0ugal has quit IRC | 12:06 | |
| *** janki has quit IRC | 12:07 | |
| *** d0ugal has joined #openstack-infra | 12:10 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: add detection of darwin (MacOS/OSX) https://review.openstack.org/546284 | 12:12 |
| chandankumar | fungi: clarkb please have a look at this patch https://review.openstack.org/#/c/529490/ | 12:12 |
| *** gongysh has joined #openstack-infra | 12:12 | |
| *** gongysh has quit IRC | 12:13 | |
| chandankumar | it is required for tempest stestr migration patch | 12:13 |
| chandankumar | fungi: clarkb and this one also https://review.openstack.org/#/c/523646/ | 12:14 |
| *** armaan has joined #openstack-infra | 12:17 | |
| *** ericyoung has quit IRC | 12:19 | |
| *** mgagne has quit IRC | 12:19 | |
| *** d0ugal has quit IRC | 12:20 | |
| *** adam_g has quit IRC | 12:20 | |
| *** d34dh0r53 has quit IRC | 12:20 | |
| *** ericyoung has joined #openstack-infra | 12:20 | |
| *** wendar has quit IRC | 12:21 | |
| *** evrardjp has quit IRC | 12:22 | |
| *** wendar has joined #openstack-infra | 12:22 | |
| *** dhellmann has quit IRC | 12:22 | |
| *** d0ugal has joined #openstack-infra | 12:22 | |
| *** kazsh has quit IRC | 12:23 | |
| *** olaph has joined #openstack-infra | 12:23 | |
| *** olaph1 has quit IRC | 12:23 | |
| *** dprince has joined #openstack-infra | 12:23 | |
| *** kazsh has joined #openstack-infra | 12:25 | |
| *** evrardjp has joined #openstack-infra | 12:26 | |
| *** d34dh0r53 has joined #openstack-infra | 12:26 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: add detection of darwin (MacOS/OSX) https://review.openstack.org/546284 | 12:26 |
| *** adam_g has joined #openstack-infra | 12:28 | |
| *** dhellmann has joined #openstack-infra | 12:30 | |
| *** d0ugal has quit IRC | 12:34 | |
| *** mgagne has joined #openstack-infra | 12:36 | |
| *** wolverineav has joined #openstack-infra | 12:36 | |
| *** mgagne is now known as Guest20946 | 12:36 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** wolverineav has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:42 | |
| *** zoli is now known as zoli|wfh | 12:42 | |
| *** zoli|wfh is now known as zoli | 12:42 | |
| *** ldnunes has quit IRC | 12:43 | |
| fungi | frickler: 529490 is related to 523646 which you already reviewed, if you have a moment to take a look | 12:45 |
| *** persia has quit IRC | 12:46 | |
| *** persia has joined #openstack-infra | 12:47 | |
| *** melwitt has quit IRC | 12:47 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** electrofelix has quit IRC | 12:49 | |
| *** adarazs is now known as adarazs_lunch | 12:49 | |
| *** fultonj has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** melwitt has joined #openstack-infra | 12:52 | |
| *** fultonj has joined #openstack-infra | 12:52 | |
| *** tosky has joined #openstack-infra | 12:52 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack/diskimage-builder master: Checking link status according to DIB_DHCP_TIMEOUT https://review.openstack.org/529976 | 12:53 |
| *** ldnunes has joined #openstack-infra | 12:56 | |
| *** cshastri has quit IRC | 12:59 | |
| *** kiennt26 has joined #openstack-infra | 13:00 | |
| *** weshay_PTO is now known as weshay | 13:04 | |
| *** mdbooth has quit IRC | 13:06 | |
| *** ari[m] has quit IRC | 13:06 | |
| *** dave-mccowan has joined #openstack-infra | 13:09 | |
| *** ari[m] has joined #openstack-infra | 13:10 | |
| *** mdbooth has joined #openstack-infra | 13:11 | |
| *** d0ugal has joined #openstack-infra | 13:13 | |
| *** d0ugal_ has joined #openstack-infra | 13:19 | |
| *** d0ugal has quit IRC | 13:20 | |
| *** rosmaita has joined #openstack-infra | 13:21 | |
| *** janki has joined #openstack-infra | 13:21 | |
| *** d0ugal_ has quit IRC | 13:25 | |
| andreaf | hello infra team :) | 13:26 |
| andreaf | I'm trying to get a multinode zuulv3 native job working, and in doing so I noticed that sometimes node get different IPs in the inventory - one public which is used by ansible and one private in the 10.* range - in other cases I get the same public IP attached to the private ip variable | 13:27 |
| *** alexchadin has quit IRC | 13:27 | |
| *** amoralej is now known as amoralej|lunch | 13:28 | |
| andreaf | and when that's the case the subnode cannot talk to the controller | 13:28 |
| andreaf | I guess it's provider specific? | 13:28 |
| *** alexchadin has joined #openstack-infra | 13:28 | |
| andreaf | I now added switch and peers groups in the nodeset to activate http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/playbooks/multinode/pre.yaml#n20 and see that it helps | 13:29 |
| andreaf | however in the zuulv2 world I was under the impression that that kind of setup was only needed for neutron-dvr and ironic jobs | 13:30 |
| *** dhajare_ has quit IRC | 13:31 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:31 | |
| andreaf | fungi clarkb corvus ianw pabelanger ^^^^ | 13:31 |
| *** rlandy has joined #openstack-infra | 13:34 | |
| *** rlandy is now known as rlandy|rover | 13:35 | |
| *** alexchadin has quit IRC | 13:35 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add additional builder debug logging https://review.openstack.org/546303 | 13:35 |
| *** d0ugal_ has joined #openstack-infra | 13:38 | |
| *** kgiusti has joined #openstack-infra | 13:40 | |
| *** wolverineav has joined #openstack-infra | 13:41 | |
| *** jpena|lunch is now known as jpena | 13:44 | |
| *** ykarel is now known as ykarel|afk | 13:45 | |
| *** wolverineav has quit IRC | 13:46 | |
| *** d0ugal_ has quit IRC | 13:48 | |
| *** d0ugal has joined #openstack-infra | 13:48 | |
| *** d0ugal has quit IRC | 13:48 | |
| *** d0ugal has joined #openstack-infra | 13:48 | |
| vabada | hi, I have a general question. Sorry if I missed something, but I set up my gerrit account some months ago and I have been able to push some patches. However, for instance, I don't get to see the messages regarding: Username proposed openstack/project: change. Any ideas? | 13:50 |
| AJaeger | pabelanger: is https://review.openstack.org/543139 fine now? | 13:51 |
| *** ykarel|afk has quit IRC | 13:52 | |
| AJaeger | vabada: is the project you're pushing setup to display those messages? Which channel and change do you talk about as example? | 13:53 |
| *** efried is now known as efried_omalley | 13:53 | |
| *** danpawlik has quit IRC | 13:53 | |
| vabada | I guess so, the channel is #openstack-magnum, where I can see other' | 13:54 |
| *** psachin has quit IRC | 13:54 | |
| vabada | I guess so, the channel is #openstack-magnum, where I can see other's messages. The change as an example: https://review.openstack.org/#/c/544896/ | 13:54 |
| vabada | sorry: #openstack-containers is the proper channel | 13:56 |
| *** alexchadin has joined #openstack-infra | 13:57 | |
| *** d0ugal has quit IRC | 13:58 | |
| *** mriedem has joined #openstack-infra | 13:58 | |
| AJaeger | vabada: no idea directly. Could be a temporary thing. But it shouldn't be related to your account at all. | 13:58 |
| AJaeger | vabada: setup of gerritbot looks fine to me | 13:58 |
| *** d0ugal has joined #openstack-infra | 13:59 | |
| *** amoralej|lunch is now known as amoralej | 14:01 | |
| *** ihrachys has joined #openstack-infra | 14:03 | |
| *** d0ugal has quit IRC | 14:04 | |
| vabada | AJaeger, ok thanks. I think I've been observing this behaviour for a while (not reporting messages in the channel), I just thought until today that my IRC client might be hiding them from me somehow (which makes no sense now that I think), so I'll keep an eye on that. Is here the proper place to report this? In case it doesn't get solved in a couple of days... | 14:04 |
| *** olaph1 has joined #openstack-infra | 14:04 | |
| AJaeger | vabada: you can double check here - http://eavesdrop.openstack.org/irclogs/%23openstack-containers/latest.log.html and if you think it's a systematic problem, yes, please report. | 14:04 |
| AJaeger | Let's see whether any of the admins has an idea ^ | 14:05 |
| *** olaph has quit IRC | 14:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove precise references https://review.openstack.org/545157 | 14:06 |
| fungi | andreaf: any configuration where one node needs to reach instances booted on another node will need an overlay network in place because the nested nova instances don't have ip addresses assigned by the underlying service provider | 14:06 |
| *** fverboso has quit IRC | 14:07 | |
| andreaf | fungi: ok that makes sense - however my issue is with openstack services talking to each other | 14:08 |
| fungi | andreaf: so for multinode-devstack/grenade jobs i would suggest just always adding the overlay network anyway | 14:08 |
| andreaf | fungi: so the overlay does not matter | 14:08 |
| fungi | andreaf: well, are the firewall rules adjusted to allow inter-service communication between nodes on those addresses? | 14:08 |
| andreaf | fungi yeah I'm doing that now but that doesn't affect my issue | 14:08 |
| fungi | we open up communication over the tunnel but not necessarily between the actual provider-assigned addresses of the nodes | 14:09 |
| odyssey4me | is there a way to trigger an ubuntu mirror update - it looks like some new packages were released after the previous update and the OSA builds are breaking due to using the base LXC images which include the new packages | 14:09 |
| fungi | odyssey4me: i'll take a look and see if it's lagging | 14:10 |
| odyssey4me | thanks fungi - appreciate it | 14:10 |
| fungi | odyssey4me: yeah, looks like it's about a day stale? http://mirror.dfw.rax.openstack.org/ubuntu/timestamp.txt | 14:11 |
| fungi | checking mirror update logs now | 14:11 |
| andreaf | fungi well in some cases it works - for instance here http://logs.openstack.org/96/537296/15/check/devstack-multinode/c63c7b3/zuul-info/inventory.yaml | 14:11 |
| andreaf | fungi the private ip is in the 10.* range | 14:11 |
| *** Goneri has joined #openstack-infra | 14:12 | |
| fungi | andreaf: i don't think we can make any guarantees about communication between rfc-1918 addresses in providers. are you able to draw any correlation between which jobs/builds are working and which aren't? | 14:13 |
| *** bhujay has joined #openstack-infra | 14:13 | |
| fungi | odyssey4me: looks like there may be a stale afs lockfile. i'm trying to make sure there's no stuck vos release and then i'll clean that up and redo it | 14:14 |
| andreaf | fungi: this role should take care of the firewall rules http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-firewall/tasks/main.yaml | 14:14 |
| odyssey4me | awesome, thanks fungi | 14:14 |
| *** lbragstad has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Handle a stestr results directory https://review.openstack.org/523646 | 14:16 |
| fungi | odyssey4me: cleaned up the lock and manually starting a fresh ubuntu mirror update now, i'll let you know in a few minutes when it completes | 14:16 |
| fungi | #status log deleted stale afs lockfile for the ubuntu mirror and restarted a fresh reprepro-mirror-update of it on mirror-update.o.o | 14:18 |
| openstackstatus | fungi: finished logging | 14:18 |
| andreaf | fungi: I haven't found any real correlation yet I'm trying but it could be something else I'm doing wrong that I fail to see | 14:18 |
| *** dsariel has quit IRC | 14:19 | |
| fungi | andreaf: correlation between service providers where it works and where it fails? | 14:21 |
| *** ykarel|afk has joined #openstack-infra | 14:21 | |
| *** d0ugal has joined #openstack-infra | 14:22 | |
| *** cshastri has joined #openstack-infra | 14:22 | |
| *** olaph has joined #openstack-infra | 14:22 | |
| andreaf | fungi: no, I thought it worked fine on rax but I have a case where it doesn't | 14:23 |
| andreaf | fungi: never mind I will dig further it must be something else I'm doing wrong | 14:23 |
| *** olaph1 has quit IRC | 14:24 | |
| fungi | andreaf: yeah, i'm not as familiar with devstack configuration options as i should be to know what addresses it ends up using... probably a stupid question on my part but have you checked the stackrc/localrc between a legacy multinode job and teh equivalent v3 native one you're trying to make work? | 14:25 |
| andreaf | fungi: yeah I've being going through the configuration the whole morning | 14:25 |
| fungi | do the legacy jobs have the services listening/communicating across the ovs bridge? | 14:26 |
| *** esberglu has joined #openstack-infra | 14:29 | |
| *** links has quit IRC | 14:29 | |
| *** wolverineav has joined #openstack-infra | 14:30 | |
| andreaf | fungi: I think I found the issue - I found proof that connectivity happens in apache logs, but something seems to be wrong with the SSL setup http://logs.openstack.org/96/545696/19/check/devstack-multinode/8092d24/controller/logs/apache/tls-proxy_error.log.gz#_2018-02-21_11_17_25_807322 | 14:31 |
| *** sree_ has joined #openstack-infra | 14:31 | |
| fungi | andreaf: oh! great, so at least it's probably not a network-related issue | 14:31 |
| *** sree_ is now known as Guest44467 | 14:31 | |
| andreaf | fungi: yeah I think it must be something else | 14:32 |
| andreaf | something in the CA setup | 14:32 |
| *** wolverineav has quit IRC | 14:35 | |
| andreaf | fungi it must this bit missing https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n756 | 14:35 |
| fungi | vabada: AJaeger: it _is_ something to do with the account, i think! see http://paste.openstack.org/show/680378/ for what i found in the bot's log | 14:36 |
| Shrews | pabelanger: cleaned up nb04 zk data | 14:36 |
| pabelanger | Shrews: thanks! | 14:36 |
| AJaeger | fungi: oh, interesting. Thanks for checking! | 14:36 |
| fungi | taking a look at the account in gerrit's db now | 14:36 |
| *** psachin has joined #openstack-infra | 14:37 | |
| *** eernst has joined #openstack-infra | 14:37 | |
| *** dmellado has quit IRC | 14:37 | |
| vabada | fungi, AJaeger: thanks. I think I just set up the name 1-2 hours ago, so I'll push a dummy update to check if it was that | 14:37 |
| fungi | vabada: yeah, i see a name in the db currently, which was confusing me | 14:38 |
| fungi | in good news, at least this doesn't crash the bot and merely skips the notification ;) | 14:38 |
| vabada | fungi, sorry for that. It has to be that | 14:38 |
| fungi | though if you wanted to patch openstack-infra/gerritbot's gerritbot/bot.py around line 191 to use something like data['patchSet']['uploader'].get('name', data['patchSet']['uploader'].get('username', '<unknown user>')) then it could fall back gracefully to username or worst report it as an unknown user | 14:41 |
| *** dmellado has joined #openstack-infra | 14:42 | |
| vabada | do you mean I could add a patchset for that? I'd be happy to | 14:43 |
| fungi | vabada: sure! if you have time that would be great | 14:43 |
| vabada | fungi, it worked now btw, everything fine. | 14:43 |
| vabada | fungi, sure, I'll do it | 14:43 |
| fungi | easier to fix simple bugs like this while they're fresh in the mind | 14:43 |
| fungi | thanks! | 14:43 |
| *** eharney has joined #openstack-infra | 14:44 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed event type validation rules https://review.openstack.org/546651 | 14:44 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed event type validation rules https://review.openstack.org/546651 | 14:48 |
| *** d0ugal has quit IRC | 14:49 | |
| *** ramishra has quit IRC | 14:50 | |
| *** oidgar has quit IRC | 14:50 | |
| *** eernst has quit IRC | 14:51 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 14:51 |
| *** eernst has joined #openstack-infra | 14:51 | |
| *** ramishra has joined #openstack-infra | 14:52 | |
| *** felipemonteiro has joined #openstack-infra | 14:53 | |
| *** bhujay has quit IRC | 14:55 | |
| *** alexchadin has quit IRC | 14:56 | |
| *** d0ugal has joined #openstack-infra | 14:57 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Revert fixes for legacy boot jobs https://review.openstack.org/543350 | 14:58 |
| *** ramishra has quit IRC | 14:59 | |
| *** kiennt26 has quit IRC | 15:00 | |
| odyssey4me | fungi apologies - I see you're stuck in another issue, I'd like confirmation of whether the mirror update is done or not when you have a moment. | 15:01 |
| *** cshastri has quit IRC | 15:02 | |
| *** wolverineav has joined #openstack-infra | 15:03 | |
| andreaf | fungi this should fix the tls issue https://review.openstack.org/#/c/528074/6/roles/sync-devstack-data/tasks/main.yaml | 15:04 |
| fungi | odyssey4me: it's still underway, looks like | 15:04 |
| odyssey4me | fungi ah ok, thanks :) | 15:05 |
| *** pcichy has quit IRC | 15:05 | |
| fungi | odyssey4me: there must have been a large security update or something... it's currently in the middle of grabbing all the qemu packages | 15:05 |
| fungi | oh, yeah there's a gcc update in the history here too | 15:06 |
| fungi | that's probably where it's been spending a lot of its time | 15:06 |
| *** felipemonteiro has quit IRC | 15:06 | |
| *** psachin has quit IRC | 15:08 | |
| *** wolverineav has quit IRC | 15:08 | |
| odyssey4me | ja, the gcc update is what's biting us so it makes sense | 15:08 |
| fungi | though it's just been sitting here for ~15 minutes after what looks like the end of the qemu updates... not sure what it's busy on now | 15:09 |
| fungi | the logs have grown silent | 15:09 |
| *** psachin has joined #openstack-infra | 15:09 | |
| fungi | i'll give it a bit longer | 15:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add individual core group for openstack-ansible-os_gnocchi https://review.openstack.org/546299 | 15:13 |
| openstackgerrit | Vabada proposed openstack-infra/gerritbot master: Fail back to username if user has no full name https://review.openstack.org/546661 | 15:13 |
| fungi | vabada: ^ thanks! reviewing now | 15:14 |
| *** felipemonteiro has joined #openstack-infra | 15:14 | |
| vabada | fungi, thank you for the suggestion. Any comments welcome! | 15:14 |
| *** dklyle has joined #openstack-infra | 15:15 | |
| *** jogo has quit IRC | 15:16 | |
| *** david-lyle has quit IRC | 15:16 | |
| odyssey4me | if any project-config cores have a gap to review https://review.openstack.org/546571 & https://review.openstack.org/546075 & https://review.openstack.org/546299 we'd like to get those imports done asap so that we can get to work on those new roles - appreciate the time! | 15:16 |
| pabelanger | dirk: AJaeger: clarkb: do we have any sort of output from tools/build-image.sh for opensuse-tumbleweed? Also, dirk if you want to rebase, we can maybe see about bring the image online today: https://review.openstack.org/504169/ | 15:17 |
| *** slaweq has quit IRC | 15:20 | |
| *** slaweq has joined #openstack-infra | 15:21 | |
| *** agopi has joined #openstack-infra | 15:24 | |
| *** yamahata has joined #openstack-infra | 15:25 | |
| *** Guest44467 has quit IRC | 15:25 | |
| *** slaweq has quit IRC | 15:26 | |
| *** psachin has quit IRC | 15:26 | |
| *** mgoddard_ has joined #openstack-infra | 15:27 | |
| *** psachin has joined #openstack-infra | 15:28 | |
| mgoddard_ | Hi infra, I'm just looking at https://docs.openstack.org/infra/manual/creators.html, to convert kayobe to an OpenStack-related project. I'm a little confused about the transition to Storyboard - is it intended to be a direct replacement for Launchpad, or should I create a project on both? | 15:28 |
| *** psachin has quit IRC | 15:29 | |
| openstackgerrit | Ifat Afek proposed openstack-infra/project-config master: Added docs and publish-to-pypi jobs to vitrage-tempest-plugin https://review.openstack.org/546667 | 15:29 |
| jlvillal | AJaeger, Thanks. The regex support is nice. Eric Fried did most of the work in implementing it. | 15:30 |
| *** sshnaidm is now known as sshnaidm|afk | 15:33 | |
| *** oidgar has joined #openstack-infra | 15:33 | |
| jlvillal | vabada, fungi: There is a chain of gerritbot patches that would be good to land: https://review.openstack.org/#/c/545469/ | 15:34 |
| jlvillal | As an FYI :) | 15:34 |
| fungi | mgoddard_: for now you can use one or the other but i wouldn't recommend having both | 15:37 |
| openstackgerrit | Merged openstack-infra/nodepool master: Only DeletedNodeWorker should delete nodes https://review.openstack.org/546146 | 15:39 |
| openstackgerrit | Merged openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 15:39 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 15:39 |
| mgoddard_ | fungi: thanks. Is storyboard the future? I'm currently using github for issue and feature tracking, and would prefer to make the switch only once | 15:40 |
| *** oidgar has quit IRC | 15:40 | |
| *** caphrim007 has quit IRC | 15:42 | |
| evrardjp | I am pulling out the ansible config_template plugin, it could be used by more than just osa (and it currently is!) does anyone here have a strong opinion on where I should put it in terms of governance? | 15:45 |
| evrardjp | pulling out of osa plugins* | 15:45 |
| jlvillal | fungi, gerritbot patch: https://review.openstack.org/#/c/318096/ has the same effect for unknown usernames in Gerrit. But already has one +2. And handles more cases it seems. | 15:45 |
| *** VW has joined #openstack-infra | 15:51 | |
| *** signed8bit has quit IRC | 15:51 | |
| *** inc0 has joined #openstack-infra | 15:54 | |
| *** oidgar has joined #openstack-infra | 15:54 | |
| *** r-daneel has joined #openstack-infra | 15:54 | |
| *** yamamoto has quit IRC | 15:55 | |
| fungi | mgoddard_: it's a possible future and one many of us are hanging our hopes on since it has a similar data model to lp but is much easier to run our own instance of rather than relying on a third party | 15:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Add nspawn_.* roles to openstack-ansible https://review.openstack.org/546075 | 15:56 |
| evrardjp | woot! | 15:57 |
| evrardjp | thanks folks! | 15:57 |
| mgoddard_ | fungi: I see, thanks. I'll hold off on the decision for now then | 15:59 |
| *** adarazs_lunch is now known as adarazs | 15:59 | |
| fungi | mgoddard_: a lot of projects are already using sb, but it's up to you whether it's feature-complete enough yet for your needs or whether you still need features from lp which sb lacks at this point | 16:00 |
| *** r-daneel_ has joined #openstack-infra | 16:01 | |
| *** r-daneel has quit IRC | 16:02 | |
| *** r-daneel_ is now known as r-daneel | 16:02 | |
| AJaeger | jlvillal: indeed - thanks efried_omalley ! | 16:02 |
| mgoddard_ | fungi: we're just a small project right now without huge project management requirements. Perhaps we should give storyboard a shot | 16:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Add individual core group for openstack-ansible-os_gnocchi https://review.openstack.org/546299 | 16:02 |
| efried_omalley | what'd I do? | 16:02 |
| *** sshnaidm|afk is now known as sshnaidm | 16:03 | |
| efried_omalley | oh, the regex thing, cool. Yeah, I guess time to start proposing patches to convert to regexes | 16:03 |
| *** dsariel has joined #openstack-infra | 16:03 | |
| AJaeger | efried_omalley: are you efried in general? Then you worked on gerritbot and added regex support - thanks | 16:03 |
| AJaeger | Yes, please! | 16:03 |
| efried_omalley | AJaeger: Yeah, I'm getting in the Irish spirit. | 16:03 |
| efried_omalley | Not a drop of gaelic blood in me, I'm afraid. | 16:04 |
| AJaeger | the Irish are welcoming people ;) | 16:04 |
| fungi | mgoddard_: you probably want to at least skim https://docs.openstack.org/infra/storyboard/user/index.html and try it out (we also have a storyboard-dev.openstack.org sandbox if you want to create test stories/tasks/worklists/boards/whatever) | 16:04 |
| fungi | efried_omalley: i credit my irish heritage with my ability to keep up with all the other drunks at openstack events | 16:05 |
| mgoddard_ | fungi: will do | 16:05 |
| efried_omalley | hahaha | 16:05 |
| *** felipemonteiro has quit IRC | 16:07 | |
| *** felipemonteiro has joined #openstack-infra | 16:07 | |
| AJaeger | infra-root, could you delete a few directories from tarballs.openstack.org, please? http://tarballs.openstack.org/training-labs/dist/build/ http://tarballs.openstack.org/dist/ and http://tarballs.openstack.org/build/ . Those are all training-labs publishing errors ;( | 16:09 |
| fungi | AJaeger: will do shortly | 16:09 |
| *** janki has quit IRC | 16:09 | |
| mnaser | AJaeger: https://review.openstack.org/#/c/546571 -- i think it seems okay but i don't trust myself with new projects yet | 16:10 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import config_template to openstack(-ansible) https://review.openstack.org/546678 | 16:10 |
| *** mayur_ind__ has joined #openstack-infra | 16:10 | |
| AJaeger | mnaser: one question for odyssey4me | 16:10 |
| AJaeger | odyssey4me: do you really want t oimport the stable branches from https://github.com/odyssey4me/openstack-ansible-os_panko/branches ? | 16:11 |
| mayur_ind__ | Hi folks, | 16:11 |
| *** manjeets has quit IRC | 16:11 | |
| mayur_ind__ | We are facing some issue in our CI, no new jobs are triggering. | 16:11 |
| mayur_ind__ | I restarted zuul services and getting following error http://paste.openstack.org/show/680387/ | 16:11 |
| mayur_ind__ | It is failing in following following command = client.connect(self.hostname, username=self.username, port=self.port, key_filename=self.keyfile) | 16:11 |
| efried_omalley | AJaeger: Are comments supported in channels.yaml? | 16:11 |
| mayur_ind__ | Any Idea what may be the issue | 16:11 |
| AJaeger | efried_omalley: at least line 1 has some ;) | 16:12 |
| efried_omalley | beaut, shoulda searched. | 16:12 |
| efried_omalley | thanks | 16:12 |
| AJaeger | efried_omalley: and there are other places - so go for it ;) | 16:12 |
| AJaeger | mnaser: so, change was fine ;9 | 16:13 |
| *** jogo has joined #openstack-infra | 16:13 | |
| *** jogo has joined #openstack-infra | 16:13 | |
| pabelanger | mayur_ind__: looks like a routing issue to gerrit | 16:13 |
| AJaeger | mnaser: could you review https://review.openstack.org/546307 , please? | 16:13 |
| AJaeger | pabelanger: is https://review.openstack.org/543139 fine now? Or should I ask others for review? | 16:14 |
| mnaser | AJaeger: done | 16:14 |
| mnaser | infra-root: is it possible that ubuntu mirrors have/had an issue? http://logs.openstack.org/13/512313/3/gate/openstack-ansible-functional-ubuntu-xenial/07f5fbe/logs/ara/result/d945685e-4122-4c8f-8cfb-9fab0a73ccb2/ | 16:14 |
| *** hongbin has joined #openstack-infra | 16:14 | |
| pabelanger | AJaeger: +2 | 16:15 |
| mnaser | that usually is hints that mirroring failed mid process? | 16:15 |
| pabelanger | mnaser: I can look now | 16:15 |
| mayur_ind__ | pabelanger: But I not did anything, this issue came up suddenly. | 16:15 |
| mnaser | pabelanger: thanks, im going to guess infra didn't hit this because gcc is preinstalled but this is inside osa containers | 16:15 |
| AJaeger | pabelanger: thanks - then I'll +A - and wait for new images... | 16:15 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import config_template to openstack(-ansible) https://review.openstack.org/546678 | 16:16 |
| mayur_ind__ | pabelanger: Its trying to reach 'review.openstack.org' , can this be related to ssh_keys | 16:16 |
| *** pcaruana has quit IRC | 16:16 | |
| openstackgerrit | Mark Goddard proposed openstack-infra/project-config master: Add new project config for kayobe https://review.openstack.org/546681 | 16:18 |
| pabelanger | yah, ubuntu mirror is stale. Cleaning up | 16:18 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Regex gerritbot notifications: #openstack-ansible https://review.openstack.org/546682 | 16:18 |
| *** openstackgerrit has quit IRC | 16:19 | |
| pabelanger | mayur_ind__: I don't think it is SSH keys, it looks like you cannot reach review.o.o from your server. I'd check your networking | 16:19 |
| efried_omalley | AJaeger: There's one ^ | 16:19 |
| AJaeger | efried_omalley: thanks | 16:19 |
| *** openstackgerrit has joined #openstack-infra | 16:19 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add config_template repo base jobs https://review.openstack.org/546683 | 16:19 |
| AJaeger | efried_omalley: i love the -59 ;) | 16:20 |
| *** dsariel has quit IRC | 16:20 | |
| efried_omalley | AJaeger: Yeah, we ought to be able to -<a bunch> in this file now. | 16:20 |
| AJaeger | evrardjp, odyssey4me, I'd like to point out https://review.openstack.org/546682 to you | 16:20 |
| mayur_ind__ | pabelanger: But its pingable. | 16:21 |
| mayur_ind__ | root@sfsshare01:~# ping review.openstack.org PING review.openstack.org (104.130.246.91) 56(84) bytes of data. 64 bytes from review.openstack.org (104.130.246.91): icmp_seq=1 ttl=48 time=272 ms 64 bytes from review.openstack.org (104.130.246.91): icmp_seq=2 ttl=48 time=272 ms | 16:21 |
| mayur_ind__ | ping review.openstack.org | 16:21 |
| mayur_ind__ | PING review.openstack.org (104.130.246.91) 56(84) bytes of data | 16:21 |
| mayur_ind__ | 64 bytes from review.openstack.org (104.130.246.91): icmp_seq=1 ttl=48 time=272 ms | 16:21 |
| pabelanger | mayur_ind__: please use pastebin | 16:21 |
| pabelanger | mayur_ind__: try to SSH into review.o.o using your key and account zuul is using | 16:21 |
| AJaeger | evrardjp: do you want to answer my question on https://review.openstack.org/#/c/546571 ? If this is fine, I can +A | 16:22 |
| fungi | mnaser: pabelanger: see the infra status log, but yes | 16:24 |
| fungi | i'm presently working on updating it manually | 16:24 |
| fungi | Interruption in progress, interrupt again to force-stop it (and risking database corruption!) | 16:24 |
| pabelanger | Oh | 16:24 |
| fungi | i wonder if our update is taking too long | 16:24 |
| pabelanger | fungi: I am running it now | 16:24 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Regex gerritbot notifications: #fuel-tracker https://review.openstack.org/546682 | 16:24 |
| *** dprince has quit IRC | 16:24 | |
| fungi | pabelanger: uh | 16:25 |
| pabelanger | fungi: sorry, I didn't know you were also working on it | 16:25 |
| fungi | did you kill the one i had running then? | 16:25 |
| pabelanger | no | 16:25 |
| pabelanger | there was no reprepro runnin | 16:25 |
| pabelanger | but, was a lock | 16:25 |
| fungi | okay, then it had just died i guess | 16:25 |
| pabelanger | which I did delete | 16:25 |
| fungi | thanks | 16:25 |
| fungi | i guess we try again? | 16:25 |
| pabelanger | yah, it is exporting indices right now | 16:26 |
| fungi | there was a gcc update and a qemu update, among others, so took a while to sync | 16:26 |
| *** slaweq has joined #openstack-infra | 16:26 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Add stable branch notification for openstack-cinder https://review.openstack.org/546686 | 16:26 |
| fungi | and i guess it timed out the kstart maybe? | 16:26 |
| *** slaweq has quit IRC | 16:26 | |
| *** kukacz_ has joined #openstack-infra | 16:26 | |
| pabelanger | fungi: yah, that could be it. might need to bump timeout a little again | 16:26 |
| *** slaweq has joined #openstack-infra | 16:26 | |
| AJaeger | efried_omalley: what did you do here? https://review.openstack.org/#/c/546682/1..2//COMMIT_MSG | 16:27 |
| *** slaweq has quit IRC | 16:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move jenkins/data/bindep-fallback.txt (1/2) https://review.openstack.org/543139 | 16:27 |
| pabelanger | I wonder if we are having routing issue on mirror-update.o.o to afs cell | 16:27 |
| mnaser | fungi, pabelanger: oh sorry, i should have checked, thank you | 16:27 |
| AJaeger | efried_omalley: better don't reuse change-id this way | 16:27 |
| pabelanger | I see a lot of AFS lost connections | 16:27 |
| efried_omalley | AJaeger: Oh, crap. I forgot to change the change ID | 16:27 |
| pabelanger | [Wed Feb 21 15:48:21 2018] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -512) (all multi-homed ip addresses down for the server) | 16:27 |
| *** slaweq has joined #openstack-infra | 16:27 | |
| efried_omalley | AJaeger: Bad copy/paste. | 16:27 |
| AJaeger | efried_omalley: download first version, git commit --amend to remove change-id and push up again... | 16:28 |
| efried_omalley | AJaeger: Yuh, thanks. | 16:28 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Regex gerritbot notifications: #openstack-ansible https://review.openstack.org/546687 | 16:29 |
| *** kukacz_ is now known as kukacz | 16:29 | |
| *** oidgar has quit IRC | 16:29 | |
| AJaeger | efried_omalley: smcginnis is helping you, see https://review.openstack.org/546686 ;) | 16:30 |
| *** sree_ has joined #openstack-infra | 16:31 | |
| persia | Am I reading reprepro's source correctly that it updates Packages.gz before it copies the packages to the pool? | 16:31 |
| *** sree_ is now known as Guest41691 | 16:31 | |
| fungi | AJaeger: should i also remove http://tarballs.openstack.org/training-labs/dist/ ? | 16:31 |
| efried_omalley | AJaeger: Thanks for the heads up. | 16:31 |
| AJaeger | fungi, no that one is ok | 16:31 |
| *** slaweq has quit IRC | 16:31 | |
| AJaeger | fungi: just the /training-labs/dist/build, please | 16:32 |
| fungi | and the other two at root level | 16:32 |
| fungi | removing now | 16:32 |
| AJaeger | fungi: and http://tarballs.openstack.org/training-labs/images/ as well, please | 16:32 |
| AJaeger | thanks, fungi | 16:32 |
| *** slaweq has joined #openstack-infra | 16:33 | |
| mayur_ind__ | pabelanger: I tried doing ssh and observed following http://paste.openstack.org/show/680392/ | 16:33 |
| *** wolverineav has joined #openstack-infra | 16:33 | |
| fungi | AJaeger: oh, wow, /training-labs/images/ is pretty large | 16:33 |
| AJaeger | fungi: just one image - but yeah LARGE | 16:34 |
| smcginnis | efried_omalley: I was just going to push up a change for all stable branches. | 16:34 |
| smcginnis | efried_omalley: You'd rather go project by project? | 16:34 |
| AJaeger | mnaser: want to update https://review.openstack.org/#/c/546432/1 to follow efried_omalley's changes? | 16:34 |
| efried_omalley | smcginnis: I considered it both ways. My thought was that it would be best to get a nod from the team whose channel we're updating. And that would be best done channel by channel. | 16:34 |
| pabelanger | mayur_ind__: you didn't use -p29418 for 2nd attempt | 16:34 |
| AJaeger | smcginnis: he does some other changes as well, see https://review.openstack.org/546687 | 16:35 |
| pabelanger | that is needed to access gerrit | 16:35 |
| fungi | AJaeger: i've moved all four of those trees out of the tarballs site now | 16:35 |
| AJaeger | fungi, thanks. | 16:35 |
| efried_omalley | smcginnis: But I'm not stuck on that philosophy. | 16:35 |
| *** Guest41691 has quit IRC | 16:35 | |
| *** dsariel has joined #openstack-infra | 16:36 | |
| smcginnis | AJaeger: Yeah, that was what I was asking. ;) | 16:36 |
| fungi | #status log manually removed /srv/static/tarballs/training-labs/dist/build, /srv/static/tarballs/training-labs/images, /srv/static/tarballs/dist and /srv/static/tarballs/build from static.openstack.org | 16:36 |
| openstackstatus | fungi: finished logging | 16:36 |
| smcginnis | I will just push up what I have since it's kind of simple and would at least take care of that part if we want to go ahead with it. | 16:36 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Regex gerritbot notifications: #openstack-nova https://review.openstack.org/546689 | 16:36 |
| mayur_ind__ | pabelanger: yah, now with port I am able to do ssh. http://paste.openstack.org/show/680393/ | 16:36 |
| mayur_ind__ | so what may be the real issue :-( | 16:37 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch notifications for stable branches to match all https://review.openstack.org/546690 | 16:37 |
| *** dprince has joined #openstack-infra | 16:37 | |
| *** slaweq has quit IRC | 16:38 | |
| *** wolverineav has quit IRC | 16:38 | |
| *** slaweq has joined #openstack-infra | 16:38 | |
| AJaeger | efried_omalley: ^ | 16:38 |
| *** yamahata has quit IRC | 16:39 | |
| pabelanger | mayur_ind__: try starting zuul again, like I said. I don't believe the failure was caused by SSH keys, but a routing issue from your zuul instance to gerrit (review.o.o) | 16:39 |
| pabelanger | you could also run tcpdump on your zuul instance to get more information | 16:40 |
| AJaeger | smcginnis: if you anyhow need to edit your change - what about not touching the repos that efried has changes up to avoid merge conflicts? | 16:42 |
| *** r-daneel_ has joined #openstack-infra | 16:43 | |
| *** r-daneel has quit IRC | 16:44 | |
| *** r-daneel_ is now known as r-daneel | 16:44 | |
| fungi | efried_omalley: smcginnis: considered as an alternative making gerritbot default to matching on all branches if none are specified? i'd be willing to bet that's the default behavior most projects want anyway | 16:44 |
| *** slaweq has quit IRC | 16:44 | |
| *** slaweq has joined #openstack-infra | 16:44 | |
| *** slaweq has quit IRC | 16:44 | |
| AJaeger | fungi: nice idea! | 16:45 |
| fungi | there are at least some which will want a ^.*$ branch matcher at least (e.g., openstack-infra/publications uses a separate branch per presentation) | 16:45 |
| *** slaweq has joined #openstack-infra | 16:45 | |
| *** tpsilva has joined #openstack-infra | 16:45 | |
| *** andreas_s has quit IRC | 16:46 | |
| AJaeger | config-core, care to review https://review.openstack.org/545234 to refactor our static publish jobs, please? | 16:47 |
| dmsimard | fungi: +1 on that idea | 16:47 |
| pabelanger | fungi: mnaser: still generating indices, but looks to be working. | 16:47 |
| mnaser | pabelanger: nice, thank you! | 16:48 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Update bot for puppet-openstack to include Queens https://review.openstack.org/546432 | 16:48 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Added check to ensure template usage https://review.openstack.org/529318 | 16:48 |
| mnaser | AJaeger: ^ voila | 16:48 |
| fungi | pabelanger: yep, i mean it was working earlier too it just seems to have taken too long | 16:49 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/openstack-zuul-jobs master: Remove now unused jobs from various sahara repositories https://review.openstack.org/546694 | 16:49 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Remove migrated jobs for sahara repositories https://review.openstack.org/546695 | 16:49 |
| *** salv-orl_ has quit IRC | 16:49 | |
| *** slaweq has quit IRC | 16:50 | |
| *** salv-orlando has joined #openstack-infra | 16:50 | |
| pabelanger | fungi: I think we lost access to AFS, there are corrupt DB errors in ubuntu-mirror.log | 16:50 |
| openstackgerrit | Mark Goddard proposed openstack-infra/project-config master: Add new project config for kayobe https://review.openstack.org/546681 | 16:50 |
| openstackgerrit | Mark Goddard proposed openstack-infra/project-config master: Add Zuul project for kayobe https://review.openstack.org/546696 | 16:50 |
| pabelanger | so, might be taking longer to rebuild DB | 16:50 |
| pabelanger | doesn't usually take this long | 16:50 |
| fungi | :( | 16:50 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Remove migrated jobs for sahara repositories https://review.openstack.org/546695 | 16:50 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/openstack-zuul-jobs master: Remove now unused jobs from various sahara repositories https://review.openstack.org/546694 | 16:50 |
| fungi | pabelanger: doesn't seem to have been a reboot of afsdb01 or 02 at least | 16:51 |
| tosky | umpf, following the "Moving Legacy Jobs to Projects" section of the manual is now more complicated | 16:51 |
| tosky | previously we could use the ID in Depends-On and Needed-By; now we need to submit both patches, check the review number, amend, resubmit | 16:51 |
| AJaeger | tosky: yes ;( | 16:52 |
| rcarrillocruz | hey folks, poking DIB and fedora images. I noticed that the sshd_config file has PermitRootLogin: yes and PasswordAuthentication: yes , but can't see any elements setting that. Is that set on the base upstream fedora image? | 16:53 |
| *** salv-orlando has quit IRC | 16:54 | |
| fungi | rcarrillocruz: probably | 16:54 |
| rcarrillocruz | hmm | 16:54 |
| fungi | rcarrillocruz: is that when using the fedora or fedora-minimal element (or both)? | 16:55 |
| rcarrillocruz | k, think i'll create an element to lock that down | 16:55 |
| mayur_ind__ | pabelanger: I think you are right, here is output of tcpdump http://paste.openstack.org/show/680395/ | 16:55 |
| rcarrillocruz | fungi: just poked at 'fedora' | 16:55 |
| rcarrillocruz | don't know if minimal is the same | 16:55 |
| mayur_ind__ | pabelanger: can you suggest where should I look for solution. | 16:55 |
| *** yamamoto has joined #openstack-infra | 16:56 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 16:56 |
| rcarrillocruz | made me nervous, cos i was injecting ssh key with 'devuser', but devuser has an envvar to set user password (disable by default tho) | 16:56 |
| fungi | mayur_ind__: when you tested ssh, you used ipv4 but this looks like ipv6... maybe v6 routing is broken between you and review.openstack.org? | 16:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: remove legacy-rally-dsvm-fakevirt-heat https://review.openstack.org/546699 | 16:57 |
| fungi | er, scratch that. i see your manual ssh test doesn't indicate whether it was v4 or v6 | 16:57 |
| *** timrc has quit IRC | 16:58 | |
| efried_omalley | fungi: How would you turn that off? | 16:58 |
| fungi | mayur_ind__: you might want to try your manual testing with ssh -6 just to confirm it's actually testing over ipv6 | 16:58 |
| efried_omalley | You mean if the section is absent, default to .*, but if present, use whatever they say? | 16:59 |
| fungi | efried_omalley: yeah | 16:59 |
| ssbarnea | https://review.openstack.org/#/c/546284/ --- bindep improvement, easy to review. | 16:59 |
| efried_omalley | that'd work | 16:59 |
| fungi | efried_omalley: currently there's no reason to omit the branches section since that would result in no reporting for changes | 16:59 |
| efried_omalley | right | 16:59 |
| fungi | efried_omalley: so i think it's something we can safely assume downstream users would only ever have configured that way by accident/in error | 16:59 |
| *** myoung is now known as myoung|food | 17:01 | |
| AJaeger | config-core, I added the abstract attribute to some base jobs in https://review.openstack.org/545603 and https://review.openstack.org/#/c/545442 - could you review them, please? | 17:01 |
| *** yamamoto has quit IRC | 17:02 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:03 | |
| *** gyee has joined #openstack-infra | 17:04 | |
| pabelanger | mayur_ind__: wonder if we are having ipv6 routing issues to review.o.o, cc fungi | 17:05 |
| mayur_ind__ | fungi: I tried this, http://paste.openstack.org/show/680397/ . Did I tried same what you were saying? | 17:05 |
| pabelanger | mayur_ind__: you could try and force ipv4 and see if that works | 17:05 |
| *** dsariel has quit IRC | 17:05 | |
| *** caphrim007 has joined #openstack-infra | 17:05 | |
| pabelanger | I can't test ipv6 myself to review.o.o | 17:06 |
| pabelanger | maybe somebody else could | 17:06 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 17:06 |
| fungi | pabelanger: i'm connected to review.o.o via ipv6 right now | 17:06 |
| pabelanger | k | 17:06 |
| jlvillal | mordred, clarkb ^^^ I'm not sure if that is correct https://review.openstack.org/546700 | 17:06 |
| pabelanger | mayur_ind__: so, sounds like ipv6 issue in your network some place. | 17:06 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update Gentoo Hardened profiles (now stable) https://review.openstack.org/546701 | 17:06 |
| *** olaph1 has joined #openstack-infra | 17:06 | |
| jlvillal | mordred, clarkb: So reviews pointing out mistakes much appreciated. I'm not a puppet master :( | 17:07 |
| fungi | mayur_ind__: yep, that looks like you probably have v6 connectivity issues... trying the same with -4 instead pf -6 works? | 17:07 |
| *** olaph has quit IRC | 17:07 | |
| *** caphrim007 has quit IRC | 17:08 | |
| *** dklyle has quit IRC | 17:08 | |
| *** caphrim007 has joined #openstack-infra | 17:09 | |
| mayur_ind__ | pabelanger: My other CI server is working properly, and it is showing same behavior with ipv6 http://paste.openstack.org/show/680399/ , I dont think it is the issue. | 17:10 |
| *** caphrim007 has quit IRC | 17:10 | |
| fungi | mayur_ind__: does tcpdump on your other server show it connecting over ipv6 or ipv4? | 17:10 |
| *** caphrim007 has joined #openstack-infra | 17:10 | |
| *** caphrim007 has quit IRC | 17:10 | |
| *** timrc has joined #openstack-infra | 17:11 | |
| *** caphrim007 has joined #openstack-infra | 17:11 | |
| pabelanger | fungi: mayur_ind__: yah, if the other server is working but ssh fails to ipv6, it must be using ipv4 | 17:13 |
| fungi | or the v6 connectivity issue is specific to the one machine having a problem | 17:13 |
| *** felipemonteiro_ has joined #openstack-infra | 17:14 | |
| pabelanger | the pastebin shows ssh -6 to review.o.o failing, I assume that is the server working | 17:14 |
| fungi | all the paste examples seem to have been from one server (sfsshare01) so i assumed that was the broken one | 17:15 |
| pabelanger | http://paste.openstack.org/show/680399/ looks to be from CIPuneMaster | 17:15 |
| *** yamahata has joined #openstack-infra | 17:15 | |
| *** felipemonteiro has quit IRC | 17:16 | |
| pabelanger | I think that is the working (zuul?) one | 17:16 |
| pabelanger | but, could be confusing things | 17:16 |
| mayur_ind__ | pabelanger : yes this is working one | 17:16 |
| pabelanger | it must be using ipv4 to review.o.o | 17:16 |
| fungi | got it. i missed that one and was only looking at the other 4 pastes | 17:17 |
| fungi | so yes, my guess is that tcpdump will show that CIPuneMaster is connecting over ipv4 and sfsshare01 is trying (but failing) to connect over ipv6 | 17:17 |
| pabelanger | agree | 17:18 |
| fungi | so you probably have some sort of global ipv6 configuration on the local network which the server started trying to use, but either it's broken or has some egress firewalling which is rejecting your attempts to connect to review.openstack.org on 29418/tcp over ipv6 | 17:19 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 17:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Import openstack-ansible-os_panko role https://review.openstack.org/546571 | 17:22 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 17:24 |
| mayur_ind__ | fungi: Both the server are trying over ipv4. http://paste.openstack.org/show/680400/ | 17:24 |
| openstackgerrit | Merged openstack-infra/gerritbot master: Add _get_username method to discover username https://review.openstack.org/318096 | 17:25 |
| *** e0ne has quit IRC | 17:25 | |
| *** florianf has quit IRC | 17:25 | |
| *** panda is now known as panda|off | 17:27 | |
| evrardjp | AJaeger: sorry I was out | 17:27 |
| evrardjp | will check right now | 17:27 |
| *** imacdonn has quit IRC | 17:27 | |
| *** imacdonn has joined #openstack-infra | 17:27 | |
| pabelanger | mayur_ind__: run tcpdump with -n flag to stop resolving dns | 17:28 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 17:28 |
| pabelanger | I suspect it will show different | 17:28 |
| *** zoli is now known as zoli|gone | 17:30 | |
| fungi | mayur_ind__: pabelanger: the top (non-working) one in that paste is definitely iv6 | 17:30 |
| *** zoli|gone is now known as zoli | 17:30 | |
| fungi | as it shows the client's source address in v6 notation | 17:30 |
| *** caromeo has quit IRC | 17:31 | |
| pabelanger | yup | 17:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add OSA os_panko repo base jobs https://review.openstack.org/546573 | 17:31 |
| fungi | the bottom one, yes, can't tell without -n because it's using resolved hostnames | 17:31 |
| mayur_ind__ | pabelanger: with -n option , http://paste.openstack.org/show/680401/ | 17:32 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add OSA nspawn host/container repo base jobs https://review.openstack.org/546294 | 17:32 |
| mayur_ind__ | fungi: yes you are right | 17:32 |
| pabelanger | mayur_ind__: yah, so that confirms working is using ipv4 and is broken on ipv6 | 17:33 |
| fungi | so options are to troubleshoot why ipv6 connectivity is broken for your network there, or attempt to disable v6 routing on the system(s) | 17:34 |
| mayur_ind__ | pabelanger: Yah you right | 17:34 |
| fungi | but i don't think there's anything for us to do at this point | 17:35 |
| fungi | (ipv6 connectivity to the same server is working for other clients) | 17:35 |
| mayur_ind__ | fungi: thats a lot of help , thanks a lot to both of you | 17:35 |
| fungi | you're welcome! | 17:35 |
| pabelanger | np | 17:36 |
| mayur_ind__ | I will try disabling v6 routing on my system | 17:36 |
| mayur_ind__ | hope that will work | 17:36 |
| *** katkapilatova has left #openstack-infra | 17:39 | |
| *** katkapilatova has quit IRC | 17:39 | |
| *** david-lyle has joined #openstack-infra | 17:41 | |
| AJaeger | evrardjp: no worries... | 17:42 |
| evrardjp | do you still have a question there? It seems one has merged and the other was okay to me at previous patchset | 17:44 |
| AJaeger | evrardjp: I'm fine now, thanks | 17:44 |
| evrardjp | if you need anything just call :) | 17:45 |
| AJaeger | evrardjp: https://review.openstack.org/546687 si the other one, efried_omalley had a wrong push so renamed now | 17:45 |
| *** openstackgerrit has quit IRC | 17:48 | |
| *** myoung|food is now known as myoung | 17:49 | |
| evrardjp | AJaeger: openstack-ansible comes between openstack-almanach and openstack-architecture, so ? | 17:49 |
| * evrardjp confused | 17:49 | |
| *** openstackgerrit has joined #openstack-infra | 17:49 | |
| openstackgerrit | Eric Fried proposed openstack-infra/gerritbot master: Match all by default https://review.openstack.org/546712 | 17:49 |
| efried_omalley | fungi, smcginnis, AJaeger: Your wish is my command ^ | 17:50 |
| AJaeger | evrardjp: I didn't know that the change was in the wrong place when I asked. The order is that ^ comes before a, so needs to be first entry. question to you is whether those regexes are fine | 17:50 |
| evrardjp | efried_omalley: ALL the things ? | 17:50 |
| evrardjp | THat's a lot. | 17:50 |
| efried_omalley | Y knot? | 17:50 |
| evrardjp | :) | 17:51 |
| efried_omalley | I got started and figured... who knows what crazypants downstream consumer will want? | 17:51 |
| evrardjp | AJaeger: okay so it makes sense if you want also the ^ to be counted in the alphabetical order | 17:51 |
| *** mgoddard_ has quit IRC | 17:53 | |
| evrardjp | let me patch that quick then | 17:54 |
| *** olaph has joined #openstack-infra | 17:54 | |
| evrardjp | not sure why it's needed to have alphabetical order though. | 17:55 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Regex gerritbot notifications: #openstack-ansible https://review.openstack.org/546687 | 17:55 |
| *** olaph1 has quit IRC | 17:55 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Regex gerritbot notifications: #openstack-ansible https://review.openstack.org/546687 | 17:57 |
| *** VW_ has joined #openstack-infra | 17:57 | |
| *** VW_ has quit IRC | 17:58 | |
| *** VW_ has joined #openstack-infra | 17:58 | |
| *** Swami has joined #openstack-infra | 17:58 | |
| *** yamamoto has joined #openstack-infra | 17:58 | |
| *** eernst has quit IRC | 17:59 | |
| *** eernst has joined #openstack-infra | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| efried_omalley | the faaahh? | 18:00 |
| efried_omalley | AJaeger: Why is alpha order important? | 18:00 |
| *** VW has quit IRC | 18:01 | |
| efried_omalley | I mean, I guess if the entry is 60 lines long it's one thing. | 18:01 |
| *** yamamoto has quit IRC | 18:04 | |
| *** xarses has joined #openstack-infra | 18:05 | |
| *** andreww has joined #openstack-infra | 18:06 | |
| *** jpich has quit IRC | 18:07 | |
| *** ykarel|afk has quit IRC | 18:07 | |
| *** ralonsoh has quit IRC | 18:09 | |
| *** xarses has quit IRC | 18:09 | |
| *** mayur_ind__ has left #openstack-infra | 18:09 | |
| AJaeger | efried_omalley, we wrote a check that looks for those - was helpful with those 60 lines... | 18:14 |
| efried_omalley | AJaeger: Where's that check? Maybe we can limit it to stanzas over a certain size. | 18:14 |
| *** felipemonteiro_ has quit IRC | 18:14 | |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| jlvillal | efried_omalley, What is the downside to having it in alphabetical order? | 18:15 |
| efried_omalley | jlvillal: Nothing really, just a pain. | 18:15 |
| efried_omalley | jlvillal: Also prevents things from being in "logical" order, if that order isn't alphabetical. | 18:16 |
| AJaeger | it helps us to keep some sanity ;) | 18:16 |
| efried_omalley | shrug | 18:16 |
| efried_omalley | TIL | 18:16 |
| AJaeger | efried_omalley: tools/check-channels-yaml.sh | 18:16 |
| efried_omalley | thx | 18:17 |
| *** jpena is now known as jpena|off | 18:18 | |
| jlvillal | Any puppet masters out there that can help me debug this build failure? http://logs.openstack.org/00/546700/3/check/legacy-infra-puppet-apply-3-ubuntu-trusty/49bebe7/job-output.txt.gz#_2018-02-21_18_13_49_076978 | 18:18 |
| jlvillal | It just says "puppet did not run". And I don't know why :( | 18:19 |
| jlvillal | Oh never mind I found this! http://logs.openstack.org/00/546700/3/check/legacy-infra-puppet-apply-3-ubuntu-trusty/49bebe7/openstack-infra/system-config/applytest/puppetapplytest01.final.out.FAILED | 18:19 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Remove migrated jobs for sahara repositories https://review.openstack.org/546695 | 18:21 |
| * jlvillal is confused... | 18:21 | |
| fungi | jlvillal: yeah, those job logs are a little interesting to have to find your way around in the first time you encounter them, but looking in openstack-infra/system-config/applytest/ for any ending in .FAILED is where i usually start | 18:21 |
| fungi | Could not find dependency Vcsrepo[https://git.openstack.org/openstack-infra/gerritbot] for Exec[install-gerritbot] at /etc/puppet/modules/gerritbot/manifests/init.pp:38 | 18:22 |
| jlvillal | fungi, Thanks. I think I am using wrong variable name... | 18:22 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 18:23 |
| * jlvillal crosses his fingers... | 18:23 | |
| *** tesseract has quit IRC | 18:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/publications zuul: Add Zuulv3 presentty presentation https://review.openstack.org/546724 | 18:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/publications zuul: Update architecture and gertty slides https://review.openstack.org/546727 | 18:30 |
| *** r-daneel_ has joined #openstack-infra | 18:31 | |
| andreaf | corvus why? http://logs.openstack.org/74/528074/6/check/devstack-multinode/3e6a464/ara/result/12dd5411-d7a6-4f02-87e0-c086e72b118f/ | 18:31 |
| corvus | andreaf: that's probably accidentally running on the executor, not the remote node | 18:31 |
| corvus | andreaf: (localhost==executor) | 18:32 |
| *** r-daneel has quit IRC | 18:32 | |
| *** r-daneel_ is now known as r-daneel | 18:32 | |
| AJaeger | config-core, could you review https://review.openstack.org/545234 to refactor static publish jobs as well as https://review.openstack.org/545442 and https://review.openstack.org/545603 to mark base jobs as abstract, please? | 18:32 |
| corvus | andreaf: what's the goal? | 18:33 |
| andreaf | corvus uhm... ARA says controller | 18:33 |
| corvus | andreaf: get a certificate from one host and put it on all others? | 18:33 |
| andreaf | corvus push CA data from controller to all other nodes | 18:33 |
| andreaf | corvus yeah | 18:33 |
| fungi | is that something devstack-gate is doing in legacy jobs? | 18:33 |
| corvus | andreaf: we need one more step | 18:33 |
| andreaf | fungi yes | 18:34 |
| corvus | andreaf: we need to pull the cert from the controller to executor, then push from executor to others | 18:34 |
| andreaf | fungi https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n756 | 18:34 |
| pabelanger | you should be able to rsync between controller and compute without executor | 18:35 |
| corvus | andreaf: so add a pull step to synchronose src=/opt/stack/whatever dest=ca-bundle.pem then change the push to src=ca-bundle.pem and dest=/opt/stack/whatever | 18:35 |
| corvus | pabelanger: nope, can't rsync to two remote hosts | 18:36 |
| corvus | or rather between two remote hosts | 18:36 |
| *** eernst has quit IRC | 18:36 | |
| fungi | you could run rsync on one to copy files to the other though (without involving the executor) | 18:37 |
| corvus | it's always push or pull with the localhost(executor) on one end or the other | 18:37 |
| corvus | fungi: yes | 18:37 |
| andreaf | corvus why is that? | 18:37 |
| fungi | assuming the nodes are set up with keys to log into one another, et cetera | 18:37 |
| corvus | andreaf: that's how rsync works, you can't do a 3-point rsync | 18:37 |
| fungi | andreaf: rsync doesn't support operating on one host to copy files between two other hosts | 18:37 |
| fungi | same as, e.g., ftp | 18:38 |
| fungi | with rsync you can copy files from remote to local or local to remote or local to local but not remote to remote | 18:38 |
| andreaf | corvus fungi well I though it would ssh into | 18:38 |
| andreaf | I though ansible would ssh into controller and rsync from there to the other host | 18:39 |
| corvus | so yeah, since we have ssh set up between nodes in this job, you could do fungi's suggestion of just running rsync yourself to sync directly from controller to others | 18:39 |
| corvus | andreaf: you might be able to have ansible treat the controller as a jump host or something... | 18:39 |
| AJaeger | where can I see nodepool log files and image list ? previously we had http://nodepool.openstack.org/image-list - what is this one now? | 18:39 |
| fungi | AJaeger: published on the corresponding builders now (nb01.o.o et al) | 18:40 |
| corvus | andreaf: i favor the pull/push thing -- it's easiest and most ansibly | 18:40 |
| AJaeger | fungi: n*b* - ah, thanks | 18:40 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Handle stestr execution for stackviz https://review.openstack.org/529490 | 18:41 |
| pabelanger | corvus: odd, I would have expected https://stackoverflow.com/questions/25505146/how-to-copy-files-between-two-nodes-using-ansible to work, 2nd answer has better reply IMO | 18:41 |
| andreaf | corvus: if I use delegate_to: controller would zuul allow that? | 18:41 |
| AJaeger | fungi: http://nb01.openstack.org/ gives logs but http://nb01.openstack.org/image-list gives 503 - do we have that still somewhere? | 18:41 |
| fungi | AJaeger: oh, i thought you were asking for the image build logs | 18:42 |
| AJaeger | fungi: for both ;) | 18:42 |
| fungi | pabelanger: that ubuntu mirror update is still going, huh? | 18:43 |
| AJaeger | build logs will help me already to figure out when images are build and I can recheck my bindep changes... | 18:43 |
| *** wolverineav has joined #openstack-infra | 18:43 | |
| corvus | andreaf: worth a try :) | 18:43 |
| openstackgerrit | Merged openstack-infra/nodepool master: Source /etc/ci/mirror_info.sh for mirrors https://review.openstack.org/546032 | 18:43 |
| pabelanger | fungi: yah :( I think it is crawling all files again to build up the database | 18:43 |
| *** wolverineav has quit IRC | 18:43 | |
| *** wolverineav has joined #openstack-infra | 18:43 | |
| *** wolverineav has quit IRC | 18:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Unset finger client timeout after connect https://review.openstack.org/546735 | 18:43 |
| *** wolverineav has joined #openstack-infra | 18:44 | |
| pabelanger | andreaf: left a comment on 528074 | 18:45 |
| fungi | AJaeger: yeah, looks like we have these rewrites http://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/templates/nodepool-log.vhost.erb#n31 but i only see that listening on the nodepool launchers (nl01.o.o et al) and we're not running a webserver there nor do our iptables rules allow access to 8005/tcp | 18:48 |
| fungi | Shrews: ^ what's the intended way to access that? or is that still a work in progress? | 18:48 |
| AJaeger | fungi: no need to fix now ;) Just curious.. Thanks! | 18:49 |
| fungi | well, you got me curious too ;) | 18:49 |
| *** tosky has quit IRC | 18:49 | |
| *** eernst has joined #openstack-infra | 18:49 | |
| Shrews | fungi: *shrug* that's sort of a new thing | 18:49 |
| Shrews | i.e., i don't know what we (in openstack infra) want to do with that (if anything) | 18:52 |
| *** dhajare_ has joined #openstack-infra | 18:53 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 18:53 |
| fungi | Shrews: got it. i guess AJaeger wants to have a way to see a list of current images | 18:54 |
| fungi | and used to hit a similar url at some point in the past (perhaps pre-v3?) | 18:55 |
| pabelanger | mordred: ianw: clarkb: fungi: https://review.openstack.org/485748 is another glean (systemd) fix, that bring debian-jessie back online for nodepool dsvm testing and fixes a long standing issue with dependency loops in systemd. Could I get a review please. | 18:57 |
| AJaeger | fungi: I just wanted to know when we have new images - looking at log files will be good as plan b - it does not show me on one page that all are *successfully* build. | 18:58 |
| *** e0ne has joined #openstack-infra | 18:59 | |
| *** yamamoto has joined #openstack-infra | 19:00 | |
| AJaeger | infra-root, https://review.openstack.org/#/c/546571/ merged two hours ago - but https://review.openstack.org/#/c/546573/ says "he project "openstack/openstack-ansible-os_panko" was not found. " - http://git.openstack.org/cgit/openstack/openstack-ansible-os_panko/ exists. | 19:00 |
| *** e0ne has quit IRC | 19:00 | |
| *** VW_ has quit IRC | 19:01 | |
| *** VW has joined #openstack-infra | 19:02 | |
| *** andreas_s has joined #openstack-infra | 19:02 | |
| *** tbarron has quit IRC | 19:02 | |
| *** VW has quit IRC | 19:03 | |
| *** salv-orlando has joined #openstack-infra | 19:03 | |
| *** gtmanfred has quit IRC | 19:03 | |
| *** VW has joined #openstack-infra | 19:03 | |
| dmsimard | that's.... interesting | 19:05 |
| AJaeger | is puppet disabled on one of the servers? | 19:05 |
| fungi | zuul shouldn't be relying on puppet to recognize config updates any longer | 19:06 |
| *** yamamoto has quit IRC | 19:06 | |
| corvus | fungi: it does for new project adds | 19:06 |
| dmsimard | It looks like it's loaded properly: 2018-02-21 17:59:34,155 INFO zuul.TenantParser: Loading configuration from openstack/openstack-ansible-os_panko/zuul.d/project.yaml@master | 19:06 |
| fungi | corvus: oh! good to know, i didn't realize that was the case | 19:06 |
| dmsimard | 17:59, it looks like it was loaded about 2 minutes after the original syntax error | 19:06 |
| dmsimard | AJaeger: ^ | 19:06 |
| dmsimard | AJaeger: I'll try a recheck and it should be good now I guess | 19:07 |
| fungi | aha, so delayed | 19:07 |
| *** andreas_s has quit IRC | 19:07 | |
| dmsimard | What timezone is that in ? | 19:07 |
| dmsimard | That doesn't look UTC | 19:07 |
| dmsimard | Ah yeah it is, my bad I'm confused | 19:08 |
| dmsimard | nope, Zuul still complaining about syntax error, that's peculiar | 19:08 |
| AJaeger | dmsimard: is there a bug in the file? Let me check | 19:09 |
| dmsimard | Zuul is like... "let me tell you about why this is a syntax error" and proceeds to print out the exact part where the project is written.. is there a non-ascii/utf8 character or something maybe ? | 19:09 |
| dmsimard | bad indentation ? digging.. | 19:10 |
| AJaeger | odyssey4me: next time, please do not import a zuul.yaml file ;) | 19:10 |
| dmsimard | AJaeger: oh you have a point, Zuul is likely confused because of https://github.com/openstack/openstack-ansible-os_panko/blob/master/zuul.d/project.yaml | 19:11 |
| AJaeger | the zuul/main.yaml change merged, so it *should* be ok... | 19:12 |
| dmsimard | odyssey4me: can you try to remove the configuration from the openstack-ansible repo before we try to add the ones in project-config ? | 19:12 |
| * AJaeger sees no errors in the file | 19:12 | |
| corvus | Exception: Job openstack-ansible-dir-ubuntu-xenial in openstack/openstack-ansible-nspawn_container_create is not permitted to shadow job openstack-ansible-dir-ubuntu-xenial in openstack/openstack-ansible-lxc_container_create | 19:12 |
| corvus | that's the error that caused zuul to abort the reconfig which added the project | 19:13 |
| *** gtmanfred has joined #openstack-infra | 19:13 | |
| corvus | so we're running with the previous tenant config now, which doesn't have that project | 19:13 |
| *** tbarron has joined #openstack-infra | 19:13 | |
| AJaeger | ah! | 19:14 |
| dmsimard | so that's another patch that merged | 19:14 |
| * AJaeger patches openstack/openstack-ansible-nspawn_container_create quickly | 19:14 | |
| AJaeger | dmsimard: taht wasn't on my radar, sorry | 19:14 |
| dmsimard | AJaeger: we couldn't have known, the error is indeed in the logs though | 19:14 |
| dmsimard | corvus: is there a way to bubble that up and make it more obvious somehow ? | 19:15 |
| AJaeger | evrardjp, odyssey4me, please see above ^ | 19:15 |
| AJaeger | should we add a check that upstream imports do *not* include zuul.yaml etc? | 19:15 |
| dmsimard | AJaeger: so we should probably make sure that there's no zuul configuration in the repos we're importing | 19:15 |
| dmsimard | I think so | 19:15 |
| *** olaph has quit IRC | 19:15 | |
| *** olaph1 has joined #openstack-infra | 19:15 | |
| * AJaeger took the big axe - https://review.openstack.org/546746 | 19:17 | |
| fungi | yeah, when i reviewed the recent ones i looked for extra branches and tags but forgot to check for zuul configs | 19:17 |
| AJaeger | infra-root, can you force merge that one, please? ^ | 19:17 |
| AJaeger | evrardjp, odyssey4me, FYI ^ | 19:17 |
| * AJaeger works on the zuul import check | 19:17 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import config_template to openstack(-ansible) https://review.openstack.org/546678 | 19:17 |
| dmsimard | AJaeger: it's a bit tricky if we need to check all branches :/ | 19:18 |
| corvus | dmsimard: there's patch in progress to make tenant config errors visible in the web | 19:18 |
| *** esberglu has quit IRC | 19:19 | |
| evrardjp | sorry I was in a meeting | 19:19 |
| evrardjp | Darn many excuses today | 19:19 |
| dmsimard | corvus: awesome | 19:19 |
| evrardjp | could someone explain what's the problem? | 19:20 |
| *** Goneri has quit IRC | 19:20 | |
| AJaeger | evrardjp: see https://review.openstack.org/546746 - your new repo has a broken zuul config. | 19:20 |
| AJaeger | evrardjp: and thus we refuse to handle that repo completely | 19:21 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git https://review.openstack.org/546700 | 19:21 |
| evrardjp | let me rephrase, what's wrong? | 19:21 |
| *** rossella_s has quit IRC | 19:22 | |
| evrardjp | "is not permitted to shadow job openstack-ansible-dir-ubuntu-xenial" | 19:22 |
| AJaeger | evrardjp: your imported repo has a broken zuul configuration - and thus we need to remove it. | 19:22 |
| evrardjp | still understood that -- I don't understand that message | 19:23 |
| *** efried_omalley has quit IRC | 19:23 | |
| evrardjp | We use to be able to redefine the jobs in the repos | 19:23 |
| fungi | zuul config changes are self-testing, but our repo import process does not pre-test your zuul configuration | 19:23 |
| AJaeger | evrardjp: http://git.openstack.org/cgit/openstack/openstack-ansible-lxc_container_create/tree/zuul.d/jobs.yaml#n24 | 19:23 |
| dmsimard | evrardjp: tl;dr, you had good intentions in preparing the zuul config in-tree but we'll need to avoid doing that in the future because we can't validate it before we import it. We'll be able to iterate and figure out what the real issue is after we revert the broken state | 19:23 |
| AJaeger | evrardjp: you defined a *duplicate* job - which is not allowed | 19:23 |
| evrardjp | AJaeger: well we do have a that in other places. | 19:23 |
| evrardjp | so I didn't think it was bad. | 19:24 |
| evrardjp | It's for all branches too I guess? | 19:24 |
| dmsimard | yeah. | 19:24 |
| evrardjp | (for this one only master is important, but it could be important for other patches I have) | 19:24 |
| evrardjp | ok | 19:24 |
| evrardjp | gotcha. | 19:25 |
| fungi | any given job can only be defined in one repository | 19:25 |
| fungi | it can be inherited by or varied by configuration in other repositories | 19:25 |
| dmsimard | jobs that are defined for the purpose of inheritance ("base jobs" or templates, if you will) and jobs that are defined in the layouts are treated differently, I believe | 19:25 |
| evrardjp | fungi: I thought overrides could be defined. | 19:25 |
| *** rossella_s has joined #openstack-infra | 19:25 | |
| evrardjp | anyway -- I know better now | 19:25 |
| fungi | yes, job variants can override things, but including an entire job definition is only possible if the job has a different name in the new repo | 19:26 |
| dmsimard | evrardjp: it's basically a protection so another project can't "hijack" your job | 19:26 |
| evrardjp | fungi: dmsimard thanks for clarifications. In this case, it would probably be okay to remove the jobs.yaml, as project.yaml would use the other repo definition | 19:27 |
| evrardjp | but I am fine with a full removal and smart addition later. | 19:27 |
| *** slaweq has joined #openstack-infra | 19:30 | |
| *** felipemonteiro has joined #openstack-infra | 19:33 | |
| evrardjp | sorry for the incovenience there. | 19:33 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add individual core group for openstack-ansible-os_neutron https://review.openstack.org/546577 | 19:34 |
| fungi | evrardjp: no apology needed, we need to start paying attention to that when we review project imports | 19:35 |
| evrardjp | yeah I guess there are always lessons learned :) | 19:36 |
| evrardjp | learnt* | 19:36 |
| fungi | it's our responsibility to try and avoid approving changes which will wedge our services | 19:36 |
| evrardjp | we can maybe add a rule to not accept a project import if .?zuul.* | 19:38 |
| *** salv-orlando has quit IRC | 19:38 | |
| *** salv-orlando has joined #openstack-infra | 19:39 | |
| evrardjp | because human enforcement there seems hard. | 19:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Support ubuntu-bionic (18.04) with nodepool dsvm https://review.openstack.org/544574 | 19:39 |
| *** lbragstad has quit IRC | 19:40 | |
| *** efried has joined #openstack-infra | 19:41 | |
| fungi | i _think_ we already have a check which rejects any change that would attempt to import a repo lacking a master branch, so may be able to extend that to do it | 19:41 |
| corvus | i think i saw AJaeger heading off in that directon :) | 19:42 |
| fungi | oh! indeed, i see that in scrollback now | 19:42 |
| *** armaan has quit IRC | 19:43 | |
| *** salv-orlando has quit IRC | 19:43 | |
| *** esberglu has joined #openstack-infra | 19:43 | |
| *** esberglu has quit IRC | 19:44 | |
| *** esberglu has joined #openstack-infra | 19:44 | |
| *** dbecker has joined #openstack-infra | 19:48 | |
| *** dhajare_ has quit IRC | 19:48 | |
| *** e0ne has joined #openstack-infra | 19:51 | |
| openstackgerrit | Emmet Hikory proposed openstack-infra/system-config master: Add ARM64 Ubuntu mirror https://review.openstack.org/546466 | 19:52 |
| *** felipemonteiro has quit IRC | 19:54 | |
| *** felipemonteiro has joined #openstack-infra | 19:58 | |
| evrardjp | AJaeger: if you're doing a manual thing to make the few patches pass, you might want to add this one: https://review.openstack.org/#/c/546758/1 | 20:01 |
| *** lbragstad has joined #openstack-infra | 20:01 | |
| *** tosky has joined #openstack-infra | 20:02 | |
| *** yamamoto has joined #openstack-infra | 20:02 | |
| *** olaph has joined #openstack-infra | 20:04 | |
| *** olaph1 has quit IRC | 20:05 | |
| andreaf | pabelanger thanks good advice I'll try that now | 20:06 |
| *** felipemonteiro has quit IRC | 20:06 | |
| andreaf | pabelanger I didn't need creating the folders on the controller anyways :S | 20:06 |
| *** yamamoto has quit IRC | 20:07 | |
| *** amoralej is now known as amoralej|off | 20:10 | |
| dirk | pabelanger: I'll look at rebasing in a bit today. Would be awesome to have an experimental job running for ptg indeed :-) | 20:11 |
| *** psachin has joined #openstack-infra | 20:11 | |
| pabelanger | dirk: yah, I think we can bring it online for then | 20:11 |
| clarkb | sorry I've been mostly afk today. Had doctor appointment and now lunch then I'm pulling out an old dishwasher so that new one can go in tomorrow. The joys of pre travel errand running | 20:13 |
| AJaeger | I've been working on not getting thos in again - sorry, no time for cleanup the mess ;( | 20:15 |
| *** eernst has quit IRC | 20:15 | |
| AJaeger | http://paste.openstack.org/show/680430/ shows the problems | 20:15 |
| *** eernst_ has joined #openstack-infra | 20:15 | |
| AJaeger | patch coming for this check | 20:15 |
| ianw | infra-root: if i could get some eyes on https://review.openstack.org/#/c/546465/ and https://review.openstack.org/#/c/546302/ that will help puppet launch non-ipv6 servers and the linaro builder | 20:16 |
| *** felipemonteiro has joined #openstack-infra | 20:16 | |
| pabelanger | AJaeger: what is that the output from? | 20:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Avoid import of zuul files https://review.openstack.org/546764 | 20:17 |
| AJaeger | pabelanger: from this change ^ | 20:18 |
| pabelanger | AJaeger: Ah, nice. Good idea | 20:18 |
| AJaeger | pabelanger: just updated project-config-gerrit check so that this will not happen again... | 20:18 |
| *** armaan has joined #openstack-infra | 20:18 | |
| *** pblaho has quit IRC | 20:18 | |
| pabelanger | AJaeger: I missed the excitement, which one broke? | 20:18 |
| AJaeger | pabelanger: see https://review.openstack.org/546746 - the new repo has a broken zuul config. | 20:19 |
| *** eharney has quit IRC | 20:19 | |
| evrardjp | dmsimard: I can remove the files if need be. | 20:19 |
| evrardjp | pabelanger: it's us! | 20:19 |
| evrardjp | woot, party! | 20:19 |
| AJaeger | pabelanger: an infra-root needs to force-merge 546746 | 20:19 |
| AJaeger | and check whether the other files are also broken | 20:20 |
| AJaeger | and remove them as well... | 20:20 |
| evrardjp | AJaeger: and https://review.openstack.org/#/c/546758/ | 20:20 |
| AJaeger | infra-root ^ | 20:20 |
| pabelanger | AJaeger: evrardjp: what if you just did a noop, wouldn't that let zuul run the jobs? | 20:21 |
| pabelanger | or is it blocking something else | 20:21 |
| AJaeger | zuul refuses to load the file and does not recognize the repo as valid | 20:22 |
| *** Goneri has joined #openstack-infra | 20:22 | |
| evrardjp | I think things are jammed now, so removing all the jobs (like done in https://review.openstack.org/546746 ) doesn't help | 20:22 |
| evrardjp | a force merge is needed | 20:22 |
| evrardjp | well that's my understanding at least. | 20:22 |
| andreaf | corvus: this should make you happy https://review.openstack.org/54676 | 20:23 |
| andreaf | corvus for multinode, once it lands, the list will have to live in host_vars as opposed to vars | 20:23 |
| pabelanger | corvus: do you have a moment to confirm, see ^ between AJaeger and evrardjp | 20:23 |
| *** hashar has quit IRC | 20:23 | |
| evrardjp | but hey, https://slack-imgs.com/?c=1&url=http%3A%2F%2Friffsy.com%2Fimage%2Fvemp_tinygif.gif | 20:23 |
| AJaeger | pabelanger: can't help further - have to step out now. | 20:24 |
| pabelanger | AJaeger: okay, I can help get evrardjp back online | 20:24 |
| pabelanger | just want to confirm with corvus before force merging zuul related things | 20:25 |
| evrardjp | well to my understanding it's blocking others, so that's why I am active :) Else it can wait! :D | 20:25 |
| pabelanger | evrardjp: do you know what it is blocking? I'm just getting up to speed myself | 20:26 |
| Shrews | c 2 | 20:27 |
| Shrews | oops | 20:27 |
| mnaser | AJaeger: so the only patch i ever tried to review for new project my first time breaks the world? darn | 20:27 |
| mnaser | evrardjp: https://review.openstack.org/#/c/546687/ is okay for you for me to +A ? | 20:28 |
| evrardjp | I think it's a little above for the real reason -- but the gist is that we imported those repos that have bad neighbours: zuul.d files that contains jobs definitions. Duplicate job definitions | 20:28 |
| evrardjp | mnaser: yup | 20:28 |
| *** lbragstad has quit IRC | 20:29 | |
| mnaser | evrardjp: donezo :> | 20:29 |
| evrardjp | woot! | 20:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: DNM: Testing zuul import detection https://review.openstack.org/546767 | 20:30 |
| *** lbragstad has joined #openstack-infra | 20:30 | |
| fungi | infra-root: i'll take bypassing ci to submit 546746 in gerrit now | 20:30 |
| AJaeger | mnaser: https://review.openstack.org/#/c/546075/ is the one - you didn't touch it. | 20:30 |
| mnaser | AJaeger: oh i thought the issue was the os_panko one, my bad | 20:31 |
| mnaser | so much imports :> | 20:31 |
| AJaeger | mnaser: we noticed on os_panko which merged later | 20:31 |
| AJaeger | bbl | 20:32 |
| *** lbragstad has quit IRC | 20:32 | |
| *** psachin has quit IRC | 20:33 | |
| *** lbragstad has joined #openstack-infra | 20:33 | |
| fungi | #status log bypassed ci testing for https://review.openstack.org/546746 to resolve a deadlock with imported zuul configuration in a new project | 20:33 |
| openstackstatus | fungi: finished logging | 20:33 |
| *** hongbin has quit IRC | 20:33 | |
| pabelanger | AJaeger: left comment on 546764 | 20:34 |
| *** myoung is now known as myoung|brb | 20:35 | |
| mnaser | infra-root: if anyone has a few cycle, https://review.openstack.org/#/admin/groups/1868,members could you add "openstack-ansible-core" group into it? | 20:36 |
| pabelanger | mnaser: can you link the review that created it | 20:36 |
| mnaser | group was just created in https://review.openstack.org/#/c/546299/ | 20:37 |
| evrardjp | yeah that would be nice :) | 20:38 |
| pabelanger | mnaser: done | 20:38 |
| evrardjp | thanks pabelanger | 20:38 |
| mnaser | pabelanger: thank you! | 20:38 |
| *** psachin has joined #openstack-infra | 20:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix filelist for project-config-gerrit https://review.openstack.org/546768 | 20:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Regex gerritbot notifications: #openstack-ansible https://review.openstack.org/546687 | 20:39 |
| AJaeger | pabelanger: ^ | 20:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Regex gerritbot notifications: #openstack-nova https://review.openstack.org/546689 | 20:39 |
| AJaeger | pabelanger: no reason to -1 546764 for a change to another repo IMHO | 20:39 |
| *** salv-orlando has joined #openstack-infra | 20:39 | |
| AJaeger | pabelanger: and https://review.openstack.org/#/c/546767 tests 546764 | 20:40 |
| pabelanger | AJaeger: oh, its not defined in project-config? yah, in that case we should fix the other repo | 20:41 |
| evrardjp | pabelanger: could you do it again with another group? | 20:41 |
| evrardjp | :D | 20:41 |
| evrardjp | https://review.openstack.org/#/admin/groups/1869,members | 20:41 |
| evrardjp | https://review.openstack.org/#/c/546571/ | 20:42 |
| pabelanger | evrardjp: done | 20:43 |
| mnaser | pabelanger: sorry for being out of the loop but did the mirror issue end up getting resolved? | 20:44 |
| mnaser | i don't see an update here https://wiki.openstack.org/wiki/Infrastructure_Status and i see a recent failure | 20:44 |
| AJaeger | mnaser: could you +2A 546764, please? | 20:44 |
| *** salv-orlando has quit IRC | 20:44 | |
| pabelanger | mnaser: still rebuilding indices, it looks like we are rebuilding the databases for reprepro | 20:45 |
| *** eernst_ has quit IRC | 20:45 | |
| evrardjp | also when we fixed the patch https://review.openstack.org/#/c/546746/ by force merge, this one wasn't forced merged: https://review.openstack.org/#/c/546758/ | 20:45 |
| mnaser | pabelanger: ah ok, no worries, i'll avoid rechecking jobs for now | 20:45 |
| evrardjp | pabelanger: thank you once again! | 20:46 |
| AJaeger | evrardjp: http://paste.openstack.org/show/680430/ shows the potential problems - please check every branch listed there and if needed remove them | 20:46 |
| pabelanger | mnaser: yah, being AFS, we are limited by bandwidth right now | 20:46 |
| mnaser | AJaeger: looked at it, looks good, thanks | 20:47 |
| AJaeger | thanks, mnaser | 20:47 |
| evrardjp | AJaeger: well we are more strict than what we should now | 20:47 |
| AJaeger | evrardjp: I'm not saying you have to remove all of them - I'm saying: Please double check! | 20:48 |
| evrardjp | https://review.openstack.org/#/c/546758/ is already fixing things but I guess it needs a force merge | 20:48 |
| evrardjp | the rest should be our standard jobs usage, no new definitions, so all clear | 20:49 |
| ianw | pabelanger: re 485748 did we find the exact cycle? something just seems ... odd. i wonder if it's related to old lsb scripts on jessie | 20:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Avoid import of zuul files https://review.openstack.org/546764 | 20:51 |
| *** felipemonteiro has quit IRC | 20:51 | |
| *** priteau has quit IRC | 20:52 | |
| *** psachin has quit IRC | 20:52 | |
| evrardjp | maybe I'll ask pabelanger to force merge https://review.openstack.org/#/c/546758/ , he is on a row right now! "Jamais deux sans trois"! | 20:52 |
| pabelanger | ianw: we've had the cycle for some time, even on ubuntu. But you can see logs at http://logs.openstack.org/74/544574/11/experimental/nodepool-functional-py35-debian-src/c8bd21f/ for debian | 20:52 |
| mnaser | i think openstack-ansible-os_panko might have the same zuul issue right? | 20:54 |
| *** myoung|brb is now known as myoung | 20:54 | |
| pabelanger | ianw: specifically in http://logs.openstack.org/74/544574/11/experimental/nodepool-functional-py35-debian-src/c8bd21f/controller/logs/screen-nodepool-launcher.txt.gz we never see glean run | 20:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix filelist for project-config-gerrit https://review.openstack.org/546768 | 20:55 |
| *** psachin has joined #openstack-infra | 20:55 | |
| andreaf | pabelanger: http://logs.openstack.org/96/545696/25/check/devstack-multinode/654230d/ara/result/9d9766f9-7fe5-4e7e-afab-59d5f93be012/ | 20:56 |
| *** eernst has joined #openstack-infra | 20:56 | |
| andreaf | pabelanger: I think I will have to do the pull / push thing | 20:56 |
| ianw | pabelanger: ok, i'll just investigate a little to convince myself of the exact loop | 20:57 |
| *** caphrim007_ has joined #openstack-infra | 20:59 | |
| fungi | infra-root: evrardjp: i'll bypass ci for https://review.openstack.org/546758 as well | 20:59 |
| *** dave-mccowan has quit IRC | 21:00 | |
| *** caphrim007_ has quit IRC | 21:00 | |
| pabelanger | andreaf: hmm, I think that might be a bug in zuul. i don't see why we'd limit that between 2 nodes: http://logs.openstack.org/96/545696/25/check/devstack-multinode/654230d/job-output.txt.gz#_2018-02-21_20_50_37_656812 | 21:00 |
| *** caphrim007_ has joined #openstack-infra | 21:00 | |
| pabelanger | andreaf: it is possible our lock down of sync module is over reaching here | 21:00 |
| andreaf | pabelanger: yeah, but ok I will work around that for now | 21:01 |
| *** caphrim007_ has quit IRC | 21:01 | |
| fungi | #status log bypassed ci testing for https://review.openstack.org/546758 to resolve a deadlock with imported zuul configuration in a new project | 21:02 |
| openstackstatus | fungi: finished logging | 21:02 |
| *** kgiusti has left #openstack-infra | 21:02 | |
| *** caphrim007 has quit IRC | 21:02 | |
| evrardjp | thanks fungi | 21:02 |
| *** dprince has quit IRC | 21:02 | |
| *** salv-orlando has joined #openstack-infra | 21:03 | |
| pabelanger | andreaf: yah, we'd need to update http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/action/synchronize.py#n21 and support delegate_to | 21:03 |
| *** openstackgerrit has quit IRC | 21:03 | |
| *** yamamoto has joined #openstack-infra | 21:04 | |
| *** armaan has quit IRC | 21:04 | |
| *** dprince has joined #openstack-infra | 21:04 | |
| *** lpetrut has quit IRC | 21:06 | |
| andreaf | pabelanger: heh right - well this should work for now https://review.openstack.org/#/c/528074/10/roles/sync-devstack-data/tasks/main.yaml | 21:07 |
| andreaf | hopefully... | 21:07 |
| *** armaan has joined #openstack-infra | 21:08 | |
| *** yamamoto has quit IRC | 21:09 | |
| AJaeger | config-core, could you review https://review.openstack.org/545234 to refactor static publish jobs as well as https://review.openstack.org/545442 and https://review.openstack.org/545603 to mark base jobs as abstract, please? | 21:11 |
| *** felipemonteiro has joined #openstack-infra | 21:13 | |
| *** dprince has quit IRC | 21:13 | |
| dhellmann | a while back we had some issues with very very long patch series causing zuul to have some performance issues. Is that still a problem with v3? If so, what's a reasonable cut-off for a series length? | 21:13 |
| dhellmann | corvus : ^^ | 21:13 |
| * fungi is headed out for a stroll, bbiaw | 21:14 | |
| *** felipemonteiro has quit IRC | 21:17 | |
| *** openstackgerrit has joined #openstack-infra | 21:18 | |
| openstackgerrit | Merged openstack-infra/puppet-iptables master: Allow allowed_hosts to not have ipv6 interfaces https://review.openstack.org/546465 | 21:18 |
| pabelanger | dhellmann: how many is there currently to push up? | 21:18 |
| dhellmann | 35 | 21:18 |
| AJaeger | dhellmann: are those touching zuul files? | 21:19 |
| dhellmann | no | 21:19 |
| dhellmann | all in openstack/releases | 21:19 |
| openstackgerrit | Merged openstack-infra/system-config master: Add nb03 matching, firewall and monitoring https://review.openstack.org/546302 | 21:20 |
| pabelanger | if they are doing dynamic reloading (touching zuul.d), i think we are fine. But maybe start with 20? | 21:20 |
| AJaeger | infra-root, what needs to be done so that zuul recognizes the repos now that the broken zuul jobs are removed? | 21:20 |
| dhellmann | pabelanger : can I tell git-review to limit the number of patches it proposes? | 21:21 |
| pabelanger | dhellmann: good question, I am unsure. Let me check docs | 21:21 |
| dhellmann | I have 2 separate branches (a was branched and patched and then later b branched from a) so I can do it that way | 21:22 |
| dhellmann | that first branch has already been submitted successfully it just needs to be rebased | 21:22 |
| pabelanger | dhellmann: I don't see anything about limiting | 21:23 |
| dhellmann | ok, I'll go with the 2 sets from the branches | 21:24 |
| pabelanger | AJaeger: not sure, does a recheck still report failures? | 21:24 |
| AJaeger | pabelanger: yes, on https://review.openstack.org/546573 | 21:24 |
| dhellmann | thanks, pabelanger | 21:24 |
| pabelanger | AJaeger: it is possible we need to do a reload for zuul, but would be good to confirm with corvus first. It is also possible we've exposed a bug in zuul, and if so I'd like to confirm before corvus had a chance to look | 21:26 |
| AJaeger | pabelanger: sure | 21:27 |
| *** xarses has joined #openstack-infra | 21:32 | |
| *** andreww has quit IRC | 21:35 | |
| *** andreww has joined #openstack-infra | 21:36 | |
| *** xarses has quit IRC | 21:40 | |
| *** hashar has joined #openstack-infra | 21:40 | |
| *** gfidente has quit IRC | 21:44 | |
| *** e0ne has quit IRC | 21:46 | |
| *** eharney has joined #openstack-infra | 21:49 | |
| *** psachin has quit IRC | 21:50 | |
| *** jtomasek has quit IRC | 21:52 | |
| *** caphrim007 has joined #openstack-infra | 21:56 | |
| *** eernst has quit IRC | 21:57 | |
| *** eernst has joined #openstack-infra | 21:58 | |
| evrardjp | I start to sleep on my keyboard, so if there is anything else, I'll see it tomorrow morning. Sorry again for the breakages! | 22:00 |
| *** stevebaker has quit IRC | 22:00 | |
| *** caphrim007 has quit IRC | 22:00 | |
| *** dmellado has quit IRC | 22:00 | |
| *** wolverineav has quit IRC | 22:02 | |
| *** wolverin_ has joined #openstack-infra | 22:02 | |
| *** caphrim007 has joined #openstack-infra | 22:03 | |
| *** yamamoto has joined #openstack-infra | 22:05 | |
| *** kjackal_ has quit IRC | 22:06 | |
| *** wolverin_ is now known as wolverineav | 22:09 | |
| *** yamamoto has quit IRC | 22:10 | |
| *** armaan has quit IRC | 22:14 | |
| *** armaan has joined #openstack-infra | 22:14 | |
| *** kjackal has quit IRC | 22:16 | |
| *** rcernin has joined #openstack-infra | 22:32 | |
| *** salv-orlando has quit IRC | 22:34 | |
| openstackgerrit | Trevor McCasland proposed openstack-infra/subunit2sql master: WIP: add subunit2sql-meta for reading metadata with runs https://review.openstack.org/546006 | 22:34 |
| *** pcichy has joined #openstack-infra | 22:34 | |
| *** salv-orlando has joined #openstack-infra | 22:34 | |
| *** yamamoto has joined #openstack-infra | 22:35 | |
| *** aeng has joined #openstack-infra | 22:35 | |
| *** salv-orlando has quit IRC | 22:39 | |
| *** r-daneel has quit IRC | 22:40 | |
| *** lbragstad has quit IRC | 22:47 | |
| *** lbragstad has joined #openstack-infra | 22:47 | |
| *** VW_ has joined #openstack-infra | 22:50 | |
| *** VW has quit IRC | 22:53 | |
| *** salv-orlando has joined #openstack-infra | 22:53 | |
| *** HeOS has quit IRC | 22:53 | |
| *** VW_ has quit IRC | 22:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: have tag-releases skip closed series https://review.openstack.org/546801 | 22:55 |
| *** HeOS has joined #openstack-infra | 22:55 | |
| *** olaph1 has joined #openstack-infra | 22:56 | |
| *** olaph has quit IRC | 22:57 | |
| *** caphrim007_ has joined #openstack-infra | 22:59 | |
| *** hashar has quit IRC | 22:59 | |
| *** caphrim007 has quit IRC | 23:02 | |
| *** andreas_s has joined #openstack-infra | 23:04 | |
| *** olaph has joined #openstack-infra | 23:04 | |
| *** olaph1 has quit IRC | 23:06 | |
| *** yamamoto has quit IRC | 23:08 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config master: Add building of openSUSE Tumbleweed nodepool images https://review.openstack.org/504169 | 23:08 |
| *** myoung is now known as myoung|afk | 23:08 | |
| *** andreas_s has quit IRC | 23:09 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Disable vhd on aarch64 https://review.openstack.org/546435 | 23:13 |
| pabelanger | dirk: do you have an output from tools/image-build.sh for opensuse-tumbleweed?^ | 23:17 |
| dirk | pabelanger: not from a recent run, but I did it when I submitted the review originally | 23:18 |
| dirk | pabelanger: the image is building and booting fine | 23:18 |
| pabelanger | k | 23:18 |
| *** rossella_s has quit IRC | 23:18 | |
| pabelanger | ianw: do you mind a quick review on https://review.openstack.org/504169/ I have some time to monitor builds this evening | 23:19 |
| clarkb | ok I'm back to a computer now. Anything I can help with? | 23:20 |
| clarkb | I guess I can review ^ | 23:20 |
| pabelanger | clarkb: yah, if you don't mind | 23:20 |
| clarkb | pabelanger: glean release was made right? | 23:20 |
| pabelanger | yes | 23:20 |
| pabelanger | 1.10.2 | 23:21 |
| clarkb | approved | 23:21 |
| pabelanger | cool, lets see what happens | 23:21 |
| dirk | pabelanger: hmpf, so the new dib doesn't work | 23:22 |
| dirk | wait | 23:22 |
| ianw | yeah, that's impossible, dib is perfect :) | 23:22 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Switch cloudkitty-dashboard to use horizon job https://review.openstack.org/546805 | 23:23 |
| clarkb | I WIP'd the change so it won't merge | 23:23 |
| andreaf | corvus pabelanger gah! again :D http://logs.openstack.org/74/528074/11/check/devstack-multinode/3fbf003/ara/result/d4217366-2e43-4f34-bdb8-ce2422a81f43/ | 23:23 |
| ianw | tubleweed has built and booted in the glean experimental tests? | 23:23 |
| pabelanger | ianw: yah | 23:24 |
| dirk | ah, nevermind, I used it wrong | 23:25 |
| clarkb | so I can unWIP? | 23:25 |
| dirk | yes | 23:25 |
| dirk | its caching the git repos | 23:25 |
| pabelanger | andreaf: yah, I can look at our file checks tomorrow, pretty sure using delegate_to in ansible is valid, but we don't allow it right now | 23:25 |
| clarkb | ok it should regate now | 23:25 |
| dirk | thats gonna take a couple of hours | 23:25 |
| pabelanger | I | 23:26 |
| pabelanger | err | 23:26 |
| andreaf | pabelanger thanks | 23:26 |
| pabelanger | I plan on adding a DIB image build jobs for project-config when we make changes to elements, to atleast ensure we can build the images in gate | 23:26 |
| ianw | dirk: yeah, i usually strip that down , unless you need it ... see https://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh#n76 | 23:27 |
| *** Goneri has quit IRC | 23:29 | |
| smcginnis | If anyone has a chance to take a look, simple release job switch for a horizon dashboard release: https://review.openstack.org/546805 | 23:30 |
| *** olaph1 has joined #openstack-infra | 23:30 | |
| *** olaph has quit IRC | 23:30 | |
| andreaf | pabelanger one last attempt, using zuul.executor.work_rook https://review.openstack.org/#/c/528074/12/roles/sync-devstack-data/tasks/main.yaml... | 23:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Add building of openSUSE Tumbleweed nodepool images https://review.openstack.org/504169 | 23:31 |
| *** salv-orlando has quit IRC | 23:32 | |
| *** salv-orlando has joined #openstack-infra | 23:33 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:34 | |
| dirk | pabelanger: so nb01/nb02.o.o is where the log files end up these days? | 23:34 |
| pabelanger | andreaf: no, think that is going to still fail. you'll have to run sync-devstack-data on host: localhost, then delegate_to: controller for Pull the CA certificate and folder, then delegate_to compute for Push the CA certificate and folder | 23:35 |
| pabelanger | dirk: yah | 23:35 |
| pabelanger | it will be image.log until we reload builders with new logging config | 23:35 |
| clarkb | smcginnis: done | 23:36 |
| *** salv-orlando has quit IRC | 23:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-nodepool master: Increase timeout on nodepool install https://review.openstack.org/546814 | 23:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Update nodepool-builder logging for opensuse-tumbleweed https://review.openstack.org/546815 | 23:38 |
| pabelanger | ianw: clarkb: ^logging changes for nodepool-builder | 23:39 |
| ianw | pabelanger: hmm, well if you restart the buidlers, they're going to pick up corvus's new stuff and put the results in the build dir | 23:41 |
| ianw | ergo i think that's all redundant | 23:41 |
| pabelanger | ianw: have we exposed them to http yet? | 23:43 |
| ianw | pabelanger: no ... the stack related to https://review.openstack.org/#/c/543671/ needs to go in | 23:43 |
| ianw | it will want some babysitting i'd say | 23:43 |
| pabelanger | k, might be too much for tonight | 23:44 |
| *** ldnunes has quit IRC | 23:44 | |
| ianw | yeah, i'll see how i go, just pushing on nb03 atm | 23:44 |
| ianw | https://review.openstack.org/#/c/546814/ if you want a quick one, just increases timeout for nodepool install as it has to build a bunch o stuff | 23:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch cloudkitty-dashboard to use horizon job https://review.openstack.org/546805 | 23:46 |
| ianw | do the adhoc playbooks not refresh /etc/puppet/modules? | 23:46 |
| *** tpsilva has quit IRC | 23:46 | |
| clarkb | ianw: I want to say they don't | 23:47 |
| clarkb | update_puppet.yaml is the playbook that does it and it doesn't appear to be called by the ad hoc playbooks | 23:48 |
| *** mriedem has quit IRC | 23:54 | |
| *** mriedem has joined #openstack-infra | 23:56 | |
| fungi | back now and catching up | 23:59 |
| *** olaph has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
