| *** d0ugal has joined #openstack-infra | 00:01 | |
| *** rossella_s has quit IRC | 00:17 | |
| *** rossella_s has joined #openstack-infra | 00:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Fix deprecation warning of ansible-2.5 https://review.openstack.org/571996 | 00:18 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 00:19 | |
| EmilienM | do we have plans to move mirrors to gitlab? | 00:23 |
| tristanC | EmilienM: why would you do that? | 00:23 |
| fungi | the replication to github has been maintained mostly on the premise that if we didn't put an "official-looking" copy of our popular repositories on github then some unofficial repo would end up getting treated as the one all the github users fork from because so may people treat github as _the_ index for all source code. if we could find a way to not mirror to github or any other proprietary service, | 00:27 |
| fungi | i'd be thrilled | 00:27 |
| fungi | there's really no point (currently at least) in mirroring to gitlab, bitbucket or any other third-party-operated service | 00:29 |
| *** rossella_s has quit IRC | 00:31 | |
| *** rossella_s has joined #openstack-infra | 00:33 | |
| openstackgerrit | Merged openstack-infra/system-config master: Substitute site name for survey OpenID callback https://review.openstack.org/571976 | 00:36 |
| fungi | #status log survey01.openstack.org is no longer in the emergency disable list now that https://review.openstack.org/571976 has merged | 00:37 |
| openstackstatus | fungi: finished logging | 00:37 |
| ianw | that this rumor (it still seem to technically be that) comes as infra is talking about how to bring more things under the umbrella is ... interesting | 00:38 |
| *** rossella_s has quit IRC | 00:43 | |
| *** rossella_s has joined #openstack-infra | 00:44 | |
| *** vtapia has quit IRC | 00:45 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update Zuul Status Page to correct URL https://review.openstack.org/567175 | 00:46 |
| *** anteaya has joined #openstack-infra | 00:50 | |
| *** kzaitsev1pi has quit IRC | 00:51 | |
| anteaya | fungi: microsoft might have just provided you the answer to your prayers: https://www.thestar.com/business/2018/06/03/microsoft-reportedly-agrees-to-buy-coding-site-github.html | 00:51 |
| fungi | yep, that's the article which had presumably spurred on the above discussion (dmsimard liked it at 20:59z) | 00:52 |
| fungi | s/liked/linked/ | 00:52 |
| fungi | well, a similar article on bloomberg's site anyway | 00:53 |
| ianw | fungi: if you've got a second to sanity check https://review.openstack.org/#/c/571646/ it just allows mirror-update to send stats. just fleshing out getting some afs monitoring | 00:54 |
| *** rossella_s has quit IRC | 00:57 | |
| *** rossella_s has joined #openstack-infra | 00:59 | |
| *** kzaitsev_pi has joined #openstack-infra | 00:59 | |
| fungi | yeah, that seems fine | 01:00 |
| ianw | fungi: thanks ... still doing some loopback testing before i send anything off, but it seems to be gathering the partition, some basic server and volume stats ok | 01:02 |
| ianw | not the first people to monitor afs like this, but probably the first to want to do it in a python environment (rather than perl) and send things via statsd | 01:02 |
| fungi | that sounds crazy useful | 01:02 |
| fungi | and yeah, i guess there's not a net-snmp backend for it? | 01:03 |
| fungi | that's how i'd expect most people to monitor such things | 01:03 |
| ianw | i might have just extended some of the existing perl stuff, but i couldn't really figure out a virtualenv analogue | 01:03 |
| ianw | + then having to automate that with some puppet etc | 01:04 |
| *** kzaitsev_pi has quit IRC | 01:05 | |
| *** kzaitsev_pi has joined #openstack-infra | 01:06 | |
| ianw | https://github.com/openafs-contrib/afs-tools/blob/master/admin/snips/afsmon is interesting, for SNIPS which i hadn't heard of | 01:06 |
| fungi | yeah, that's a new one on me | 01:06 |
| ianw | i've tried to just pull the info into python objects, so hopefully if somone were interested in throwing the output to something other than ascii / statsd, it would be fairly trivial | 01:07 |
| *** felipemonteiro has joined #openstack-infra | 01:09 | |
| fungi | yeah, the callout interface in net-snmpd is pretty straightforward. wouldn't be too hard if we decided later we wanted to use cacti to monitor it in a pull fashion instead | 01:10 |
| dmsimard | EmilienM: mordred has been meaning to give a serious look at gitea at some point | 01:10 |
| dmsimard | EmilienM: Gitlab is venture-backed and has an open core business model, it's not immune to acquisitions. It's actually very much an acquisition target :) | 01:11 |
| dmsimard | oh the irony, gitea is hosted on github | 01:13 |
| anteaya | ah, beg pardon dmsimard | 01:14 |
| *** xinliang has quit IRC | 01:14 | |
| anteaya | obviously I hadn't got far enough through the back scroll | 01:14 |
| EmilienM | at least gitlab code is open-source (not sure about EE though) | 01:15 |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Allow mirror-update.o.o to produce stats https://review.openstack.org/571646 | 01:15 |
| fungi | the hosted version of gitlab is not open source | 01:15 |
| EmilienM | fungi made a good point of the repo discovery and the "official looking" aspect | 01:15 |
| fungi | follow debian-devel discussions for fun tidbits of features they want to use but can't because gitlab that particular gitlab enterprise feature isn't available in gitlab community | 01:16 |
| *** rossella_s has quit IRC | 01:17 | |
| dmsimard | my first interaction with the ceo of gitlab was him trying to monetize the development of the ldap feature if we subscribed to the entreprise edition on a long term, it left a rather bad taste in my mouth | 01:17 |
| anteaya | if only there were a group of people who had the ability to host open source projects, open to working with people interesting in supporting their efforts | 01:17 |
| EmilienM | how is bitbucket ? | 01:18 |
| dmsimard | They highlighted in their new version of the open source project and congratulated themselves on open sourcing a feature that was previously only available in entreprise edition : | 01:18 |
| fungi | anyway, we have plenty of mirroring to systems we control. the github mirroring is simply because people expect to find something on github, not because we need it. i'd rather we found some way to just stop actively mirroring to third parties at all | 01:19 |
| *** rossella_s has joined #openstack-infra | 01:20 | |
| dmsimard | fungi: I would like that as well but only if we have a viable alternative for users. I don't mind cgit/gitweb but in terms of UX there's better out there. Is it time to draft an official spec ? | 01:21 |
| fungi | sure, i'll agree cgit is not great for viewing source code in a web browser | 01:22 |
| fungi | though it's better than gitweb, which is what we'd been using previously | 01:23 |
| dmsimard | ++ | 01:23 |
| dmsimard | Fedora has been using Pagure to varying amounts of success https://pagure.io/ | 01:23 |
| fungi | yeah, i've seen it | 01:23 |
| fungi | unfortunately most of the fancier git browsing interfaces out there want to also be patch submission and code review and everything else too | 01:24 |
| dmsimard | Yeah, they want integration and an ecosystem. I'll spin up a gitea instance to check it out. | 01:25 |
| dmsimard | oh hey it supports sqlite \o/ | 01:27 |
| *** dpawlik has joined #openstack-infra | 01:27 | |
| *** hongbin has joined #openstack-infra | 01:29 | |
| *** xinliang has joined #openstack-infra | 01:30 | |
| *** xinliang has quit IRC | 01:30 | |
| *** xinliang has joined #openstack-infra | 01:30 | |
| *** weshay has quit IRC | 01:30 | |
| *** dpawlik has quit IRC | 01:32 | |
| *** namnh has joined #openstack-infra | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:35 | |
| *** weshay has joined #openstack-infra | 01:36 | |
| *** anteaya has quit IRC | 01:37 | |
| *** gongysh has joined #openstack-infra | 01:38 | |
| *** kiennt26 has joined #openstack-infra | 01:39 | |
| *** germs has quit IRC | 01:40 | |
| *** germs has joined #openstack-infra | 01:41 | |
| *** germs has quit IRC | 01:41 | |
| *** germs has joined #openstack-infra | 01:41 | |
| *** gagehugo has quit IRC | 01:44 | |
| dmsimard | wow, it's actually pretty good | 01:44 |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| dmsimard | it also supports transparently mirroring repositories out of the box | 01:46 |
| dmsimard | I'm sold: https://i.imgur.com/BlXfpek.png | 01:49 |
| bkero | fungi: I lamented about the same with clarkb a few days ago | 01:51 |
| bkero | https://github.com/jonashaag/klaus was what I found when I complained | 01:52 |
| *** kiennt26 has quit IRC | 01:54 | |
| *** hamzy has quit IRC | 01:56 | |
| *** hamzy has joined #openstack-infra | 01:56 | |
| *** pahuang has quit IRC | 02:03 | |
| *** yamamoto has joined #openstack-infra | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:13 | |
| *** gagehugo has joined #openstack-infra | 02:17 | |
| *** pcrews has quit IRC | 02:18 | |
| *** pahuang has joined #openstack-infra | 02:20 | |
| *** pcrews has joined #openstack-infra | 02:32 | |
| *** namnh has quit IRC | 02:33 | |
| clarkb | fwiw I'm not sure we have to change anything? github has always been non open/free and is only a mirror for us | 02:34 |
| clarkb | we've long said we'd prefer to not use github | 02:34 |
| *** namnh has joined #openstack-infra | 02:34 | |
| *** zhangfei has joined #openstack-infra | 02:36 | |
| *** gcb has joined #openstack-infra | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** annp has joined #openstack-infra | 02:47 | |
| *** zhongjun_ has joined #openstack-infra | 02:49 | |
| *** pahuang has quit IRC | 03:06 | |
| *** lifeless_ has joined #openstack-infra | 03:08 | |
| *** lifeless has quit IRC | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:10 | |
| *** slaweq has joined #openstack-infra | 03:14 | |
| *** pahuang has joined #openstack-infra | 03:18 | |
| *** slaweq has quit IRC | 03:19 | |
| *** EmilienM has quit IRC | 03:22 | |
| *** gyankum has joined #openstack-infra | 03:22 | |
| *** EmilienM has joined #openstack-infra | 03:23 | |
| *** gyankum has quit IRC | 03:24 | |
| *** lifeless has joined #openstack-infra | 03:29 | |
| *** lifeless_ has quit IRC | 03:29 | |
| *** eernst has joined #openstack-infra | 03:35 | |
| *** ramishra has joined #openstack-infra | 03:37 | |
| *** jamesmcarthur has joined #openstack-infra | 03:38 | |
| *** ramishra has quit IRC | 03:42 | |
| *** ramishra has joined #openstack-infra | 03:43 | |
| *** udesale has joined #openstack-infra | 03:54 | |
| *** vivsoni has joined #openstack-infra | 04:01 | |
| *** kzaitsev1pi has joined #openstack-infra | 04:07 | |
| *** kzaitsev_pi has quit IRC | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** jogo has quit IRC | 04:09 | |
| *** ramishra has quit IRC | 04:13 | |
| *** ramishra has joined #openstack-infra | 04:13 | |
| *** kiennt26 has joined #openstack-infra | 04:23 | |
| *** jogo has joined #openstack-infra | 04:24 | |
| *** jogo has joined #openstack-infra | 04:24 | |
| scas | there's a reason i've not seriously considered migrating openstack-chef off of openstack infra, even with so few involved. i like github being a mirror, not the lynchpin | 04:29 |
| *** gongysh has quit IRC | 04:31 | |
| *** germs has quit IRC | 04:34 | |
| *** dhajare has joined #openstack-infra | 04:38 | |
| *** stakeda has joined #openstack-infra | 04:38 | |
| *** psachin has joined #openstack-infra | 04:41 | |
| *** felipemonteiro has quit IRC | 04:43 | |
| *** hongbin has quit IRC | 04:44 | |
| *** vtapia has joined #openstack-infra | 04:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move planet unittests in-tree https://review.openstack.org/572016 | 05:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove openstack-planet-unittest legacy job https://review.openstack.org/572017 | 05:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Add a zuul test to run on github3.py changes https://review.openstack.org/558323 | 05:03 |
| *** dpawlik has joined #openstack-infra | 05:05 | |
| *** dpawlik has quit IRC | 05:05 | |
| *** rossella_s has quit IRC | 05:05 | |
| *** rossella_s has joined #openstack-infra | 05:07 | |
| *** pgadiya has joined #openstack-infra | 05:07 | |
| *** pgadiya has quit IRC | 05:07 | |
| *** eernst has quit IRC | 05:08 | |
| *** jamesmcarthur has joined #openstack-infra | 05:09 | |
| *** dhajare has quit IRC | 05:10 | |
| *** hamzy has quit IRC | 05:12 | |
| *** jamesmcarthur has quit IRC | 05:14 | |
| *** udesale_ has joined #openstack-infra | 05:35 | |
| *** dpawlik has joined #openstack-infra | 05:37 | |
| *** dpawlik has quit IRC | 05:37 | |
| *** dpawlik has joined #openstack-infra | 05:37 | |
| *** udesale has quit IRC | 05:37 | |
| *** links has joined #openstack-infra | 05:38 | |
| *** cshastri has joined #openstack-infra | 05:41 | |
| *** dpawlik has quit IRC | 05:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/grafyaml master: Test case for multiple targets with hidden https://review.openstack.org/572020 | 05:53 |
| openstackgerrit | Quique Llorente proposed openstack-infra/grafyaml master: Add 'database' to the datasource schema https://review.openstack.org/570172 | 05:56 |
| *** rossella_s has quit IRC | 05:56 | |
| *** rossella_s has joined #openstack-infra | 05:57 | |
| *** iranzo has joined #openstack-infra | 05:58 | |
| *** rossella_s has quit IRC | 06:03 | |
| *** rossella_s has joined #openstack-infra | 06:03 | |
| *** udesale__ has joined #openstack-infra | 06:16 | |
| *** rossella_s has quit IRC | 06:17 | |
| *** rossella_s has joined #openstack-infra | 06:18 | |
| *** udesale_ has quit IRC | 06:18 | |
| *** dhajare has joined #openstack-infra | 06:25 | |
| *** pgadiya has joined #openstack-infra | 06:25 | |
| *** pgadiya has quit IRC | 06:25 | |
| *** rossella_s has quit IRC | 06:27 | |
| *** aojea has joined #openstack-infra | 06:27 | |
| *** rossella_s has joined #openstack-infra | 06:29 | |
| *** armaan has joined #openstack-infra | 06:30 | |
| *** e0ne has joined #openstack-infra | 06:34 | |
| *** ccamacho has joined #openstack-infra | 06:40 | |
| *** wznoinsk has quit IRC | 06:43 | |
| *** ssbarnea has quit IRC | 06:43 | |
| *** ssbarnea has joined #openstack-infra | 06:44 | |
| *** bdodd has joined #openstack-infra | 06:47 | |
| *** wznoinsk has joined #openstack-infra | 06:48 | |
| *** bdodd_ has quit IRC | 06:50 | |
| *** slaweq has joined #openstack-infra | 06:54 | |
| *** slaweq has quit IRC | 06:55 | |
| *** slaweq has joined #openstack-infra | 06:55 | |
| *** rossella_s has quit IRC | 06:57 | |
| *** pgadiya has joined #openstack-infra | 06:58 | |
| *** pgadiya has quit IRC | 06:58 | |
| *** florianf has joined #openstack-infra | 06:59 | |
| *** rossella_s has joined #openstack-infra | 06:59 | |
| *** AJaeger has joined #openstack-infra | 07:00 | |
| *** zoli|gone is now known as zoli | 07:01 | |
| *** zoli is now known as zoli|wfh | 07:01 | |
| *** zoli|wfh is now known as zoli | 07:01 | |
| *** armaan has quit IRC | 07:01 | |
| *** armaan has joined #openstack-infra | 07:02 | |
| *** jbadiapa has joined #openstack-infra | 07:06 | |
| *** armaan has quit IRC | 07:07 | |
| *** andreas_s has joined #openstack-infra | 07:08 | |
| *** sshnaidm has joined #openstack-infra | 07:10 | |
| *** rossella_s has quit IRC | 07:11 | |
| *** pcaruana has joined #openstack-infra | 07:12 | |
| *** rossella_s has joined #openstack-infra | 07:13 | |
| *** pcaruana is now known as pcaruana|worksho | 07:14 | |
| *** jtomasek has joined #openstack-infra | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:16 | |
| *** janki has joined #openstack-infra | 07:17 | |
| *** rossella_s has quit IRC | 07:18 | |
| *** shardy has quit IRC | 07:19 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: nodepool: Set min-ready to '1' for opensuse-150 https://review.openstack.org/572030 | 07:19 |
| *** shardy has joined #openstack-infra | 07:19 | |
| *** yolanda_ has joined #openstack-infra | 07:20 | |
| *** rossella_s has joined #openstack-infra | 07:21 | |
| *** armaan has joined #openstack-infra | 07:21 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: Add support for specifying security_group in nodepool https://review.openstack.org/570857 | 07:23 |
| *** yolanda has quit IRC | 07:23 | |
| *** yamamoto has quit IRC | 07:24 | |
| *** tosky has joined #openstack-infra | 07:29 | |
| *** rossella_s has quit IRC | 07:30 | |
| *** e0ne has quit IRC | 07:30 | |
| *** e0ne_ has joined #openstack-infra | 07:30 | |
| *** rossella_s has joined #openstack-infra | 07:31 | |
| *** amoralej|off is now known as amoralej | 07:31 | |
| *** kiennt26 has quit IRC | 07:34 | |
| *** sshnaidm is now known as sshnaidm|brq | 07:38 | |
| *** jistr is now known as jistr|mtgs | 07:38 | |
| *** roman_g has joined #openstack-infra | 07:44 | |
| *** rossella_s has quit IRC | 07:44 | |
| *** rossella_s has joined #openstack-infra | 07:45 | |
| *** tesseract-RH has joined #openstack-infra | 07:45 | |
| *** tesseract has quit IRC | 07:48 | |
| *** alexchadin has joined #openstack-infra | 07:48 | |
| *** tesseract-RH has quit IRC | 07:48 | |
| *** tesseract has joined #openstack-infra | 07:49 | |
| *** roman_g has quit IRC | 07:49 | |
| *** roman_g has joined #openstack-infra | 07:50 | |
| *** jpena|off is now known as jpena | 07:53 | |
| hwoarang | ianw:clarkb opensuse-150 now builds fine. thanks a lot for your help! | 07:54 |
| hwoarang | clarkb: i have now submitted https://review.openstack.org/#/c/572030/ to add it to nodepool | 07:55 |
| hwoarang | clarkb: i will read the ntp + suse backlog in a bit | 07:55 |
| *** hashar has joined #openstack-infra | 07:56 | |
| *** rossella_s has quit IRC | 07:59 | |
| *** rossella_s has joined #openstack-infra | 08:01 | |
| *** jaosorior has joined #openstack-infra | 08:05 | |
| *** vickyc has joined #openstack-infra | 08:13 | |
| *** rcernin has quit IRC | 08:13 | |
| *** rpittau has joined #openstack-infra | 08:13 | |
| *** gongysh has joined #openstack-infra | 08:16 | |
| *** yamamoto has joined #openstack-infra | 08:25 | |
| *** yamamoto has quit IRC | 08:33 | |
| *** gfidente has joined #openstack-infra | 08:34 | |
| *** dingyichen has quit IRC | 08:36 | |
| *** rossella_s has quit IRC | 08:56 | |
| *** rossella_s has joined #openstack-infra | 08:57 | |
| *** rossella_s has quit IRC | 09:02 | |
| *** rossella_s has joined #openstack-infra | 09:07 | |
| *** jamesmcarthur has joined #openstack-infra | 09:10 | |
| *** armaan has quit IRC | 09:19 | |
| *** pblaho has joined #openstack-infra | 09:19 | |
| *** armaan has joined #openstack-infra | 09:20 | |
| *** yamamoto has joined #openstack-infra | 09:27 | |
| *** armaan has quit IRC | 09:30 | |
| *** armaan has joined #openstack-infra | 09:30 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Remove direct tagging/branching for yaql https://review.openstack.org/572065 | 09:39 |
| *** pbourke has quit IRC | 09:41 | |
| *** pbourke has joined #openstack-infra | 09:41 | |
| *** vabada has joined #openstack-infra | 09:54 | |
| *** vabada has quit IRC | 09:54 | |
| ianw | hwoarang: is that to bring up a suse-based builder? | 09:54 |
| *** cshastri has quit IRC | 09:59 | |
| hwoarang | ianw: i believe that's what's needed for project tests to start using opensuse-150 | 09:59 |
| hwoarang | i am not sure how to bring up a static suse builder for wheels yet | 09:59 |
| ianw | hwoarang: hmm, i may be missing something, but i'm not seeing the connection between puppet-ntp -> system-config -> bringing up control plane hosts (i.e. nodepool-builder) and project tests | 10:04 |
| ianw | i think we also don't need a static node for wheel builds? pabelanger made it all zuul jobs iirc | 10:04 |
| ianw | may also be wrong | 10:04 |
| *** namnh has quit IRC | 10:05 | |
| hwoarang | ianw: it's irrelevant i think. two different problems :) the common thing between them is just the opensuse-150 dib images | 10:07 |
| hwoarang | if we dont need special config for wheels that's even better | 10:08 |
| *** cshastri has joined #openstack-infra | 10:13 | |
| *** gongysh has quit IRC | 10:13 | |
| *** gongysh has joined #openstack-infra | 10:14 | |
| *** pleia2_ has joined #openstack-infra | 10:15 | |
| *** gongysh has quit IRC | 10:15 | |
| *** bramwelt1 has joined #openstack-infra | 10:15 | |
| *** darvon_ has joined #openstack-infra | 10:16 | |
| *** frickler_ has joined #openstack-infra | 10:16 | |
| *** jamesmcarthur has quit IRC | 10:17 | |
| *** alex_xu_ has joined #openstack-infra | 10:17 | |
| *** eglute_s has joined #openstack-infra | 10:19 | |
| *** mpeterson_ has joined #openstack-infra | 10:19 | |
| *** sambetts_ has joined #openstack-infra | 10:19 | |
| *** ddurst- has joined #openstack-infra | 10:19 | |
| *** zzzeek_ has joined #openstack-infra | 10:19 | |
| *** kazsh_ has joined #openstack-infra | 10:19 | |
| *** gouthamr_ has joined #openstack-infra | 10:20 | |
| *** sai- has joined #openstack-infra | 10:20 | |
| *** sambetts|afk has quit IRC | 10:20 | |
| *** zzzeek has quit IRC | 10:20 | |
| *** gouthamr has quit IRC | 10:20 | |
| *** sai_ has quit IRC | 10:20 | |
| *** pleia2 has quit IRC | 10:20 | |
| *** mpeterson has quit IRC | 10:20 | |
| *** eglute has quit IRC | 10:20 | |
| *** darvon has quit IRC | 10:20 | |
| *** alex_xu has quit IRC | 10:20 | |
| *** 7GHAARR89 has quit IRC | 10:20 | |
| *** bramwelt has quit IRC | 10:20 | |
| *** frickler has quit IRC | 10:20 | |
| *** ddurst has quit IRC | 10:20 | |
| *** kazsh has quit IRC | 10:20 | |
| *** betherly_ has joined #openstack-infra | 10:20 | |
| *** udesale__ has quit IRC | 10:23 | |
| *** armaan has quit IRC | 10:24 | |
| *** armaan has joined #openstack-infra | 10:24 | |
| *** jamesmcarthur has joined #openstack-infra | 10:25 | |
| *** eyalb has joined #openstack-infra | 10:25 | |
| *** yamamoto has quit IRC | 10:26 | |
| eyalb | Hi I cannot see the zuul check in the gerrit review in the browser console I see this error: hideci.js:1 Failed to load resource: the server responded with a status of 404 (Not Found) | 10:26 |
| *** hamzy has joined #openstack-infra | 10:27 | |
| eyalb | for this file /static/hideci.js?e=9096ec95ae8a6287d076931f52f98b20:1 | 10:27 |
| *** alexchadin has quit IRC | 10:28 | |
| *** jamesmcarthur has quit IRC | 10:29 | |
| *** darvon_ is now known as darvon | 10:30 | |
| *** psachin has quit IRC | 10:32 | |
| *** zoli is now known as zoli|lunch | 10:36 | |
| *** tesseract has quit IRC | 10:41 | |
| *** alexchadin has joined #openstack-infra | 10:41 | |
| *** dave-mccowan has joined #openstack-infra | 10:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/grafyaml master: Update documentation https://review.openstack.org/572076 | 10:43 |
| *** tesseract has joined #openstack-infra | 10:44 | |
| *** jamesmcarthur has joined #openstack-infra | 10:45 | |
| *** dave-mcc_ has joined #openstack-infra | 10:45 | |
| *** dave-mccowan has quit IRC | 10:47 | |
| *** jamesmcarthur has quit IRC | 10:49 | |
| ianw | eyalb ... hmm, we merged a change to that | 10:51 |
| ianw | eyalb: try with a hard refresh now? i touched the header and hoepfully gerrit picks it up | 10:53 |
| eyalb | yes works now thanks | 10:54 |
| *** alexchadin has quit IRC | 11:02 | |
| *** alexchadin has joined #openstack-infra | 11:03 | |
| *** alexchadin has quit IRC | 11:03 | |
| *** alexchadin has joined #openstack-infra | 11:03 | |
| *** alexchadin has quit IRC | 11:04 | |
| *** alexchadin has joined #openstack-infra | 11:04 | |
| *** kashyap has left #openstack-infra | 11:04 | |
| *** alexchadin has quit IRC | 11:05 | |
| *** alexchadin has joined #openstack-infra | 11:05 | |
| *** alexchadin has quit IRC | 11:05 | |
| *** jpena is now known as jpena|lunch | 11:09 | |
| *** rossella_s has quit IRC | 11:09 | |
| *** psachin has joined #openstack-infra | 11:10 | |
| *** rossella_s has joined #openstack-infra | 11:11 | |
| *** bramwelt1 is now known as bramwelt | 11:12 | |
| *** bramwelt has quit IRC | 11:12 | |
| *** bramwelt has joined #openstack-infra | 11:12 | |
| andreaf | corvus thanks! | 11:17 |
| *** eyalb has left #openstack-infra | 11:17 | |
| *** jamesmcarthur has joined #openstack-infra | 11:19 | |
| *** jrollinhatin is now known as jroll | 11:20 | |
| *** rossella_s has quit IRC | 11:21 | |
| *** lifeless_ has joined #openstack-infra | 11:21 | |
| *** rossella_s has joined #openstack-infra | 11:21 | |
| *** lifeless has quit IRC | 11:22 | |
| *** zhangfei has quit IRC | 11:24 | |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** amoralej is now known as amoralej|lunch | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** dpawlik has joined #openstack-infra | 11:33 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** dpawlik has quit IRC | 11:38 | |
| *** slaweq has quit IRC | 11:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Initial grafana AFS dashboard https://review.openstack.org/572083 | 11:39 |
| *** slaweq has joined #openstack-infra | 11:39 | |
| *** jbadiapa has quit IRC | 11:39 | |
| *** alexchadin has joined #openstack-infra | 11:39 | |
| *** Bhujay has joined #openstack-infra | 11:40 | |
| *** stakeda has quit IRC | 11:40 | |
| *** alexchadin has quit IRC | 11:45 | |
| *** verdurin has joined #openstack-infra | 11:46 | |
| *** zoli|lunch is now known as zoli|wfh | 11:49 | |
| *** rfolco has joined #openstack-infra | 11:55 | |
| *** udesale has joined #openstack-infra | 11:57 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove trusty beaker job template https://review.openstack.org/571025 | 11:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Create ansible-role-openstacksdk role https://review.openstack.org/568915 | 12:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Run openstacksdk ansible jobs on ansible devel https://review.openstack.org/570681 | 12:04 |
| *** armaan has quit IRC | 12:06 | |
| *** armaan has joined #openstack-infra | 12:06 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Make openstackci-beaker xenial jobs voting https://review.openstack.org/571965 | 12:07 |
| *** jbadiapa has joined #openstack-infra | 12:07 | |
| *** wolverineav has joined #openstack-infra | 12:07 | |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** rh-jelabarre has joined #openstack-infra | 12:10 | |
| *** edmondsw has joined #openstack-infra | 12:12 | |
| *** rossella_s has quit IRC | 12:13 | |
| *** alexchadin has joined #openstack-infra | 12:14 | |
| *** rossella_s has joined #openstack-infra | 12:15 | |
| *** psachin has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** amoralej|lunch is now known as amoralej | 12:19 | |
| *** tpsilva has joined #openstack-infra | 12:20 | |
| *** wolverineav has quit IRC | 12:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 12:22 |
| *** apetrich has quit IRC | 12:23 | |
| *** apetrich has joined #openstack-infra | 12:24 | |
| *** armaan has quit IRC | 12:26 | |
| *** armaan has joined #openstack-infra | 12:27 | |
| *** rossella_s has quit IRC | 12:28 | |
| *** rossella_s has joined #openstack-infra | 12:29 | |
| *** jroll has quit IRC | 12:29 | |
| *** jroll has joined #openstack-infra | 12:30 | |
| *** armaan has quit IRC | 12:31 | |
| *** rosmaita has joined #openstack-infra | 12:31 | |
| *** armaan has joined #openstack-infra | 12:32 | |
| *** dpawlik has joined #openstack-infra | 12:33 | |
| *** openstackgerrit has quit IRC | 12:34 | |
| *** rlandy has joined #openstack-infra | 12:34 | |
| *** rlandy is now known as rlandy|rover | 12:35 | |
| *** frickler_ is now known as frickler | 12:39 | |
| *** dbecker has joined #openstack-infra | 12:40 | |
| *** jamesdenton has joined #openstack-infra | 12:41 | |
| *** openstackgerrit has joined #openstack-infra | 12:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 12:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 12:44 |
| *** hrw has left #openstack-infra | 12:53 | |
| *** cshastri has quit IRC | 12:53 | |
| *** jcoufal has joined #openstack-infra | 13:05 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 13:13 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 13:13 |
| *** caphrim007 has quit IRC | 13:15 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** goutham has joined #openstack-infra | 13:17 | |
| goutham | hi does Zuul have support for Nested Virtualization | 13:17 |
| goutham | we have jobs which need nested virtualization | 13:18 |
| *** Guest85905 is now known as zeus | 13:19 | |
| *** zeus has quit IRC | 13:19 | |
| *** zeus has joined #openstack-infra | 13:19 | |
| *** dpawlik has quit IRC | 13:20 | |
| *** beisner is now known as beisner-sick | 13:22 | |
| *** cshastri has joined #openstack-infra | 13:23 | |
| mnaser | goutham: is this for an openstack project? | 13:24 |
| goutham | yes | 13:24 |
| *** Guest34116 has quit IRC | 13:24 | |
| mnaser | so by zuul, you mean the openstack deployment of zuul rather than zuul itself as a software :) | 13:24 |
| mnaser | i think it depends on providers.. some have it and some don't, most code will gracefully fallback to qemu emulation | 13:24 |
| mnaser | can i ask what sort of use case you have? | 13:24 |
| goutham | https://github.com/openstack/airship-promenade | 13:24 |
| mnaser | goutham: why does this need nested virt? | 13:25 |
| goutham | basically in this what we do is deploy 3 vms and deploy a kubernetes cluster across all the 3 vms | 13:26 |
| goutham | so we have support for KVM | 13:26 |
| mnaser | goutham: why don't you instead use zuuls multinode features and have it deploy 3 vms instead and depoy against them? | 13:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 13:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 13:27 |
| *** dhajare has quit IRC | 13:27 | |
| goutham | so u mean to say there we can use all the nodes instead of vms and then test ?? | 13:27 |
| *** armaan has quit IRC | 13:27 | |
| mnaser | goutham: zuul can give you multiple nodes so you can run jobs against them | 13:28 |
| goutham | hmm ok mnaser | 13:29 |
| mnaser | goutham: i think maybe you can look at a few projects like nova or devstack which do some sort of multinode | 13:29 |
| goutham | are there any scripts or document of how to use?? | 13:29 |
| mark-burnett | mnaser: indeed, the promenade gate doesn't require nested virt and should be able to work with normal multi-node | 13:29 |
| mnaser | let me try and find a sample multinode job for you.. | 13:29 |
| mark-burnett | we do have one project that it would be good to use nested virt to test with (airship-drydock), but we haven't started working on building out that testing in zuul yet | 13:30 |
| *** kgiusti has joined #openstack-infra | 13:32 | |
| mnaser | i am not finding something right away.. but maybe if you try in an hour or two there might be more infra people who will be able to help out | 13:32 |
| goutham | yes mnaser :) thanks | 13:32 |
| *** yamamoto has quit IRC | 13:36 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** Bhujay has quit IRC | 13:39 | |
| *** dpawlik has joined #openstack-infra | 13:39 | |
| *** Bhujay has joined #openstack-infra | 13:39 | |
| *** slaweq_ has joined #openstack-infra | 13:40 | |
| *** slaweq has quit IRC | 13:40 | |
| smcginnis | I know there's a cinder multinode job. Let me see if I can find that. | 13:41 |
| *** elbragstad is now known as lbragstad | 13:42 | |
| *** jpena|lunch is now known as jpena | 13:42 | |
| smcginnis | goutham: I think part of the trick is this flag: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/grenade-dsvm-cinder-mn-sub-bak/run.yaml#n38 | 13:43 |
| smcginnis | goutham: You can probably check out the rest of that job to see how it's all done. | 13:43 |
| goutham | sure i will do it thanks smcginnis | 13:43 |
| goutham | :) | 13:43 |
| smcginnis | Good luck. ;) | 13:44 |
| *** rossella_s has quit IRC | 13:44 | |
| *** camunoz has joined #openstack-infra | 13:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 13:47 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove getRepoPermission from FakeGithubConnection https://review.openstack.org/570428 | 13:47 |
| *** esberglu has joined #openstack-infra | 13:47 | |
| *** r-daneel has joined #openstack-infra | 13:48 | |
| *** rossella_s has joined #openstack-infra | 13:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add supercedent pipeline manager https://review.openstack.org/571932 | 13:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix link in mqtt reno https://review.openstack.org/572110 | 13:55 |
| corvus | goutham: you might start by just writing an ansible playbook to deploy k8s on 3 nodes, then making that the run playbook for a job that requests a nodeset with 3 nodes | 13:58 |
| *** anteaya has joined #openstack-infra | 13:59 | |
| *** mriedem has joined #openstack-infra | 14:00 | |
| corvus | goutham: (devstack is mostly concerned with deploying devstack, so probably won't be much use here) | 14:00 |
| *** VW has joined #openstack-infra | 14:01 | |
| *** r-daneel_ has joined #openstack-infra | 14:04 | |
| *** felipemonteiro has joined #openstack-infra | 14:05 | |
| pabelanger | I'd recommend using kubespray, I did some work for single node at https://review.openstack.org/549768 but would just need a new inventory file to use 3 nodes from zuul. | 14:06 |
| *** beekneemech is now known as bnemec | 14:06 | |
| pabelanger | plan was to create a base job for k8s deployment for projects to use, but never finished i | 14:06 |
| *** dave-mcc_ is now known as dave-mccowan | 14:06 | |
| *** r-daneel has quit IRC | 14:06 | |
| corvus | goutham: ^ that sounds very much like what you need? | 14:06 |
| goutham | Sure corvus i will do that thanks :) | 14:07 |
| goutham | thanks pabelanger :) | 14:08 |
| *** rossella_s has quit IRC | 14:09 | |
| *** r-daneel_ has quit IRC | 14:09 | |
| mnaser | https://blog.github.com/2018-06-04-github-microsoft/ | 14:09 |
| mnaser | oh its a thing now | 14:09 |
| *** r-daneel has joined #openstack-infra | 14:10 | |
| *** links has quit IRC | 14:10 | |
| *** dpawlik has quit IRC | 14:11 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Remove jobs from beaker-* projects https://review.openstack.org/571926 | 14:12 |
| *** rossella_s has joined #openstack-infra | 14:13 | |
| *** pblaho has quit IRC | 14:13 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:13 | |
| scas | the news was already out last night, but, yep. i'm a bit disappointed by tech shrinking in that way | 14:14 |
| *** felipemonteiro has quit IRC | 14:14 | |
| scas | nothing against the players, just the game being played | 14:14 |
| *** felipemonteiro_ has joined #openstack-infra | 14:15 | |
| *** r-daneel has quit IRC | 14:15 | |
| *** r-daneel has joined #openstack-infra | 14:16 | |
| *** e0ne_ has quit IRC | 14:16 | |
| *** e0ne has joined #openstack-infra | 14:17 | |
| scas | it reminds me of the recent visual rendition of the self-fulfilling prophecy ready player one. developers are all part of the game, but not the players in this, just the friendly zombies getting danced with and on </soapbox> | 14:18 |
| *** TheJulia is now known as needsleep | 14:18 | |
| *** felipemonteiro__ has quit IRC | 14:18 | |
| *** alex_xu_ has quit IRC | 14:20 | |
| *** alex_xu has joined #openstack-infra | 14:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add supercedent pipeline manager https://review.openstack.org/571932 | 14:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix link in mqtt reno https://review.openstack.org/572110 | 14:23 |
| *** goutham has left #openstack-infra | 14:24 | |
| *** felipemonteiro_ has quit IRC | 14:25 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:25 | |
| *** hashar is now known as hasharAway | 14:26 | |
| *** hamzy has quit IRC | 14:36 | |
| *** felipemonteiro_ has quit IRC | 14:36 | |
| *** felipemonteiro has joined #openstack-infra | 14:37 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:38 | |
| *** felipemonteiro has quit IRC | 14:42 | |
| *** alexchadin has quit IRC | 14:42 | |
| *** neiloy has joined #openstack-infra | 14:43 | |
| *** dbecker has quit IRC | 14:44 | |
| zigo | Guys, the VM on hold for puppet-nova can be released. | 14:46 |
| zigo | 104.239.240.37 aka debian-stretch-rax-dfw-0004211310 | 14:46 |
| *** eernst has joined #openstack-infra | 14:47 | |
| *** alex_xu has quit IRC | 14:47 | |
| *** gcb has quit IRC | 14:47 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: CAlDav 409 error handling https://review.openstack.org/571902 | 14:48 |
| *** alex_xu has joined #openstack-infra | 14:49 | |
| *** eernst has quit IRC | 14:49 | |
| *** eernst has joined #openstack-infra | 14:50 | |
| fungi | zigo: thanks for the update, i'll clear the autohold and delete the held node | 14:50 |
| zigo | fungi: Grafana shows there's only a single node on hold, so that should be the only one. | 14:51 |
| fungi | looks like the autohold was already cleared, so i deleted the node | 14:51 |
| zigo | Cheers. | 14:51 |
| zigo | fungi: I was able to fix puppet thanks to getting in, so thanks ! | 14:51 |
| fungi | yup, ianw had left a comment on the hold, but also i was able to match it by ip address | 14:51 |
| zigo | fungi: Now, I'm getting kind of close, I got only an iptables issue and the remaining 3 functional tests will pass. | 14:52 |
| zigo | fungi: For some reason, there's an iptables FORWARD problem. | 14:52 |
| zigo | As if the default ACCEPT was missing or something. | 14:52 |
| zigo | This should be a neutron config problem, maybe ... | 14:52 |
| fungi | multi-node or single-node job? | 14:52 |
| zigo | fungi: Single node. | 14:53 |
| zigo | puppet-openstack scenario001 | 14:53 |
| zigo | I got the same problem on my virtualbox VM when I run the test script. | 14:53 |
| fungi | i think our default iptables rules only block traffic into the real interfaces of the test node, but not the loopback or any virtual interfaces/bridges you create | 14:53 |
| zigo | fungi: Oh, I don't doubt the issue is on me ... | 14:54 |
| zigo | Just trying to figure out what's wrong on my side of things. | 14:54 |
| fungi | though we also don't set up any forwarding chains by default, so yeah, sounds like a neutron configuration problem | 14:54 |
| *** gcb has joined #openstack-infra | 14:54 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: CAlDav 409 error handling https://review.openstack.org/571902 | 14:54 |
| *** kiennt26 has joined #openstack-infra | 14:54 | |
| zigo | If I do "iptables -I FORWARD -j ACCEPT", then everything just work ... | 14:54 |
| zigo | Of course, that's the an acceptable way to do things. | 14:54 |
| *** dpawlik has joined #openstack-infra | 14:55 | |
| *** needsleep is now known as TheJulia | 14:56 | |
| andreaf | tobiash ianw corvus the way we setup /etc/hosts in zuulv3 multinode jobs does not work for nova live migration http://paste.openstack.org/show/722655/ | 14:58 |
| *** cshastri has quit IRC | 14:58 | |
| andreaf | in d-g we associate the proper hostname with the private IP - nova uses the hostname for live migration | 14:59 |
| andreaf | today instead we only associate the inventory name to the private IP | 14:59 |
| *** dpawlik has quit IRC | 14:59 | |
| andreaf | I think we should extend that to associate the private IP to both inventory and hostname, however that would conflict with the hostname being associated with the public IP as it is today | 15:00 |
| corvus | andreaf: what's the proper hostname? "ubuntu-xenial-rax-iad-0004273356" ? | 15:00 |
| andreaf | yeah | 15:00 |
| andreaf | that's ansible_hostname, and what you get if you run $(hostname) on the host | 15:00 |
| *** rpioso|afk is now known as rpioso | 15:00 | |
| corvus | do we set that in the job, or is that glean? | 15:01 |
| *** tesseract-RH has joined #openstack-infra | 15:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add driver interface documentation https://review.openstack.org/571538 | 15:01 |
| *** tesseract has quit IRC | 15:02 | |
| corvus | andreaf: here's where we do the private ip part: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-hosts-file/tasks/main.yaml#n1 | 15:02 |
| corvus | did devstack-gate just rewrite the whole hosts file? | 15:02 |
| *** pcichy has quit IRC | 15:03 | |
| corvus | no, it looks like it appended as well | 15:03 |
| *** pcichy has joined #openstack-infra | 15:03 | |
| andreaf | d-g appends to /etc/hosts http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n197 | 15:04 |
| andreaf | corvus yeah | 15:04 |
| *** kiennt26 has quit IRC | 15:04 | |
| *** armaan has joined #openstack-infra | 15:05 | |
| corvus | andreaf: adding the full hostname sounds fine to me, i'm just confused about why it's there already (and presumably isn't there when run under devstack-gate -- are we sure about that -- do you have a devstack-gate /etc/hosts file handy?) | 15:05 |
| *** armaan has quit IRC | 15:06 | |
| andreaf | corvus: I don't but I can make a tmp patch to create one | 15:06 |
| *** armaan has joined #openstack-infra | 15:06 | |
| *** armaan has quit IRC | 15:06 | |
| andreaf | unless we save it to logs | 15:06 |
| *** armaan has joined #openstack-infra | 15:07 | |
| *** camunoz has quit IRC | 15:07 | |
| andreaf | no we don't | 15:07 |
| andreaf | we have resolv.conf | 15:08 |
| corvus | also, we should ping clarkb for this conversation ^ | 15:08 |
| *** lifeless_ has quit IRC | 15:08 | |
| *** esberglu has quit IRC | 15:10 | |
| *** sshnaidm|brq has quit IRC | 15:10 | |
| *** felipemonteiro_ is now known as felipemonteiro | 15:14 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: DNM Debug /etc/hosts on controller https://review.openstack.org/572129 | 15:14 |
| *** felipemonteiro has quit IRC | 15:14 | |
| andreaf | corvus clarkb ^^^ | 15:14 |
| *** jbadiapa has quit IRC | 15:15 | |
| *** ramishra has quit IRC | 15:18 | |
| *** andreas_s has quit IRC | 15:19 | |
| *** e0ne has quit IRC | 15:19 | |
| * clarkb catches up | 15:21 | |
| *** hamzy has joined #openstack-infra | 15:21 | |
| *** whizzer has joined #openstack-infra | 15:22 | |
| clarkb | the thing that devstack-gate does is add the other hosts hostnames to /etc/hosts | 15:22 |
| clarkb | the current host's hostname should already be in there but for live migration to work you need working "dns" for those names on all computes | 15:23 |
| *** esberglu has joined #openstack-infra | 15:23 | |
| *** esberglu has quit IRC | 15:25 | |
| whizzer | I installed a 3 node devstack and assigned FLOATING_RANGE=192.168.2.0/24, it created an interface with 192.168.2.1 on the controller node and I set up some static routes and was off and running. After I rebooted the controller node though, it lost the ip address and I don't know where to re-configure it. There didn't seem to be a router associated with the ip when it worked or when it didn't work | 15:25 |
| clarkb | whizzer: that might be a question for neutron? | 15:26 |
| whizzer | okay | 15:26 |
| *** hongbin has joined #openstack-infra | 15:26 | |
| clarkb | whizzer: in general though devstack isn't expected to survive reboots | 15:26 |
| whizzer | I see | 15:26 |
| whizzer | everything else seems to survive... I understand that point though | 15:26 |
| whizzer | certainly I could re-stack | 15:26 |
| *** Bhujay has quit IRC | 15:27 | |
| *** esberglu has joined #openstack-infra | 15:27 | |
| *** caphrim007 has joined #openstack-infra | 15:27 | |
| *** esberglu_ has joined #openstack-infra | 15:28 | |
| *** caphrim007 has quit IRC | 15:29 | |
| *** esberglu has quit IRC | 15:32 | |
| *** benj_ has quit IRC | 15:32 | |
| *** caphrim007 has joined #openstack-infra | 15:33 | |
| *** caphrim007 has quit IRC | 15:33 | |
| *** caphrim007 has joined #openstack-infra | 15:33 | |
| *** pcaruana|worksho has quit IRC | 15:35 | |
| *** rfolco_ has joined #openstack-infra | 15:36 | |
| *** r-daneel_ has joined #openstack-infra | 15:36 | |
| *** janki has quit IRC | 15:37 | |
| *** rfolco has quit IRC | 15:37 | |
| *** r-daneel has quit IRC | 15:38 | |
| *** r-daneel_ is now known as r-daneel | 15:38 | |
| *** pleia2_ is now known as pleia2 | 15:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add doc build job for Chef OpenStack https://review.openstack.org/569572 | 15:42 |
| clarkb | ianw: hwoarang the connection wtih ntp and wheel builds after I thought they weren't connected is that the wheel builds use puppet-afs to setup the afs connectivity to the cluster and that uses puppet-ntp? | 15:43 |
| clarkb | its weird and may not need to be coupled like that | 15:44 |
| *** benj_ has joined #openstack-infra | 15:44 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Move pbr-installation jobs in-tree https://review.openstack.org/571571 | 15:45 |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Use upper-constraints in integration tests https://review.openstack.org/572150 | 15:45 |
| *** slaweq_ is now known as slaweq | 15:45 | |
| *** diablo_rojo has joined #openstack-infra | 15:52 | |
| *** yamamoto has quit IRC | 15:55 | |
| whizzer | clarkb: all I needed to do was reassign the IP with ifconfig on the br-ex interface | 15:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Directly link providers and request handlers https://review.openstack.org/571584 | 15:55 |
| clarkb | whizzer: I'm guessing that doesn't get written to the host's persistent network config | 16:02 |
| clarkb | andreaf: corvus I'm still a little confused about what 572129 is hoping to show us. Cna you expand on that a bit? | 16:02 |
| *** rajinir has joined #openstack-infra | 16:08 | |
| *** gyee has joined #openstack-infra | 16:08 | |
| *** pcichy has quit IRC | 16:09 | |
| *** pcichy has joined #openstack-infra | 16:10 | |
| *** neiloy has quit IRC | 16:11 | |
| corvus | clarkb: whether the "real hostname" (eg ubuntu-xenial-rax-iad-0004273356) appears in /etc/hosts with the public ip address | 16:13 |
| clarkb | corvus: it should, but glean does that | 16:14 |
| corvus | clarkb: (it does under new-style multinode; andreaf wants that hostname to be attached to the private ip) | 16:14 |
| clarkb | no I don't think that is right | 16:14 |
| corvus | clarkb: that's what i thought. after confirming that, i'm curious how we end up with that hostname mapped to the private ip (which andreaf says nova needs). | 16:15 |
| clarkb | at least from glean's perspective. Maybe we were undoing that with d-g | 16:15 |
| *** gcb has quit IRC | 16:15 | |
| clarkb | d-g updates /etc/hosts so that we don't have to have dns for the test environment and it adds all the nodes to each node's /etc/hosts but I don't recall if that is using the private ip or public | 16:15 |
| clarkb | however in this case it would be for the other nodes that matter not the current node (which adds a slight gotcha to the problem I guess) | 16:16 |
| *** sshnaidm|brq has joined #openstack-infra | 16:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: zuul-migrate: run / post-run playbooks need .yaml extensions https://review.openstack.org/572160 | 16:17 |
| *** felipemonteiro has joined #openstack-infra | 16:19 | |
| *** chkumar246 is now known as chandankumar | 16:24 | |
| *** eernst has quit IRC | 16:27 | |
| *** trown is now known as trown|lunch | 16:27 | |
| *** njohnston_ has quit IRC | 16:27 | |
| *** harlowja has joined #openstack-infra | 16:28 | |
| *** udesale has quit IRC | 16:30 | |
| *** ianychoi has quit IRC | 16:32 | |
| *** tesseract-RH has quit IRC | 16:34 | |
| clarkb | corvus: I'm going to grab some breakfast but then am around to help with zuul restarts if we want to do that soon | 16:38 |
| openstackgerrit | Merged openstack-infra/zuul master: Gracefully handle broken .gitmodule files https://review.openstack.org/567227 | 16:39 |
| *** edmondsw has quit IRC | 16:41 | |
| *** felipemonteiro has quit IRC | 16:43 | |
| *** ihar has quit IRC | 16:45 | |
| *** pcichy has quit IRC | 16:46 | |
| *** pcichy has joined #openstack-infra | 16:46 | |
| *** edmondsw has joined #openstack-infra | 16:46 | |
| *** ihar has joined #openstack-infra | 16:46 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix ConfigValue comparisons https://review.openstack.org/567927 | 16:46 |
| openstackgerrit | Merged openstack-infra/nodepool master: Make imagesAvailable() part of the driver API https://review.openstack.org/568702 | 16:46 |
| *** tosky has quit IRC | 16:47 | |
| *** e0ne has joined #openstack-infra | 16:47 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Use ProviderConfig iface to validate labels https://review.openstack.org/568703 | 16:48 |
| *** ianychoi has joined #openstack-infra | 16:48 | |
| *** felipemonteiro has joined #openstack-infra | 16:48 | |
| *** pcaruana|worksho has joined #openstack-infra | 16:51 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Simplify driver API https://review.openstack.org/568704 | 16:54 |
| *** yamamoto has joined #openstack-infra | 16:55 | |
| *** gouthamr_ is now known as gouthamr | 16:56 | |
| clarkb | corvus: quick skim of the git log for zuul makes me think we should restart all the zuul services | 16:58 |
| clarkb | quite a bit of updates to mergers, and the scheduler and web | 16:58 |
| fungi | clarkb: corvus: i'm around too if assistance is needed | 16:58 |
| *** whizzer has quit IRC | 16:59 | |
| *** caphrim007_ has joined #openstack-infra | 17:00 | |
| *** gfidente is now known as gfidente|afk | 17:02 | |
| *** yamamoto has quit IRC | 17:03 | |
| *** caphrim007 has quit IRC | 17:04 | |
| corvus | i think i'd like to restart zuul-web alone first. | 17:04 |
| clarkb | that is fine with me | 17:05 |
| *** sambetts_ is now known as sambetts|afk | 17:05 | |
| *** ldnunes has quit IRC | 17:05 | |
| *** ldnunes has joined #openstack-infra | 17:06 | |
| clarkb | corvus: zuul==3.0.4.dev70 # git sha 6c302e6 is the installed zuul on zuul01 according to pbr. That is the change that merged half an hour ago | 17:08 |
| clarkb | CherryPy==15.0.0 is installed too implying the installation picked up the new deps successfully | 17:08 |
| corvus | cool | 17:09 |
| corvus | i'll restart it now | 17:09 |
| *** esberglu_ has quit IRC | 17:09 | |
| *** esberglu has joined #openstack-infra | 17:10 | |
| *** harlowja has quit IRC | 17:10 | |
| *** jamesmcarthur has quit IRC | 17:12 | |
| *** jamesmcarthur has joined #openstack-infra | 17:13 | |
| *** esberglu has quit IRC | 17:14 | |
| *** iranzo has quit IRC | 17:15 | |
| *** jamesmcarthur has quit IRC | 17:15 | |
| *** neiloy has joined #openstack-infra | 17:16 | |
| *** eernst has joined #openstack-infra | 17:16 | |
| corvus | sorry, got sidetracked | 17:17 |
| corvus | restarting now | 17:17 |
| corvus | oops, a bug on startup, i'll fix locally real quick | 17:18 |
| *** kjackal has joined #openstack-infra | 17:18 | |
| *** pcaruana|worksho is now known as pcaruana | 17:18 | |
| *** jpena is now known as jpena|off | 17:19 | |
| corvus | i changed the config file to listen on "::" | 17:19 |
| corvus | it's running now | 17:19 |
| clarkb | corvus: what was it listening on before? or was it implied before? | 17:20 |
| corvus | the config file said "listen_address=" | 17:20 |
| corvus | and the error said "don't use the empty string" | 17:20 |
| clarkb | ah | 17:20 |
| clarkb | a hard refresh in my browser produces subway graphs | 17:20 |
| corvus | i'm sorting through whether we need to fix something in zuul, docs, or just local deployment | 17:20 |
| corvus | https://zuul-ci.org/docs/zuul/admin/components.html#attr-web.listen_address | 17:21 |
| corvus | so it looks like we were using an undocumented feature where the empty string meant listen on all v4/v6 | 17:21 |
| *** germs has joined #openstack-infra | 17:21 | |
| *** germs has quit IRC | 17:21 | |
| *** germs has joined #openstack-infra | 17:21 | |
| clarkb | that seems like a reasonable default if no value is supplied, however is foo= implying empty string or default value | 17:21 |
| *** yamahata has joined #openstack-infra | 17:21 | |
| corvus | seems like empty str | 17:22 |
| corvus | so perhaps we make no changes to zuul | 17:22 |
| corvus | since the cherrypy error is self-explanatory | 17:22 |
| corvus | and just update our local config | 17:23 |
| clarkb | ya either set :: in our config or drop the setting of '' | 17:23 |
| *** jesslampe has joined #openstack-infra | 17:23 | |
| corvus | i'll drop the config line and restart | 17:23 |
| corvus | restarted again | 17:24 |
| *** neiloy has quit IRC | 17:29 | |
| *** neiloy has joined #openstack-infra | 17:30 | |
| *** jamesmcarthur has joined #openstack-infra | 17:30 | |
| *** jamesmcarthur has quit IRC | 17:30 | |
| *** jamesmcarthur has joined #openstack-infra | 17:30 | |
| openstackgerrit | Merged openstack-infra/zuul master: Status branch protection checking for github https://review.openstack.org/535680 | 17:31 |
| clarkb | corvus: are you working on a patch to update the puppetry for that? | 17:31 |
| corvus | yep | 17:31 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add multilabel support to ZooKeeper Node model https://review.openstack.org/570227 | 17:31 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add driver interface documentation https://review.openstack.org/571538 | 17:31 |
| *** neiloy has quit IRC | 17:32 | |
| *** neiloy has joined #openstack-infra | 17:32 | |
| *** zoli|wfh is now known as zoli|gone | 17:32 | |
| *** zoli|gone is now known as zoli | 17:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add web listen arguments to zuul class https://review.openstack.org/572172 | 17:33 |
| corvus | clarkb, fungi: lemme know if that makes sense and you think it's the write fix ^ | 17:33 |
| openstackgerrit | Merged openstack-infra/nodepool master: Directly link providers and request handlers https://review.openstack.org/571584 | 17:33 |
| clarkb | corvus: wfm | 17:35 |
| fungi | yeah, that should do the trick nicely | 17:35 |
| *** neiloy has quit IRC | 17:36 | |
| corvus | i'm going to afk for 30m or so | 17:36 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Outlook Cal Sync Token revocation error https://review.openstack.org/572174 | 17:39 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Outlook Cal Sync Token revocation error https://review.openstack.org/572174 | 17:40 |
| *** neiloy has joined #openstack-infra | 17:42 | |
| clarkb | corvus: andreaf http://logs.openstack.org/29/572129/1/check/legacy-tempest-dsvm-multinode-full/69db533/job-output.txt.gz#_2018-06-04_15_25_32_766962 d-g is writing the public IP addrs to etc/hosts however that is a cloud wtihout private ip addresses so not sure that really tells us anything | 17:43 |
| clarkb | corvus: andreaf http://logs.openstack.org/29/572129/1/check/neutron-grenade-multinode/169f066/job-output.txt.gz#_2018-06-04_15_34_10_018829 that job ran on rax whcih has private interfaces and it used private IPs there | 17:45 |
| clarkb | I guess that confirms andreaf's suspicions | 17:45 |
| *** amoralej is now known as amoralej|off | 17:45 | |
| *** harlowja has joined #openstack-infra | 17:49 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Update presentation model - Berlin CFP https://review.openstack.org/572177 | 17:51 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Update presentation model - Berlin CFP https://review.openstack.org/572177 | 17:53 |
| *** trown|lunch is now known as trown | 17:54 | |
| *** shardy has quit IRC | 17:55 | |
| *** r-daneel has quit IRC | 17:58 | |
| *** markvoelker has quit IRC | 17:59 | |
| *** markvoelker has joined #openstack-infra | 18:00 | |
| clarkb | corvus: when you get back, thoughts on restarting the rest of the zuul services? | 18:03 |
| *** r-daneel has joined #openstack-infra | 18:09 | |
| corvus | clarkb, andreaf: so apparently it's okay to have the public and private ip associated with the hostname in /etc/hosts. so i think that means just a simple change to the multinode etc hosts role to add the hostname (along with the inventory name) to /etc/hosts should be sufficient. | 18:09 |
| clarkb | ya based on existing d-g behavior that should be fine | 18:10 |
| corvus | clarkb: restarting zuul wfm | 18:10 |
| *** esberglu has joined #openstack-infra | 18:11 | |
| clarkb | ok I'll ping the release team | 18:11 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add web listen arguments to zuul class https://review.openstack.org/572172 | 18:13 |
| clarkb | corvus: process is stop scheduler, while waiting for it to actually stop stop the mergers and executors, start mergers, start scheduler, start executors ? | 18:15 |
| clarkb | should restart fingergw too I guess | 18:17 |
| *** camunoz has joined #openstack-infra | 18:17 | |
| corvus | clarkb: scheduler stops pretty quickly. i say we try: stop sched, stop mergers, stop execs, stop finger, start sched, start finger, wait for mergers to stop, start mergers, wait for execs to stop, start execs. | 18:19 |
| corvus | clarkb: (i think the scheduler stops in < 15 seconds) | 18:20 |
| clarkb | wfm. release team is good for us to do that now | 18:21 |
| clarkb | I'm going to queue some of this up in a terminal on puppetmaster | 18:22 |
| clarkb | corvus: fungi ready to do this now? I can drive and will ping if anything comes up | 18:22 |
| clarkb | I guess I should double check the git sha1 on all the other nodes first | 18:23 |
| *** roman_g has quit IRC | 18:24 | |
| *** caphrim007_ has quit IRC | 18:24 | |
| *** caphrim007 has joined #openstack-infra | 18:25 | |
| corvus | clarkb: ready when you are. also ++ git sha check | 18:25 |
| clarkb | zuul==3.0.4.dev72 # git sha fcd468d appears to be the sha1 across the board which lgtm | 18:26 |
| fungi | yeah, i'm ready | 18:26 |
| clarkb | if that sha1 looks right to you all I'll go ahead with the process outlined above | 18:27 |
| clarkb | oh we need to capture queues /me adds taht to process | 18:28 |
| fungi | confirmed, fcd468d is what i have for origin/master as of seconds ago | 18:29 |
| clarkb | ok capturing check and gate now then proceeding with the plan corvus wrote down | 18:29 |
| clarkb | scheduler taking longer than 15 seconds to stop | 18:31 |
| *** aojea has quit IRC | 18:31 | |
| corvus | oh, it looks like it's in the middle of a bunch of layout generations; bad luck. it'll get there though. | 18:32 |
| clarkb | ok should I go ahead and stop the other services or wait? | 18:32 |
| clarkb | I guess its safe to stop the other services since this won't be successful layout generation (rather not used) | 18:32 |
| corvus | should be | 18:32 |
| corvus | if not, happy to learn that :) | 18:32 |
| clarkb | mergers and executors asked to stop now | 18:33 |
| clarkb | mergers are all stopped | 18:34 |
| corvus | i'm starting to be confused about why the scheduler hasn't stopped | 18:35 |
| fungi | i hope we haven't exposed a signal handling race | 18:36 |
| clarkb | `service zuul-scheduler stop` is what I told it | 18:36 |
| fungi | oh, right, it's an rpc these days | 18:36 |
| corvus | clarkb: i say kill it with fire; i'll look at logs | 18:36 |
| clarkb | ok | 18:36 |
| corvus | (i saw it get the stop signal) | 18:36 |
| fungi | do we want a thread dump first? | 18:36 |
| corvus | fungi: good idea clarkb ^ | 18:37 |
| clarkb | sigusr2 right? | 18:37 |
| corvus | yep | 18:37 |
| clarkb | kill -USR2 3194 issued | 18:37 |
| clarkb | did you see the dump in the lgos? | 18:38 |
| corvus | yep | 18:38 |
| clarkb | I'm going to kill then kill -9 if I have to | 18:38 |
| fungi | yeah, i see threads mentioned in the debug log | 18:38 |
| clarkb | ok that stopped it | 18:39 |
| fungi | 2018-06-04 18:37:37,928 DEBUG zuul.stack_dump: Starting Yappi | 18:39 |
| fungi | 2018-06-04 18:37:37,929 DEBUG zuul.stack_dump: End debug handler | 18:39 |
| clarkb | I'm stopping finger then starting sched and fingergw | 18:39 |
| fungi | so looks like it dumped it all | 18:40 |
| clarkb | I did not need kill -9 fwiw | 18:40 |
| corvus | Thread-1 seems to be the scheduler going about its merry business | 18:40 |
| clarkb | sched is coming back | 18:40 |
| *** mriedem has quit IRC | 18:40 | |
| clarkb | starting finger and mergers | 18:40 |
| clarkb | and executors were stopped so starting them too | 18:41 |
| fungi | cool, so the initscript's stop mechanism didn't dtrt but regular old sigterm did? | 18:41 |
| corvus | fungi: the init script stop dtrt, but zuul did not | 18:42 |
| clarkb | fungi: well it did the socket stop command according to corvus' reading of the logs | 18:42 |
| clarkb | right that | 18:42 |
| clarkb | once sched is done reconfiguring I will reenqueue things | 18:42 |
| fungi | er, sure that ;) | 18:42 |
| *** mriedem has joined #openstack-infra | 18:43 | |
| fungi | i should have said asking it to stop with the initscript seems to have not done the right thing | 18:43 |
| corvus | i just want to be clear that it's not like systemd/init failed here | 18:43 |
| fungi | yes | 18:43 |
| *** myoung is now known as myoung|lunch | 18:43 | |
| clarkb | hrm does zuul-web need to be restarted too? | 18:44 |
| fungi | i see we also have a /etc/init.d/zuul on zuul01... is that cruft? | 18:44 |
| clarkb | I think I can requeue but the web isn't showing me that | 18:44 |
| corvus | clarkb: give it a shot | 18:44 |
| fungi | yeah, zuul-web has needed restarting in the past if the scheduler is restarted | 18:44 |
| corvus | (i wasn't sure how it would be have since it's new) | 18:44 |
| clarkb | that seems to have done it | 18:45 |
| clarkb | requeuing now | 18:45 |
| fungi | i guess the new zuul-web also needs it (for now) | 18:45 |
| clarkb | ok two things learned here. zuul-scheduler stop needs debugging and we should try to make zuul-web resilient to scheduler restarts | 18:45 |
| fungi | is it possible that old zuul-web was making assumptions which were not true for newer zuul-scheduler? | 18:46 |
| clarkb | well we restarted zuul-web independently prior to global restart so I don't think so | 18:47 |
| clarkb | more likely it was hanging on gearman connection going down or similar | 18:47 |
| fungi | ahh, so it was already on the new version | 18:47 |
| fungi | i missed that in the sequence | 18:47 |
| fungi | sorry | 18:47 |
| *** Sukhdev has joined #openstack-infra | 18:47 | |
| clarkb | fungi: corvus did it about an hour ago independent of this since the web server was massively rewritten | 18:47 |
| fungi | aha | 18:48 |
| clarkb | gate is enqueued check is going in now | 18:48 |
| fungi | right, hence the puppet-zuul change i approved | 18:48 |
| fungi | my short-term memory is apparently terrible today | 18:48 |
| clarkb | it is one of those days | 18:48 |
| corvus | oh interesting; it got 2 stop signals | 18:49 |
| corvus | https://etherpad.openstack.org/p/iuidl8MHuE | 18:49 |
| clarkb | Shrews: can you push or recheck an ansible change that we third party ci? just to make sure its all still happy with latest zuul | 18:50 |
| clarkb | I only ran it the once | 18:50 |
| corvus | oh, sorry, i misread that | 18:50 |
| corvus | the second was the internal _stop command | 18:51 |
| corvus | ah, i have a theory | 18:52 |
| corvus | i think shutting down the executors put a whole bunch of events on the queues, and we were just working through those | 18:53 |
| corvus | so for the moment, i think clarkb's approach of wait until scheduler stops before stopping executors wins | 18:53 |
| clarkb | interesting | 18:53 |
| corvus | we might want to put in a short circuit for stopping in there :) | 18:53 |
| clarkb | mordred: see message above to shrews, can you trigger things that will be third party checked by us in ansible to double check it is still happy after the update? | 18:54 |
| clarkb | on the zuul-web front, if I had to guess the cherrypythreading pool model runs threads as non daemon threads and the threads asking gearman for data will hang | 18:55 |
| mordred | clarkb: yes - one sec | 18:56 |
| clarkb | the gerrit stuff seems happy | 18:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop scheduler more quickly https://review.openstack.org/572186 | 18:57 |
| fungi | yeah, looks like stuff is queued | 18:57 |
| fungi | about to start paying attention to the diversity wg meeting, but still around so ping me if you need anything | 18:58 |
| clarkb | fungi: I think we are in good shape. Will ping if that changes | 18:58 |
| clarkb | now to review the fix above | 18:58 |
| fungi | same | 18:58 |
| mordred | clarkb, corvus: I left a recheck comment here: https://github.com/ansible/ansible/pull/37170 but I don't see anything queued | 18:59 |
| clarkb | mordred: I'm double checking the files list on the jobs | 19:00 |
| *** pcichy has quit IRC | 19:01 | |
| *** eernst has quit IRC | 19:01 | |
| *** wadadli has left #openstack-infra | 19:01 | |
| clarkb | seens to match the current list | 19:01 |
| *** eernst has joined #openstack-infra | 19:02 | |
| clarkb | I suppose that implies something in tobiash's github refactor may be at fault | 19:02 |
| corvus | or zuul-web | 19:02 |
| corvus | i don't see 37170 in web-debug.log | 19:02 |
| fungi | since it has to receive the events from the hook, right? | 19:02 |
| corvus | ya | 19:02 |
| clarkb | ah ya if the web server isn't getting it then tobiash's refactor won't have much to work off of | 19:03 |
| corvus | we should log into github and check their transmission log | 19:03 |
| corvus | see if we told github we got the event | 19:03 |
| tobiash | clarkb: most of the github stuff was actually github tedt refactoring | 19:03 |
| corvus | i need to grab lunch; can pitch in on that when i return | 19:03 |
| clarkb | tobiash: ya as corvus points out it could be zuul-web | 19:03 |
| tobiash | s/tedt/test | 19:03 |
| clarkb | I'm nto sure I know what to log in as? can I see that as a github admin on things? | 19:04 |
| fungi | lack of mention in web-debug.log makes that sound more likely | 19:04 |
| corvus | clarkb: yeah, as an admin of the openstack-infra org which owns the openstack-zuul app, you can see the webhook calls in the openstack-zuul app's debug tag | 19:04 |
| *** salv-orlando has joined #openstack-infra | 19:05 | |
| clarkb | cool I'll see if I can dig that up | 19:05 |
| corvus | gotta run, back asap | 19:05 |
| mordred | clarkb: if you go to https://github.com/organizations/openstack-infra/settings/profile | 19:05 |
| mordred | you can navigate to "github apps" then click on the app - which should be: | 19:06 |
| mordred | https://github.com/organizations/openstack-infra/settings/apps/openstack-zuul | 19:06 |
| mordred | then 'advanced' is the tab you want for webhook delivery things | 19:06 |
| clarkb | " Something went really wrong. " when asking for my password, nice | 19:06 |
| mordred | it looks like we have green deliveries | 19:06 |
| clarkb | oh it wants my yubikey and apparently gives a nasty warning if its not there immediately | 19:07 |
| mordred | I don't see an event for the recheck comment at all | 19:08 |
| clarkb | mordred: ya I confirm that, based on timestamps and grepping for the PR number I don't see anything | 19:08 |
| *** VW_ has joined #openstack-infra | 19:09 | |
| tobiash | That's odd | 19:09 |
| *** eernst has quit IRC | 19:09 | |
| tobiash | Is the app still installed in the ansible repo/org? | 19:09 |
| *** lpetrut has joined #openstack-infra | 19:09 | |
| mordred | yah - we're getting other events | 19:09 |
| *** salv-orlando has quit IRC | 19:09 | |
| mordred | let me try again | 19:09 |
| clarkb | tobiash: ya there are other events for that repo | 19:09 |
| mordred | still nothing | 19:10 |
| tobiash | Maybe github changed some permission settings? | 19:10 |
| * fungi blames microsoft ;) | 19:10 | |
| tobiash | You may want to check the permissions page of the app | 19:10 |
| mordred | yah - checked that - it shows pull request comments | 19:11 |
| clarkb | we had a bunch of timeouts while we restarted things, is it possible they've blacklisted us as a result? though that wouldn't explain why we have other events from the shippable comments | 19:11 |
| mordred | but maybe something got unset on the installation in ansible/ansible ? | 19:11 |
| clarkb | also there is a redeliver button implying they are happy to accomodate these types of outages | 19:11 |
| mordred | yah - they'll happily deliver to a broken endpoint for a long time | 19:12 |
| *** VW has quit IRC | 19:12 | |
| *** rossella_s has quit IRC | 19:12 | |
| *** VW_ has quit IRC | 19:13 | |
| *** rossella_s has joined #openstack-infra | 19:14 | |
| clarkb | is it possible there is lag in getting those events out? | 19:16 |
| clarkb | github says all systems are running properly | 19:16 |
| clarkb | there are comments from regular users being posted too according to the log | 19:18 |
| *** lpetrut has quit IRC | 19:19 | |
| *** diablo_rojo_ has joined #openstack-infra | 19:19 | |
| *** jaosorior has quit IRC | 19:19 | |
| *** armaan has quit IRC | 19:19 | |
| *** lpetrut has joined #openstack-infra | 19:19 | |
| *** armaan has joined #openstack-infra | 19:20 | |
| *** diablo_rojo_ has quit IRC | 19:20 | |
| *** lpetrut has quit IRC | 19:20 | |
| fungi | are there backlogged timestamps on the events we are receiving? | 19:20 |
| fungi | maybe the pipes are clogged with everyone deleting their repos and accounts ;) | 19:20 |
| *** diablo_rojo has quit IRC | 19:20 | |
| *** lpetrut has joined #openstack-infra | 19:21 | |
| clarkb | not according to the events themselves | 19:21 |
| clarkb | looking at the PRs that we are getting eevnts for they don't appear to be backlogged | 19:21 |
| *** lpetrut has quit IRC | 19:22 | |
| *** sambetts|afk has quit IRC | 19:22 | |
| *** lpetrut has joined #openstack-infra | 19:22 | |
| *** rkukura has quit IRC | 19:24 | |
| *** sambetts_ has joined #openstack-infra | 19:25 | |
| *** diablo_rojo has joined #openstack-infra | 19:25 | |
| clarkb | I double checked the perms requests on the main app setup. Can projects like ansible trim it down from that? | 19:26 |
| clarkb | I think http://zuul.openstack.org/builds.html is not working either. That may be due to updates in how we config the db? | 19:28 |
| tobiash | clarkb: not that I know (except the perms of the app itself have been changed) | 19:28 |
| *** harlowja has quit IRC | 19:29 | |
| *** diablo_rojo has quit IRC | 19:33 | |
| *** lifeless has joined #openstack-infra | 19:34 | |
| *** rkukura has joined #openstack-infra | 19:34 | |
| *** rwsu has joined #openstack-infra | 19:34 | |
| *** rossella_s has quit IRC | 19:35 | |
| *** diablo_rojo has joined #openstack-infra | 19:35 | |
| *** diablo_rojo has quit IRC | 19:35 | |
| *** diablo_rojo has joined #openstack-infra | 19:37 | |
| *** slaweq has quit IRC | 19:38 | |
| *** rossella_s has joined #openstack-infra | 19:38 | |
| *** kjackal has quit IRC | 19:38 | |
| *** slaweq has joined #openstack-infra | 19:38 | |
| clarkb | poking at the builds issue we don't appear to be logging the cherrypy tracebacks? | 19:38 |
| clarkb | at least they don't show up in web-debug.log when I curl directly against localhost to ensure I'm hitting cherrypyo | 19:39 |
| *** Sukhdev has quit IRC | 19:39 | |
| clarkb | ok I think I see why, we have a special logging config for zuul-web and it defaults root logging to warning or higher | 19:41 |
| clarkb | I'll work on updating that to aid in debuggin | 19:41 |
| corvus | shouldn't a traceback be at error? | 19:43 |
| clarkb | you would think so | 19:44 |
| clarkb | but its definitely not coming through | 19:44 |
| corvus | i've read backlog, but i don't fully understand -- does our zuul see any github events? | 19:45 |
| clarkb | corvus: it sees the events that github is publishing. | 19:45 |
| clarkb | corvus: github is not publishing the event for mordred's recheck comment | 19:46 |
| corvus | ok, so we didn't see the ansible event because github didn't publish it. neat | 19:46 |
| clarkb | (at least according to github's logs) | 19:46 |
| corvus | clarkb: so the only thing we currently suspect is wrong with zuul-web is the builds thing? | 19:46 |
| *** calebb has quit IRC | 19:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zuul master: Configure cherrypy logging https://review.openstack.org/572194 | 19:47 |
| clarkb | corvus: yes | 19:47 |
| clarkb | that change isn't tested but is based on the docs linked to in the commit message | 19:48 |
| corvus | i'd like to hold off on that while i look into why we're not already seeing it | 19:48 |
| clarkb | ok | 19:49 |
| clarkb | corvus: reading the docs they imply cherrypy.errors is the place errors are scoped at then they log them at INFO level | 19:49 |
| *** jamesmcarthur has quit IRC | 19:49 | |
| clarkb | which is not going to match WARNING or higher. It is possible they just have their own ideas for how to manage log levels and errors | 19:49 |
| corvus | clarkb: i raised an exception and got 2018-06-04 12:48:14,934 cherrypy.error.140257947233920 ERROR [04/Jun/2018:12:48:14] HTTP | 19:50 |
| corvus | Traceback (most recent call last): | 19:50 |
| clarkb | that is error level so we should see that I guess | 19:50 |
| clarkb | fwiw I curl'd directly to the endpoint which gives me a cherrypy specific 500 error so I am fairly confident it isn't somewhere else that is failing | 19:51 |
| *** calebb has joined #openstack-infra | 19:51 | |
| *** eernst has joined #openstack-infra | 19:54 | |
| *** Sukhdev has joined #openstack-infra | 19:55 | |
| corvus | clarkb: ah, i see. the log config has the root logger only going to the console, not the files | 19:57 |
| *** myoung|lunch is now known as myoung | 19:57 | |
| *** kzaitsev1pi has quit IRC | 19:57 | |
| corvus | clarkb: how about we do 'cherrypy' at WARN instead of cherrypy.error at info? | 19:57 |
| *** kzaitsev_pi has joined #openstack-infra | 19:58 | |
| *** harlowja has joined #openstack-infra | 19:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add cherrypy to built-in logging config https://review.openstack.org/572200 | 19:59 |
| corvus | (that ^ will fix the default zuul log config, but we don't use that (yet) so we should still merge clarkb's change) | 19:59 |
| clarkb | corvus: that should work want to update my change? I'm making lunch | 19:59 |
| corvus | clarkb: will do | 20:00 |
| *** lpetrut has quit IRC | 20:00 | |
| clarkb | does journalctl show the error we need? | 20:00 |
| clarkb | systemd should put the console logging there | 20:01 |
| *** lpetrut has joined #openstack-infra | 20:01 | |
| *** eernst has quit IRC | 20:01 | |
| corvus | nope, because of the way we background | 20:01 |
| clarkb | oh right | 20:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Configure cherrypy logging https://review.openstack.org/572194 | 20:02 |
| *** eernst has joined #openstack-infra | 20:02 | |
| corvus | i'll go ahead and make that change manually on the server and restart | 20:02 |
| *** eernst has quit IRC | 20:02 | |
| *** eernst_ has joined #openstack-infra | 20:02 | |
| corvus | Exception: Unable to find connection for tenant openstack | 20:03 |
| *** eernst_ has quit IRC | 20:03 | |
| corvus | so it couldn't find the appropriate sql connection | 20:04 |
| *** salv-orlando has joined #openstack-infra | 20:05 | |
| clarkb | because we scope them niw? that needs a config update too I think | 20:05 |
| corvus | it should be automatic; | 20:06 |
| corvus | it tries to find the first sql connection that shows up in a tenant's pipelines | 20:07 |
| corvus | (basically, we're just assuming there's only going to be one per tenant for now. which is better than what was there before -- it assumed one per installation) | 20:07 |
| corvus | there is a unit test of this, so i'm a bit puzzled | 20:08 |
| *** slaweq has quit IRC | 20:09 | |
| *** slaweq has joined #openstack-infra | 20:09 | |
| corvus | i think i see the problem | 20:10 |
| corvus | zuul-web is looking at the tenant's layout, but it's zuul-web, not the scheduler, so there is no layout loaded. | 20:10 |
| corvus | (the unit test probably works because we erroneously share connection objects between the scheduler and web) | 20:10 |
| *** salv-orlando has quit IRC | 20:10 | |
| corvus | so zuul-web needs to ask the scheduler which sql connection it should use for a given tenant | 20:11 |
| corvus | this is ... awkward | 20:11 |
| *** jamesmcarthur has joined #openstack-infra | 20:12 | |
| *** e0ne has quit IRC | 20:13 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: [DNM] Test puppet 4 beaker jobs https://review.openstack.org/571987 | 20:14 |
| *** felipemonteiro has quit IRC | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:17 | |
| corvus | i'm working on patches to untangle it | 20:19 |
| *** felipemonteiro has joined #openstack-infra | 20:21 | |
| anteaya | under the category of news articles that mention zuul: http://www.bbc.com/news/blogs-news-from-elsewhere-44302596 | 20:25 |
| *** Sukhdev has quit IRC | 20:31 | |
| *** jcoufal has quit IRC | 20:33 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/openstack-zuul-jobs master: Add experimental puppet 4 beaker-rspec jobs https://review.openstack.org/571966 | 20:34 |
| *** rossella_s has quit IRC | 20:36 | |
| *** rossella_s has joined #openstack-infra | 20:37 | |
| *** e0ne has joined #openstack-infra | 20:38 | |
| *** rossella_s has quit IRC | 20:46 | |
| *** lifeless_ has joined #openstack-infra | 20:49 | |
| *** larainema has quit IRC | 20:50 | |
| *** armaan has quit IRC | 20:50 | |
| *** rossella_s has joined #openstack-infra | 20:50 | |
| *** lifeless has quit IRC | 20:50 | |
| *** lpetrut has quit IRC | 20:52 | |
| *** janki has joined #openstack-infra | 20:54 | |
| *** armaan has joined #openstack-infra | 20:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix zuul-web sql connections https://review.openstack.org/572210 | 20:56 |
| corvus | clarkb: ^ i think that should do the trick; want to give it a quick glance and i can manually install it on zuul01 to check? | 20:56 |
| clarkb | yup looking now | 20:57 |
| *** kjackal has joined #openstack-infra | 20:58 | |
| *** pcaruana has quit IRC | 20:59 | |
| corvus | also, ftr, when i updated the tests to use separate connection registries, the unit test broke in the same way as prod, then with the rest of that change, it works again | 21:00 |
| clarkb | corvus: left a couple of comments you should check before deploying it | 21:02 |
| fungi | that's a promising development | 21:02 |
| *** e0ne has quit IRC | 21:02 | |
| *** hasharAway has quit IRC | 21:02 | |
| *** larainema has joined #openstack-infra | 21:03 | |
| *** eernst has joined #openstack-infra | 21:03 | |
| corvus | clarkb: the break is inside the "for driver_class" loop, but the continue is outside of it, but inside of "for section_name" | 21:03 |
| *** harlowja has quit IRC | 21:04 | |
| corvus | clarkb: so that should mean "if this driver isn't sql, go to the next one without initializing it", right? | 21:04 |
| *** stewie925 has quit IRC | 21:04 | |
| clarkb | oh right it is skipping everything in the outer loop with that continue | 21:05 |
| *** trown is now known as trown|outtypewww | 21:06 | |
| *** salv-orlando has joined #openstack-infra | 21:06 | |
| corvus | clarkb: replied on your second too. i think that's good enough to install now, so i'll do that. | 21:06 |
| *** jamesmcarthur has joined #openstack-infra | 21:07 | |
| clarkb | ya I'm happy with it given those responses | 21:07 |
| clarkb | we need to update both scheduler and web right? | 21:08 |
| corvus | oh, yep, since the scheduler has the new rpc method | 21:08 |
| *** dpawlik has joined #openstack-infra | 21:09 | |
| corvus | clarkb: i'll go ahead and restart the scheduler now | 21:09 |
| corvus | i just did zuul-web | 21:09 |
| clarkb | ok | 21:09 |
| *** janki has quit IRC | 21:10 | |
| *** felipemonteiro has quit IRC | 21:10 | |
| *** salv-orlando has quit IRC | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:11 | |
| *** aeng_ has joined #openstack-infra | 21:12 | |
| *** gfidente|afk has quit IRC | 21:13 | |
| *** dpawlik has quit IRC | 21:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 21:14 |
| clarkb | looking at the github events list more closely I'm not sure we are getting any comments from users. There are reviews, CI status updates, PR state changes like closed, reopened, merged, but none are plain comments from users I think | 21:15 |
| corvus | hey i have data | 21:15 |
| clarkb | mordred: ^ makes me wonder if the permissions are wrong | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | however my read of the perms was that we have everything necessary listed | 21:15 |
| clarkb | corvus: me too | 21:15 |
| corvus | i will re-enqueue now | 21:15 |
| clarkb | corvus: we have 404s on github events queue now though | 21:17 |
| clarkb | <p>The path '/api/connection/github/payload' was not found.</p> | 21:17 |
| clarkb | are we overaggressively filtering the connections? | 21:17 |
| corvus | yep we are | 21:17 |
| *** VW has joined #openstack-infra | 21:18 | |
| corvus | we do need the github connection | 21:18 |
| corvus | i'll patch that in real quick; we should clean up this api after this though | 21:20 |
| clarkb | ++ | 21:20 |
| corvus | i'll run the github unit tests locally this time too | 21:23 |
| corvus | (they do indeed catch that error) | 21:24 |
| *** felipemonteiro has joined #openstack-infra | 21:27 | |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| *** myoung is now known as myoung|off | 21:31 | |
| *** jamesmcarthur has quit IRC | 21:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix zuul-web sql connections https://review.openstack.org/572210 | 21:32 |
| corvus | clarkb: ^ there's a patch. again, this is weird because the connections were shared in testing before. zuul-web needs to shut down the sql connection in its tests, but doesn't need to shut down the github connection. :| | 21:33 |
| corvus | so that's a quick fix | 21:34 |
| corvus | i'm going to try a complete local test run before i install it this time | 21:34 |
| *** edmondsw has quit IRC | 21:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 21:35 |
| clarkb | corvus: the gearman_worker attribute indicates that this is running outside of testing? | 21:36 |
| *** felipemonteiro has quit IRC | 21:36 | |
| *** harlowja has joined #openstack-infra | 21:36 | |
| corvus | clarkb: no it means that the connection was "started" (the onLoad method called) | 21:37 |
| clarkb | gotcha so it is checking if you can stop the connection by checking if it has been started | 21:37 |
| corvus | that only happens in the scheduler | 21:37 |
| corvus | yep | 21:37 |
| *** esberglu_ has joined #openstack-infra | 21:42 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Configure cherrypy logging https://review.openstack.org/572194 | 21:43 |
| corvus | okay, i'm going to restart the scheduler again | 21:44 |
| *** neiloy has quit IRC | 21:44 | |
| *** esberglu has quit IRC | 21:44 | |
| *** felipemonteiro has joined #openstack-infra | 21:45 | |
| *** slaweq has quit IRC | 21:48 | |
| corvus | clarkb: how's github lookin? | 21:49 |
| *** dave-mccowan has quit IRC | 21:50 | |
| clarkb | last event was after you said you are restarting and was 404 | 21:50 |
| clarkb | er | 21:51 |
| clarkb | was before you said you were restarting | 21:51 |
| clarkb | I think we just need to wait for new events | 21:51 |
| *** kien-ha has joined #openstack-infra | 21:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 21:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 21:53 |
| *** inc0 has quit IRC | 21:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Stop falling back to job name for missing url https://review.openstack.org/554056 | 21:58 |
| clarkb | it has been 17 minutes without a github event :/ | 21:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove dashboard workaround for missing log_url https://review.openstack.org/554066 | 21:59 |
| *** felipemonteiro has quit IRC | 22:00 | |
| corvus | something is querying the build db for tripleo jobs every 20 seconds | 22:00 |
| *** felipemonteiro has joined #openstack-infra | 22:00 | |
| mordred | clarkb: the ansible repo is more active than that :( | 22:02 |
| clarkb | mordred: something weird seems to be going on and if our config is correct it must be in github itself | 22:02 |
| mordred | lovely | 22:02 |
| clarkb | corvus: still seeing "<p>The path '/api/connection/github/payload' was not found.</p>" | 22:03 |
| clarkb | there were events last minute | 22:03 |
| *** rcernin has joined #openstack-infra | 22:03 | |
| andreaf | clarkb corvus ok thanks, I'll propose a patch to zuul-jobs tomorrow - I think it should be fine to inject the inventory name there along with the hostname | 22:05 |
| corvus | clarkb: i'll patch zuul-web locally to do further debugging (won't need a scheduler restart) | 22:06 |
| clarkb | corvus: ok | 22:06 |
| clarkb | I have double checked web-debug.log and the warning level isn't high enoguh to show us why that is unhappy | 22:07 |
| clarkb | (I assume 404s are logged at debug or info) | 22:07 |
| *** salv-orlando has joined #openstack-infra | 22:07 | |
| *** rossella_s has quit IRC | 22:09 | |
| openstackgerrit | Merged openstack-infra/grafyaml master: Test case for multiple targets with hidden https://review.openstack.org/572020 | 22:10 |
| openstackgerrit | Merged openstack-infra/grafyaml master: Update documentation https://review.openstack.org/572076 | 22:10 |
| *** rossella_s has joined #openstack-infra | 22:11 | |
| *** felipemonteiro has quit IRC | 22:12 | |
| corvus | clarkb: okay. i installed it incorrectly last time. should be actually installed and running now. | 22:12 |
| *** salv-orlando has quit IRC | 22:12 | |
| *** r-daneel has quit IRC | 22:12 | |
| corvus | clarkb: want to resend one of the github events? | 22:12 |
| clarkb | corvus: ya I can resend the last failed event | 22:12 |
| corvus | that did something in zuul-web | 22:13 |
| clarkb | was a 200 | 22:13 |
| clarkb | I think it is happy now | 22:13 |
| *** r-daneel has joined #openstack-infra | 22:13 | |
| clarkb | except for the events that don't ever fire in the first place | 22:13 |
| *** felipemonteiro has joined #openstack-infra | 22:14 | |
| mordred | yah. which we can't do a ton about right now | 22:14 |
| mordred | clarkb: want me to send a DNM PR to an openstack module in ansible to make sure that event is firing properly? | 22:14 |
| clarkb | ya I can make a change to one of the whitelisted files and make a pull request. May take a few minutes to sort out my fork of ansible and get it updated and send PR | 22:16 |
| mordred | clarkb: I can do it quickly | 22:16 |
| openstackgerrit | Merged openstack-infra/zuul master: Add cherrypy to built-in logging config https://review.openstack.org/572200 | 22:16 |
| openstackgerrit | Merged openstack-infra/zuul master: Stop scheduler more quickly https://review.openstack.org/572186 | 22:16 |
| mordred | clarkb: https://github.com/ansible/ansible/pull/41106 | 22:17 |
| clarkb | mordred: :P I actually set up my local clone properly to do updates against origin being the right thing | 22:17 |
| clarkb | that worked | 22:17 |
| mordred | \o/ | 22:17 |
| mordred | clarkb: I think most people would find my git config for my ansible/ansible clone to be confusing | 22:18 |
| clarkb | I know I don't use github repos the way you are supposed to | 22:18 |
| clarkb | I effectively have the non fork as origin, keep "master" in sync with that then push to my fork directly on new branches to do PRs | 22:19 |
| clarkb | and never keep my repo fork up to date | 22:19 |
| *** dave-mccowan has joined #openstack-infra | 22:21 | |
| mordred | clarkb: http://paste.openstack.org/show/722682/ | 22:21 |
| mordred | clarkb: that's basically what I do too | 22:21 |
| mordred | I just do it by setting pushurl to my fork | 22:21 |
| kmalloc | mordred, clarkb: paste.o.o seems to have died? | 22:22 |
| mordred | (this is also how all of my gerrit clones are set up) | 22:22 |
| kmalloc | isn't loading for a few of us. | 22:22 |
| kmalloc | at all. | 22:22 |
| mordred | kmalloc: I agree - it's not loading now for me either | 22:22 |
| kmalloc | yay! it's not just me | 22:22 |
| kmalloc | wait... not yay | 22:22 |
| kmalloc | =/ | 22:22 |
| *** felipemonteiro has quit IRC | 22:23 | |
| mordred | the server itself seems fine | 22:23 |
| *** VW has quit IRC | 22:23 | |
| *** VW has joined #openstack-infra | 22:23 | |
| clarkb | it eventually loaded for me | 22:24 |
| kmalloc | hey it loaded | 22:24 |
| kmalloc | eventually like 5m wait | 22:24 |
| mordred | weird | 22:25 |
| kmalloc | it's happy now | 22:25 |
| kmalloc | loading fast | 22:25 |
| kmalloc | DB backup? | 22:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Add playbook to upgrade puppet https://review.openstack.org/572218 | 22:25 |
| mordred | maybe so? | 22:25 |
| clarkb | we do backups at 0000UTC iirc so that probably isn't the problem | 22:25 |
| mordred | not sure why that would impact the 'make new paste' page | 22:25 |
| kmalloc | maybe some jerk was scranning pastes for E_EVIL reasons | 22:26 |
| kmalloc | anyway, seems happy now | 22:26 |
| kmalloc | *shrug* | 22:26 |
| * mordred shakes fists at world | 22:26 | |
| kmalloc | right?! | 22:26 |
| kmalloc | ;) | 22:26 |
| * kmalloc goes back to keystone WTFs... like the whole role_assignment api ... and cries | 22:26 | |
| clarkb | where are the logs for this service... | 22:27 |
| mordred | clarkb: /var/log/apache2 | 22:27 |
| mordred | are the best ones I could find | 22:27 |
| fungi | i have a feeling those paste.o.o delays have more to do with some sort of db connection timeout/problem | 22:28 |
| clarkb | but its proxying to the python service | 22:28 |
| mordred | fungi: me too | 22:28 |
| clarkb | looks like it may be set up to syslog? | 22:29 |
| *** kgiusti has left #openstack-infra | 22:29 | |
| *** mriedem has quit IRC | 22:30 | |
| *** lifeless has joined #openstack-infra | 22:31 | |
| *** lifeless_ has quit IRC | 22:31 | |
| fungi | how surprisingly... sane? | 22:32 |
| clarkb | well except ther eisn't anything in there eiterh | 22:32 |
| clarkb | just the GET requests which is good I guess | 22:33 |
| *** yamamoto has joined #openstack-infra | 22:34 | |
| *** jamesmcarthur has joined #openstack-infra | 22:37 | |
| *** yamamoto has quit IRC | 22:38 | |
| *** VW has quit IRC | 22:41 | |
| clarkb | corvus: are we good to give the release team the go ahead? | 22:42 |
| corvus | clarkb: yep, i think we're done touching zuul for the day | 22:44 |
| *** jamesmcarthur has quit IRC | 22:45 | |
| corvus | i'm going to eod | 22:47 |
| clarkb | I've approved the fix for sql and github in zuul-web | 22:47 |
| fungi | yep, lgtm | 22:48 |
| *** mriedem has joined #openstack-infra | 22:48 | |
| fungi | was mostly waiting until it was running in production, which seems to have done the trick | 22:48 |
| clarkb | I guess we should look into why things are looking up tripleo constantly | 22:48 |
| fungi | likely their tripleo dashboard collecting job results | 22:48 |
| clarkb | it is a red hat ip | 22:49 |
| fungi | sounds about right | 22:49 |
| mordred | clarkb: if you're in the mood to approve things: https://review.openstack.org/#/c/571240/ is working now | 22:50 |
| mordred | clarkb: (and I verified that it's installing the right version of things) | 22:50 |
| clarkb | actually before I do anything else thoughts on switching to verified ssl with our github app now? | 22:50 |
| clarkb | I think we just go for it | 22:50 |
| mordred | clarkb: yah. I say let's just go for it | 22:51 |
| mordred | we have ssl now - so nothing should break | 22:51 |
| clarkb | I did it | 22:52 |
| mordred | woot | 22:52 |
| clarkb | mordred: why were those jobs failing before and needing shade? | 22:54 |
| clarkb | was that just a tox thing? | 22:54 |
| mordred | well - it was them actually not grabbing the right branch | 22:54 |
| *** sshnaidm|brq has quit IRC | 22:55 | |
| mordred | clarkb: so it was running the sdk tests against the version of ansible that needs still needs shade directly - and thus sadness ensued | 22:55 |
| clarkb | ah | 22:55 |
| clarkb | there is at least one http 200 response from our zuul after telling github to verify ssl | 22:56 |
| mordred | \o/ | 22:56 |
| clarkb | do we care if red hat is asking for tripleo build updates every 20 seconds? | 22:58 |
| clarkb | it seems liek not a nice thing to do but zuul doesn't seem to be breaking as a result | 22:58 |
| *** jamesmcarthur has joined #openstack-infra | 22:58 | |
| *** hongbin has quit IRC | 22:59 | |
| mordred | honestly - I'd say if that *does* break zuul we should fix zuul | 22:59 |
| mordred | since that query should not be super costly ... however, we might also want to suggest mqtt to them once that's available | 23:00 |
| clarkb | ya but when you consider the jobs take 3 hours to run asking for newer status updates ever 20 seconds is silly | 23:00 |
| clarkb | mordred: we are running mqtt able code now. Just need to update config I think | 23:00 |
| mordred | yah | 23:00 |
| clarkb | memory looks good on zuul01 too so cherrypy hasn't drastically chagned that | 23:02 |
| mordred | \o/ | 23:03 |
| *** r-daneel has quit IRC | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:07 | |
| *** jamesmcarthur has joined #openstack-infra | 23:07 | |
| *** salv-orlando has joined #openstack-infra | 23:08 | |
| clarkb | following up on zuul-web needing to be restarted after restarting zuul-scheduler, looks like we have a 5 minute timeout on the gearman rpc client calls from zuul-web to scheduler | 23:10 |
| clarkb | I expect that if we wait for 5 minutes it may recover on its own | 23:10 |
| *** salv-orlando has quit IRC | 23:12 | |
| clarkb | and by default there is no max limit for threads that cherrypy can run | 23:14 |
| *** jamesmcarthur has quit IRC | 23:14 | |
| *** rwsu has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-infra | 23:17 | |
| openstackgerrit | Merged openstack-dev/pbr master: Revert "Remove win32/nt checks for wrapper script gen" https://review.openstack.org/571503 | 23:19 |
| *** caphrim007_ has joined #openstack-infra | 23:24 | |
| *** caphrim007_ has quit IRC | 23:24 | |
| *** esbergl__ has joined #openstack-infra | 23:27 | |
| *** caphrim007 has quit IRC | 23:28 | |
| *** tpsilva has quit IRC | 23:29 | |
| *** esberglu_ has quit IRC | 23:30 | |
| clarkb | mordred: can you review https://review.openstack.org/#/c/571263/ probably a bit late to babysit that one today, but can get to it tomorrow if you think it is ready | 23:40 |
| clarkb | AJaeger: when you get back your thoughts on https://review.openstack.org/#/c/571520/ woudl be much appreciated | 23:40 |
| *** mriedem has quit IRC | 23:48 | |
| *** masber has joined #openstack-infra | 23:49 | |
| *** AJaeger has quit IRC | 23:49 | |
| *** stakeda has joined #openstack-infra | 23:51 | |
| *** masuberu has quit IRC | 23:52 | |
| *** camunoz has quit IRC | 23:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Handle multiple stats in assertReportedStats https://review.openstack.org/572230 | 23:55 |
| *** jamesmcarthur has quit IRC | 23:58 | |
| *** rpioso is now known as rpioso|afk | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
