| *** jamesmcarthur has quit IRC | 00:12 | |
| *** jamesmcarthur has joined #openstack-infra | 00:12 | |
| *** zxiiro has quit IRC | 00:13 | |
| *** jamesmcarthur has quit IRC | 00:18 | |
| *** jamesmcarthur has joined #openstack-infra | 00:24 | |
| *** igordc has quit IRC | 00:25 | |
| *** igordc has joined #openstack-infra | 00:25 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: docs: remove generated toc from the main index https://review.opendev.org/703468 | 00:28 |
|---|---|---|
| *** igordc has quit IRC | 00:49 | |
| *** ricolin has joined #openstack-infra | 00:55 | |
| *** rlandy is now known as rlandy|bbl | 00:55 | |
| *** artom has quit IRC | 01:00 | |
| ianw | clarkb: LGTM, thanks! | 01:01 |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:04 | |
| *** jamesmcarthur has quit IRC | 01:06 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** yamamoto has joined #openstack-infra | 01:15 | |
| *** jistr has quit IRC | 01:17 | |
| *** jistr has joined #openstack-infra | 01:19 | |
| ricolin | fungi, can you help to create multi-arch-sig-core gerrit group, thanks!:) | 01:31 |
| *** lseki has quit IRC | 01:40 | |
| *** rfolco has quit IRC | 01:41 | |
| *** Lucas_Gray has quit IRC | 01:49 | |
| clarkb | ricolin: our tooling automatically creates new groups when they are added in acl files | 01:56 |
| clarkb | ricolin: that means you should update openstack/project-config/gerrit/acls as appropriate and the group will be created | 01:56 |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** jamesmcarthur has joined #openstack-infra | 01:57 | |
| ricolin | clarkb, so I guess I should ask to put me in that group right?:) (if this is correct config https://review.opendev.org/#/c/703323/3/gerrit/acls/openstack/multi-arch-sig.config ) | 01:59 |
| ricolin | https://review.opendev.org/#/admin/groups/2079,members | 02:02 |
| *** jamesmcarthur has quit IRC | 02:02 | |
| clarkb | ricolin: done | 02:03 |
| ricolin | clarkb, awesome! | 02:05 |
| ricolin | thank you | 02:05 |
| *** xinranwang has joined #openstack-infra | 02:06 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** jamesmcarthur has joined #openstack-infra | 02:24 | |
| *** goldyfruit has quit IRC | 02:30 | |
| *** roman_g has quit IRC | 02:34 | |
| *** ociuhandu has joined #openstack-infra | 02:47 | |
| *** ociuhandu has quit IRC | 02:51 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:05 | |
| *** dklyle has joined #openstack-infra | 03:07 | |
| *** apetrich has quit IRC | 03:10 | |
| *** jamesmcarthur has quit IRC | 03:11 | |
| *** yamamoto has joined #openstack-infra | 03:14 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove unused linaro credentials https://review.opendev.org/703534 | 03:16 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add Linaro US cloud https://review.opendev.org/703535 | 03:16 |
| *** jamesmcarthur has joined #openstack-infra | 03:30 | |
| *** jamesmcarthur has quit IRC | 03:33 | |
| *** hwoarang has quit IRC | 03:39 | |
| *** rlandy|bbl has quit IRC | 03:39 | |
| *** hwoarang has joined #openstack-infra | 03:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add Linaro US cloud https://review.opendev.org/703535 | 03:45 |
| *** hongbin has joined #openstack-infra | 04:03 | |
| *** tetsuro has quit IRC | 04:18 | |
| *** tetsuro has joined #openstack-infra | 04:19 | |
| *** tetsuro has quit IRC | 04:23 | |
| *** goldyfruit has joined #openstack-infra | 04:34 | |
| *** hongbin has quit IRC | 04:39 | |
| *** jamesmcarthur has joined #openstack-infra | 04:42 | |
| *** udesale has joined #openstack-infra | 04:44 | |
| *** udesale has quit IRC | 04:44 | |
| *** udesale has joined #openstack-infra | 04:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add Linaro US cloud https://review.opendev.org/703535 | 04:55 |
| *** tetsuro has joined #openstack-infra | 05:03 | |
| *** ykarel|away is now known as ykarel | 05:23 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** udesale_ has joined #openstack-infra | 05:34 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** udesale has quit IRC | 05:37 | |
| *** jamesmcarthur has joined #openstack-infra | 05:37 | |
| *** jamesmcarthur has quit IRC | 05:43 | |
| *** raukadah is now known as chandankumar | 05:45 | |
| openstackgerrit | Merged openstack/diskimage-builder master: dib-lint: test elements have README.rst file https://review.opendev.org/177832 | 05:50 |
| *** surpatil has joined #openstack-infra | 06:03 | |
| *** SurajPatil has joined #openstack-infra | 06:04 | |
| *** yolanda has quit IRC | 06:04 | |
| *** jamesmcarthur has joined #openstack-infra | 06:06 | |
| *** lpetrut has joined #openstack-infra | 06:08 | |
| *** lpetrut has quit IRC | 06:09 | |
| *** lpetrut has joined #openstack-infra | 06:10 | |
| *** jamesmcarthur has quit IRC | 06:13 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Handle event id in node requests https://review.opendev.org/703406 | 06:23 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Centralize logging adapters https://review.opendev.org/703407 | 06:23 |
| *** adriant has quit IRC | 06:40 | |
| *** adriant has joined #openstack-infra | 06:41 | |
| *** dchen has quit IRC | 06:43 | |
| *** dchen has joined #openstack-infra | 06:44 | |
| *** lpetrut has quit IRC | 06:49 | |
| AJaeger | config-core, please review https://review.opendev.org/698091 | 06:58 |
| *** icey has joined #openstack-infra | 07:04 | |
| *** lmiccini has joined #openstack-infra | 07:04 | |
| *** jamesmcarthur has joined #openstack-infra | 07:09 | |
| *** jamesmcarthur has quit IRC | 07:14 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Don't publish doctrees when building docs https://review.opendev.org/703544 | 07:24 |
| *** jtomasek has joined #openstack-infra | 07:24 | |
| *** yolanda has joined #openstack-infra | 07:25 | |
| *** roman_g has joined #openstack-infra | 07:30 | |
| *** ociuhandu has joined #openstack-infra | 07:30 | |
| *** yolanda has quit IRC | 07:33 | |
| *** yolanda has joined #openstack-infra | 07:34 | |
| *** ociuhandu has quit IRC | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:36 | |
| *** lpetrut has joined #openstack-infra | 07:38 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: fetch-sphinx: Exclude doctrees directory https://review.opendev.org/703547 | 07:39 |
| *** yolanda has quit IRC | 07:48 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Pass node request handler to launcher base class https://review.opendev.org/703549 | 07:50 |
| *** pgaxatte has joined #openstack-infra | 08:00 | |
| *** florianf has joined #openstack-infra | 08:02 | |
| *** slaweq has joined #openstack-infra | 08:03 | |
| *** bnemec has joined #openstack-infra | 08:06 | |
| *** jamesmcarthur has joined #openstack-infra | 08:10 | |
| *** tkajinam has quit IRC | 08:10 | |
| *** iurygregory has joined #openstack-infra | 08:11 | |
| *** jamesmcarthur has quit IRC | 08:14 | |
| *** lmiccini has quit IRC | 08:17 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| *** yolanda has joined #openstack-infra | 08:21 | |
| *** lmiccini has joined #openstack-infra | 08:25 | |
| *** priteau has joined #openstack-infra | 08:29 | |
| *** ralonsoh has joined #openstack-infra | 08:30 | |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| *** hashar has joined #openstack-infra | 08:40 | |
| openstackgerrit | Merged openstack/project-config master: IRC #openstack-ironic gerritbot CI failed messages https://review.opendev.org/698091 | 08:43 |
| *** rpittau|afk is now known as rpittau | 08:48 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** yamamoto has quit IRC | 08:53 | |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Annotate logs in launcher https://review.opendev.org/703558 | 08:57 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Annotate logs in node request handler https://review.opendev.org/703559 | 08:57 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Include event id in node request listings https://review.opendev.org/703560 | 08:57 |
| *** pkopec has joined #openstack-infra | 08:59 | |
| *** iurygregory has quit IRC | 08:59 | |
| *** florianf has quit IRC | 09:02 | |
| *** tosky has joined #openstack-infra | 09:07 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Annotate logs in zk module https://review.opendev.org/703561 | 09:10 |
| *** goldyfruit has quit IRC | 09:11 | |
| *** tommylikehu has joined #openstack-infra | 09:11 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 09:11 |
| *** iurygregory has joined #openstack-infra | 09:13 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** lucasagomes has joined #openstack-infra | 09:14 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 09:15 |
| *** yamamoto has quit IRC | 09:18 | |
| *** apetrich has joined #openstack-infra | 09:18 | |
| *** jaosorior has joined #openstack-infra | 09:19 | |
| *** gfidente has joined #openstack-infra | 09:23 | |
| *** xek has joined #openstack-infra | 09:24 | |
| *** xinranwang has quit IRC | 09:26 | |
| *** yolanda has quit IRC | 09:27 | |
| *** udesale_ has quit IRC | 09:28 | |
| *** SurajPatil has quit IRC | 09:28 | |
| *** udesale_ has joined #openstack-infra | 09:28 | |
| *** derekh has joined #openstack-infra | 09:28 | |
| *** SurajPatil has joined #openstack-infra | 09:28 | |
| *** SurajPatil has quit IRC | 09:29 | |
| *** surpatil has quit IRC | 09:29 | |
| *** surpatil has joined #openstack-infra | 09:30 | |
| *** yolanda has joined #openstack-infra | 09:33 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Docs: fix stestr run example https://review.opendev.org/703566 | 09:40 |
| *** yolanda has quit IRC | 09:49 | |
| *** jaosorior has quit IRC | 09:54 | |
| *** tetsuro has quit IRC | 09:56 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tox: pass --slowest to stestr https://review.opendev.org/703571 | 09:58 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Divide concurrent tests by classes https://review.opendev.org/703575 | 10:08 |
| *** ykarel is now known as ykarel|afk | 10:11 | |
| *** openstackgerrit has quit IRC | 10:12 | |
| *** priteau has quit IRC | 10:41 | |
| *** ykarel|afk is now known as ykarel | 10:43 | |
| *** goldyfruit has joined #openstack-infra | 10:47 | |
| *** udesale_ has quit IRC | 10:51 | |
| *** openstackgerrit has joined #openstack-infra | 11:10 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Add ironic jobs to the CI https://review.opendev.org/702474 | 11:10 |
| *** goldyfruit has quit IRC | 11:10 | |
| *** rpittau is now known as rpittau|bbl | 11:15 | |
| *** surpatil has quit IRC | 11:26 | |
| *** Wasaac has quit IRC | 11:32 | |
| *** Wasaac has joined #openstack-infra | 11:34 | |
| *** Lucas_Gray has joined #openstack-infra | 11:40 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Add ironic jobs to the CI https://review.opendev.org/702474 | 11:49 |
| *** rfolco has joined #openstack-infra | 12:00 | |
| *** dtantsur is now known as dtantsur|bbl | 12:01 | |
| AJaeger | stevebaker, infra-root, https://review.opendev.org/#/c/698091/ merged to update IRC notifications for ironic but I don't see the new notifations at http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/latest.log.html . Do we need to restart gerritbot? Or the change not working? | 12:06 |
| *** priteau has joined #openstack-infra | 12:06 | |
| *** aedc has quit IRC | 12:09 | |
| *** aedc has joined #openstack-infra | 12:09 | |
| *** yolanda has joined #openstack-infra | 12:10 | |
| *** yolanda has quit IRC | 12:10 | |
| *** yolanda has joined #openstack-infra | 12:11 | |
| *** Wasaac has quit IRC | 12:12 | |
| *** ociuhandu has joined #openstack-infra | 12:12 | |
| *** Wasaac has joined #openstack-infra | 12:12 | |
| *** ociuhandu has quit IRC | 12:13 | |
| *** Lucas_Gray has quit IRC | 12:13 | |
| *** tkajinam has joined #openstack-infra | 12:14 | |
| *** Lucas_Gray has joined #openstack-infra | 12:17 | |
| *** TomStappaerts has joined #openstack-infra | 12:18 | |
| *** aedc has quit IRC | 12:20 | |
| *** aedc has joined #openstack-infra | 12:20 | |
| *** jpena is now known as jpena|lunch | 12:21 | |
| *** artom has joined #openstack-infra | 12:24 | |
| TomStappaerts | Hi guys, seems like pip 20 is breaking some (if not all) of our CI jobs? | 12:26 |
| *** TomStappaerts has quit IRC | 12:27 | |
| *** TomStappaerts has joined #openstack-infra | 12:27 | |
| *** rcernin has quit IRC | 12:28 | |
| TomStappaerts | eg: https://81b633e2c5fe858f8400-d324a81a71d524d51ede3dc5aee27774.ssl.cf5.rackcdn.com/702831/4/check/networking-ovn-tempest-dsvm-ovs-release/c062094/ | 12:29 |
| tkajinam | TomStappaerts, this one ? https://github.com/pypa/pip/issues/7217 | 12:29 |
| TomStappaerts | yes | 12:30 |
| *** TomStappaerts has quit IRC | 12:35 | |
| *** TomStappaerts has joined #openstack-infra | 12:36 | |
| *** dpawlik has joined #openstack-infra | 12:46 | |
| *** ykarel is now known as ykarel|afk | 12:47 | |
| *** lmiccini has quit IRC | 12:48 | |
| *** nicolasbock has joined #openstack-infra | 12:50 | |
| *** ociuhandu has joined #openstack-infra | 12:52 | |
| frickler | AJaeger: was there a -2 event since it merged? also very likely gerritbot needs restarting, I can look into that in a bit | 12:54 |
| *** lseki has joined #openstack-infra | 12:57 | |
| *** udesale has joined #openstack-infra | 12:59 | |
| iurygregory | Hey infra team, anyone aware of problems in stable branches such as "ImportError: cannot import name 'SourceDistribution'" ? | 13:01 |
| *** ociuhandu has quit IRC | 13:01 | |
| tkajinam | iurygregory, I'm not so familiar with the infra stuffs, but just sent an e-mail to share that error on ml | 13:01 |
| yoctozepto | also TomStappaerts has seen this issue | 13:02 |
| tkajinam | iurygregory, so hopefully somebody will see it and set pin on pip to fix the issue... I hope | 13:02 |
| yoctozepto | https://pypi.org/project/pip/#history | 13:03 |
| iurygregory | tkajinam, tks, I'm trying to test locally to see if it's a problem on infra or not since we are getting a lot of FAILURE and POST_FAILURE in ironic CI | 13:03 |
| yoctozepto | they fixed 19 mintues ago | 13:03 |
| iurygregory | yoctozepto, tks! | 13:03 |
| iurygregory | so a recheck would work? | 13:03 |
| yoctozepto | looks like they b0rked something and quickly fixed | 13:03 |
| yoctozepto | iurygregory: if it's the same issue they just fixed | 13:03 |
| iurygregory | yoctozepto, ack I will try to trigger a recheck | 13:04 |
| yoctozepto | https://github.com/pypa/pip/commit/8f3687cfd9977039f953c9a6216fb62bbb6b4848 | 13:04 |
| yoctozepto | looks like it, iurygregory, tkajinam, TomStappaerts | 13:04 |
| *** priteau has quit IRC | 13:05 | |
| *** rpittau|bbl is now known as rpittau | 13:05 | |
| tkajinam | yoctozepto ahhh, thanks. | 13:06 |
| yoctozepto | tkajinam: btw, which ml did you mean? nothing appearing on os-discuss... | 13:06 |
| tkajinam | I meant openstack-discuss | 13:06 |
| *** lmiccini has joined #openstack-infra | 13:06 | |
| tkajinam | yoctozepto, ^^^ | 13:07 |
| *** rlandy has joined #openstack-infra | 13:07 | |
| yoctozepto | tkajinam: yeah, seeing now, thanks for confirming, must have been greylisted | 13:07 |
| iurygregory | there is an email on openstack-discuss just now =) | 13:08 |
| tkajinam | iurygregory, yoctozepto good to heat that. I've not sent e-mail on that list for a long time, so I was a little bit afraid I made something wrong :-P | 13:08 |
| tkajinam | and I know see that follow-up mail was sent telling that the they fixed the issue in pip | 13:10 |
| yoctozepto | I replied to close the topic | 13:10 |
| tkajinam | yoctozepto, thanks !! | 13:10 |
| yoctozepto | good old pip likes to break from time to time :-) | 13:10 |
| yoctozepto | but they have good response times, really | 13:11 |
| tkajinam | yeah it's surprisingly quick | 13:11 |
| tkajinam | they were so quick that they didn't update the issue info before releasing the fix :-) | 13:12 |
| yoctozepto | tkajinam: indeed! well, if you break half the internets you first fix the issue, then post about it :-) | 13:12 |
| tkajinam | yoctozepto, yeah, that is much appreciated behavior. | 13:13 |
| tkajinam | all we have to do is to check commit logs first :-) | 13:13 |
| tkajinam | by the way I found an interesting file in devstack while looking for the way to pin pip | 13:14 |
| yoctozepto | our natural habitat, wouldn't you say? | 13:14 |
| tkajinam | yoctozepto, definitely | 13:14 |
| tkajinam | https://github.com/openstack/devstack/blob/master/tools/cap-pip.txt | 13:14 |
| frickler | o.k., I just watched one job pass the location where pip 20.0.0 broke things, so looks like the 20.0.1 fix is working | 13:14 |
| yoctozepto | devstack is fun | 13:14 |
| tkajinam | frickler, good to hear that news | 13:15 |
| frickler | infra-root: broken stream for http://zuul.openstack.org/stream/11236c64bee34787a854c896a005642f?logfile=console.log , maybe someone has time to check the executors later | 13:15 |
| *** jamesmcarthur has joined #openstack-infra | 13:18 | |
| frickler | AJaeger: gerritbot was restarted at 10:12, well after the change merged | 13:18 |
| *** zbr|drover has quit IRC | 13:18 | |
| *** zbr has joined #openstack-infra | 13:19 | |
| *** jpena|lunch is now known as jpena | 13:23 | |
| AJaeger | frickler: thanks for confirming | 13:24 |
| AJaeger | frickler: I was looking for commented | 13:24 |
| frickler | AJaeger: I think the patch is wrong, the event type is "comment-added", not "comments-added". I also don't find any "x-vrif-*" events | 13:27 |
| *** whoami-rajat_ has joined #openstack-infra | 13:28 | |
| frickler | oh, that's in https://review.opendev.org/#/c/698089/2/gerritbot/bot.py , will need to crosscheck the logs again | 13:29 |
| frickler | iiuc it needs to have the 'comment-added' tag in order to trigger that code | 13:30 |
| frickler | I'm doing a follow-up patch | 13:31 |
| *** exsdev0 has joined #openstack-infra | 13:32 | |
| *** AJaeger has quit IRC | 13:33 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** exsdev has quit IRC | 13:33 | |
| *** exsdev0 is now known as exsdev | 13:33 | |
| *** dpawlik has quit IRC | 13:33 | |
| *** icey has quit IRC | 13:33 | |
| *** AJaeger has joined #openstack-infra | 13:34 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Fix use of 'comment-added' event type https://review.opendev.org/703614 | 13:34 |
| *** icey has joined #openstack-infra | 13:34 | |
| frickler | config-core: ^^ | 13:34 |
| *** iurygregory has quit IRC | 13:35 | |
| AJaeger | thanks, frickler ! | 13:38 |
| *** Lucas_Gray has quit IRC | 13:42 | |
| sshnaidm | is SourceDistribution error known? | 13:44 |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| frickler | sshnaidm: yes, pip 20.0.0 error. pip 20.0.1 was just released and fixes it | 13:45 |
| sshnaidm | frickler, thanks! | 13:45 |
| sshnaidm | zbr, ^^ | 13:45 |
| sshnaidm | weshay|ruck, ^^ | 13:46 |
| *** ociuhandu has joined #openstack-infra | 13:47 | |
| *** ociuhandu has quit IRC | 13:53 | |
| *** aaronsheffield has joined #openstack-infra | 13:54 | |
| *** dtantsur|bbl is now known as dtantsur | 14:01 | |
| *** iurygregory has joined #openstack-infra | 14:05 | |
| fungi | yoctozepto: tkajinam: we don't do greylisting on mailman, but posts to openstack-discuss take a few minutes to show up in folks inboxes just because of how many copies it needs to send out for its ~1300 subscribers | 14:15 |
| tkajinam | fungi, it makes sense. yeah I know we have many subscribes to the list. | 14:16 |
| AJaeger | fungi, could you review https://review.opendev.org/#/c/703614/, please? Quick IRC fix... | 14:17 |
| tkajinam | fungi, thanks for the info | 14:17 |
| openstackgerrit | Ilya Etingof proposed opendev/glean master: Fix a handful of bugs in config-drive processing https://review.opendev.org/703623 | 14:21 |
| *** yamamoto has joined #openstack-infra | 14:21 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed https://review.opendev.org/703624 | 14:25 |
| openstackgerrit | Merged openstack/project-config master: Fix use of 'comment-added' event type https://review.opendev.org/703614 | 14:29 |
| *** yamamoto has quit IRC | 14:36 | |
| *** dtroyer has joined #openstack-infra | 14:38 | |
| *** kjackal has joined #openstack-infra | 14:46 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed https://review.opendev.org/703631 | 14:46 |
| zbr | clarkb: we are going to disable gzipping of logs on https://review.opendev.org/#/c/702862/ -- are you sure that this will not impact the log-servers? | 14:46 |
| *** TomStappaerts has quit IRC | 14:56 | |
| *** TomStappaerts has joined #openstack-infra | 14:57 | |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| *** yamamoto has quit IRC | 14:59 | |
| fungi | zbr: they are compressed on upload to swift | 14:59 |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| zbr | fungi: mainly server side compression is tansparent, so we should not care. | 15:00 |
| fungi | zbr: the only place i can think they might pose a problem is if they exceed the available transfer space on the executor during log collection | 15:00 |
| fungi | which is something i keep wondering about with this plan | 15:00 |
| zbr | fungi: we will find out, we can implement some truncation if needed. | 15:01 |
| fungi | then again, if there are super large logfiles, precompressing them may make sense because you're not going to view them with a browser anyway (if you're sane) | 15:01 |
| zbr | fungi: i agree. once we sport the first issue, i will try to implement extra logic. | 15:02 |
| zbr | until then, we should be ok, I am not aware of normal huge files. | 15:02 |
| *** yamamoto has quit IRC | 15:04 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tox: reduce deps used for pep8 env https://review.opendev.org/703634 | 15:05 |
| *** kjackal has quit IRC | 15:06 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tox: rename pep8 to linters https://review.opendev.org/703635 | 15:13 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tox: do not install bindep for linters https://review.opendev.org/703636 | 15:13 |
| *** tkajinam has quit IRC | 15:16 | |
| *** electrofelix has joined #openstack-infra | 15:16 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed https://review.opendev.org/703624 | 15:17 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add networking.k8s.io apiGroups rbac for service account https://review.opendev.org/703637 | 15:17 |
| *** jamesmcarthur has quit IRC | 15:25 | |
| *** kjackal has joined #openstack-infra | 15:30 | |
| *** ociuhandu has joined #openstack-infra | 15:30 | |
| *** electrofelix has quit IRC | 15:31 | |
| noonedeadpunk | hey everyone:) | 15:32 |
| noonedeadpunk | how can I ask for a job hold to look into what's happening in vm? For instance for that build? https://zuul.opendev.org/t/openstack/build/021f80901599483ab2e28c977506b677 | 15:33 |
| *** jamesmcarthur has joined #openstack-infra | 15:35 | |
| *** electrofelix has joined #openstack-infra | 15:36 | |
| *** electrofelix has quit IRC | 15:36 | |
| *** electrofelix has joined #openstack-infra | 15:36 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add spec for enhanced regional executor distribution https://review.opendev.org/663413 | 15:38 |
| fungi | noonedeadpunk: looks like that job was successful... what's the issue you're investigating? | 15:39 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 15:39 |
| AJaeger | noonedeadpunk: we can only hold a job that fails... | 15:39 |
| noonedeadpunk | Ah, I see. | 15:39 |
| noonedeadpunk | Like it's just missing gathered data, so I was thinking I would be able to get it that way... | 15:40 |
| fungi | yep, i just checked the output of `zuul autohold --help` and we don't seem to have an option for holding on success or on non-failure results | 15:40 |
| noonedeadpunk | actually then https://zuul.opendev.org/t/openstack/build/76dc975e3ea74ac3a2a8ff1d791f1633 would be helpful as well | 15:40 |
| noonedeadpunk | it what I'm actually investigatig :p | 15:41 |
| *** Lucas_Gray has joined #openstack-infra | 15:42 | |
| fungi | remember that you can of course usually extend the job to run whatever commands you're hoping to run manually to poke around on the filesystem | 15:42 |
| jrosser | following up the neutron-lib adventure from yesterday, will the existing wrong .py2.py3 wheels persist in the wheel cache? | 15:44 |
| fungi | noonedeadpunk: zuul autohold --tenant openstack --project openstack/openstack-ansible-os_manila --job openstack-ansible-deploy-aio_metal-ubuntu-bionic --change 675934 --reason "noonedeadpunk investigating btrfs-related issues for lxc" --count 1 | 15:45 |
| fungi | does that capture what you're looking for? | 15:45 |
| *** lpetrut has quit IRC | 15:46 | |
| fungi | jrosser: if https://review.opendev.org/703487 works (i haven't checked a recent build log yet but will shortly) then we can safely delete that wheel from our cache and it shouldn't reappear | 15:46 |
| fungi | noonedeadpunk: i've set that autohold if you want to recheck change 675934 | 15:47 |
| fungi | noonedeadpunk: if i should make adjustments to the autohold to better capture what you're looking for, let me know | 15:48 |
| *** chandankumar is now known as raukadah | 15:48 | |
| noonedeadpunk | Oh, I've started searching for the way I should launch that :p | 15:48 |
| clarkb | fungi: re precompressing making sense for large files, that is why we xz the serialized journal file. its like half a gig uncompressed and you have to pass it through journald anyway to view it so may as well compress it down to like 30MB | 15:48 |
| fungi | noonedeadpunk: once that job fails again i can ssh into the node and add your public ssh key for root access | 15:49 |
| noonedeadpunk | fungi: oh, thanks! It's just the first time I'm asking for the hold, so wasn't sure how it works:) | 15:49 |
| noonedeadpunk | thanks for explaining | 15:50 |
| fungi | yw | 15:50 |
| *** ociuhandu has quit IRC | 15:51 | |
| AJaeger | config-core, infra-root, I created a change for system-config to not publish doctrees ( https://review.opendev.org/703544 ) but then thought, let's fix it for every job with https://review.opendev.org/703547 in zuul-jobs. What do you think? | 15:52 |
| clarkb | AJaeger: ++ to fixing globally | 15:53 |
| AJaeger | ;) | 15:54 |
| corvus | what's a .doctree dir? | 15:54 |
| corvus | https://stackoverflow.com/questions/33904042/is-doctrees-folder-required-for-displaying-html-docs-with-sphinx | 15:55 |
| *** zxiiro has joined #openstack-infra | 15:57 | |
| *** jtomasek has quit IRC | 15:58 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 15:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed https://review.opendev.org/703624 | 15:58 |
| *** udesale has quit IRC | 15:59 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Enable E741 flake8 check https://review.opendev.org/703650 | 16:00 |
| corvus | Shrews: can we just make that change without enabling that check? | 16:01 |
| corvus | i think we can make a judgement call that "for x in list" is fine. | 16:01 |
| *** iurygregory has quit IRC | 16:01 | |
| *** hashar has quit IRC | 16:02 | |
| *** auristor has joined #openstack-infra | 16:04 | |
| fungi | technically e741 doesn't mind variables named "x" | 16:08 |
| corvus | oh, is this the "my font doesn't show the difference between l and 1" check? | 16:08 |
| fungi | it specifically cares that variables are not named l, I or O due to typographical similarities with digits 1 and 0 | 16:09 |
| fungi | i agree the check is of questionable value, but its reach is fairly limited | 16:09 |
| corvus | sorry i misremembered the check | 16:09 |
| Shrews | it's not an important change. we can live without it | 16:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-sphinx: Exclude doctrees directory https://review.opendev.org/703547 | 16:12 |
| *** jackedin has joined #openstack-infra | 16:13 | |
| *** openstackgerrit has quit IRC | 16:13 | |
| *** openstackgerrit has joined #openstack-infra | 16:14 | |
| openstackgerrit | Clément Mondion proposed zuul/nodepool master: add tags support for aws provider https://review.opendev.org/703651 | 16:14 |
| jrosser | ive just had two jobs fail with unable to get to opendev git repos "fatal: unable to access 'https://opendev.org/openstack/cinder/': Encountered end of file" and fatal: unable to access 'https://opendev.org/openstack/neutron/': GnuTLS recv error (-110): The TLS connection was non-properly terminated." | 16:18 |
| jrosser | both on here https://review.opendev.org/703389 | 16:19 |
| clarkb | jrosser: it helps if you can link to the job logs | 16:19 |
| jrosser | https://7727e75b0735fce8b288-3578f4b3c7df6e8f4dbcf87a4a72da28.ssl.cf1.rackcdn.com/703389/1/check/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/0eedacd/logs/ara-report/result/0adb4a8b-f1fd-4c28-8697-e9bc44161742/ | 16:20 |
| *** ykarel|afk is now known as ykarel | 16:20 | |
| jrosser | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_530/703389/1/check/openstack-ansible-deploy-aio_lxc-centos-7/530721f/logs/ara-report/result/80f168c2-e363-4a54-8b99-6fd96f2175df/ | 16:21 |
| clarkb | as a first pass sanity check the http top level page for neutron and cinder loads for me from all 8 backends | 16:22 |
| jrosser | ok, normally i'd just recheck but two together like that looks suspicious | 16:22 |
| clarkb | jrosser: out of curiousity why are those jobs cloning the repos and not using the zuul supplied repos? | 16:22 |
| *** ociuhandu has joined #openstack-infra | 16:23 | |
| clarkb | going to try git clones against all backends now | 16:23 |
| *** jtomasek has joined #openstack-infra | 16:26 | |
| jrosser | clarkb: i don't have a better answer than "it'd be complicated", it's been like this forever so there must be some justification | 16:26 |
| clarkb | ok, we go out of our way to ensure that our CI jobs don't become a self inflicted DDoS | 16:27 |
| jrosser | because the repos of osa itself are definately picked up from the zuul supplied ones | 16:27 |
| clarkb | jobs should ideally only clone direclty if they are testing that those clones work | 16:27 |
| *** ociuhandu has quit IRC | 16:28 | |
| clarkb | I'm still digging around to see if there is a smoking gun for this, but we recognize this as a faulty configuration and have alternatives in place as a result | 16:28 |
| *** lmiccini has quit IRC | 16:30 | |
| clarkb | gitea04 appears to have been hit erally hard during that period and ran out of memory, swapped, and had high load average as a result | 16:30 |
| noonedeadpunk | fungi: it has failed, but not where I was expecting... | 16:31 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66699&rra_id=all | 16:31 |
| clarkb | I expect that is the cause of the problem | 16:31 |
| noonedeadpunk | but it might be enough actually | 16:31 |
| jrosser | clarkb: i will try to find out why it is like this | 16:32 |
| fungi | noonedeadpunk: okay, where can i get a copy of your public ssh key? | 16:32 |
| noonedeadpunk | https://launchpad.net/~noonedeadpunk/+sshkeys | 16:33 |
| *** tosky has quit IRC | 16:33 | |
| fungi | noonedeadpunk: ssh root@198.72.124.91 | 16:34 |
| noonedeadpunk | fungi: thanks! | 16:34 |
| *** eharney has quit IRC | 16:35 | |
| openstackgerrit | Clément Mondion proposed zuul/nodepool master: add tags support for aws provider https://review.opendev.org/703651 | 16:35 |
| fungi | clarkb: oh, yeah, check out the swap graph | 16:35 |
| fungi | [Tue Jan 21 15:00:50 2020] Out of memory: Kill process 11821 (gitea) score 817 or sacrifice child | 16:36 |
| *** ociuhandu has joined #openstack-infra | 16:36 | |
| fungi | i guess the health checks we added to haproxy aren't sophisticated enough to catch that | 16:37 |
| clarkb | fungi: ya I'm looking at haproxy logs for connections to gitea04 between 1300UTC and 1600UTC to see if I can make sense of what may have caused it | 16:37 |
| clarkb | fungi: I think they are, but any existing connections would be lost | 16:37 |
| fungi | ahh, okay, so maybe impact was somewhat limited | 16:37 |
| *** gyee has joined #openstack-infra | 16:39 | |
| zbr | is by design or bug that .sh files are downloaded while .bash ones are loaded as text? (logs) | 16:40 |
| *** iurygregory has joined #openstack-infra | 16:40 | |
| clarkb | zbr: we use a python mimetype lib to figure that out iirc | 16:41 |
| clarkb | zbr: possibly a bug in that tool | 16:41 |
| *** mattw4 has joined #openstack-infra | 16:41 | |
| clarkb | where it decides .bash is a text type and .sh isn't? | 16:41 |
| zbr | clarkb: thanks, i will look into, i guess everyone wants to be able to see them without downloading them. | 16:42 |
| weshay|ruck | zbr, k.. so we're fixing here? cool | 16:43 |
| *** jpena is now known as jpena|brb | 16:46 | |
| *** bnemec has quit IRC | 16:48 | |
| *** tommylikehu has quit IRC | 16:48 | |
| fungi | infra-root: if anyone has a moment to approve a new ml request from last week for lists.opendev.org, these folks are hoping to start using it soon to plan an upcoming collaboration at the mass open cloud workshop: https://review.opendev.org/703145 | 16:51 |
| clarkb | fungi: I've +2'd but not approved in case we can get another non staffer to review, but probably fine to proceed | 16:52 |
| corvus | i wondere where we are on our MOC credentials | 16:56 |
| clarkb | there are a lot of huawei IPs hitting gitea04 | 16:56 |
| * clarkb tries to finish cleaning up this data so it is easier to understand | 16:57 | |
| *** kjackal has quit IRC | 16:58 | |
| fungi | mordred: if you're around, have you heard any more about moc creds for nodepool? | 16:59 |
| fungi | you're probably asleep right now though | 17:00 |
| *** tesseract has quit IRC | 17:01 | |
| *** rpittau is now known as rpittau|afk | 17:04 | |
| *** ricolin has quit IRC | 17:04 | |
| *** pgaxatte has quit IRC | 17:04 | |
| *** lucasagomes has quit IRC | 17:05 | |
| *** ricolin has joined #openstack-infra | 17:05 | |
| clarkb | huawai was almost half the total connections during that ~hour period where gitea04 was swapping | 17:10 |
| clarkb | red hat nat has most connections for a single IP | 17:11 |
| clarkb | I expect that the huawei IPs are a CI system (maybe openlab?) they are all within like the same /22 | 17:12 |
| clarkb | anyone know who to talk to at openlab now? smcginnis maybe? | 17:12 |
| clarkb | it would be great if other CI systems didn't DDoS us in addition to our efforts to rpevent our CI system from DDoSing us | 17:12 |
| clarkb | jrosser: ^ fyi that is my quick read of the haproxy logs | 17:12 |
| jrosser | clarkb: we just had a loooong discussion about this in #openstack-ansible | 17:13 |
| jrosser | and its a sort of mashup of "history" and "it needs to behave like it would for end users" | 17:14 |
| *** jtomasek has quit IRC | 17:14 | |
| clarkb | jrosser: the way we address "behave like it would for end users" is to have a pre step put the git repos in place, then the job/tool/whatever only clones if that repo isn't alredy there | 17:14 |
| clarkb | jrosser: this should work for CI and not CI | 17:14 |
| jrosser | but thats not to say we can't look at moving to the zuul cloned repos, but i'm a bit wary of never testing the code path that folk in the wild would use | 17:14 |
| clarkb | jrosser: also that prevents you from doing cross project testing | 17:15 |
| clarkb | which is a very powerful tool | 17:15 |
| fungi | the concern is that you would cease testing whether git breaks its ability to do a git clone? | 17:15 |
| fungi | i hope the git maintainers test their code | 17:15 |
| fungi | and don't rely on us to test that for them | 17:16 |
| *** rfolco is now known as rfolco|brb | 17:16 | |
| clarkb | other advantages include being able to run mutliple times without incurring clone costs (or failing beacuse code is already there). Also users can preset up their git repos this way as well if they know they want a specific version of something | 17:16 |
| fungi | yeah, they gain the ability to set up wiregapped environments that way | 17:17 |
| clarkb | I need to find breakfast but as far as addressing the OOM I think we either A) make bigger gitea nodes and/or B) ask huawei (maybe it is openlab) to use cached repos | 17:17 |
| fungi | er, airgapped i mean | 17:17 |
| fungi | yeah, i guess since there's no separate webserver in front of gitea on the same host, we don't have the ability to perform resource management throttles | 17:18 |
| clarkb | oh there is a C) try the least conns lb method again | 17:18 |
| fungi | i'm wary of least connections until we have shared backend clustering working | 17:19 |
| clarkb | ya it will probably make some clients unhappy again if the pack files get out of sync | 17:19 |
| clarkb | (I think pack vs object was causing the problems before because if you think the file is in a pack or an object then request the other you'll fail?) | 17:20 |
| fungi | even if packfiles don't get out of sync, clients are racing replication events from gerrit | 17:20 |
| openstackgerrit | Merged opendev/system-config master: Add mailing list for OpenInfra Labs https://review.opendev.org/703145 | 17:20 |
| fungi | invariably, some fetches will go to backends which don't have those refs yet | 17:20 |
| clarkb | thats a good point | 17:21 |
| fungi | gerrit doesn't provide any guarantees that the same refs are replicated to the same destinations at the exact same times | 17:22 |
| smcginnis | clarkb: That's probably Huawei proper and not OpenLab. OpenLab (at least was) spread out across different providers. | 17:22 |
| smcginnis | Not sure who would be a good contact there now. | 17:23 |
| smcginnis | Maybe mnaser is still working with them on some things? | 17:23 |
| smcginnis | Most of the team was still all Huawei employees spread out between India and China. | 17:23 |
| openstackgerrit | Clément Mondion proposed zuul/nodepool master: add tags support for aws provider https://review.opendev.org/703651 | 17:23 |
| fungi | clarkb: what about request rate limiting or bandwidth throttles per ip address in haproxy? | 17:24 |
| fungi | we might be able to tune it so that only addresses which are overusing the git farm get poor performance | 17:25 |
| *** jpena|brb is now known as jpena | 17:26 | |
| AJaeger | ianw: I think you have some packaging background as well, could you review https://review.opendev.org/#/c/703495, please? | 17:26 |
| fungi | though i know bandwidth utilization and request frequency don't directly map to memory utilization on the backend (that depends a lot on the request type) | 17:26 |
| clarkb | fungi: that might work butmay make it worse since the impact is memory by git operations. Completing those as quickly as possible is best | 17:26 |
| *** jtomasek has joined #openstack-infra | 17:26 | |
| clarkb | slowing them down will only increase memory demand I think | 17:26 |
| fungi | that's a great point | 17:26 |
| fungi | sounds more like a feature request for gitea itself: client rate limits or some sort of resource management so it can start rejecting connections when it's overloaded | 17:27 |
| *** gfidente is now known as gfidente|dinner | 17:29 | |
| fungi | AJaeger: specifically, anyone with background on how setuptools/pip shells out to gcc for compiling python extensions would be a huge help. i wasn't having a lot of luck tracking down where/how that happens | 17:31 |
| fungi | ianw: ^ | 17:31 |
| clarkb | smcginnis: thanks for the info. Maybe we can reach out via our board member there if this persists | 17:32 |
| AJaeger | fungi: yeah, couldn't find it either. | 17:32 |
| clarkb | I think its up to the package? | 17:34 |
| *** evrardjp has quit IRC | 17:34 | |
| clarkb | it has been a long time since I fiddled with C linked packages though | 17:34 |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| fungi | yeah, i was looking at pyyaml as my initial example and couldn't track down how "pip install pyyaml" or more specifically "pip wheel pyyaml" (which should be the same codepath for that part) winds up compiling the libyaml extension .so file | 17:36 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tox: do not install bindep for linters https://review.opendev.org/703636 | 17:36 |
| openstackgerrit | Merged zuul/zuul master: Docs: fix stestr run example https://review.opendev.org/703566 | 17:41 |
| AJaeger | config-core, do we want all openstack-tox-p36/37 jobs to increase timeout to 1h, or ask neutron team to do that in-repo? See https://review.opendev.org/703386 | 17:42 |
| AJaeger | it's 40 minutes now - and they run sometimes in timeouts | 17:43 |
| clarkb | I think I'm ok with a global setting but I havent checked runtimes of jobs to seehow close others are to that limit. I expect nova is close too | 17:44 |
| *** TomStappaerts has quit IRC | 17:44 | |
| *** eharney has joined #openstack-infra | 17:46 | |
| AJaeger | nova has ~15 mins | 17:46 |
| AJaeger | https://review.opendev.org/#/c/697153/ | 17:46 |
| *** ykarel is now known as ykarel|away | 17:47 | |
| clarkb | oh wow that is quicker than I though | 17:48 |
| AJaeger | yep - looking at http://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py37 now... | 17:49 |
| AJaeger | cyborg timed out | 17:49 |
| AJaeger | cyborg, neutron, keystonemiddleware is what I see | 17:50 |
| *** iurygregory has quit IRC | 17:51 | |
| *** harlowja has quit IRC | 17:51 | |
| *** harlowja has joined #openstack-infra | 17:52 | |
| *** yolanda has quit IRC | 17:52 | |
| *** yolanda has joined #openstack-infra | 17:54 | |
| fungi | i wonder if nova's were sped up by the mox to mock transition | 17:59 |
| fungi | they used to run a lot longer | 17:59 |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** roman_g has quit IRC | 18:01 | |
| *** TomStappaerts has joined #openstack-infra | 18:02 | |
| fungi | WOAH, i wasn't getting a response from lists.o.o for longer than seemed healthy. i finally managed to ssh in and its load average is over 100 | 18:08 |
| fungi | there may be another subscription spamming event underway | 18:08 |
| fungi | digging into it now | 18:08 |
| clarkb | fungi: thank you. Let me know if I can help | 18:09 |
| clarkb | otherwise I'm going to catch up on the governance change reviews now | 18:09 |
| fungi | though there were tons of www-data owned python processes so it might be mass crawling of pipermail archives on it | 18:09 |
| *** dtantsur is now known as dtantsur|afk | 18:11 | |
| *** jtomasek has quit IRC | 18:13 | |
| *** ociuhandu_ has joined #openstack-infra | 18:13 | |
| *** TomStappaerts has quit IRC | 18:14 | |
| *** Lucas_Gray has quit IRC | 18:15 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update project doc to reflect OpenDev changes https://review.opendev.org/703488 | 18:15 |
| clarkb | frickler: ^ thank you for the reviews | 18:15 |
| *** ociuhandu has quit IRC | 18:17 | |
| fungi | no sign of mass subscription activity | 18:17 |
| *** ociuhandu_ has quit IRC | 18:18 | |
| fungi | there was a spike in web requests to the openstack.org vhost but that started after the load average cleared up | 18:18 |
| fungi | also i think our ansible wheel may be getting bogged down by something | 18:19 |
| fungi | the last entry in /var/log/ansible/run_all_cron.log was 17:01:45z | 18:19 |
| fungi | so over an hour ago | 18:20 |
| fungi | looks like it's been trying to ssh to the ipv6 address corresponding to zm06.openstack.org for all that time | 18:21 |
| fungi | i'll see what's up (or down rather) with zm06 | 18:21 |
| clarkb | ianw: small thing on https://review.opendev.org/#/c/703535/3 otherwise that looks good to go | 18:21 |
| fungi | zm06 responds to icmp ping but not ssh | 18:22 |
| fungi | might be something nasty has happened to its rootfs. will try to check out the oob console for any i/o errors from the kerneol | 18:22 |
| fungi | INFO: task jbd2/xvda1-8:309 blocked for more than 120 seconds. | 18:25 |
| fungi | looks like the rootfs to me | 18:26 |
| fungi | i'll try to force reboot it | 18:26 |
| *** slaweq_ has joined #openstack-infra | 18:26 | |
| *** slaweq has quit IRC | 18:27 | |
| fungi | looks like the last time cacti was able to get a response from zm06 was 2020-01-16:05:00z | 18:28 |
| fungi | so ~5.5 days ago | 18:28 |
| clarkb | I think that is what we see from live migrations | 18:28 |
| fungi | #status log performed a hard reboot of zm06 after it lost the use of its rootfs (likely 2020-01-16:05:00z per gap in cacti graphs) | 18:29 |
| openstackstatus | fungi: finished logging | 18:29 |
| fungi | it's up now | 18:29 |
| *** ralonsoh has quit IRC | 18:31 | |
| fungi | zuul-merger service won't start either... lockfile.LockFailed: failed to create /var/run/zuul/merger.pid | 18:32 |
| clarkb | probably leaked that file | 18:32 |
| fungi | there's a /var/run/zuul-merger directory but no /var/run/zuul directory | 18:32 |
| clarkb | hrm | 18:33 |
| fungi | did we change initscripts recently? | 18:33 |
| fungi | or puppetry? | 18:33 |
| fungi | looks like zm05 has both | 18:33 |
| clarkb | not that I was aware of. I seem to recall /var/run/zuul/merger.pid being the correct location. We also restarted zuul semi recnetly and owuld've expected that to fail | 18:33 |
| *** electrofelix has quit IRC | 18:34 | |
| clarkb | /var/run is not persistent fs though iirc | 18:34 |
| clarkb | possible you need a puppet pulse to create that dir then it can start the service | 18:34 |
| *** jpena is now known as jpena|off | 18:34 | |
| fungi | on zm05 /var/run/zuul-merger was modified 2019-01-09 and /var/run/zuul was last modified 2019-01-14 | 18:34 |
| fungi | so we must have switched which directory we're using between a restart on the 9th and a restart on the 14th? | 18:35 |
| clarkb | semi related, zuul schedulers memory has been stable since we reverted that change | 18:35 |
| clarkb | corvus: ^ we should probably consider reverting it soon if this holds up? | 18:35 |
| clarkb | fungi: no this change was a long time ago iirc | 18:35 |
| clarkb | run/zuul-merger was the old location then we consolidated everything in run/zuul/ | 18:36 |
| clarkb | but run/zuul is wiped on every reboot so we have to wait for puppet to run to write it down iirc | 18:36 |
| fungi | clarkb: well, what's odd is that zm06 has a /var/run/zuul-merger created/modified at the time i tried to start the zuul-merger service but then it tries to lock a pidfile in /var/run/zuul which dne | 18:36 |
| clarkb | we might want to have the init script write the dir if not there as an alternative | 18:36 |
| clarkb | fungi: ya I think the init script wasn't updated to create the proper dirwhen they changed a while back | 18:37 |
| clarkb | puppet was though | 18:37 |
| clarkb | ya we need to update the PIDFILE arg of the init script | 18:38 |
| clarkb | looks like the other services need similar help | 18:39 |
| *** ramishra has quit IRC | 18:41 | |
| corvus | clarkb: ack, i'll propose the revert, thx | 18:50 |
| fungi | i'll work on the pidfile handling fix for the initscript now | 18:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Extract an abstract base Parser class" https://review.opendev.org/703669 | 18:52 |
| fungi | infra-root: i manually killed the hung ssh process on bridge.o.o which was trying to reach the (since rebooted) zm06, and ansible has continued past it now | 18:55 |
| frickler | FYI if someone comes along with weird pip failures in devstack master, this got merged half an hour ago and might be the cause openstack/devstack master: Revert "Do not use pip 10 or higher" https://review.opendev.org/561597 | 19:09 |
| clarkb | frickler: thank you for the heads up | 19:09 |
| fungi | nice!!! | 19:12 |
| fungi | that's been a long time coming | 19:12 |
| *** dustinc|PTO is now known as dustinc | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| stevebaker | frickler, AJaeger: thanks for the comment-added followup | 19:25 |
| *** tosky has joined #openstack-infra | 19:25 | |
| *** jackedin has quit IRC | 19:27 | |
| *** iurygregory has joined #openstack-infra | 19:33 | |
| openstackgerrit | Merged opendev/storyboard-webclient master: Remove unused imagemin build step https://review.opendev.org/691050 | 19:34 |
| iurygregory | Hello Infra o/ is there any know issues with openstack-tox-* jobs? in ironic I noticed a lot of FAILURE / RETRY_LIMIT for this jobs in in master and stable branches | 19:35 |
| clarkb | iurygregory: I am not aware of any known issues. Examples (links to logs) can be helpful | 19:37 |
| fungi | iurygregory: were they from much earlier today? there was a broken pip release for a few hours | 19:37 |
| clarkb | oh huh I missed that but that would cause retry limits if pip fails early in the job | 19:37 |
| fungi | pip 20.0.0 is bad, 20.0.1 solved it | 19:37 |
| frickler | iurygregory: a couple of hours there was an issue caused by pip 20.0.0, which should now be fixed by 20.0.1, some error with SourceDistribution? | 19:37 |
| iurygregory | https://review.opendev.org/703381 https://review.opendev.org/703380 | 19:37 |
| openstackgerrit | Merged opendev/storyboard-webclient master: Update selenium-standalone and gifsicle https://review.opendev.org/691051 | 19:37 |
| openstackgerrit | Merged opendev/storyboard-webclient master: Reinstate "Add transpiling as a step in the build process" https://review.opendev.org/691477 | 19:37 |
| iurygregory | I got this one earlier but now seems to bee different | 19:37 |
| fungi | iurygregory: some of us can probably dig deeper on that when the infra meeting finishes | 19:38 |
| iurygregory | fungi, tks, i will try to keep the irc open in my notebook o/ | 19:39 |
| frickler | looks like some other pip issue https://zuul.opendev.org/t/openstack/build/5a6e757cd7024b5cba39f0bf64f77efe/console#2/0/0/ubuntu-bionic | 19:41 |
| *** gfidente|dinner is now known as gfidente | 19:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add Linaro US cloud https://review.opendev.org/703535 | 19:44 |
| ianw | clarkb: ^ also found another typo ... i feel sure we could more programatically create some of these files | 19:45 |
| ianw | probably falls into the cost-of-automating v new clouds actually added equation though | 19:45 |
| clarkb | ianw: if the clouds run horizon I think it will output a clouds.yaml to you | 19:46 |
| *** jamesmcarthur has quit IRC | 19:46 | |
| prometheanfire | looks like requirements bot is failing, still with the tempest issues? | 19:46 |
| fungi | frickler: that looks suspiciously like something which just changed in the zuul-jobs ensure-pip role | 19:48 |
| fungi | frickler: zbr: https://review.opendev.org/702978 merged 4 days ago which added that logic | 19:50 |
| *** jamesmcarthur has joined #openstack-infra | 19:50 | |
| openstackgerrit | Sorin Sbarnea proposed opendev/gear master: packaging: updated project urls https://review.opendev.org/703422 | 19:50 |
| fungi | ensure-tox not ensure-pip i mean | 19:50 |
| fungi | no stdout nor stderr but returns an exit code of 141 | 19:51 |
| zbr | interesting error | 19:53 |
| zbr | afaik that was command not found or something similar | 19:53 |
| zbr | is related to pipefail https://stackoverflow.com/questions/22464786/ignoring-bash-pipefail-for-error-code-141 | 19:54 |
| *** smarcet has joined #openstack-infra | 19:55 | |
| *** nicolasbock has quit IRC | 19:57 | |
| zbr | is there any chance that /bin/bash not being real bash? | 19:58 |
| ianw | no, if it's explicitly called like that it's bash | 19:58 |
| zbr | tbh, is would be possible to avoid the pipefail if we want, it not really the most complex pieace of bash I seen. | 19:58 |
| zbr | what is interesting is that I am unable to replicate the failure manually | 19:59 |
| fungi | any idea if it's happening reproducibly or just intermittently? | 19:59 |
| fungi | yeah, i tried the same code locally in an interactive shell and wasn't seeing any issues with it | 19:59 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul-jobs master: ensure-tox: Output tox version https://review.opendev.org/701236 | 20:00 |
| *** jamesmcarthur has quit IRC | 20:01 | |
| zbr | i would remove pipefail, add -x, just in case. | 20:01 |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| zbr | but i confess, is frustrating to look at this task output and to guess what could have went wrong. | 20:03 |
| zbr | it seems like an one-off, based on 703381 | 20:03 |
| zbr | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-functional | 20:03 |
| zbr | i think i remember, that happened when ssh connection dropped, mainly the error indicates that one of std??? stream was closed prematurely | 20:05 |
| zbr | i am sure i seen the same error before | 20:05 |
| ianw | it doesn't seem like "command -v" could return anything other than 0/1 | 20:06 |
| *** hashar has joined #openstack-infra | 20:07 | |
| *** jamesmcarthur has quit IRC | 20:08 | |
| *** jtomasek has joined #openstack-infra | 20:10 | |
| frickler | if both commands are installed, "command -v pip pip3" outputs two lines, but "head -n1" exists after the first is processed | 20:10 |
| openstackgerrit | Merged zuul/zuul-website master: Remove some redirects https://review.opendev.org/703457 | 20:11 |
| frickler | so there may be a race that sometimes causes command output to trigger a EPIPE | 20:11 |
| *** yamamoto has joined #openstack-infra | 20:11 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Fix releasenotes redirects https://review.opendev.org/703687 | 20:14 |
| *** jamesmcarthur has joined #openstack-infra | 20:14 | |
| ianw | frickler: yeah ... | 20:15 |
| ianw | $ while [ 1 ] ; do bash ./set.sh; r=$?; if [[ $r != 0 ]]; then echo $r; fi; done | 20:15 |
| ianw | 141 | 20:15 |
| *** yamamoto has quit IRC | 20:16 | |
| ianw | that's a very interesting trap, not sure i've seen that before | 20:16 |
| ianw | frickler: are you writing a change? | 20:17 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Speed up ansible plugin tests https://review.opendev.org/703688 | 20:19 |
| frickler | ianw: no I'm off for today, feel free to take over | 20:19 |
| ianw | frickler: np, ttyl! | 20:23 |
| *** Lucas_Gray has joined #openstack-infra | 20:26 | |
| *** eharney has quit IRC | 20:27 | |
| smarcet | fungi: afternoon, could u stop the puppet agent for openstackid production ? i would need to update puppet script and would like to test first at dev :) | 20:29 |
| *** yolanda has quit IRC | 20:29 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Speed up ansible plugin tests https://review.opendev.org/703688 | 20:29 |
| *** yolanda has joined #openstack-infra | 20:30 | |
| openstackgerrit | Merged zuul/zuul master: tox: pass --slowest to stestr https://review.opendev.org/703571 | 20:33 |
| fungi | smarcet: i have added openstackid01.openstack.org (the server which hosts the openstackid.org site) to our emergency disable list for ansible so it will not run puppet there further until we remove it from the list again | 20:34 |
| smarcet | fungi: thx u :) | 20:34 |
| fungi | ianw: frickler: zbr: maybe... echo `command -v pip pip3` | cut -d' ' -f1 | 20:36 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: fix pipe race https://review.opendev.org/703689 | 20:36 |
| fungi | also possible putting the first command in a subshell could work around it? | 20:36 |
| ianw | fungi: heh, or maybe ^ ? two things i thought ... 1) is that we could have a "else echo "i could not find pip"; exit 1" ... but that seems possibly backwards incompatible because it would currently work without pip, *if* tox was installed anyway | 20:37 |
| ianw | and 2, should we use pip3 in preference to pip if both are found? | 20:37 |
| ianw | again, backwards incompat change maybe, but it is 2020 ... | 20:37 |
| fungi | very good points | 20:37 |
| openstackgerrit | Merged zuul/zuul master: doc: add links to components documentation https://review.opendev.org/703105 | 20:38 |
| fungi | reversing the parameter order in the command command solves which one appears first, btw | 20:39 |
| *** rcernin has joined #openstack-infra | 20:42 | |
| openstackgerrit | Felipe Reyes proposed openstack/project-config master: Add charm-interface-keystone-notifications project https://review.opendev.org/703691 | 20:43 |
| openstackgerrit | Felipe Reyes proposed openstack/project-config master: Add charm-interface-keystone-notifications project https://review.opendev.org/703691 | 20:44 |
| *** yolanda has quit IRC | 20:46 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: use pip3 in preference to pip https://review.opendev.org/703694 | 20:46 |
| *** yolanda has joined #openstack-infra | 20:50 | |
| zbr | fungi: backtick syntax is discouraged | 20:50 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: fix pipe race https://review.opendev.org/703689 | 20:50 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: use pip3 in preference to pip https://review.opendev.org/703694 | 20:50 |
| ianw | fungi: if you get a sec for the arm64 cloud https://review.opendev.org/#/c/703535/ i'll watch it on bridge today, to be sure i didn't typo anything in the host_vars bit on bridge.o.o (someone can check bef2aeda30f496e4b32523dcc3e85d0b269dc505 on bridge) | 20:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-website master: Update redirects https://review.opendev.org/703456 | 20:56 |
| *** whoami-rajat_ has quit IRC | 20:58 | |
| *** jamesmcarthur has quit IRC | 20:58 | |
| fungi | zbr: consider it shorthand for $(), point is to evaluate in a subshell so its output is not truncated while it's running | 21:02 |
| openstackgerrit | Merged zuul/zuul master: Limit parallelity when installing ansible https://review.opendev.org/703126 | 21:04 |
| *** harlowja has quit IRC | 21:04 | |
| openstackgerrit | Merged zuul/zuul-website master: Update redirects https://review.opendev.org/703456 | 21:07 |
| *** kjackal has joined #openstack-infra | 21:08 | |
| openstackgerrit | Antoine Musso proposed x/gearman-plugin master: Add maven/java8 jdk as test bindeps https://review.opendev.org/518284 | 21:10 |
| *** smarcet has quit IRC | 21:15 | |
| *** kjackal has quit IRC | 21:18 | |
| openstackgerrit | Merged opendev/gear master: packaging: updated project urls https://review.opendev.org/703422 | 21:19 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: fix pipe race https://review.opendev.org/703689 | 21:20 |
| *** armax has joined #openstack-infra | 21:22 | |
| *** diablo_rojo has quit IRC | 21:27 | |
| *** diablo_rojo has joined #openstack-infra | 21:28 | |
| *** eharney has joined #openstack-infra | 21:28 | |
| openstackgerrit | Merged zuul/zuul master: Docs: change "config" title https://review.opendev.org/703471 | 21:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: Output tox version https://review.opendev.org/701236 | 21:32 |
| *** jamesmcarthur has joined #openstack-infra | 21:35 | |
| *** gfidente has quit IRC | 21:36 | |
| *** jtomasek has quit IRC | 21:38 | |
| openstackgerrit | Merged zuul/zuul master: docs: improve job.role documentation https://review.opendev.org/703372 | 21:51 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: test_repo_repr does not need to clone https://review.opendev.org/703698 | 22:02 |
| fungi | ianw: if you get a moment, can you take a look at 703495? it's still not entirely clear to me what AJaeger is saying about setting CFLAGS in the script causing optimizations to not be applied when the compiler is invoked | 22:05 |
| *** mattw4 has quit IRC | 22:06 | |
| *** eharney has quit IRC | 22:06 | |
| *** mattw4 has joined #openstack-infra | 22:06 | |
| ianw | fungi: was just looking :) i think the concern might be, and i've certainly seen it before, that you set CFLAGS=... and it completely overrides whatever defaults something like autoconf/make provide | 22:08 |
| ianw | which i should say i think is right, if you're trying to debug something and set CFLAGS= and it goes off and does it's own thing, that's very annoying | 22:13 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: tests: remove test_repo_repr https://review.opendev.org/703698 | 22:14 |
| *** mattw4 has quit IRC | 22:14 | |
| fungi | as in makefiles will only conditionally apply (some) compiler flags in the absence of a CFLAGS value? | 22:15 |
| *** mattw4 has joined #openstack-infra | 22:15 | |
| fungi | rather than appending the supplied CFLAGS to the compiler flags they would normally apply? | 22:15 |
| fungi | i'm going to guess this is highly project-dependent | 22:16 |
| *** slaweq_ has quit IRC | 22:16 | |
| fungi | and means that my decades-old assumptions about how CFLAGS is used in practice is now very dated | 22:16 |
| *** pkopec has quit IRC | 22:18 | |
| fungi | aha, looks like a number of projects "standardized" XCFLAGS (and XLIBS) as an alternative | 22:19 |
| ianw | yeah, i think an additional complication might be that some things try to replicate the build flags that the python library it's building against were built with | 22:20 |
| fungi | " Some package install scripts, like SDL, allow CFLAGS settings to override their normal settings (instead of append to them), so setting CFLAGS can cause harm in this case." https://en.wikipedia.org/wiki/CFLAGS | 22:23 |
| * fungi grumbles | 22:23 | |
| ianw | i really only have experience with automake, where IIRC you're supposed to use AM_ flags for "invariant" things, things that would "cause harm" if modified | 22:25 |
| ianw | then things like optimize flags go in CFLAGS, so if the user wants them off they override | 22:26 |
| fungi | i'm honestly not even sure whether c extensions for python modules regularly rely on autotools | 22:27 |
| fungi | we observed in the pyyaml case at least that exporting CFLAGS in the environment did not result in a removal of unique gcc options (though without exporting CFLAGS there were a number of redundant options appearing in the command line) | 22:28 |
| ianw | no, i'm sure they don't -- i don't have much experience on that side | 22:28 |
| *** hwoarang has quit IRC | 22:28 | |
| fungi | but i had little luck tracking down where the cc was getting invoked | 22:29 |
| fungi | so many layers of tooling | 22:29 |
| *** hwoarang has joined #openstack-infra | 22:30 | |
| ianw | :/ | 22:33 |
| fungi | clarkb: corvus: after a while of looking at the puppet-zuul module, i think the problem may be that the zuulv3 initscript variants didn't get the logic from the v2 initscripts to create pidfile dirs. do you happen to know if there was a specific reason? | 22:39 |
| fungi | er, rather, they do but they hard-code a specific path and didn't get https://review.openstack.org/530820 | 22:40 |
| fungi | i think that's the real problem | 22:40 |
| corvus | fungi: i don't recall that being intentional | 22:41 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 22:41 |
| fungi | ahh, yep, they seem to have raced that improvement | 22:41 |
| fungi | the v3 initscripts were created before that fix merged to the v2 initscripts | 22:41 |
| fungi | okay, i'll just port it so we start creating the correct pidfile directories | 22:42 |
| *** jamesmcarthur has quit IRC | 22:42 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed https://review.opendev.org/703624 | 22:46 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed https://review.opendev.org/703631 | 22:48 |
| clarkb | fungi: note the pidfule var would still be wrong there I think | 22:50 |
| clarkb | since $NAME is eg zuul-executor and what we want is /var/run/zuul/executor.pid iirc | 22:51 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-zuul master: Make v3 pidfile directory creation dynamic https://review.opendev.org/703705 | 22:51 |
| fungi | clarkb: we override it with /etc/default/* | 22:51 |
| clarkb | ah | 22:52 |
| fungi | so the underlying issue seems to simply be that the hard-coded path for the directory to create isn't the parent directory of the pidfile value we override with in /etc/defaults | 22:52 |
| fungi | anyway, there's the fix, i think, so hopefully it doesn't bite us on future reboots | 22:54 |
| *** aaronsheffield has quit IRC | 22:54 | |
| fungi | and you were right that puppet would eventually create the correct directory for us (we could probably delete that from the manifest now?) so i've started zuul-merger successfully on zm06 again | 22:55 |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| openstackgerrit | Merged zuul/zuul master: Revert "Extract an abstract base Parser class" https://review.opendev.org/703669 | 23:01 |
| corvus | =win 12 | 23:05 |
| corvus | grr | 23:05 |
| clarkb | I'll review that change just as soon as my laptop starts cooperating again | 23:07 |
| *** Lucas_Gray has quit IRC | 23:08 | |
| *** Lucas_Gray has joined #openstack-infra | 23:09 | |
| fungi | i don't think there's any hurry | 23:09 |
| fungi | it's an infrequent annoyance at most, i just wanted to get to the bottom of it | 23:10 |
| clarkb | ya but if we don't fix it now we'll forget until next time :) | 23:10 |
| fungi | well, worst case it sits in review and gets ignored until we hit the problem again, forget there's a fix already written, repeat the investigation, push up a new fix, then see the review conflict ;) | 23:11 |
| fungi | (not the first time that's happened to me... this week) | 23:11 |
| *** hashar has quit IRC | 23:16 | |
| ianw | fungi: ok, yeah i'll add it to the spec i'm forming for the wheel modernisation | 23:37 |
| *** dychen has joined #openstack-infra | 23:37 | |
| fungi | ianw: i can try to find time tomorrow to set up a temp server to test wheel builds on and see how many are not reproducible simply with a static SOURCE_DATE_EPOCH | 23:38 |
| fungi | and then that gives us a list of candidates to see what CFLAGS or other workarounds we might need to try | 23:39 |
| *** armax has quit IRC | 23:41 | |
| *** dychen has quit IRC | 23:41 | |
| *** rfolco|brb is now known as rfolco | 23:41 | |
| *** dychen has joined #openstack-infra | 23:42 | |
| *** tosky has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!