| melwitt | cool! | 00:00 |
|---|---|---|
| melwitt | this is exciting | 00:00 |
| openstackgerrit | Merged zuul/nodepool master: Switch to collect-container-logs https://review.opendev.org/701869 | 00:03 |
| openstackgerrit | Merged opendev/system-config master: Add job dependencies to haproxy-statsd https://review.opendev.org/690505 | 00:11 |
| openstackgerrit | Merged opendev/system-config master: Update python-base image upload job depends https://review.opendev.org/703001 | 00:11 |
| *** tetsuro has joined #openstack-infra | 00:13 | |
| *** smarcet has quit IRC | 00:16 | |
| *** mattw4 has quit IRC | 00:25 | |
| *** cjloader has quit IRC | 00:32 | |
| *** armax has joined #openstack-infra | 00:40 | |
| *** tkajinam_ has quit IRC | 00:43 | |
| *** tkajinam has joined #openstack-infra | 00:43 | |
| melwitt | testing out a couple changes here https://review.opendev.org/703005 | 00:44 |
| openstackgerrit | Merged zuul/zuul master: Report buildset result in MQTT reporter https://review.opendev.org/702838 | 00:50 |
| openstackgerrit | Merged zuul/zuul master: Document the buildsets endpoint https://review.opendev.org/702127 | 00:56 |
| *** openstackgerrit has quit IRC | 00:57 | |
| *** tkajinam_ has joined #openstack-infra | 01:00 | |
| *** tkajinam has quit IRC | 01:00 | |
| *** diablo_rojo has quit IRC | 01:09 | |
| *** ociuhandu has joined #openstack-infra | 01:31 | |
| *** bnemec has quit IRC | 01:31 | |
| *** ociuhandu has quit IRC | 01:35 | |
| *** gyee has quit IRC | 02:07 | |
| *** zxiiro has quit IRC | 02:33 | |
| *** roman_g has quit IRC | 02:34 | |
| *** logan- has quit IRC | 02:48 | |
| *** logan_ has joined #openstack-infra | 02:50 | |
| *** logan_ is now known as logan- | 02:50 | |
| *** osmanlicilegi has quit IRC | 02:56 | |
| *** osmanlicilegi has joined #openstack-infra | 02:56 | |
| *** edausq has quit IRC | 02:57 | |
| *** kaisers has quit IRC | 02:57 | |
| *** rfolco has joined #openstack-infra | 02:57 | |
| *** edausq has joined #openstack-infra | 02:57 | |
| *** kaisers has joined #openstack-infra | 02:57 | |
| *** rfolco has quit IRC | 03:02 | |
| *** openstackgerrit has joined #openstack-infra | 03:11 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 03:11 |
| *** apetrich has quit IRC | 03:13 | |
| *** psachin has joined #openstack-infra | 03:19 | |
| *** rh-jelabarre has quit IRC | 03:22 | |
| *** rlandy has quit IRC | 04:06 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add main configuration file https://review.opendev.org/703013 | 04:21 |
| *** raukadah is now known as chandankumar | 04:46 | |
| *** udesale has joined #openstack-infra | 04:49 | |
| *** ykarel|afk is now known as ykarel | 04:50 | |
| *** dustinc is now known as dustinc|PTO | 04:52 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** kevinz has quit IRC | 05:33 | |
| *** rpittau|afk has quit IRC | 05:34 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** lathiat has quit IRC | 05:34 | |
| *** kevinz has joined #openstack-infra | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** rpittau|afk has joined #openstack-infra | 05:34 | |
| *** lathiat has joined #openstack-infra | 05:34 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** surpatil has joined #openstack-infra | 05:48 | |
| *** udesale has quit IRC | 05:56 | |
| *** udesale has joined #openstack-infra | 05:56 | |
| *** jamesdenton has quit IRC | 06:10 | |
| *** jamesdenton has joined #openstack-infra | 06:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed opendev/storyboard master: Imported Translations from Zanata https://review.opendev.org/700328 | 06:14 |
| *** SurajPatil has joined #openstack-infra | 06:36 | |
| *** tkajinam__ has joined #openstack-infra | 06:37 | |
| *** SurajPatil has quit IRC | 06:37 | |
| *** kjackal has joined #openstack-infra | 06:38 | |
| *** tkajinam_ has quit IRC | 06:39 | |
| *** surpatil has quit IRC | 06:40 | |
| *** tkajinam_ has joined #openstack-infra | 06:42 | |
| *** tkajinam__ has quit IRC | 06:44 | |
| *** surpatil has joined #openstack-infra | 06:49 | |
| *** jaosorior has joined #openstack-infra | 06:51 | |
| *** michael-beaver has quit IRC | 06:53 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: DNM: Fix yumdownloader cache dir https://review.opendev.org/698788 | 06:58 |
| *** lmiccini has joined #openstack-infra | 07:04 | |
| *** calbers_ has joined #openstack-infra | 07:09 | |
| *** ondrejburian has joined #openstack-infra | 07:12 | |
| *** pgaxatte has joined #openstack-infra | 07:15 | |
| *** ramishra has quit IRC | 07:18 | |
| *** sdoran has quit IRC | 07:18 | |
| *** petevg has quit IRC | 07:18 | |
| *** knikolla has quit IRC | 07:18 | |
| *** trident has quit IRC | 07:18 | |
| *** jrist has quit IRC | 07:19 | |
| *** lifeless has quit IRC | 07:19 | |
| *** JayF has quit IRC | 07:19 | |
| *** SotK has quit IRC | 07:19 | |
| *** sklnk has quit IRC | 07:19 | |
| *** tbarron has quit IRC | 07:19 | |
| *** tobias-urdin has quit IRC | 07:19 | |
| *** ianw has quit IRC | 07:19 | |
| *** calbers has quit IRC | 07:19 | |
| *** tobberydberg has quit IRC | 07:19 | |
| *** zigo has quit IRC | 07:19 | |
| *** EmilienM has quit IRC | 07:19 | |
| *** antonym has quit IRC | 07:19 | |
| *** amorin has quit IRC | 07:19 | |
| *** brtknr has quit IRC | 07:19 | |
| *** calbers_ is now known as calbers | 07:19 | |
| *** kjackal has quit IRC | 07:19 | |
| *** openstackstatus has quit IRC | 07:20 | |
| *** kjackal has joined #openstack-infra | 07:21 | |
| AJaeger | evrardjp: is https://review.opendev.org/#/c/701854/ ready to merge? | 07:23 |
| *** hwoarang has quit IRC | 07:34 | |
| *** hwoarang_ has joined #openstack-infra | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:35 | |
| openstackgerrit | David Pursehouse proposed opendev/git-review master: Discontinue support for draft workflow https://review.opendev.org/685533 | 07:38 |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** ramishra has joined #openstack-infra | 08:05 | |
| *** sdoran has joined #openstack-infra | 08:05 | |
| *** petevg has joined #openstack-infra | 08:05 | |
| *** knikolla has joined #openstack-infra | 08:05 | |
| *** trident has joined #openstack-infra | 08:05 | |
| *** jrist has joined #openstack-infra | 08:05 | |
| *** JayF has joined #openstack-infra | 08:05 | |
| *** lifeless has joined #openstack-infra | 08:05 | |
| *** tbarron has joined #openstack-infra | 08:05 | |
| *** SotK has joined #openstack-infra | 08:05 | |
| *** tobias-urdin has joined #openstack-infra | 08:05 | |
| *** ianw has joined #openstack-infra | 08:05 | |
| *** tobberydberg has joined #openstack-infra | 08:05 | |
| *** zigo has joined #openstack-infra | 08:05 | |
| *** EmilienM has joined #openstack-infra | 08:05 | |
| *** antonym has joined #openstack-infra | 08:05 | |
| *** amorin has joined #openstack-infra | 08:05 | |
| *** brtknr has joined #openstack-infra | 08:05 | |
| *** jamesdenton has quit IRC | 08:08 | |
| *** zzzeek has quit IRC | 08:08 | |
| evrardjp | AJaeger: good morning | 08:11 |
| *** tkajinam_ has quit IRC | 08:11 | |
| *** zzzeek has joined #openstack-infra | 08:11 | |
| evrardjp | yes indeed , merging it now, except if you have a problem with it | 08:11 |
| *** jamesdenton has joined #openstack-infra | 08:11 | |
| evrardjp | I assume it was for merging it, else you would have commented negatively. Pressing the big button. | 08:12 |
| AJaeger | evrardjp: yes, I wanted it in to do some cleanups - thanks! | 08:14 |
| *** tesseract has joined #openstack-infra | 08:16 | |
| *** ahosam has joined #openstack-infra | 08:20 | |
| *** ahosam has quit IRC | 08:21 | |
| *** dchen has quit IRC | 08:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder master: Remove trusty jobs https://review.opendev.org/703030 | 08:23 |
| AJaeger | ianw: that one (and its dependency) is needed for trusty removal as well | 08:23 |
| *** pkopec has joined #openstack-infra | 08:24 | |
| *** rpittau|afk is now known as rpittau | 08:28 | |
| *** tosky has joined #openstack-infra | 08:29 | |
| *** tetsuro has quit IRC | 08:31 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** iurygregory has joined #openstack-infra | 08:32 | |
| openstackgerrit | Merged openstack/project-config master: Remove old openstack/js-openstack-lib jobs https://review.opendev.org/702030 | 08:34 |
| *** dmellado has quit IRC | 08:34 | |
| *** dmellado has joined #openstack-infra | 08:35 | |
| *** ccamacho has joined #openstack-infra | 08:35 | |
| *** gfidente has joined #openstack-infra | 08:38 | |
| *** ahosam has joined #openstack-infra | 08:42 | |
| *** jpena|off is now known as jpena | 08:48 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 08:49 |
| *** ralonsoh has joined #openstack-infra | 08:50 | |
| *** iurygregory_ has joined #openstack-infra | 09:01 | |
| *** iurygregory has quit IRC | 09:03 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove jobs and templates used by js-openstack-lib https://review.opendev.org/701510 | 09:04 |
| *** rcernin_ has joined #openstack-infra | 09:04 | |
| *** rcernin has quit IRC | 09:04 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 09:06 |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 09:08 |
| *** Lucas_Gray has joined #openstack-infra | 09:13 | |
| yoctozepto | morning | 09:14 |
| *** lucasagomes has joined #openstack-infra | 09:16 | |
| *** roman_g has joined #openstack-infra | 09:20 | |
| *** xek has joined #openstack-infra | 09:21 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Handle Erroneous Cron Strings in TimerDriver https://review.opendev.org/702237 | 09:23 |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 09:23 |
| yoctozepto | writing here because there are good storytellers in here - I am curious about devstack and devstack-gate - am I getting it right that devstack-gate is part of this dsvm/legacy world that everyone is trying to slowly kill with zuulv3? | 09:23 |
| tosky | yoctozepto: native zuulv3 devstack (and tempest) jobs don't use devstack-gate | 09:26 |
| yoctozepto | tosky: yup | 09:27 |
| *** Lucas_Gray has quit IRC | 09:28 | |
| *** Lucas_Gray has joined #openstack-infra | 09:29 | |
| AJaeger | yoctozepto: yes - there's an OpenStack goal for Victoria to not use devstack-gate etc anymore. | 09:31 |
| AJaeger | (at least not for master and stable/victoria) | 09:32 |
| yoctozepto | AJaeger: cool, you already answered the second question I was preparing! | 09:32 |
| AJaeger | ;) | 09:33 |
| *** psachin has quit IRC | 09:35 | |
| *** derekh has joined #openstack-infra | 09:36 | |
| openstackgerrit | Andreas Jaeger proposed opendev/glean master: Move opensuse jobs to experimental for now https://review.opendev.org/703044 | 09:36 |
| openstackgerrit | Andreas Jaeger proposed opendev/glean master: Remove trusty job https://review.opendev.org/702817 | 09:37 |
| *** ociuhandu has joined #openstack-infra | 09:38 | |
| *** kjackal has quit IRC | 09:39 | |
| AJaeger | clarkb, ianw: we need those two for trusy removal ^ | 09:39 |
| *** ociuhandu has quit IRC | 09:39 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Handle jobs with dependencies on job page https://review.opendev.org/703045 | 09:43 |
| *** kjackal has joined #openstack-infra | 09:43 | |
| *** Lucas_Gray has quit IRC | 09:44 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Remove trusty testing https://review.opendev.org/703046 | 09:45 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Remove ubuntu-trusty nodeset https://review.opendev.org/702818 | 09:46 |
| *** apetrich has joined #openstack-infra | 09:47 | |
| *** tetsuro has joined #openstack-infra | 09:47 | |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Remove ubuntu-trusty nodeset https://review.opendev.org/702818 | 09:49 |
| *** Jeffrey4l has quit IRC | 09:50 | |
| *** tetsuro has quit IRC | 09:51 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Remove trusty testing https://review.opendev.org/703047 | 09:53 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Remove ubuntu-trusty nodeset https://review.opendev.org/702818 | 09:55 |
| *** tetsuro has joined #openstack-infra | 09:56 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 09:59 |
| *** ociuhandu has joined #openstack-infra | 10:00 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Optionally support mitogen for job execution https://review.opendev.org/657024 | 10:01 |
| *** Lucas_Gray has joined #openstack-infra | 10:01 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 10:03 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 10:03 |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** tetsuro has quit IRC | 10:07 | |
| *** hashar has joined #openstack-infra | 10:12 | |
| *** ociuhandu has quit IRC | 10:19 | |
| *** Jeffrey4l has joined #openstack-infra | 10:21 | |
| *** ociuhandu has joined #openstack-infra | 10:28 | |
| *** jaosorior has quit IRC | 10:29 | |
| *** ociuhandu has quit IRC | 10:29 | |
| *** ykarel is now known as ykarel|afk | 10:31 | |
| *** rcernin_ has quit IRC | 10:34 | |
| *** apetrich has quit IRC | 10:38 | |
| *** Lucas_Gray has quit IRC | 10:38 | |
| *** apetrich has joined #openstack-infra | 10:40 | |
| *** Lucas_Gray has joined #openstack-infra | 10:42 | |
| *** ociuhandu has joined #openstack-infra | 10:46 | |
| openstackgerrit | Fatih Degirmenci proposed openstack/diskimage-builder master: Enable possibility to select HWE kernel for Ubuntu minimal https://review.opendev.org/699107 | 10:47 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: docker-install: workaround for centos-8 conflicts https://review.opendev.org/703053 | 10:49 |
| tonyb | Silly zuul question, is there any way to see why a job ran? There is one that AFAICT shouldn't run (as none of the files mentioned in files were touched in the change but it ran? | 10:50 |
| *** ociuhandu has quit IRC | 10:51 | |
| tonyb | The best I can come up with is that files has a default vaule and is appended/merged into? | 10:51 |
| AJaeger | tonyb: add "debug: true" to a queue and it shows why jobs are not-run - and I think why it's run as well. | 10:56 |
| AJaeger | tonyb: do you have a link? | 10:56 |
| tonyb | AJaeger: https://review.opendev.org/#/c/702272/1 | 10:57 |
| tonyb | AJaeger: Thanks | 10:57 |
| AJaeger | infra-root, there's a cinder job in check since 44hours - waiting for tempest-slow. | 10:58 |
| AJaeger | tonyb: which job? | 10:59 |
| *** Lucas_Gray has quit IRC | 10:59 | |
| tonyb | openstack-tox-bashate | 10:59 |
| AJaeger | the bashate one? That was run since you changed it ;) | 10:59 |
| tonyb | Ahh okay | 10:59 |
| AJaeger | a newish feature: If you change a job, it is run so that you're sure you didn't break it | 11:00 |
| tonyb | That explains that, and I didn't know | 11:00 |
| tonyb | AJaeger: that is a really awesome feature | 11:00 |
| tonyb | AJaeger: thanks as always | 11:00 |
| AJaeger | it is, indeed | 11:00 |
| AJaeger | you're welcome | 11:00 |
| yoctozepto | 12:00:09 <AJaeger> a newish feature: If you change a job, it is run so that you're sure you didn't break it | 11:02 |
| yoctozepto | no more # I had to touch it | 11:02 |
| AJaeger | yoctozepto: exactly | 11:03 |
| *** ociuhandu has joined #openstack-infra | 11:03 | |
| *** Lucas_Gray has joined #openstack-infra | 11:04 | |
| AJaeger | yoctozepto, tonyb, see https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.match-on-config-updates | 11:04 |
| tonyb | AJaeger: cool | 11:06 |
| openstackgerrit | Sorin Sbarnea proposed opendev/git-review master: Add labels on change submission https://review.opendev.org/666301 | 11:08 |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 11:15 |
| openstackgerrit | Merged opendev/git-review master: Discontinue support for draft workflow https://review.opendev.org/685533 | 11:22 |
| openstackgerrit | Merged opendev/git-review master: Install commit hook into submodules https://review.opendev.org/678428 | 11:25 |
| openstackgerrit | Antoine Musso proposed opendev/bindep master: Ensure dpkg-query uses C/English https://review.opendev.org/703055 | 11:25 |
| *** ociuhandu_ has joined #openstack-infra | 11:26 | |
| *** ociuhandu_ has quit IRC | 11:28 | |
| *** ociuhandu has quit IRC | 11:28 | |
| *** ociuhandu has joined #openstack-infra | 11:28 | |
| *** surpatil has quit IRC | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:35 | |
| *** Lucas_Gray has joined #openstack-infra | 11:38 | |
| *** ykarel|afk is now known as ykarel | 11:59 | |
| *** pcaruana has joined #openstack-infra | 11:59 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 12:00 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Don't expand change panel on middle click https://review.opendev.org/703064 | 12:08 |
| *** zbr|rover is now known as zbr|drover | 12:09 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 12:09 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 12:11 |
| *** ociuhandu has quit IRC | 12:23 | |
| *** Lucas_Gray has quit IRC | 12:24 | |
| *** Lucas_Gray has joined #openstack-infra | 12:26 | |
| *** rfolco has joined #openstack-infra | 12:27 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 12:34 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 12:34 |
| *** eharney has joined #openstack-infra | 12:37 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 12:37 |
| *** iurygregory_ is now known as iurygregory | 12:40 | |
| *** dmellado has quit IRC | 12:41 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: DNM: docker-install: test existing jobs https://review.opendev.org/703068 | 12:43 |
| *** dmellado has joined #openstack-infra | 12:44 | |
| *** udesale_ has joined #openstack-infra | 12:45 | |
| *** jpena is now known as jpena|lunch | 12:46 | |
| *** udesale has quit IRC | 12:48 | |
| *** ykarel is now known as ykarel|afk | 12:50 | |
| *** rh-jelabarre has joined #openstack-infra | 12:51 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 12:54 |
| *** ociuhandu has joined #openstack-infra | 12:57 | |
| *** ociuhandu has quit IRC | 13:03 | |
| zbr|drover | AJaeger: tristanC : can you please check https://review.opendev.org/#/c/703053/ | 13:05 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 13:09 |
| *** rlandy has joined #openstack-infra | 13:11 | |
| tristanC | zbr|drover: commented | 13:21 |
| *** lpetrut has joined #openstack-infra | 13:23 | |
| openstackgerrit | Alan Pevec proposed zuul/zuul-jobs master: Add phoronix-test-suite job https://review.opendev.org/679082 | 13:25 |
| *** gfidente has quit IRC | 13:27 | |
| *** diablo_rojo has joined #openstack-infra | 13:28 | |
| *** ricolin has joined #openstack-infra | 13:30 | |
| *** Lucas_Gray has quit IRC | 13:31 | |
| zbr|drover | tristanC: replied with more info, i hope it explains the reasoning better. | 13:32 |
| *** derekh has quit IRC | 13:32 | |
| *** dave-mccowan has joined #openstack-infra | 13:36 | |
| openstackgerrit | Stamatis Katsaounis proposed openstack/project-config master: Add dependent charm for watcher https://review.opendev.org/703081 | 13:37 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: remoke-sudo: improve sudo removal https://review.opendev.org/703065 | 13:38 |
| *** jpena|lunch is now known as jpena | 13:39 | |
| *** kozhukalov has joined #openstack-infra | 13:39 | |
| *** gfidente has joined #openstack-infra | 13:42 | |
| *** pcaruana has quit IRC | 13:45 | |
| *** ykarel|afk is now known as ykarel | 13:51 | |
| *** pkopec has quit IRC | 13:52 | |
| *** liuyulong has joined #openstack-infra | 13:59 | |
| *** yamamoto has joined #openstack-infra | 13:59 | |
| *** derekh has joined #openstack-infra | 14:02 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Manage operator scaffolding using a function and configuration file https://review.opendev.org/703013 | 14:10 |
| *** ociuhandu has joined #openstack-infra | 14:14 | |
| *** pcaruana has joined #openstack-infra | 14:22 | |
| *** ociuhandu has quit IRC | 14:27 | |
| *** dave-mccowan has quit IRC | 14:32 | |
| *** pkopec has joined #openstack-infra | 14:32 | |
| *** ociuhandu has joined #openstack-infra | 14:33 | |
| *** dave-mccowan has joined #openstack-infra | 14:34 | |
| *** aedc has joined #openstack-infra | 14:36 | |
| *** ociuhandu has quit IRC | 14:37 | |
| *** ociuhandu has joined #openstack-infra | 14:38 | |
| *** aedc has quit IRC | 14:55 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: revoke-sudo: improve sudo removal https://review.opendev.org/703065 | 15:02 |
| openstackgerrit | Merged zuul/zuul-registry master: Switch to collect-container-logs https://review.opendev.org/701868 | 15:04 |
| *** ociuhandu has quit IRC | 15:04 | |
| *** zbr|drover has quit IRC | 15:08 | |
| openstackgerrit | Lee Yarwood proposed openstack/devstack-gate master: WIP Remove g-api from subnodes https://review.opendev.org/703099 | 15:09 |
| *** lmiccini has quit IRC | 15:10 | |
| *** lmiccini has joined #openstack-infra | 15:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Shard py35 and py37 test cases https://review.opendev.org/702473 | 15:13 |
| *** zbr has joined #openstack-infra | 15:16 | |
| *** zbr is now known as zbr|drover | 15:16 | |
| *** hwoarang_ is now known as hwoarang | 15:16 | |
| *** kozhukalov has quit IRC | 15:23 | |
| *** electrofelix has joined #openstack-infra | 15:26 | |
| *** yamamoto has quit IRC | 15:28 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: doc: add links to components documentation https://review.opendev.org/703105 | 15:28 |
| *** ociuhandu has joined #openstack-infra | 15:31 | |
| *** ociuhandu has quit IRC | 15:32 | |
| *** ociuhandu has joined #openstack-infra | 15:32 | |
| AJaeger | infra-root, could you review https://review.opendev.org/#/c/703044/1 and https://review.opendev.org/#/c/703044 for trusty removal, please? | 15:35 |
| *** pgaxatte has quit IRC | 15:35 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: add admin reference section https://review.opendev.org/702997 | 15:35 |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Fix release note for a 3.0.2 feature https://review.opendev.org/703109 | 15:41 |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| *** yamamoto has quit IRC | 15:45 | |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| *** lyarwood has joined #openstack-infra | 15:46 | |
| *** aedc has joined #openstack-infra | 15:52 | |
| *** jackedin has joined #openstack-infra | 15:52 | |
| *** kjackal has quit IRC | 15:59 | |
| *** udesale_ has quit IRC | 16:01 | |
| *** udesale has joined #openstack-infra | 16:03 | |
| *** lpetrut has quit IRC | 16:08 | |
| AJaeger | infra-root, there's a cinder job in check since 49 hours - waiting for tempest-slow starting. That looks broken ;( Anything we can do? | 16:08 |
| AJaeger | Or continue to ignore it? | 16:08 |
| *** udesale has quit IRC | 16:10 | |
| *** hashar has quit IRC | 16:11 | |
| fungi | we can probably manually dequeue it, or it will likely go away the next time someone pushes a new revision of that change | 16:16 |
| fungi | (or abandons it) | 16:16 |
| *** slaweq has quit IRC | 16:16 | |
| fungi | though my suspicion is we lost an executor. that's been the cause for those most times i've looked | 16:16 |
| fungi | i'll check out executors in a bit | 16:16 |
| *** mattw4 has joined #openstack-infra | 16:20 | |
| AJaeger | fungi: indeed, we have 11 instead of 12 executors currently | 16:21 |
| AJaeger | (if I interpret grafana correctly) | 16:22 |
| fungi | and that job was probably accepted by the dead one shortly before it went to lunch | 16:23 |
| AJaeger | might be | 16:23 |
| AJaeger | that looks plausible ;) | 16:23 |
| *** lmiccini has quit IRC | 16:24 | |
| AJaeger | hemna: https://review.opendev.org/#/c/701542/ is your change which is not getting a job since 49 hours (see last 10 lines backscroll) | 16:25 |
| *** gyee has joined #openstack-infra | 16:27 | |
| *** udesale has joined #openstack-infra | 16:37 | |
| *** ykarel is now known as ykarel|away | 16:37 | |
| *** yamamoto has quit IRC | 16:40 | |
| clarkb | fungi: I want to say tobiash sets up timeouts in gearman somehow | 16:43 |
| * clarkb asks tobiash in #zuul | 16:43 | |
| hemna | AJaeger I just rechecked it | 16:53 |
| *** rpittau is now known as rpittau|afk | 16:53 | |
| AJaeger | hemna: recheck will not help, it needs abandon and restore or a rebase, change. | 16:53 |
| AJaeger | hemna: let's ask fungi first whether he needs some more time for debugging | 16:53 |
| hemna | I can rebase it and see if that helps | 16:53 |
| hemna | ok | 16:53 |
| hemna | I'll wait then | 16:53 |
| AJaeger | fungi: can hemna rebase or should he wait? | 16:54 |
| fungi | once we get the executor rebooted it should clear up on its own after a few minutes | 16:58 |
| *** dtantsur is now known as dtantsur|afk | 16:58 | |
| fungi | which executor looked hung? | 16:58 |
| *** lucasagomes has quit IRC | 16:59 | |
| AJaeger | fungi: I couldn't get hat information from grafana | 16:59 |
| AJaeger | hemna: ok, so it should clear out itself hopefully... | 16:59 |
| AJaeger | clarkb: could you put the trusty-removal topic on your review list, please? We're very close now... | 17:00 |
| *** udesale has quit IRC | 17:00 | |
| clarkb | AJaeger: yes | 17:00 |
| AJaeger | thanks | 17:01 |
| fungi | the dead executor can generally be spotted in cacti, but i'll just see if one isn't responding to ssh | 17:05 |
| clarkb | AJaeger: I went ahead and approved the glean one since its just a job move | 17:05 |
| AJaeger | clarkb: thanks - that one is prerequisite for https://review.opendev.org/702817 which removes the trusty job | 17:06 |
| fungi | i can ssh into all of them. checking uptimes now | 17:07 |
| fungi | shortest uptime is 57 days (ze08) so doesn't look like any rebooted recently | 17:08 |
| fungi | checking process lists now | 17:09 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Limit parallelity when installing ansible https://review.opendev.org/703126 | 17:10 |
| openstackgerrit | Stephen Finucane proposed openstack/devstack-gate master: Stop installing g-api on subnodes https://review.opendev.org/703129 | 17:15 |
| fungi | nothing out of the ordinary with regard to executor process entries or pidfiles. now trying to work out which executor accepted that stuck build | 17:16 |
| *** yamamoto has joined #openstack-infra | 17:17 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Limit parallelity when installing ansible https://review.opendev.org/703126 | 17:18 |
| openstackgerrit | Stephen Finucane proposed openstack/devstack-gate master: Stop installing g-api on subnodes https://review.opendev.org/703129 | 17:19 |
| *** chandankumar is now known as raukadah | 17:20 | |
| fungi | okay, rotated executor log on ze08 mentions build 7f3f0f8e86324b22968daec67d7ef28c | 17:20 |
| *** tosky has quit IRC | 17:21 | |
| AJaeger | and is ze08 still alive and running? | 17:22 |
| fungi | last entry in its executor.log is from 2020-01-15 18:28:06 so i suspect it's the one which has died | 17:22 |
| fungi | there's still a running process | 17:22 |
| fungi | zuul 26814 4.7 0.7 1805696 60012 ? Sl Jan14 188:19 /usr/bin/python3 /usr/local/bin/zuul-executor | 17:22 |
| fungi | nothing in dmesg about child processes getting killed either (last entry in dmesg is from the 9th) | 17:23 |
| *** liuyulong has quit IRC | 17:25 | |
| *** ociuhandu_ has joined #openstack-infra | 17:25 | |
| *** yamamoto has quit IRC | 17:26 | |
| openstackgerrit | Merged zuul/zuul master: Handle Erroneous Cron Strings in TimerDriver https://review.opendev.org/702237 | 17:27 |
| clarkb | fungi: ah so that lines up with tobiash's stuck executor theory | 17:27 |
| *** ociuhandu has quit IRC | 17:27 | |
| fungi | there are a couple of sleeping `git cat-file ...` processes forked from it, present since the 15th | 17:27 |
| clarkb | fungi: we could try killing them and see if it gets unstuck? | 17:28 |
| clarkb | those merges will probably fail, but we'd be able to observe if that was what held things up | 17:28 |
| tobiash | clarkb: maybe take a thread dump before? | 17:28 |
| clarkb | good idea | 17:29 |
| *** ociuhandu_ has quit IRC | 17:29 | |
| fungi | `stat /proc/9574/fd` (that's the pid of the latter of the two) says "Modify: 2020-01-15 16:07:57.879332330 +0000" which is a couple hours before the logs stop | 17:30 |
| fungi | strace on them shows they're hung on a read | 17:30 |
| tobiash | We also have sometimes (once a month or so) a stuck executor, so finding the root cause would be awesome | 17:30 |
| fungi | "read(0, " | 17:30 |
| clarkb | fungi: I think tobiash means zuul's threadump functionality | 17:30 |
| clarkb | its a sigusr 1 or 2 iirc | 17:31 |
| clarkb | I forget which one | 17:31 |
| fungi | yeah, getting to that | 17:31 |
| fungi | note for later, that might be good info to link in https://zuul-ci.org/docs/zuul/howtos/admins/troubleshooting.html | 17:32 |
| *** aedc has quit IRC | 17:32 | |
| *** evrardjp has quit IRC | 17:34 | |
| clarkb | ++ | 17:34 |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| clarkb | remote: https://review.opendev.org/703134 Split OpenDev out of OpenStack Infra | 17:35 |
| clarkb | email to be sent to openstack-discuss momentarily | 17:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: flatten directory structure https://review.opendev.org/703135 | 17:38 |
| *** jpena is now known as jpena|off | 17:45 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: re-order reference index https://review.opendev.org/702962 | 17:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: move project config docs to user reference https://review.opendev.org/702992 | 17:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: move overview section to reference https://review.opendev.org/702995 | 17:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: add admin reference section https://review.opendev.org/702997 | 17:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: flatten directory structure https://review.opendev.org/703135 | 17:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: fix styling in reconfigure commands https://review.opendev.org/703138 | 17:46 |
| *** michael-beaver has joined #openstack-infra | 17:50 | |
| *** gfidente has quit IRC | 17:51 | |
| *** zxiiro has joined #openstack-infra | 17:59 | |
| *** aedc has joined #openstack-infra | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| *** ociuhandu has joined #openstack-infra | 18:03 | |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add mailing list for OpenInfra Labs https://review.opendev.org/703145 | 18:04 |
| *** aedc has quit IRC | 18:06 | |
| *** ralonsoh has quit IRC | 18:11 | |
| *** jackedin has quit IRC | 18:16 | |
| *** ociuhandu has quit IRC | 18:18 | |
| *** ociuhandu has joined #openstack-infra | 18:18 | |
| *** ociuhandu has quit IRC | 18:19 | |
| *** hashar has joined #openstack-infra | 18:19 | |
| *** smarcet has joined #openstack-infra | 18:20 | |
| *** ociuhandu has joined #openstack-infra | 18:20 | |
| *** ociuhandu has quit IRC | 18:21 | |
| *** ociuhandu has joined #openstack-infra | 18:21 | |
| *** dave-mccowan has quit IRC | 18:23 | |
| *** ociuhandu has quit IRC | 18:26 | |
| *** smarcet has quit IRC | 18:27 | |
| *** smarcet has joined #openstack-infra | 18:27 | |
| *** dtroyer has joined #openstack-infra | 18:29 | |
| fungi | #status log restarted hung zuul-executor service on ze08, moving suspect git trees openstack/charm-vault and openstack/nova into /home/fungi/ for further investigation | 18:29 |
| openstackgerrit | David Ostrovsky proposed opendev/system-config master: Update bazel to version 2.0.0 https://review.opendev.org/703156 | 18:32 |
| *** smarcet has quit IRC | 18:32 | |
| openstackgerrit | Merged opendev/glean master: Move opensuse jobs to experimental for now https://review.opendev.org/703044 | 18:33 |
| *** bnemec has joined #openstack-infra | 18:44 | |
| *** bnemec is now known as beekneemech | 18:45 | |
| *** artom has joined #openstack-infra | 18:45 | |
| *** ociuhandu has joined #openstack-infra | 18:46 | |
| artom | o/ | 18:46 |
| *** electrofelix has quit IRC | 18:46 | |
| artom | Is FN still down? And what would be a better way of tracking that, than asking here? | 18:46 |
| artom | (FN = the Fort Nebula nodepool provider) | 18:46 |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-24h&to=now | 18:48 |
| donnyd | It looks up to me | 18:48 |
| donnyd | :) | 18:48 |
| artom | donnyd, my information is clearly out of date :) | 18:49 |
| artom | Cheers! | 18:49 |
| donnyd | It was brought back up by clarkb late yesterday | 18:49 |
| donnyd | no worries | 18:49 |
| donnyd | did you need it for a particular reason or just curious? | 18:50 |
| artom | donnyd, our whitebox tempest plugin is currently entirely dependant on FN | 18:50 |
| artom | Since some or our nests require your multi-numa flavor | 18:50 |
| artom | (Others don't, but we haven't gotten around to splitting jobs) | 18:50 |
| donnyd | Ah I see - well it makes me truly happy that people find FN useful | 18:51 |
| *** pcaruana has quit IRC | 18:51 | |
| donnyd | even if that is the only thing its used for :) | 18:51 |
| fungi | artom: which plug-in, our of curiosity>? | 18:52 |
| artom | fungi, it's called just that - "whitebox-tempest-plugin" :) | 18:52 |
| fungi | ahh, that's the repo name. okay | 18:52 |
| artom | https://opendev.org/x/whitebox-tempest-plugin | 18:52 |
| *** ociuhandu has quit IRC | 18:52 | |
| fungi | thanks! i was assuming you were describing the plug-in, didn't occur to me that was its actual name | 18:53 |
| artom | All good, thanks for asking :) | 18:53 |
| fungi | yeah, i was mostly interested in seeing where numa specifics were coming into play in tempest tests | 18:54 |
| fungi | sounds neat | 18:54 |
| AJaeger | fungi, the restart unblocked the cinder test - node is now allocated and testing started. thanks | 18:55 |
| fungi | AJaeger: yep, thanks for raising it! | 18:55 |
| AJaeger | hemna: all good - nothing for you to do besides waiting ;) | 18:55 |
| artom | fungi, the stuff we do would never fly in vanilla tempest, hence the plugin | 18:55 |
| fungi | artom: testing a pinning feature i guess? | 18:55 |
| artom | And I think in general there's a gap in whitebox-style testing, when the API isn't enough | 18:55 |
| artom | fungi, yeah, for now it's mostly to do with pinning and NUMA-y things | 18:56 |
| *** lpetrut has joined #openstack-infra | 18:58 | |
| *** diablo_rojo has quit IRC | 18:58 | |
| *** smarcet has joined #openstack-infra | 18:59 | |
| *** smarcet has quit IRC | 19:03 | |
| mordred | morning all - anything it would be useful for me to look at? | 19:07 |
| AJaeger | what is openstackci-images in https://opendev.org/opendev/system-config/src/branch/master/playbooks/clouds_layouts.yml#L112 ? that references "ubuntu-trusty". Is that used and needed? | 19:07 |
| AJaeger | Can we remove trusty testing of system-config with https://review.opendev.org/703047 ? | 19:07 |
| AJaeger | good morning, mordred | 19:07 |
| *** hashar has quit IRC | 19:08 | |
| mordred | AJaeger: that's telling cloud launcher how to put trusty base images into clouds. I think we can remove it | 19:10 |
| AJaeger | I don't see openstackci-images used anywhere, so remove that whole stanza? | 19:12 |
| mordred | AJaeger: lemme look real quick | 19:12 |
| AJaeger | mordred: let me prepare change in parallel... | 19:12 |
| mordred | AJaeger: yes | 19:13 |
| AJaeger | thanks, mordred | 19:13 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Remove openstackci-images for ubuntu-trusty https://review.opendev.org/703159 | 19:13 |
| AJaeger | mordred: ^ | 19:13 |
| *** pkopec_ has joined #openstack-infra | 19:15 | |
| *** pkopec has quit IRC | 19:16 | |
| AJaeger | infra-root, around 16:24 and 17:44, it seems that Zuul terminated all images - and restarted, see http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&fullscreen&panelId=21 | 19:18 |
| AJaeger | and again 18:37 | 19:18 |
| AJaeger | that does not look healthy | 19:18 |
| clarkb | that can happen if zk connectivity is bad | 19:19 |
| jrosser | something went horribly wrong here, looks like log collection ssh failed across the board https://review.opendev.org/#/c/702133/ | 19:20 |
| AJaeger | clarkb: three times in a 130 minutes? ;( | 19:20 |
| clarkb | currebtly finishing brunch but can look closer after | 19:20 |
| AJaeger | jrosser: see my comment just above yours | 19:20 |
| clarkb | AJaeger: ya basically the dump everything behavio is associated eith that becauseit kills things globally | 19:20 |
| AJaeger | clarkb: "ssh: connect to host 2001:470:e045:8000:f816:3eff:febb:ab7f port 22: No route to host | 19:21 |
| AJaeger | (from jrosser's change) | 19:21 |
| AJaeger | clarkb: understood | 19:21 |
| AJaeger | jrosser: feel free to recheck | 19:22 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Add notes on thread dumping and yappi https://review.opendev.org/703185 | 19:26 |
| clarkb | I think that happens if the scheduler runs out of memory | 19:27 |
| fungi | checking in on that now | 19:27 |
| fungi | it is running pretty tight on available memory | 19:28 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 19:29 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all | 19:29 |
| dmsimard | the nodepool graph suggests high rates of launch failures as well: http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1 | 19:29 |
| clarkb | zuul updates introduced aleak maybe | 19:29 |
| mordred | oh - that restart twice makes sense with the "why has this job restarted twice" issue I was just looking at | 19:29 |
| fungi | looks like it began climbing around 04:00 yesterday | 19:29 |
| clarkb | what happens is zk connwction times out due to memory pressure and swapping. Then the ephemeral zk nodes get deleted whoch resulys in all nodes being deleted which causes ssh failures | 19:30 |
| *** ociuhandu has joined #openstack-infra | 19:30 | |
| clarkb | I suspect a change in latest zuul is to blame given the timing | 19:30 |
| fungi | restarts were on tuesday, this began climbing on thursday, so for some reason whatever triggers it didn't seem to occur on wednesday | 19:31 |
| mordred | do we happen to know what sha we were running pre-restart? | 19:32 |
| *** tesseract has quit IRC | 19:32 | |
| fungi | should i try to get a thread dump before it's completely out to lunch? | 19:32 |
| fungi | mordred: hopefully the previous restart was mentioned in the status log | 19:32 |
| AJaeger | "2019-12-18 22:00:49 UTC restarted all of zuul at commit 84f6ea667c3453a75a7a0210ee08228c9eec167a" | 19:33 |
| clarkb | fungi: its possible the triggering state didnt occur until thursday | 19:33 |
| clarkb | but pre restart zuul was very consistent in memory use | 19:33 |
| AJaeger | that looks like the last restart - according to https://wiki.openstack.org/wiki/Infrastructure_Status | 19:33 |
| dmsimard | fwiw logs on nl01: lots of "Not enough quota remaining to satisfy request" with quota exceeded exceptions: http://paste.openstack.org/show/788550/ | 19:33 |
| dmsimard | timeouts waiting for server to come up as well | 19:34 |
| *** pkopec_ has quit IRC | 19:34 | |
| clarkb | dmsimard: ya I think those are "normal" nova has gotten confused about quota usage state :/ | 19:34 |
| clarkb | we hit them a lot when at capacity | 19:34 |
| clarkb | AJaeger: that lines up with my memory | 19:35 |
| fungi | i'm going to risk it and try to do a thread dump while the scheduler still isn't swapping | 19:35 |
| *** pkopec_ has joined #openstack-infra | 19:35 | |
| *** ociuhandu has quit IRC | 19:35 | |
| mordred | there are not too many scheduler related changes since the last restart | 19:36 |
| corvus | 84f6ea667c3453a75a7a0210ee08228c9eec167a..e6d8b210cc416ed494b0b0248404e3e6d7ce337c if i'm reading the status log right | 19:37 |
| corvus | a mere 17 changes to inspect | 19:38 |
| mordred | corvus: yeah - with several being docs related | 19:38 |
| tobiash | nothing catches my eyes there at first glance | 19:40 |
| tobiash | I'll see if I can cross check with my deployment | 19:40 |
| corvus | i'm producing a list of candidates | 19:40 |
| fungi | the thread dump is in the debug log but it's still working on yappi's object type counting looks like | 19:40 |
| fungi | i have a feeling that may take a while with the scheduler using some 28gb of memory | 19:41 |
| *** tosky has joined #openstack-infra | 19:41 | |
| corvus | https://etherpad.openstack.org/p/3joixMs0Lz | 19:42 |
| corvus | okay i think we're down to two changes we should look closely at | 19:44 |
| openstackgerrit | Merged zuul/zuul master: Defer setting build result to event queue https://review.opendev.org/666643 | 19:45 |
| mordred | maybe the getSchema call vs the schema = vs.Schema in the base parser change could leak schemas? | 19:46 |
| fungi | yappi type counting completed and it's all in the debug log now | 19:46 |
| fungi | do we want a second sigusr2 to stop yappi and do the profiling summary? | 19:47 |
| fungi | i'm going to guess yes. more info is better for now | 19:48 |
| fungi | looks like yappi stops much more quickly than it starts. full yappi analysis is in the debug log now too | 19:49 |
| fungi | should we save queues and restart the scheduler, or revert a couple changes locally on it first? | 19:50 |
| clarkb | mordred: also maybe the logging change is creating a new logger for each instance? (though I think that .__class__.__name__ is correct and shouldn't do that) | 19:50 |
| corvus | mordred: i'm not seeing how that could happen right now... even if a parser object leaked a schema, there aren't that many parser objects | 19:50 |
| corvus | fungi: i'd like to revert one change before restarting | 19:50 |
| fungi | shall i let you handle the restart in that case, for expediency? | 19:51 |
| fungi | or i'm happy to do it if you let me know what you want reverted | 19:51 |
| corvus | i'm thinking we should revert the abstract base class change. even though i don't currently understand why it would cause a memory leak, it touches the code that, in the past, has been most associated with leaks | 19:51 |
| mordred | yeah | 19:52 |
| corvus | it also has no impact on functionality, so if we're shooting in the dark, it's a good one to start with | 19:52 |
| clarkb | wfm | 19:52 |
| fungi | sound reasoning | 19:52 |
| mordred | ++ | 19:52 |
| corvus | okay, i will manually revert that and restart the scheduler | 19:52 |
| fungi | thanks!!! | 19:52 |
| corvus | anything else need to happen during the restart, or just scheduler ok? | 19:52 |
| clarkb | I think just scheduler is ok | 19:52 |
| fungi | i know of nothing else | 19:52 |
| *** aedc has joined #openstack-infra | 19:52 | |
| tobiash | corvus: yes, that patch would be my hunch as well | 19:53 |
| fungi | i can't remember if zuul-web still needs restarting after a scheduler restart | 19:53 |
| fungi | guess we'll find out | 19:53 |
| corvus | er... | 19:53 |
| tobiash | fungi: it shouldn't need a restart | 19:53 |
| corvus | i'm also going to rewind from master a bit | 19:53 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: IRC #openstack-ironic gerritbot CI failed messages https://review.opendev.org/698091 | 19:53 |
| corvus | so i will start with 3.15.0 and revert the base class change, as opposed to starting with master | 19:53 |
| clarkb | corvus: ++ that way we don't introduce other variance | 19:54 |
| corvus | (i don't want "Defer setting build result to event queue" to be in the mix -- even though that fixes a known bug) | 19:54 |
| fungi | yeah, that sounds good to me | 19:54 |
| mordred | agree | 19:54 |
| corvus | actually, let me revise that again -- i'm going to start at 84f6ea667c3453a75a7a0210ee08228c9eec167a and revert 10190257f (because 3.15.0 isn't exactly what we were running, though it was only docs changes) | 19:55 |
| corvus | nope, e6d8b210cc416ed494b0b0248404e3e6d7ce337c and revert 10190257f :) | 19:56 |
| fungi | even better. whatever status log said we restarted at | 19:56 |
| mordred | :) | 19:56 |
| corvus | yeah. pretty sure that's right. :) | 19:57 |
| fungi | confirmed, https://wiki.openstack.org/wiki/Infrastructure_Status says e6d8b210cc416ed494b0b0248404e3e6d7ce337c | 19:57 |
| corvus | restarting now | 19:58 |
| *** pkopec_ has quit IRC | 19:58 | |
| corvus | #status log restarted zuul scheduler at commit e6d8b210cc416ed494b0b0248404e3e6d7ce337c with 10190257f reverted to debug memory leak | 19:58 |
| corvus | fungi: if you have a sec, can you resuscitate statusbot? | 19:59 |
| fungi | on it | 19:59 |
| *** michael-beaver has quit IRC | 20:00 | |
| fungi | it should rejoin any minute | 20:00 |
| *** openstackstatus has joined #openstack-infra | 20:01 | |
| *** ChanServ sets mode: +v openstackstatus | 20:01 | |
| corvus | #status log restarted zuul scheduler at commit e6d8b210cc416ed494b0b0248404e3e6d7ce337c with 10190257f reverted to debug memory leak | 20:01 |
| openstackstatus | corvus: finished logging | 20:01 |
| corvus | fungi: thanks! | 20:01 |
| mordred | corvus: zuul status is now unhappy | 20:01 |
| clarkb | in that change JobParser goes from having a single class level schema to a per instance scham | 20:01 |
| clarkb | * per instance Schema | 20:02 |
| fungi | #status log restarted statusbot service on eavesdrop.o.o to recover following a 07:20z ctcp ping timeout | 20:02 |
| openstackstatus | fungi: finished logging | 20:02 |
| clarkb | considering we have many many jobs, it is possible that if those were leaked we would have problems? | 20:02 |
| clarkb | if this revert fixes the problem ^ is my hunch at what is leaking | 20:02 |
| clarkb | many of the other objects use per instance schemas but they are also far less common in a zuul than jobs | 20:03 |
| AJaeger | clarkb: I'm surprised that it stayed flat for 2 days and then exploded suddenly | 20:03 |
| mordred | yeah- although it still wouldn't explain the tuesday-thursday delay in issue | 20:03 |
| clarkb | maybe we added a bunch of new jobs recently? | 20:03 |
| clarkb | but ya I dunno | 20:03 |
| corvus | oh there was a delay? | 20:03 |
| corvus | that will make it hard to determine if the revert is responsible | 20:03 |
| *** jtomasek has quit IRC | 20:03 | |
| AJaeger | corvus: wait until Monday and enjoy the weekend ;) | 20:04 |
| corvus | actually | 20:04 |
| corvus | now that i see the graph | 20:04 |
| corvus | i would like to change my vote | 20:04 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=2&view_type=&graph_start=1579023141&graph_end=1579289671&graph_height=120&graph_width=500&title_font_size=12 | 20:04 |
| fungi | yes, we restarted tuesday, this started climbing thursday | 20:05 |
| clarkb | delay is like 16 hours | 20:05 |
| fungi | (around 04:00z) | 20:05 |
| fungi | i think it's more than 16 hours? | 20:05 |
| fungi | unless my math is terrible | 20:05 |
| clarkb | fungi: see link above | 20:05 |
| clarkb | restart seems to be 0000 wednesday? | 20:05 |
| clarkb | (roughly | 20:06 |
| clarkb | oh sorry 26 hours | 20:06 |
| fungi | that sounds more like what i thought | 20:06 |
| fungi | restart was logged by statusbot at 2020-01-14 23:30:32 | 20:06 |
| fungi | so pretty close, yep | 20:07 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Set ubuntu-trusty in nodepool to -1 https://review.opendev.org/703189 | 20:12 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Bye, Bye, Trusty https://review.opendev.org/703190 | 20:12 |
| AJaeger | FYI, just 10 changes - plus the two above - to remove trusty | 20:12 |
| corvus | hrm. i'm not seeing anything unusual in the logs around then, but i don't know what i'd be looking for. | 20:12 |
| fungi | i've cleaned up the stack dumps and yappi profiling and they're in zuul.o.o:~fungi/stack_dump.log now | 20:13 |
| fungi | in case that's of any help whatsoever | 20:14 |
| Shrews | anyone else find it odd that the mem increase seems to happen right on 04:00? | 20:14 |
| fungi | do we have scheduled jobs which fire at that time? | 20:15 |
| Shrews | wondering the same thing | 20:15 |
| fungi | though it didn't happen at 04:00 on wednesday | 20:15 |
| fungi | only thursday | 20:15 |
| AJaeger | the daily jobs fire at 6:00 | 20:15 |
| fungi | so if there's some sort of scheduled daily event triggering things, it's not 100% reproducing | 20:16 |
| *** smarcet has joined #openstack-infra | 20:20 | |
| *** dtroyer has quit IRC | 20:21 | |
| *** aedc has quit IRC | 20:26 | |
| *** dtroyer has joined #openstack-infra | 20:28 | |
| *** haleyb has quit IRC | 20:28 | |
| *** haleyb has joined #openstack-infra | 20:32 | |
| *** kjackal has joined #openstack-infra | 20:35 | |
| *** rh-jelabarre has quit IRC | 20:36 | |
| *** aedc has joined #openstack-infra | 20:36 | |
| Shrews | nothing is standing out to me in thursday's log at 04:00. then again, don't know what i'd be looking for | 20:41 |
| *** zxiiro has quit IRC | 20:42 | |
| *** ociuhandu has joined #openstack-infra | 20:46 | |
| *** diablo_rojo has joined #openstack-infra | 20:47 | |
| *** ociuhandu has quit IRC | 20:48 | |
| *** ociuhandu has joined #openstack-infra | 20:48 | |
| *** dave-mccowan has joined #openstack-infra | 20:49 | |
| *** jamesmcarthur has joined #openstack-infra | 20:51 | |
| *** smarcet has quit IRC | 20:52 | |
| *** jamesmcarthur has quit IRC | 20:53 | |
| *** jamesmcarthur has joined #openstack-infra | 20:55 | |
| *** eharney has quit IRC | 20:59 | |
| clarkb | is that stack dump showing we have almost a million lists ? | 21:00 |
| clarkb | and we gained 1.2k between the two yappi invocations? | 21:00 |
| clarkb | ya ok I think it is showing the greatest growth in object counts between the two | 21:02 |
| clarkb | we have 4 million mapping proxies | 21:02 |
| clarkb | and two million dicts but those were stable between the two | 21:02 |
| *** ociuhandu has quit IRC | 21:03 | |
| *** ociuhandu has joined #openstack-infra | 21:04 | |
| *** hwoarang has quit IRC | 21:09 | |
| *** ociuhandu has quit IRC | 21:13 | |
| *** hwoarang has joined #openstack-infra | 21:15 | |
| fungi | yeah, yappi ran from 19:36:31,021-19:48:19,850 so 708.829 seconds | 21:17 |
| clarkb | ok 120k project configs | 21:17 |
| clarkb | and 151k FileMatchers | 21:17 |
| clarkb | it does seem that we are leaking configs because I doubt we have 120k of them | 21:17 |
| *** jamesmcarthur has quit IRC | 21:20 | |
| openstackgerrit | Merged zuul/zuul master: Fix release note for a 3.0.2 feature https://review.opendev.org/703109 | 21:22 |
| *** yamamoto has joined #openstack-infra | 21:24 | |
| clarkb | also that shows 202 logger instances which means I'm pretty sure we aren't leaking those | 21:26 |
| *** dave-mccowan has quit IRC | 21:28 | |
| AJaeger | we retired a repo at 4:00 on Thursday - would that trigger it? | 21:28 |
| AJaeger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-01-16.log.html#t2020-01-16T04:04:24 | 21:28 |
| AJaeger | otherwise no ideas ;( | 21:28 |
| clarkb | maybe the removal from the tenant config? | 21:29 |
| *** yamamoto has quit IRC | 21:29 | |
| AJaeger | maybe ;) | 21:29 |
| *** dave-mccowan has joined #openstack-infra | 21:30 | |
| AJaeger | here're two system-config changes for trusty removal: https://review.opendev.org/703047 and https://review.opendev.org/703159 - reviews welcome. topic is trusty-removal - https://review.opendev.org/703046 (zuul-jobs) is also ready to go next. | 21:30 |
| AJaeger | Have a great weekend everybody | 21:31 |
| * AJaeger waves good night | 21:31 | |
| clarkb | AJaeger: you too! | 21:31 |
| fungi | enjoy your weekend AJaeger! | 21:31 |
| AJaeger | thanks! | 21:31 |
| mordred | clarkb: between the filematchers count and the fact that it wasn't an immediate increase but instead had a delay now kind of makes me think it was the filematcher change | 21:33 |
| clarkb | mordred: ya, though I think the filematchers may leak because the jobs leak | 21:34 |
| mordred | I don't really understand the mechanics of why that change would result in that - or why the increase would have not coincided more directly with a timer trigger | 21:34 |
| clarkb | mordred: since jobs refer to file matchers | 21:34 |
| mordred | yeah | 21:34 |
| openstackgerrit | Merged zuul/zuul master: Docs: re-order reference index https://review.opendev.org/702962 | 21:35 |
| clarkb | possible it goes the other way around though | 21:35 |
| clarkb | I did check that voluptuous hasn't made a recent release (it hasn't, last was in august) | 21:35 |
| corvus | we did perform some full reconfigs before the one at 4:27 on 1-16 (which would be the one for the project removal) | 21:36 |
| clarkb | just in case maybe something in there is leaking and thus keeping the configs around | 21:36 |
| *** rlandy has quit IRC | 21:36 | |
| corvus | so merely "full reconfig" alone doesn't seem to explain it. also, the full reconfig for the project removal was 27m after the increase started. | 21:36 |
| corvus | (^ thinking about project removal as a trigger) | 21:37 |
| mordred | ++ | 21:37 |
| clarkb | I think chrome just crashed my desktop | 21:39 |
| mordred | not related to the zuul memory leak - but a recent bump to bazel for gerrit combined with _not_ bumping bazel for stable-2.15 makes me think I need to rework the gerrit image build a bit :( | 21:41 |
| *** aedc has quit IRC | 21:42 | |
| clarkb | thinking out loud here I think we should avoid compiling schemas everytime we build configs | 21:43 |
| clarkb | the schema is fixed so should be safe to compile once | 21:43 |
| clarkb | ? | 21:43 |
| *** ociuhandu has joined #openstack-infra | 21:44 | |
| fungi | maybe worth repeating in #zuul | 21:48 |
| *** ociuhandu has quit IRC | 21:48 | |
| *** kjackal has quit IRC | 21:50 | |
| openstackgerrit | Merged zuul/zuul master: Docs: move project config docs to user reference https://review.opendev.org/702992 | 21:55 |
| *** kozhukalov has joined #openstack-infra | 22:00 | |
| *** hashar has joined #openstack-infra | 22:04 | |
| openstackgerrit | Merged zuul/zuul master: Docs: move overview section to reference https://review.opendev.org/702995 | 22:12 |
| *** smarcet has joined #openstack-infra | 22:15 | |
| *** smarcet has quit IRC | 22:17 | |
| *** eharney has joined #openstack-infra | 22:26 | |
| *** rh-jelabarre has joined #openstack-infra | 22:33 | |
| *** ociuhandu has joined #openstack-infra | 22:35 | |
| *** xek has quit IRC | 22:36 | |
| openstackgerrit | Merged zuul/zuul master: Docs: add admin reference section https://review.opendev.org/702997 | 22:37 |
| *** ociuhandu has quit IRC | 22:40 | |
| *** dklyle has quit IRC | 22:48 | |
| *** david-lyle has joined #openstack-infra | 22:48 | |
| *** david-lyle has quit IRC | 22:49 | |
| *** dklyle has joined #openstack-infra | 22:49 | |
| *** slaweq has joined #openstack-infra | 22:50 | |
| *** KeithMnemonic1 has quit IRC | 22:51 | |
| openstackgerrit | Merged opendev/glean master: Remove trusty job https://review.opendev.org/702817 | 23:00 |
| *** hashar has quit IRC | 23:02 | |
| openstackgerrit | Merged zuul/zuul master: Docs: flatten directory structure https://review.opendev.org/703135 | 23:03 |
| *** kozhukalov has quit IRC | 23:13 | |
| *** slaweq has quit IRC | 23:14 | |
| *** dave-mccowan has quit IRC | 23:15 | |
| clarkb | I added notes about what I found to the etherpad. I'm not sure any of it is super useful | 23:25 |
| *** rh-jelabarre has quit IRC | 23:27 | |
| fungi | entirely possible it's useful and we just don't know yet, so thanks | 23:32 |
| *** hwoarang has quit IRC | 23:33 | |
| *** hwoarang has joined #openstack-infra | 23:34 | |
| *** ahosam has quit IRC | 23:46 | |
| *** mattw4 has quit IRC | 23:57 | |
| *** tetsuro has joined #openstack-infra | 23:59 | |
| *** rcernin_ has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!