Thursday, 2020-02-20

openstackgerritIan Wienand proposed opendev/system-config master: static: Add service-types.openstack.org  https://review.opendev.org/70876900:11
ianwhttps://7fef9cb88655f9afd142-a5696e911ff4bae58cd7af702019a98e.ssl.cf1.rackcdn.com/708507/3/check/system-config-run-static/65c7bc3/bridge.openstack.org/ara-report/00:15
ianwanother case of failed job due to remote host id changed :/00:15
fungiif they're semi-frequent it should be possible to scrape logstash and correlate ip addresses. if we see the same ip addresses appear over and over, that's usually a smoking gun for an orphaned guest instance00:22
ianwhttps://news.ycombinator.com/item?id=22368804 <- a good HN comment that explains what i saw with testinfra system-config and host testing where apache would serve the first site, turns out people were using that as an abuse vector00:26
fungigood read, i hadn't stopped to consider the potential exploit scenarios there00:32
fungialso reminds me why i run my own webservers, even for my personal sites00:32
*** mattw4 has quit IRC00:33
*** diablo_rojo has quit IRC00:33
*** stevebaker has quit IRC00:36
*** matt_kosut has joined #openstack-infra00:47
*** gyee has quit IRC00:48
*** tetsuro has joined #openstack-infra00:50
*** matt_kosut has quit IRC00:52
ianwhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_088/706900/5/check/openstack-tox-docs/0887adc/job-output.txt seems to be the last use of 23.253.164.158, maybe00:54
ianwhttp://paste.openstack.org/show/789772/ i can't see any conflicts01:02
ianwrax ord for future reference01:03
*** stevebaker has joined #openstack-infra01:15
*** yamamoto has joined #openstack-infra01:24
*** Xuchu has quit IRC01:26
*** ociuhandu has quit IRC01:27
*** ociuhandu has joined #openstack-infra01:28
*** ociuhandu has quit IRC01:34
openstackgerritMerged opendev/system-config master: Add specs.openstack.org  https://review.opendev.org/70850701:39
*** rfolco has quit IRC01:44
*** smarcet has joined #openstack-infra02:13
*** larainema has joined #openstack-infra02:21
*** ricolin has quit IRC02:25
*** hwoarang has quit IRC02:29
*** hwoarang has joined #openstack-infra02:31
*** Goneri has quit IRC02:34
*** igordc has quit IRC02:45
openstackgerritMerged opendev/system-config master: static: Add service-types.openstack.org  https://review.opendev.org/70876902:45
openstackgerritMerged openstack/project-config master: Switch specs publishing to AFS  https://review.opendev.org/70850002:53
ianw#status log switched CNAME entry for specs.openstack.org to static.opendev.org where it is now published from /afs/openstack.org/project/specs.openstack.org02:54
*** ijw has quit IRC02:55
*** openstackstatus has joined #openstack-infra02:56
*** ChanServ sets mode: +v openstackstatus02:56
ianw#status log switched CNAME entry for specs.openstack.org to static.opendev.org where it is now published from /afs/openstack.org/project/specs.openstack.org02:58
openstackstatusianw: finished logging02:58
ianw(and i restarted statusbot, again :)02:58
*** xinranwang has joined #openstack-infra03:00
*** ricolin has joined #openstack-infra03:02
*** rlandy|bbl is now known as rlandy03:03
*** yamamoto has quit IRC03:11
*** ociuhandu has joined #openstack-infra03:11
*** yamamoto has joined #openstack-infra03:14
*** jamesmcarthur has joined #openstack-infra03:14
openstackgerritIan Wienand proposed opendev/base-jobs master: opendev-zuul-tarballs: switch to new AFS site  https://review.opendev.org/70879203:15
*** ociuhandu has quit IRC03:16
*** ijw has joined #openstack-infra03:23
*** ijw has quit IRC03:28
ianw#status log switched A/AAAA records for service-types.openstack.org to a CNAME for static.opendev.org, where it is now published from /afs/openstack.org/projects/service-types.openstack.org03:32
*** psachin has joined #openstack-infra03:32
openstackstatusianw: finished logging03:32
*** tetsuro has quit IRC03:33
*** tetsuro has joined #openstack-infra03:33
*** ykarel|away is now known as ykarel03:34
*** yamamoto has quit IRC03:36
*** yamamoto has joined #openstack-infra03:36
*** lbragstad has quit IRC03:39
*** stevebaker has quit IRC03:40
*** evrardjp has quit IRC03:42
*** evrardjp has joined #openstack-infra03:45
*** jamesmcarthur has quit IRC03:48
*** jamesmcarthur has joined #openstack-infra03:49
*** jamesmcarthur has quit IRC03:54
ianwAJaeger: want to have a quick look at @ https://review.opendev.org/#/c/708792/ and i think we can migrate the tarballs sites?03:57
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: Migrate tarballs.opendev.org to static.opendev.org  https://review.opendev.org/70879504:01
ianwhave to merge that too04:01
*** hwoarang has quit IRC04:01
*** smarcet has left #openstack-infra04:01
*** migawa is now known as migawa|lunch|AFK04:10
*** ijw has joined #openstack-infra04:11
*** migawa|lunch|AFK is now known as migawa|lunch04:12
*** jamesmcarthur has joined #openstack-infra04:13
*** udesale has joined #openstack-infra04:20
*** nicolasbock has quit IRC04:21
*** jamesmcarthur has quit IRC04:25
*** ramishra has joined #openstack-infra04:29
*** HenryG has quit IRC04:35
*** HenryG has joined #openstack-infra04:36
*** rlandy has quit IRC04:41
*** dave-mccowan has joined #openstack-infra04:42
*** imacdonn has quit IRC04:44
*** imacdonn has joined #openstack-infra04:45
*** matt_kosut has joined #openstack-infra04:49
*** matt_kosut has quit IRC04:53
*** ramishra has quit IRC04:58
AJaegerianw: done.05:05
AJaegerianw: thanks! Want to +2 https://review.opendev.org/#/c/708518/ ?05:05
*** ramishra has joined #openstack-infra05:06
ianwAJaeger: i say we move it now, since it's quiet enough.  i can check back in a few hours in case of issues05:11
ianw#status log tarballs.openstack.org CNAME updated to static.opendev.org which is now serving its content from /afs/openstack.org/projects/tarballs.openstack.org/openstack/05:14
openstackstatusianw: finished logging05:14
openstackgerritMerged opendev/zone-opendev.org master: Migrate tarballs.opendev.org to static.opendev.org  https://review.opendev.org/70879505:17
*** ccamacho has quit IRC05:17
openstackgerritMerged openstack/project-config master: Remove publish-service-types-authority  https://review.opendev.org/70851805:18
ianw#status log tarballs.opendev.org CNAME updated to static.opendev.org (change #708795) which is now serving its content from /afs/openstack.org/projects/tarballs.openstack.org/openstack/05:19
openstackstatusianw: finished logging05:19
ianwonce 708792 merges i will since /afs/openstack.org/project/opendev.org/tarballs to /afs/openstack.org/project/tarballs.opendev.org and move the old one to an archived name; then i think we're done!05:20
*** ijw has quit IRC05:22
openstackgerritMerged opendev/base-jobs master: opendev-zuul-tarballs: switch to new AFS site  https://review.opendev.org/70879205:22
ianwok, as of now all opendev jobs will publish to the new location05:23
ianw... and synced05:24
*** ijw has joined #openstack-infra05:25
ianw#status log tarballs.<openstack|opendev>.org are now served by static.opendev.org from /afs/openstack.org/project/tarballs.opendev.org.  all jobs updated05:28
openstackstatusianw: finished logging05:28
*** ijw has quit IRC05:29
*** ociuhandu has joined #openstack-infra05:30
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-infra05:34
*** ociuhandu has quit IRC05:35
*** dave-mccowan has quit IRC05:47
AJaegerianw: really done? Great! Congrats and thanks for driving this!05:49
AJaegerianw: I just approved https://review.opendev.org/#/c/70673405:49
AJaegeranything else to do?05:49
ianwAJaeger: well, i think releases.openstack.org is next :)05:50
AJaegernext - and last?05:52
ianwi think of the publishing bits.  then the redirects, retirement of logs.openstack.org (corvus should do a ceremonial commit on that :)05:54
AJaegerreleases is only the docs from releases repo? I'll try to do that later today...05:55
AJaegerianw: have a great evening, I'll be back online in a bit05:55
*** ijw has joined #openstack-infra05:56
openstackgerritMerged openstack/project-config master: Remove tarball artifact publishing to static site  https://review.opendev.org/70673405:58
*** ijw has quit IRC06:01
ianwAJaeger: yeah, double checking releases it only looks like docs so should be easy-ish.  i had it in my head there was more being published there06:01
*** udesale_ has joined #openstack-infra06:03
*** udesale has quit IRC06:06
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873106:17
*** lmiccini has joined #openstack-infra06:19
*** xinranwang has quit IRC06:19
openstackgerritSimon Westphahl proposed zuul/zuul master: Diable misfire grace time of apscheduler job  https://review.opendev.org/70784206:22
*** yolanda has joined #openstack-infra06:22
openstackgerritSimon Westphahl proposed zuul/zuul master: Disable misfire grace time of apscheduler job  https://review.opendev.org/70784206:22
*** Lucas_Gray has joined #openstack-infra06:24
*** jamesmcarthur has joined #openstack-infra06:27
*** ijw has joined #openstack-infra06:28
*** priteau has joined #openstack-infra06:30
*** jamesmcarthur has quit IRC06:31
*** ramishra has quit IRC06:32
*** ijw has quit IRC06:32
*** ramishra has joined #openstack-infra06:32
*** ahosam has joined #openstack-infra06:33
*** Lucas_Gray has quit IRC06:41
*** Lucas_Gray has joined #openstack-infra06:42
*** matt_kosut has joined #openstack-infra06:45
*** ijw has joined #openstack-infra06:59
*** ijw has quit IRC07:03
fricklerkevinz: ianw: this is the fix for the worlddump issue you've been seeing, it comes from py3 passing e.g. b'br-ex' as argument https://review.opendev.org/70857907:12
kevinzfrickler: thx a lot!07:14
*** pgaxatte has joined #openstack-infra07:24
*** Lucas_Gray has quit IRC07:25
openstackgerritJan Kubovy proposed zuul/zuul master: Scheduler test app factory  https://review.opendev.org/70881207:27
*** ijw has joined #openstack-infra07:30
*** dpawlik has joined #openstack-infra07:31
*** priteau has quit IRC07:34
*** ijw has quit IRC07:35
*** ociuhandu has joined #openstack-infra07:40
*** Lucas_Gray has joined #openstack-infra07:43
*** ociuhandu has quit IRC07:45
*** snapiri has joined #openstack-infra07:47
*** dchen has quit IRC07:50
*** slaweq has joined #openstack-infra07:50
*** jpena|off is now known as jpena07:51
*** ykarel is now known as ykarel|lunch08:01
*** ijw has joined #openstack-infra08:02
*** tesseract has joined #openstack-infra08:05
*** ijw has quit IRC08:06
*** yamamoto has quit IRC08:08
*** yamamoto has joined #openstack-infra08:11
*** ociuhandu has joined #openstack-infra08:13
*** ociuhandu has quit IRC08:17
*** ociuhandu has joined #openstack-infra08:19
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873108:21
*** tetsuro_ has joined #openstack-infra08:25
*** rpittau|afk is now known as rpittau08:26
*** tetsuro has quit IRC08:29
*** tosky has joined #openstack-infra08:29
*** iurygregory has joined #openstack-infra08:31
*** ccamacho has joined #openstack-infra08:32
*** ijw has joined #openstack-infra08:33
*** yamamoto has quit IRC08:36
*** yamamoto has joined #openstack-infra08:38
*** ijw has quit IRC08:38
*** tesseract has quit IRC08:38
*** tesseract has joined #openstack-infra08:39
*** amoralej|off is now known as amoralej08:40
*** dayou has quit IRC08:41
*** gfidente has joined #openstack-infra08:41
*** dayou has joined #openstack-infra08:42
mnaserinfra-root: are executors seeing issues?  i see _a lot_ of "waiting on logger" messages08:42
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873108:43
*** tetsuro_ has quit IRC08:43
openstackgerritJan Kubovy proposed zuul/zuul master: Scheduler test app factory  https://review.opendev.org/70881208:48
*** yamamoto has quit IRC08:50
*** ociuhandu has quit IRC08:50
openstackgerritMohammed Naser proposed zuul/zuul master: ci: use bionic for functional tests  https://review.opendev.org/70882008:53
*** ralonsoh has joined #openstack-infra08:56
*** ociuhandu has joined #openstack-infra08:56
openstackgerritMohammed Naser proposed zuul/zuul master: DNM: exercising usage of bionic for stream jobs  https://review.opendev.org/70882109:01
*** pkopec has joined #openstack-infra09:01
*** Lucas_Gray has quit IRC09:02
*** jaicaa has quit IRC09:05
*** ijw has joined #openstack-infra09:05
*** jaicaa has joined #openstack-infra09:07
*** lucasagomes has joined #openstack-infra09:09
*** ijw has quit IRC09:09
*** carli has joined #openstack-infra09:10
*** ykarel|lunch is now known as ykarel09:13
*** tkajinam has quit IRC09:19
*** priteau has joined #openstack-infra09:20
*** lajoskatona has joined #openstack-infra09:21
lajoskatonaHi, I have an issue with some neutron-stadium project zuul jobs09:24
lajoskatonaexample: https://review.opendev.org/708648 (for pep8: https://92342067336e42252139-0bd2acfdbec8f6316ef91de37dd7cea6.ssl.cf2.rackcdn.com/708648/1/check/openstack-tox-pep8/9efee46/job-output.txt )09:25
lajoskatonait seems that the job tries to install train bgpvpn, and for that recently came out a release which caps horizon (https://review.opendev.org/699456)09:26
lajoskatonasee the irclog: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2020-02-19.log.html#t2020-02-19T14:38:2709:26
lajoskatonaWhat I don't understand if I add bgpvpn to required projects why the job at a point tries to install numbered release (which have only for train at the moment)09:27
*** hwoarang has joined #openstack-infra09:27
*** derekh has joined #openstack-infra09:29
lajoskatonaSo actually master bagpipe is now broken as a new train release capped some version. Is there a way to avoid this kind of issue as the problem appears for the step [tox : Run tox without tests]09:30
*** ijw has joined #openstack-infra09:36
AJaegerlajoskatona: we install what you specify via requirements and test-requirements in tox.ini09:37
lajoskatonaAJaeger: Hi09:38
lajoskatonaAJaeger: https://opendev.org/openstack/networking-bagpipe/src/branch/master/requirements.txt#L2609:38
*** ysastri has joined #openstack-infra09:39
lajoskatonaAJaeger: here we have bgpvpn >=9.0.0, if I change it to >=11.0.1 i will still have latest train release, or am I wrong? And in that case I will have the same problem09:39
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873109:39
AJaegerlajoskatona: you have https://opendev.org/openstack/networking-bagpipe/src/branch/master/.zuul.yaml#L17 - and that forces it to install bgpvpn from git09:40
*** ijw has quit IRC09:40
AJaegerso, the master branch, not a release09:41
AJaegerlajoskatona: check https://zuul.opendev.org/t/openstack/build/9efee469a6cf4dbbb4992c6d658b951c/log/job-output.txt#110209:41
lajoskatonaAJaeger: that was my understanding as well, but the job result points to the other direction with train installed in this case09:41
AJaegeris it dragging in horizon latest release? Maybe you need to install horizon from git as well - add it as required project09:42
lajoskatonaAJaeger: yes, I have seen that line, but I lost my way why later I have a contextualversion conflict09:42
lajoskatonaAJaeger: I tried yesterday09:43
*** Lucas_Gray has joined #openstack-infra09:43
lajoskatonaAJaeger: https://review.opendev.org/#/c/703949/8/.zuul.yaml09:43
AJaegerlajoskatona: I have not time to dig further, so let me explain briefly what happens: If there's a required-project line, we install the corresponding branch (so master - master, stable/train - stable/train) instead of a release. You configured some required-project liens09:44
AJaegerinteresting that that one still fails ;(09:44
AJaegerlajoskatona: you need to add *horizon* to requirements file as well, did you do that?09:46
*** xek_ has joined #openstack-infra09:46
AJaegerlajoskatona: the job parses requirements/test-requirements and installs those packages from git that are also in required-projects, otherwise a package.09:46
lajoskatonaAJaeger: you mean to add horizon to bagpipe requirements? I can do a try09:46
AJaegerlajoskatona: So, update 703949 and add horizon as requirement.09:46
AJaegerYes, please try.09:47
lajoskatonaAJaeger: ok thanks09:47
AJaegerIf that does not help, I hope somebody else with more time can help you - I need to step out again.09:47
*** StevenK has quit IRC09:52
*** StevenK has joined #openstack-infra09:52
*** tesseract has quit IRC09:58
*** jaosorior has joined #openstack-infra09:59
openstackgerritMatthieu Huin proposed zuul/zuul master: Authorization rules: add templating  https://review.opendev.org/70519310:01
*** matt_kosut has quit IRC10:02
*** matt_kosut has joined #openstack-infra10:02
*** Lucas_Gray has quit IRC10:03
*** ijw has joined #openstack-infra10:07
openstackgerritTobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS  https://review.opendev.org/70758510:08
*** matt_kosut has quit IRC10:08
*** tesseract has joined #openstack-infra10:11
*** ijw has quit IRC10:11
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests ensure-tox on all-platforms  https://review.opendev.org/70864210:14
*** Lucas_Gray has joined #openstack-infra10:14
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests bindep role on all-platforms  https://review.opendev.org/70870410:14
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests bindep role on all-platforms  https://review.opendev.org/70870410:15
*** Wryhder has joined #openstack-infra10:17
*** Lucas_Gray has quit IRC10:18
*** Wryhder is now known as Lucas_Gray10:19
*** matt_kosut has joined #openstack-infra10:36
*** ijw has joined #openstack-infra10:38
*** ijw has quit IRC10:43
AJaegerlajoskatona: I checked a bit - and we need mordred's help.10:46
*** yamamoto has joined #openstack-infra10:48
AJaegermordred, lajoskatona, amotoki, check  https://review.opendev.org/#/c/703949 - why is pep8 failing? Seems that "Run tox without tests" will install latest packages and those do not work together, so we never install the siblings ;( The "error" might be the release of horizon 18.0 - and no release of networking-bgpvpn since then (latest release has requirement "< 17.0").  So, would a new tag of10:49
AJaegernetworking-bgpvpn solve this?10:49
*** yamamoto has quit IRC10:51
amotokiAJaeger: networking-bgpvpn train release is installed in the job. networking-bgpvpn has a horizon plugin and it installs horizon train which is incompatible with u-c master.10:51
amotokiAJaeger: networking-bgpvpn train has the upper bound for horizon, so it fails.10:52
amotokiAJaeger: we are in a deadlock as the current gate forces us to install dependencies from PyPI first. neutron stadium projects uses cycle-with-rc so there is no surpise we have no beta release yet. It leads to no way to handle the situation.....10:55
AJaegeramotoki: horizon 18.0.0 is Ussuri, isn't it? I think we need a release of networking-bpgvpn that allows install of horizon 18.0.0 to untangle it - or a different way to handle the siblings.10:55
AJaegeramotoki: I see - mordred is the expert here, please discuss with him once he's awake and around.10:55
amotokione option is to uncap horizon in networking-bgpvpn train sounds reasonable to me as we depends on u-c and don't cap dependencies in stabler branches.10:57
*** Lucas_Gray has quit IRC11:02
*** udesale_ has quit IRC11:03
*** ijw has joined #openstack-infra11:09
*** ijw has quit IRC11:14
amotokiAnother option is to release ussuri beta networking-bgpvpn to include the right horizon dependency and update requirements in bagpipe to use the beta bgpvpn.11:19
*** rpittau is now known as rpittau|bbl11:20
*** adriancz has quit IRC11:21
*** nicolasbock has joined #openstack-infra11:23
*** sshnaidm|afk is now known as sshnaidm11:23
*** ykarel is now known as ykarel|afk11:24
*** jaosorior has quit IRC11:30
*** Lucas_Gray has joined #openstack-infra11:39
*** ijw has joined #openstack-infra11:41
*** ijw has quit IRC11:45
*** lpetrut has joined #openstack-infra11:50
*** dtantsur|afk is now known as dtantsur11:51
*** rfolco has joined #openstack-infra11:52
carliGood day! I have a question, about the logstash (on http://logstash.openstack.org). Are logs deleted at a regular time? I wanted to do a query on elasticsearch and it told me the index for logstash-2020.02.13 does not exist (so only 6 days of logs of the openstack ci are kept?)11:56
openstackgerritTobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS  https://review.opendev.org/70758511:59
*** jpena is now known as jpena|lunch12:00
*** amoralej is now known as amoralej|lunch12:01
lajoskatonaAJaeger amotoki: Hi, thanks12:09
*** ccamacho has quit IRC12:14
*** yamamoto has joined #openstack-infra12:17
*** yamamoto has quit IRC12:26
*** yamamoto has joined #openstack-infra12:32
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add zuul-operator-upload-image job  https://review.opendev.org/70886012:34
*** udesale has joined #openstack-infra12:34
*** lajoskatona has left #openstack-infra12:34
*** beagles is now known as beagles_pto12:35
elodamotoki AJaeger lajoskatona : horizon was capped in stable because it was said (as far as i remember) to not to add it to upper constraints. without that unfortunately networking-bgpvpn was failing (due to incompatibility with newer horizon if i remember correctly). Maybe one option is to add horizon to upper constraints in Train, if that is allowed and doesn't break anything else12:37
amotokielod: you can find horizon in u-c https://opendev.org/openstack/requirements/src/branch/stable/train/upper-constraints.txt#L69512:38
elodamotoki: hmmm sounds interesting12:39
amotokielod: it was added around Jan 7-8 https://review.opendev.org/#/q/I421ee2d498ea5100d9b52e356b46f2d594638b5b12:39
elodok, then the cap should/can be removed from bgpvpn12:40
*** lajoskatona has joined #openstack-infra12:40
amotokielod: did you see the discussion on dependency capping in stable branches in openstack-discuss?12:40
elodi guess I've missed that, looking now12:41
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873112:42
*** ijw has joined #openstack-infra12:43
*** yamamoto has quit IRC12:45
*** ijw has quit IRC12:47
*** ysastri has quit IRC12:49
*** psachin has quit IRC12:49
*** ysastri has joined #openstack-infra12:50
*** rlandy has joined #openstack-infra12:51
*** sshnaidm has quit IRC13:00
*** sshnaidm has joined #openstack-infra13:01
*** rh-jelabarre has joined #openstack-infra13:03
elodamotoki: thanks for the pointers!13:03
*** yamamoto has joined #openstack-infra13:04
*** rpittau|bbl is now known as rpittau13:07
*** artom has quit IRC13:08
fricklercarli: yes, we delete after 7 days because of limited storage capacity13:08
*** matt_kosut has quit IRC13:10
*** jamesmcarthur has joined #openstack-infra13:14
*** ijw has joined #openstack-infra13:15
carliokay, I just wanted to be sure :) thanks for the confirmation! :)13:15
*** jamesmcarthur has quit IRC13:15
*** ccamacho has joined #openstack-infra13:17
openstackgerritFelix Edel proposed zuul/zuul master: Report retried builds via sql reporter.  https://review.opendev.org/63350113:17
*** ijw has quit IRC13:19
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base cleanup run.  https://review.opendev.org/70887113:21
AJaegermordred: could you review https://review.opendev.org/708515, please?13:23
AJaegerconfig-core, please review https://review.opendev.org/708727 to unblock zuul-jobs testing13:24
*** jpena|lunch is now known as jpena13:24
*** lbragstad has joined #openstack-infra13:26
*** amoralej|lunch is now known as amoralej13:28
*** jamesmcarthur has joined #openstack-infra13:29
*** jamesmcarthur has quit IRC13:31
*** jamesmcarthur has joined #openstack-infra13:32
*** psachin has joined #openstack-infra13:36
*** Lucas_Gray has quit IRC13:38
*** jamesmcarthur has quit IRC13:44
openstackgerritTristan Cacqueray proposed zuul/nodepool master: Implement zookeeper-auth  https://review.opendev.org/61915513:45
*** ijw has joined #openstack-infra13:46
toskyout of curiosity, have you bumped the ansible version used by the opendev.org zuul instance between ~half of December and the beginning of January?13:46
tosky(background story: I'm trying to understand a job failure in an unmerged patch which started somewhere between end of December and beginning of January, and a change in the environment is a possible reason)13:47
*** ociuhandu has quit IRC13:50
*** ijw has quit IRC13:51
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests ensure-tox on all-platforms  https://review.opendev.org/70864213:52
openstackgerritTristan Cacqueray proposed zuul/zuul master: Implement zookeeper-auth  https://review.opendev.org/61915613:53
*** jamesmcarthur has joined #openstack-infra13:54
*** Goneri has joined #openstack-infra13:55
*** ociuhandu has joined #openstack-infra13:58
*** ociuhandu has quit IRC14:02
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873114:05
mordredtosky: I believe we did, yes - but I was travelling a lot and don't 100% remember14:10
mordredAJaeger: do you remember?14:10
mnaseri believe there was a bump14:12
openstackgerritMohammed Naser proposed zuul/zuul master: ci: use bionic for functional tests  https://review.opendev.org/70882014:12
openstackgerritMohammed Naser proposed zuul/zuul master: DNM: exercising usage of bionic for stream jobs  https://review.opendev.org/70882114:12
mordredmnaser: cool14:13
mnaseror actually14:13
mordredamotoki, elod: sounds like you have a path forward?14:13
mnaseri think we started using python3 in the target hosts maybe?14:13
mordredmnaser: might have done both?14:13
mnaserimagine you could store the state of your infrastructure in git and go back and check that14:14
mnaserhttps://opendev.org/openstack/project-config/commit/46ec754fda87ff9d694832040bf35b36a30f103c14:14
mnaserhttps://opendev.org/openstack/project-config/commit/dff7fccc130a7c64d422b402d85d1f8669640a3914:14
mnaserso i believe zuul bumped the default to 2.8, but then openstack broke, so we pinned to 2.7 until we unfixed ourself and then went back to the zuul default, which seems to be 2.814:15
mnasercc tosky ^14:15
amotokimordred: yeah, neutron team is thinking to cut beta release to fix the gate failure (+ address other requirements like ovn-octavia-provider). what elod and I discussed on networking-bpnvpn stable/train would be optional.14:15
mordredamotoki: cool - that sounds good14:16
mordredmnaser: imagine that! that seems almost crazy to consider :)14:16
toskymordred, mnaser: thanks; was 2.7 the starting version (in December)?14:16
mnaserhttps://zuul-ci.org/docs/zuul/reference/releasenotes.html#relnotes-3-13-0 "The default version of Ansible used by Zuul jobs has been changed to 2.8. See job.ansible-version for more information."14:17
* mnaser looks for release date14:17
mnasertosky: http://lists.zuul-ci.org/pipermail/zuul-announce/2019-December/000061.html14:18
*** ijw has joined #openstack-infra14:18
mnaserdecember 11 might be your timeline14:18
*** ysastri has quit IRC14:21
*** eharney has joined #openstack-infra14:21
toskymnaser: I've seen the releases for zuul, but I wasn't sure if the code deployed on the main instance was immediately updated, and also on which ansible version (as several are supported)14:21
mnasertosky: afaik the openstack deployment is historically running the release before its even released as a smoke test (and yeah, as those 2 commits showed, we unpinned the config so its running 2.8)14:22
*** ijw has quit IRC14:22
mnasertosky: if you still have the logs, zuul logs the current ansible version when it starts too afaik14:23
*** ykarel|afk is now known as ykarel14:23
toskymnaser: sadly when the issue started the logs from the older rechecks were not available anymore14:23
mnaserahh, bummer14:24
*** jamesmcarthur has quit IRC14:26
*** ociuhandu has joined #openstack-infra14:29
*** jaosorior has joined #openstack-infra14:34
*** sshnaidm is now known as sshnaidm|afk14:34
fricklertosky: which patch/job are you talking about? if the history is gone anyway, let's just look at what we have now and try to make it work14:37
mnaseri'm seeing a lot of 'waiting for logger' messages ..14:38
openstackgerritFelix Edel proposed zuul/zuul master: Report retried builds via sql reporter.  https://review.opendev.org/63350114:39
toskyfrickler: this one https://review.opendev.org/#/c/672804/ and namely https://review.opendev.org/#/c/672804/14:39
toskythe only relevant change in cinderlib is https://review.opendev.org/#/c/701572/ (by me)14:40
toskybut the affected playbooks are not used by the failing job, if I'm not mistaken14:40
toskyalso, I've seen the issue in a test job for train, so py2 too: https://review.opendev.org/#/c/705191/14:41
*** jamesmcarthur has joined #openstack-infra14:42
toskyit seems like the user used to run the tests changed and the current one does not have the proper permissions14:42
toskyI kind of reproduced locally too on ubuntu/devstack when using a different user than the "stack" user14:42
smcginnismnaser: I've seen several patches with that too.14:45
*** snapiri has quit IRC14:46
smcginnisI think starting early last week.14:46
*** ijw has joined #openstack-infra14:49
*** dave-mccowan has joined #openstack-infra14:49
*** piotrowskim has quit IRC14:53
*** ijw has quit IRC14:54
*** yamamoto has quit IRC14:54
*** diablo_rojo has joined #openstack-infra14:56
fungii'm still in an all-day meeting today so probably don't have an opportunity to dig into it15:09
clarkbme too15:15
*** lpetrut has quit IRC15:20
*** ijw has joined #openstack-infra15:21
*** psachin has quit IRC15:22
*** ijw has quit IRC15:25
openstackgerritMerged zuul/zuul-jobs master: set jobs for installing openvswitch  https://review.opendev.org/70872715:31
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add ensure-bazelisk role  https://review.opendev.org/70889915:34
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add ensure-bazelisk role  https://review.opendev.org/70889915:36
*** pgaxatte has quit IRC15:36
mordred#status log rebooted gitea05 - adding back to haproxy15:39
openstackstatusmordred: finished logging15:39
mordredinfra-root: I maybe should have said "also triggered a replication and waited for it to be done"15:39
clarkbmordred: a single host replicates pretty quickly if you do it now15:41
clarkb(its when we do a bunch at once that it seems to bog down15:41
mordredyeah. I did it already15:42
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add ensure-java role  https://review.opendev.org/70890115:42
mordredclarkb: incidentally - that makes me think - should we make the gitea playbook do a disable/enable in haproxy - and trigger a single-host replication?15:43
mordredclarkb: would make rollouts go slightly slower but would be totally batteries included15:43
*** sshnaidm|afk is now known as sshnaidm15:43
clarkbthe tricky bit would be checking if all clients have stopped talking to a backend, but maybe its good enough to try and be graceful since right now its slightly less than graceful15:43
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests ensure-tox on all-platforms  https://review.opendev.org/70864215:47
*** artom has joined #openstack-infra15:47
mordredclarkb: yeah. that said - I'm going to go ahead and roll out the gitea change15:48
*** ijw has joined #openstack-infra15:53
*** ijw has quit IRC15:58
*** lajoskatona has quit IRC15:59
fungithanks mordred!16:02
*** rfolco is now known as rfolco|doctor16:04
*** ykarel is now known as ykarel|afk16:05
*** jpena is now known as jpena|off16:07
*** mattw4 has joined #openstack-infra16:08
*** gyee has joined #openstack-infra16:14
*** eernst has joined #openstack-infra16:18
*** eernst has quit IRC16:19
*** ijw has joined #openstack-infra16:24
*** ijw has quit IRC16:29
*** ijw has joined #openstack-infra16:30
*** ccamacho has quit IRC16:32
* prometheanfire is not sure if last message made it through16:38
fungiprometheanfire: i've seen nothing recent from you other than that one line16:40
prometheanfirefrickler: clarkb: for making gentoo build faster, there's quite a few things that can help, I tend to build with -j5 -l16 and also enable parallel-fetch parallel-install in FEATURES in make.conf16:42
prometheanfireok, fixed :D16:42
clarkbshould we edit make.conf in our images then?16:42
prometheanfireeh, atm I'm not sure it's worth it, depends on how much pain not doing it is causing16:43
prometheanfireI generally favor not touvhing things unless needed16:43
*** slaweq has quit IRC16:43
*** lmiccini has quit IRC16:44
*** ociuhandu has quit IRC16:44
*** ociuhandu has joined #openstack-infra16:46
*** dtantsur is now known as dtantsur|afk16:51
*** ociuhandu has quit IRC16:52
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801816:52
*** rpittau is now known as rpittau|afk16:53
*** jaosorior has quit IRC16:55
*** slaweq has joined #openstack-infra16:55
*** lucasagomes has quit IRC16:58
*** slaweq has quit IRC17:00
*** tesseract has quit IRC17:00
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add AFS publish job for releases repo  https://review.opendev.org/70891617:00
openstackgerritMerged opendev/system-config master: Upgrade gitea to 1.10.3  https://review.opendev.org/70580417:06
AJaegerianw: I do not see releases.openstack.org in AFS, check http://files.openstack.org/project/ and https://review.opendev.org/708916 , please17:08
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add AFS publish job for releases repo  https://review.opendev.org/70891617:09
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove obsolete releases publishing  https://review.opendev.org/70892017:09
*** ociuhandu has joined #openstack-infra17:12
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Tests ensure-tox on all-platforms  https://review.opendev.org/70864217:13
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove unused files in playbooks/publish  https://review.opendev.org/70892517:14
*** diablo_rojo has quit IRC17:22
AJaegerconfig-core, please review https://review.opendev.org/#/c/708916/17:22
mordredinfra-root: https://review.opendev.org/#/c/708023/ landed 3 days ago but polymer-bridges still doesn't show up in the zuul project list - shouldn't that show up automatically?17:25
mordredAJaeger: ^^ ?17:25
*** jpena|off is now known as jpena17:27
AJaegermordred: it should - if ansible runs17:28
AJaegermordred: indeed, missing from http://zuul.opendev.org/t/opendev/projects17:29
*** udesale has quit IRC17:33
*** ijw has quit IRC17:34
mordredAJaeger: interesting. well - maybe the gitea05 issues were blocking it17:34
mordredthe gitea upgrade is running now - let's see if it clears up after that17:34
*** evrardjp has quit IRC17:34
*** carli has quit IRC17:34
*** evrardjp has joined #openstack-infra17:35
openstackgerritJames E. Blair proposed ttygroup/gertty master: Handle binary data in diffs  https://review.opendev.org/70893217:50
openstackgerritJames E. Blair proposed ttygroup/gertty master: Hide errors when syncing conflicts  https://review.opendev.org/70893317:50
openstackgerritJames E. Blair proposed ttygroup/gertty master: Display line comments from all patchsets in change view  https://review.opendev.org/70893417:50
*** gfidente has quit IRC17:52
*** ociuhandu_ has joined #openstack-infra18:00
*** derekh has quit IRC18:02
*** ociuhandu has quit IRC18:04
zbrfungi: clarkb: please look at https://review.opendev.org/#/c/708642/ -- i had to make some modificationns to ensure-tox to make it run on all platforms.18:05
*** ociuhandu_ has quit IRC18:05
zbrin fact I spent a lot of time tunning them in order to get all green.18:05
zbrthere were some assumptions there that didn't play well when run against 9 different platforms18:06
zbrit was an interesting whack-a-mole game18:07
mordredAJaeger: oh - well - easy explanation - the polymer-bridges line is now gone from main.yaml18:08
AJaegerwhy that?18:08
mordredor ... hrm18:08
*** ralonsoh has quit IRC18:08
AJaegerit should be there - at least in my local copy I see it18:08
mordredthe copy on zuul.o.o is - digging further18:08
AJaegerthen ansible did not run somehow18:09
mordredyeah - project-config repo is up to date on bridge18:12
mordredbut not up to date since feb 12 on zuul.o.o18:12
AJaegersince over a week? ;(18:14
mordredFeb 20 18:05:09 zuul01 puppet-user[16549]: (Facter) error while resolving custom fact "virtualenv_version": undefined method `[]' for nil:NilClass18:14
mordredit's the virtualenv thing again18:14
mordredfix coming18:14
AJaegerThanks, mordred !18:15
mordredmaybe fix coming ...18:15
AJaegerstill thanks ;)18:16
*** ijw has joined #openstack-infra18:27
openstackgerritJames E. Blair proposed ttygroup/gertty master: Support robot comments  https://review.opendev.org/70894418:28
*** ijw has quit IRC18:32
*** jpena is now known as jpena|off18:34
*** ykarel|afk is now known as ykarel|away18:34
*** igordc has joined #openstack-infra18:34
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add ensure-bazelisk role  https://review.opendev.org/70889918:35
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Add ensure-java role  https://review.opendev.org/70890118:36
*** amoralej is now known as amoralej|off18:36
openstackgerritMohammed Naser proposed zuul/zuul master: configloader: validate playbook-less + secret jobs  https://review.opendev.org/70873118:46
*** matt_kosut has joined #openstack-infra18:48
mordredcorvus: I just did "git pull --ff-only" in /etc/project-config on zuul  - because I was going to then re-run puppet and try to see what stanza was causing it to get reset backwards by a week18:50
mordredit think's it's up to date - and it thinks it's on master18:51
mordredit's gitea0518:52
mordredit's out of date18:52
mordredalthough I did a gerrit replication earlier18:52
clarkbmordred: did it come up with a stale git lockfile18:52
corvusmordred: yeah, i was just wondering if the lb had it pinned to a bad gitea host18:52
clarkbthat prevents replication from completing but it looks like it completed18:52
corvus(fwiw, i just cloned a new copy into my homedir on zuul01 and get the same ref)18:53
clarkb-rw-r--r--  1 1000 1000  123 Feb 20 17:40 .gitconfig18:53
mordred*awesome*18:53
clarkbthat seems not stale18:53
mordredclarkb: should we try a replicate again?18:53
clarkbmordred: maybe try replicate just that repo? and we can narrow down the problem space18:53
* clarkb goes back to being distracted by meetings18:53
mordredok. I replicated project-config and it's happy now18:55
mordredso maybe I shoudl try --all again18:55
corvusmordred: ++  (but just with the gitea05 target, yeah?)18:55
mordredcorvus: in the meantime - I'm running puppet on zuul again18:55
mordredcorvus: yeah18:55
mordredcorvus: well that was exciting18:57
mordredmain.yaml is now up to date18:57
*** ijw has joined #openstack-infra18:59
*** ijw has quit IRC19:03
openstackgerritMohammed Naser proposed zuul/zuul master: cleanup: move AnsibleZuulTestCase to test_ansible  https://review.opendev.org/70895219:06
*** ijw has joined #openstack-infra19:08
*** ijw has quit IRC19:13
*** stevebaker has joined #openstack-infra19:15
*** rlandy is now known as rlandy|mtg19:16
*** larainema has quit IRC19:16
*** ijw has joined #openstack-infra19:30
*** rlandy|mtg is now known as rlandy19:34
*** ijw has quit IRC19:35
*** igordc has quit IRC19:38
*** jamesmcarthur has quit IRC19:45
mordred#status log gitea upgraded to 1.10.319:46
openstackstatusmordred: finished logging19:46
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801819:49
*** ijw has joined #openstack-infra20:02
*** ijw has quit IRC20:07
openstackgerritMichael Daitzman proposed openstack/project-config master: Create openinfralabs/contrib repo  https://review.opendev.org/70896120:20
*** eharney has quit IRC20:30
*** priteau has quit IRC20:44
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove obsolete releases publishing  https://review.opendev.org/70892020:47
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801820:50
*** artom has quit IRC20:54
corvusmordred: \o/ thanks!20:56
corvusmordred: oh look at this: https://opendev.org/explore/repos?q=zuul&tab=&sort=recentupdate20:57
corvusthe last updated times seem to be old, and not updated by new commits20:57
corvusi wonder what changed there (i'm pretty sure they used to update after each ref update)20:58
fungizuul/zuul "updated 7 months ago"20:58
AJaegerbut https://opendev.org/zuul/zuul shows "22hours ago"21:00
mordredcorvus: weird.21:02
*** Lucas_Gray has joined #openstack-infra21:03
*** Lucas_Gray has quit IRC21:03
corvusmordred: might be worth waiting till we're done upgrading to latest to debug21:07
mordredcorvus: yeah21:07
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801821:08
*** Nizars has joined #openstack-infra21:09
fungialso it's possible not every gitea backend shows the same numbers21:20
fungimight be worth comparing21:21
openstackgerritMichael Daitzman proposed openstack/project-config master: Create openinfralabs/contrib repo  https://review.opendev.org/70896121:22
openstackgerritAkihiro Motoki proposed openstack/hacking master: Disable old-style-class check by default  https://review.opendev.org/70896921:24
*** Goneri has quit IRC21:25
*** xek_ has quit IRC21:26
*** igordc has joined #openstack-infra21:28
*** eharney has joined #openstack-infra21:29
ianwAJaeger: ahh yes still need to create that volume.  will do, thanks for looking in21:35
openstackgerritIan Wienand proposed opendev/system-config master: static: add default site  https://review.opendev.org/70897221:37
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801821:43
openstackgerritJeremy Stanley proposed opendev/jeepyb master: Overhaul default welcome message for OpenDev  https://review.opendev.org/70897521:51
*** Nizars has quit IRC21:59
openstackgerritAdrian Turjak proposed openstack/project-config master: Add pypi publish to Adjutant UI  https://review.opendev.org/70898322:26
*** dpawlik has quit IRC22:33
*** iurygregory has quit IRC22:36
openstackgerritMerged opendev/system-config master: launch-node.py : make sure new inventory comes last  https://review.opendev.org/70642422:37
openstackgerritMerged opendev/system-config master: openafs-client: use dnf for CentOS 8  https://review.opendev.org/70703722:38
openstackgerritIan Wienand proposed opendev/system-config master: static: add default site  https://review.opendev.org/70897222:45
*** pkopec has quit IRC22:47
*** ijw has joined #openstack-infra22:51
openstackgerritJames E. Blair proposed ttygroup/gertty master: Add support for checks plugin  https://review.opendev.org/70899222:57
*** tkajinam has joined #openstack-infra22:58
*** sshnaidm is now known as sshnaidm|afk23:02
*** slaweq has joined #openstack-infra23:10
*** ahosam has quit IRC23:10
*** ahosam has joined #openstack-infra23:11
*** slaweq has quit IRC23:15
*** ociuhandu has joined #openstack-infra23:16
openstackgerritIan Wienand proposed opendev/system-config master: static: add releases.openstack.org site  https://review.opendev.org/70900923:17
*** nicolasbock has quit IRC23:18
*** ociuhandu has quit IRC23:22
*** slaweq has joined #openstack-infra23:26
openstackgerritMerged openstack/project-config master: Add AFS publish job for releases repo  https://review.opendev.org/70891623:28
*** eharney has quit IRC23:30
*** slaweq has quit IRC23:31
ianwfungi: mind a quick look @ https://review.opendev.org/#/c/708972/ which adds a default site for static.opendev.org, somewhat in relation to that apache-config-choose-the-first-thing discussion yesterday23:32
ianwit exceeds the original spec, but i'm of the feeling we should just move all the sites on files02.openstack.org to static.opendev.org now, and get rid of one more host23:33
ianwotherwise it's just kicking the can on it until xenial removal time23:34
openstackgerritMonty Taylor proposed opendev/system-config master: Build gerrit images with bazelisk  https://review.opendev.org/70801823:37
*** dchen has joined #openstack-infra23:40
openstackgerritMerged openstack/project-config master: check-release-approval: retry once when Gerrit fails  https://review.opendev.org/70839723:40
openstackgerritJames E. Blair proposed ttygroup/gertty master: Add support for checks plugin  https://review.opendev.org/70899223:40
*** ahosam has quit IRC23:40
*** dave-mccowan has quit IRC23:51
*** dave-mccowan has joined #openstack-infra23:52
fungiianw: sure, taking a look now. need a break from reviewing conference talk abstracts anyway23:55
* fungi is suffering buzzword overload23:55
openstackgerritIan Wienand proposed opendev/system-config master: static: move site setup to a loop  https://review.opendev.org/70901623:55
fungiianw: also absolutely in favor of rehoming things off files.o.o, the two servers are filling essentially identical roles, just for different sets of sites23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!