Wednesday, 2020-01-15

*** rfolco has quit IRC00:06
fungithe purpose of timesyncd seems to be for systemd to put ntp out of business?00:06
fungi"systemd-timesyncd does no clock discipline: the clock is not trained or compensated, and internal clock drift over time is not reduced. It has rudimentary logic to adjust poll interval but without disciplining the host will end up with uneven time forever as systemd-timesyncd pushes or pulls at whatever interval it thinks the near-term drift requires. It also can't assess the quality of the remote00:08
fungitime source. You're unlikely to get accuracy much greater than 100ms."00:08
fungihttps://unix.stackexchange.com/questions/305643/ntpd-vs-systemd-timesyncd-how-to-achieve-reliable-ntp-syncing00:08
corvus+/- 7 minutes according to recent experience00:09
fungiindeed00:09
ianwwell i would be almost certain that was when my external screen refused to wake up (long standing bug) so i suspended my work vm and rebooted00:10
clarkbianw: is that screen hooked up via displayport and you are using intel integrated graphics?00:11
clarkbif so I've had reasonable luck turning the monitor off and on again a few times and eventually the right events get through and it starts working again :/00:11
fungii run into that occasionally, and can solve it by blindly typing `xset dpms force on` into a terminal or over ssh from a separate system00:12
ianwit's https://bugzilla.redhat.com/show_bug.cgi?id=150633900:14
openstackbugzilla.redhat.com bug 1506339 in xorg-x11-drv-intel "[drm:intel_cpu_fifo_underrun_irq_handler [i915]] *ERROR* CPU pipe B FIFO underrun" [High,New] - Assigned to ajax00:14
ianwone of those great 2+ year bugs with no answer00:14
ianwi think this laptop is probably old enough for a lease refresh ... however, the probability a new laptop will bring even more issues is probably higher :)00:16
ianwclarkb: on your point with the le certs refreshing at the same time -- possibly they do do a no-op task and then call a notify for a common restart handler?00:18
clarkbianw: my laptop is now 5 years old and ya the problems you know seem safer than new problems00:18
clarkbianw: I think that concern is far less of an issue with the switch to graceful apache relods00:18
clarkbwe should be able to queue those up several deep and not worry00:18
ianwyeah, i should cycle back on the listen: thing at the top of that file now we're using later ansible versions00:18
*** ricolin has joined #openstack-infra00:37
openstackgerritMerged openstack/project-config master: copy-wheels : generate indexes version 2  https://review.opendev.org/70253100:46
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: package-afs-centos: fix centos8 build  https://review.opendev.org/70256201:05
*** adriant has quit IRC01:36
*** iokiwi has quit IRC01:36
*** jamesmcarthur has joined #openstack-infra01:37
*** michael-beaver has quit IRC01:40
*** jamesmcarthur has quit IRC01:44
*** adriant has joined #openstack-infra01:48
*** hongbin has joined #openstack-infra02:02
*** hongbin has quit IRC02:02
*** hongbin has joined #openstack-infra02:03
*** hongbin has quit IRC02:03
*** hongbin has joined #openstack-infra02:04
*** hongbin has quit IRC02:04
*** hongbin has joined #openstack-infra02:06
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: package-afs-centos: fix centos8 build  https://review.opendev.org/70256202:09
openstackgerritMerged openstack/devstack-gate master: Use python3 for virtualenv if exists  https://review.opendev.org/70212602:12
*** hamzy_ has joined #openstack-infra02:15
openstackgerritsebastian marcet proposed openstack/project-config master: Removed OpenstackId legacy jobs  https://review.opendev.org/70257202:23
openstackgerritsebastian marcet proposed openstack/openstack-zuul-jobs master: Removed OpenstackId Legacy Jobs  https://review.opendev.org/70257302:24
*** jamesmcarthur has joined #openstack-infra02:25
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Update ansible-lint  https://review.opendev.org/70257402:26
*** jamesmcarthur_ has joined #openstack-infra02:28
*** jamesmcarthur has quit IRC02:28
openstackgerritsebastian marcet proposed openstack/project-config master: Removed OpenstackId legacy jobs  https://review.opendev.org/70257202:30
*** roman_g has quit IRC02:34
openstackgerritsebastian marcet proposed openstack/project-config master: Removed OpenstackId legacy jobs  https://review.opendev.org/70257202:35
openstackgerritsebastian marcet proposed openstack/openstack-zuul-jobs master: Removed OpenstackId Legacy Jobs  https://review.opendev.org/70257302:35
openstackgerritMerged openstack/openstack-zuul-jobs master: package-afs-centos: fix centos8 build  https://review.opendev.org/70256202:44
openstackgerritIan Wienand proposed openstack/project-config master: Update ansible-lint to version 4  https://review.opendev.org/70257502:45
openstackgerritIan Wienand proposed openstack/project-config master: copy-wheels: more fixes  https://review.opendev.org/70257602:50
ianwthis time for sure02:50
*** dychen has joined #openstack-infra02:56
*** dchen has quit IRC02:59
*** rlandy has quit IRC03:20
*** factor has quit IRC03:27
*** dave-mccowan has quit IRC03:29
openstackgerritMerged openstack/project-config master: copy-wheels: more fixes  https://review.opendev.org/70257603:34
*** tkajinam has quit IRC03:48
*** dave-mccowan has joined #openstack-infra03:53
*** dingyichen has joined #openstack-infra03:54
*** dychen has quit IRC03:56
*** gyee has quit IRC03:56
openstackgerritIan Wienand proposed opendev/system-config master: openafs-client: add centos8  https://review.opendev.org/70257904:07
*** jamesmcarthur_ has quit IRC04:08
*** jamesmcarthur has joined #openstack-infra04:10
*** rh-jelabarre has quit IRC04:11
*** hongbin_ has joined #openstack-infra04:12
*** hongbin has quit IRC04:15
*** tkajinam has joined #openstack-infra04:28
*** hongbin_ has quit IRC04:37
*** dave-mccowan has quit IRC04:42
*** jamesmcarthur has quit IRC04:43
*** jamesmcarthur has joined #openstack-infra04:44
*** zxiiro has quit IRC04:48
*** jamesmcarthur has quit IRC04:50
openstackgerritIan Wienand proposed opendev/system-config master: openafs-client: add centos8  https://review.opendev.org/70257905:03
openstackgerritIan Wienand proposed openstack/project-config master: copy-wheel: package install needs to be root  https://review.opendev.org/70258305:06
*** jamesmcarthur has joined #openstack-infra05:15
openstackgerritIan Wienand proposed opendev/system-config master: openafs-client: add centos8  https://review.opendev.org/70257905:15
*** rosmaita has quit IRC05:16
*** jamesmcarthur has quit IRC05:23
openstackgerritMerged openstack/project-config master: copy-wheel: package install needs to be root  https://review.opendev.org/70258305:24
openstackgerritMerged openstack/openstack-zuul-jobs master: Require JDK 8 for translations  https://review.opendev.org/70251505:27
*** rosmaita has joined #openstack-infra05:30
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-infra05:34
*** jamesmcarthur has joined #openstack-infra05:57
*** jamesmcarthur has quit IRC06:02
*** mugsie has quit IRC06:11
*** mugsie has joined #openstack-infra06:14
*** lmiccini has joined #openstack-infra06:24
*** jtomasek has joined #openstack-infra06:28
*** auristor has quit IRC06:29
*** jamesmcarthur has joined #openstack-infra06:36
*** auristor has joined #openstack-infra06:39
*** jamesmcarthur has quit IRC06:42
*** dmsimard has quit IRC06:50
*** dmsimard has joined #openstack-infra06:54
openstackgerritIan Wienand proposed openstack/project-config master: copy-wheels: fix index build  https://review.opendev.org/70259107:07
*** apetrich has quit IRC07:11
*** iurygregory has quit IRC07:13
*** kjackal has joined #openstack-infra07:15
*** KeithMnemonic1 has joined #openstack-infra07:15
*** KeithMnemonic has quit IRC07:19
openstackgerritMerged openstack/project-config master: Use Bionic for translation jobs  https://review.opendev.org/70251607:23
*** ralonsoh has joined #openstack-infra07:27
*** jtomasek has quit IRC07:29
*** jtomasek has joined #openstack-infra07:29
AJaegeramotoki: thanks for the Pint change - we'll go to python 3.6 anyway ^07:37
*** jamesmcarthur has joined #openstack-infra07:38
*** jamesmcarthur has quit IRC07:43
openstackgerritMerged openstack/project-config master: copy-wheels: fix index build  https://review.opendev.org/70259107:45
*** tesseract has joined #openstack-infra07:58
*** tesseract has quit IRC07:58
*** tesseract has joined #openstack-infra07:59
*** kjackal has quit IRC08:01
*** lxkong has quit IRC08:06
*** slaweq has joined #openstack-infra08:09
*** pgaxatte has joined #openstack-infra08:17
*** tosky has joined #openstack-infra08:19
*** dingyichen has quit IRC08:19
*** arxcruz|off is now known as arxcruz08:20
*** iurygregory has joined #openstack-infra08:21
*** rpittau|afk is now known as rpittau08:22
*** dchen has joined #openstack-infra08:22
*** dchen has quit IRC08:23
*** dpawlik has joined #openstack-infra08:25
*** sshnaidm|afk is now known as sshnaidm08:38
*** jamesmcarthur has joined #openstack-infra08:39
*** kjackal has joined #openstack-infra08:42
*** jamesmcarthur has quit IRC08:45
*** tkajinam has quit IRC08:48
*** jpena|off is now known as jpena08:50
*** dmellado has quit IRC08:51
openstackgerritJan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler  https://review.opendev.org/62147908:55
*** dmellado has joined #openstack-infra08:55
*** gfidente is now known as gfidente|afk08:56
*** aedc has quit IRC09:07
*** ccamacho has joined #openstack-infra09:15
*** lucasagomes has joined #openstack-infra09:16
*** apetrich has joined #openstack-infra09:21
*** xek has joined #openstack-infra09:25
openstackgerritIan Wienand proposed openstack/project-config master: copy-wheels: fix index  https://review.opendev.org/70260409:31
*** derekh has joined #openstack-infra09:36
*** tonyb has joined #openstack-infra09:38
openstackgerritIan Wienand proposed openstack/project-config master: copy-wheels: fix index path, remove f-strings  https://review.opendev.org/70260409:40
*** jamesmcarthur has joined #openstack-infra09:41
*** pcaruana has joined #openstack-infra09:42
*** jamesmcarthur has quit IRC09:46
*** iurygregory is now known as iury_mtg09:50
openstackgerritMerged openstack/project-config master: copy-wheels: fix index path, remove f-strings  https://review.opendev.org/70260410:04
*** jaosorior has joined #openstack-infra10:12
openstackgerritSorin Sbarnea proposed openstack/project-config master: Update ansible-lint to version 4.2  https://review.opendev.org/70257510:14
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227510:26
AJaegeryeah, translation jobs are working again!10:30
AJaegeramotoki: ^10:30
*** dtantsur|afk is now known as dtantsur10:32
openstackgerritSorin Sbarnea proposed openstack/project-config master: Update ansible-lint to version 4.2  https://review.opendev.org/70257510:33
openstackgerritMatthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect  https://review.opendev.org/70197210:34
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227510:34
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder master: DNM try breaking the ironic jobs  https://review.opendev.org/70264010:37
*** ahosam has joined #openstack-infra10:38
*** exsdev has quit IRC10:39
*** jamesmcarthur has joined #openstack-infra10:42
*** jamesmcarthur has quit IRC10:46
openstackgerritMerged openstack/devstack-gate master: nova: Enable n-cauth on stable/rocky for cells v1 jobs  https://review.opendev.org/70208911:03
*** melwitt has quit IRC11:09
*** pcaruana has quit IRC11:12
*** dchen has joined #openstack-infra11:21
openstackgerritTobias Henkel proposed zuul/zuul master: Document the buildsets endpoint  https://review.opendev.org/70212711:30
openstackgerritTobias Henkel proposed zuul/zuul master: Support dynamic badges  https://review.opendev.org/70212811:30
*** pkopec has joined #openstack-infra11:34
*** tbachman has quit IRC11:39
*** jamesmcarthur has joined #openstack-infra11:43
*** jaosorior has quit IRC11:46
*** gfidente|afk is now known as gfidente11:47
*** dave-mccowan has joined #openstack-infra11:48
*** jamesmcarthur has quit IRC11:48
dulekWe're seeing some missing packages on CentOS 8 mirror used by the gate (glibc, glibc-common, glibc-minimal-langpack, platform-python, python3-libs). Is there an OpenDev mirror, or is it more of a real mirror issue?11:49
*** dchen has quit IRC11:59
*** iury_mtg has quit IRC12:00
*** iurygregory has joined #openstack-infra12:01
zbrianw: https://review.opendev.org/#/c/702575/ is ready.  if you need help with ansible-lint in any other place ask me.12:03
*** rfolco has joined #openstack-infra12:06
*** sshnaidm is now known as sshnaidm|afk12:09
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227512:18
*** ociuhandu has joined #openstack-infra12:26
*** jpena is now known as jpena|lunch12:33
jrossergetting some unexpected log-files-rendered-as-gzip, like this https://zuul.opendev.org/t/openstack/build/2f9ff8b43a094518a2094471f7fadb61/log/logs/host/python_venv_build.log.txt.gz12:35
*** jamesmcarthur has joined #openstack-infra12:44
*** udesale has joined #openstack-infra12:45
amotokiAJaeger: you're welcome. it was a simple fix.12:46
amotokiAJaeger: btw, both changes on Pint have not been merged yet. how did you fix the translation jobs?12:47
*** pcaruana has joined #openstack-infra12:49
amotokiah.. I found https://review.opendev.org/702516 has been merged.12:49
*** jamesmcarthur has quit IRC12:49
AJaegeramotoki: yes. Still your Pint changes is good to have.12:56
amotokiAJaeger: yeah. it was a good chance to know Pint is no longer needed in horizon :)12:57
amotokiAJaeger: re: translation jobs, if we cannot change ubuntu version due to JDK reqs of zanata-cli, we can use PPT from "deadsnakes" team who provides other python versions for ubuntu https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa12:58
*** cgoncalves has quit IRC12:58
*** ociuhandu has quit IRC12:59
AJaegeramotoki: we are now on Bionic, so we're fine for a bit - but question is what to do with zanata going forward since it's dead...12:59
*** cgoncalves has joined #openstack-infra13:00
*** rh-jelabarre has joined #openstack-infra13:00
*** cgoncalves has quit IRC13:04
amotokiAJaeger: yes, I know we succeeded to migrate the translation job node to bionic.13:05
amotokiAJaeger:  honestly I have no idea on the future of zanata...... I don't have priority on i18n now unfortunately, so I can say nothing.13:05
*** rlandy has joined #openstack-infra13:05
AJaegersame here13:09
*** cgoncalves has joined #openstack-infra13:10
*** jamesmcarthur has joined #openstack-infra13:15
*** eharney has quit IRC13:20
*** dpawlik has quit IRC13:23
*** jamesmcarthur has quit IRC13:24
*** jamesmcarthur has joined #openstack-infra13:24
*** aedc has joined #openstack-infra13:25
*** ociuhandu has joined #openstack-infra13:27
smcginnisNo one is active on Zanata anymore?13:29
*** dave-mccowan has quit IRC13:33
*** jpena|lunch is now known as jpena13:35
*** jamesmcarthur has quit IRC13:36
amotokiAFAIK redhat closed zanata development two? years ago, so zanata no longer has active developers.13:36
smcginnisYikes13:37
smcginnisI didn't see anything in zanata-cli that looked too complicated to not be able to work with JDK 11, but it's been many years since I've touched Java code.13:37
smcginnisMaybe worth sending something to the ML looking for anyone interested in picking up some work there to get it updated?13:38
*** dave-mccowan has joined #openstack-infra13:39
*** liuyulong has joined #openstack-infra13:42
amotokiif it is only about zanata-cli, we can directly talk with zanata, for example, using requests. I think most worried part is the server side of zanata.13:46
*** jamesmcarthur has joined #openstack-infra13:47
amotokiI agree it is worth sending a mail for volunteer(s). I've not be involved in Zanata in OpenStack. I think Ian or Frank can raise it.13:51
*** mnaser has quit IRC13:54
*** mnaser has joined #openstack-infra13:55
*** rfolco has quit IRC14:00
*** rfolco has joined #openstack-infra14:01
*** jackedin has joined #openstack-infra14:01
*** eharney has joined #openstack-infra14:01
*** hrybacki has quit IRC14:03
*** hrybacki has joined #openstack-infra14:03
*** rcernin has quit IRC14:04
*** jackedin_ has joined #openstack-infra14:15
*** avladu has joined #openstack-infra14:17
*** jackedin has quit IRC14:17
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227514:18
avladuhello, can someone review and +2/workflow, please? https://review.opendev.org/#/c/702259/ It already has one +214:19
*** jungleboyj has quit IRC14:20
*** jungleboyj has joined #openstack-infra14:21
*** jackedin_ has quit IRC14:21
*** jackedin has joined #openstack-infra14:24
*** sshnaidm|afk is now known as sshnaidm14:39
openstackgerritLucian Petrut proposed openstack/project-config master: Update compute-hyperv test jobs  https://review.opendev.org/70267314:40
*** jamesmcarthur has quit IRC14:40
*** lseki has quit IRC14:44
*** diablo_rojo_phon has quit IRC14:44
*** lseki has joined #openstack-infra14:44
*** dpawlik has joined #openstack-infra14:45
*** diablo_rojo_phon has joined #openstack-infra14:49
*** electrofelix has joined #openstack-infra14:49
*** smarcet has joined #openstack-infra14:50
*** aedc has quit IRC14:52
openstackgerritLucian Petrut proposed openstack/project-config master: Update compute-hyperv test jobs  https://review.opendev.org/70267314:57
*** ociuhandu has quit IRC15:01
openstackgerritMerged openstack/project-config master: New charm for watcher  https://review.opendev.org/70207115:01
*** rfolco is now known as rfolco|bbl15:03
*** avladu has quit IRC15:03
*** apetrich has quit IRC15:07
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Bump ansible-lint to 4.2.0  https://review.opendev.org/70267915:07
*** eernst has quit IRC15:07
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application  https://review.opendev.org/70210615:07
fungismcginnis: amotoki: yes, we've already had a fairly major security vulnerability crop up in zanata after it was abandoned: https://review.opendev.org/69347715:11
fungidulek: do you have an example log where it shows the failure to retrieve those packages? i can take a look15:11
smcginnisYeee15:12
dulekfungi: Sure, on the bottom: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_408/696657/18/check/kuryr-kubernetes-tempest-containerized/408c49d/controller/logs/devstacklog.txt15:12
openstackgerritMerged zuul/zuul-helm master: Add option to manage secrets outside of helm  https://review.opendev.org/70205215:13
dulekfungi: But now I wonder if it wasn't caused by recent update to centos containers on DockerHub and 8.1 release.15:13
dulekfungi: I'm trying to figure that out at the moment.15:13
dulekfungi: BTW some context - this happens during container build.15:13
fungijrosser: there was a necessary change in handling of compressed log files, jobs should not compress logs on their own now if you want them to be viewable in a browser (this was to solve a problem where tarballs were getting silently decompressed on download without changing their filenames to drop the .gz extension)15:13
jrosserhmm ok we need to change a *lot* of stuff then :)15:14
openstackgerritMerged zuul/zuul-helm master: Add Zuul charts  https://review.opendev.org/70046015:14
openstackgerritMerged zuul/zuul-helm master: Allow tenant config file to be managed externally  https://review.opendev.org/70205715:14
openstackgerritMerged zuul/zuul-helm master: Allow override of service type for zuul-web  https://review.opendev.org/70210115:14
fungijrosser: zuul will still compress the logs when it uploads them to swift, and swift will serve them with appropriate content encoding headers15:14
fungiwe patched the common cases we knew about, but there are likely some project-specific jobs which also need adjusting15:15
jrossersure, i think i know where ours are15:15
fungidulek: i think you're right. our mirror is for centos 8.1 not 8.0, so has for example glibc-2.28-42.el8.1.x86_64.rpm (el8.1 not el8)15:19
fungiwe mirror and build base images for whatever the latest centos 8 version is, we don't maintain separate images and mirrors for minor release versions of centos15:19
fungihttp://files.openstack.org/mirror/centos/8/BaseOS/x86_64/os/Packages/15:20
fungiin case you wanted to inspect what's in there15:20
dulekfungi: Okay, I'll try to figure this out and will update you if I'll make any sense of it. At the moment it seems to me like DockerHub just doesn't have x86 images for 8.1, but I'm getting more and more confused here.15:20
dulekfungi: Thanks!15:20
*** jamesmcarthur has joined #openstack-infra15:22
fungican't 8.0 install 8.1 packages?15:22
*** ociuhandu has joined #openstack-infra15:28
*** pkopec_ has joined #openstack-infra15:29
*** pkopec has quit IRC15:29
dulekThose packages have versions defined in dependencies, so I'm guessing there's a mismatch there.15:30
*** dpawlik has quit IRC15:33
openstackgerritMerged openstack/project-config master: x/cloudbase-init: remove openstack gates  https://review.opendev.org/70225915:37
*** ociuhandu has quit IRC15:39
*** ociuhandu has joined #openstack-infra15:40
*** eharney has quit IRC15:48
*** eharney has joined #openstack-infra15:52
*** udesale_ has joined #openstack-infra15:54
*** efried is now known as efried_afk15:56
*** ykarel has joined #openstack-infra15:57
*** udesale has quit IRC15:57
*** zxiiro has joined #openstack-infra15:57
clarkbdulek: fungi is it possible the centos repo is in a transition state between 8.0 and 8.1?15:59
clarkbwe've seen that happen in the past with other centos releases I think15:59
fungii wouldn't rule it out15:59
fungibasically centos periodically breaks the ability of their users to perform package updates while they're switching between minor versions?16:00
*** melwitt has joined #openstack-infra16:00
fungidebian does this, but the impact is limited to the unstable suite16:00
dulekclarkb: Well, technically it all works fine on my local system. How often are infra mirrors updated?16:01
* fungi checks16:01
clarkbI believe every 4 hours16:01
clarkband we pull from a second level mirror too16:01
clarkbso depending on when that mirror updates it could compound16:01
fungi43 */2 * * *16:01
clarkbits pretty easy to compare the package lists upstream16:02
fungiso 43 minutes after the hour every two hours (on even hours)16:02
dulekOkay, so that would get fixed if that was the only reason.16:02
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227516:02
fungilast vos release for that volume was 2020-01-15T15:37:4716:02
fungiutc16:02
fungiso less than half an hour ago16:03
*** gyee has joined #openstack-infra16:03
clarkbcompare http://mirror.dfw.rax.openstack.org/centos/8/BaseOS/x86_64/os/Packages/ to http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/16:03
clarkbthe first package in the list is different16:03
fungion that pulse, rsync said:16:04
clarkbso ya that implies to me that there is desync happening16:04
fungisent 581 bytes  received 2,336,497 bytes  150,779.23 bytes/sec16:04
fungitotal size is 43,538,322,424  speedup is 18,629.3816:04
fungiwe retrieved from mirror.dal.nexril.net16:05
fungiso it may have synced from elsewhere mid-update too16:05
openstackgerritMerged zuul/zuul master: Fix nits from initial reorg  https://review.opendev.org/70253216:06
fungihttp://mirror.dal.nexril.net/centos/8/BaseOS/x86_64/os/Packages/16:06
*** armax has joined #openstack-infra16:07
fungiin theory our mirrors ~match that one for the moment16:07
dulekJust a data point - this happens for at least 7 hours.16:07
fungiyeah, i'm starting to suspect that mirror.dal.nexril.net grabbed a snapshot of the master centos mirror when it was in the middle of getting centos 8.0 packages swapped out for 8.116:08
fungiso we're synchronizing to a broken "official" centos mirror16:08
fungi(in theory)16:09
*** kjackal has quit IRC16:09
*** kjackal has joined #openstack-infra16:09
dulekmirror.dal.nexril.net seems different than mirror.centos.org - i.e. it has el8.1 packages.16:10
clarkbdulek: yes see above :)16:10
clarkbdulek: and we then sync from that16:10
clarkbso we are different16:10
clarkb(we don't pull directly from upstream because only those official mirrors are supposed to do that aiui)16:11
dulekclarkb: Yes, yes, I'm just trying to figure out if you will keep mirroring 8.0 or want to switch to 8.1?16:11
clarkbdulek: we want to track "8" which means latest 8.x16:11
clarkbbecause older 8.y is no longer supported when newer 8.x is released16:11
clarkbwe can wait for our upstream to catch up or look for a better upstream16:12
dulekclarkb: Okay, good. And the centos.org centos/8/BaseOS works for us.16:12
clarkbanother option would be to contact that mirror and ask them to resync16:12
dulekclarkb: I guess making that decision is our of my competence level. This breaks kuryr-kubernetes CI, but most contributors are from Europe and it's already EOD here, so I guess no hurry.16:14
fungior we could switch to a different rsync mirror which isn't broken16:14
clarkbfungi: ya I want to say we've used limestone in the past due to locality and having logan- here16:15
clarkbI don't recall what prompted the switch from that mirror16:15
fungithere's also a long-standing todo to add a package repository checker which would analyze the dependencies in the rpms and refuse to release the volume until they reach consistency16:15
*** lmiccini has quit IRC16:16
openstackgerritMerged openstack/project-config master: New repo: "ideas"  https://review.opendev.org/70153016:16
*** jackedin has quit IRC16:17
*** hrw has joined #openstack-infra16:21
hrwmorning16:21
hrwcan someone help me with understanding http://zuul.openstack.org/buildset/2e228315d13f4c7a887734e74fded1d5 failure? was because kolla-publish-debian-source-multiarch-aarch64 ended with POST_FAILURE because it took 5h? or was it other reason?16:22
hrwif it is this then I will remove that job16:22
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Making zuul-jobs-test-multinode-roles-opensuse-tumbleweed nv  https://review.opendev.org/70269316:23
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Bump ansible-lint to 4.2.0  https://review.opendev.org/70267916:24
clarkbhrw: 2020-01-15 10:46:04.526946 | POST-RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/kolla/tests/playbooks/publish.yml@master]16:25
clarkbfrom the job-output.txt file16:25
hrwthanks. somehow missed that part16:25
evrardjpthanks clarkb AJaeger16:25
openstackgerritMatthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect  https://review.opendev.org/70197216:26
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227516:26
clarkbseems it spent three hours pushing docker images16:26
fungivery large images?16:26
hrwI suspect that linaro-london pipe is new bottleneck there16:26
hrwsame amount pushed from x86-64 job in ~50 minutes16:27
hrwwill discuss it internally and remove job from CI as it makes no sense to keep it16:27
*** pgaxatte has quit IRC16:30
openstackgerritDavid Shrewsbury proposed zuul/zuul master: Extract project config YAML into ref docs  https://review.opendev.org/70197716:32
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application  https://review.opendev.org/70210616:32
funginow that the zuul operator call has wrapped up, i'm going to go out for lunch but should be back in an hour16:34
openstackgerritsebastian marcet proposed openstack/project-config master: Removed OpenstackId legacy jobs  https://review.opendev.org/70257216:38
*** hrw has left #openstack-infra16:38
*** ociuhandu has quit IRC16:39
*** ociuhandu has joined #openstack-infra16:39
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Making zuul-jobs-test-multinode-roles-opensuse-tumbleweed nv  https://review.opendev.org/70269316:39
dulekfungi, clarkb: Seems like mirror.dal.nexril.net just resynched correctly.16:40
zbrAJaeger: does https://review.opendev.org/#/c/702693 makes sense?16:41
zbrit should prevent other accidental modification of that file.16:41
*** lsell has quit IRC16:42
*** lsell has joined #openstack-infra16:42
*** openstackgerrit has quit IRC16:43
*** pcaruana has quit IRC16:45
*** udesale_ has quit IRC16:45
clarkbdulek: cool and I think at 16:43UTC we would've started a new sync so we should be up to date shortly16:46
dulekclarkb: Yup! Meanwhile it seems that Kuryr gate works, so probably it was using another mirror, which suffered the same desynch.16:47
dulekWeird that yum does not indicate which mirror is used.16:48
*** openstackgerrit has joined #openstack-infra16:48
openstackgerritMerged zuul/zuul-jobs master: Stop compressing files during intermediate steps part 2  https://review.opendev.org/70128216:48
dulekAnyway thanks for help!16:48
*** clayg has quit IRC16:49
*** clayg has joined #openstack-infra16:49
*** beisner has quit IRC16:50
*** beisner has joined #openstack-infra16:50
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227516:50
*** eernst has joined #openstack-infra16:55
*** ociuhandu has quit IRC16:59
*** ykarel is now known as ykarel|away17:01
*** lucasagomes has quit IRC17:02
openstackgerritClĂ©ment Mondion proposed zuul/nodepool master: add requirements for json logging  https://review.opendev.org/70270417:03
*** iurygregory has quit IRC17:04
openstackgerritMerged opendev/system-config master: afs-client: move reduced cache to group variable  https://review.opendev.org/69812817:05
openstackgerritMerged opendev/system-config master: Periodically release additional AFS volumes  https://review.opendev.org/69916317:05
*** ykarel|away has quit IRC17:07
*** ociuhandu has joined #openstack-infra17:08
*** gmann has quit IRC17:09
*** gmann has joined #openstack-infra17:10
*** iurygregory has joined #openstack-infra17:14
*** jcoufal has joined #openstack-infra17:15
*** rpittau is now known as rpittau|afk17:15
*** tesseract has quit IRC17:26
*** iurygregory has quit IRC17:26
*** arxcruz is now known as arxcruz|off17:29
*** guilhermesp has quit IRC17:31
*** guilhermesp has joined #openstack-infra17:32
*** ociuhandu has quit IRC17:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-infra17:34
*** dtantsur is now known as dtantsur|afk17:35
fungidulek: i would take that as a sign that it's worth looking into why those jobs aren't using the local ci mirrors as well. you may get some helpful speed improvements from fixing that17:49
fungiand possibly stability improvements17:49
*** npochet has quit IRC17:49
fungibut also that's a great demonstration that our mirrors are only as good as whatever they're mirroring from17:49
*** smarcet has quit IRC17:49
*** npochet has joined #openstack-infra17:50
*** jpena is now known as jpena|off17:57
*** eernst has quit IRC17:58
*** eernst has joined #openstack-infra18:00
*** electrofelix has quit IRC18:07
*** derekh has quit IRC18:08
clarkbthinking out loud here. I wonder if centos updates so infrequently that we should just update after each minor release?18:08
clarkbI don't know how often packages get updated between releases18:09
clarkbI guess the risk with that is if our images update before our mirror updates then we have newer packages on the images than in the mirror18:10
*** ociuhandu has joined #openstack-infra18:14
*** smarcet has joined #openstack-infra18:15
*** xek_ has joined #openstack-infra18:18
*** eernst has quit IRC18:18
*** eernst has joined #openstack-infra18:19
*** ociuhandu has quit IRC18:19
*** xek has quit IRC18:20
*** smarcet has quit IRC18:21
openstackgerritClark Boylan proposed openstack/project-config master: Revert "Disable compression of files in stage-output role"  https://review.opendev.org/70271518:22
fungiwell, and more specifically that attempts to install additional packages result in the packages on the mirror depending on older versions of things than are already installed on the rootfs, leading to yum/dnf failing to install them18:22
*** smarcet has joined #openstack-infra18:23
clarkbinfra-root ^ that should conclude our compression settings changes. zuul-jobs now sets that var to false by default so we can rely on role defaults18:23
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Update sdk to v0.13.0 and install dhall-to-json  https://review.opendev.org/70210318:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Import dhall-operator  https://review.opendev.org/70210418:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Import zuul application from dhall-zuul  https://review.opendev.org/70210518:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application  https://review.opendev.org/70210618:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Generate TLS certificats for the gearman service  https://review.opendev.org/70271618:25
*** eharney has quit IRC18:26
sean-k-mooneydonnyd: is FN still disabled as a provider by the way or have you been able to resolve your power issues18:26
*** eernst has quit IRC18:28
*** eernst has joined #openstack-infra18:28
*** rfolco|bbl is now known as rfolco18:28
sean-k-mooneylooking at http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 ill assume it is disabled18:29
*** TheJulia has quit IRC18:29
*** mwhahaha has quit IRC18:30
*** rpittau|afk has quit IRC18:30
*** vdrok has quit IRC18:30
*** cjohnston has quit IRC18:30
*** seongsoocho has quit IRC18:30
*** masayukig has quit IRC18:30
*** evgenyl has quit IRC18:30
*** coreycb has quit IRC18:30
*** cyberpear has quit IRC18:30
*** zxiiro has quit IRC18:30
*** donnyd has quit IRC18:30
*** csatari has quit IRC18:30
*** jamespage has quit IRC18:30
*** thedac has quit IRC18:30
*** tdasilva has quit IRC18:30
*** sdoran has quit IRC18:30
*** npochet has quit IRC18:30
*** ildikov has quit IRC18:31
*** srwilkers has quit IRC18:31
*** arne_wiebalck_ has joined #openstack-infra18:31
*** rpioso has quit IRC18:31
*** mnaser has quit IRC18:31
*** thedac has joined #openstack-infra18:31
*** MarkMaglana has quit IRC18:31
*** donnyd has joined #openstack-infra18:31
*** davidlenwell has quit IRC18:31
*** tonyb[m] has quit IRC18:31
*** arne_wiebalck has quit IRC18:31
*** Shrews has quit IRC18:31
*** arne_wiebalck_ is now known as arne_wiebalck18:31
*** jamespage has joined #openstack-infra18:32
*** diablo_rojo_phon has quit IRC18:32
*** TheJulia has joined #openstack-infra18:32
*** cjohnston has joined #openstack-infra18:32
*** cyberpear has joined #openstack-infra18:32
*** seongsoocho has joined #openstack-infra18:32
*** openstackstatus has quit IRC18:32
*** mwhahaha has joined #openstack-infra18:32
*** coreycb has joined #openstack-infra18:33
*** vdrok has joined #openstack-infra18:33
*** ildikov has joined #openstack-infra18:33
*** tdasilva has joined #openstack-infra18:33
*** csatari has joined #openstack-infra18:33
*** johnsom has quit IRC18:33
*** crodriguez has quit IRC18:33
*** tosky has quit IRC18:33
*** npochet has joined #openstack-infra18:33
clarkbsean-k-mooney: yes I believe it was disabled due to the mirror flakyness after power work18:34
*** mnaser has joined #openstack-infra18:34
*** eernst has quit IRC18:34
*** srwilkers has joined #openstack-infra18:34
*** crodriguez has joined #openstack-infra18:34
*** smarcet has quit IRC18:34
*** rpioso has joined #openstack-infra18:34
*** davidlenwell has joined #openstack-infra18:34
*** tonyb[m] has joined #openstack-infra18:35
*** Shrews has joined #openstack-infra18:35
*** sdoran has joined #openstack-infra18:35
*** johnsom has joined #openstack-infra18:35
*** masayukig has joined #openstack-infra18:35
*** sshnaidm is now known as sshnaidm|afk18:38
*** csatari has quit IRC18:38
*** eharney has joined #openstack-infra18:38
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Fix Yum repositories and GPG keys for CentOS 8.1  https://review.opendev.org/70272118:39
*** rcernin has joined #openstack-infra18:40
*** crodriguez has quit IRC18:40
*** TheJulia has quit IRC18:40
*** jamespage has quit IRC18:40
*** thedac has quit IRC18:40
*** lathiat has quit IRC18:40
*** donnyd has quit IRC18:40
*** thedac has joined #openstack-infra18:42
openstackgerritClark Boylan proposed openstack/diskimage-builder master: DNM debugging centos 8.1 updates  https://review.opendev.org/70272218:42
*** donnyd has joined #openstack-infra18:42
*** tdasilva has quit IRC18:43
*** cyberpear has quit IRC18:43
openstackgerritClark Boylan proposed openstack/diskimage-builder master: DNM debugging centos 8.1 updates  https://review.opendev.org/70272218:43
cgoncalvesclarkb, see https://review.opendev.org/70272118:43
clarkbcgoncalves: oh gool18:44
clarkber cool18:44
*** jamesmcarthur has quit IRC18:44
clarkbcgoncalves: that populates /etc/yum.repos.d/ I take it18:45
cgoncalvescorrect. tested building an octavia image too18:45
openstackgerritMerged openstack/project-config master: Revert "Disable compression of files in stage-output role"  https://review.opendev.org/70271518:46
*** slaweq has quit IRC18:47
sean-k-mooneyclarkb: cool thanks if you think of it can you ping me when its back. no rush just we have 1 job that only runs there however we dont have anything urgnet either so it can wait18:48
clarkbcgoncalves: re paul's question on that change shoudl we install those packages only if the version is exactl 8 or >=8.1?18:51
clarkbcgoncalves: excluding 8.0*18:52
cgoncalvesclarkb, do we need to support 8.0 when a newer .y version is out?18:52
clarkbcgoncalves: I think users shouldn't install 8.0 when a newer .y is available, but my hunch is that people will need it18:53
cgoncalvescentos-minimal installs centos "8" and that is latest .y18:53
clarkbcgoncalves: ya but you can set the dib version to 8.0 and get 8.0 I think18:53
clarkbor rather 8.0.190518:53
cgoncalvesnot so sure users can set different DIB_RELEASE other than "7" and "8"18:55
*** efried_afk is now known as efried18:55
clarkbcgoncalves: ok confirmed that centos-minimal README says set to 7 or 818:56
openstackgerritMerged opendev/system-config master: Update registry testing to use LE  https://review.opendev.org/70254118:56
clarkbgiven that I think it is ok to merge as is then if people want to add support for 8.0.* its another set of changes18:57
cgoncalvesright18:57
*** thedac has quit IRC18:58
clarkbI'll leave that note on the change18:58
zbrafaik, "centos-8" reads as latest 8.x version, a not 8.0 (since two hours ago). if anyone wants 8.0 exactly they will need new jobs. Makes sense?18:58
clarkbzbr: they will need to update dib yes18:58
zbrclarkb: I am watching your change.18:59
clarkbzbr: cgoncalves change is the fix https://review.opendev.org/70272118:59
*** thedac has joined #openstack-infra19:00
*** cyberpear has joined #openstack-infra19:00
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: install-kubernetes: disable docker userland proxy  https://review.opendev.org/70272319:01
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy  https://review.opendev.org/70272419:02
AJaegerclarkb: could you review https://review.opendev.org/#/c/702571/ , https://review.opendev.org/#/c/702572/ and https://review.opendev.org/#/c/702573/ for openstackid jobs, please?19:06
openstackgerritMerged zuul/zuul master: Add note about GitHub Action limitations  https://review.opendev.org/70250319:08
*** sshnaidm|afk is now known as sshnaidm19:09
*** thedac has quit IRC19:10
*** openstackstatus has joined #openstack-infra19:22
*** ChanServ sets mode: +v openstackstatus19:22
fungi#status log restarted statusbot service on eavesdrop.o.o to recover following a 18:32z ctcp ping timeout19:22
openstackstatusfungi: finished logging19:22
rosmaitahello infra people ... i need some help configuring the .zuul.yaml for python-cinderclient19:22
rosmaitai've got the functional-py36 job working, but the functional-py37 job is failing with a retry_limit19:22
rosmaitaonly difference in config is that the py37 job specifies nodeset: ubuntu-bionic19:22
rosmaitai'm seeing this when it does the write-devstack-local-conf task:19:22
rosmaitaThe task includes an option with an undefined variable. The error was: \"hostvars['controller']\" is undefined19:22
rosmaitaand then it skips the RUN phase and goes directly to POST-RUN19:22
rosmaitahere's the patch where it's happening: https://review.opendev.org/#/c/691467/19:22
rosmaitathanks!19:22
clarkbrosmaita: chances are there is no python3.7 available19:24
* clarkb loks19:24
clarkbrosmaita: ah yup its going to be the nodeset19:25
clarkbthis is a devstack based job so it expects a devstack compatible nodeset19:25
rosmaitashoot, i thought bionic always had py3.719:25
clarkbrosmaita: its installable but not there by default19:25
openstackgerritTristan Cacqueray proposed zuul/zuul master: Dockerfile: add support for arbritary uid  https://review.opendev.org/65024619:25
rosmaitaclarkb: can you point me to an example of how i would install it?19:26
clarkbrosmaita: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L11-L19 try using that nodeset19:26
rosmaitaty19:26
clarkbI think the tox py37 jobs are expected to list python37 packages in bindep19:27
clarkbI don't know if that has been plumbed into devstack at all19:27
clarkbI would see if the qa team knows19:27
rosmaitaclarkb: thanks19:28
fungiclarkb: i thought we got to the point where the ensure_python role was actually installing it, but a lot of the tox jobs in the openstack-zuul-jobs repo weren't setting the right variable to make that happen19:34
clarkboh that could be19:34
fungisimply because they predated that feature19:34
fungiand didn't get revisited19:34
clarkbrosmaita: ^ you may be able to use the ensure_python role in those jobs to get the version you want19:34
AJaegerrosmaita: see change I46d6780a5a32c443c37ef040b2af2459641dd123 as well, maybe setting python_version is enough19:36
*** iurygregory has joined #openstack-infra19:36
*** dklyle has quit IRC19:36
rosmaitaclarkb: ok, i will try that too -- will i still need to specify a nodeset, or will zuul figure that out?19:36
AJaeger(if your job uses ensure_python already)19:36
*** ociuhandu has joined #openstack-infra19:36
rosmaitaAJaeger: thanks, i am not using ensure_python19:36
clarkbrosmaita: you need to specify the nodeset because the devstack jobs expect a specific host topology19:36
clarkbrosmaita: you need the controller group at the very least (which the nodeset I linked to provides)19:37
rosmaitaok, cool, so both changes -- thanks!19:37
*** dklyle has joined #openstack-infra19:37
*** dklyle has quit IRC19:37
AJaegerrosmaita: openstack-zuul-jobs sets python_version for the functional job, so it should be installed...19:37
*** dklyle has joined #openstack-infra19:38
AJaegerrosmaita: no, just change the nodeset.19:38
AJaegerah, you're not parenting to openstack-tox-functional-py37, so then you need python_version19:39
*** jcoufal has quit IRC19:39
rosmaitaok19:40
AJaegerrosmaita: but all depends on the job inheritance. Sorry, no time to check this further.19:41
AJaegerI meant: no time tonight.19:41
rosmaitaok, well y'all have given me some stuff to try out19:41
rosmaitai will bug you tomorrow if i don't get it working :)19:41
AJaegerquick look: Seems ensure-python is not used19:41
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy  https://review.opendev.org/70272419:51
*** jamespage has joined #openstack-infra19:52
*** jamespage has quit IRC19:52
*** MarkMaglana has joined #openstack-infra20:07
*** TheJulia has joined #openstack-infra20:08
smcginnisI thought there was a patch up to make sure ensure_python installed the version if needed.20:12
smcginnisI at least remember it being discussed a few weeks back.20:13
clarkbsmcginnis: in devstack jobs?20:13
clarkbsmcginnis: I think that is the disconnnect here. That job is a devstack job20:13
smcginnisWherever that role is defined.20:13
clarkbthe ensure_python stuff that happened is in the tox jobs20:13
clarkbthey are disjoint because one basically just runs tox and the other installs openstack with devstack then runs stuff20:13
smcginnisI think it was the tox one. I can't recall the situation, but I think the failure under question was not a devstack job. Someone at least said "we should make sure the base ensure_python role installs the version if it's needed", but I don't recall the specifics now.20:14
*** ccamacho has quit IRC20:16
AJaegersmcginnis: see change I46d6780a5a32c443c37ef040b2af2459641dd123 for what we did20:17
ianwclarkb: i've made a mess of the wheel generation.  i now realise that we only create the final structure on afs which means very slow indexing.  i'm going to have to go for version 3 with larger changes to create and move folders rather than files20:17
ianwAJaeger: i know you looked at trusty some more last year ... can we kill it?20:18
smcginnisAJaeger: Thanks - that was the one.20:18
ianwit would be great to not build wheels for it as it's the only python3.5 (even centos7 has python3.6 in epel)20:18
clarkbianw: I think at this point trusty is largely a small number of infra jobs20:19
clarkbAJaeger cleaned up fueld which was a big consumer20:19
clarkband we've deleted the infra trusty jobs we don't need20:19
*** jamesmcarthur has joined #openstack-infra20:19
AJaegerianw: there are a few uses for infra jobs ;(20:19
clarkbAJaeger: ianw chances are we don't need the wheels in those jobs though20:20
clarkb(and simply rely on pypi)20:20
ianwok, well wheels probably aren't important there, i think we can kill that path20:20
rosmaitaclarkb: when you have a minute -- what would be the exact syntax for adding ensure-python to the job defined at line 22 here: https://review.opendev.org/#/c/691467/17/.zuul.yaml20:20
AJaegerianw: check openstack-zuul-jobs repo - that's the main user20:20
AJaegerhttps://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/zuul-legacy-jobs.yaml20:20
ianwthanks, will do20:20
clarkbrosmaita: you want to add it to https://opendev.org/openstack/python-cinderclient/src/branch/master/playbooks/python-cinderclient-functional.yaml#L220:21
ianwinfra-root: also, some eyes on https://review.opendev.org/#/c/702579/ to add centos8 openafs builds would be great20:21
fungiany trusty jobs which are lingering solely due to wiki.openstack.org can go away. it's not puppeted anyway, and the -dev server (which is somewhat puppeted) runs on xenial20:21
ianwthe only trick is that for some reason kstart isn't in epel8, i have a temporary copr repo with it and have filed and upstream bug20:21
AJaegerlegacy-gearman-plugin-maven-build-ubuntu-trusty, legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty, legacy-zmq-event-publisher-maven-build-ubuntu-trusty20:22
clarkbianw: see also dib centos-8 problems. cgoncalves wrote a fix which I have approved. Will need a release20:22
clarkbAJaeger: zmq-event-publisher can be removed20:22
clarkbAJaeger: someone on the internet forked it from us and took it over for jenkins20:22
clarkb(can probably treat gearman plugin similarly though I don't think anyone took it over)20:23
AJaegerso, should we retire that repo?20:23
AJaegerlet me remove jobs...20:23
clarkbrosmaita: I think you can drop the nodeset entirely too fwiw. Now that I see the python36 job relies on the devstack default20:23
rosmaitaclarkb: well, the nodeset did make the py37 job pass finally20:24
clarkbrosmaita: have you run it with no nodeset though?20:24
clarkbrosmaita: specifying the wrong nodeset is what broke it before. I think specifying no nodest will have it use the default for devstack which should work20:24
ianwclarkb: will do on dib.  just have to decamp as movers are coming soon, bib20:25
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add noop-jobs for zmq-event-publisher  https://review.opendev.org/70274520:26
AJaegerclarkb, ianw, ^20:26
clarkbianw: one minor thing on https://review.opendev.org/#/c/702579/3 if it ends up needing a new ps20:26
AJaegerclarkb: shall we retire completely?20:26
rosmaitaclarkb: with no nodeset the functional-py37 job skipped all the tests (couldn't find py3.7 interpreter)20:26
clarkbAJaeger: I think we can, people would be consuming the built artifacts and not git if still using our stuff20:27
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove zmq-event-publisher jobs  https://review.opendev.org/70274720:28
AJaegerclarkb: ok, let me send an email and do the next steps then...20:28
clarkbrosmaita: ok. That surprises me because we don't install python37 by default but maybe devstack is grabbing it20:28
clarkbrosmaita: it also doesn't hurt to have the nodeset there20:29
rosmaitaclarkb: ok, cool20:32
AJaegerclarkb: do you know the new name and location?20:34
openstackgerritAndreas Jaeger proposed x/zmq-event-publisher master: Retire zmq-event-publisher  https://review.opendev.org/70274820:35
*** eharney has quit IRC20:36
*** slaweq has joined #openstack-infra20:36
*** iurygregory has quit IRC20:37
openstackgerritAndreas Jaeger proposed openstack/project-config master: Retire x/zmq-event-publisher  https://review.opendev.org/70274920:38
AJaegerclarkb: pushed everything ^, somebody needs to abandon the two open jobs in https://review.opendev.org/#/q/project:x/zmq-event-publisher+is:open20:39
openstackgerritAndreas Jaeger proposed openstack/project-config master: Retire x/zmq-event-publisher  https://review.opendev.org/70274920:42
clarkbAJaeger: I'm not finding the new location. I think it may hve just been a github fork?20:45
*** aedc has joined #openstack-infra20:49
openstackgerritJames E. Blair proposed zuul/zuul master: Re-elect James Blair as project lead  https://review.opendev.org/70126720:51
smcginnisI see clarkb's - https://github.com/cboylan/zmq-event-publisher20:52
smcginnisAnd a related project - https://github.com/KuraudoTama/jenkins-events-handlers20:53
*** aedc has quit IRC20:54
*** aedc has joined #openstack-infra20:54
clarkbya I'm not finding the fork, also searching email isn't turning it up. I may have misremembered20:54
clarkbhowever we haven't used that software in many years20:54
clarkbI think we can retire it20:54
smcginnisYeah, even that last link that was working along with it doesn't look like it's been touched since 2015.20:55
*** ralonsoh has quit IRC20:57
openstackgerritMatthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option  https://review.opendev.org/70227520:58
*** tdasilva has joined #openstack-infra21:00
AJaegerclarkb: Antoine Musso had a patch for it recently, what was his nick?21:03
fungihashar21:03
AJaegernot around - thanks, fungi21:03
clarkbin 201721:03
AJaegeroh, indeed21:04
clarkbI think either this will not cause any problems or someone will notice and then we can resurrect it properly with them leading the way21:04
clarkbeither way I think this is a good next step21:04
AJaegerok21:04
AJaegerthat leaves two more jobs on trusty...21:05
AJaegerlegacy-gearman-plugin-maven-build-ubuntu-trusty, legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty21:05
fungioof21:05
fungithose are some lengthy job names21:06
*** jamesmcarthur has quit IRC21:06
AJaegersecond one used here: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L803 - on opendev/puppet-openstack_infra_spec_helper21:06
AJaegerfirst one on x/gearman-plugin, https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L535921:07
AJaegerdo we need the repos - or should we just drop jobs?21:07
*** smarcet has joined #openstack-infra21:09
* AJaeger calls it a day, if somebody wants to cleanup, please go for it. I'll happily review tomorrow21:09
*** jamesmcarthur has joined #openstack-infra21:10
clarkbAJaeger: I've abandoned the zmq changes21:12
openstackgerritMerged openstack/project-config master: Add noop-jobs for zmq-event-publisher  https://review.opendev.org/70274521:13
clarkbAJaeger: for infra-spec helper we should make sure there is a xenial job which is legacy-puppet-openstack-infra-spec-helper-unit so I think we can delete that trusty job21:13
*** lseki has quit IRC21:14
openstackgerritMerged x/zmq-event-publisher master: Retire zmq-event-publisher  https://review.opendev.org/70274821:15
*** jamesmcarthur has quit IRC21:15
*** jamesmcarthur has joined #openstack-infra21:15
harlowjayo yo folks (long time, yes i know)21:18
harlowjado folks know if there is any document/blog about how openstack moved from jenkins -> zuul using jenkins -> zuulv2 ....?21:18
harlowjai'd like to share it with folks at my place, and i'm pretty sure such a thing exists21:18
harlowja(i just don't recall where, ha)21:18
clarkbAJaeger: gearman-plugin is in similar situation as zmq-event-publisher, we basically don't use it and its been effectively dead. That one is actually required to do zuulv2 so might be worth keeping alive ?21:19
clarkbharlowja: fungi is your thing available yet?21:19
clarkbharlowja: ^ I meant to say fungi was working on something along those lines21:19
*** slaweq has quit IRC21:19
harlowjaohhh21:20
harlowjaniceeee21:20
fungiyeah, i have an article submitted to opensource.com which details all that history21:20
harlowjapublished soon?21:20
funginot sure where it is on their publication schedule yet21:20
harlowjak21:20
harlowjacool though, i'll keep a lookout for it21:20
fungimy guess is likely end of january/early february21:21
fungii touch on where we started, outline the various things we built up around jenkins, the pitfalls and jenkins shortcomings we worked around, and the eventual reasons we chose to eliminate it from our environment21:22
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: DNM: debug docker command line options  https://review.opendev.org/70275221:23
fungibut with an aim of not being particularly negative about jenkins itself. it's another free/libre open source project after all, and it's useful for the things it's designed to do21:23
fungii mostly try to underscore that even though people refer to jenkins as a "ci system" it's really not, it's an automation engine people frequently build ci systems around21:24
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: disable docker userland proxy  https://review.opendev.org/70275321:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: DNM: test without docker userland proxy  https://review.opendev.org/70272421:27
harlowjafungi, sounds good21:27
harlowjasounds just like what i want21:28
fungii'm just sorry it's not already published to point you at it21:28
*** igordc has joined #openstack-infra21:29
openstackgerritMerged openstack/openstack-zuul-jobs master: Remove zmq-event-publisher jobs  https://review.opendev.org/70274721:33
rosmaitaclarkb: sorry to be a PITA, but when you have a minute can you look at https://review.opendev.org/#/c/691467/17..18/.zuul.yaml and tell me what i did wrong? the functional-py37 job is running, but it's using py3621:34
harlowjafungi, no worries, better late than never :)21:35
rosmaitaharlowja: what are you up to these days?21:36
*** tdasilva has quit IRC21:36
harlowjahttps://plus.ai/21:37
harlowjaseeing how that works out, haha21:37
harlowjawe also delivered some butter run, lol https://www.bbc.com/news/technology-5074208021:38
rosmaitavery cool21:38
harlowjareddit thread on that was funny as heck21:38
harlowjahttps://www.reddit.com/r/Futurology/comments/e8xwm2/a_selfdriving_truck_delivered_butter_from/21:38
harlowjacomments are awesome21:38
rosmaitawe have torc robotics in my town doing self-driving cars, every so often you see them driving around -- not sure they have gone cross-country, though21:39
harlowjaneats21:40
*** rcernin has quit IRC21:42
*** tommylikehu has quit IRC21:42
*** piotrowskim has quit IRC21:42
*** eharney has joined #openstack-infra21:48
*** eernst has joined #openstack-infra21:53
*** ociuhandu has quit IRC21:54
*** gfidente has quit IRC21:55
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: disable docker userland proxy  https://review.opendev.org/70275321:57
fungiaround here the horses sometimes wander into the road. they're fairly autonomous21:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Import dhall-operator  https://review.opendev.org/70210421:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Import zuul application from dhall-zuul  https://review.opendev.org/70210521:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application  https://review.opendev.org/70210621:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Generate TLS certificats for the gearman service  https://review.opendev.org/70271621:58
harlowjaautonomous horses, powered by horseAI22:04
fungithough to be clear, most of the routines in horseai seem to be centered around rooting through unsecured trashcans22:05
*** jamesmcarthur has quit IRC22:06
clarkbrosmaita: will do though I need to grab late lunch22:06
clarkbso give me a few :)22:06
*** pkopec_ has quit IRC22:08
clarkbrosmaita: your comment is spot on I think22:10
clarkbrosmaita: delete the line you highlighted the nthe vars you set at the higher level will be passed through I think22:10
*** kjackal has quit IRC22:11
*** jamesmcarthur has joined #openstack-infra22:16
*** jamesmcarthur has quit IRC22:19
*** jtomasek has quit IRC22:19
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add Openshift SCC and functional test  https://review.opendev.org/70275822:23
*** xek_ has quit IRC22:24
tristanCzuul tenant container build job are failing unexpectedly with a https://zuul.opendev.org/t/zuul/build/None url22:26
clarkbtristanC: that happens if there is a failed image build for the parent22:28
clarkbtristanC: this is a zuul issue not an infra issue22:28
clarkb(I ran into it yesterday too and discovered zuul doesn't report this to you you just get an empty build)22:28
tristanCclarkb: i see thanks, one of the parent review does have a 'net/http: TLS handshake timeout' image build failure22:29
clarkbtristanC: I think it would be good for zuul to report a better error message back but I haven't had time to debug why it doesnt'22:30
rosmaitaclarkb: thanks for looking!22:31
fungiinfra-root: anybody else care to weigh in on whether we should revert the replication.autoReload addition in gerrit? https://review.opendev.org/69145222:34
clarkbfungi: done22:34
openstackgerritMerged opendev/zone-opendev.org master: Reorganize zonefile  https://review.opendev.org/70183322:34
fungithanks clarkb!22:35
*** icey has quit IRC22:36
*** dosaboy has quit IRC22:36
*** dklyle has quit IRC22:40
*** diablo_rojo has quit IRC22:45
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add Openshift SCC and functional test  https://review.opendev.org/70275822:46
*** smarcet has quit IRC22:46
*** rfarr has joined #openstack-infra22:47
*** thedac has joined #openstack-infra22:48
*** dklyle has joined #openstack-infra22:49
openstackgerritIan Wienand proposed openstack/project-config master: Remove infra trusty jobs  https://review.opendev.org/70276422:50
*** icey has joined #openstack-infra22:51
*** exsdev has joined #openstack-infra22:51
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Remove legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty  https://review.opendev.org/70276522:52
*** rfarr has quit IRC22:53
*** dosaboy has joined #openstack-infra22:53
*** rcernin has joined #openstack-infra22:55
openstackgerritMerged opendev/system-config master: Revert "Enable gerrit replication.autoReload for review"  https://review.opendev.org/69145222:56
*** eernst has quit IRC23:01
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Move legacy-gearman-plugin-mavin-build away from Trusty  https://review.opendev.org/70276823:03
*** tkajinam has joined #openstack-infra23:03
openstackgerritIan Wienand proposed openstack/project-config master: Remove legacy-gearman-plugin-mavin-build-ubuntu-trusty  https://review.opendev.org/70276923:04
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Remove legacy-ubuntu-trusty nodeset  https://review.opendev.org/70277123:07
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Remove legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty  https://review.opendev.org/70276523:09
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Move legacy-gearman-plugin-mavin-build away from Trusty  https://review.opendev.org/70276823:09
*** igordc has quit IRC23:09
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Remove legacy-ubuntu-trusty nodeset  https://review.opendev.org/70277123:09
*** igordc has joined #openstack-infra23:10
openstackgerritIan Wienand proposed openstack/project-config master: Remove trusty wheel jobs  https://review.opendev.org/70277423:19
*** calebb has joined #openstack-infra23:20
ianwit's a very tangled graph but working through https://review.opendev.org/#/q/topic:trusty-removal from https://review.opendev.org/702764 should unravel to remove a big chunk of remaining trusty23:23
clarkbianw: k I'll get to reviews on that shortly23:24
clarkbianw: https://review.opendev.org/#/c/702721/1 failed in the gate so I've rechecked it (this is the dib centos 8 fix)23:24
clarkbas soon as that lands we should get a release out23:24
*** jamespage has joined #openstack-infra23:27
ianw++ yep23:28
clarkbianw: I think we should add a change to the mirror configurator rule to stop adding the wheel mirror to trusty nodes?23:32
clarkbianw: at least until we completely remove the trusty nodes23:32
clarkbotherwise we may get unexpected behavior if we stop updating those?23:33
clarkbits probably ok because pip will use an sdist if the package is newer23:33
clarkbya the more I think of it the more I think its probably ok23:33
clarkbas long as we remove the mirror itself after the nodepool nodes23:34
ianwyeah, i'm hoping it will be very brief.  i think only dib jobs remain, which i only keep because we're building it in the gate23:34
ianwthe only other one i can find is https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/zuul.d/jobs.yaml#L7623:35
clarkbianw: do we not have a puppet job that runs it either?23:35
clarkbI half expected ^ but maybe we've gotten past that already23:35
clarkbthe system-config puppet apply all job or whatever its called23:35
clarkb(does the noop puppet)23:35
ianwmmm, yeah ok looking at system-config23:36
clarkbthat gives us a small bit of coverage over static.o.o and status and refstack I think23:37
*** tdasilva has joined #openstack-infra23:37
clarkbmaybe we need to hunker down and just finish those upgrades.23:37
ianwyeah, i guess they fall more into "control plane" jobs rather than project jobs23:38
*** tobiash has quit IRC23:38
clarkbif we haven't yet a first good step is to test those on xenial23:39
*** tobiash has joined #openstack-infra23:39
clarkbthen if those pass maybe we just delete the trusty jobs and plan to redeploy on xenial real soon now23:39
ianwit's mostly the wheel jobs i want to stop right now, to make one less target there23:40
clarkbya I think that should be safe23:41
clarkbsince control plane doesn't rely on libvirt-python and lxml23:41
clarkbwe can just use pypi23:41
ianwyeah, i don't think any of the system-config jobs setup the mirror?23:42
clarkbianw: they may get it from our base job?23:42
ianwpossibly, we can do the "drop mirror if trusty"23:43
clarkbianw: when do we get you back from moving?23:44
clarkbmaybe we can plan a sprint day or two and focus on finishing the control plane upgrades23:44
openstackgerritMerged zuul/zuul master: Re-elect James Blair as project lead  https://review.opendev.org/70126723:45
ianw21st (.au time)23:45
clarkbmaybe give you a bit of time to settle in and plan for the 27th and 28th (thats 28th and 29th au time I think)23:45
clarkbfungi: ^ will you have departed for fosdem at that point yet? or would you be able to help then?23:46
ianwyeah, i'd be happy to focus on it then23:46
clarkbk lets pencil that in23:46
clarkbI think we are really close and its mostly just a matter of writing the changes and booting some nodes23:46
clarkbfocused time on that should help us get it done23:47
fungiclarkb: checking, but it's close anyway23:47
fungiclarkb: looks like my first flight leaves on thursday 2020-01-30 so i could do the 27th and 28th23:48
clarkbgreat23:48
ianwof course, there is the incredibly complex issue of running a cable from the street to the new house thru pre-laid specific conduit for wired internet, which i'm sure will take the ISP weeks to figure out23:48
ianwand then, when they eventually arrive, the correct person 15 minutes to actually do23:49
funginone of my talks got selected, so i don't have any last-minute material to prepare, just need to practice looking good so when folks come by the osf table asking about zuul or something they won't be too bad when i just smile and nod23:49
fungier, won't be too mad23:49
fungihere, it's all explained in this very glossy onesheet23:49
fungisee the high sheen on the print? that's a quality onesheet23:50
clarkbfungi: also you'll need one liners for the kids looking for stickers23:50
fungiin fact, have two. give one to a friend!23:50
*** dchen has joined #openstack-infra23:50
clarkbianw: oof lets hope they get it right the first time around23:50
fungii'll have to learn how to say "we're out of zuul stickers" in 5 or 6 different languages23:50
ianwclarkb: almost certain they won't ... i have a 4g hotspot i can enable month-to-month, which stupidly is faster than the HFC available anyway.  i could probably even live with the data cap, but the cgnat it does annoys me23:51
fungii still don't understand how anyone puts up with the down-under isps. is it just one of the many outcomes of being settled as a penal colony?23:54
fungihere, have some prison internet23:54
fungii mean, internet is pretty terribad in the usa compared to lots of places, but nothing like au23:55
clarkbI just upgraded to half gig symettric service. I get about quarter gig in reality23:56
clarkbfungi: I think it has to do with limited bw to the outside world23:56
ianwso we have the NBN which is supposed to connect everyone.  originally everyone was getting fibre, but now it's a complete mess of fibre-to-curb, HFC, etc.  then ISP's sell access to the NBN23:57
ianwAFAICT, the only thing ISPs can really compete on is "how little backhaul can i buy to support X customers without them getting so annoyed at the shitty service they leave"23:58
fungiso race to the bottom23:58
ianwi mean, they're all selling access to the same NBN network, once connected to that i can theoretically call any isp and just switch whenever23:58
clarkbah here its all separate infrastructure. So I can get fiber with frontier up to gigabit, comcast does cable with docsis up to 1000/10 I think, and centurylink will do adsl up to like 8mbps23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!