Tuesday, 2020-03-10

*** tetsuro has joined #openstack-infra00:06
openstackgerritMerged opendev/system-config master: Add initial Ansible for nodepool hosts  https://review.opendev.org/71088800:10
*** jamesmcarthur has joined #openstack-infra00:11
openstackgerritMerged opendev/system-config master: Add install zookeeper role; use for nodepool-builder testing  https://review.opendev.org/71090800:12
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-builder: deploy from container  https://review.opendev.org/71089100:14
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-builder: Add webserver  https://review.opendev.org/71157400:14
openstackgerritIan Wienand proposed opendev/system-config master: Add nb01.opendev.org  https://review.opendev.org/69311900:14
*** Goneri has quit IRC00:14
openstackgerritTristan Cacqueray proposed zuul/zuul master: Implement zookeeper-auth  https://review.opendev.org/61915600:15
clarkbianw: https://review.opendev.org/693119's dependency needs a rebase too. 693119 got a -1 because its dpeends on can't be merged00:15
*** matt_kosut has joined #openstack-infra00:15
clarkb(its gonna be the serial on the zone file)00:15
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: Add nb01.opendev.org  https://review.opendev.org/69311800:17
*** tetsuro has quit IRC00:18
*** tetsuro has joined #openstack-infra00:18
ianwclarkb: ^thanks, should get it00:18
*** jamesmcarthur has quit IRC00:19
*** matt_kosut has quit IRC00:20
*** jamesmcarthur has joined #openstack-infra00:21
*** tetsuro has quit IRC00:21
*** tetsuro has joined #openstack-infra00:22
*** dklyle has quit IRC00:22
*** jamesmcarthur has quit IRC00:26
clarkbcool I think I've +2'd the whole stack at this point and we can make changes to mounts and config copies as followons00:30
clarkb(assuming we decide those changes make sense)00:31
ianwthanks!00:32
*** diablo_rojo has quit IRC00:42
*** jamesmcarthur has joined #openstack-infra00:44
*** yamamoto has joined #openstack-infra00:54
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add ensure-venv element, install glean with it  https://review.opendev.org/70751300:57
ianwclarkb: ^ all the boot tests passed, but i still added the mkdir -p you mentioned00:58
ianwif you're ok with that, i think we release dib with it soon.  couple of other changes to merge too00:58
*** jamesmcarthur has quit IRC01:08
*** jamesmcarthur has joined #openstack-infra01:12
openstackgerritMerged openstack/diskimage-builder master: Uncap hacking  https://review.opendev.org/70933201:16
openstackgerritMerged opendev/glean master: Switch to Fedora 30 jobs  https://review.opendev.org/71197001:17
openstackgerritTristan Cacqueray proposed zuul/zuul master: Implement zookeeper-auth  https://review.opendev.org/61915601:18
*** jamesmcarthur has quit IRC01:24
*** jamesmcarthur has joined #openstack-infra01:24
openstackgerritGhanshyam Mann proposed openstack/project-config master: Rename x/devstack-plugin-nfs to openstack/devstack-plugin-nfs  https://review.opendev.org/71183401:24
*** ccamacho has joined #openstack-infra01:25
gmannfungi: ^^ updated.01:26
fungilookin'01:28
*** hongbin has joined #openstack-infra01:31
*** ricolin_ has joined #openstack-infra01:39
*** rosmaita has left #openstack-infra01:40
*** rkukura has quit IRC01:40
*** armax has quit IRC01:43
*** rlandy|bbl is now known as rlandy01:53
*** yamamoto has quit IRC02:01
*** happyhemant has quit IRC02:05
*** jamesmcarthur has quit IRC02:06
*** jamesmcarthur has joined #openstack-infra02:07
*** rkukura has joined #openstack-infra02:07
*** yamamoto has joined #openstack-infra02:07
*** nicolasbock has joined #openstack-infra02:26
openstackgerritMerged openstack/diskimage-builder master: Remove Fedora 29 job  https://review.opendev.org/71197202:50
*** yamamoto has quit IRC02:52
*** yamamoto has joined #openstack-infra02:54
*** rkukura has quit IRC02:56
*** rkukura has joined #openstack-infra03:01
*** nicolasbock has quit IRC03:06
*** jamesmcarthur has quit IRC03:21
*** rlandy has quit IRC03:36
*** hongbin has quit IRC03:47
*** lbragstad_ has joined #openstack-infra03:54
*** roman_g has quit IRC04:02
*** rcernin has quit IRC04:04
*** rcernin has joined #openstack-infra04:04
*** matt_kosut has joined #openstack-infra04:15
*** matt_kosut has quit IRC04:20
*** ianychoi has quit IRC04:39
*** ianychoi has joined #openstack-infra04:40
*** dave-mccowan has quit IRC04:40
*** yamamoto has quit IRC04:57
openstackgerritMerged openstack/diskimage-builder master: Add ensure-venv element, install glean with it  https://review.opendev.org/70751305:08
*** yamamoto has joined #openstack-infra05:27
*** lbragstad_ has quit IRC05:34
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-infra05:35
*** ociuhandu has joined #openstack-infra06:07
*** ijw_ has quit IRC06:08
*** ociuhandu has quit IRC06:12
openstackgerritIan Wienand proposed openstack/project-config master: Move fedora builds to nb01.opendev.org  https://review.opendev.org/69312006:12
*** ricolin_ has quit IRC06:23
*** ricolin has joined #openstack-infra06:23
*** eandersson has quit IRC06:23
*** eandersson has joined #openstack-infra06:24
*** ijw has joined #openstack-infra06:33
*** lmiccini has joined #openstack-infra06:34
*** ijw has quit IRC06:38
*** udesale has joined #openstack-infra06:41
*** AJaeger has joined #openstack-infra07:03
*** xinranwang has joined #openstack-infra07:08
AJaegergmann, fungi, clarkb, https://review.opendev.org/#/c/711834 (project rename) needs still a gerrit downtime, correct? clarkb, want to bring this up in the team meeting today?07:13
AJaegerconfig-core, before we can disable fedora-29, we need to update fedora-latest, please review https://review.opendev.org/71196907:14
*** dpawlik has joined #openstack-infra07:21
*** ociuhandu has joined #openstack-infra07:28
*** tetsuro has quit IRC07:29
*** Lucas_Gray has joined #openstack-infra07:30
*** yamamoto has quit IRC07:33
*** ociuhandu has quit IRC07:35
*** ccamacho has quit IRC07:35
*** ccamacho has joined #openstack-infra07:35
*** yamamoto has joined #openstack-infra07:35
*** pgaxatte has joined #openstack-infra07:38
*** tetsuro has joined #openstack-infra07:43
*** Lucas_Gray has quit IRC07:45
*** Lucas_Gray has joined #openstack-infra07:46
*** AJaeger has quit IRC07:49
*** Wryhder has joined #openstack-infra07:51
*** Lucas_Gray has quit IRC07:51
*** Wryhder is now known as Lucas_Gray07:52
*** slaweq has joined #openstack-infra07:56
*** xek_ has joined #openstack-infra07:57
*** AJaeger has joined #openstack-infra08:00
*** yboaron has joined #openstack-infra08:02
*** ociuhandu has joined #openstack-infra08:05
*** tkajinam has quit IRC08:08
*** ijw has joined #openstack-infra08:08
*** yboaron has quit IRC08:10
*** tesseract has joined #openstack-infra08:11
*** ijw has quit IRC08:13
*** ociuhandu has quit IRC08:18
*** tosky has joined #openstack-infra08:18
*** matt_kosut has joined #openstack-infra08:21
*** jcapitao has joined #openstack-infra08:22
*** pkopec has joined #openstack-infra08:28
*** amoralej|off is now known as amoralej08:29
*** ijw has joined #openstack-infra08:39
*** rpittau|afk is now known as rpittau08:43
*** ijw has quit IRC08:44
*** priteau has joined #openstack-infra08:44
*** ralonsoh has joined #openstack-infra08:50
*** jpena|off is now known as jpena08:50
*** Wryhder has joined #openstack-infra08:50
*** Lucas_Gray has quit IRC08:51
*** Wryhder is now known as Lucas_Gray08:51
*** ociuhandu has joined #openstack-infra08:53
*** yamamoto has quit IRC09:02
*** yamamoto has joined #openstack-infra09:06
*** ociuhandu has quit IRC09:09
*** ijw has joined #openstack-infra09:10
*** ijw has quit IRC09:15
*** priteau has quit IRC09:28
*** tetsuro has quit IRC09:31
openstackgerritTobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check  https://review.opendev.org/64455709:41
*** ijw has joined #openstack-infra09:42
openstackgerritTobias Henkel proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535409:45
*** ijw has quit IRC09:46
openstackgerritTobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs  https://review.opendev.org/67384009:49
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove Fedora 29 from nodepool  https://review.opendev.org/71203209:51
AJaegerianw: are you ok with the change above to remove Fedora 29 (once dependencies are merged)? ^09:51
*** derekh has joined #openstack-infra09:58
*** udesale has quit IRC10:01
*** gfidente|afk is now known as gfidente10:01
openstackgerritTobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches  https://review.opendev.org/64027210:01
ianwAJaeger: yes ... working to get f31 up soon with the builder changes10:01
*** sshnaidm|afk is now known as sshnaidm10:01
rpittauhey good morning! Is there a way to get a bionic image with kernel version 5.x for devstack in ci ?10:02
*** dtantsur|afk is now known as dtantsur10:06
ianwrpittau: umm, like a hwe kernel?  not really i'm afraid.  iirc reboots weren't handled too well when i last tried it10:10
*** jcapitao has quit IRC10:13
*** yamamoto has quit IRC10:13
*** jcapitao has joined #openstack-infra10:15
*** ociuhandu has joined #openstack-infra10:17
openstackgerritBenjamin Schanzel proposed zuul/nodepool master: Kubernetes/OpenShift Provider: Don't Require Bash in Container Images  https://review.opendev.org/71203410:17
rpittauianw: thanks, yeah didn't want to deal with restarts.... I see that bionic is shipping with 5.x by default but I guess that's not valid for the cloud image10:17
*** xinranwang has quit IRC10:18
AJaegerianw: Yeah, I've seen those f31 changes.10:24
zbrianw: low hanging https://review.opendev.org/#/c/708704/  thanks.10:26
zbrhttps://review.opendev.org/#/c/703053/ is also useful, we already use this for 2+ months on two other instances (rdo and ansible)10:28
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: DNM: test triggers  https://review.opendev.org/71203710:33
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: DNM: test triggers  https://review.opendev.org/71203710:33
zbrAJaeger: see my last comment on https://review.opendev.org/#/c/708642/16/zuul-tests.d/python-jobs.yaml@1810:36
AJaegerzbr: let's discuss with corvus, it looks to me like you found a bug...10:38
zbri doubt is a bug because some files have a huge number of jobs defined in them, and user may not want to trigger 100 jobs when they fix a typo in one description.10:39
zbrbut i have no problem waiting10:39
zbrideally zuul could be smart to figure-out that job definition dictionary was updated, but i have no idea if this was ever implemented.10:41
AJaegerzbr, we have https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.match-on-config-updates , so it should run10:41
AJaegerzbr, I agree, it does not work - and a change of description might not be enough. Let's change some more - let me do that quickly...10:41
zbrsure, feel free to re-use my ^ DNM change.10:41
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: DNM: test triggers  https://review.opendev.org/71203710:42
AJaegerzbr:now they run...10:42
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: DNM: test triggers  https://review.opendev.org/71203710:43
AJaegerzbr, so all is fine ;)10:43
AJaegerzbr, the test for change in job does not seem to take description into account, which is fine for me10:43
zbri guess the am_i_changed() logic is quite smart, ignoring minor changes10:44
*** ijw has joined #openstack-infra10:44
AJaegerzbr, so http://zuul.opendev.org/t/zuul/status/change/712037,4/ LGTM - to you as well?10:44
AJaegerzbr, yes, seems the logic only looks for real change - and a description change does not count.10:45
zbryep, looks well10:45
AJaegerzbr, so I leave my -1 on the other change ;)10:45
*** ijw has quit IRC10:49
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Improve ensure-tox role  https://review.opendev.org/70864210:51
AJaegerzbr, please update commit message as well10:54
*** andrewbonney has joined #openstack-infra10:55
*** yamamoto has joined #openstack-infra10:57
zbrno need, replied.11:01
*** Lucas_Gray has quit IRC11:04
openstackgerritMerged zuul/zuul-jobs master: Tests bindep role on all-platforms  https://review.opendev.org/70870411:06
*** yamamoto has quit IRC11:07
*** Lucas_Gray has joined #openstack-infra11:08
*** jcapitao is now known as jcapitao_lunch11:11
*** tkajinam has joined #openstack-infra11:15
*** ianychoi has quit IRC11:16
*** rpittau is now known as rpittau|bbl11:16
*** ianychoi has joined #openstack-infra11:16
*** jamesmcarthur has joined #openstack-infra11:19
openstackgerritSlawek Kaplonski proposed openstack/project-config master: Remove neutron-tempest-dvr job from Neutron's dashboard  https://review.opendev.org/71204811:22
*** yamamoto has joined #openstack-infra11:24
*** jamesmcarthur has quit IRC11:24
*** udesale has joined #openstack-infra11:25
*** Lucas_Gray has quit IRC11:26
*** Lucas_Gray has joined #openstack-infra11:28
*** udesale_ has joined #openstack-infra11:29
openstackgerritTobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs  https://review.opendev.org/67384011:30
*** udesale has quit IRC11:32
openstackgerritTobias Henkel proposed zuul/zuul master: Add spec for enhanced regional executor distribution  https://review.opendev.org/66341311:34
openstackgerritTobias Henkel proposed zuul/zuul master: Move fingergw config to fingergw  https://review.opendev.org/66494911:36
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway  https://review.opendev.org/66496511:36
openstackgerritTobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw  https://review.opendev.org/66495011:36
*** yamamoto has quit IRC11:37
*** rosmaita has joined #openstack-infra11:38
openstackgerritBalazs Gibizer proposed opendev/irc-meetings master: Reschedule Nova Team meeting  https://review.opendev.org/71205211:38
*** udesale_ has quit IRC11:38
*** ociuhandu has quit IRC11:40
*** auristor has quit IRC11:40
*** ociuhandu has joined #openstack-infra11:40
*** udesale has joined #openstack-infra11:41
*** ociuhandu has quit IRC11:46
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway  https://review.opendev.org/66496511:47
openstackgerritTobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw  https://review.opendev.org/66495011:47
*** udesale has quit IRC11:47
*** Goneri has joined #openstack-infra11:49
*** nicolasbock has joined #openstack-infra11:54
*** dchen has quit IRC11:59
*** rh-jelabarre has joined #openstack-infra11:59
AJaegerzbr ,want to abandon https://review.opendev.org/712037 now?12:09
zbrdone12:10
AJaegerthanks12:11
*** Lucas_Gray has quit IRC12:12
*** Goneri has quit IRC12:12
*** auristor has joined #openstack-infra12:13
*** nicolasbock has quit IRC12:13
*** nicolasbock has joined #openstack-infra12:15
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: bindep: Add missing virtualenv and fixed repo install  https://review.opendev.org/69363712:17
*** Lucas_Gray has joined #openstack-infra12:19
*** jamesmcarthur has joined #openstack-infra12:21
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make ensure-tox pass cross-platform  https://review.opendev.org/70743912:24
*** jpena is now known as jpena|lunch12:25
openstackgerritTristan Cacqueray proposed zuul/zuul master: Implement zookeeper-auth  https://review.opendev.org/61915612:25
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: use failed_when  https://review.opendev.org/71206212:28
*** ociuhandu has joined #openstack-infra12:29
*** rh-jelabarre has quit IRC12:36
*** jamesmcarthur has quit IRC12:36
*** happyhemant has joined #openstack-infra12:40
*** ociuhandu has quit IRC12:43
*** Goneri has joined #openstack-infra12:43
*** ociuhandu has joined #openstack-infra12:44
*** ociuhandu has quit IRC12:49
*** ianychoi has quit IRC12:49
*** ijw has joined #openstack-infra12:49
*** ianychoi has joined #openstack-infra12:50
*** gfidente has quit IRC12:54
*** rpittau|bbl is now known as rpittau12:54
*** piotrowskim has joined #openstack-infra12:54
*** ijw has quit IRC12:54
*** jcapitao_lunch is now known as jcapitao12:57
*** eharney has quit IRC12:59
*** yamamoto has joined #openstack-infra12:59
*** xek_ is now known as xek13:01
*** lbragstad_ has joined #openstack-infra13:01
*** jpena|lunch is now known as jpena13:04
*** Lucas_Gray has quit IRC13:07
*** matt_kosut has quit IRC13:10
*** gfidente has joined #openstack-infra13:10
*** yamamoto has quit IRC13:11
*** lbragstad_ is now known as lbragstad13:12
*** lbragstad has quit IRC13:13
*** Lucas_Gray has joined #openstack-infra13:13
*** lbragstad has joined #openstack-infra13:16
*** zxiiro has joined #openstack-infra13:18
*** yamamoto has joined #openstack-infra13:19
*** yamamoto has quit IRC13:19
*** ijw has joined #openstack-infra13:21
*** Lucas_Gray has quit IRC13:21
*** amoralej is now known as amoralej|lunch13:22
*** yamamoto has joined #openstack-infra13:23
*** Lucas_Gray has joined #openstack-infra13:25
*** ijw has quit IRC13:25
*** yamamoto has quit IRC13:30
*** rkukura has quit IRC13:30
*** rkukura has joined #openstack-infra13:37
*** Wryhder has joined #openstack-infra13:37
*** Lucas_Gray has quit IRC13:38
*** Wryhder is now known as Lucas_Gray13:38
fungiianw: what problem did you run into with reboots, besides needing to restart the log streamer?13:38
*** eharney has joined #openstack-infra13:43
*** ociuhandu has joined #openstack-infra13:44
smcginnisHowdy folks. We have a release job pending on getting a publish-to-pypi job added for it. Can someone other than Andreas take a look at https://review.opendev.org/#/c/711612/13:50
*** yamamoto has joined #openstack-infra13:50
*** dklyle has joined #openstack-infra13:51
openstackgerritLance Bragstad proposed openstack/project-config master: Add a new project and repository for tripleo-ipa  https://review.opendev.org/71111413:54
openstackgerritLance Bragstad proposed openstack/project-config master: Add jobs for tripleo-ipa project  https://review.opendev.org/71111513:54
*** Liang__ has quit IRC14:00
*** dave-mccowan has joined #openstack-infra14:02
*** rh-jelabarre has joined #openstack-infra14:06
*** matt_kosut has joined #openstack-infra14:07
*** ianychoi has quit IRC14:10
*** ianychoi has joined #openstack-infra14:11
*** matt_kosut has quit IRC14:12
*** Lucas_Gray has quit IRC14:12
openstackgerritCédric Jeanneret (Tengu) proposed openstack/project-config master: Add new Validation Framework projects  https://review.opendev.org/71191014:17
*** yamamoto has quit IRC14:18
*** Lucas_Gray has joined #openstack-infra14:18
*** ijw has joined #openstack-infra14:23
openstackgerritLance Bragstad proposed openstack/project-config master: Add a new project and repository for tripleo-ipa  https://review.opendev.org/71111414:25
openstackgerritLance Bragstad proposed openstack/project-config master: Add jobs for tripleo-ipa project  https://review.opendev.org/71111514:25
*** ijw has quit IRC14:28
clarkbrpittau: ianw fungi ya you can reboot in jobs, the reason we dont have the hwe kernel is they dont give it by default (and we install the default kernel). I think only desktop installs swotch by default?14:28
*** amoralej|lunch is now known as amoralej14:28
clarkbI know I had to opt in on my server install in order to get some driver fix for hardware14:28
rpittauclarkb: interesting, any help on how we can handle that ?14:30
*** nicolasbock has quit IRC14:30
clarkbrpittau: as fungi mentions you need to restart the zuul console streamer (there is a role you can use to do that)14:30
clarkbotherwise should be as normal, install kernel and reboot14:31
rpittauclarkb: thanks, didn't see fungi comment, I'll search for the role :)14:31
openstackgerritMerged openstack/project-config master: Add publish-to-pypi template for tenks  https://review.opendev.org/71161214:32
*** jamesmcarthur has joined #openstack-infra14:39
*** matt_kosut has joined #openstack-infra14:39
*** iurygregory has quit IRC14:39
*** iurygregory has joined #openstack-infra14:43
*** jamesmcarthur has quit IRC14:46
*** ijw has joined #openstack-infra14:46
*** armax has joined #openstack-infra14:51
*** jamesmcarthur has joined #openstack-infra14:54
*** jamesmcarthur has quit IRC14:56
*** jamesmcarthur has joined #openstack-infra14:56
*** yamamoto has joined #openstack-infra14:57
*** yamamoto has quit IRC15:04
*** Lucas_Gray has quit IRC15:10
*** mattw4 has joined #openstack-infra15:14
*** Lucas_Gray has joined #openstack-infra15:14
*** ociuhandu has quit IRC15:17
*** tkajinam has quit IRC15:17
*** ociuhandu has joined #openstack-infra15:17
openstackgerritClark Boylan proposed opendev/base-jobs master: Cleanup workaround for importlib-resources bad version  https://review.opendev.org/71210715:22
*** ociuhandu has quit IRC15:23
clarkbinfra-root ^ fyi I think all the images that are building right now have caught up after lceaning up builders yesterday and we can land that change. I also linked to a base-test run you can confirm with in the change15:23
clarkbrpittau: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/start-zuul-console/README.rst is the role. Not sure if you've found it yet15:26
rpittauclarkb: thanks, no! :)15:27
*** armax has quit IRC15:33
*** pgaxatte has quit IRC15:37
*** KeithMnemonic has joined #openstack-infra15:38
*** gyee has joined #openstack-infra15:43
*** matt_kosut has quit IRC15:45
*** ociuhandu has joined #openstack-infra15:48
*** pkopec has quit IRC16:02
*** lbragstad has quit IRC16:07
openstackgerritMerged opendev/base-jobs master: Switch nodeset fedora-latest to fedora 30  https://review.opendev.org/71196916:08
*** lbragstad has joined #openstack-infra16:08
*** yamamoto has joined #openstack-infra16:10
clarkbAJaeger: correct on the gerrit downtime for project renames16:11
clarkbAJaeger: maybe we should do openstack/infra-manual -> opendev/infra-manual now that there is another rename queued up16:11
*** ianychoi has quit IRC16:11
*** ianychoi has joined #openstack-infra16:12
*** yamamoto has quit IRC16:15
TenguAJaeger: heya! do I need to do anything for https://review.opendev.org/#/c/711910/ to get merged? I think everything should be OK (the needed-by is there, just running a recheck on it)16:15
clarkbTengu: what sort of renaming might happen later? renaming of the git repos?16:19
*** michael-mc-aleer has joined #openstack-infra16:23
*** michael-mc-aleer has quit IRC16:23
*** michael-mc-aleer has joined #openstack-infra16:24
*** jcapitao has quit IRC16:25
Tenguclarkb: heya! nope, not the repositories - more the "tripleo-validations" part that will be dropped later, so we might want to rename the... err... "group"? to validation-framework instead.16:26
michael-mc-aleerHi Infrastructure team, I'm running into some issues stacking Devstack from master, Horizon has a dependency pyScss==1.3.4 that tries to import Fearure from setup tools but it was removed in setup tools 46.0.0, any advice on where I can go to raise it to get fixed? Thanks!16:26
*** aarents has quit IRC16:27
*** aarents has joined #openstack-infra16:27
michael-mc-aleergithub.com/Kronuz/pyScss/issues/38516:27
michael-mc-aleerStacking process broken as a result16:28
Tenguclarkb: but maybe the "tox" error here might be due to the grouping I'm trying to do in the governance chance... https://zuul.opendev.org/t/openstack/build/1c34c13f61154a0d9ee7fd8b4892963f16:28
clarkbTengu: I'm not sure, just wanted to make sure we didn't merge the repo creation early if the names were still up for debate16:29
Tenguclarkb: ah, nah, the repositories are fixed for good16:29
Tengupackages will get that name and all16:29
*** bdodd has quit IRC16:29
clarkbmichael-mc-aleer: you can disable horizon and bypass the issue entirely. Otherwise you'll probably need pyScss to update16:29
clarkbmichael-mc-aleer: Setuptools isn't a library we can easily control the version of so addressing bugs that come up with interactions with it is usually the best option16:30
clarkb(bceause users local version of setuptools is arbitrary)16:30
Tenguthanks clarkb :)16:30
michael-mc-aleerSure, I can make that work for our dev team but our quality engineers may be impacted because they use horizon16:30
michael-mc-aleerThanks for the heads up, that pyScss repo looks pretty stagnant, last commit was 2018 and there is a tonne of issues left unanswered16:31
clarkbmichael-mc-aleer: possible that horizon may need to consider a different css compiler then16:31
michael-mc-aleerYeah that's what my thoughts are too, I'll give their channel a msg to see if they are aware16:32
michael-mc-aleerThanks again!16:32
*** bdodd has joined #openstack-infra16:33
*** trident has quit IRC16:34
*** jcapitao has joined #openstack-infra16:35
*** matt_kosut has joined #openstack-infra16:36
*** trident has joined #openstack-infra16:36
*** yamamoto has joined #openstack-infra16:47
openstackgerritClark Boylan proposed opendev/system-config master: Add periodic job to run goaccess reporting  https://review.opendev.org/71212116:47
*** ianychoi has quit IRC16:47
clarkbinfra-root ^ thats a first pass at using go access for website stats. corvus I've chosen zuul-ci.org as the first site to run this16:48
*** ianychoi has joined #openstack-infra16:48
*** pkopec has joined #openstack-infra16:49
clarkbthe job is set up to not publicly publsih the reports initially. That will give us the opportunity to review them first (to ensure the version on bionic does what we want and that things otherwise are as expected)16:50
fungiclarkb: michael-mc-aleer: i think there was a recent ml thread (last month?) on openstack-discuss about options for replacing it. i'll see if i can find that in the archive16:51
*** yamamoto has quit IRC16:52
openstackgerritJames E. Blair proposed zuul/zuul master: Implement zookeeper-auth  https://review.opendev.org/61915616:54
*** armax has joined #openstack-infra16:55
lbragstadAJaeger i think we finally reached consensus on https://review.opendev.org/#/c/711114/ - curious if you'd like to take another look when you get a minute?16:55
clarkboh apparently I can use copy: with a content attribute instead of shell and a heredoc16:56
Tenguclarkb: yep, that's pretty nice :)16:56
Tenguansible is full of tricks :]16:56
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Implement zookeeper-auth  https://review.opendev.org/61915516:57
openstackgerritClark Boylan proposed opendev/system-config master: Add periodic job to run goaccess reporting  https://review.opendev.org/71212116:58
clarkbnow with better ansibling16:58
fungiclarkb: michael-mc-aleer: nevermind, i was misremembering the discussion about finding a replacement for django-babel16:58
clarkbfrickler: ^ sorry, I decided to get rid of the hacky heredoc once I realized the copy module could do what I wanted. Thank you for the review16:59
michael-mc-aleerDowngrading setuptools to N-1 release 45.3.0 works if anyone else runs into a similar problem, I was stacking in a clean environment suit was pulling the latest setuptools during setup17:04
michael-mc-aleer*it was pulling17:05
*** tesseract has quit IRC17:07
*** matt_kosut has quit IRC17:09
clarkbfriendly reminder: For many of us in north america that observe dst the clocks shifted a coupel days ago. This means that our meeting is an hour later than it was last week if going off your local timezone clock17:09
clarkbOur meeting is in just under 2 hours17:09
clarkbunless you are fungi and only use UTC clocks17:10
*** rpittau is now known as rpittau|afk17:10
*** ociuhandu has quit IRC17:10
fungiyup17:11
fungithough i had a conference call earlier today which follows usa/canada dst so had to remember that it occurred an hour earlier relative to utc17:12
* fungi lobbies his congresscritters again to abolish dst17:13
*** ccamacho has quit IRC17:15
*** jamesmcarthur has quit IRC17:20
*** ianychoi has quit IRC17:26
*** mattw4 has quit IRC17:26
*** mattw4 has joined #openstack-infra17:27
*** jamesmcarthur has joined #openstack-infra17:27
*** ianychoi has joined #openstack-infra17:27
fungiapparently it currently costs us$800k/mo to operate pypi? https://groups.google.com/forum/#!topic/pypa-dev/Pdnoi8UeFZ817:27
*** sshnaidm is now known as sshnaidm|afk17:31
corvusfungi: fascinating!  we have experience with at least 2 of their suggestions....17:33
corvusfungi: i wonder if the reason we abandoned one of them (bandersnatch -- because of tensorflow data packages) has anything to do with the 800k/month17:33
fungicorvus: yeah, i've sent a reply detailing our prior and current approaches/experience there (per the reference implementation section) along with links to the relevant configs17:34
corvusfungi: is that moderated?17:35
fungino idea how long it takes replies to threads on pypa-dev to show up, may be moderated (google's mta accepted my reply near-instantly)17:35
fungiso doesn't seem they're greylisting me17:35
corvusfungi: ack, thanks for doing that, i think we can be helpful17:35
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-infra17:35
fungii hope so. if my message doesn't make it through to the list i'll pull it out of my fcc and stick it in an etherpad or somewhere for someone else to try to send17:36
clarkbcorvus: judging based on freedesktop's rough cost break down it does seem like storage and transfer is not cheap17:36
clarkbcorvus: and having multi gigabytes of tensorflow related packages pushed everyday would make a big dent I Think17:36
*** dtantsur is now known as dtantsur|afk17:36
fungiyeah, moc said they're currently paying amazon something like us$2m/yr and almost all of it is data transfer on storage of research data17:37
*** matt_kosut has joined #openstack-infra17:37
clarkbit does make me happy to know that our attempts at being friendly neighbors are appreciated17:37
fungior rather fetching the research data they've stored (amazon storage is priced so that it's cheap to put things there and expensive to read them back)17:38
corvusclarkb: yeah, i mean, if they're pushing that TF data up there, i bet someone is downloading it too17:38
clarkbcorvus: ya exactly17:38
clarkbfungi: thats how they get you! :)17:39
fungiindeed17:39
clarkbalso ++ for providing feedback on how bandersnatch is basically unusable at this point unless your goal is to make a complete mirror17:39
fungiwell, it grew a feature to filter specific packages around the time we stopped using it, but 1. that would have required a full rebuild of the mirror to filter out earlier files we'd already copied and 2. it would still need keeping on top of for any future packages following the same pattern17:40
fungii mentioned those in my reply17:40
fungiwhich has still not appeared17:41
clarkbarg my change hit the single ' in comment in ansible playbook issue17:41
clarkbfungi: exactly17:41
fungi2020-03-10 17:24:22 1jBic8-0003nD-SU => pypa-dev@googlegroups.com R=dnslookup T=remote_smtp H=gmr-smtp-in.l.google.com [2607:f8b0:400d:c0f::e] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes DN="C=US,ST=California,L=Mountain View,O=Google LLC,CN=mx.google.com" C="250 2.0.0 OK  1583861062 q1si355258pff.4 - gsmtp"17:41
fungic'mon google, it's been over 15 minutes now17:42
*** michael-mc-aleer has quit IRC17:42
openstackgerritClark Boylan proposed opendev/system-config master: Add periodic job to run goaccess reporting  https://review.opendev.org/71212117:42
clarkbyou know when I was taught ot write a compiler the first thing we were told to do was to strip all the comments out and completely ignore them17:42
clarkbbecause you shouldn't apply your syntax rules to comment content17:42
*** jpena is now known as jpena|off17:43
fungiunfortunately it's the yaml parser which knows what should be a comment, and i guess ansible's substitution engine takes precedence?17:43
*** jamesmcarthur has quit IRC17:43
*** jcapitao is now known as jcapitao_off17:44
clarkbfungi: I'm not sure what layer is hitting it. The linter catches it, but I'm pretty sure real ansible fails on it too (so yay linter does catch issues)17:44
fungior is it a shell comment? that's even harder to catch since the comment identification happens at runtime not during deserialization17:44
clarkbfungi: in this case it was actually a shell comment in a block string. But it happens in yaml comments too17:45
clarkb(so ya it must be evaluating the entire string before yaml parses any of it)17:45
clarkbthe docker suggestions in that pep draft really only shift the problem to the docker registries17:47
mordredfungi: if only both vexxhost and ovh's object storage weren't half the cost17:47
clarkbwhich wouldn't be so bad if docker registries weren't so painful to cache17:48
clarkbbut they are17:48
clarkb(so bad that lennart even wrote an entire new toolchain to address it as he does)17:48
fungii did mention caching dockerhub and quay (and it's happening in the same file i linked to for our pypi caching example), but i didn't explicitly mention the struggles with caching them since they were less the topic of the pep17:49
mordredclarkb: to be fair - he also writes entire new toolchains to replace good software - so I don't know that that adds more credibility to the point that it's hard to cache17:49
clarkbmordred: ya17:49
*** lmiccini has quit IRC17:49
clarkbfungi: ya I think thats fine. I'm mostly just a little concerned the fix is to make it dockerhubs problem17:50
clarkbwhich will only make dockerhub even less reliable17:50
fungiright, i touched on the need to cache basically everything you can (within reason, so long as it doesn't undermine the integrity of the build itself of course)17:51
clarkb++17:51
fungiand that the internet is terribad17:51
fungialbeit with more elegant phrasing17:51
*** jamesmcarthur has joined #openstack-infra17:56
clarkbfungi: I've found documentation that says you must have a google account to post to a google group.17:58
clarkbhttps://support.google.com/groups/answer/1067205?hl=en17:58
clarkbthat may explain the non post?17:59
*** pkopec has quit IRC17:59
*** derekh has quit IRC18:00
openstackgerritMerged openstack/os-testr master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/70552218:01
fungiclarkb: i wonder if that's irrespective of whether it comes from a subscribed e-mail address?18:01
clarkbfungi: hrm it lists join a group as being somethign that requires a googel account too. Not sure if that is the same as subscribing though18:03
*** Lucas_Gray has quit IRC18:03
fungiright, i've been treating it as a public mailing list to which my e-mail address is subscribed. pretty sure i've posted to it in the past18:03
fungivia smtp18:04
*** ijw_ has joined #openstack-infra18:05
fungii suspect it's just moderated18:06
zbrisn18:06
zbrisn't easier to reply to https://discuss.python.org/t/draft-pep-pypi-cost-solutions-ci-mirrors-containers-and-caching-to-scale/3681 ?18:07
*** ijw has quit IRC18:07
fungino for two reasons, first i'm not subscribed to discuss.python.org and second the pep says feedback should go to the pypa-dev google group forum/ml18:07
*** trident has quit IRC18:08
*** amoralej is now known as amoralej|off18:08
fungibut if it doesn't eventually show up i can try signing into my infrequently-used google account and doing unspeakable $things18:08
*** ianychoi has quit IRC18:10
zbrplease do, i am going to make some popcorn18:10
AJaegerclarkb: let me add opendev/infra-manual to the queue...18:11
*** matt_kosut has quit IRC18:15
zbrby looking at the stats of the google ML, it appears to be far less popular than the discuss "packaging" category. i kinda prefer discuss because is not isolated from python, which also have a PEPs section and more important: i can subscribe to the topics I care about, not as in the ML which is all or nothing.18:15
*** armax has quit IRC18:15
*** ianychoi has joined #openstack-infra18:17
*** trident has joined #openstack-infra18:18
openstackgerritAndreas Jaeger proposed openstack/project-config master: Move openstack/infra-manuals to opendev  https://review.opendev.org/71214918:22
AJaegerclarkb: ^18:22
openstackgerritAndreas Jaeger proposed openstack/project-config master: Move openstack/infra-manuals to opendev  https://review.opendev.org/71214918:24
clarkbAJaeger: thanks. I'm eating early lunch then will catch up on meeting prep18:27
*** diablo_rojo has joined #openstack-infra18:28
AJaegerclarkb: enjoy!18:30
*** factor has joined #openstack-infra18:36
openstackgerritMerged openstack/project-config master: Add new Validation Framework projects  https://review.opendev.org/71191018:38
rm_workmtreinish: hey do you have "core" on https://github.com/testing-cabal/subunit ?18:39
rm_workmtreinish: it's non-py3-able right now because the required fixes haven't been released yet... IE https://github.com/testing-cabal/subunit/commit/24a2465c92368784800562e105c920db0c99787018:39
fungirm_work: lifeless may also be worth trying to reach on that18:40
rm_workhmm looks like you're not in the maintainers list, just saw your name come up on an issue there and figured you might know the right people at least -- i've pinged a numbber of them previously18:40
rm_workyeah I think last time lifeless told me to ping jelmer? and I tracked him down in some channel (he has PMs disabled and is hidden) and he said "sure sometime" but hasn't done it18:41
rm_workalso looking for a release of https://github.com/testing-cabal/testtools18:42
rm_workthey release like once every two years and I'm dieing T_T18:42
*** gfidente is now known as gfidente|Dafk18:49
*** gfidente|Dafk is now known as gfidente|afk18:49
fungiclarkb: corvus: zbr: i think i figured it out. google groups was tossing my replies because they were pgp+mime signed18:52
mordredfungi: and they don't like signatures?18:52
fungiwell, at least setting the message to not be signed got it to go through to the list18:52
mordredsigh18:52
mtreinishrm_work: I think I have commit and merge access on both subunit and testtools. But, I haven't been able to keep up with the maint on either package19:00
rm_work:(19:00
mtreinishbefore I push a release I need to check that everthing is working on master, and everytime I go to check things tend to be kind of stale19:01
mtreinishon the subunit side before we push a release I'd like to see stephenfin's PR: https://github.com/testing-cabal/subunit/pull/40 reach a conclusion19:01
*** ijw has joined #openstack-infra19:01
mtreinishbecause that's the top bug people are hitting with stestr/subunit on py319:02
*** ijw_ has quit IRC19:05
mtreinishrm_work: I will try to find time to review things this week (but I have a pretty busy schedule, I've got a release crunch on a work project)19:05
rm_workmtreinish: i hear you :D19:06
rm_workand yeah that makes sense19:06
rm_workno point doing a py3 release when not all the stuff is solved19:06
*** ralonsoh has quit IRC19:07
*** eharney has quit IRC19:08
*** igordc has joined #openstack-infra19:09
rm_workah speaking of stestr: http://paste.openstack.org/show/790522/ <-- `stestr run` leads to `run: error: the following arguments are required: command`19:10
rm_workanyone seen this before?19:10
rm_workit's like it's getting a completely different set of args for "run" than it normally should19:11
mtreinishrm_work: I have vague memory of that, there was a bug in glance something or an oslo lib that was messing with argparse19:16
rm_workhmm19:16
mtreinish(or I remember getting pinged about something like that not too long ago)19:16
mtreinishbnemec: ^^^ do you remember the details? (I think you pinged me about it)19:17
bnemecI know there was a patch. I'm looking for it now.19:17
*** matt_kosut has joined #openstack-infra19:18
bnemecmtreinish: rm_work: I think this was the fix in glance: https://review.opendev.org/#/c/701580/19:18
mtreinishbnemec: thanks, that's what I was sort-of remembering :)19:20
openstackgerritTristan Cacqueray proposed zuul/zuul master: spec: add a zuul-runner cli  https://review.opendev.org/68127719:22
*** yamamoto has joined #openstack-infra19:23
*** mugsie has quit IRC19:23
rm_workoh, so pretty recent... hmm19:25
*** matt_kosut has quit IRC19:25
*** mugsie has joined #openstack-infra19:26
*** yamamoto has quit IRC19:28
*** jamesmcarthur has quit IRC19:30
*** openstackgerrit has quit IRC19:32
*** ahosam has joined #openstack-infra19:35
*** armax has joined #openstack-infra19:36
*** lbragstad_ has joined #openstack-infra19:55
*** eharney has joined #openstack-infra19:56
*** lbragstad has quit IRC19:58
AJaegerconfig-core, we can remove fedora-29 nodes now, please review https://review.opendev.org/#/c/712032/20:01
*** slaweq has quit IRC20:03
ShrewsAJaeger: I do not recommend removing from the builder nodepool.yaml at the same time you remove from the launcher nodepool.yaml unless you've confirmed that there are no active fedora-29 nodes in any provider20:05
clarkbShrews: whats the concern there?20:06
clarkbthe builder manages the images and as long as we don't try to boot new nodes that shouldn't cause any problems20:06
* AJaeger sets a WIP quickly20:06
clarkband since we aren't removing any providers themselves it should know how to clean up nodes if told to directly20:06
Shrewsif there are live fedora29 instances when the builder tries to delete an uploaded image that is in use, it will error20:07
clarkbShrews: only in vexxhost and it will retry until the instance goes away20:07
clarkbit isn't fatal thankfully20:07
AJaegerShrews: there should not be a job running, we rmoeved them 12 hours ago20:07
AJaeger(more than 12 hours even)20:08
Shrewsthere is a min-ready of 1 though20:08
AJaegerso, first set that to 0, then merge?20:08
clarkbShrews: ya that just means we'll retry on deleting the image from vexxhost sjc1 until all volumes using the image there delete20:08
clarkbAJaeger: that won't acutllay delete the instances either. I'm checking now if there are any fedora-29 nodes and will manaully issue deletes if so20:08
*** jcapitao_off has quit IRC20:09
Shrewsclarkb: i suppose it will retry, true. I guess it's ok then20:09
clarkb| 0003587456 | vexxhost-sjc1       | fedora-29-vexxhost            | 62e01c16-019d-46b8-b92b-b3e409161e97 | 38.108.68.35    |                                         | ready    | 368:07:00:35 | unlocked |20:09
AJaegerclarkb: I see , so I'll remove the WIP20:09
clarkbsomething tells me that that server isn't going to delete on its own20:09
clarkbAJaeger: ya I think you are fine to merge it20:09
AJaegerthanks, clarkb and Shrews20:09
*** openstackgerrit has joined #openstack-infra20:12
openstackgerritMerged openstack/project-config master: Remove Fedora 29 from nodepool  https://review.opendev.org/71203220:12
AJaegerand gone ^20:12
AJaeger#status log Fedora 29 images are retired20:12
AJaegerdid we loose the status bot again?20:13
clarkbseems like only gerritbot and meetbot are here20:13
ianwcorvus: repsonded in change, but there's quite a few things mapped into the container that are (correctly, i would say) owned by the local "nodepool" user20:15
*** jamesmcarthur has joined #openstack-infra20:17
*** ianychoi has quit IRC20:17
ianwalthough there is probably a case for things like dib_tmp being some sort of ephemerally attached volume storage in the container20:17
ianwhowever, i'd like to KISS for the initial deployment20:18
*** ianychoi has joined #openstack-infra20:18
clarkbAJaeger: it last logged entries in its log file on march 5, I have restarted it, but it may take a minute to join this channel20:19
*** openstackstatus has joined #openstack-infra20:19
*** ChanServ sets mode: +v openstackstatus20:19
clarkbthere it is20:19
clarkbAJaeger: if you want to rerun your status now20:19
AJaeger#status log Fedora 29 images are retired20:19
openstackstatusAJaeger: finished logging20:19
AJaegerthanks, clarkb !20:19
clarkbthe process itself was still running, I guess it got netsplit and stopped having messages to process and thus no more logging20:21
corvusianw: i think you've convinced me on 3/4 of those; does what i wrote make sense?  (i want to make sure i'm following the situation and requirements correctly)20:25
corvusmordred: can you take a look at the conversation between ianw and i on https://review.opendev.org/710891 -- it relates to container user ids, and i wonder if you have any left-field ideas20:26
mordredcorvus: reading20:26
*** dave-mccowan has quit IRC20:27
*** rh-jelabarre has quit IRC20:28
mordredcorvus: I left a comment which may be useful or may be completely off base20:34
ianwmordred: so we export the build logs via http so people can see their build status20:37
ianwwhile that could also be containerised ... yeah i'm back to KISS :)20:37
corvusmordred: i think the main thing is that we need to set the uid of the logs so that nodepool can write to it when it's mounted20:38
corvusmordred: i agree, we probably don't have to create a user for it, we could just set the uid to 1001 and be done -- but we do need to set it to 1001 if we're going to mount it in the host20:38
corvus(and i think we should mount it in the host so we don't have to exec into the container to debug)20:39
ianwcorvus/mordred: i'm putting some tasks @ https://storyboard.openstack.org/#!/story/2007407 about what i think we can look at progressively20:39
corvusianw: iiuc, your objection to making dib_tmp a volume right now is that it would be backed by the wrong filesystem on the host?20:40
*** mattw4 has quit IRC20:40
ianwyes, and that i feel like it's something we can try out once we have a POC20:42
ianwi'm just proposing we build fedora on this for now, as we learn how it will work20:42
openstackgerritClark Boylan proposed zuul/nodepool master: Install zypper on the nodepool-builder image  https://review.opendev.org/71217720:43
corvusianw: fwiw, i agree with your kiss, i'm trying to see if we can make it simpler.  :)20:43
clarkbmordred: if you have a moment https://review.opendev.org/#/c/712121/3 is a quick rereview after I fixed the unmatch ' in ansible yaml bug20:44
clarkbI can trigger that directly instead of waiting for the periodic run once it lands too20:44
corvusianw: change lgtm, i think mordreds +0 comment about tag may be a -1; you should take a look at that20:44
ianwthanks, will do soon20:45
mordredyeah - I think otherwise we'd have to have the : in the tag name if we used it20:45
mordred(it's academic right now - there are no tags)20:45
corvusbut ideally code review helps us avoid landing future bugs :)20:48
clarkbhrm actually thinking about https://review.opendev.org/#/c/712121/ a bit more do I need to worry about that zcat somehow ending up in the console log?20:51
clarkbfungi: ^ you probably know if that is a risk with bash20:51
corvusclarkb: zcat where?20:52
clarkbcorvus: https://review.opendev.org/#/c/712121/3/playbooks/periodic/goaccess.yaml line 5820:52
*** rcernin has quit IRC20:53
clarkbif goaccess were to crash early maybe? though in that case I think you get an error and not output to stdout?20:53
corvusclarkb: ah, you're asking if that command can somehow go wrong and, in that case, something be output to the zuul console log for the job20:54
*** Lucas_Gray has joined #openstack-infra20:54
clarkbyes20:54
mordredyeah - the only way I can see the zcat going to stdout would be if goaccess decided to crash with an error message and included the contents of stdin in that error message20:55
corvusclarkb: i can't think of one, but i'm in the same boat as you, and would love fungi's opinion on that.  but if we want to be paranoid, we could put that in a subshell that redirects everything to a file or /dev/null -- so that if something goes wrong, there's something one level up to catch it.20:55
corvusmordred: yeah, which seems crazy20:56
clarkbmordred: ya I've done some testing of the goaccess side and its pretty quiet20:56
fungii don't think the shell is at risk, but it's possible goaccess itself might echo loglines in error messages?20:56
clarkbfungi: I havne't encountered that yet but will double check their man page for clues20:56
mordredlike - I've never seen bash start sending a stream to stdout when it was redirected to stdin of a process after that process failed20:56
mordredand I think if it did it would be a huge bug :)20:57
clarkbI can actually feed it junk pretty trivially and see if it echos those lines20:57
clarkbgoaccess that is20:57
fungii do still think it might be cleaner to cat/zcat all the files in a subshell (similar to corvus's suggestion) and then only process -20:58
fungibut that's really just from a readability perspective20:59
fungiif you were worried about it, you could concatenate them all into a tempfile and then clean up the tempfile after processing20:59
fungii assume goaccess doesn't care that the lines it processes are traversed in chronological order21:01
clarkbfungi: ya it uses the time stamps in the logs themselves21:01
clarkbdoing some testing locally feeding it bad input it gives you a very limited error (likely for this reason)21:01
corvusgiven that, i think the task as written is probably good?21:01
clarkbbasically gives you the single token that doesn't match and then the parser function that tried to match it21:01
clarkbcorvus: ya I think its probably safe given this testing locally21:02
clarkbfungi: what does that subshell look like $(cat log log.1 ; zcat log.*.gz) | goaccess - ?21:03
fungiclarkb: almost, drop the $21:04
fungiand put spaces around the ( and )21:04
fungi(i think they need spaces anyway)21:04
clarkbk, can do that in a followon (which we need anyway to publish results once happy with them)21:04
fungiyeah, spaces are optional. you can also log{,.1}21:06
fungiwhich may be more readable depending on if there's a lengthy leading path21:06
*** xek has quit IRC21:07
*** lbragstad_ has quit IRC21:11
rm_workhmmm are folks dealing with this already? https://github.com/pallets/markupsafe/issues/5721:16
rm_workmay hit stable branches, like glance and designate at least on rocky21:16
openstackgerritMerged opendev/system-config master: Add periodic job to run goaccess reporting  https://review.opendev.org/71212121:16
rm_worki wonder if we're seeing it downstream due to some other bad pinning though...21:17
rm_workoh nm https://review.opendev.org/#/c/711930/ is happening already21:18
clarkbrm_work: horizon hit it with pyscss21:18
rm_workdid setuptools actually un-deprecate?21:18
rm_worki assumed they wouldn't do that and it'd be up to us to upgrade other libs21:19
*** factor has quit IRC21:19
clarkbnot sure I understand the question. un-deprecate?21:19
rm_worki mean, the issue is they deprecated (and removed) some code21:19
rm_workthis is skipping (!=) release 46.0.021:20
clarkbrm_work: ya that won't work21:20
rm_workbut when they release 46.0.1 we'll be in the same boat21:20
clarkbyou either need to cap or fix the libs21:20
rm_workright21:20
rm_workso what's the point of this patch21:20
rm_workbuying is an undetermined (could be MINUTES?) amount of time? lol21:20
rm_work*buying us21:20
clarkbrm_work: I don't think people understand the underlying problem if that is the proposed fix21:20
clarkbianw: do you hvae an example zuul enqueue command for periodic jobs? (or is it enqueue-ref?)21:21
clarkbzuul enqueue-ref --tenant openstack --pipeline periodic --project opendev/system-config --ref refs/heads/master maybe?21:25
clarkbthe vars in an example inventory file don't exactly line up with enqueue or enqueue-ref21:26
*** jamesmcarthur has quit IRC21:28
*** jamesmcarthur has joined #openstack-infra21:28
rm_worksmcginnis: ^^ or see comment on https://review.opendev.org/#/c/711930/21:30
rm_workprometheanfire: ^^21:30
ianwclarkb: umm, did you check my history?21:30
clarkbianw: I hadn't but will do so now21:31
smcginnisrm_work: It's a stopgap while the actual fix is done. We don't want to cap setuptools forever.21:31
smcginnisrm_work: Is there a replacement for psScss21:31
rm_worksmcginnis: ok, so we just hope they don't release in the next ... day?21:31
smcginnis*pyScss21:31
clarkbianw: ok its there thanks21:31
rm_worksmcginnis: this is also a problem for markupsafe21:31
rm_workhttps://github.com/pallets/markupsafe/issues/5721:31
ianwclarkb: heh, yeah that's the only way i remember :)21:31
rm_workwhich has a fix in 1.1.1 but most stable branches cap at 1.1.021:31
rm_worksorry, 1.021:32
clarkbianw: though I don't see system-config in the list as I expected, but can adapt from the others easily enough21:32
smcginnisI see MarkupSafe in upper-constraints, but not in global-requirements.21:32
rm_workfor example, designate and glance in their rocky branches21:32
rm_workyeah it's probably an indirect-dep only21:33
*** ahosam has quit IRC21:33
rm_workhttps://github.com/openstack/requirements/blob/stable/rocky/upper-constraints.txt#L10921:33
rm_workstein+ is fine21:34
*** ahosam has joined #openstack-infra21:34
clarkboh neat we don't start a console log streamer on the remote in those jobs so we don't actualyl stream that into the console log21:34
clarkbthat might be advantageous here? in any case I think it failed because goaccess hasn't been installed yet21:34
rm_workI don't know that we can bump a requirement in rocky, may have to pin setuptools in rocky?21:35
clarkbI'll re enqueue once ansible installs goaccess21:35
*** ociuhandu has joined #openstack-infra21:37
clarkbrm_work: you can't reliably pin setuptools, it tends to already be there when you start doing things and when it downgrades itself that applies to the next process invocation not the current one21:40
fungiwhich means you can't easily control it in tox invocations either because tox wants to install everything in one shot21:42
rm_workeugh yeah21:43
rm_workso ... can we bump upper-constraints for things in stable branches? :/21:43
*** jamesmcarthur has quit IRC21:44
rm_workwe will need markupsafe==1.1.1 and probably pyscss greater than whatever is there (assuming they fix the issue?)21:44
*** Lucas_Gray has quit IRC21:45
*** jamesmcarthur has joined #openstack-infra21:45
prometheanfirehi21:48
rm_work:)21:48
prometheanfirerm_work: pyscss is the problem for horizon at least, and it may be a dead project21:50
prometheanfirefixing the actual issue is preferred but may not be possible, so stop using deps that are bad, mmmmk?21:51
prometheanfirestop hitting yourself :P21:51
rm_worklol21:52
rm_workyeah so how do we resolve this permanently? we can all agree this fix that is merging is not going to last, right? :D21:52
rm_workmaybe not even more than a day or so21:52
*** jamesmcarthur has quit IRC21:53
rm_workfor markupsafe (which hits a few projects in their stable/rocky branches) it is already fixed, so we'd just need to bump the upper-constraints for it in rocky requirements21:55
*** matt_kosut has joined #openstack-infra21:55
*** matt_kosut has quit IRC21:56
prometheanfirein python nothing is permament21:56
rm_worki'm not sure about the fix for pyscss21:56
prometheanfiremarkupsafe bump is fine21:56
prometheanfirenot sure about pyscss21:56
rm_workwe might need to submit PRs and rouse the authors from their slumber21:56
prometheanfirethe skip patch just merged, so we are good now til we aren't again21:57
prometheanfirethen again, we don't track setuptools in constraints, this may be fixed there, if only we didn't have the chicken/egg problem21:57
*** verdurin has quit IRC21:58
*** sreejithp has joined #openstack-infra21:59
rm_workdo i just need to change upper-constraints or is there more?21:59
prometheanfireno, it's not in upper-constraint, setuptools chicken/egg etc22:00
prometheanfireit'd be nice if we could include it imo22:00
clarkbmordred won't be happy with this but the pyproject stuff would address it I think22:00
clarkb(because you can set the libs you need to do an install and their versions22:01
*** jcapitao_off has joined #openstack-infra22:01
*** sreejithp has quit IRC22:02
prometheanfireya22:02
prometheanfireI've suggested that before22:03
rm_workprometheanfire: i mean this: https://review.opendev.org/#/c/712187/22:03
fungiwell, a big part of the solution for this is and has been the deprecation of setuptools and its eventual removal, yes22:04
fungiat least the "solution" as endorsed by the folks actively discussing in distutils-sig and pypa-dev and the like22:05
*** jamesmcarthur has joined #openstack-infra22:05
fungiand maintaining a lot of the related toolchain pieces22:05
clarkbare they actually replacing setuptools? the new tooling allows you to but I hvaen't really see any movement towarsd that22:05
fungii forget the name of the new thing which is basically "splat files into a directory"22:06
prometheanfirerm_work: thanks22:07
fungii think it gets a mention in pep 51822:07
rm_workprometheanfire: actually, I *assume* this is an issue in upstream gates... but i've only actually seen it downstream when attempting to run tests given that it's really hard to ensure the right version of setuptools everywhere...22:08
rm_worki am THINKING it'll probably show up in the gate too but i haven't proven it. it'll definitely show up for random people doing builds (as it did for me)22:08
clarkbrm_work: it could be delay in image build cycle. Depending on when that setuptools release happened it could take 24 hours for the gate to update22:09
rm_workyeah, k22:09
*** jamesmcarthur has quit IRC22:10
fungiclarkb: nevermind, it was mentioned in discussions of pep 517: flit22:14
fungihttps://pypi.org/project/flit/22:14
clarkbthanks22:15
fungii don't think the plan is for flit to be *the* setuptools replacement, but rather for peps 517/518 to make it possible to have a variety of setuptools replacements emerge filling different niches while still all interoperating at the package installation level22:15
fungiflit just happened to be the first viable (albeit trivial) example22:16
fungireally it's separating build-time operations from install-time operations which is the goal, and making sure install-time operations can be handled by pip directly22:17
*** evrardjp has quit IRC22:17
*** matt_kosut has joined #openstack-infra22:17
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-builder: deploy from container  https://review.opendev.org/71089122:17
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-builder: Add webserver  https://review.opendev.org/71157422:17
openstackgerritIan Wienand proposed opendev/system-config master: Add nb01.opendev.org  https://review.opendev.org/69311922:17
clarkbfungi: ya but you still need tools that can make sdists and wheels22:17
clarkb(so that pip can then consume that from pypi)22:17
fungiright22:18
*** evrardjp has joined #openstack-infra22:18
fungiidea being setuptools is one of many (and hopefully increasingly abandoned in favor of simpler alternatives not encumbered by its legacy of api backward-compatibility obligations)22:19
*** ianychoi has quit IRC22:19
*** ianychoi has joined #openstack-infra22:20
*** matt_kosut has quit IRC22:21
clarkbgoaccess has been installed so I've reenqueued the periodic job again to see how it goes22:26
clarkbit seems to have worked. I'm trying to view the report now22:27
ianwcool, that was quick22:28
*** weifan has joined #openstack-infra22:29
*** weifan has quit IRC22:29
clarkbthe gui view filtered as expected. I'm reading through the json export now to see if it did as well22:31
*** evrardjp has quit IRC22:31
clarkbstatic.opendev.org:/home/zuul/zuul-ci.org_goaccess_report.html if others want to see it22:31
*** evrardjp has joined #openstack-infra22:33
*** dciabrin_ has joined #openstack-infra22:33
clarkbthe json looks clean too22:33
ianwlgtm22:34
ianwi wonder if you can filter out .css files ... it doesn't seem that useful to know they're the most requested22:35
ianwyou really want to know what bits of content are accessed22:35
clarkbianw: ya thats why they break it out into two different graphs22:35
clarkbrequests and static requests22:35
clarkbI think you can largely ignore "static requests"22:35
*** dciabrin__ has quit IRC22:35
ianwohh, yeah ok i see now22:36
*** dpawlik has quit IRC22:40
*** ianychoi has quit IRC22:41
*** ociuhandu has quit IRC22:41
*** ianychoi has joined #openstack-infra22:42
mordredclarkb: I'd been thinking that once 517/518 were a real thing we could teach pbr to not use setuptools and instead just make sdists and wheels directly, since we need WAY less flexibility anyway22:48
mordredbut - you know - ENOTIME22:49
*** jcapitao_off has quit IRC22:49
mordredclarkb, prometheanfire: my biggest pushback against pyproject at _this_ point is that it's less featureful for a few use cases we actively use22:49
mordredthe model is not a bad model - but it only has 2 sets of pacakges, prod and dev, so we'd lose having docs requirements split out22:51
openstackgerritClark Boylan proposed opendev/system-config master: Simplify goaccess command  https://review.opendev.org/71220122:51
openstackgerritClark Boylan proposed opendev/system-config master: Pull goaccess report into zuul logs  https://review.opendev.org/71220222:51
clarkbmordred: ^ I've sort of taken your work on making log stuff templated for zuul jobs and untemplated it because static.o.o isn't a test node with all those assumptions built in. Can you review that and make sure its sane that way?22:51
openstackgerritIan Wienand proposed openstack/project-config master: nodepool-elements: Use venv for utilities  https://review.opendev.org/70775022:51
mordredalso - last I checked, you can't create the lock file while referencing a shared constraints file - so while the lock construct solves things on a per-project basis it doesn't really let us express the same thing22:51
clarkbmordred: ya we'd probably have to produce our own lockfiles22:52
clarkb(not sure if that is a good idea or not)22:52
mordredwell - we could for sure22:53
*** tkajinam has joined #openstack-infra22:53
mordredalong with the de-setuptools-ify pbr mentioned earlier22:53
mordredbut I think we need for some of that to still solidify a bit more22:53
clarkbbut also I think you can use pyproject.toml without a lockfile22:53
mordredclarkb: I'm assuming that report gets replaced each time right?22:53
clarkband still feed it the -c constraints when piping22:53
clarkbmordred: yes22:53
mordredclarkb: you can - but the only real benefit to pyproject.toml is the lockfile, or?22:54
clarkbmordred: you can specify you need older setuptools as an example related to the current situation without a lockfile22:54
clarkbmordred: then you can furhter lock that specificaton to one version if you want22:54
mordredI don't think that really gets us anything22:55
mordredI mean - at least not quickly22:55
mordredand if we're gonna go into setuptools-less 518 land, I think we can do better than pyproject.toml for our use cases22:55
clarkbya I don't actually know what the answer is here22:56
mordredclarkb: the logs thing looks fine assuming that's making a whole new report22:56
clarkbmordred: ya its not appending it replaces22:57
mordredcool. then looks great22:57
clarkb(I tested that locally with goaccess, ot will overwrite the output)22:57
mordredprobably shoudl be an artifact22:57
mordredjust for, you know, completeness - but that could be a followup22:57
*** igordc has quit IRC22:58
clarkbya I didn't want to figure out the zuul_return right away22:58
clarkb(thats mostly me being lazy and wanting something published though)22:58
mordredclarkb: both are 2x+2 - I +A'd the second one but left off the +A on the first22:58
mordredso - fire away whenever22:58
clarkbok. ianw ^ I've got about an hour left in my day which should be enough time to monitor that but if not any chance you are able to trigger the periodic job again and take a look at it?22:58
ianwclarkb: sure22:59
mordredclarkb: I think before we go anywhere near 518 or pyproject we need to step back and assess our goals an our actual current contraints22:59
clarkbthanks!22:59
mordredbecause 518/pyproject is essentially a COMPLETE redesign for us due to how complex our current thing is - and I think if we're going to do that it's worth stepping back and whiteboarding for a while22:59
*** bdodd has quit IRC23:00
*** lbragstad_ has joined #openstack-infra23:00
ianwinfra-root: if i could get final eyes on https://review.opendev.org/#/c/711574 (export logs via web for new containered nodepool-builder) and https://review.opendev.org/#/c/693119/6 (add host to ansible) i can fiddle with that today and make sure it's coming up23:00
mordredit's possible pyproject might be the path forward - or there might be a door number 323:00
clarkbmordred: I've heard good things about poetry (which gloms itself into the plugin system that you describe in pyproject.toml)23:01
ianwinfra-root: and i'd also like to get in https://review.opendev.org/707750 which will switch our images to venv tools, and build a new image type deployed in rax without any of the pip/virtuatlenv stuff ... then we can look at a job to restore it via ansible23:01
mordredclarkb: yeah -I looked at it a while ago - it wasn't super happy making - but I can look at it again23:01
ianw(dib has been released with changes and should deploy to builders soon)23:01
fungimordred: the recommendation i saw recently was to use extras to specify documentation build and test requirements (apparently extras named "docs" and "test" are special in some way)23:04
clarkbmordred: I think it does address the global lockfile issue fwiw23:04
clarkbmordred: because you can install all the things into an env with it then lock that set rather than a specific package23:04
clarkbianw: that is failing linting23:05
ianwyeah looking23:05
clarkbhttps://zuul.opendev.org/t/openstack/build/bb696536b6ae47988f65434657b2b9ef/log/job-output.txt#50323:05
clarkbianw: ^23:05
mordredclarkb: I don't think it does - but I don't have the actual brainspace to dig in fully and express why right now23:06
ianwoh, that's because the new dib release uncapped hacking23:06
*** Tengu has quit IRC23:11
openstackgerritIan Wienand proposed openstack/project-config master: nodepool-elements: Use venv for utilities  https://review.opendev.org/70775023:12
openstackgerritIan Wienand proposed openstack/project-config master: Fix new dib-lint errors  https://review.opendev.org/71220423:12
ianwclarkb: ^ that should fix the project-config gate23:12
*** Tengu has joined #openstack-infra23:12
*** zxiiro has quit IRC23:15
*** dchen has joined #openstack-infra23:19
*** dchen has quit IRC23:19
*** dchen has joined #openstack-infra23:21
*** lbragstad_ has quit IRC23:25
openstackgerritMerged opendev/system-config master: nodepool-builder: deploy from container  https://review.opendev.org/71089123:25
*** rcernin has joined #openstack-infra23:26
*** armax has quit IRC23:32
mordredclarkb: https://review.opendev.org/#/c/712177/ has 3x+223:32
clarkbmordred: we actually build that image as part of testing so it should be safe to approve it right?23:33
clarkbshould I hit +A?23:33
openstackgerritMerged openstack/project-config master: Fix new dib-lint errors  https://review.opendev.org/71220423:34
openstackgerritMerged openstack/project-config master: nodepool-elements: Use venv for utilities  https://review.opendev.org/70775023:34
mordredclarkb: yeah23:35
clarkbdone23:35
mordredwoot23:35
mordredbetween that and ianw's work - I'm pretty excited there23:36
ianwmordred: if you could just look over the webserver bits at https://review.opendev.org/#/c/711574 the rest is just mechanical to add the host23:37
*** lbragstad_ has joined #openstack-infra23:38
ianwgrabbing a tea then hopefully non-pip-virtualenv images will be building23:43
*** ahosam has quit IRC23:46
clarkbmm tea23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!