Tuesday, 2019-02-19

*** smarcet has joined #openstack-infra00:06
*** emccormick has joined #openstack-infra00:08
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675900:15
*** emccormick has quit IRC00:21
*** wolverineav has joined #openstack-infra00:22
*** jamesmcarthur has quit IRC00:22
*** rascasoft has joined #openstack-infra00:23
*** emccormick has joined #openstack-infra00:24
openstackgerritMerged openstack-infra/zuul-jobs master: Fix build-docker-image when using buildset_registry  https://review.openstack.org/63765000:25
*** sdake has joined #openstack-infra00:25
*** wolverineav has quit IRC00:26
*** chandankumar has quit IRC00:28
*** chandankumar has joined #openstack-infra00:29
*** smarcet has quit IRC00:32
*** rascasoft has quit IRC00:34
*** ijw has quit IRC00:35
*** ijw has joined #openstack-infra00:35
*** jamesmcarthur has joined #openstack-infra00:35
*** jamesmcarthur has quit IRC00:40
*** slaweq has quit IRC00:41
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675900:42
*** ijw has quit IRC00:52
*** tosky has quit IRC00:57
*** sdake has quit IRC00:58
openstackgerritClark Boylan proposed openstack-infra/zuul master: Don't request PR issue data  https://review.openstack.org/63672800:59
*** wolverineav has joined #openstack-infra00:59
corvusclarkb, fungi, mordred: https://review.openstack.org/637037 succeeded!  next up is a job that depends on that one to test out provides/requires!01:00
clarkbnice01:00
openstackgerritJames E. Blair proposed openstack-infra/zuul-preview master: WIP: test docker registry  https://review.openstack.org/63703701:01
*** wolverineav has quit IRC01:04
*** markvoelker has joined #openstack-infra01:10
*** sdake has joined #openstack-infra01:15
*** sdake has quit IRC01:16
*** rlandy has quit IRC01:18
openstackgerritJames E. Blair proposed openstack-infra/system-config master: WIP: Run zuul-preview  https://review.openstack.org/63765401:21
*** emccormick has quit IRC01:22
*** whoami-rajat has joined #openstack-infra01:22
*** sdake has joined #openstack-infra01:22
corvushrm.  that change doesn't show up in the builds ui, so i think there may be a bug there01:24
corvushttp://zuul.openstack.org/builds?change=63703701:24
*** ijw has joined #openstack-infra01:29
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: amqp: add basic trigger  https://review.openstack.org/63745801:32
*** hongbin has joined #openstack-infra01:33
*** rascasoft has joined #openstack-infra01:37
*** ijw has quit IRC01:41
*** ijw has joined #openstack-infra01:41
*** sdake has quit IRC01:42
*** markvoelker has quit IRC01:44
*** sdake has joined #openstack-infra01:44
*** jamesmcarthur has joined #openstack-infra01:46
*** rascasoft has quit IRC01:49
*** zaneb has joined #openstack-infra01:50
*** wolverineav has joined #openstack-infra01:53
*** sdake has quit IRC01:54
*** jamesmcarthur has quit IRC01:56
*** jamesmcarthur has joined #openstack-infra01:58
*** sdake has joined #openstack-infra01:59
fungiclarkb: thanks for the heads up on the storyboard migration for karbor02:03
fungiclarkb: for some reason we're not tracking dig repos in https://git.openstack.org/cgit/openstack/governance-sigs/tree/sigs.yaml and have them in https://git.openstack.org/cgit/openstack/governance/tree/reference/sigs-repos.yaml02:17
fungier, sig repos02:18
*** wolverineav has quit IRC02:20
*** wolverineav has joined #openstack-infra02:20
*** sdake has quit IRC02:22
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables  https://review.openstack.org/63766602:27
*** wolverineav has quit IRC02:30
*** markvoelker has joined #openstack-infra02:41
*** jamesmcarthur has quit IRC02:41
*** wolverineav has joined #openstack-infra02:47
*** wolverineav has quit IRC02:51
*** psachin has joined #openstack-infra02:59
*** hwoarang has quit IRC02:59
*** wolverineav has joined #openstack-infra03:00
*** hwoarang has joined #openstack-infra03:02
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables  https://review.openstack.org/63766603:08
openstackgerritIan Wienand proposed openstack-infra/nodepool master: [dnm] testing devstack fix  https://review.openstack.org/63766903:11
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add triggers information to pipeline list  https://review.openstack.org/63767003:13
*** markvoelker has quit IRC03:13
*** apetrich has quit IRC03:14
*** ricolin has joined #openstack-infra03:15
*** dklyle has joined #openstack-infra03:23
*** rascasoft has joined #openstack-infra03:26
*** roman_g has quit IRC03:27
*** ricolin has quit IRC03:33
*** wolverineav has quit IRC03:34
*** rascasoft has quit IRC03:34
*** rfolco has quit IRC03:36
*** ykarel|away has joined #openstack-infra03:36
*** ykarel|away is now known as ykarel03:37
*** dave-mccowan has joined #openstack-infra03:45
*** psachin has quit IRC03:50
*** wolverineav has joined #openstack-infra03:54
*** udesale has joined #openstack-infra03:54
*** wolverineav has quit IRC03:54
*** psachin has joined #openstack-infra04:00
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add initial driver and event  https://review.openstack.org/55515304:01
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add web route and rpclistener  https://review.openstack.org/55483904:01
prometheanfirewhat happened to make the images grow between itterations? https://nb01.openstack.org/images/04:02
*** hrubi has quit IRC04:02
*** hrubi_ has joined #openstack-infra04:02
*** agopi has quit IRC04:06
*** agopi has joined #openstack-infra04:07
*** andreaf has quit IRC04:08
*** andreaf has joined #openstack-infra04:09
*** hamzy has quit IRC04:13
*** jlvillal has quit IRC04:14
*** janki has joined #openstack-infra04:14
*** jlvillal has joined #openstack-infra04:14
*** jamesdenton has quit IRC04:16
*** jamesdenton has joined #openstack-infra04:16
*** Shrews has quit IRC04:17
*** Shrews has joined #openstack-infra04:19
*** Shrews has quit IRC04:24
*** wolverineav has joined #openstack-infra04:25
*** Shrews has joined #openstack-infra04:25
*** hrubi_ has quit IRC04:26
*** jamesmcarthur has joined #openstack-infra04:26
*** zaneb has quit IRC04:27
*** wolverineav has quit IRC04:29
*** jamesmcarthur has quit IRC04:31
*** chandankumar is now known as chkumar|ruck04:32
*** zaneb has joined #openstack-infra04:42
*** zzzeek has quit IRC04:49
*** zzzeek has joined #openstack-infra04:52
*** irclogbot_1 has quit IRC04:57
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: amqp: add message informations to the job variables  https://review.openstack.org/63766604:57
*** jamesmcarthur has joined #openstack-infra04:58
*** bhavikdbavishi has joined #openstack-infra05:00
*** jamesmcarthur has quit IRC05:03
*** jrist has quit IRC05:06
*** ykarel has quit IRC05:07
*** jrist has joined #openstack-infra05:10
*** dave-mccowan has quit IRC05:13
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675905:22
*** ykarel has joined #openstack-infra05:23
*** wolverineav has joined #openstack-infra05:27
*** ijw has quit IRC05:27
*** ijw has joined #openstack-infra05:29
*** wolverineav has quit IRC05:32
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Keep git after ironic-agent post  https://review.openstack.org/63716205:32
*** udesale has quit IRC05:33
*** ijw has quit IRC05:34
*** jamesmcarthur has joined #openstack-infra05:38
*** ricolin has joined #openstack-infra05:40
*** hongbin has quit IRC05:41
*** markvoelker has joined #openstack-infra05:41
*** udesale has joined #openstack-infra05:43
*** jamesmcarthur has quit IRC05:43
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix kube config permission  https://review.openstack.org/63768205:48
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: DNM: test install-kubernetes fix  https://review.openstack.org/63768305:49
*** wolverineav has joined #openstack-infra05:52
*** kjackal has joined #openstack-infra05:59
*** wolverineav has quit IRC05:59
*** wolverineav has joined #openstack-infra06:01
*** wolverineav has quit IRC06:06
*** jamesmcarthur has joined #openstack-infra06:10
*** ricolin has quit IRC06:13
*** markvoelker has quit IRC06:14
ianwprometheanfire: this is what DIB_SHOW_IMAGE_USAGE would help with, i feel like we turned it off at some point06:14
*** jamesmcarthur has quit IRC06:15
openstackgerritIan Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation  https://review.openstack.org/63675906:19
*** e0ne has joined #openstack-infra06:25
openstackgerritIan Wienand proposed openstack-infra/infra-specs master: letsencrypt spec  https://review.openstack.org/58728306:26
*** e0ne has quit IRC06:26
*** quiquell|off is now known as quiquell|rover06:29
*** ricolin has joined #openstack-infra06:34
*** ccamacho has quit IRC06:35
*** hrubi has joined #openstack-infra06:37
*** apetrich has joined #openstack-infra06:38
*** jtomasek has joined #openstack-infra06:38
*** wolverineav has joined #openstack-infra06:42
*** wolverineav has quit IRC06:47
*** bgmccollum has quit IRC06:53
*** wolverineav has joined #openstack-infra06:56
*** kopecmartin|off is now known as kopecmartin07:02
*** slaweq has joined #openstack-infra07:11
*** markvoelker has joined #openstack-infra07:11
*** jamesmcarthur has joined #openstack-infra07:11
*** wolverineav has quit IRC07:12
*** bgmccollum has joined #openstack-infra07:14
*** jamesmcarthur has quit IRC07:15
*** e0ne has joined #openstack-infra07:23
*** ricolin has quit IRC07:24
*** rcernin has quit IRC07:25
*** quiquell|rover is now known as quique|rover|r--07:26
*** aojea has joined #openstack-infra07:29
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission  https://review.openstack.org/63768207:32
*** takamatsu has joined #openstack-infra07:39
*** ociuhandu has joined #openstack-infra07:40
openstackgerritSorin Sbarnea proposed openstack-infra/project-config master: Add tripleo-ci members as irc ops in oooq  https://review.openstack.org/63443807:42
*** markvoelker has quit IRC07:43
*** ociuhandu has quit IRC07:47
*** ccamacho has joined #openstack-infra07:47
*** jamesmcarthur has joined #openstack-infra07:48
*** jamesmcarthur has quit IRC07:53
*** rpittau has joined #openstack-infra07:57
*** udesale has quit IRC07:58
*** ginopc has joined #openstack-infra08:01
*** udesale has joined #openstack-infra08:03
*** wolverineav has joined #openstack-infra08:06
*** ykarel is now known as ykarel|lunch08:06
*** udesale has quit IRC08:07
*** udesale has joined #openstack-infra08:08
*** rascasoft has joined #openstack-infra08:10
*** pcaruana has joined #openstack-infra08:11
*** tkajinam has quit IRC08:12
*** e0ne has quit IRC08:13
*** pgaxatte has joined #openstack-infra08:14
*** shardy has joined #openstack-infra08:15
*** shardy has quit IRC08:15
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission  https://review.openstack.org/63768208:19
*** bhavikdbavishi has quit IRC08:23
*** sdake has joined #openstack-infra08:27
*** shardy has joined #openstack-infra08:31
*** evrardjp_ is now known as evrardjp08:31
*** jtomasek has quit IRC08:36
*** sdake has quit IRC08:37
*** electrofelix has joined #openstack-infra08:38
*** wolverineav has quit IRC08:39
*** markvoelker has joined #openstack-infra08:40
*** gfidente has joined #openstack-infra08:41
*** tosky has joined #openstack-infra08:42
*** sdake has joined #openstack-infra08:45
*** quique|rover|r-- is now known as quiquell|rover08:46
*** jamesmcarthur has joined #openstack-infra08:50
*** jpich has joined #openstack-infra08:52
*** ykarel|lunch is now known as ykarel08:54
*** jamesmcarthur has quit IRC08:55
*** bhavikdbavishi has joined #openstack-infra08:57
*** priteau has joined #openstack-infra08:59
*** jpena|off is now known as jpena08:59
*** ociuhandu has joined #openstack-infra09:02
openstackgerritzhongshengping proposed openstack/os-testr master: add python 3.7 unit test job  https://review.openstack.org/63774909:06
openstackgerritzhongshengping proposed openstack/os-performance-tools master: add python 3.7 unit test job  https://review.openstack.org/63775909:06
*** panda|off is now known as panda09:11
*** markvoelker has quit IRC09:14
*** ociuhandu has quit IRC09:23
*** yamamoto has joined #openstack-infra09:23
*** sdake has quit IRC09:25
*** sdake has joined #openstack-infra09:28
*** ociuhandu has joined #openstack-infra09:31
*** slaweq has quit IRC09:31
*** ccamacho has quit IRC09:31
*** slaweq has joined #openstack-infra09:32
*** dtantsur|afk is now known as dtantsur09:34
*** ociuhandu has quit IRC09:35
*** wolverineav has joined #openstack-infra09:37
*** derekh has joined #openstack-infra09:42
*** sdake has quit IRC09:49
*** jamesmcarthur has joined #openstack-infra09:51
*** e0ne has joined #openstack-infra09:53
*** jamesmcarthur has quit IRC09:55
*** ociuhandu has joined #openstack-infra09:56
*** roman_g has joined #openstack-infra09:56
*** ociuhandu has quit IRC09:57
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project  https://review.openstack.org/63666209:57
*** ociuhandu has joined #openstack-infra09:58
*** xek has joined #openstack-infra10:01
openstackgerritMerged openstack/diskimage-builder master: Keep git after ironic-agent post  https://review.openstack.org/63716210:08
*** wolverineav has quit IRC10:10
*** markvoelker has joined #openstack-infra10:11
openstackgerritJan Kundrát proposed openstack-infra/git-review master: Support usernames that contain '@' and ssh Git URLs  https://review.openstack.org/42870010:11
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Show animated progress bar in preparation phase  https://review.openstack.org/63781010:11
*** sajauddin has joined #openstack-infra10:18
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Log to job output when running Ansible setup  https://review.openstack.org/63781310:24
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add jobs list filter  https://review.openstack.org/63365210:25
*** jamesmcarthur has joined #openstack-infra10:27
*** jamesmcarthur has quit IRC10:32
*** priteau has quit IRC10:41
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project  https://review.openstack.org/63666210:42
*** markvoelker has quit IRC10:43
*** lpetrut has joined #openstack-infra10:46
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: config: add statsd-server config parameter  https://review.openstack.org/53556010:53
*** yamamoto has quit IRC11:00
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project  https://review.openstack.org/63666211:01
*** luizbag has joined #openstack-infra11:01
openstackgerritMerged openstack-infra/bindep master: Fix bindep --brief for arch linux  https://review.openstack.org/63742111:01
*** gfidente has quit IRC11:04
*** yamamoto has joined #openstack-infra11:06
*** wolverineav has joined #openstack-infra11:07
*** udesale has quit IRC11:10
*** yamamoto has quit IRC11:11
*** yamamoto has joined #openstack-infra11:11
*** jamesmcarthur has joined #openstack-infra11:29
*** jtomasek has joined #openstack-infra11:32
*** jamesmcarthur has quit IRC11:33
*** bhavikdbavishi has quit IRC11:36
*** bhavikdbavishi has joined #openstack-infra11:37
*** wolverineav has quit IRC11:39
*** markvoelker has joined #openstack-infra11:40
*** gfidente has joined #openstack-infra11:55
*** xek has quit IRC11:56
*** xek has joined #openstack-infra11:57
*** xek_ has joined #openstack-infra11:59
*** xek has quit IRC11:59
mordredianw: yay! thank you for fixing devstack for nodepool12:06
*** bhavikdbavishi has quit IRC12:08
openstackgerritBrendan proposed openstack-infra/zuul-jobs master: Use zuul_workspace_root variable for Git workspace prep  https://review.openstack.org/63687012:10
*** prometheanfire has quit IRC12:12
*** prometheanfire has joined #openstack-infra12:14
*** markvoelker has quit IRC12:14
*** smarcet has joined #openstack-infra12:14
*** asettle has joined #openstack-infra12:15
*** sajauddin has quit IRC12:20
*** jpena is now known as jpena|lunch12:30
*** rfolco has joined #openstack-infra12:33
*** wolverineav has joined #openstack-infra12:36
*** jcoufal has joined #openstack-infra12:37
*** janki has quit IRC12:41
*** janki has joined #openstack-infra12:44
*** dpawlik has quit IRC12:51
*** janki has quit IRC12:52
*** janki has joined #openstack-infra12:53
*** janki has quit IRC12:54
*** janki has joined #openstack-infra12:54
*** udesale has joined #openstack-infra12:55
*** bhavikdbavishi has joined #openstack-infra12:57
*** weshay_PTO is now known as weshay12:58
*** xek_ has quit IRC13:01
*** xek_ has joined #openstack-infra13:02
*** jamesmcarthur has joined #openstack-infra13:06
*** wolverineav has quit IRC13:09
*** trown|outtypewww is now known as trown13:10
*** markvoelker has joined #openstack-infra13:11
*** jamesmcarthur has quit IRC13:11
*** jamesmcarthur has joined #openstack-infra13:12
*** bhavikdbavishi has quit IRC13:14
*** dpawlik has joined #openstack-infra13:23
*** rh-jelabarre has joined #openstack-infra13:24
*** sdake has joined #openstack-infra13:25
*** kgiusti has joined #openstack-infra13:27
*** jamesmcarthur has quit IRC13:32
*** agopi has quit IRC13:33
*** rlandy has joined #openstack-infra13:34
*** jpena|lunch is now known as jpena13:35
roman_gianw: RTD folks have just merged the patch for hooks to work w/o CSRF token. Look forward to successful builds http://zuul.openstack.org/builds?job_name=trigger-readthedocs-webhook13:40
openstackgerritNir Magnezi proposed openstack/diskimage-builder master: [wip] rhel8 beta support  https://review.openstack.org/62313713:40
*** zul has joined #openstack-infra13:41
*** markvoelker has quit IRC13:43
mordredroman_g: \o/13:44
*** kgiusti has quit IRC13:44
*** hamzy has joined #openstack-infra13:44
*** kgiusti has joined #openstack-infra13:45
*** chkumar|ruck is now known as chandankumar13:45
roman_gmordred: all it took was to /j #readthedocs and politely ask them to review the patch they have wrote.13:47
roman_gVery nice guys.13:47
*** jamesmcarthur has joined #openstack-infra13:49
*** amorin has quit IRC13:50
*** dave-mccowan has joined #openstack-infra13:50
*** janki has quit IRC13:54
*** eharney has joined #openstack-infra13:55
*** jamesmcarthur has quit IRC13:56
*** ykarel is now known as ykarel|away14:02
*** sthussey has joined #openstack-infra14:03
*** sdake has quit IRC14:05
*** ccamacho has joined #openstack-infra14:05
*** wolverineav has joined #openstack-infra14:07
*** rfolco has quit IRC14:07
*** rfolco has joined #openstack-infra14:08
*** yamamoto has quit IRC14:10
*** boden has joined #openstack-infra14:11
*** ykarel|away has quit IRC14:13
*** sdake has joined #openstack-infra14:13
*** jtomasek has quit IRC14:17
*** ccamacho has quit IRC14:18
*** sdake has quit IRC14:19
*** ccamacho has joined #openstack-infra14:22
*** yamamoto has joined #openstack-infra14:24
*** zul has quit IRC14:25
*** mriedem has joined #openstack-infra14:25
*** yamamoto has quit IRC14:26
*** zul has joined #openstack-infra14:26
*** yamamoto has joined #openstack-infra14:27
*** xek__ has joined #openstack-infra14:28
*** xek_ has quit IRC14:31
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project  https://review.openstack.org/63666214:32
*** sdake has joined #openstack-infra14:38
*** wolverineav has quit IRC14:39
*** mriedem has quit IRC14:40
*** mriedem has joined #openstack-infra14:40
*** markvoelker has joined #openstack-infra14:41
*** munimeha1 has joined #openstack-infra14:46
*** ykarel|away has joined #openstack-infra14:49
*** kjackal has quit IRC14:51
*** ykarel|away is now known as ykarel14:52
*** sdake has quit IRC14:53
*** udesale has quit IRC14:53
*** kjackal has joined #openstack-infra14:57
openstackgerritSean McGinnis proposed openstack-infra/openstack-zuul-jobs master: Add per-cycle Python job templates  https://review.openstack.org/63786614:58
*** sdake has joined #openstack-infra14:59
*** whoami-rajat has quit IRC15:01
bodenhi, hoping for a little help with the legacy-tempest-dsvm-networking-bagpipe and legacy-tempest-dsvm-networking-bgpvpn-bagpipe jobs15:14
bodenbased on the logs, they appear to be picking up an older version of networking-sfc; a required project and are failing due to a bad import in sfc http://logs.openstack.org/62/636962/2/check/legacy-tempest-dsvm-networking-bagpipe/0fabd66/logs/devstacklog.txt.gz#_2019-02-15_14_44_19_05615:14
*** markvoelker has quit IRC15:14
bodenrecent versions of sfc don't have this import https://github.com/openstack/networking-sfc/blob/master/networking_sfc/db/flowclassifier_db.py#L3215:14
bodenbased on the inventory of that failing job it seems sfc is coming from source, but I don't see how that can be http://logs.openstack.org/62/636962/2/check/legacy-tempest-dsvm-networking-bagpipe/0fabd66/zuul-info/inventory.yaml15:14
*** rpittau has quit IRC15:14
fungijust because zuul is preparing and making that repository state available, that doesn't necessarily mean devstack is installing from there15:15
bodenfungi: is that directed at me?15:16
fungiues15:16
fungier, yes15:16
bodenfungi so you are saying its an issue with the devstack setup I assume? I will take it up with the bagpipe team15:17
fungilooks that way: http://logs.openstack.org/62/636962/2/check/legacy-tempest-dsvm-networking-bagpipe/0fabd66/logs/devstacklog.txt.gz#_2019-02-15_14_37_30_84415:17
*** gfidente has quit IRC15:17
fungibut you might want to get input from the devstack maintainers in #openstack-qa15:17
bodenfungi ok thanks... I'm not up to speed on the devstack aspect (yet)15:18
*** whoami-rajat has joined #openstack-infra15:18
fungii have a feeling this would be easier to troubleshoot if you replaced that legacy job with a modern devstack job15:18
bodenfungi ack... I will suggest that to the bagpipe team.. I was just trying to fix their gate (quickly)15:19
fungithe new-style devstack jobs can infer a lot of this sort of thing from zuul. legacy jobs are straight ports of the scripting from our old jjb configs and as such require a lot more direct configuring15:19
bodenfungi I see15:20
fungii get the impression the qa team is planning to cease supporting "legacy" devstack jobs as of the stein release, though they can confirm what the situation is there15:21
*** gfidente has joined #openstack-infra15:21
fungii think it's tied in with the switch from ubuntu trusty to xenial for devstack/tempest jobs15:22
openstackgerritMerged openstack-infra/nodepool master: Properly handle TaskManagerStopped exception  https://review.openstack.org/63639315:26
*** sdake has quit IRC15:28
*** smarcet has quit IRC15:33
openstackgerritMiguel Lavalle proposed openstack-infra/irc-meetings master: Propose a new time for L3 subteam meeting  https://review.openstack.org/63790015:34
*** sdake has joined #openstack-infra15:35
*** wolverineav has joined #openstack-infra15:37
*** sreejithp has joined #openstack-infra15:37
dmsimardI'm trying to understand why the "python3-virtualenv" package is not found on our Fedora 29 images: http://logs.openstack.org/39/635339/1/check/ara-server-role-integration-fedora/532762b/job-output.txt.gz#_2019-02-14_16_37_19_87576315:38
dmsimardThe package is definitely there: http://mirror.regionone.limestone.openstack.org/fedora/releases/29/Everything/x86_64/os/Packages/p/python3-virtualenv-16.0.0-5.fc29.noarch.rpm15:38
dmsimardIt's even installed already, probably supplied by DIB's pip/virtualenv element15:38
ykareldmsimard, possibly because of exclude in /etc/dnf.conf15:40
dmsimardykarel: oh wow, thanks15:41
ykareldmsimard, last time i saw dib element installing pip/virtualenv set the exclude for it15:41
dmsimardykarel: the virtualenv package is indeed excluded in /etc/dnf/dnf.conf15:41
ykareldmsimard, yup15:41
fungii guess that's so as to use the version from pypi?15:41
fungiand not accidentally overwrite it with an rpm?15:41
dmsimardfungi: I would think not because the rpm package for pip and virtualenv are installed15:41
dmsimardhttp://paste.openstack.org/show/745372/15:42
fungipossible that's cruft from when we installed them from pypi i suppose15:42
dmsimardThis problem has been a headache because my role attempts to install virtualenv and it can't find it :(15:42
fungiout of curiosity, why not use python3's builtin venv module instead?15:43
dmsimardfungi: ideally I'd use the ansible pip module and call it a day but it's been pretty hard to get things to work consistently across distros and versions of ansible15:44
openstackgerritMerged openstack-infra/zuul-jobs master: install-kubernetes: fix minikube config permission  https://review.openstack.org/63768215:45
*** lpetrut has quit IRC15:45
fungilooks like it does try to use rpms instead of source on fedora, unless i'm misreading the script: https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n2715:45
dmsimardthe ansible pip module supports specifying a "virtualenv_python" (which would be "python3") or a "virtualenv_command" (which would be something like /usr/bin/python3 -m venv)15:46
dmsimardI had the weird wheels issue I brought up a while back when using python3 -m venv15:47
dmsimardBeen wasting a lot of time on what feels like a trivial issue :(15:49
fungiaha, so that element first installs python3-virtualenv from a distro package, then does a `pip3 install virtualenv` over top of it, and adds the exclusion to dnf.conf to prevent it from being subsequently re-overwritten by a later distro package manager command15:50
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Ignore files at timer trigger  https://review.openstack.org/63791615:51
fungihttps://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n2715:51
fungier, i meant https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n14915:51
dmsimardThat's not necessarily bad but very opinionated15:52
*** sdake has quit IRC15:52
fungiyeah, i personally think we should stop installing pip and virtualenv on all our images and expect jobs to install what they need instead15:52
dmsimardperhaps a zuul-jobs role ?15:53
fungithough we do i think still preinstall a couple of tools into the images using pip and virtualenv, which would explain why we haven't excised them just yet15:53
openstackgerritClark Boylan proposed openstack-infra/zuul master: Rename project to project_name in getPullBySha  https://review.openstack.org/63721815:55
openstackgerritClark Boylan proposed openstack-infra/zuul master: Test GithubShaCache  https://review.openstack.org/63722815:55
openstackgerritClark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache  https://review.openstack.org/63761515:55
corvusclarkb, mordred: are ya'll around to chat about this here gitea thing?15:56
* fungi is around as well, though has a conference call starting in 2 minutes so will be multi-tasking15:57
*** smarcet has joined #openstack-infra15:58
*** sdake has joined #openstack-infra15:59
*** e0ne has quit IRC16:00
*** ianychoi has quit IRC16:00
*** jamesmcarthur has joined #openstack-infra16:01
clarkbI'll be in that same conference call16:01
clarkbbut can probably follow along16:02
*** sdake has quit IRC16:03
*** e0ne has joined #openstack-infra16:03
mordredclarkb, corvus: I am around - but will be around for a while, so also happy to wait a bit if needed16:03
openstackgerritClark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache  https://review.openstack.org/63761516:04
corvusclarkb, fungi: would you rather wait a bit, or get started now?16:05
corvuswe could start at 1700 if that's better16:06
clarkbwaiting until 1700 would be easiest for me16:06
*** smarcet has quit IRC16:06
corvusmordred, clarkb, fungi: let's gather in #opendev at 1700 and get into the weeds on gitea/k8s16:06
fungi1700z in #opendev wfm, thought i might also be holding a sandwich and typing one-handed ;)16:07
*** wolverineav has quit IRC16:10
mordredcorvus: ++16:10
*** ginopc has quit IRC16:11
*** kjackal has quit IRC16:11
corvuscool, in the mean time, i'm going to try to figure out why we got no sql result for my registry test job16:11
*** markvoelker has joined #openstack-infra16:11
*** ianychoi has joined #openstack-infra16:12
openstackgerritJan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver  https://review.openstack.org/53555616:16
*** ginopc has joined #openstack-infra16:17
*** kjackal has joined #openstack-infra16:23
*** smarcet has joined #openstack-infra16:25
corvusi restarted zuul-web to pick up a bugfix that recently landed16:25
corvusoy.  it has not landed.16:26
*** gfidente has quit IRC16:28
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Fix typo in build api endpoint  https://review.openstack.org/63622716:28
*** gfidente has joined #openstack-infra16:29
*** yamamoto has quit IRC16:31
*** yamamoto has joined #openstack-infra16:33
*** yamamoto has quit IRC16:33
*** ginopc has quit IRC16:33
*** yamamoto has joined #openstack-infra16:33
*** yamamoto has quit IRC16:33
*** ykarel is now known as ykarel|away16:37
*** ginopc has joined #openstack-infra16:37
*** quiquell|rover is now known as quiquell|off16:38
*** rh-jelabarre has quit IRC16:41
*** markvoelker has quit IRC16:44
openstackgerritJan Kundrát proposed openstack-infra/nodepool master: Implement a Runc driver  https://review.openstack.org/53555616:45
*** dtantsur is now known as dtantsur|afk16:48
*** hamzy has quit IRC16:50
*** pgaxatte has quit IRC16:50
*** rh-jelabarre has joined #openstack-infra16:52
*** mattw4 has joined #openstack-infra16:57
*** pcaruana has quit IRC16:57
*** mattw4 has quit IRC16:59
* clarkb is here and ready to talk gitea17:00
*** wolverineav has joined #openstack-infra17:01
fungii thought we were doing that in #opendev17:03
corvusyeah, let's go over there17:03
*** ginux has joined #openstack-infra17:06
*** ginopc has quit IRC17:07
*** ginux is now known as ginopc17:07
*** yamamoto has joined #openstack-infra17:10
*** jpich has quit IRC17:11
*** chandankumar is now known as kmrchdn17:12
*** aojea has quit IRC17:16
*** yamamoto has quit IRC17:16
*** lucasagomes is now known as lucas-afk17:20
*** lucas-afk is now known as lucasagomes17:20
*** Vadmacs has joined #openstack-infra17:24
*** lpetrut has joined #openstack-infra17:27
*** jcoufal has quit IRC17:30
openstackgerritMerged openstack-infra/zuul master: Fix typo in build api endpoint  https://review.openstack.org/63622717:30
openstackgerritClark Boylan proposed openstack-infra/zuul master: Switch to LRU based sha to PR cache  https://review.openstack.org/63761517:32
*** ociuhandu_ has joined #openstack-infra17:33
*** wolverineav has quit IRC17:34
*** e0ne has quit IRC17:35
*** jcoufal has joined #openstack-infra17:35
*** trown is now known as trown|lunch17:36
*** ociuhandu has quit IRC17:36
*** ykarel|away has quit IRC17:37
*** eharney has quit IRC17:37
*** smarcet has quit IRC17:37
*** ociuhandu_ has quit IRC17:38
*** psachin has quit IRC17:39
*** lpetrut has quit IRC17:40
*** markvoelker has joined #openstack-infra17:41
*** agopi has joined #openstack-infra17:41
openstackgerritMerged openstack-infra/zuul master: Don't request PR issue data  https://review.openstack.org/63672817:42
*** smarcet has joined #openstack-infra17:43
*** gfidente has quit IRC17:46
*** ccamacho has quit IRC17:46
*** ginopc has quit IRC17:48
roman_gmordred: quick win was not that quick. RTD has delayed acceptance of the patch set, doing some investigation https://github.com/rtfd/readthedocs.org/pull/5009 Cc: ianw clarkb17:48
*** ociuhandu has joined #openstack-infra17:48
*** ociuhandu has quit IRC17:52
*** derekh has quit IRC17:53
*** kopecmartin is now known as kopecmartin|off17:53
openstackgerritClark Boylan proposed openstack-infra/zuul master: Clarify project vs repository in getPullBySha  https://review.openstack.org/63795617:54
fungiinfra-root: when a couple of you have a moment, can you review https://review.openstack.org/637633 to add the replacement xenial server for production openstackid to our inventory?18:03
*** ociuhandu has joined #openstack-infra18:04
*** tjgresha has left #openstack-infra18:04
*** ociuhandu has quit IRC18:08
*** ijw has joined #openstack-infra18:08
*** markvoelker has quit IRC18:14
*** sreejithp has quit IRC18:14
*** sreejithp has joined #openstack-infra18:14
*** hamzy has joined #openstack-infra18:17
*** jamesmcarthur has quit IRC18:17
clarkbcorvus: couple of comments on https://review.openstack.org/#/c/637330/5 to go along with fungi's I don't think any are fatal but probably worth looking at18:18
*** jamesmcarthur has joined #openstack-infra18:18
openstackgerritMerged openstack-infra/zuul master: Add dockerized test setup  https://review.openstack.org/63642418:21
*** jpena is now known as jpena|off18:22
*** kmrchdn is now known as chandankumar18:26
*** mattw4 has joined #openstack-infra18:29
*** ociuhandu has joined #openstack-infra18:30
corvusclarkb: about your 45m docker image comment -- are you saying that every time docker-compose runs, it checks if the image is up to date, and if it isn't restarts the container?18:31
corvusclarkb: i've never tested that when the containers are already running18:31
clarkbcorvus: yes I think so based on my local running of docker compose for next cloud18:34
clarkbcorvus: I think we can sort that out once it is up and running, but I want to say it will auto do that update as it is how I do upgrades locally18:34
*** smarcet has quit IRC18:39
*** lpetrut has joined #openstack-infra18:39
*** e0ne has joined #openstack-infra18:41
*** trown|lunch is now known as trown18:42
*** e0ne has quit IRC18:42
*** ccamacho has joined #openstack-infra18:42
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Docs addition on decrypting Zuul secrets  https://review.openstack.org/63796918:43
*** jtomasek has joined #openstack-infra18:45
*** wolverineav has joined #openstack-infra18:50
*** wolverineav has quit IRC18:50
*** wolverineav has joined #openstack-infra18:50
openstackgerritMerged openstack-infra/zuul master: Rename project to project_name in getPullBySha  https://review.openstack.org/63721818:51
openstackgerritMerged openstack-infra/zuul master: Test GithubShaCache  https://review.openstack.org/63722818:51
openstackgerritMerged openstack-infra/zuul master: Switch to LRU based sha to PR cache  https://review.openstack.org/63761518:51
*** jamesmcarthur has quit IRC18:53
*** e0ne has joined #openstack-infra18:54
openstackgerritMerged openstack-infra/zuul master: Clarify project vs repository in getPullBySha  https://review.openstack.org/63795618:55
clarkbfriendly reminder, we'll have our weekly team meeting in #openstack-meeting in about 5 minutes18:57
*** electrofelix has quit IRC19:03
*** xek_ has joined #openstack-infra19:07
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Remove reference to localhost in zuul_return docs  https://review.openstack.org/63797419:09
*** xek__ has quit IRC19:10
*** markvoelker has joined #openstack-infra19:11
*** ijw has quit IRC19:15
*** xek_ has quit IRC19:16
*** panda is now known as panda|off19:17
*** lpetrut has quit IRC19:18
openstackgerritMerged openstack-infra/system-config master: Add new openstackid01 host to inventory and cacti  https://review.openstack.org/63763319:22
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Switch artifact return back to list  https://review.openstack.org/63797919:23
*** jamesmcarthur has joined #openstack-infra19:24
*** smarcet has joined #openstack-infra19:24
*** sdake has joined #openstack-infra19:33
*** eharney has joined #openstack-infra19:34
*** dougwig has joined #openstack-infra19:38
*** e0ne has quit IRC19:39
*** e0ne has joined #openstack-infra19:39
*** diablo_rojo has joined #openstack-infra19:42
*** markvoelker has quit IRC19:44
*** e0ne has quit IRC19:45
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Have gitea sshd listen on 222  https://review.openstack.org/63733319:46
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Use host networking for gitea  https://review.openstack.org/63733419:46
*** sdake has quit IRC19:52
*** wolverineav has quit IRC19:53
*** ekultails has quit IRC19:54
*** wolverineav has joined #openstack-infra19:54
*** sdake has joined #openstack-infra19:55
*** sdake has quit IRC19:58
*** wolverineav has quit IRC20:00
openstackgerritMerged openstack-infra/project-config master: Add cinderlib project  https://review.openstack.org/63761320:00
*** openstackgerrit has quit IRC20:09
*** luizbag has quit IRC20:11
fungiyeah, so to follow up from the discussion in the meeting, as of a couple years ago when the topic came up on debian-devel there was no official solution to configure the gpg-agent behavior we want (automatically terminating when the invoking gpg2 process itself terminates) https://lists.debian.org/debian-devel/2016/08/msg00079.html20:12
*** ekultails has joined #openstack-infra20:13
prometheanfirefungi: yarp, I asked gpg upstream about this too (back when it originally came up)20:13
fungii guess that leaves us with wrapper script, funky command lines or continuing to snipe stale agent daemons as our options20:13
prometheanfirealias?20:14
fungimaybe, though it would need to alias the emacs command in that case20:24
fungioh, perhaps a custom elisp script which gets used instead of the one emacs normally calls on opening encrypted files?20:24
*** openstackgerrit has joined #openstack-infra20:27
openstackgerritMerged openstack-infra/system-config master: Deploy gitea with docker-compose  https://review.openstack.org/63733020:27
*** efried has quit IRC20:29
*** efried has joined #openstack-infra20:29
*** emccormick has joined #openstack-infra20:31
*** wolverineav has joined #openstack-infra20:31
*** wolverineav has quit IRC20:36
*** e0ne has joined #openstack-infra20:37
openstackgerritJan Kundrát proposed openstack-infra/zuul master: More parallelism for git clones and checkouts  https://review.openstack.org/63799620:37
clarkbI tend to not like aliases beacuse they stop working when you don't load the right env, but running a consistent command/script would20:38
*** Vadmacs has quit IRC20:38
clarkbre elisp I've never had to write emacs plugins so I have no idea if that is a sane thing to do20:38
*** markvoelker has joined #openstack-infra20:41
fungiluckily one is already written i expect, and we just need to for it to alter the command line it uses to invoke gpg2?20:43
fungier, need to fork it to20:46
*** smarcet has quit IRC20:49
*** wolverineav has joined #openstack-infra20:50
*** wolverineav has quit IRC20:53
*** smarcet has joined #openstack-infra20:53
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Combine zuul.artifacts lists in zuul_return  https://review.openstack.org/63800520:55
*** wolverineav has joined #openstack-infra20:55
*** wolverineav has quit IRC20:55
*** wolverineav has joined #openstack-infra20:56
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Use list form of zuul artifact return  https://review.openstack.org/63800721:00
corvusclarkb, fungi, mordred: ^ we should be able to go ahead and merge that so i can continue my test even without the zuul change (since my test only involves one image)21:00
clarkbI've +2'd both changes21:01
ianwprometheanfire: hrm, we added all this size tracking stuff ... which i don't think is making it to graphite21:01
clarkbianw: iirc the raeson we stopped with the du was it listed every file in the image which took a long time and filled logs21:02
clarkbianw: a summary of the first two dir levels might be a good compromise instead if we want to go bac kto doing that21:02
ianwclarkb: oh, well that stopped and it only shows files over 10mb by default ...21:03
clarkbah21:03
prometheanfireianw: :D21:03
ianwbut i'm talking about stats, i'm now wondering if we ever let the builders actually talk to graphite21:03
clarkbanother good compromise21:03
*** eharney has quit IRC21:03
ianwyeah, the answer seems to be no, which is a shame21:05
fungicatching up21:05
openstackgerritIan Wienand proposed openstack-infra/system-config master: Allow nb*.openstack.org to talk to graphite  https://review.openstack.org/63801121:07
*** yamamoto has joined #openstack-infra21:13
*** markvoelker has quit IRC21:13
*** kgiusti has left #openstack-infra21:14
*** emccormick has quit IRC21:17
ianwclarkb: ok, here's where we turned off reporting in logs ... https://review.openstack.org/#/c/357393/21:17
ianwnot sure what was thinking iwth flag in nodepool ... but it was a different era21:18
*** yamamoto has quit IRC21:18
*** whoami-rajat has quit IRC21:21
ianwand here's relevant discussion : http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-18.log.html#t2016-08-18T20:49:5121:25
*** wolverineav has quit IRC21:27
*** wolverineav has joined #openstack-infra21:28
*** e0ne has quit IRC21:28
*** xek has joined #openstack-infra21:31
*** hamzy has quit IRC21:41
*** wolverineav has quit IRC21:43
*** wolverineav has joined #openstack-infra21:47
openstackgerritMerged openstack-infra/zuul master: Switch artifact return back to list  https://review.openstack.org/63797921:49
openstackgerritMerged openstack-infra/zuul-jobs master: Use list form of zuul artifact return  https://review.openstack.org/63800721:52
fungismarcet: looks like openstackid01.openstack.org has the expected apache configuration for serving the openstackid.org site as of the puppet update a few minutes ago21:54
smarcetfungi: yes thx sire i was testing it21:54
smarceti think that perhaps tomorrow we could do the dns change21:54
smarcetlooks good to me21:54
fungiexcellent, i can make the dns update whenever you like tomorrow21:55
openstackgerritMerged openstack-infra/zuul master: Combine zuul.artifacts lists in zuul_return  https://review.openstack.org/63800521:56
*** jtomasek has quit IRC21:57
clarkbnice21:58
*** sdake has joined #openstack-infra21:58
*** dave-mccowan has quit IRC22:07
openstackgerritClark Boylan proposed openstack-infra/infra-specs master: Update priority effort Gerrit topics  https://review.openstack.org/63802422:07
clarkbcorvus: ^ I think that gets us a working rendered url while not sacrificing human readability of that (though mostly that is because chrome stopped converting to an escaped version for me so this was easier than pulling out urlparse)22:08
*** sdake has quit IRC22:09
*** markvoelker has joined #openstack-infra22:10
prometheanfirehttps://paste.pound-python.org/show/tmBPCdFpfPKG4lNGmBr6/ are the large dirs, the os stuff looks ok (image size22:13
ianwroman_g: yay!  green again -> http://zuul.openstack.org/builds?job_name=trigger-readthedocs-webhook22:13
*** jcoufal has quit IRC22:14
clarkbprometheanfire: daisycloud-core hasn't been updated in months so the biggest repo isn't newly biggest22:15
ianwpromethanfire: i think those fuel directories have been there forever and have always been big22:15
clarkbprometheanfire: I wonder if this is a compression thing and the actual image size hasn't changed much22:15
clarkbwhat is the true image size? iirc 18-20GB was normal22:16
smarcetfungi: thx u i will let u know22:16
prometheanfireuncompressed? checking22:16
prometheanfire-rw-r--r-- 1 qemu qemu  16G Feb 19 21:31 f22dd92e145ff3dbf6b33ddd0b037901d6d2a66922:16
prometheanfire-rw-r--r-- 1 root root 8.9G Feb 19 21:22 f22dd92e145ff3dbf6b33ddd0b037901d6d2a669.qcow222:16
*** xek_ has joined #openstack-infra22:16
fungilooks about right on the uncompressed end22:17
*** rlandy is now known as rlandy|brb22:18
*** xek has quit IRC22:18
clarkbinfra-root it would probably be a good idea for us to restart the zuul scheduler to pick up the github caching improvements22:19
clarkb(as well as reduction in github api calls overall)22:19
clarkbthe gate isn't too busy right now22:20
mordred++22:20
clarkbcorvus: any reason to not do a scheduler only restart in the near future?22:20
clarkb(like maybe picking up zuul_return/artifacts fixes that aren't quite in yet)22:20
*** xek_ has quit IRC22:21
fungii think those are in now22:21
fungithough i thought we were looking at a new zuul release rsn, in which case i assume we'll want to restart everything just to be sure we're running latest and having no problems22:21
clarkbya if we wnat to do a release soon probably do want a full restart22:22
fungibut no idea quite how soon rsn is22:22
*** rcernin has joined #openstack-infra22:26
*** rlandy|brb is now known as rlandy22:27
*** rh-jelabarre has quit IRC22:28
*** smarcet has quit IRC22:29
corvusclarkb: we should restart all of zuul at 6bc25035dd8a41c0522fbe43d149303c426cbc5a22:30
corvusclarkb: if you have time to take care of that, that'd be great22:30
fungii should be able to help between now and when tc nominations end in another 1.25 hours22:31
clarkbI can likely drive that if others are around to remind me where I've done thinsg wrong22:31
clarkblet me check what is currently installed22:31
fungihappy to22:31
corvusyeah, i'm around, i just have too many windows open right now :)22:31
corvusthat's sort of a metaphor, but sort of literal22:32
*** agopi has quit IRC22:32
clarkbok 6bc250 is current zuul HEAD22:33
clarkbnow to see what is installed on zuul cluster nodes22:33
clarkboh ze08 is still disabled in puppet iirc. Is it ok if one executor is on the wrong thing? I'm guessing not with the zuul return work22:34
clarkbcan probably just out of band bring that one up to speed22:34
fungiis there any reason to keep it disabled now?22:35
clarkbfungi: it would be nice to continue to collect memory usage data on it, though that isn't super critical22:35
clarkbI think initial results are that jemalloc uses less memory but not enough less that we don't swap22:36
corvusmaybe after this we should go ahead and switch them all to jemalloc?22:36
fungioh, right, i forgot this was the jemalloc experiment22:37
clarkbI'd be happy with that. It doesn't appear to have a negative impact, but does appear to have a small positive impact22:37
fungiwfm22:37
clarkbI've just confirmed that all zuul and zm servers are up to date and all but ze08 is up to date on the executors22:37
*** smarcet has joined #openstack-infra22:37
clarkbshould I go ahead and manually update zuul install on ze08?22:37
clarkbthen as a followup later we'll jemalloc across the board? or do we want to incorporate that in this restart?22:38
clarkb(that will likely make this take more than a few minutes)22:38
corvusclarkb: manually updating ze08 now should be fine22:38
corvusjust remember to pip3 install it :)22:38
fungiwhat configuration management changes are needed to make the switch?22:38
corvusfungi: i think a package install and an LD_ env var line in /etc/default22:39
corvusor something like that22:39
corvuswould need to be in puppet-zuul22:39
*** smarcet has quit IRC22:40
fungii guess those were manually added on ze08 and the disable list was used to prevent reverting /etc/default/whatever?22:40
clarkbcorvus: yup export LD_PRELOAD and install the package22:40
clarkbfungi: yes22:40
clarkbupdating ze08 now22:40
fungiyep, i see it in /etc/default/zuul-executor22:41
fungitaking a look to see how to shoehorn it into puppet-zuul now22:41
fungido we want this to be optional?22:42
openstackgerritIan Wienand proposed openstack-infra/project-config master: Enable DIB_SHOW_IMAGE_USAGE for build  https://review.openstack.org/63803122:42
corvusfungi: i think so, cause i think folks who aren't us may still be using that22:43
fungiyeah, i figured22:43
corvuscould *probably* just get away with using the zuulv3 flag though22:43
clarkbze08 is now up to date22:43
fungioh, that's not a terrible idea22:43
corvusso if you're on the puppet-zuul zuulv3 train with us, you get jemalloc.  i don't see a problem with that.22:43
*** markvoelker has quit IRC22:44
clarkbcorvus: ++22:44
clarkbnow for restarting things I can run the zuul_restart.yaml playbook right?22:44
fungiinterestingly, i don't see where templates/zuul.default.erb includes the PIDFILE= line22:44
clarkband I want to save queues before I start executing that22:44
clarkbthen restore them as soon as zuul-scheduler has loaded its config?22:45
fungiis /etc/default/zuul-executor being written some other way?22:45
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea  https://review.openstack.org/63803322:45
corvusclarkb: yes22:46
corvusfungi: hrm.  not that i'm aware of.22:46
clarkbalright going to notify the release team now then I'll save queues and start the playbook22:47
fungioh, hah, we don't use that template on the executors22:49
fungiwe use a literal content string in a file resource22:49
clarkbfungi: do I need to wait for the election change that is in the gate to merge?22:49
clarkbor are we ok to reenqueue that after?22:49
fungiclarkb: if you could that would be great, but if not there's still time to get it merged22:49
clarkb(I don't know how strict about merging vs timelines we are)22:49
clarkbif getting that merged before the deadlien is required I can wait22:50
fungiit's mostly just that people can get confused if we send the end of nominations announcement linking to a page which hasn't been updated for all the candidates yet22:50
fungiit's not a strict requirement though, no22:50
clarkbok I'll wait then, there aren't any other candidates that need to be merged right now are there?22:50
fungiyou could i suppose just tweak the enueuing order when you bring things back up22:50
corvus(i'm going to restart zuul-web because it will let me make a step forward on my testing)22:51
funginope, no other candidate changes pending approval22:51
clarkbfungi: nah it should merge quickly once it gets this last node assigned I'll just wait22:51
corvus(done restarting zuul-web)22:51
fungicool, the jobs for those run pretty quickly anyway22:51
fungithe slowest jobs for that repo are still under 4 minutes22:52
clarkbya just need a xenial node for py36 I think22:52
clarkbalso we may not need to test python36 and 35 on the elections repo :)22:53
fungiand then it will trigger a post job to update the published candidate list22:53
fungiwhich is also fast (and high priority)22:53
clarkbwhile I'd like to just skip the queue I'd rather not have to worry about reenquing that and having everything work so better to let it finish first:)22:54
fungiwfm22:54
fungithanks!22:54
fungiwe're gearing up to send the next phase announcements for the election22:55
*** tkajinam has joined #openstack-infra22:55
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Allow extra data in artifact schema validation  https://review.openstack.org/63803822:59
*** boden has quit IRC22:59
corvusclarkb, fungi: while we're waiting, could you review that?22:59
corvusi don't want to hold the restart for that merging.  so i can either do another scheduler restart later, or if we're feeling lucky, i can manually install that on our scheduler to get it in this restart.23:01
fungion it23:01
clarkbthe election merge is ~3 minute away then post jobs23:02
clarkbmight be best to do a followup restart later?23:02
*** munimeha1 has quit IRC23:02
fungiwe could just approve and enqueue into the gate23:02
corvushrm.  we could probably do that, and a manual update of the repo and install.....23:03
fungior kick.sh23:03
fungiwhich i guess is basically what you said23:03
clarkbup to you, I can wait a few more minutes since waiting on election things anyway23:03
corvusheh, i was about to say "clarkb: your call" :)23:04
corvusi still think it's clarkb's show23:04
clarkbI'd like to not juggle it myself, but if someone else juggles it so I can focus on the restart itself that would be fine23:04
corvusbut i've popped enough things off my stack that i'm ready to help23:04
corvusokay, i'll direct-enqueue it and get it installed for the restart23:04
clarkbok23:05
corvusshould only delay us a few more mins23:05
*** smarcet has joined #openstack-infra23:05
fungisounds good23:05
corvusenqueued23:05
corvusyou wanted to wait for the election post jobs anyway, right?23:06
clarkb`ansible-playbook -f 20 playbooks/zuul_restart.yaml` is the command I'll run from /opt/system-config on bridge once ready23:06
clarkbcorvus: yes23:06
corvusthat's more important actually, since we don't typically re-enqueue those23:06
corvusclarkb: lgtm23:06
*** rascasoft has quit IRC23:07
fungiworks case i could have done an enqueue-ref to run the post jobs for that repo after23:10
*** ccamacho has quit IRC23:13
clarkbfungi: post jobs are done. Not sure if we need an afs release pulse for those to be visible (should happen in a couple minutes if so)23:13
fungiit's on static.o.o so shouldn't23:13
clarkbbut I think we are ready on the election side so just waiting on zuul chagne to merge and then updating of the scheduler23:13
fungihttps://governance.openstack.org/election/ is already updated23:13
clarkbgreat23:13
fungiso all good23:13
*** ccamacho has joined #openstack-infra23:14
corvuseta 15m on the new change23:15
*** kjackal has quit IRC23:15
openstackgerritJeremy Stanley proposed openstack-infra/puppet-zuul master: Use jemalloc on Zuul v3 executors  https://review.openstack.org/63804423:16
fungithere's the change to make jemalloc default on our executors23:16
*** sreejithp has quit IRC23:16
fungisince the executors are zuulv3-only anyway, no new conditional logic was needed23:16
corvusoh good point :)23:16
eanderssonAnyone know a better way of handling this? https://review.openstack.org/#/c/638027/2/tox.ini23:18
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Run an haproxy load balancer for gitea  https://review.openstack.org/63803323:19
eanderssonMitaka upper constraints needs to be applied when the unit-tests are running23:20
fungieandersson: but only for python2.7?23:20
*** ekultails has quit IRC23:20
eanderssonfor all python versions23:20
fungioh, i see, you're reusing it23:21
eanderssonThis did work at somepoint, but broke recently23:21
eanderssonbut not clue how it was supposed to work tbh23:21
clarkbthis is mitaka uppter contstrainst against master of yuor project?23:21
* fungi counts letters of the alphabet23:21
fungiyeah, that's a 3-year-old constraints file. wow23:22
clarkbyou could set requirements as a required project (if not already on those jobs) then override the checkout for requirements to be mitaka-eol then test against that local zuul checkout23:22
eanderssonclarkb, yea no clue why it was setup this way, but they have py27 (mitak), py27-newton etc23:22
fungiso i guess the goal is for master of networking-cisco to be backwards-compatible with older branches of requirements?23:23
eanderssonI am just a good samaritan in this case :p23:23
eanderssonfungi, yea23:23
eanderssone.g. https://review.openstack.org/#/c/638000/23:23
eandersson(see the Zuul tests)23:23
fungilooks like neutron_lib may have ceased to be compatible there?23:26
*** eharney has joined #openstack-infra23:26
clarkbas far as why the mitaka constraints aren't used, it used http://logs.openstack.org/00/638000/1/check/networking-cisco-tox-py27-mitaka/1218c47/job-output.txt.gz#_2019-02-19_20_54_00_995298 the local copy23:28
clarkbwhich is probably cheked out to master?23:28
clarkbthe job might be setting UPPER_CONSTRAINTS_FILE in that case23:28
*** mattw4 has quit IRC23:29
*** ociuhandu has quit IRC23:30
*** ociuhandu_ has joined #openstack-infra23:30
fungiyeah,  http://logs.openstack.org/00/638000/1/check/networking-cisco-tox-py27-mitaka/1218c47/tox/py27-5.log says neutron-lib is at the version in https://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n58 (master)23:31
fungias opposed to https://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=mitaka-eol#n18623:32
eanderssonthe odd thing is that it does initially respect the upper-constraints for mitaka23:32
eanderssonhttp://logs.openstack.org/00/638000/1/check/networking-cisco-tox-py27-mitaka/1218c47/tox/py27-1.log23:33
openstackgerritMerged openstack-infra/zuul master: Allow extra data in artifact schema validation  https://review.openstack.org/63803823:33
fungihttp://zuul.openstack.org/builds?job_name=networking-cisco-tox-py27-mitaka shows the history for that job23:34
*** wolverineav has quit IRC23:34
clarkbcorvus: ^ change has merged23:34
clarkbcorvus: I'm ready to grab queues and restart once you are ready23:35
fungilooks like https://review.openstack.org/629794 succeeded with that job on january 10 and then failed today23:35
fungiunfortunately the job logs from over a month ago have aged out now23:35
*** wolverineav has joined #openstack-infra23:35
corvusclarkb: okay, HEAD is installed on the scheduler, i think we can restart now.23:35
corvus(that change shouldn't affect any other components, so we can skip the install i think23:36
corvusbut if you want, i could install globally real quick23:36
clarkbno I think its fine if it was only the scheduler that needs it23:36
corvusclarkb: cool, then you should be gtg23:37
clarkbI'll grab queues now and then start the restart playbook23:37
clarkbplaybook is running now23:37
clarkbcurrently waiting for zuul-web to stop23:38
corvusclarkb: run ps, and rm /var/run/zuul/web.pid when it's stopped23:38
clarkbdone23:38
clarkbnow it is waiting on executors to stop23:39
*** wolverineav has quit IRC23:40
clarkbscheduler is on its way back according to its log23:40
*** markvoelker has joined #openstack-infra23:41
*** mriedem has quit IRC23:41
clarkbthe sequencing on this is nice. We process the slow scheduler startup along with the slow executor shutdown23:42
corvusyeah, the gap in event reception is very small23:42
corvus(because as soon as the scheduler is back up, it starts queueing events even before it loads its config)23:43
clarkbhalf of the executors have stooped23:43
clarkbok scheduler seems up I can queue pipelines now right?23:45
clarkbthen when executors come up they'll take the work23:45
corvusclarkb: yep23:45
clarkboh and executors are already back23:45
clarkbgate is back in and check is on its way23:46
*** smarcet has quit IRC23:46
clarkbhow is #status log Restarted zuul-scheduler at 5271b592afe708d33fc4b3d08d9a2cc97ae0ddfc and zuul mergers + executors at 6bc25035dd8a41c0522fbe43d149303c426cbc5a.23:49
clarkbI'm not sure if the web reported version will properly capture that split so an explicit log is worthwhile?23:49
clarkbcheck is done being enqueued23:50
fungilgtm23:50
*** smarcet has joined #openstack-infra23:52
clarkb#status log Restarted zuul-scheduler at 5271b592afe708d33fc4b3d08d9a2cc97ae0ddfc and zuul mergers + executors at 6bc25035dd8a41c0522fbe43d149303c426cbc5a.23:53
openstackstatusclarkb: finished logging23:53
clarkbnow someone go get ansible to merge a bunch of code :)23:53
*** mattw4 has joined #openstack-infra23:53
clarkbcorvus: thinking out loud here, zuul may want to document that full cluster restart sequence since it seems to work quite well?23:55
clarkbI guess other zuuls probably aren't as busy with as many repos to load configs from so maybe less of a concern23:55
*** smarcet has quit IRC23:55
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!